Parallel Processing and Applied Mathematics: 7th International Conference, PPAM 2007, Gdansk, Poland, September 9-12, 2007, Revised Selected papers

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Massachusetts Institute of Technology, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany

4967

Roman Wyrzykowski Jack Dongarra Konrad Karczewski Jerzy Wasniewski (Eds.)

Parallel Processing and Applied Mathematics 7th International Conference, PPAM 2007 Gdansk, Poland, September 9-12, 2007 Revised Selected Papers

13

Volume Editors Roman Wyrzykowski Konrad Karczewski Czestochowa University of Technology Institute of Computer and Information Science Dabrowskiego 73 42-200 Czestochowa, Poland E-mail:{roman, xeno}@icis.pcz.pl Jack Dongarra University of Tennessee, Electrical Engineering and Computer Science Department Volunteer Boulevard 1122 37996-3450 Knoxville, TN, USA E-mail: [email protected] Jerzy Wasniewski Technical University of Denmark, Department of Informatics and Mathematical Modelling Richard Petersens Plads, Building 321 2800 Kongens Lyngby, Denmark E-mail: [email protected]

Library of Congress Control Number: 2008927189 CR Subject Classification (1998): D, F.2, G, B.2-3, C.2, J.2 LNCS Sublibrary: SL 1 – Theoretical Computer Science and General Issues ISSN ISBN-10 ISBN-13

0302-9743 3-540-68105-1 Springer Berlin Heidelberg New York 978-3-540-68105-2 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. Springer is a part of Springer Science+Business Media springer.com © Springer-Verlag Berlin Heidelberg 2008 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12270521 06/3180 543210

Preface

This volume comprises the proceedings of the 7th International Conference on Parallel Processing and Applied Mathematics - PPAM 2007, which was held in Gda´ nsk, Poland, September 9–12, 2007. It was organized by the Department of Computer and Information Sciences of the Czestochowa University of Technol ogy, with the help of the TASK Academic Computer Centre in Gda´ nsk. The main organizer was Roman Wyrzykowski. PPAM is a biennial conference. Six previous events have been held in different places in Poland since 1994. The proceedings of the last three conferences have been published by Springer in the Lecture Notes in Computer Science series (Nalecz´  ow, 2001, vol. 2328; Czestochowa, 2003, vol. 3019; Pozna´ n , 2005, vol. 3911).  The PPAM conferences have become an international forum for exchanging ideas between researchers involved in parallel and distributed computing, including theory and applications, as well as applied and computational mathematics. The focus of PPAM 2007 was on software tools which facilitate efficient and convenient utilization of modern computing architectures, as well as grid computing, and large-scale applications. This meeting gathered more than 230 participants from 35 countries. A strict refereeing process resulted in the acceptance of 141 contributed presentations, while approximately 45% of the submissions were rejected. Regular tracks of the conference covered such important fields of parallel/distributed/grid computing and applied mathematics as: – – – – – –

Parallel/distributed architectures and mobile computing Numerical algorithms and parallel numerics Parallel and distributed non-numerical algorithms Environments and tools for parallel/distributed/grid computing Applications of parallel/distributed/grid computing Evolutionary computing, meta-heuristics and neural networks The plenary and invited talks were presented by:

– – – – – – – – – –

David A. Bader from the Georgia Institute of Technology (USA) Ben Bennet from ClearSpeed Technology (UK) Ewa Deelman from the University of Southern California (USA) Jack Dongarra from the University of Tennessee and Oak Ridge National Laboratory (USA) Richard Dracott from Intel (USA) Erik Elmroth from the Ume˚ a University (Sweden) Fabrizio Gagliardi from Microsoft (USA) Angel E. Garcia from the Rensselaer Polytechnic Institute (USA) Fred Gustavson from the IBM T.J. Watson Research Center (USA) Hans-Christian Hoppe from Intel (Germany)

VI

Preface

– Vladik Kreinovich from the University of Texas at El Paso (USA) – Jarek Nieplocha from the Pacific Northwest National Laboratory (USA) – Jennifer Schopf from the Argonne National Laboratory (USA) and Science Institute (UK) – Masha Sosonkina from the Ames Laboratory and Iowa State University (USA) – Boleslaw Szyma´ nski from the Rensselaer Polytechnic Institute (USA) – Jerzy Wa´sniewski from the Technical University of Denmark (Denmark)

Workshops and Minisymposia Important and integral parts of the PPAM 2007 conference were the workshops: – The Second Minisymposium on Novel Data Formats and Algorithms for Dense Linear Algebra Computations organized by Fred Gustavson from the IBM T.J. Watson Research Center (USA), and Jerzy Wa´sniewski from the Technical University of Denmark (Denmark) – Combinatorial Tools for Parallel Sparse Matrix Computations Workshop organized by Laura Grigori from INRIA (France), and Masha Sosonkina from the Ames Laboratory and Iowa State University (USA) – The Third Grid Application and Middleware Workshop - GAMW ’07 organized by Ewa Deelman from the USC Information Sciences Institute (USA), and Norbert Meyer from the Pozna´ n Supercomputing and Networking Center (Poland) – The Third Workshop on Large Scale Computations on Grids - LaSCoG’07 organized by Marcin Paprzycki from SWPS in Warsaw (Poland), Dana Petcu from the Western University of Timisoara (Romania), and Przemyslaw Stpiczy´ nski from Marie Curie-Sklodowska University in Lublin (Poland) – Workshop on Models, Algorithms and Methodologies for Grid-Enabled Computing Environments organized by Giovanni Aloisio from the University of Lecce (Italy), and Giuliano Laccetti from the University of Naples Federico II (Italy) – Workshop on Scheduling for Parallel Computing - SPC’07 organized by Maciej Drozdowski from the Pozna´ n University of Technology (Poland) – The Second Workshop on Language-Based Parallel Programming Models WLPP’07 organized by Ami Marowka from the Shenkar College of Engineering and Design in Ramat-Gan (Israel) – Workshop on Performance Evaluation of Parallel Applications on LargeScale Systems organized by Jan Kwiatkowski, Dariusz Konieczny and Marcin Pawlik from the Wroclaw University of Technology (Poland) – Workshop on Parallel Computational Biology - PBC’2007 organized by David A. Bader from the Georgia Institute of Technology in Atlanta (USA), Denis Trystram from ID-IMAG in Grenoble (France), and Jaroslaw Zola from the Iowa State University (USA) – Workshop on High Performance Computing for Engineering Applications organized by Piotr Doerffer from the Institute of Fluid Machinery in Gda´ nsk (Poland), and Jacek Rokicki from the Warsaw University of Technology (Poland)

Preface

VII

– Minisymposium on Interval Analysis organized by Vladik Kreinovich from the University of Texas at El Paso (USA), Pawel Sewastjanow from the Czes tochowa University of Technology, Bartlomiej J. Kubica from the Warsaw University of Technology (Poland), and Jerzy Wa´sniewski from the Technical University of Denmark (Denmark)

Tutorials The PPAM 2007 meeting began with four half-day tutorials: – Globus: fundamental tools for building your Grid, by Jennifer Schopf from the Argonne National Laboratory (USA) and eScience Institute (UK) – Intel tools for grid programming, by Ralf Ratering from Intel (Germany) – New data structures for the Cell processor, by Fred Gustavson from the the IBM T.J. Watson Research Center (USA), and Jerzy Wa´sniewski from the Technical University of Denmark (Denmark) – Grid computing with GridWay on Globus infrastructures: porting applications using the DRMAA standard, by GridWay Team (Spain)

The New Topics at PPAM 2007 The New Computer Architectures:1 Power consumption, heat dissipation and other physical limitations are pushing the microprocessor industry towards multicore design patterns. Most of the processor manufacturers, such as Intel and AMD, are following more conventional approaches, which consist of homogeneous, symmetric multicores where execution units are replicated on the same die; multiple execution units share some cache level (generally L2 and L3) and the bus to memory. Other manufacturers proposed still homogeneous approaches but with a stronger emphasis on parallelism and hyperthreading. Yet other chip manufacturers started exploring heterogeneous designs where cores have different architectural features. One such example is the Cell Broadband Engine (see2 ,3 , and4 ) developed by STI, a consortium formed by Sony, Toshiba and IBM. The Cell BE has outstanding floating-point computational power, which makes it a considerable candidate for high performance computing systems. IBM shipped the first Cell-based system, the BladeCenter QS20, on September 12th 2006. 1

2

3 4

Taken from the Introduction of the LAPACK Working Note Number 185 of Alfredo Buttari, Jack Dongarra, and Jakub Kurzak; Limitations of the Play Station 3 for High Performance Cluster Computing, UT-CS-07-597, May 2007; ”http://www.netlib.org/ lapack/lawnspdf/lawn185.pdf”. H.P. Hofstee. Power efficient processor architecture and the Cell process or. In Proceedings of the 11th Intel Symposium on High-Performance Computer Architecture, 2005. J.A. Kahle, M.N. Day, H.P. Hofstee, C.R. Johns, T.R. Maeurer, and D. Shippy. Introduction to the Cell multiprocessor. IBM J. Res. & Dev., 49(4/5):589-604, 2005. IBM. Cell Broadband Engine Architecture, Version 1.0, August 2005.

VIII

Preface

This blade was equipped with two Cell processors with a 512 MB memory each and connected in a NUMA configuration; the external connectivity was achieved through a Gigabit and an InfiniBand network interface. Now QS20 blade is replaced with a new QS21 Blade Center with larger memory. Its impressive computational power, coupled with high speed network interfaces, makes it a good candidate for high performance cluster computing. At almost the same period (November 11th 2006), Sony released the Play Station 3 (PS3) gaming console. Even if this console is not meant for high performance computing, it is still equipped with a (stripped down) Cell processor and its price ($500) definitely makes it an attractive solution for building a Cell-based cluster. Four speakers Alfredo Buttari, Jack Dongarra, Fred Gustavson and Adrianto Wirawan spoke, in their lectures, about the new Cell architecture. Also, there was a tutorial on New Data Structures and Cell related work by Jack Dongarra team, and a minisymposium on ”Novel Data Formats and Algorithms for Dense Linear Algebra Computations” both organized by Fred Gustavson and Jerzy Wa´sniewski. This latter material partly featured the Cell architecture as an example of the new Multi-Core / Many Core environment and role that New Data Structures could play for Cell. Ben Bennett, Richard Dracott, and Jerzy Wa´sniewski were also discussing the new Multi-Core / Many Core environment in their lectures. Interval analysis methods: Numerical methods – such as optimization methods, methods for solving systems of equations, etc. – usually produce approximate solutions. Often, there are no guaranteed bounds on the accuracy of these approximate solutions – or there are bounds but these bounds are too wide to be practically useful. In such situations, it is desirable to have verified numerical computations, i.e., computations that would produce verified (provable) accuracy. When a (final or intermediate) approximate result x  of the computation comes with a verified bound Δ, it means that the actual (unknown) value of the estimated quantity belongs to the interval [ x − Δ, x  + Δ]. In view of this fact, verified numerical computing is also known as interval analysis. To address these issues, a minisymposium on interval analysis method was organized by Vladik Kreinovich, Pawel Sewastjanow, Bartlomiej J. Kubica, and Jerzy Wa´sniewski. The intent was to present a state-of-the-art overview on the challenging and dynamic field of verified computing techniques and interval analysis for researchers, experts, and scientists who are current and future users of these techniques. This minisymposium included a tutorial on interval techniques and their application, and several research talks. Some of these talks described new interval (verification) techniques in solving various numerical problems: computing the value of a given special function (Evgueni Petrov), estimating the sum of an infinite series (Boguslaw Bo˙zek, Wieslaw Solak, and Zbigniew Szydelko), solving algebraic (Pawel Sevastjanov and Ludmila Dymowa) and differential equations (Karol Gajda, Malgorzata Jankowska, Andrzej Marciniak, and Barbara Szyszka), and checking properties of the solutions – such as their monotonicity (Iwona Skalna). Two talks discussed decision making under interval uncertainty: Bartlomiej J. Kubica and

Preface

IX

Adam Wo´zniak showed how to compute the Pareto set in a multi-criterial problem, and Van Nam Huynh, Vladik Kreinovich, Yoshiteru Nakamori, and Hung T. Nguyen showed how to efficiently predict human decisions under such uncertainty. Eva Dyllong, Cornelius Grimm, Jorge Fl´ orez, Mateu Sbert, Miguel A. Sainz, and Josep Veh´ı applied interval techniques to problems of computer vision, computer graphics, and computer-aided design and manufacturing. Finally, Piotr Orantek and Antoni John used interval techniques in biomedical engineering: namely, in the analysis of the human pelvic bone.

Acknowledgements The organizers are indebted to PPAM 2007 sponsors, whose support was vital to the success of the conference. The main sponsor was the Intel Corporation. The other sponsors were: Microsoft Corporation, IBM Corporation, Action S.A., and SIAM. We thank to all members of the International Program Committee and additional reviewers for their diligent work in refereeing the submitted papers. Finally, we thank to all of the local organizers from the Czestochowa University of  Technology, TASK Academic Computer Centre in Gda´ nsk, and Academy of Music in Gda´ nsk, who helped us to run the event very smoothly. We are especially indebted to Gra˙zyna Kolakowska, Urszula Kroczewska, L  ukasz Kuczy´ nski, and Marcin Wo´zniak from the Czestochowa University of Technology; to M´scislaw  Nakonieczny, Ewa Politowska and Rafal Tylman from the TASK Academic Computer Centre; and to Henryk Maczka from the Academy of Music in Gda´ nsk. 

PPAM 2009 We hope that this volume will be useful to you. We would like everyone who reads it to feel invited to the next conference, PPAM 2009, which will be held in Wroclaw (Poland), September 2009.

February 2008

Roman Wyrzykowski Jack Dongarra Konrad Karczewski Jerzy Wa´sniewski

Organization

Program Committee Jan Weglarz  Roman Wyrzykowski Boleslaw Szyma´ nski Peter Arbenz Piotr Bala Radim Blaheta Jacek Bla˙zewicz Tadeusz Burczy´ nski Peter Brezany Jerzy Brzezi´ nski Marian Bubak ˇ Raimondas Ciegis Bogdan Chlebus Zbigniew Czech Jack Dongarra Maciej Drozdowski Jacek Gondzio Andrzej Go´sci´ nski Frederic Guinand Marta Fairen Ladislav Hluchy Alexey Kalinov Ayse Kiper Jacek Kitowski Jozef Korbicz Stanislaw Kozielski Dieter Kranzlmueller Henryk Krawczyk Piotr Krzy˙zanowski Jan Kwiatkowski Marco Lapegna Alexey Lastovetsky Aleksandr Legalov Vyacheslav Maksimov Victor E. Malyshkin Tomas Margalef

Pozna´ n University of Technology, Poland Honorary Chair Czestochowa University of Technology, Poland  Chair of Program Committee Rensselaer Polytechnic Institute, USA Vice-Chair of Program Committee ETH, Zurich, Switzerland N. Copernicus University, Poland Institute of Geonics, Czech Academy of Sciences Pozna´ n University of Technology, Poland Silesia University of Technology, Poland University of Vienna, Austria Pozna´ n University of Technology, Poland Institute of Computer Science, AGH, Poland Vilnius Gediminas Tech. University, Lithuania University of Colorado at Denver, USA Silesia University of Technology, Poland University of Tennessee and ORNL, USA Pozna´ n University of Technology, Poland University of Edinburgh, Scotland, UK Deakin University, Australia Universite du Havre, France Univer. Polit. de Catalunya, Barcelona, Spain Slovak Academy of Sciences, Bratislava Institute for System Programming, Russia Middle East Technical University, Turkey Institute of Computer Science, AGH, Poland University of Zielona G´ ora, Poland Silesia University of Technology, Poland Johannes Kepler University Linz, Austria Gda´ nsk University of Technology, Poland University of Warsaw, Poland Wroclaw University of Technology, Poland University of Naples, Italy University College Dublin, Ireland Krasnoyarsk State Technical University, Russia Ural Branch, Russian Academy of Sciences Siberian Branch, Russian Academy of Sciences Universitat Autonoma de Barcelona, Spain

XII

Organization

Ami Marowka Norbert Meyer Jarek Nabrzyski Marcin Paprzycki Dana Petcu Edwige Pissaloux Jacek Rokicki Leszek Rutkowski Jaroslaw Rybicki Franciszek Seredy´ nski Robert Schaefer Norbert Sczygiol Jurij Silc Peter M.A. Sloot Przemyslaw Stpiczy´ nski Maciej Stroi´ nski Domenico Talia Andrei Tchernykh Roman Trobec Denis Trystram Marek Tudruj Pavel Tvrdik Jens Volkert Jerzy Wa´sniewski Bogdan Wiszniewski Krzysztof Zieli´ nski Jianping Zhu

Shenkar College of Eng. and Design, Israel PSNC, Pozna´ n, Poland PSNC, Pozna´ n, Poland SWPS, Warsaw, Poland Western University of Timisoara, Romania Universite de Rouen, France Warsaw University of Technology, Poland Czestochowa University of Technology, Poland  Gda´ nsk University of Technology, Poland Polish Academy of Sciences, Warsaw Institute of Computer Science, AGH, Poland Czestochowa University of Technology, Poland  Jozef Stefan Institute, Slovenia University of Amsterdam, The Netherlands UMCS, Lublin, Poland Supercomp. and Networking, Pozna´ n, Poland University of Calabria, Italy CICESE, Ensenada, Mexico Jozef Stefan Institute, Slovenia ID-IMAG, Grenoble, France Polish Academy of Sciences, Warsaw Czech Technical University, Prague Johannes Kepler University Linz, Austria Technical University of Denmark Gda´ nsk University of Technology, Poland Institute of Computer Science, AGH, Poland University of Akron, USA

Table of Contents

Parallel/Distributed Architectures and Mobile Computing Safety of a Session Guarantees Protocol Using Plausible Clocks . . . . . . . . Jerzy Brzezi´ nski, Michal Kalewski, and Cezary Sobaniec

1

On Checkpoint Overhead in Distributed Systems Providing Session Guarantees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Arkadiusz Danilecki, Anna Kobusi´ nska, and Marek Libuda

11

Performance Evolution and Power Benefits of Cluster System Utilizing Quad-Core and Dual-Core Intel Xeon Processors . . . . . . . . . . . . . . . . . . . . . Pawel Gepner, David L. Fraser, and Michal F. Kowalik

20

Skip Ring Topology in FAST Failure Detection Service . . . . . . . . . . . . . . . Jacek Kobusi´ nski, Filip Gorski, and Stanislaw Stempin

29

Inter-processor Communication Optimization in Dynamically Reconfigurable Embedded Parallel Systems . . . . . . . . . . . . . . . . . . . . . . . . . . Eryk Laskowski and Marek Tudruj

39

An Algorithm to Improve Parallelism in Distributed Systems Using Asynchronous Calls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rouzbeh Maani and Saeed Parsa

49

IEBS Ticketing Protocol as Answer to Synchronization Issue . . . . . . . . . . Barbara Palacz, Tomasz Milos, Lukasz Dutka, and Jacek Kitowski

59

Analysis of Distributed Packet Forwarding Strategies in Ad Hoc Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marcin Seredynski, Pascal Bouvry, and Mieczyslaw A. Klopotek

68

Implementation and Optimization of Dense LU Decomposition on the Stream Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ying Zhang, Tao Tang, Gen Li, and Xuejun Yang

78

Numerical Algorithms and Parallel Numerics An Adaptive Interface for the Efficient Computation of the Discrete Sine Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pedro Alonso, Miguel O. Bernabeu, and Antonio-Manuel Vidal-Maci´ a

89

XIV

Table of Contents

Incomplete WZ Factorization as an Alternative Method of Preconditioning for Solving Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . Beata Bylina and Jaroslaw Bylina

99

A Block-Based Parallel Adaptive Scheme for Solving the 4D Vlasov Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Olivier Hoenen and Eric Violard

108

On Optimal Strategies of Russia’s Behavior on the International Market for Emissions Permits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alexey Kadiyev, Vyacheslav Maksimov, and Valeriy Rozenberg

118

Message-Passing Two Steps Least Square Algorithms for Simultaneous Equations Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jose-Juan L´ opez-Esp´ın and Domingo Gim´enez

127

Parallel Implementation of Cholesky LLT -Algorithm in FPGA-Based Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Oleg Maslennikow, Volodymyr Lepekha, Anatoli Sergiyenko, Adam Tomas, and Roman Wyrzykowski

137

Dimensional Analysis Applied to a Parallel QR Algorithm . . . . . . . . . . . . . Robert W. Numrich

148

Sparse Matrix-Vector Multiplication - Final Solution? . . . . . . . . . . . . . . . . ˇ Ivan Simeˇ cek and Pavel Tvrd´ık

156

Parallel and Distributed Non-numerical Algorithms Petascale Computing for Large-Scale Graph Problems . . . . . . . . . . . . . . . . David A. Bader The Buffered Work-Pool Approach for Search-Tree Based Optimization Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Faisal N. Abu-Khzam, Mohamad A. Rizk, Deema A. Abdallah, and Nagiza F. Samatova Parallel Scatter Search Algorithm for the Flow Shop Sequencing Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wojciech Bo˙zejko and Mieczyslaw Wodecki Theoretical and Practical Issues of Parallel Simulated Annealing . . . . . . . Agnieszka Debudaj-Grabysz and Zbigniew J. Czech Modified R-MVB Tree and BTV Algorithm Used in a Distributed Spatio-temporal Data Warehouse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marcin Gorawski and Michal Gorawski

166

170

180

189

199

Table of Contents

XV

Towards Stream Data Parallel Processing in Spatial Aggregating Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marcin Gorawski and Rafal Malczok

209

On Parallel Generation of Partial Derangements, Derangements and Permutations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zbigniew Kokosi´ nski

219

Parallel Simulated Annealing Algorithm for Graph Coloring Problem . . . ´ eto´ Szymon L  ukasik, Zbigniew Kokosi´ nski, and Grzegorz Swi  n

229

Parallel Algorithm to Find Minimum Vertex Guard Set in a Triangulated Irregular Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Masoud Taghinezhad Omran

239

JaCk-SAT: A New Parallel Scheme to Solve the Satisfiability Problem (SAT) Based on Join-and-Check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Daniel Singer and Anthony Monnet

249

Environments and Tools for Parallel/Distributed/Grid Computing Designing Service-Based Resource Management Tools for a Healthy Grid Ecosystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Erik Elmroth, Francisco Hern´ andez, Johan Tordsson, and ¨ Per-Olov Ostberg BC-MPI: Running an MPI Application on Multiple Clusters with BeesyCluster Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pawel Czarnul

259

271

Managing Distributed Architecture with Extended WS-CDL . . . . . . . . . . Konrad Dusza and Henryk Krawczyk

281

REVENTS: Facilitating Event-Driven Distributed HPC Applications . . . Dawid Kurzyniec, Vaidy Sunderam, and Magdalena Slawi´ nska

291

Empowering Automatic Semantic Annotation in Grid . . . . . . . . . . . . . . . . ˇ Michal Laclav´ık, Marek Ciglan, Martin Seleng, and Ladislav Hluch´y

302

Fault Tolerant Record Placement for Decentralized SDDS LH* . . . . . . . . Grzegorz L  ukawski and Krzysztof Sapiecha

312

Grid Services for HSM Systems Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . Darin Nikolow, Renata Slota, and Jacek Kitowski

321

XVI

Table of Contents

The Vine Toolkit: A Java Framework for Developing Grid Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Michael Russell, Piotr Dziubecki, Piotr Grabowski, Michal Krysin´ski, Tomasz Kuczy´ nski, Dawid Szjenfeld, Dominik Tarnawczyk, Gosia Wolniewicz, and Jaroslaw Nabrzyski Enhancing Productivity in High Performance Computing through Systematic Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Magdalena Slawi´ nska, Jaroslaw Slawi´ nski, and Vaidy Sunderam

331

341

A Formal Model of Multi-agent Computations . . . . . . . . . . . . . . . . . . . . . . . Maciej Smolka

351

An Approach to Distributed Fault Injection Experiments . . . . . . . . . . . . . Janusz Sosnowski, Andrzej Tymoczko, and Piotr Gawkowski

361

Applications of Parallel/Distributed/Grid Computing Parallel Solution of Nonlinear Parabolic Problems on Logically Rectangular Grids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Andr´es Arrar´ as, Laura Portero, and Juan Carlos Jorge

371

Provenance Tracking in the ViroLab Virtual Laboratory . . . . . . . . . . . . . . Bartosz Bali´s, Marian Bubak, and Jakub Wach

381

Efficiency of Interactive Terrain Visualization with a PC-Cluster . . . . . . . Dariusz Dalecki, Jacek Lebied´z, Krzysztof Mieloszyk, and Bogdan Wiszniewski

391

Implementing Commodity Flow in an Agent-Based Model E-Commerce System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Maria Ganzha, Maciej Gawinecki, Pawel Kobzdej, Marcin Paprzycki, and Tomasz Serzysko MPI and OpenMP Computations for Nuclear Waste Deposition Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ondˇrej Jakl, Roman Kohut, and Jiˇr´ı Star´y A Pipelined Parallel Algorithm for OSIC Decoding . . . . . . . . . . . . . . . . . . . Francisco-Jose Mart´ınez-Zald´ıvar, Antonio-Manuel Vidal-Maci´ a, and Pedro Alonso A Self-scheduling Scheme for Parallel Processing in Heterogeneous Environment: Simulations of the Monte Carlo Type . . . . . . . . . . . . . . . . . . Grzegorz Musial, Lech D¸ebski, Dorota Jeziorek-Kniola, and Krzysztof Gola¸b

400

409

419

429

Table of Contents

Asynchronous Parallel Molecular Dynamics Simulations . . . . . . . . . . . . . . . Jaroslaw Mederski, L  ukasz Mikulski, and Piotr Bala Parallel Computing of GRAPES 3D-Variational Data Assimilation System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xiaoqian Zhu, Weimin Zhang, and Junqiang Song

XVII

439

447

Evolutionary Computing, Meta-Heuristics and Neural Networks The Effects of Heterogeneity on Asynchronous Panmictic Genetic Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Boleslaw K. Szymanski, Travis Desell, and Carlos Varela

457

A Parallel Sensor Selection Technique for Identification of Distributed Parameter Systems Subject to Correlated Observations . . . . . . . . . . . . . . . Przemyslaw Baranowski and Dariusz Uci´ nski

469

Distributed Segregative Genetic Algorithm for Solving Fuzzy Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Octav Brudaru and Octavian Buzatu

479

Solving Channel Borrowing Problem with Coevolutionary Genetic Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Krzysztof Gajc and Franciszek Seredynski

489

Balancedness in Binary Sequences with Cryptographic Applications . . . . Candelaria Hern´ andez-Goya and Amparo F´ uster-Sabater

499

A Cost-Benefit-Based Adaptation Scheme for Multimeme Algorithms . . . Wilfried Jakob

509

Optimizing the Shape of an Impeller Using the Differential Ant-Stigmergy Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ˇ Peter Koroˇsec, Jurij Silc, Klemen Oblak, and Franc Kosel

520

Parallel Algorithm for Simulation of Circuit and One-Way Quantum Computation Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marek Sawerwain

530

Modular Rough Neuro-fuzzy Systems for Classification . . . . . . . . . . . . . . . . Rafal Scherer, Marcin Korytkowski, Robert Nowicki, and Leszek Rutkowski

540

Tracing SQL Attacks Via Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . Jaroslaw Skaruz, Franciszek Seredynski, and Pascal Bouvry

549

Optimization of Parallel FDTD Computations Using a Genetic Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Adam Smyk and Marek Tudruj

559

XVIII

Table of Contents

Modular Type-2 Neuro-fuzzy Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Janusz Starczewski, Rafal Scherer, Marcin Korytkowski, and Robert Nowicki Evolutionary Viral-type Algorithm for the Inverse Problem for Iterated Function Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Barbara Strug, Andrzej Bielecki, and Marzena Bielecka Tackling the Grid Job Planning and Resource Allocation Problem Using a Hybrid Evolutionary Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Karl-Uwe Stucky, Wilfried Jakob, Alexander Quinte, and Wolfgang S¨ uß Evolutionary Algorithm with Forced Variation in Multi-dimensional Non-stationary Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dariusz Wawrzyniak and Andrzej Obuchowicz Hybrid Flowshop with Unrelated Machines, Sequence Dependent Setup Time and Availability Constraints: An Enhanced Crossover Operator for a Genetic Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Victor Yaurima, Larisa Burtseva, and Andrei Tchernykh

570

579

589

600

608

The Second Minisymposium on Novel Data Formats and Algorithms for Dense Linear Algebra Computations The Relevance of New Data Structure Approaches for Dense Linear Algebra in the New Multi-Core / Many Core Environments . . . . . . . . . . . Fred G. Gustavson Three Versions of a Minimal Storage Cholesky Algorithm Using New Data Structures Gives High Performance Speeds as Verified on Many Computers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jerzy Wa´sniewski and Fred G. Gustavson Hardware-Oriented Implementation of Cache Oblivious Matrix Operations Based on Space-filling Curves . . . . . . . . . . . . . . . . . . . . . . . . . . . Michael Bader, Robert Franz, Stephan G¨ unther, and Alexander Heinecke Parallel Tiled QR Factorization for Multicore Architectures . . . . . . . . . . . Alfredo Buttari, Julien Langou, Jakub Kurzak, and Jack Dongarra Application of Rectangular Full Packed and Blocked Hybrid Matrix Formats in Semidefinite Programming for Sensor Network Localization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jacek Blaszczyk, Ewa Niewiadomska-Szynkiewicz, and Michal Marks

618

622

628

639

649

Table of Contents

New Data Structures for Matrices and Specialized Inner Kernels: Low Overhead for High Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jos´e R. Herrero

XIX

659

The Implementation of BLAS for Band Matrices . . . . . . . . . . . . . . . . . . . . . Alfredo Rem´ on, Enrique S. Quintana-Ort´ı, and Gregorio Quintana-Ort´ı

668

Parallel Solution of Band Linear Systems in Model Reduction . . . . . . . . . Alfredo Rem´ on, Enrique S. Quintana-Ort´ı, and Gregorio Quintana-Ort´ı

678

Evaluating Linear Recursive Filters Using Novel Data Formats for Dense Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Przemyslaw Stpiczy´ nski

688

Combinatorial Tools for Parallel Sparse Matrix Computations Workshop Application of Fusion-Fission to the Multi-way Graph Partitioning Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Charles-Edmond Bichot

698

A Parallel Approximation Algorithm for the Weighted Maximum Matching Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fredrik Manne and Rob H. Bisseling

708

Heuristics for a Matrix Symmetrization Problem . . . . . . . . . . . . . . . . . . . . . Bora U¸car

718

A Supernodal Out-of-Core Sparse Gaussian-Elimination Method . . . . . . . Sivan Toledo and Anatoli Uchitel

728

The Third Grid Applications and Middleware Workshop (GAMW’07) A Large-Scale Semantic Grid Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marian Babik and Ladislav Hluchy

738

Scientific Workflow: A Survey and Research Directions . . . . . . . . . . . . . . . . Adam Barker and Jano van Hemert

746

A Light-Weight Grid Workflow Execution Engine Enabling Client and Middleware Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Erik Elmroth, Francisco Hern´ andez, and Johan Tordsson Supporting NAMD Application on the Grid Using GPE . . . . . . . . . . . . . . Rafal Kluszczy´ nski and Piotr Bala

754 762

XX

Table of Contents

A Grid Advance Reservation Framework for Co-allocation and Co-reservation Across Heterogeneous Local Resource Management Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Changtao Qu Using HLA and Grid for Distributed Multiscale Simulations . . . . . . . . . . . Katarzyna Rycerz, Marian Bubak, and Peter M.A. Sloot

770

780

The OpenCF: An Open Source Computational Framework Based on Web Services Technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Adrian Santos, Francisco Almeida, and Vincente Blanco

788

Service Level Agreement Metrics for Real-Time Application on the Grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . L  ukasz Skital, Maciej Janusz, Renata Slota, and Jacek Kitowski

798

Dynamic Control of Grid Workflows through Activities Global State Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marek Tudruj, Damian Kopanski, and Janusz Borkowski

807

Transparent Access to Grid-Based Compute Utilities . . . . . . . . . . . . . . . . . Constantino V´ azquez, Javier Font´ an, Eduardo Huedo, Rub´en S. Montero, and Ignacio M. Llorente Towards Secure Data Management System for Grid Environment Based on the Cell Broadband Engine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Roman Wyrzykowski and Lukasz Kuczynski Ontology Alignment for Contract Based Virtual Organizations Negotiation and Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Joanna Zieba, Bartosz Kryza, Renata Slota, Lukasz Dutka, and Jacek Kitowski

817

825

835

The Third Workshop on Large Scale Computations on Grids (LaSCoG’07) On Service-Oriented Symbolic Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . Alexandru Cˆ arstea, Marc Frˆıncu, Alexander Konovalov, Georgiana Macariu, and Dana Petcu

843

CPPC-G: Fault-Tolerant Applications on the Grid . . . . . . . . . . . . . . . . . . . Daniel D´ıaz, Xo´ an C. Pardo, Mar´ıa J. Mart´ın, Patricia Gonz´ alez, and Gabriel Rodr´ıguez

852

Garbage Collection in Object Oriented Condensed Graphs . . . . . . . . . . . . Sunil John and John P. Morrison

860

Table of Contents

XXI

MASIPE: A Tool Based on Mobile Agents for Monitoring Parallel Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . David E. Singh, Alejandro Miguel, F´elix Garc´ıa, and Jes´ us Carretero

870

Geovisualisation Service for Grid-Based Assessment of Natural Disasters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Peter Sl´ıˇzik and Ladislav Hluch´y

880

Web Portal to Make Large-Scale Scientific Computations Based on Grid Computing and MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Assel Zh. Akzhalova and Daniar Y. Aizhulov

888

Workshop on Models, Algorithms and Methodologies for Grid-Enabled Computing Environments The GSI Plug-In for gSOAP: Building Cross-Grid Interoperable Secure Grid Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Massimo Cafaro, Daniele Lezzi, Sandro Fiore, Giovanni Aloisio, and Robert van Engelen Implementing Effective Data Management Policies in Distributed and Grid Computing Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Luisa Carracciuolo, Giuliano Laccetti, and Marco Lapegna

894

902

Data Mining on Desktop Grid Platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . Valerie Fiolet, Richard Olejnik, Eryk Laskowski, L  ukasz Masko, Marek Tudruj, and Bernard Toursel

912

Distributed Resources Reservation Algorithm for GRID Networks . . . . . . Matviy Il’yashenko

922

A PMI-Aware Extension for the SSH Service . . . . . . . . . . . . . . . . . . . . . . . . Giuliano Laccetti and Giovanni Schmid

932

An Integrated ClassAd-Latent Semantic Indexing Matchmaking Algorithm for Globus Toolkit Based Computing Grids . . . . . . . . . . . . . . . . Raffaele Montella, Giulio Giunta, and Angelo Riccio

942

A Grid Computing Based Virtual Laboratory for Environmental Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Raffaele Montella, Giulio Giunta, and Giuliano Laccetti

951

Exploring the Behaviour of Fine-Grain Management for Virtual Resource Provisioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fernando Rodr´ıguez-Haro, Felix Freitag, Leandro Navarro, and Rene Brunner

961

XXII

Table of Contents

Workshop on Scheduling for Parallel Computing (SPC’07) Parallel Irregular Computations with Dynamic Load Balancing through Global Consistent State Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Janusz Borkowski and Marek Tudruj On-Line Partitioning for On-Line Scheduling with Resource Conflicts . . . Piotr Borowiecki A Multiobjective Evolutionary Approach for Multisite Mapping on Grids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ivanoe De Falco, Antonio Della Cioppa, Umberto Scafuri, and Ernesto Tarantino

971

981

991

Scheduling with Precedence Constraints: Mixed Graph Coloring in Series-Parallel Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1001 ˙ nski Hanna Furma´ nczyk, Adrian Kosowski, and Pawel Zyli´ A New Model of Multi-installment Divisible Loads Processing in Systems with Limited Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1009 Maciej Drozdowski and Marcin Lawenda Scheduling DAGs on Grids with Copying and Migration . . . . . . . . . . . . . . 1019 Israel Hernandez and Murray Cole Alea – Grid Scheduling Simulation Environment . . . . . . . . . . . . . . . . . . . . . 1029 Dalibor Klus´ aˇcek, Ludˇek Matyska, and Hana Rudov´ a Cost Minimisation in Unbounded Multi-interface Networks . . . . . . . . . . . . 1039 Adrian Kosowski and Alfredo Navarra Scheduling in Multi-organization Grids: Measuring the Inefficiency of Decentralization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1048 Krzysztof Rzadca Tightness Results for Malleable Task Scheduling Algorithms . . . . . . . . . . . 1059 Ulrich M. Schwarz

The Second Workshop on Language-Based Parallel Programming Models (WLPP’07) Universal Grid Client: Grid Operation Invoker . . . . . . . . . . . . . . . . . . . . . . . 1068 Tomasz Barty´ nski, Maciej Malawski, Tomasz Gubala, and Marian Bubak Divide-and-Conquer Parallel Programming with Minimally Synchronous Parallel ML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1078 Radia Benheddi and Fr´ed´eric Loulergue

Table of Contents

XXIII

Cloth Simulation in the SILC Matrix Computation Framework: A Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1086 Tamito Kajiyama, Akira Nukada, Reiji Suda, Hidehiko Hasegawa, and Akira Nishida Computing the Irregularity Strength of Connected Graphs by Parallel Constraint Solving in the Mozart System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1096 Adam Meissner, Magdalena Niwi´ nska, and Krzysztof Zwierzy´ nski DPSKEL: A Skeleton Based Tool for Parallel Dynamic Programming . . . 1104 Ignacio Pel´ aez, Francisco Almeida, and Fernando Su´ arez SkelJ: Skeletons for Object-Oriented Applications . . . . . . . . . . . . . . . . . . . . 1114 Joao L. Sobral Formal Semantics of DRMA-Style Programming in BSPlib . . . . . . . . . . . . 1122 Julien Tesson and Fr´ed´eric Loulergue A Container-Iterator Parallel Programming Model . . . . . . . . . . . . . . . . . . . 1130 Gerhard Zumbusch

Workshop on Performance Evaluation of Parallel Applications on Large-Scale Systems Semantic-Oriented Approach to Performance Monitoring of Distributed Java Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1140 Wlodzimierz Funika, Piotr Godowski, and Piotr P¸egiel Using Experimental Data to Improve the Performance Modelling of Parallel Linear Algebra Routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1150 Luis-Pedro Garc´ıa, Javier Cuenca, and Domingo Gim´enez Comparison of Execution Time Decomposition Methods for Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1160 Jan Kwiatkowski, Marcin Pawlik, and Dariusz Konieczny An Extensible Timing Infrastructure for Adaptive Large-Scale Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1170 Dylan Stark, Gabrielle Allen, Tom Goodale, Thomas Radke, and Erik Schnetter End to End QoS Measurements of TCP Connections . . . . . . . . . . . . . . . . . 1180 Witold Wysota and Jacek Wytrebowicz Performance Evaluation of Basic Linear Algebra Subroutines on a Matrix Co-processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1190 Ahmed S. Zekri and Stanislav G. Sedukhin

XXIV

Table of Contents

Workshop on Parallel Computational Biology (PBC’2007) High Throughput Comparison of Prokaryotic Genomes . . . . . . . . . . . . . . . 1200 Luciana Carota, Lisa Bartoli, Piero Fariselli, Pier L. Martelli, Ludovica Montanucci, Giorgio Maggi, and Rita Casadio A Parallel Classification and Feature Reduction Method for Biomedical Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1210 Mario R. Guarracino, Salvatore Cuciniello, and Davide Feminiano Applying SIMD Approach to Whole Genome Comparison on Commodity Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1220 Arpith Jacob, Marcin Paprzycki, Maria Ganzha, and Sugata Sanyal Parallel Multiprocessor Approaches to the RNA Folding Problem . . . . . . 1230 ´ Etienne Ogoubi, David Pouliot, Marcel Turcotte, and Abdelhakim Hafid Protein Similarity Search with Subset Seeds on a Dedicated Reconfigurable Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1240 Pierre Peterlongo, Laurent No´e, Dominique Lavenier, Gilles Georges, Julien Jacques, Gregory Kucherov, and Mathieu Giraud Parallel DNA Sequence Alignment on the Cell Broadband Engine . . . . . . 1249 Adrianto Wirawan, Kwoh Chee Keong, and Bertil Schmidt

Workshop on High Performance Computing for Engineering Applications Scalability and Performance Analysis of a Probabilistic Domain Decomposition Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1257 Juan A. Acebr´ on and Renato Spigler Scalability Analysis for a Multigrid Linear Equations Solver . . . . . . . . . . . 1265 Krzysztof Bana´s A Grid-Enabled Lattice-Boltzmann-Based Modelling System . . . . . . . . . . 1275 G´erard Dethier, Cyril Briquet, Pierre Marchot, and P.A. de Marneffe Parallel Bioinspired Algorithms in Optimization of Structures . . . . . . . . . 1285 Waclaw Ku´s and Tadeusz Burczy´ nski 3D Global Flow Stability Analysis on Unstructured Grids . . . . . . . . . . . . . 1293 Marek Morzy´ nski and Frank Thiele Performance of Multi Level Parallel Direct Solver for hp Finite Element Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1303 Maciej Paszy´ nski

Table of Contents

XXV

Graph Transformations for Modeling Parallel hp-Adaptive Finite Element Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1313 Maciej Paszy´ nski and Anna Paszy´ nska Acceleration of Preconditioned Krylov Solvers for Bubbly Flow Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1323 Jok Man Tang and Kees Vuik Persistent Data Structures for Fast Point Location . . . . . . . . . . . . . . . . . . . 1333 Michal Wichulski and Jacek Rokicki

Minisymposium on Interval Analysis A Reliable Extended Octree Representation of CSG Objects with an Adaptive Subdivision Depth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1341 Eva Dyllong and Cornelius Grimm Efficient Ray Tracing Using Interval Analysis . . . . . . . . . . . . . . . . . . . . . . . . 1351 Jorge Fl´ orez, Mateu Sbert, Miguel A. Sainz, and Josep Veh´ı A Survey of Interval Runge–Kutta and Multistep Methods for Solving the Initial Value Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1361 Karol Gajda, Malgorzata Jankowska, Andrzej Marciniak, and Barbara Szyszka Towards Efficient Prediction of Decisions under Interval Uncertainty . . . . 1372 Van Nam Huynh, Vladik Kreinovich, Yoshiteru Nakamori, and Hung T. Nguyen Interval Methods for Computing the Pareto-Front of a Multicriterial Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1382 Bartlomiej Jacek Kubica and Adam Wo´zniak Fuzzy Solution of Interval Linear Equations . . . . . . . . . . . . . . . . . . . . . . . . . 1392 Pavel Sevastjanov and Ludmila Dymova On Checking the Monotonicity of Parametric Interval Solution of Linear Structural Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1400 Iwona Skalna Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1411

Safety of a Session Guarantees Protocol Using Plausible Clocks Jerzy Brzeziński, Michał Kalewski, and Cezary Sobaniec Institute of Computing Science Poznań University of Technology, Poland {Jerzy.Brzezinski,Michal.Kalewski,Cezary.Sobaniec}@cs.put.poznan.pl

Abstract. Session guarantees is a group of consistency models used to manage replica consistency in a distributed system from the client’s perspective. In this paper we present and prove safety of a novel protocol implementing session guarantees. The protocol uses server-based version vectors conceptually based on plausible clocks. The version vectors are constant-size and accept dynamic reconfigurations, which is the main advantage of this approach. The cost is reduced accuracy of representation of sets of operations, which, however, does not violate session guarantees. Keywords: consistency models, session guarantees, plausible clocks.

1

Introduction

Replication in distributed systems is used for achieving high performance and availability. Unfortunately, replication introduces inconsistency of replicas in case of replica updates. The users are usually not interested in the internal mechanisms providing high availability; they rather want the system to behave in a transparent manner, as if it was not replicated. However, transparent replication means that it is necessary to apply a strong consistency model, i.e. the system must provide strong guarantees related to ordering of modifying operations, which usually makes the system inefficient. There are several weaker consistency models than strong consistency, developed mainly in the context of Distributed Shared Memory systems. However, these data-centric consistency models [1] do not take into account mobility; they assume that the clients are bound to the servers maintaining the replicas. There is also another class of consistency models — client-centric consistency models, also known as session guarantees [2]. Session guarantees maintain data consistency from a single client point of view, but the client can switch from one server to another. The data observed by the user are consistent with the previous sequence of operations issued by the client within a session. There are four different guarantees ordering different types of operations. Definitions of session guarantees refer to sets of operations issued by clients. A consistency protocol of session guarantees has to efficiently represent the sets, which is not trivial because the sets are monotonically growing. The original authors of session guarantees have proposed application of version vectors for R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 1–10, 2008. c Springer-Verlag Berlin Heidelberg 2008 

2

J. Brzeziński, M. Kalewski, and C. Sobaniec

efficient representation of sets of operations. Version vectors are conceptually based on vector clocks [3,4] used for keeping track of causal dependencies in distributed systems. In a static system, version vectors have a constant size, representing all operations performed up to the logical time represented by the value of a given vector. However, in a dynamic system the set of servers participating in the processing may change: some servers may crash, others may be added to the system. As a consequence, the structure of version vectors must reflect such reconfigurations. New positions in the version vector must be allocated, and some may become unused. In general, frequent reconfigurations may lead to difficult to maintain increase of size of version vectors. In this paper we propose adoption of plausible clocks [5,6] which are used to approximate causal dependencies in distributed systems. The main advantage of version vectors based on plausible clocks is constant size, regardless of the number of servers, and their reconfigurations. The cost is decreased accuracy of representation of sets of operations, which results in additional — not required by session guarantees — synchronization between servers, which, however, do not violate the model. This paper presents a consistency protocol of session guarantees, called VsRSG, and formally proves ifs safety. In this context safety means that clients’ requirements concerning session guarantees are preserved throughout the whole execution of the protocol.

2

Session Guarantees

We consider a weakly consistent replicated storage system. The system consists of a number of servers holding a full copy of a set of shared objects, and clients running applications that access the objects. Clients are separated from servers, i.e. a client application may run on a separate computer than the server. A client may access a shared object after selecting a single server and sending a direct request to the server. Clients are mobile, i.e. they can switch from one server to another during application execution. Session guarantees are expected to take care of data consistency observed by a migrating client. The set of shared objects replicated by the servers does not imply any particular data model or organization. Operations performed on shared objects are divided into reads and writes. A read does not change states of the shared objects, while a write does. A write may cause an update of an object, it may create a new object, or delete an existing one. A write may also atomically update states of several objects. C Operations on shared objects issued by a client Ci are ordered by a relation i called client issue order. A server Sj performs operations in an order represented Sj

by a relation . Writes and reads on objects will be denoted by w and r, respectively. An operation performed by a server Sj will be denoted by w|Sj or r|Sj . Definition 1. Relevant writes RW (r) of a read operation r is a set of writes that has influenced the current state of objects observed by the read r.

Safety of a Session Guarantees Protocol Using Plausible Clocks

3

The exact meaning of relevant writes will strongly depend on the characteristics of a given system or application. For example, in case of simple isolated objects (i.e. objects with methods that access only their internal fields), relevant writes of a read on object x may be represented by all previous writes on object x. Session guarantees have been defined in [2]. The following more formal definitions are based on those concepts. The definitions assume that operations are unique, i.e. they are labeled by some internal unique identifiers. Definition 2. Read Your Writes (RYW) session guarantee is defined as follows:   Sj C ∀Ci ∀Sj w i r|Sj =⇒ w  r Definition 3. Monotonic Writes (MW) session guarantee is defined as follows:   Sj C ∀Ci ∀Sj w1 i w2 |Sj =⇒ w1  w2 Definition 4. Monotonic Reads (MR) session guarantee is defined as follows:   Sj C ∀Ci ∀Sj r1 i r2 |Sj =⇒ ∀wk ∈ RW (r1 ) : wk  r2 Definition 5. Writes Follow Reads (WFR) session guarantee is defined as follows:   Sj Ci ∀Ci ∀Sj r  w|Sj =⇒ ∀wk ∈ RW (r) : wk  w

3

VsRSG Protocol

The proposed VsRSG protocol implementing session guarantees intercepts communication between clients and servers; at the client side before sending a request, at the server side after receiving the request and before sending a reply, and at the client side after receiving the reply. These interceptions are used to exchange and maintain additional data structures necessary to preserve appropriate session guarantees. After receipt of a new request a server checks whether its state is sufficiently up to date to satisfy client’s requirements. If the server’s state is outdated then the request is postponed and will be resumed after updating the server. Servers periodically exchange information about writes performed in the past in order to synchronize the states of replicas. This synchronization procedure eventually causes total propagation of all writes directly submitted by clients. It does not influence safety of the VsRSG protocol but rather its liveness, therefore it will not be discussed in this paper (example procedure is presented in [7]). Every server Sj records all writes performed locally in a history. The writes result from direct client requests, or are incorporated from other servers during synchronization procedure. The writes are performed sequentially, therefore the history is totally ordered. Formally, histories are defined as follows:

4

J. Brzeziński, M. Kalewski, and C. Sobaniec

Definition 6. A history HSj

  Sj is a linearly ordered set OSj ,  where OSj is Sj

a set of writes performed by a server Sj , and relation  represents an execution order of the writes. During synchronization of servers the histories are concatenated. A concatenation of two histories is constructed as a sum of the first history, and new writes found in the second history. The orderings of respective histories are preserved, and new writes are added at the end of the first history. A write in VsRSG protocol is labeled with a vector timestamp set to the current value of the version vector VSj of the Sj server performing the write for the first time. In the presentation of the VsRSG protocol the vector timestamp of a write w is returned by a function T : O → V . A single i-th position of the version vector timestamp associated with a write will be denoted by T (w)[i]. VsRSG protocol is presented in Fig. 2. A request sent by a client is a couple op, SG, where op is an operation to be performed, and SG is a set of session guarantees required for this operation. Before sending to the server, the request is supplemented with a vector W representing the client’s requirements. A reply is a triple op, res, W  where op is the operation just performed, res represents the results of the operation (delivered to the application), and W is a vector representing the state of the server just after performing the operation. Before sending a request by a client Ci , a vector W representing its requirements is calculated based on the type of operation, and the set SG of session guarantees required for the operation. The vector W is set to either 0, or WCi — a vector representing writes issued by client Ci , or RCi — a vector representing writes relevant to reads issued by the client, or to a maximum of these two vector (lines 1, 3 and 6). The maximum of two vectors V1 and V2 is a vector V = max (V1 , V2 ), such that V [i] = max (V1 [i], V2 [i]). On receipt of a new request a server Sj checks whether its local version vector VSj dominates the vector W sent by the client (line 14), which is expected to be sufficient for providing appropriate session guarantees. A version vector V1 dominates a version vector V2 , which is denoted by V1 ≥ V2 , when ∀i : V1 [i] ≥ V2 [i]. If the state of the server is not sufficiently up to date, the request is postponed (line 15), and will be resumed after synchronization with another server (line 41). As a result of writes performed by a server Sj , its version vector VSj is updated (line 19), and a timestamped operation is recorded in history HSj (lines 20 and 21). The current value of the server version vector VSj is returned to the client (line 23) and updates the client’s vector WCi in case of writes (line 26), or RCi in case of reads (line 28). Version vectors are used for efficient representation of sets of writes required by clients and necessary to check at the server side. Version vectors used by standard VsSG protocol have the following form: v1 v2 . . . vN , where N is the total number of servers in the system. A single position vj denotes the number of writes performed by a server Sj , and changes whenever a new write request is performed by the server. Because every server increments the version vector for every write, and the changes are done at different positions, the values of version

Safety of a Session Guarantees Protocol Using Plausible Clocks

C1

S1

w(x)1

[1 0] [1 0]

S3

C2 RYW

5

w(x)2

r(x)1

(a) C1

S1

w(x)1

[2 0] [1 0]

S3

C2 RYW

C1

w(x)2

r(x)2

(b)

[1 0]

S1

S3

[1 0] [2 0]

w(x)1

C2 RYW

[2 0] [1 0]

w(x)2

r(x)1

(c)

Fig. 1. Ordering of writes in a protocol using server-based constant size version vectors

vectors at servers during execution of writes are unique. The version vectors must be unique because they are used for identifying write operations. VsRSG protocol uses constant size server-based version vectors, consisting of R-positions, and a server Sj is expected to update a single position j mod R of the server’s version vector VSj . Obviously, for R < N there are servers that share a single position of the version vectors, and thus certain writes may result in indistinguishable values of version vectors. Let us consider a case shown in Fig. 1(a). There are 2 clients, and 3 servers (server S2 is not shown). Let R = 2, which means that servers S1 and S3 share the first position of the version vector. Client C1 issues a write of value 1 to variable (object) x, which is denoted by w(x)1, and, as a result of the write, VS1 is updated to [ 1 0 ]. Client C2 concurrently issues another write w(x)2 at server S3 , which causes an update of the server version vector VS3 to [ 1 0 ]. Because these two servers share the first position of the version vector, the vector timestamps of the writes are identical. As a result, client C2 after migrating to server S1 cannot observe its previous write despite RYW session guarantee was requested. In order to solve the problem additional ordering must be forced. The values of version vectors will be unique if writes performed by different servers sharing a common position in the version vector are globally ordered. This can be achieved by means of additional sequencers used by groups of servers. VsRSG protocol generates a sequence number for every write (see line 10 of Fig. 2). The sequence numbers are generated for servers sharing the same position of the version vectors. Sequence numbers are checked

6

J. Brzeziński, M. Kalewski, and C. Sobaniec

before performing writes (line 14), and thus writes that are requested before performing all previous writes of appropriate group of servers are suspended. In the case shown in Fig. 1(a) it results in performing either w(x)1 before w(x)2 — Fig. 1(b), or w(x)2 before w(x)1 — Fig. 1(b). Regardless of the ordering of the writes, the requested RYW session guarantee is provided for client C2 . Depending on the length of the version vector, the additional ordering forced by VsRSG protocol limits possible concurrency of request processing; only R write requests can be performed concurrently. However, in a typical scenario reads are much more frequent than writes. Reads are processed by VsRSG protocol in a very similar manner to the standard VsSG protocol, without additional ordering. The version vectors are constant size, therefore their management is greatly simplified. They are also ready for changes: new participants of the processing may come and go at minimal additional cost. The price of these advantages is decreased accuracy of the version vectors resulting from their limited capacity. The sets of operations represented by plausible version vectors are larger than sets of writes of full version vectors, which results in more “eager” propagation of updates. A positive consequence of this fact is a reduction of the number of cases where client requests must wait due to missing writes at servers. VsRSG protocol uses a simple mapping between servers and positions in the version vector consisting in “wrapping” the server’s number around the length of the version vector by modulo function. In general, the mapping can be an arbitrary function that best suits the system, or application needs (see examples in [6]). The system designer can take into account additional specific knowledge concerning its characteristics, and propose a more appropriate mapping.

4

Safety of VsRSG Protocol

 Definition 7. A supremum of a set of writes OSj , denoted by V OSj , is a vector that is set  to 0 for an empty set, and for nonempty sets its i-th position is defined as V OSj [i] = maxw∈OSj T (w)[i]. Lemma 1. For every server Sj running VsRSG protocol at every moment:  V OSj = VSj Proof. By induction. 1. Basis. At the very beginning VSj = 0, and the set of writes OSj = ∅, therefore   V OSj = 0, hence V OSj = VSj .  2. Induction step. Let us assume a state where condition V OSj = VSj holds. The set OSj and the version vector VSj can change only in the following two situations: (a) Server Sj accepts a new write requested by a client. This causes the value of VSj [j mod R] to be incremented by 1, next the write is timestamped with the current value of vector VSj , and the write is added to OSj . This  causes V OSj to be also incremented at position j mod R by 1 (lines  19 and 21 of Fig. 2). As a result, the condition V OSj = VSj still holds.

Safety of a Session Guarantees Protocol Using Plausible Clocks Upon sending a request op, SG to server Sj at client Ci 1: W ← 0 2: if (iswrite(op) and MW ∈ SG) or (not iswrite(op) and RYW ∈ SG) then 3: W ← max (W, WCi ) 4: end if 5: if (iswrite(op) and WFR ∈ SG) or (not iswrite(op) and MR ∈ SG) then 6: W ← max (W, RCi ) 7: end if 8: send op, W  to Sj Upon receiving a request op, W  from client Ci at server Sj 9: if iswrite(op) then 10: seq ← getSeqNumber(j mod R) 11: else 12: seq ← 0 13: end if   14: while VSj ≥ W ∨ seq > VSj [j mod R] + 1 do 15: wait() 16: end while 17: perform op and store results in res 18: if iswrite(op) then 19: VSj [j mod R] ← VSj [j mod R] + 1 20: timestamp op with VSj 21: HSj ← HSj ∪ {op} 22: end if 23: W ← VSj 24: send op, res, W  to Ci Upon receiving a reply op, res, W  from server Sj at client Ci 25: if iswrite(op) then 26: WCi ← max (WCi , W ) 27: else 28: RCi ← max (RCi , W ) 29: end if 30: deliver res Every Δt at server Sj 31: foreach Sk = Sj  do 32: send Sj , HSj to Sk 33: end for Upon receiving an update Sk , H at server Sj 34: foreach wi ∈ H do 35: if VSj ≥ T (wi ) then 36: perform wi  37: VSj ← max VSj , T (wi ) 38: HSj ← HSj ∪ {wi } 39: end if 40: end for 41: signal() Fig. 2. VsRSG consistency protocol of session guarantees

7

8

J. Brzeziński, M. Kalewski, and C. Sobaniec

(b) Server Sj incorporates a write w received from another server. This causes the current value of VSj to be maximized with the vector T (w) of the write being added (line 37). The new write  is then added to OSj (line 38). As a result, values of VSj and V OSj will be incremented  at the same positions by the same values, therefore the condition V OSj = VSj still holds.  A set of writes represented by a version vector will be denoted by WS(V ), and is formally defined in the following manner. Definition 8. A write-set WS (V ) of a given version vector V is defined as  NS WS (V ) = j=1 w ∈ OSj : T (w) ≤ V . Lemma 2. For any two vectors V1 and V2 used by servers and clients of VsRSG protocol: V1 ≥ V2 ⇐⇒ WS (V1 ) ⊇ WS (V2 ) Proof. 1) Sufficient condition. By contradiction, let us assume that: V1 ≥ V2 ∧ WS (V1 ) ⊇ WS (V2 ) which means that: ∃w ∈ O [w ∈ WS (V1 ) ∧ w ∈ WS (V2 )] and, according to Definition 8: ∃k (T (w)[k] > V1 [k] ∧ T (w)[k] ≤ V2 [k]) =⇒ V1 [k] < V2 [k] =⇒ V1 ≥ V2 2) Necessary condition. By contradiction, let us assume that: WS (V1 ) ⊇ WS (V2 ) ∧ V1 ≥ V2 which means that: ∃k : V1 [k] < V2 [k] Version vectors at position k are only incremented when a new write is performed by a server Sj , where j mod R = k (line 19). The new value at position k is unique between all version vectors identifying writes, because it is generated by the function getSeqNumber(k), returning consecutive numbers for a given position (line 10). Before the server version vector is incremented, its value must be exactly 1 less than the sequence number, therefore the incrementation at line 19 has the same effect as assignment VSj [j mod R] = seq. As a result, every value of the version vector at position k is associated with a write accepted by some server. Based on the observation ∃k : V1 [k] < V2 [k], it can be concluded that: ∃w ∈ OSj [w ∈ WS (V2 ) ∧ w ∈ WS (V1 )] and hence WS (V1 ) ⊇ WS (V2 ).



Safety of a Session Guarantees Protocol Using Plausible Clocks

9

The implication V1 ≥ V2 =⇒ WS (V1 ) ⊇ WS (V2 ) is true for any vectors V1 and V2 , while the other implication WS (V1 ) ⊇ WS (V2 ) =⇒ V1 ≥ V2 is true only for vectors maintained by servers and clients, i.e. for vectors VSj , WCi , and RCi .  Lemma 3. At any time during execution of VsRSG protocol OSj = WS VSj .  Proof. By contradiction: 1) Let us assume that ∃w ∈ OSj : w ∈ WS VSj . According to Definition 8, a write w does not belong to WS VSj when T (w) ≤ VSj . This implies that ∃k : T (w)[k] > VSj [k], and, according to Lemma 1, T (w)[k] >   V OSj [k], which implies V OSj ≥ T (w). Basedon Definition 7, w ∈ OSj — a contradiction. 2) Let us assume that ∃w ∈ WS VSj : w ∈ OSj . According  to Definition 7, a write w does not belong to OSj when V OSj ≥ T (w). This  implies that ∃k : T (w)[k] > V OSj , and, according to Lemma 1, T (w)[k] > VSj [k], which implies T (w) ≤ VSj . Based on Definition 8, w ∈ WS VSj — a contradiction.  Lemma 4. At any time during execution of VsRSG protocol WS (WCi ) contains all writes issued by a client Ci . Proof. A write issued by a client Ci and performed by a server Sj updates the client’s vector WCi by calculating a maximum of its current value and value of the server version vector VSj (lines 23 and 26). Hence, afterperforming the write WCi ≥ VSj , and (according to Lemma 2) WS (WCi ) ⊇ WS VSj , and (according to Lemma 3) WS (WCi ) ⊇ OSj . It means that the write-set WS (WCi ) contains all writes requested directly at server Sj , including also writes requested by client Ci at server Sj . The vector WCi monotonically increases, therefore no past write is lost in case of a migration to another server.  Lemma 5. At any time during execution of VsRSG protocol WS (RCi ) contains all writes relevant to reads issued by a client Ci . Proof. A read issued by a client Ci and performed by a server Sj updates the client’s vector RCi by calculating a maximum of its current value and value of the server version vector VSj (lines 23 and 28). Hence (according to Lemmata 2 and 3) RCi ≥ VSj =⇒ WS (RCi ) ⊇ WS VSj = OSj . It means that the write-set WS (RCi ) contains all writes performed at server Sj , therefore also writes relevant to reads requested by client Ci at server Sj . The vector RCi monotonically increases, therefore no past write is lost in case of a migration to another server.  Theorem 1. RYW session guarantee is preserved by VsRSG protocol for clients requesting it. Proof. Let us consider two operations w and r, issued by a client Ci requiring RYW session guarantee. Let the read follow the write in the client’s issue order, C and let the read be performed by a server Sj , i.e. w i r|Sj . After performing w we have (according to Lemma 4) w ∈ WS (WCi ). Because VSj ≥ WCi is

10

J. Brzeziński, M. Kalewski, and C. Sobaniec

fulfilled performing r (lines 3 and 14), we get (according to Lemma 2)  before WS VSj ⊇ WS (WCi ) =⇒ w ∈ WS VSj . Because local operations at servers Sj

are totally ordered, we get w  r. This will happen for any client Ci requiring  Sj Ci RYW and any server Sj , so ∀Ci ∀Sj w  r|Sj =⇒ w  r , which means that RYW session guarantee is preserved.



Theorem 2. MR session guarantee is preserved by VsRSG protocol for clients requesting it. Proof. Let us consider two reads r1 and r2 , issued by a client Ci requiring MR session guarantee. Let the second read follow the first read in the client’s issue C order, and let the second read be performed by a server Sj , i.e. r1 i r2 |Sj . After performing r1 we have (according to Lemma 5) ∀wk ∈ RW (r1 ) : wk ∈ WS (RCi ). Because VSj ≥ RCi is fulfilled  before performing r2 (lines 6 and 14), we get (according to Lemma 2) WS V ⊇ WS (RCi ) =⇒ ∀wk ∈ RW (r1 ) : S j  wk ∈ WS VSj . Because local operations at servers are totally ordered, we get Sj

∀wk ∈ RW (r1 ) : wk  r2 . This will happen for any client  Ci and any server Sj , S j C so ∀Ci ∀Sj r1 i r2 |Sj =⇒ ∀wk ∈ RW (r1 ) : wk  r2 , which means that MR session guarantee is preserved.



A theorem and a proof for MW are analogous to RYW. A theorem and a proof for WFR are analogous to MR.

References 1. Tanenbaum, A.S., van Steen, M.: Distributed Systems — Principles and Paradigms. Prentice Hall, New Jersey (2002) 2. Terry, D.B., Demers, A.J., Petersen, K., Spreitzer, M., Theimer, M., Welch, B.W.: Session guarantees for weakly consistent replicated data. In: Proc. of the Third Int. Conf. on Parallel and Distributed Information Systems (PDIS 1994), Austin, USA, pp. 140–149. IEEE Computer Society, Los Alamitos (1994) 3. Mattern, F.: Virtual time and global states of distributed systems. In: Cosnard, Q., Raynal, R. (eds.) Proc. of the Int. Conf. on Parallel and Distributed Algorithms, pp. 215–226. Elsevier Science Publishers B.V, Amsterdam (1988) 4. Fidge, C.: Logical time in distributed computing systems. Computer 24, 28–33 (1991) 5. Torres-Rojas, F.J., Ahamad, M.: Plausible clocks: Constant size logical clocks for distributed systems. Distributed Computing 12, 179–196 (1999) 6. Gidenstam, A., Papatriantafilou, M.: Adaptive plausible clocks. In: Proc.of the 24th Int. Conf. on Distributed Computing Systems (ICDCS 2004), Tokyo, Japan, pp. 86–93 (2004) 7. Petersen, K., Spreitzer, M.J., Terry, D.B., Theimer, M.M., Demers, A.J.: Flexible update propagation for weakly consistent replication. In: Proc. of the 16th ACM Symp. on Operating Systems Principles (SOSP-16), Saint Malo, France, pp. 288– 301 (1997)

On Checkpoint Overhead in Distributed Systems Providing Session Guarantees Arkadiusz Danilecki, Anna Kobusi´ nska, and Marek Libuda Institute of Computing Science Pozna´ n University of Technology, Poland {Arkadiusz.Danilecki,Anna.Kobusinska,Marek.Libuda}@cs.put.poznan.pl

Abstract. This paper presents the performance evaluation of the checkpointing and rollback-recovery protocols for distributed mobile systems, guarantying client-centric consistency models, despite failures of servers. The performance of considered protocols is evaluated by estimating the checkpointing overhead. Additionally, the paper disscusses the influence of the consistency models provided by the system on the moments of taking checkpoints. The impact of the obtained results on strategies for checkpointing in the mobile environment guarantying client-centric consistency models is assessed. Keywords: checkpointing, rollback-recovery, performance evaluation, distributed mobile systems, session guarantees.

1

Introduction

The variety of computational and networking capabilities, introduced by mobile computing, results in our growing reliance on mobile applications. It also mandates deploying mobile systems that should meet not only performance demands imposed on them, but also reliability requirements. The well known method to achieve fault tolerance in distributed systems is checkpointing and rollback recovery. This technique reduces the overall expected time of completing the computation in case of failures, at the same time increasing the efficiency and enhancing the reliability of the system. In general, checkpointing and rollback-recovery is based on the idea of saving the current system state during a failure-free execution, to be able to restore the error-free state from the saved data in case of failures. Each time the system’s state is recorded into a stable storage, able to survive all failures, a checkpoint is said to be taken. The benefit of checkpointing, however, comes at a price — the excessive checkpointing would result in performance degradation, while deficient checkpoining would incur an expensive recovery overhead. Thus, the appropriate checkpoint interval minimizes the expected recovery time of the aplication in the presence of failures and reduces the time added to the application as a result of checkpointing in the failure free execution. Therefore, the problem of placing checkpoints 

This work was supported in part by the State Committee for Scientific Research (KBN), Poland, under grant KBN 3 T11C 073 28.

R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 11–19, 2008. c Springer-Verlag Berlin Heidelberg 2008 

12

A. Danilecki, A. Kobusi´ nska, and M. Libuda

“optimally” in time to meet the system performance objective is of paramount importance. Determining the appropriate checkpoint interval is a field of research with a rich history. The first papers on the topic appeard in the 70’s in the context of transaction processing systems. Later work was concentrated on real-time and distributed systems. It was suggested that the checkpointing frequency should be a function of the failure rate [1,2]. On the other hand, it was advanced that the number of checkpoints should rely on the distribution of the program execution time [3]. In turn, in this paper, we state that the moment of taking a checkpoint, depends on a consistency model provided by the system. We consider the client-centric consistency models (called session guarantees), proposed for distributed mobile systems and affirm that the most desirable moments of taking checkpoints in such systems depend on the semantics of required session guarantees. To prove this statemement we evaluate the performance of checkpoining and rollback-recovery protocols, proposed in [4], providing session guarantees despite failures of servers. The checkpointing scenarios considered in these protocols are evaluated by estimating the checkpointing overhead. Additionally, the impact of the obtained results on strategies for checkpointing in the mobile environment guaranteeing client-centric consistency models is assessed. The realistic performance evaluation of checkpointing and rollback-recovery is difficult due to numerous factors that must be taken into account, involving a detailed characterization of the system and of a client’s application. Therefore, the performance of the proposed rollback-recovery protocols is in this paper evaluated quantitatively, by carrying out appropriate simulation experiments. The rest of the paper is organized as follows. We describe the system model and session guarantees in Section 2. An informal presentation of the rollbackrecovery protocol appears in Section 3. The results of experiments follow in Section 5, and finally we conclude in Section 6.

2

Distributed Mobile System Model

Throughout this paper, a replicated distributed storage system is considered. The system consists of a number of unreliable servers holding a full copy of a shared objects and clients running applications that access these objects. Clients are mobile, i.e. they can switch from one server to another during an application execution. To access the shared object, clients select a single server and send a direct request to this server. Operations are issued by clients sequentially, i.e. a new operation may be issued after the results of the previous one have been obtained. The storage replicated by servers does not imply any particular data model or organization. Operations performed on shared objects are basically divided into reads and writes. Clients can concurrently submit conflicting writes at different servers, e.g. writes that modify the overlapping parts of the data storage. It is assumed that clients perceive the data from the replicated storage according to session guarantees [5], called also client-centric consistency models, which specify consistency conditions of the state of replicas that must hold at a

On Checkpoint Overhead in Distributed Systems

13

server to execute an operation requested by the client. The informal definitions of session guarantees in the context of replicated shared objects are given below: – Read Your Writes (RYW) guarantee states that a read operation requested by a client can be executed on a server that has performed all write operations previously issued by the requesting client. – Monotonic Reads (MR) guarantee states that a read operation requested by a client can be executed by a server that has performed all relevant writes preceeding past read operations of the client. The term relevant writes has been introduced in [5]. The interpretation of this term for the purpose of this paper is that all write operations that influence the current state of a given object are relevant to a read operation performed by the server. – Monotonic Writes (MW) guarantee states that a write operation issued by a client can be executed on a server if it has performed all write operations previously issued by the client. – Writes Follow Reads (WFR) guarantee states that a write requested by a client can be executed on a server that has performed all writes relevant to reads previously requested by the client. In this paper, we focus on failures of servers and assume the crash-recovery failure model, i.e. servers may crash and recover after crashing a finite number of times [6]. Additionally, we assume that servers have access to a stable storage, able to survive all failures. To ensure fault-tolerance, objects are saved in the stable storage in the form of the log or the checkpoint. A log is a set of operations issued by the client, while a checkpoint is a couple of a vector describing writes performed by the server and the ordered set of operations performed by the server and called history.

3

The Rollback-Recovery Protocols

To preserve the required session guarantee, the rollback-recovery protocol must ensure that writes issued by the client and essential to preserve this guarantee are not lost in the result of the server’s failure. In the proposed protocols, it is achieved by logging in the stable storage appropriate operations obtained by servers. To optimize the protocol, servers save only some of the obtained operations, namely those received directly from clients. Servers periodically exchange information about writes performed in the past in order to synchronize states of their replicas. This synchronization procedure eventually causes total propagation of all writes directly submitted by clients. The operations obtained during synchronization procedure are performed by the server, but they need not to be logged, because they have already been saved in the stable storage of other servers. Servers periodically take a checkpoint of its state. When the checkpoint is taken, the log is cleared. After the failure, the failed server restarts from the latest checkpoint and replays operations from the log.

14

A. Danilecki, A. Kobusi´ nska, and M. Libuda

The moments of taking checkpoints are determined by session guarantees requirements [4]. Therefore, for each guarantee, sets of operations essential to preserve it are distinguished and a desirable moment of taking a checkpoint, denoted by DMTC, is defined. DMTC indicates such a moment, before which there is no need to take a checkpoint, because the server has not performed any operation required by a given session guarantee. In the case of RYW guarantee, DMTC is determined by obtaining a read request from a client. For MR guarantee, DMTC is determined by obtaining a read request, which follows the read previously issued by the same client. With reference to MW session guarantee, DMTC is determined by obtaining a write request, which follows the write previously issued by the same client. Finally, for WFR session guarantee, DTMC is determined by obtaining a write request, which follows the read request issued by the same client. Depending on system characteristics, in general, checkpoints can be taken every n-th DMTC, where n = 1, 2, 3, ... . But, there is no need to take checkpoints between two consecutive DMTC. In this paper, several rollback-recovery protocols, called rVsSG, rVsRYW, rVsMW, rVsMR, rVsWFR and rVsAll, proposed in [4], are considered. In rVsSG protocol, servers take checkpoints periodically, every K operation issued by clients. Protocols rVsRYW, rVsMW, rVsMR, and rVsWFR take checkpoints accordingly to DTMC for RYW, MW, MR and WFR session guarantees, respectively. Finally, rVsSGOpt protocol gathers features of rVsRYW, rVsMW, rVsMR, and rVsWFR protocols and determines the moment of taking a checkpoint in accordance with DTMC of all session guarantees.

4

Input Parameters of Simulation Experiments

The performance of the proposed rollback-recovery protocols is quantitatively evaluated in terms of checkpointing overhead. Since the checkpoint overhead is the increase in the amount of time added to the application in the failure-free run as the result of checkpointing, the protocol total execution time was a main criterion of the evaluation. The experiments were performed with respect to two points of reference: the coherence protocol, called VsSG [7] and rollback-recovery protocol rVsPes [4], implementing typical pessimistic checkpointing, where every server logs every write operation it receives, and takes checkpoints from time to time. For the performance analysis the refined simulator, called Distributed Algorithms Simulator (DAS) [8], has been developed. Since, in general, the simulation parameters depend on the system configuration and hardware characteristics, the values used in DAS were estimated through benchmark tests, which assessed the memory read/write transfer time, hard drive read/write transfer time, LAN, WAN and WIFI network delays, failure model, intensity of client migrations, and finally log and checkpoint delays. To ensure that the simulator works as expected, the mean values obtained during the validation tests were used as DAS input values. In the performed experiments we assumed that hard drive write and read access time is set to 3 and 2, respectively. These values represent the

On Checkpoint Overhead in Distributed Systems

15

time of saving a single operation in the stable storage and reading it from the storage while retrieving this operation during rollback-recovery. The server’s local read/write operation execution time is characterized by the memory access time and is set to 0. The cost of logging and checkpointing is calculated on the basis of the definitions of these mechanisms and the above values. Thus, the cost of logging the operation in the stable storage is simply the hard drive access time, while for the checkpoint it is a number of objects saved in the checkpoint plus the number of operations in the history multiplied by the hard drive access time. The simulation experiments were repeated 150 times for each protocol and each combination of the following parameters: 10, 50, 100, 200, 300, 400 clients, 15 servers, and two client’s application scenarios: with 20 and 80 percentage of all access operations that are writes (from now on these scenarios will be denoted by 0.2 and 0.8, respectively). The obtained results are presented in the following section. For each of the conducted experiments the 99% confidence limits for a mean time value never exceed 1.81% of the mean value, while 95% confidence limits never exceed 1.37% of the mean value. Therefore, each point of the figures presented in the next section corresponds with the 99% probability to the range of 1.81% width of the mean time value, and with 95% probability to the range of 1.37% width.

5

Simulation Experiments and Performance Evaluation

In the figure 1 the comparison of VsSG, rVsSG, rVsAll and rVsPes protocols is presented. The vertical axes of the graphs use a logarithmic scale and represent the time, while the horizontal axes represent the number of clients. In the failure-free run, the best performance, regardless of the application scenario and the number of clients, is achieved by VsSG consistency protocol which does not bear any additional expenses connected with taking a checkpoint and accessing the stable storage. On the other hand, the highest checkpoint overhead has rVsPes rollback-recovery protocol, saving in the stable storage every 10000 VsSG rVsSG rVsAll rVsPes

0.20 0.20 0.20 0.20

VsSG rVsSG rVsAll rVsPes

0.80 0.80 0.80 0.80

Execution time

Execution time

10000

1000 1000 0

50

100

150

200 Clients

250

300

350

400

0

50

100

150

200 250 Number of clients

300

Fig. 1. Checkpoint overhead for 0.2 and 0.8 client’s application scenario

350

400

16

A. Danilecki, A. Kobusi´ nska, and M. Libuda

obtained write operation, i.e. the one issued by the client as well as the one obtained during synchronization with other servers. Since the frequent access to stable storage is very time-consuming, the performance penalty in the case of rVsPes protocol is very high. Obviously, the higher the percentage of all access operations that are writes in the client’s application scenario, the higher the performance degradation. This fact is visible in figure 1, where the curve characterizing rVsPes protocol for the application scenario with 0.8 ratio of writes to all operations is higher and grows faster than in the case of the application scenario with 0.2 write ratio. The important observations are related to rVsSG and rVsAll rollback-recovery protocols. They both log only operations received directly from clients. Thus, the number of taken checkpoints and the overall execution time for these protocols is significantly smaller than for rVsPes. In our experiments, rVsSG protocol takes a checkpoint only when the server’s log contains 10 operations, while in the case of rVsAll protocol, the moment of checkpointing server’s state is determined by the required session guarantee. In general, checkpoints in rVsAll protocol are taken every: – read operation r required with RYW, if, after the latest checkpoint, r was preceded in the server’s execution order by writes issued by the same client – second write w required with MW, if, since the latest checkpoint, the server has already performed one write issued by the same client – second read r required with MR, if, since the latest checkpoint, the server has already performed one read request issued by the same client having not empty set of relevant writes – write w required with WFR, if the latest checkpoint w follows in the server’s execution order a read operation issued by the same client and having not empty set of relevant writes. When clients issue significantly more reads than writes, the checkpoint overhead of rVsAll protocol is higher than of rVsSG. It comes from the fact that rVsAll protocol takes checkpoints also while receiving read requests issued with RYW or MR session guarantee. For the opposite client’s application scenario, where clients issue considerably more writes than reads, the execution time for rVsAll protocol is higher than for rVsSG one, only for the small number of clients. With the growing number of clients, the number of obtained requests also grows, so the execution time, and thus, checkpoint overhead becomes higher for rVsSG protocol. In the case of our experiment, the number of clients that influences the checkpoint overhead of rVsSG and rVsAll protocols is close to 200 (Fig. 1). As we mentioned in the Section 3, the moment of taking a checkpoint in the case of rVsAll protocol may be changed, and checkpoints may be taken every multiple of DMTC. Figure 2 presents a comparison of checkpoint overhead of the protocol rVsAll with checkpoints taken every DMTC, 5th -DTMC, 10th DTMC and 20th -DTMC. The obtained results depend on a number of clients in the system. For both types of the client’s application scenario, rescheduling the moment of taking a checkpoint results in the significant decrease of the

On Checkpoint Overhead in Distributed Systems

1800

17

3400 rVsAll rVsAll5 rVsAll10 rVsAll20

1700

0.20 0.20 0.20 0.20

rVsAll rVsAll5 rVsAll10 rVsAll20

3200 3000

0.80 0.80 0.80 0.80

1600 2800 1500 Execution time

Execution time

2600 1400

1300

2400 2200 2000

1200 1800 1100 1600 1000

1400

900

1200 0

50

100

150

200

250

300

350

400

0

50

100

150

200

Clients

250

300

350

400

Clients

Fig. 2. Checkpoint overhead of rVsAll protocol with checkpoints taken every 1, 5, 10, 20-DMTC for 0.2 and 0.8 application scenario 4000 rVsSG rVsSG rVsAll rVsAll rVsAll20 rVsAll20

3500

0.20 0.80 0.20 0.80 0.20 0.80

Execution time

3000

2500

2000

1500

1000

500 0

50

100

150

200 Clients

250

300

350

400

Fig. 3. Checkpoint overhead of protocols: rVsSG, rVsAll and rVsAll taking checkpoints every 20th -DMTC

checkpoint overhead, which is the effect of the decrease of checkpoint costs, when the checkpoints are taken less frequently. The experiments show that differences in the overhead of rollback-recovery protocols, where checkpoints are taken every nth -DMTC (where n = 5, 10, 20), are not high (Fig. 2). Our another observation concerns the fact that, in the case of 0.2 client’s application scenario, the checkpoint overhead of rVsSG protocol and of rVsAll protocols that take checkpoints every 5, 10, 20th-DMTC is lower than the one of rVsAll protocol taking checkpoints every single DMTC. Thus, a natural question regarding relationship between the overheads of the first two protocols arises. The comparison of the execution time of protocols rVsSG, rVsAll and rVsAll taking checkpoints every 20th -DMTC, for the 0.2 and 0.8 client’s application scenarios, is shown in Figure 3. Again, the obtained results depend on the number of clients in the system. In the case of both application scenarios, the execution time of rVsAll protocol that takes checkpoints every 20th -DMTC is the shortest, so the checkpoint overhead of this protocol is the lowest. The obtained results demonstrate that rVsAll protocol, for which the moment of taking a checkpoint

18

A. Danilecki, A. Kobusi´ nska, and M. Libuda

may be rescheduled, is an efficient approach for providing fault-tolerance for distributed applications with session guarantees (Fig. 3). The last experiment compares the performance of protocols taking checkpoints according to the requirements of only one, pre-defined session guarantee. The obtained results are shown in Figure 4. The overhead of rVsAll protocol is the highest in the case of 0.2 client’s application scenario, which results from the fact that this protocol takes checkpoints when both types of client’s requests are received: namely reads and writes. 1800

4000 VsSG rVsSG rVsAll rVsMR rVsMW rVsRYW rVsWFR

1700 1600 1500

0.20 0.20 0.20 0.20 0.20 0.20 0.20

VsSG rVsSG rVsAll rVsMR rVsMW rVsRYW rVsWFR

3500

0.80 0.80 0.80 0.80 0.80 0.80 0.80

Execution time

Execution time

3000 1400 1300 1200

2500

2000 1100 1000 1500 900 800

1000 0

50

100

150

200 Clients

250

300

350

400

0

50

100

150

200 250 Number of clients

300

350

400

Fig. 4. The execution time of rollback-recovery protocols taking checkpoints according to the requirements of only one of session guarantee for 0.2 and 0.8 application scenario

According to the considered application scenario, the overhead of rVsMR and rVsRYW protocols which take checkpoints on read operations is higher than for rVsMW and rVsWFR protocols. On the contrary, when we consider 0.8 client’s application scenario, then the overhead of the protocols rVsMW and rVsWFR is higher than the overhead of rVsMR and rVsRYW.

6

Conclusions

This paper has described the experimental performance evaluation of checkpointing and rollback-recovery protocols, done by estimating the checkpoint overhead. The obtained empirical results show directly that rollback-recovery mechanisms integrated with consistency protocols of session guarantees efficiently ensure that session guarantees are provided despite server’s failures. These results also clearly indicate that the proposed rVsSG, rVsRYW, rVsMW, rVsMR, rVsWFR and rVsAll protocols dominate the pessimistic rollback-recovery protocol. The advantage of the proposed protocols over rVsPes protocol results from the smaller number of operations saved by these protocols in the stable storage and thus their less frequent access to such a storage. However, the experiences with the simulation experiments show that the significant impact on the protocol performance has also the moment of taking a checkpoint, determined by both — the applied consistency model and the client application scenario. Rescheduling the moment of taking a checkpoint results in the significant decrease of the

On Checkpoint Overhead in Distributed Systems

19

checkpoint overhead of the proposed protocols. Thus, in the paper the evaluation of the cost of various moments of taking a checkpoint was done. Our future work is closely related with described in this paper experimantal results and encompassess the mechanism of adapting (tuning) rollback-recovery protocols to specific characteristics of clients requests.

References 1. Bruno, J., Coffman, E.: Moptimal fault-tolerant computing on multiprocessor systems. Acta Informatica 34, 881–904 (1997) 2. Plank, J., Li, K., Puening, M.: Tmdiskless checkpointing. IEEE Trans. Parallel and Distributed Systems 9, 972–986 (1998) 3. Ling, Y., Mi, J., Lin, X.: A variational calculus approach to optimal checkpoint placement. IEEE Trans. on Computers 50(7), 699–708 (2001) 4. Kobusinska, A.: Rollback-Recovery Protocols for Distributed Mobile Systems Providing Session Guarantees. PhD thesis, Institute of Computing Science, Poznan University of Technology (2006) 5. Terry, D.B., Demers, A.J., Petersen, K., Spreitzer, M., Theimer, M., Welch, B.W.: Session guarantees for weakly consistent replicated data. In: Proc. of the Third Int. Conf. on Parallel and Distributed Information Systems (PDIS 1994), Austin, USA, pp. 140–149. IEEE Computer Society, Los Alamitos (1994) 6. Guerraoui, R., Rodrigues, L.: Introduction to distributed algorithms. Springer, Heidelberg (2004) 7. Sobaniec, C.: Consistency Protocols of Session Guarantees in Distributed Mobile Systems. PhD thesis, Institute of Computing Science, Poznan University of Technology (2005) 8. Gorski, F., Kobusi´ nska, A., Marciniak, P., Plaza, M., Stempin, S.: Das - the distributed algorithms simulator. Technical Report RA-021/06, II-PP (2006)

Performance Evolution and Power Benefits of Cluster System Utilizing Quad-Core and Dual-Core Intel Xeon Processors Pawel Gepner, David L. Fraser, and Michal F. Kowalik Intel Corporation {pawel.gepner,david.l.fraser,michal.f.kowalik}@intel.com

Abstract. Multi-core processors represent an evolutionary change in conventional computing as well setting the new trend for high performance computing. The chip-level multiprocessing architectures with a large number of cores continue to offer dramatically increased performance and power savings characteristics. Energy efficiency and scalability in performance have become more important to many enterprises and play important role in a cluster environment as well. This paper will describe how much we may expect from the cluster systems in terms of performance and power saving if we based the installation on the servers which are founded on the Intel Xeon Quad-Core and Intel Xeon DualCore processors family. Keywords: HPC, multi-core processors, dual-core processors, quad-core processors, parallel processing, benchmarks.

1

Introduction

Since the launch of the first microprocessors based on Intel Core Microarchitecture in June 2006, the high-performance computing (HPC) community have had an interesting choice. The new Xeon processor based on this energyefficient philosophy with the performance driven architecture delivers not only record performance but also significantly reduces the power. The HPC industry is becoming increasingly aware of the impact that power consumption has on the total cost of HPC installation. Operational cost produced as a result of running the cluster and cooling it in a huge server room, or maintaining specialized buildings to prevent system failure increase as server power consumption levels rise. New metrics of HPC installation success are no longer focused just on a pure performance, but rather on delivering a system which provides leadership in both raw performance and on performance per watt. The new generation of Intel processor Xeon Dual-Core and Xeon Quad- Core are focused on providing leadership in both aspects and both benefit a broad spectrum of applications including HPC optimized code. R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 20–28, 2008. c Springer-Verlag Berlin Heidelberg 2008 

Performance Evolution and Power Benefits of Cluster System

2

21

Performance and Performance-Per-Watt Consideration

True performance is a combination of both clock frequency and Instruction Per Clock (IPC). This shows that the performance can be improved by increasing frequency and IPC. The frequency is a function of both the manufacturing process and the microarchitecture. On our existing 65nm CMOS process technology and micro-architecture optimized for that frequency (e.g. long pipelining design) such as NetBurst we can achieve today 3.8 GHz maximally. Unfortunately high clock ratio has some implications in power consumption. If we analyze the NetBursts based processors running today we observe highest available speed 3.8 GHz and the thermal guideline 115 W. Dealing with such a thermal consideration is not an easy task. Assuming that a new process technology, (45 nm CMOS), will change the situation dramatically is wrong. Unfortunately leakage power limits frequency scaling (Figure 1) and it is the most important constraint of frequency acceleration.

Fig. 1. Leakage Power (% of total) vs. process technology

If the frequency can not be easily accelerated and it is an issue to deal with the thermal management then we need to focus on increasing instructions per clock, whilst fitting within an acceptable thermal envelope. Pure performance is important but we need to always consider the implications on power, when measuring the performance of the system. More and more we look for the best ratio of performance per watt. If the power consumption is related to the dynamic capacitance, the square of the voltage with which the transistors and I/O buffers are supplied times the frequency at which the transistors and signals are switching then we can express: power = (dynamiccapacitance) ∗ voltage2 ∗ f requency

(1)

22

P. Gepner, D.L. Fraser, and M.F. Kowalik

Taking into account performance and power equations, CPU designers need to balance IPC efficiency from one side and voltage and frequency from the other to offer a compromise of performance and power efficiency of the processor.

3

Multi-core Processors

In addition to all the methods described above and all of the considerations we have discussed so far, there is also one extra way to build a high performance system. Dual and multi-core processor systems are going to change the dynamics of the market and enable new innovative designs delivering high performance with an optimized power characteristic. They drive multithreading and parallelism at a higher than instruction level, and provide it to mainstream computing on a massive scale. From an operating system level (OS) they look like a symmetric multiprocessor system (SMP) but they bring a lot more advantage than typical dual or multi processor systems which we know from the classic server architecture. Multi-core processing is a long-term strategy for Intel that began more than a decade ago. Intel has more than 15 multicore processor projects underway and we are on track to deliver multi-core processors in high volume across multiple platform families. Intels multi-core architecture will possibly feature dozens or even hundreds of processor cores on a single die. In addition to general-purpose cores, Intel multi-core processors will eventually include specialized cores for processing graphics, speech recognition algorithms, communication protocols, and more. Many new and significant innovations designed to optimize the power, performance, and scalability are implemented into the new multi-core processors. How all of these innovations reflect to the overall system performance and accelerating performance per watt ratio is described below. In the testing environment we have been testing single system performance and performance per Watt as well as the cluster configuration as found in a typical HPC workload.

4

Selected Benchmarks and Platforms Configuration Details

Benchmarks selected for the testing environment contain: Fluent, LS-DYNA, Amber, Star-CD and LINPACK. These application and synthetic benchmarks represent a board spectrum of HPC workloads and seem to be a typical representation of testing suite for this class of calculation. Fluent is a commercial engineering application used to model computational fluid dynamics. The benchmark consists of 9 standard workloads organized into small, medium and large models. These comparisons use all but the largest of the models which do not fit into the 8GB of memory available on the platforms. The Rating, the default Fluent metric, was used in calculating the ratio of the platforms by taking a geometric mean of the 8 workload ratings measured.

Performance Evolution and Power Benefits of Cluster System

23

LS-DYNA is a commercial engineering application used in finite element analysis such as a car collision. The workload used in these comparisons is called 3 Vehicle Collision and is publicly available from http://www.topcrunch.org/. The metric for the benchmark is elapsed time in seconds. Amber is a package of molecular simulation programs. The workload measures the number of problems solved per day (PS) using eight standard molecular dynamic simulations. See http://amber.ch.ic.ac.uk/amber8.bench1.html for more information Star-CD is a suite of test cases, selected to demonstrate the versatility and robustness of STAR-CD in computational fluid dynamic solutions. The metric produced is elapsed seconds (wall clock) converted to jobs per day. For more information go to http://www.cd-adapco.com/products/STAR-CD LINPACK is a floating-point benchmark that solves a dense system of linear equations in parallel. The metric produced is Giga-FLOPS or billions of floating point operations per second. The benchmark is used to determine the world’s fastest computers published at the website http://www.top500.org/. Performance was measured using these benchmarks comparing system configurations of Quad-Core Intel Xeon Processor X3220 (2.40 GHz, 8 MB L2 cache, 1066 MHz FSB) to Intel Xeon processor 3070 (2.67 GHz, 4 MB L2 cache, 1066 MHz FSB) and to Intel Pentium D processors 950 (3.40 GHz, 2 MB L2 cache, 800 MHz FSB). Configurations details: – Quad-Core Intel Xeon Processor X3220 based platform: Dell PowerEdge 860 using Quad-Core Intel Xeon processor X3220 (2.40 GHz, 8 MB L2 cache, 1066 MHz system bus), 4 x 1GB (dual-ranked) ECC 533MHz DDR2 SDRAM; Red Hat Enterprise AS Linux 4, Update 3, EM64T, TR673 – Intel Xeon Processor 3070 based platform: Intel preproduction customer reference board Whitney A1 BIOS EXTWM212 with Intel Xeon processor 3070, 2.67 GHz with 4M L2 Cache, 1066 MHz system bus) 4 x 1GB (dualranked) ECC 533MHz DDR2 SDRAM DDR2 ; HW Prefetch Enabled. Red Hat Enterprise AS Linux 4, Update 3, EM64T, TR673. – Intel Pentium D Processor 950 based platform: Intel SR1475NH1-E E7230 chipset server with Intel Pentium D Processor 950, (3.40 GHz with 2x 2M L2 Cache, 800 MHz system bus), 4GB (4x1GB) 533MHz DDR2 ; Red Hat Enterprise AS Linux 4, Update 3, EM64T, TR626.

5

Comparing Single Processor Performance

In this section we have focused on single processor performance based on different type of CPU microarchitecture operating in typical HPC workload. We have compared NetBurst based dual core processor Intel Pentium D Processor 950 versus CPUs based on Intel Core Microarchitecture like Intel Xeon processor 3070 We also observe how the new microarchitecture accelerates performance in the HPC environment and how different dual core products can scale. We may

24

P. Gepner, D.L. Fraser, and M.F. Kowalik

Fig. 2. LINPACK: Dense Floating-Point Operations

also see how Quad-Core Intel Xeon processor X3220 benefits HPC application comparing to dual core CPUs. Using LINPACK HPL we see 50% performance improvement between system based on NetBurst CPU and Intel Core Microarchitecture. Both products are dual core but Intel Xeon 3070 significantly improves floating-point calculation utilizing 128 Bit SSE instructions and result is very visible. Quad Core product performance is even more spectacular we see 250% performance improvement versus Intel Pentium D Processor 950 and 66% versus Intel Xeon 3070. This shows great scaling for applications where we solve a dense system of linear equations in parallel. Molecular simulations also show great scaling.

Fig. 3. Amber: Molecular Modeling Application

Performance Evolution and Power Benefits of Cluster System

25

Fig. 4. Fluent: Computational Fluid Dynamics Application

Fig. 5. Star-CD: Computational Fluid Dynamics Application

Amber shows almost double performance advantage when we use dual core and quad core products. As we see figure 3 Intel Core Microarchitecture brings 30% improvement versus product based on NetBurst. In the computational fluid dynamics modeling Quad core can deliver between 80% and 210% more then dual core based product and the new Intel Core Microarchitecture brings 40% of improvement comparing NetBurst Dual core to Intel Core Microarchitecture core based dual core. Automotive HPC departments may find the multi core solution very useful as well. Commercial engineering application used in finite element analysis such

26

P. Gepner, D.L. Fraser, and M.F. Kowalik

Fig. 6. LS-Dyna: Finite Element Analysis for Crash Simulations

as a car collision LS-Dyna shows more then 60% performance improvement on Quad core vs. Intel Pentium D 950 also Intel Xeon 3070 gives 20% performance increase than previous generation Dual Core products.

6

Cluster Performance Study

Cluster configuration is built on 4 typical two processors nodes connected with Gigabit Ethernet. The cluster was installed using Rocks Cluster Package. Intel Xeon Processor 5150 has been selected for node population. We have been testing

Fig. 7. Cluster speedup across benchmarks

Performance Evolution and Power Benefits of Cluster System

27

scalability of the system running 4, 8, and 16 cores (threads) in one, two, four servers respectively. Figure 7 shows the results of the test running Benchmarks HPL, Fluent, Amber, Star-CD. As we can see Fluent and HPL will scale very well because of the nature of those benchmarks. Both of them are highly CPU intensive benchmarks and the type interconnect used does not play an important role. Star-CD is more communication intensive benchmark and with Gigabit as the interconnect interface we see less scalability. Changing the interconnect type to a faster interface solution will definitely deliver better scaling. So for compute intensive benchmarks we see great scalability but also for communication intensive benchmarks dual core are also a good solution, because of the reduced number of nodes. As we can see degradation inside the node is less observable than outside when we connect nodes. The gap between one and two nodes is not big like the transition from two to four when interconnect plays a more important role.

7

Performance/Watt Consideration

Intel Core Microarchitecture was designed in principle to drive energy-efficiency. Quad-Core based product delivers better performance than Dual-Core on the same platform. Generally we get problems solved quicker and save money on energy. Taking in to consideration that both products have a small difference in thermal TDP we may anticipate the results. Figure 8 shows how Quad-Core Intel Xeon Processor provide best in class performance and energy efficiency in a single socket configuration in typical HPC benchmark workloads.

Fig. 8. Performance per system watt - Dual Core and Quad Core comparison

8

Conclusion

With the release of the first dual-core processor we have entered a new era of processor architecture. Dual-core and multi-core processors will become the

28

P. Gepner, D.L. Fraser, and M.F. Kowalik

standard for delivering greater performance, improved performance per watt, and new capabilities. New Multi-core products provide a tremendous advantage in the HPC environment. We see a significant performance advantage, ranging from 20% - 210% on single CPU system performance and in addition observe significant performance per watt improvement. In a cluster environment we see great scaling when comparing to clusters built on previous generations of processor architecture, resulting in a reduction in the number of nodes necessary to achieve the intend performance level Multi-core processors do more work per clock cycle, and can be designed to operate at lower frequencies than their singlecore counterparts. All of this drives significant improvements in user experiences for the HPC environment and the same time extends Moores Law well into the future.

References 1. Gepner, P., Kowalik, M.F.: Multi-Core Processors: New Way to Achieve High System Performance. In: PARELEC 2006, pp. 9–13 (2006) 2. Ramanathan, R.M.: Intel Multi-Core Processors: Leading the Next Digital Revolution, Technology @ Intel Magazine 3. Wechsler, O.: Inside Intel Core Microarchitecture: Setting New Standards for Energy- Efficient Performance, Technology @ Intel Magazine 4. Smith, J.E., Sohi, G.S.: The Microarchitecture of superscalar processors. Proc. IEEE 83, 1609–1624 (1995) 5. Ronen, R., Mendelson, A., Lai, K., Lu, S.-L., Pollack, F., Shen, J.P.: Coming challenges in microarchitecture and architecture. Proc. IEEE 89, 325–340 (2001) 6. Moshovos, A., Sohi, G.S.: Microarchitectural innovations: Boosting microprocessor performance beyond semiconductor technology scaling. Proc. IEEE 89, 1560–1575 (2001) 7. Tullsen, D.M., Eggers, S.J., Levy, H.M.: Simultaneous multithreading: Maximizing onchip parallelism. In: Proc. 22th Annu. Int. Symp. Computer Architecture, pp. 392–403 (1995) 8. Eggers, S.J., Emer, J.S., Levy, H.M., Lo, J.L., Stamm, R.L., Tullsen, D.M.: Simultaneous multithreading

Skip Ring Topology in FAST Failure Detection Service Jacek Kobusiński, Filip Gorski, and Stanisław Stempin Institute of Computing Science Poznań University of Technology, Poland {jkobusinski,fgorski}@cs.put.poznan.pl,[email protected]

Abstract. This paper addresses the problem of communication among loosely coupled groups of nodes in distributed systems. We describe a novel proposal of logical communication topology based on skip list data structure. We enhance this structure to make it more resilient to failures. Its good self-stabilization characteristics are shown through extensive simulation experiments. We present this new concept in the context of our failure detection service, where we use it at a local communication level. Keywords: failure detection, distributed systems, fault tolerance, probabilistic communication.

1

Introduction

The use of distributed systems as a computing platform constitutes a very promising research and business area due to their availability, economic aspects and scalability. The intense development of Grids, P2P networks, cluster and high-speed network topologies gave the possibility to allocate an enormous amount of resources to distributed applications at a reasonably low cost. Their level of parallelism can improve the performance of existing applications and raise the processing power of distributed systems to a new higher level. Unfortunately, those systems are failure prone and the probability that a failure occurs during computations is higher than in traditional systems. Moreover, failures of underlying communication network have a great impact on system behavior and condition. There are many techniques to prolong the time in which the component will work correctly or even avoid the failure by introducing new rigorous technology procedures. These efforts are often very expensive and cannot ensure the full reliability of the system. Therefore, to overcome the problem one should construct a fault tolerant mechanism to detect unavoidable failures and make their effects transparent to a user. Fault tolerance can be achieved by introducing a certain degree of redundancy, either in time or in space. A common approach consists in replicating vulnerable components of the systems. The replication can guarantee, to a large 

This work was supported by France Telecom, under project "Brain" No. 21/06.

R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 29–38, 2008. c Springer-Verlag Berlin Heidelberg 2008 

30

J. Kobusiński, F. Gorski, and S. Stempin

extent, continuous availability of resources and continuity of work, but must be complemented with a failure detection mechanism. Chandra and Toueg presented a concept of failure detectors [1], as an abstract mechanism that supports asynchronous system model . To imagine the basic idea behind this mechanism, consider two processes p1 and p2 . Process p1 using its failure detector module tries to evaluate if process p2 is crashed or alive. If failure detector’s response is crashed, we can say that process p1 suspects p2 . A failure detector that makes no mistakes and eventually suspects all failed processes is called perfect failure detector. However, this requirements are very difficult to be fulfilled in practice. So, one can weaken safety (completeness) or liveness (accuracy) property to construct unreliable failure detector [1], which can also be used to solve agreement problems in an asynchronous system. In [2] authors propose a new type of a failure detector, called accrual failure detector. It provides a non-negative accrual value representing the current suspicion level of the monitored process, which can be further interpreted by the application. A general survey of the above mentioned failure detectors can be found in [3,4,5]. The failure detection mechanism [6,7,8,9,10,11,12] can be build into a distributed application or implemented as a service. The first approach allows to adjust the parameters of the detection to certain needs. However, it must be implemented as an application integral component. When more applications want to use such a mechanism, there is no possibility to share it. Besides, the resources like CPU power or link bandwidth allocated by this mechanism are used inefficiently. Considering these drawbacks, the failure detection service seems to be a good alternative. In this approach one can create environment for end users with common services that can be shared among different users working concurrently. In our recent research we focus on building flexible, adaptable, scalable and timely (in short FAST) failure detection service, which is composed of the following modules: failure detection, inter-group communication and local communication. While the inter-group communication module is responsible for efficient message dissemination in large scale environment, the local communication module has to provide a fast, scalable and failure resilient communication mechanism, as well as support for failure detection and low maintenance cost. Unfortunately, the existing communication topologies do not fulfill these requirements. Thus, we propose a novel approach called skip ring, which we present in the context of FAST. The remaining part of the paper is structured in the following way. In section 2, we define the system model. Section 3 describes the architecture of the FAST failure detection service. Description and experimental results concerning the skip ring topology are presented in section 4. Finally, section 5 brings our concluding remarks.

2

System Model

Throughout this paper, a distributed system is considered. For reasons of complexity and overhead, we model the system as consisting of processes and

Skip Ring Topology in FAST Failure Detection Service

31

computers (hosts). We do not include a network, computer elements or low level software abstractions as monitored components because the primary objective is not to construct a service that will diagnose the cause of the failure but to enable the construction of a reliable distributed application. So, the system consists of a set of nodes connected by quasi-reliable communication channels which are subject to crash failures. It means that there is no message corruption, message loss, and finally no creation of spurious messages. We assume a crash-stop failure model of nodes and consider the system to be asynchronous because there exists no bound either on communication delays or on process speed. Moreover, according to [13], we assume that the network performs in a synchronous way “most of the time”, i.e. long stable periods, where the system behaves like a synchronous one and short instability periods, where no timing guarantees can be made, occur alternately. This phenomenon, called partial synchrony [14], justifies the use of a “weaker” failure detectors because their properties are eventually satisfied.

3

FAST Failure Detection Service

FAST failure detection service architecture is modular, flexible and open. It allows swapping any single component without the need of changing the whole service (Figure 1), as long as other components’ requirements are met. This feature can be used to compare different failure detection methods and communication schema.

Fig. 1. Fast service architecture

Fig. 2. Inter-group pattern

communication

Considering the scalability issues, we assume a twolevel structure. To reduce the number of messages exchanged over internetwork links and to speed up the process of dissemination we divide the entire system into groups of hosts. In general one can imagine the simplest architecture without any hierarchy, where all modules are replaced by single module. On the other hand, one can extend this architecture and add some hierarchy levels by adding new modules. In the following subsections we shortly present general assumptions and ideas concerning the three components of our service: failure detection, inter-group communication and local communication.

32

3.1

J. Kobusiński, F. Gorski, and S. Stempin

Failure Detection

Failure detection protocol is the key component of the designed service. It will use all other components to disseminate acquired information about node failures. A node can learn about other node failure in two ways, by detecting it by itself or by getting this information from others. While the former method is faster, it can incur high cost and suffer poor scalability. The latter method is based on distribution of monitoring responsibility among nodes. Alas, the drawback of this dispersal is latency arising from information dissemination between the monitor and other nodes. As scalability issues are more important, we decide to disperse the monitoring responsibility. Another important aspect is the type of information maintained and shared by failure detectors. It can be either positive (correct nodes) or negative (suspected nodes). Regardless of the method, the issue of incorrect suspicions has to be considered as failure detectors are unreliable. The former approach handles this issue transparently, while the latter should be enhanced by the routine of sending single message carrying positive information once a false suspicion is discovered. On the other hand, it is reasonable to expect a majority of nodes to be correct most of the time and therefore, the latter approach is less bandwidth and memory consuming. Taking these facts into consideration, we decided to apply negative information scenario. To increase the failure resilience of the proposed detection mechanism we impose a lower limit of m on the number of monitors for each node. Despite that, in order to avoid creating a single point of failure, we are interested in more or less equal distribution of the monitoring responsibility among all the nodes. Therefore, we restrict the number of monitored nodes for each failure detector to a multiple of m. Finally, to satisfy various requirements of a user’s application we decided to use the accrual failure detector [2]. Instead of binary (correct/suspected) decision, it provides a non-negative suspicion value, which can be further interpreted by a user’s application. The higher value it returns, the more probable the node crashed. This kind of failure detection requires that monitoring relationships be stable and do not change frequently. 3.2

Inter-group Communication

At the inter-group communication level, nodes are organized into groups based on some metric. In our case, it is similarity of IP addresses, which can be replaced with a more complex method if required. Each group is assigned a unique identifier — GID. When a group’s size reaches a value from a priori defined range, split process occurs. As a result, two disjoint groups are created from its members. Their GIDs are unambiguously derived from the GID of the original group. We expect the communication at this level to be scalable, fast and failure resilient. Despite the speed and reliability of all-to-all communication, it does not guarantee scalability. On the other hand, the hierarchical approach, which

Skip Ring Topology in FAST Failure Detection Service

33

guarantees scalability, is difficult to maintain due to its lack of failure resilience. With regard to the above facts, we decided to use gossiping, which is a randomized communication pattern that fulfills our requirements. We have evaluated many different gossiping patterns based on rumor age, timeout, probability and feedback mechanism. Moreover, we decided to use a cooperative bidirectional push-pull approach which makes the dissemination faster. Finally, we modify the basic algorithm to limit the number of messages sent while the dissemination efficiency remains high. This was possible due to combination of gossip aggregation and different characteristics of push-pull approach. While gossiping makes our service more resilient to any communication failure and easily adoptable to the dynamic changes of membership or network topology, it cannot guarantee deterministic broadcast to all nodes. To solve this problem, we additionally introduced a simple ring communication pattern that will be used only occasionally. The maintenance of this structure induces almost no extra cost since it is done during exchange of gossip messages. Each node maintains randomly chosen addresses of two other nodes that are members of adjacent groups (one per group). As nodes of one group with high probability have different contact addresses, there are many parallel connections between those groups. This fact is very positive in the context of self-stabilization and fault tolerance. Logical schema of described the inter-group communication is presented on Figure 2, where each cloud represents a group, arrows depict gossiping, while dotted arcs show a logical ring structure. 3.3

Local Communication

At the local communication level, we handle the issues of communication between the nodes constituting a single autonomous group, which is self-sufficient, i.e. it does not relay on any other components. The term group refers to a set of loosely coupled nodes, with a limited size and no guarantees about consistent group view or message delivery. We require that nodes made some long term connections due to reasons mentioned in section 3.1. We do not assume the complete knowledge about group membership or group size at any single node. However, it should be possible to estimate that size at least by some nodes in order to perform a split procedure mentioned above (section 3.2). In order to disseminate information among all the nodes in the group, we considered communication schema similar to those discussed in the context of inter-group communication (section 3.2). We think that all-to-all pattern is not acceptable due to lack of scalability. We also claim that gossiping does not satisfy our long term connection requirement. Thus, we decided to use a hierarchical structure, which offers acceptable message propagation delays. As existing solutions incur high cost, stronger guarantees about group membership or centralized knowledge, we propose a novel logical topology called skip ring, which is described in details in section 4.

34

J. Kobusiński, F. Gorski, and S. Stempin

Fig. 3. Skip list

4

Fig. 4. Skip ring

Skip Ring Topology

Our service uses the skip ring topology which is based on a skip list structure proposed in [15]. Skip list is similar to a balanced tree regarding good lookup performance (O(log n)) and does not suffer high costs of inserting and deleting items at the same time (Figure 3). This is achieved by constructing this ordered list from items of different levels randomly chosen by each element independently. Each item starts from level 1 and repeatedly tries to increase it with probability p (we assume p = 12 ) till the first unsuccessful try. It is easy to notice that the levels are randomly distributed in the following proportion: 50% are of level 1, 25% are of level 2, 12.5% are of level 3 and so on. An item that has l forward pointers is called a level l item. Each such a pointer points to the next element on the list of level l or higher. As mentioned earlier, we want to use the skip list as a communication pattern. The first modification we made was removing head and tail of the list and making it cyclic. This topology is used as a communication pattern at a group level, where items represent the group members and pointers stand for network links. This structure is fragile as nodes can fail. To increase the failure resilience, it should be enhanced by adding extra pointers. The final structure, called skip ring, augmented with those pointers to the previous items is presented in Figure 4. For clarity, newly added pointers are shown only for one selected element. Selection strategy and the number of backward pointers is a very important issue in the context of self-stabilization of our structure. We carry out a set of experiments and their results prove that the resilience to failure can be increased by choosing more than two (k > 2) backward pointers. The more we use, the more resilient to failure the skip ring structure is (Figures 5def). On the other hand, the cost of maintenance will grow as the number ofpointers increases.  Our theoretical analysis shows that k should be O log1/f n where n is the size of the group and f is the number of failed nodes. These theoretical results have been confirmed by empirical data obtained from simulations (Figures 5ef) where we present failure resilience of skip ring for groups of different size assuming the number of backward pointers scale accordingly to theoretical results. We have considered two alternative selection strategies of backward nodes: random (RND) and deterministic (LAST). In the former method, the node randomly chooses k other nodes (RND), while in the latter, the node selects the

100

100

80

80

% of stabilized runs

% of stabilized runs

Skip Ring Topology in FAST Failure Detection Service

60

40 f=1 f=5 f=10 f=20

20

0

0

10

60

40 f=1 f=10 f=10 f=20

20

20 30 40 50 60 70 80 90 Buffer size of known nodes as % of total nodes

0

100

0

(a) rnd k=3 m=25% 100

40

20

60

40

20

0

10

20

30

40

0

50

0

Concurrently failed nodes as % of total nodes

10

80

80

% of stabilized runs

% of stabilized runs

100

60

40 n=64 f=5 n=128 f=10 n=256 f=20 n=512 f=40 0

1

2 3 4 number of backward pointers

(e) last m=25%

30

40

50

(d) m=25%

100

20

20

Concurrently failed nodes as % of total nodes

(c) k=3

0

100

LAST k=1 LAST k=3 LAST k=5 RND k=1 RND k=3 RND k=5

80 % of stabilized runs

% of stabilized runs

20 30 40 50 60 70 80 90 Buffer size of known nodes as % of total nodes

(b) last k=3 m=25%

60

0

10

100

LAST m=25% LAST m=50% RND m=25% RND m=50% RND m=100%

80

35

5

60

40

LAST f=5 LAST f=10 LAST f=20 LAST f=30 LAST f=40 LAST f=50

20

6

0

0

1

2 3 4 number of backward pointers

5

6

(f) last m=25% f/n=const.

Fig. 5. Skip ring self-stabilization experimental results

closest k nodes behind (LAST). In both scenarios, the node chooses candidates from all locally known nodes. The experimental simulation shows that the second scenario is much better in the context of the self-stabilization property. All the experiments were conducted for groups of size 128 (n = 128), unless specified otherwise. We measured the percentage of stabilized runs for different parameters configuraitons. By “stabilized run” we mean a run, in which the structure overcomes the problem of nodes crashes and reconfigure properly.

36

J. Kobusiński, F. Gorski, and S. Stempin

Figure 5a presents the percentage of stabilized runs for RND scenario while changing the size of the locally known nodes buffer (m) for different number of concurrent failures (f ) and constant number of backward pointers (k = 3). It is easy to notice a strong relation between the failure rate f and the number of successfully stabilized runs. It is more difficult to achieve stabilization when f increases. For a given f this situation can be improved by extending the amount of information about other members of the ring gathered by each node. Figure 5b shows corresponding results for the LAST scenario. In this case, the relation between the failure rate f and the number of successfully stabilized runs is hard to observe, which is a consequence of a better fault resilience in such a skip ring. We can also see that the size of the buffer has a little influence on successful stabilization and for most configurations it would be unreasonable to keep it high. Comparing those two plots, it is evident that LAST scenario behaves better than RND scenario for any given f and buffer size. It is clear that a rational choice of the scenario can improve the self-stabilization property of the system while decreasing its memory requirements. We believe that there is possibility for further improvements in this area. In Figure 5c we present the percentage of stabilized runs (k = 3) for both scenarios, with different buffer sizes (25%, 50%, 100% for RND and 25%, 50% for LAST) while changing the percentage of concurrent failures. This plot demonstrates once more a better quality of LAST scenario over RND. We can also see that LAST for m = 50% is comparable to RND with full knowledge of the ring. Improvements in stabilization observed for LAST scenario are less significant comparing to RND scenario. We think it is reasonable to set m to 25% for further analysis of LAST scenario. In Figure 5d we analyze backward pointers impact on skip ring stabilization when (m = 25%) and failure rate changes. Not surprisingly, there is a strong impact for LAST scenario. The more backward pointers, the easier to stabilize the ring. It is also noticeable that selecting a sophisticated scenario like LAST should be connected with careful considering other parameter values, especially k. Data series for LAST with k = 1 shows almost no improvement to RND with k = 1. The following two plots highlight the relation between the success of stabilization and the number of backward pointers (k). First of them, (Figure 5e), presents a positive effect of increasing k for different sizes of the skip ring with the same percentage of concurrent failures and buffer size. It is evident that for constant f there exists a value k  such that for k > k  further stabilization improvements are insignificant. The same regularity can be observed on the next plot (Figure 5f), however, instead of variable ring sizes, we analyze various numbers of concurrent failures f . The shape of the plot suggests again that there exists the value k  and it is also clear that it depends on f . That would be an empirical validation of our expectations derived from the theoretical analysis.

5

Conclusion

Our recent studies concentrate on FAST — a failure detection service. In the paper, we described its architecture and characterized its components briefly.

Skip Ring Topology in FAST Failure Detection Service

37

During the design and implementation of FAST we tried to merge some promising techniques presented recently in the literature and overcome difficulties in finding appropriate topology for local communication. Therefore, we proposed a novel probabilistic communication pattern based on the skip list — the skip ring. We showed how to transform the skip list, a data structure, into the skip ring, a communication topology. As the next step we enhanced it with additional pointers to make it failure resilient. We also showed the results of theoretical and experimental studies of the proposed solution. They confirm that the proposed the skip ring is a scalable and failure resilient topology that provides fast message dissemination and incurs low maintenance cost. Moreover, it is designed to support failure detection. We believe that the skip ring is an interesting solution with very good characteristics, which make it suitable for a wide variety of distributed applications including the local management component of FAST. As the future work we plan to extend our studies concerning efficient ring size estimation, improved scenarios of backward pointer selection and integration of accrual failure detection mechanism into the skip ring.

References 1. Chandra, T.D., Toueg, S.: Unreliable failure detectors for reliable distributed systems. Journal of the ACM 43, 225–267 (1996) 2. Hayashibara, N., Défago, X., Yared, R., Katayama, T.: The ϕ accrual failure detector. In: SRDS, pp. 66–78 (2004) 3. Brzeziński, J., Kobusiński, J.: A survey of failure detector protocols. Foundations of Computing and Decision Sciences 28, 65–81 (2003) 4. Reynal, M.: A short introduction to failure detectors for asynchronous distributed systems. SIGACT News, 53–70 (2005) 5. Freiling, F., Guerraoui, R., Kouznetsov, P.: The failure detector abstraction. Technical Report TR 2006-003, Department for Mathematics and Computer Science, University of Mannheim (2006) 6. van Renesse, R., Minsky, Y., Hayden, M.: A gossip-based failure detection service. In: Proc. of the Int. Conf. on Distributed Systems Platforms and Open Distributed Processing, pp. 55–70 (1998) 7. Stelling, P., DeMatteis, C., Foster, I.T., Kesselman, C., Lee, C.A., von Laszewski, G.: A fault detection service for wide area distributed computations. Cluster Computing 2, 117–128 (1999) 8. Gupta, I., Chandra, T.D., Goldszmidt, G.S.: On scalable and efficient distributed failure detectors. In: Proc. of 20th Annual ACM Symp. on Principles of Distributed Computing, pp. 170–179. ACM Press, New York (2001) 9. Bertier, M., Marin, O., Sens, P.: Implementation and performance evaluation of an adaptable failure detector. In: Proc. of the Int. Conf. on Dependable Systems and Networks, Washington, DC, pp. 354–363 (2002) 10. Hayashibara, N., Cherif, A., Katayama, T.: Failure detectors for large-scale distributed systems. In: Proc. of the 1st Workshop on Self-Repairing and SelfConfigurable Distributed Systems (RCDS), 21st IEEE Int’l Symp. on Reliable Distributed Systems (SRDS-21), Osaka, Japan, pp. 404–409 (2002) 11. Dunagan, J., Harvey, N.J.A., Jones, M.B., Kosti, D., Theimer, M., Wolman, A.: FUSE: Lightweight guaranteed distributed failure notification. In: Proc. of the 6th Symp. on Operating Systems Design and Implementation, pp. 151–166 (2004)

38

J. Kobusiński, F. Gorski, and S. Stempin

12. Horita, Y., Taura, K., Chikayama, T.: A scalable and efficient self-organizing failure detector for grid applications. In: Proc. of 6th Int. Workshop on Grid Computing, pp. 202–210 (2005) 13. Cristian, F., Fetzer, C.: The timed asynchronous distributed system model. IEEE Trans. on Parallel and Distributed Systems 10, 642–657 (1999) 14. Dwork, C., Lynch, N., Stockmeyer, L.: Consensus in the presence of partial synchrony. Journal of the ACM 35, 288–323 (1988) 15. Pugh, W.: Skip lists: a probabilistic alternative to balanced trees. Communication of the ACM 33, 668–676 (1990)

Inter-processor Communication Optimization in Dynamically Reconfigurable Embedded Parallel Systems Eryk Laskowski1 and Marek Tudruj1,2 2

1 Institute of Computer Science PAS, 01-237 Warsaw, Ordona 21, Poland Polish-Japanese Institute of Information Technology, ul. Koszykowa 86, 02-008 Warsaw, Poland {laskowsk,tudruj}@ipipan.waw.pl

Abstract. The paper concerns program design methods for a new kind of parallel embedded systems in which communication infrastructure is dynamically run-time adaptable to particular application program needs. The new system architecture assumes a fairly large number of autonomous communication links in each executive processor. Interprocessor link connections are subdue to dynamic reconfiguration according to the compile-time elaborated strategy based on the application program graph analysis. Automatic program structuring methods are used for defining the structuring of reconfigurable processor link sets which enables the look-ahead connection reconfiguration that overlaps with the current program execution including data communication. Algorithms for respective program task scheduling and dynamic program decomposition into sections executed with the look-ahead created connections of subsets of processor links are presented. Simulation experiment results with structuring of parallel numerical programs of matrix multiplication are presented. The experiments compare program structuring quality of the look ahead connection reconfiguration based on multiple crossbar switches with the quality of reconfiguration in a single crossbar switch but with the use of multiple link subsets reconfigured in advance.

1

Introduction

In previous papers we have investigated novel message-passing circuit-switching system architecture based on redundant communication resources [3, 5, 6]. This architecture involves new program execution paradigm based on the look-ahead dynamic reconfiguration of inter-processor connections. It is based on preparing inter-processor connections in advance in spare (redundant) communication resources (e.g. additional crossbar switches). The experimental results have shown [5, 6] that the look-ahead reconfiguration in redundant communication resources gives very good speedups, especially for fine-grain parallel programs. The look-ahead reconfiguration is particularly beneficial when applied in embedded parallel systems based on user-level communication [1]. The user-level R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 39–48, 2008. c Springer-Verlag Berlin Heidelberg 2008 

40

E. Laskowski and M. Tudruj

communication provides a very low software overhead (sometimes as low as 1 CPU cycle time, e.g. DIMMnet network interface [2]) through accessing the inter-processor network directly from user program without intervention from the operating system. This imposes the tough requirements for efficiency of interprocessor connection setting in high performance parallel embedded systems. It is interesting to check whether it is possible to exploit some ideas coming from the look-ahead reconfiguration paradigm in standard message-passing parallel systems based on single reconfigurable communication network. The lack of network hardware redundancy in standard systems directs our effort to a still better reconfiguration control approach, namely to a careful program graph structuring in a non-redundant communication infrastructure. In this paper, we propose a new strategy of inter-processor connections reconfiguration control for message-passing multiprocessor systems, where application programs are partitioned into sections executed with fixed inter-processor connections to reduce reconfiguration control time overheads. In [7] the first attempt to use the look-ahead reconfiguration paradigm in standard message-passing system was presented. Unfortunately, the solution presented there was based on static processor link sets partitioning, which gave disappointing speedups in comparison to classical on-request approach. The paper presents how program execution control, which has originated from the look-ahead reconfigurable systems can be applied in a classic circuitswitching on-request reconfigurable parallel system. Although the program execution paradigm of look-ahead reconfiguration is closely connected to/inherently based on hardware redundancy, it is possible to use specific program execution control based on partitioning of an application program into sections in standard hardware environment. Thanks to this, we can expect to achieve speedup improvements without investment into sophisticated, and thus expensive, system architecture. In the paper, we show that parallel program graph partitioning for optimized reconfiguration control is justified and confirmed by the results of experimental research. The paper consists of three parts. In the first part, the inter-processor connection reconfiguration principles are discussed. In the second part, main features of the applied graph partitioning algorithm are discussed. In the third part, experimental results of efficiency measures of program execution based on the dynamic reconfiguration are shown and discussed.

2

Look-Ahead Dynamically Reconfigurable Multiprocessor Systems

In the paper, we discuss multiprocessor systems with distributed memory and communication based on message passing. In such systems, reconfigurable through inter-processor connection switching, before a message can be sent, the connection of the sender processor link with the receiving processor link has to be created. With the dynamic on request reconfiguration each communication in an application program provokes a connection request directed to the

Inter-processor Communication Optimization

41

operating system. The system creates the requested connection and acknowledges the requesting processes. When the communication is completed, link release messages are sent to the operating system to enable new connections. The look-ahead dynamic connection reconfiguration paradigm is based on anticipated connection setting in some redundant communication resources provided in the system. Some subsets of the redundant resources are allocated to program execution and some others to the look-ahead connection setting. The redundant resources considered in this paper are multiple processor link sets and, for comparison purposes, link connection switches (crossbar switches, multistage connection networks). In look-ahead dynamic reconfigurable systems an application program is divided into communication-disjoint sections. Inter-processor connections for currently executed sections are created in advance in some connection switching devices. The connections do not change during execution of the current sections. In parallel with the execution of current sections, a configuring control process sets the connections for next sections in redundant connection switches. The partitioning of application programs into sections is performed automatically at compile time, based on an analysis of the application program graph. 2.1

Systems with Many Link Connection Switches

Creation of link connections in parallel with communication can be enabled by the use of several link connection switches (crossbars) for program execution and anticipated connection setting. The general structure of a look-ahead link connection reconfigurable system with several connection switches is shown in Fig. 1. When WPi executes a current program section using links connected in one connection switch, the global control subsystem (GCS) sets, in another switch, connections required for execution of next sections. GCS controls the link set config.

Crossbar

switch

S1

Crossbar

switch

...

switch S2

...

config.

Crossbar

... ... ... ...

links

config.

SX

links

... ... ... ... ...

... ... ... ...

Processor link set switch

...

links

WP1

...

links

WP2

control

...

... ... ...

links

...

links

WPN

Synchronization Path

GCS Communication Control Path

Fig. 1. Structure of the look-ahead reconfigurable system with multiple connection switches

42

E. Laskowski and M. Tudruj

switch, which is used to change connections between the WPi ’s links and the crossbar switches. When the connections for next sections are ready and the links of WPi will be no more used in the current section, the link set switch connects WPi links to the configured connection switch. We assume that switching time of the link set switch is very small comparing the connection setting time in the crossbar switch, so new connections are made available to program execution in almost time transparent way. Depending on the strategy used for program execution, the link set switch can switch all or selected links of a worker processor subsystem. The control bus connects worker processor subsystems with the global control subsystem. It is used for transferring section completion (end of use of links) messages from WPi to GCS and section activation messages from GCS to WPi . The simplest implementation of the Communication Control Path is a bus but a more efficient solution can assume direct links (data lines) connecting worker processors with the GCS. Besides the Communication Control Path, the Synchronization Path is used in this system. The path is used to synchronize end of section reports coming from worker subsystems. 2.2

Systems with Multiple Processor Link Sets

Instead of multiple connection switches it is possible to use multiple processor link sets connected to a single link connection switch (e.g. a single crossbar). In such a configuration, links of each worker processor subsystem are divided into two subsets: one used for the execution of current program sections and the other used for the reconfiguration. The complete structure of the look-ahead reconfigurable system with partitioned processor link sets is shown in Fig. 2. This solution requires less hardware, but, it requires processors (WPi ) with a larger link connection switch controlled by the global control subsystem (GCS). The worker processor subsystems (WPi ) are controlled by GCS using a communication control path. It can be a bus, which transfers the end of link use/section activation messages.

config.

Crossbar switch S

links c.l.

... ... w.l

links c.l.

WP1

...

... ... w.l WP2

links c.l.

...

... ... w.l WPN

... Communication Control Path w.l. – working links

GCS control c.l. – configured links

Fig. 2. Look-ahead reconfigurable system with partitioned processor link sets

Inter-processor Communication Optimization

43

The partitioning of processor links into the subset of working links and subset of configured links can be done using two strategies. The set of all processor links is statically divided into two equal size subsets [7] or the set is dynamically divided into two usually non-equal size subsets depending on the current program needs. In the paper we will examine the performance obtained with the dynamic link partitioning strategy in simulation experiments and we will compare them to the strategy based on statically divided two equal size subsets of processor links.

3

Parallel Program Scheduling and Partitioning into Sections

An application program is represented as a weighted Directed Acyclic Graph (DAG) with computation task nodes and communication (i.e. node data dependency) edges. Programs are executed according to the macro data flow model. The graph is static and deterministic. A two-phase approach is used to tackle the problem of scheduling and graph partitioning in presented environments [5, 6]. In the first phase, a scheduling algorithm reduces the number of communications and minimizes program execution time, based on program DAG. The scheduling algorithm is an improved version of ETF (Earliest Task First) scheduling [4]. In the second phase, scheduled program graph is partitioned into sections for the look-ahead execution in the assumed environment. A scheduled program is specified by the Assigned Program Graph (APG) [5,6], Fig. 4. Program graph partition into sections is defined using the Communication Activation Graph (CAG) [5,6]. Program sections correspond to such sub-graphs in the CAG for which the following conditions are fulfilled: 1. section sub graphs are mutually disjoint in respect to communication edges connecting processes allocated to different processors, 2. sub-graphs are connected in respect to activation and communication edges, 3. inter-processor connections do not change inside a section sub-graph, 4. section sub-graphs are complete in respect to activation paths and include all communication edges incident to all communication nodes on all activation paths between any two communication nodes which belong to a section, 5. all connections for a section are prepared before this section activation, which simultaneously concerns all processors involved in section execution. 3.1

Program Graph Partitioning into Sections

In the second phase of the program structuring, the program graph partitioning heuristics is used to find program sections, Fig. 3. Besides, the heuristics assigns a crossbar switch to each section. The algorithm starts with an initial partition, which consists of sections built of single communication and assigned to the same crossbar switch. In each step, a vertex of CAG is selected and then the algorithm

44

E. Laskowski and M. Tudruj

Begin Initialize the set of sections /each section is composed of a single communication and assigned to a crossbar 1/ While the stop condition is not satisfied /stop condition: each vertex of CAG is visited, no execution time improvement during given number of steps/ Select a vertex v of CAG, which maximizes the selection function and which is not placed in tabu list Try to merge section of v with some sections of predecessors of v Select the merging, which gives the shortest partitioned program execution time EndWhile End

Fig. 3. The general scheme of the graph partitioning algorithm

tries to include this vertex to a union of existing sections determined by edges of the current vertex. The heuristics tries to find such a union of sections, which doesn’t break rules of graph partitioning. The union, which gives the shortest program execution time is selected. See [5, 6] for detailed description of the partitioning heuristics. 3.2

Simulated Execution of Programs

The program execution time is obtained during the graph structuring by simulated execution of the partitioned graph in a look-ahead (or on-request) reconfigurable system. A simulator for symbolic execution of APG graphs, written in C language, has been used. For simulated execution of the program graph in a look-ahead reconfigurable system, an APG graph with a valid partition is extended by subgraphs, which model the look-ahead reconfiguration control, Fig. 4. The functioning of the Communication Control Path, Synchronization Path and the Global Control Subsystem GCS are modeled as subgraphs executed on virtual additional processors. Weights in the graph nodes correspond to latencies of respective control actions, such as crossbar switch reconfiguration, bus latency, connection setting and activation of program sections. These weights are expressed as control parameters of the simulation (tS , tF , tR , tA ).

4

Experimental Results

During experiments we performed evaluation and comparative studies of execution efficiency of programs for different program execution control strategies and system parameters. We have examined program execution speedup versus parameters of reconfiguration control (tR and tV ), the number of crossbar switches, the number of processor links, for the parallel program of Strassen matrix multiplication (two recursion levels). The results presented in the paper are obtained

Inter-processor Communication Optimization

P1 10

S1

5

L1 1

L1

P3

P4

P5

20

30

5

5

S2

5 5

5 5

P2

L1 5 L1

L1 2

L1

5

10

L1 5

5 L2

L2

P6 10 L1 5

10

4

5 5

5

10

end of link use messages

3

5 L2

6

Control Bus Process

10 L2

5

15 15

L2

L2

7

5

tS L2

10

8

L1

S3

5

10 L1 9

10 L2

S4

L1 5 L1

5

10 5 L1 L2

5

5

10

S6 20

14

10 L2

L2 L1

L2 11

10 5

10

L1

S5

10

12

5

15

tF

10 15

10

20

45

L2

13

L1

120 5

activation messages

synchronisation for activation

tR tA

15

L1 20

Activation Process

Reconfiguration Process

Fig. 4. Modeling the reconfiguration control in an APG

for two graphs of Strassen matrix multiplication program: Strassen (two recursion levels), Strassen(cg) (Strassen multiplication with coarse–grain parallelism – one recursion level). Table below shows parameters of program graphs (the average distance between reconfigurations and the granularity, defined as the communication/computation ratio): graph name nb. of nb. of commu- granularity vertices nications Strassen 209 352 1.2 Strassen (cg) 27 44 4.2

distance between reconfigurations 80-130 500-1300

Program execution was simulated using the look-ahead paradigm in the following architectural variants: a system with dynamically partitioned multiple processor link sets (multiple-link-sets), a system with single connection switch with on-request dynamically established connections (on-request ). The number of processors (4, 8, 12) and the number of processor links (2, 4, 8) were used as basic parameters in the experiments. Additional parameters were: tR – reconfiguration time of a single connection in the range 1-100 and tV – section activation time overhead (tV = tF + tS + tA , for components of tV see Fig. 4) in the range 1-100. Fig. 5 shows maximal speedup for fine-grain version of Strassen program graph. Both architectures obtained similar maximal speedup values. This outcome was obtained for very small values of time parameters representing reconfiguration control overhead (tV ) and creation of a single connection (tR ). For these very small values of tV and tR , the general impact of overheads coming from reconfiguration control on total execution time of the program is minimal or even negligible. Thus, regardless the efficiency of reconfiguration control, both architectures give similar results. Fig. 5 shows also the average speedup for wide range of values of parameters tV and tR . The range of tR and tV used during experiments included values for which

46

E. Laskowski and M. Tudruj

Speedup 10

Max (multiple-link-set) Max (on-request) Ave (multiple-link-set) Ave (on-request) Tr=1, Tv=18 (multiple-link-set) Tr=1, Tv=18 (on-request)

9 8 7 6 5 4 3 2 1 0 2

4

2

4

4

2

8

4 12

Fig. 5. Maximal and average speedup for Strassen program for full range of tR and tV , and speedup for tR = 1, tV = 18

the efficiency of reconfiguration control is very low, thus the average speedup was substantially lower than maximal. The architecture based on multiple link sets yields better results in presence of higher reconfiguration time overheads, due to more effective reconfiguration subsystem based on optimized program control and section partitioning. The analysis of experimental results for different values of tV and tR reveals that the better results of multiple link set architecture were obtained when reconfiguration control overhead (tV ) had significant influence of total program execution time. An example of such a situation is shown in Fig. 5, where the speedup for selected values of tR = 1 (very small value) and tV = 18 (very big value) is presented. It should be noticed that for these values of tV and tR parameters, the system based on classical on-request reconfiguration control completely fails, especially for higher number of processors. In the system based on multiple link sets, the number of control messages is reduced, comparing the classical on-request control. So, for the higher number of processors in the system (and thus the higher number control messages transmitted), the efficiency of reconfiguration control is preserved for bigger values of tV parameter.

Speedup 7

Max (multiple-link-set) Max (on-request) Ave (multiple-link-set) Ave (on-request) Tr=10, Tv=48 (multiple-link-set) Tr=10, Tv=48 (on-request)

6 5 4 3 2 1 0 2

4 4

2

4 8

2

4 12

Fig. 6. Maximal and average speedup for Strassen (coarse-grain) program for full range of tR and tV , and speedup for tR = 10, tV = 48

Inter-processor Communication Optimization

47

Speedup 3,5 3 2,5 dyn. mls stat. mls 4 crossbars on-request

2 1,5 1 0,5 0 4

8

12

Fig. 7. Comparison of average speedup of Strassen program in look-ahead reconfigurable system (4 crossbars – 4 crossbar switches), dynamically partitioned multiplelink-sets (dyn. mls), statically partitioned multiple-link-sets (stat. mls), on-request system (on-request)

The afore-mentioned results were confirmed for coarse-grain version of Strassen program graph (Fig. 6). The coarse-grain version of Strassen graph yields lower values of maximal speedup since we were unable to parallelize it for the number of processor bigger than 8 (the same results were obtained for 8 and 12 processors, Fig. 6). Using the coarse-grain parallelization entails smaller impact of reconfiguration control overheads on total program execution time. Fig. 6 shows that the average speedup was similar to the maximal. So, as expected, the difference between classical on-request reconfiguration control and multiple link set control is smaller than for fine-grain parallelism, but still the architecture based on multiple link sets gives better results. Fig. 7 shows the average speedup of the fine-grain Strassen program for all system architectures discussed in the paper.

5

Conclusions

In the paper, we have discussed the idea of the look-ahead dynamic connection reconfiguration in multi-processor systems implemented as parallel embedded systems whose communication behavior is tailored to program needs. We have shown that the system based on multiple link sets and optimized reconfiguration control through program graph partitioning into sections executed with fixed inter-processor connections gives better speedups than the classic on-request reconfiguration control, especially when the reconfiguration subsystem is not enough efficient or not adjusted to application’s communication requirements. It is important, that better performance is achieved without changing the standard system architecture. So, the proposed parallel program optimization method provides a much better program behavior without any additional cost from the hardware point of view.

48

E. Laskowski and M. Tudruj

Experiments with Strassen matrix multiplication have shown that the lookahead reconfiguration with multiple crossbar switches gives in general a better program execution speedup and larger applicability area in respect to various system time parameters than the reconfiguration with link set partitioning. The lower is the reconfiguration efficiency of the system, the better results are obtained from the application of the look-ahead dynamic connection reconfiguration. It confirms the value of the look-ahead reconfiguration for fine-grained parallel programs, which have time-critical reconfiguration requirements. The presented experiments show that for low values of unitary connection reconfiguration time and program section activation time, performance of the architecture with the multiple crossbar switches and link set partitioning is similarly good.

References 1. Chen, J., Watson, W.: User-level communication on Alpha Linux Systems. In: Intl. Symp. On Parallel Architectures, Algorithms and Networks, ISPAN, IEEE Cs Press, Los Alamitos (2000) 2. Tanabe, N., et al.: Low Latency Communication on DIMNET-I network Interface Plugged into a DIMM slot. In: Intl. Conf. On Parallel Computing in Electrical Engineering, Warsaw, pp. 9–14. IEEE CS press, Los Alamitos (2002) 3. Tudruj, M.: Look-Ahead Dynamic Reconfiguration of Link Connections in MultiProcessor Architectures. In: Parallel Computing 1995, Gent, September 1995, pp. 539–546 (1995) 4. Hwang, J.-J., Chow, Y.-C., Angers, F.D., Lee, C.-Y.: Scheduling Precedence Graphs in Systems with Interprocessor Communication Times. Siam J. Comput. 18(2) (1989) 5. Laskowski, E.: New Program Structuring Heuristics for Multi-Processor Systems with Redundant Communication Resources. In: Proceedings of PARELEC 2002, Warsaw, September 2002, pp. 183–188 (2002) 6. Laskowski, E.: Program Scheduling in Look-Ahead Reconfigurable Parallel Systems with Multiple Communication Resources. In: Proceedings of PARELEC 2004, Dresden, September 2004, pp. 256–261 (2004) 7. Laskowski, E., Tudruj, M.: Efficient Parallel Embedded Computing Through LookAhead Configured Dynamic Inter-Processor Connections. In: Proceedings of 5-th International Symposium on Parallel Computing in Electrical Engineering PARc ELEC 2006, Bialystok, Poland, September 13-17 (2006) 2006 IEEE

An Algorithm to Improve Parallelism in Distributed Systems Using Asynchronous Calls Rouzbeh Maani and Saeed Parsa Iran University of Science and Technology [email protected], [email protected]

Abstract. Distributed systems use different methods for making parallelism. A common approach is using asynchronous calls. In this method the caller method and called method are located in different workstations and the caller method continues to run even after calling the remote method. Although the caller and called methods can be executed concurrently, the dependency of the instructions after the call instruction to the values affected by the called method make the caller to stop. In this article, an instruction scheduling algorithm is presented to achieve more concurrency in the execution of these distributed codes.

1

Introduction

The use of asynchronous calls is a common approach to create concurrency in distributes programs. The concurrent execution of the caller and the called methods, in this approach, results in speeding up the program execution time [1][2]. However, the problem is that programmers are used to think sequentially in such a way that when they write a call or any other statement of code they want to use the results of the statement immediately after. This way of thinking contradicts the idea of speeding up the execution of a distributed code by applying asynchronous calls, because, the concurrent execution of the caller and called methods in an asynchronous call ends up with the very first positions within the caller where any value affected by the called method is required. Instruction scheduling algorithms has been conventionally used to enhance parallelism in the execution of program codes [3][4][5]. These algorithms mostly depend on the architecture of the underlying system. As a case in point, in pipelined architectures, instruction scheduling is applied to speed up programs by keeping the pipeline full. This is achieved by finding sequences of unrelated instructions that can be overlapped in the pipeline [6]. As an another example, Nystrom and Eichenberger have presented an instruction scheduling algorithm, mainly focusing on minimizing execution overhead due to inter-cluster communications in VLIW architectures [7]. Instruction scheduling has been also applied to speed up programs code execution by increasing throughput [8][9], minimizing register pressure [10][11], reducing the effect of the cache misses or a combination of these [12][13][14]. Our main motivation has been to enhance the degree of concurrency in the execution of remote asynchronous calls. This is achieved by maximizing R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 49–58, 2008. c Springer-Verlag Berlin Heidelberg 2008 

50

R. Maani and S. Parsa

the distance between the call statement and the very first positions within the caller where any value affected by the called method is used. The proposed instruction scheduling algorithm needs to know the time of each statement. In this regard, a time estimation approach can be used [15]. The remaining parts of this article are organized as follows. In section 2, different type of dependencies between statements and their effects on reordering of instructions is discussed. In section 3, the instruction scheduling algorithm is presented. Section 4, includes a case study which surveys applying the proposed algorithm to a sample distributed code. The conclusion is brought in section 5.

2

Instruction Scheduling, Challenges and Requirements

As mentioned before, the aim is to gain the maximum possible concurrency in remote asynchronous calls by reordering the statements. The maximum amount of concurrency can be achieved when the caller method uses the values affected by the called method, just after the completion time of the asynchronous call. Therefore, the distance between each asynchronous call statement and the statements using any values affected by the called method should be increased, if possible. An instruction may be moved within a program code as far as control and data dependencies amongst the instructions are not violated [6]. Control and data dependencies and their effects on instruction reordering are further described in the next subsections. 2.1

Control Dependency

By definition an instruction is control dependent on another instruction if its execution is determined by that instruction. As an example, all the instructions within the body of an If statement or a For statement are control dependent on the instruction with which these statements begin. Control dependencies between instructions can be represented by a directed graph in which, each node addresses an instruction and the edges indicate control dependencies between the instructions. Any Instruction scheduling algorithm must preserve control dependencies between the instructions. To preserve control dependencies, instructions are only moved within their control dependency scope. A control dependency scope consists of all the instructions which are directly control dependent on a same instruction. In Fig. 1 a sample code and the corresponding control dependency scopes are shown. In this graph solid arrows represent control dependencies. As an example, all the instructions which are control dependent on instruction number 2 are in scope2. Instruction number 2, itself, is in scope1. 2.2

Data Dependency

To preserve the program semantics, data dependencies should be preserved, while reordering the program instructions. Generally, there are three kinds of data dependencies between two instructions: Read after Write, Write after Read and

An Algorithm to Improve Parallelism in Distributed Systems

51

Fig. 1. A sample code and its corresponding task graph

1 2 3

10

Algorithm:ResolveDependency(TheInstruction) Begin Find the parents of TheInstruction in DATA DEPENDENCY GRAPH and put them in ListOfParents For each Parent in ListOfParents Begin Find the CommonScope between TheInstruction and Parent InstructionParentInCommonScope ← Ancestor of TheInstruction in the CONTROL DEPENDENCY GRAPH the scope of which is CommonScope ParentParentInCommonScope ← Ancestor of Parent in the CONTROL DEPENDENCY GRAPH the scope of which is CommonScope If InstructionParentInCommonScope locates before ParentParentInCommonScope, put it after ParentParentInCommonScope End

11

End

4 5 6 7 8 9

Fig. 2. Resolve Dependency Algorithm

Write after Write[6]. Data dependencies are represented by a graph in which, each node addresses an instruction and edges indicate data dependencies between the instructions. Data dependencies are shown by dotted arrows in Fig. 1. A graph representing both data and control dependencies is called a task graph. To prevent any dependency violation while reordering the instructions, a procedure called ResolveDependency is used. This procedure gets an instruction and checks whether all the instructions which are the parents of the instruction in the data dependency graph, execute before the instruction. In Fig. 2, the pseudo code description of the ResolveDependency procedure is given.

3

Instruction Scheduling, Challenges and Requirements

In this section the proposed instruction scheduling algorithm is presented. In this algorithm, instructions are categorizes into 3 types: – Call Instructions: Asynchronous remote call instructions

52

R. Maani and S. Parsa

– Use Instructions: Instructions which use any value affected by a Call Instruction – Common Instructions: Other instructions The algorithm firstly moves Call instructions and their parents in the caller task graph, to the reordered code. In the second step, Use instructions and their parents are moved. In the third step Common instructions are moved into the reordered code and between Calls and Uses. In the forth step of the algorithm the reordered code is further optimized. Before moving an instruction to the reordered code, the ResolveDependency procedure, described above, is invoked to resolve any dependency conflicts caused by the move. These four steps are described more in the next subsections. 3.1

First Step

In the first step of the algorithm, presented in Fig.3, asynchronous call instructions within each method are moved to the top positions in the reordered code of the method. Apparently, the longer the execution time of a called method the longer the caller may wait for the call results; hence, if there are several Call instructions within a method, the Call instruction with the longest completion time is moved to the reordered code, first. Before moving a Call instruction, all its parents in the task graph are moved to the reordered code. Again, within the parents the priority is given to the Call instructions with the longest execution time. In the first step, after the Call instructions, Common instructions and finally all the Use instructions within the parent nodes are selected. A selected node is moved to the reordered code if it is not already moved. 3.2

Second Step

In the second step of the algorithm, presented in Fig.3, all the remaining Use instructions within each method are moved to the bottom of the reordered code of the method. While moving a Use instruction, all its parents in the task graph are moved to the reordered code. A selected instruction is moved to the reordered code if it is not already moved. 3.3

Third Step

As mentioned before, in an asynchronous remote call the caller has to wait for the completion of the call at the first position, where any value affected by the called method is used. To reduce the wait time, in the third step, the remaining Common instructions are positioned in between each Call and its corresponding Use instruction, if possible. Here, the priority is given to a Use instruction whose wait time is relatively longer than the others. The wait time is equal to completion time of the Call minus the time elapsed to reach the Use position from the Call position. If the wait time is computed less than zero it is set to zero (Fig. 4). In order to find suitable positions for inserting Common

An Algorithm to Improve Parallelism in Distributed Systems

1 2 3

53

22 23 24 25

Algorithm Step 1: ScheduleTheCalls(ListOfInstructions:List) Begin While there is a CallInstruction in ListOfInstructions which is not Moved To Reordered Code Begin Find a CallInstruction with the longest execution time from ListOfInstructions which is not Moved To Reordered Code Find all the parent nodes of the found CallInstruction in Task Graph and put them in the NewListOfInstructions ScheduleTheCalls(NewListOfInstructions) MoveToReorderedCode(CallInstruction) End While there is a CommonInstruction in ListOfInstructions which is not Moved To Reordered Code Begin Take a CommonInstruction Find all control dependent parents of the CommonInstruction and put them in ListOfControlParents For each ParentInstruction in ListOfControlParents MoveToReorderedCode(ParentInstruction) MoveToReorderedCode(CommonInstruction) End While there is a UseInstruction in ListOfInstructions which is not Moved To Reordered Code Begin Take a UseInstruction Find all control dependent parents of the UseInstruction and put them in ListOfControlParents For each ParentInstruction in ListOfControlParents MoveToReorderedCode(ParentInstruction) MoveToReorderedCode(UseInstruction) End

26

End

4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

1 2 3 4 5 6 7 8 9 10

Algorithm Step2: SelectUseInstructions Begin While there is a UseInstruction which is not Moved To Reordered Code Begin Take a UseInstruction Find all control dependent parents of the UseInstruction and put them in ListOfControlParents For each ParentInstruction in ListOfControlParents MoveToReorderedCode(ParentInstruction) MoveToReorderedCode(UseInstruction) End

11

End

Fig. 3. Step 1 and 2 of the algorithm

instructions a procedure called FindCandidatePositions is invoked for each Call and its corresponding Use instruction. Here, suitable positions are defined as all the positions within each scope located in between the Call and its corresponding Use instruction.

54

1 2 3

R. Maani and S. Parsa

23 24 25 26

Algorithm Step3: ScheduleInstructionsFillWaitingTimes(ListOfUseInstructions) Begin While there is an Instruction in ListOfUseInstructions which is not Moved To Reordered Code Begin Find the UseInstruction with the longest wait time from ListOfUseInstructions Remove the UseInstruction form ListOfUseInstructions CandidatePlacesList ← FindCandidatePositions(CallInstruction, UseInstruction) for each place in CandidatePlacesList, if it is possible to put an Instruction which is not Moved To Reordered Code Begin Take the Instruction Find all control dependent parents of the Instruction and put them in ListOfControlParents For each ParentInstruction in ListOfControlParents MoveToReorderedCode(ParentInstruction) MoveToReorderedCode(Instruction) Subtract the time of the Instruction form the wait time of the UseInstruction If the wait time of the UseInstruction is less than or equal to 0 then go to 3 End End While there is an Instruction which is not Moved To Reordered Code Begin Take an Instruction Find all control dependent parents of the Instruction and put them in ListOfControlParents For each ParentInstruction in ListOfControlParents MoveToReorderedCode(ParentInstruction) MoveToReorderedCode(Instruction) End

27

End

4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

1 2 3

17 18 19 20 21 22

Algorithm Step4: MoveInstructionBeforWaitAndAfterCall(ListOfUseInstructions) Begin While there is an Instruction in ListOfUseInstructions which is not Moved To Reordered Code Begin Find the UseInstruction with the lowest wait time from ListOfUseInstructions Remove the UseInstruction form ListOfUseInstructions Add all instructions of the scope which are placed after the UseInstruction to MoveBeforeUseList Find the corresponding call Instruction of the UseInstruction and Name it CallInstruction Add all instructions of the scope which are placed after the CallInstruction to MoveAfterCallList For each instruction in MoveBeforeUseList, if it is possible to move it before the UseInstruction Begin Move the instruction to the place before the UseInstruction Subtract the time of the instruction form the wait time of the UseInstruction If the wait time of the UseInstruction is less than or equal to 0 then go to 3 End For each instruction in MoveAfterCallList, if it is possible to move it before the CallInstruction Begin Move the instruction to the place after the CallInstruction Subtract the time of the instruction form the wait time of the UseInstruction If the wait time of the UseInstruction is less than or equal to 0 then go to 3 End End

23

End

4 5 6 7 8 9 10 11 12 13 14 15 16

Fig. 4. Step 3 and 4 of the algorithm

An Algorithm to Improve Parallelism in Distributed Systems

3.4

55

Fourth Step

In the first step of the algorithm when moving the dependent parents of a Call instruction, there maybe Use instructions within the parents. Moving a Use instruction at this step to the reordered code, may increase the wait time for the Call results. Also because of some other reasons the wait time of Use instructions might be increased. To resolve this problem, in the forth step, it is attempted to move all the instructions after each Use instruction to a position immediately before it. Similarly, it is attempted to move all the instructions, within the reordered code, which are positioned before a Call instruction to a position after it. Here, again the priority is given to the Call and Use instructions with relatively lowest wait time. It should be noted that it is not always possible to move an instruction before or after another one. In general, it is possible to move instruction 2 to a position before instruction 1 if and only if instruction 2 and all its successors in control dependency graph have no data dependencies on instruction 1 and all the instructions in between instructions 1 and 2. In Fig. 4, the pseudo code description of the step 4 of the proposed instruction reordering algorithm is presented.

4

Case Study

In this section the effect of applying the proposed algorithm to the Traveling Salesman Problem (TSP) program, is demonstrated. The TSP program code includes 17 classes and 129 methods. After extracting the call flow graph from within the program code, the graph is partitioned into two clusters, as shown in Fig. 5. These clusters are then distributed and executed on separate computing nodes. In order to evaluate the proposed instruction scheduling algorithm the

Fig. 5. Clustered Call Flow Graph of TSP program

56

R. Maani and S. Parsa

(a) Execution Time in Java Virtual Machine Cycles

(b) Speed Up before and after using Instruction Scheduling

(c) Ratio of speedup after using instruction scheduling algorithm to before

(d) Amount of concurrent time in Java Virtual Machine Cycles

(e) Ratio of concurrent time after using algorithm to before

Fig. 6. TSP program experimental results

execution time of the program is obtained statically by applying the time estimation algorithm presented in [16] which estimates programs execution time in Java Virtual Machine Cycles, platform independently. Java Virtual Machine Cycles can be translated into the seconds on different platforms, easily [17]. The execution time of the program, in Java Virtual Machine Cycles, for different input graphs is shown in Fig. 6.a. The serial execution time and parallel execution time before and after applying the proposed algorithm for each input graph is depicted in this figure. This is clear that the execution time is reduced when

An Algorithm to Improve Parallelism in Distributed Systems

57

the program runs as a distributed program and can be improved by using the instruction scheduling algorithm. Fig. 6.b shows the amount of speedup before and after applying the scheduling algorithm. It is observable that the presented algorithm improves the speedup amongst different inputs. The highest speedup is obtained by the input graph with 120 nodes and 250 edges which is 1.45 before and 1.49 after applying the algorithm. The ratio of speedup after using instruction scheduling algorithm to before applying it is brought in Fig. 6.c. Another parameter is comparing the amount of concurrent time before and after using the presented algorithm. Fig. 6.d shows this measurement. Also the ratio of concurrent times after using instruction scheduling algorithm to before applying it is depicted in Fig. 6.e.This shows that the amount of concurrent time can be increased 1.06 times.

5

Conclusion

Programmers may develop distributed code without bothering about concurrency in execution of the program code. It is possible to efficiently exploit the inherent concurrency in execution of distributed code by applying an instruction scheduling technique. Maximum concurrency in execution of distributed code can be achieved, by maximizing the distance between asynchronous call instructions and instructions applying the results of the calls, as much as possible. In this paper an algorithm for enhancement of concurrency in execution of distributed program code is presented. The experimental results demonstrate a relatively high improvement of concurrency in the execution of a distributed program code for solving the TSP problem.

References 1. Parsa, S., Bushehrian, O.: Automatic Translation of Serial into Distributed Code ¨ Using CORBA Event Channels. In: Yolum, p., G¨ ung¨ or, T., G¨ urgen, F., Ozturan, C. (eds.) ISCIS 2005. LNCS, vol. 3733, pp. 152–161. Springer, Heidelberg (2005) 2. Parsa, S., Khalilpoor, V.: Automatic Distribution of Sequential Code Using JavaSymphony Middleware. In: Wiedermann, J., Tel, G., Pokorn´ y, J., Bielikov´ a, ˇ M., Stuller, J. (eds.) SOFSEM 2006. LNCS, vol. 3831, pp. 440–450. Springer, Heidelberg (2006) 3. Aleta, A., Codina, J., Sanchez, J., Gonzalez, A., Kaeli, D.: Exploiting pseudoschedules to guide data dependence graph partitioning. In: Procedings of the 2002 International Conference on Parallel Architectures and Compilation Techniques, Charlottesville, VA (September 2002) 4. Golumbic, M.C., Rainish, V.: Instruction scheduling beyond basic blocks. IBM 1. Res. Develop. 34(I) (January 1990) 5. Hagog, M., Zaks, A.: Swing Modulo Scheduling for GCC. In: GCC Developers Summit (2004) 6. Hennessy, J., Patterson, D.: Computer Architecture: A Quantitative Approach, 3rd edn. Morgan Kaufmann Publishers, San Francisco (2003)

58

R. Maani and S. Parsa

7. Nystrom, E., Eichenberger, A.: Effective cluster assignment for modulo scheduling. In: Proceedings of the 31st International Symposium on Microarchitecture (MICRO-31), Dallas, TX, December 1998, pp. 103–114 (1998) 8. Jain, S.: Circular Scheduling: A New Technique to Perform Software Pipelining. In: Proc. of the Int. Conf. on Programming Languages, Design and Implementation (1991) 9. Rau, B.: Iterative Modulo Scheduling: An Algorithm for Software Pipelining Loops. In: Proc. of 27th Int. Symp. on Microarchitecture, November 1994, pp. 67–74 (1994) 10. Eichenberger, A., Davidson, E.: Step Scheduling: A Technique to Reduce the Register Requirements of a Module Schedule. In: Proc. of the 28th Int. Symposium on Microarchitecture, pp. 338–349 (1995) 11. Eichenberger, A., Davidson, E., Abraham, S.: Optimum Module Schedules for Minimum Register Requirements. In: Proc. of Supercomputing 1995 (1995) 12. Huff, R.: Lifetime-Sensitive Modulo Scheduling. In: Proc. of the Int. Conf. on Programming Languages, Design and Implementation, pp. 318–328 (1993) 13. Llosa, J., Valero, M., Ayguade, E., Gonzalez, A.: Modulo Scheduling with Reduced Register Pressure. IEEE Transactions on Computers 47(6), 625–638 (1998) 14. Sanchez, J., Gonzalez, A.: Cache Sensitive Modulo Scheduling. In: Proc. of 30th Int. Symp. on Microarchitecture, December 1997, pp. 338–348 (1997) 15. Corti, M., Gross, T.: Instruction duration estimation by partial trace evaluation. In: Proc., W.I.P. (ed.) Session 10th Real-Time and Embedded Technology and Applications Symposium, Toronto, Canada, IEEE, Los Alamitos (2004) 16. Parsa, S., Bushehrian, O.: On the Optimal Object-Oriented Program Remodularization. In: Shi, Y., van Albada, G.D., Dongarra, J., Sloot, P.M.A. (eds.) ICCS 2007. LNCS, vol. 4487, pp. 599–602. Springer, Heidelberg (2007) 17. Schoeberl, M.: JOP: A Java Optimized Processor for Embedded Real-Time Systems. PhD thesis, Vienna University of Technology (2005)

IEBS Ticketing Protocol as Answer to Synchronization Issue Barbara Palacz1 , Tomasz Milos1 , Lukasz Dutka2 , and Jacek Kitowski1,2 1

Institute of Computer Science, AGH University of Science and Technology, al. Mickiewicza 30, 30-059 Cracow, Poland 2 Academic Computer Centre CYFRONET-AGH, ul. Nawojki 11, 30-950 Cracow, Poland {palacz,milos}@student.agh.edu.pl, {dutka,kito}@agh.edu.pl

Abstract. In this paper we present a novel synchronization technology, which is designed for distributed computer systems that use replication as means of redundancy. Our mechanism was primarily created for Intelligent ExaByte Storage, which is a network storage system with multiuser access. The protocol is based on the concept of digital identity certificates and supports real-time update of replicas and annulment of operations that might have produced out-of-date results. It also guarantees that the number of replicas stays the same at all times during the system’s performance.

1

Introduction

Nowadays, there exists a great number of distributed computer systems that require a mechanism for managing synchronization of their resources. Most of these systems try to improve their reliability, performance or fault-tolerance through the use of redundancy. In context of storage systems, redundancy often involves data replication, which can be regarded as the process of managing copies of data, as well as a caching strategy where identical files are available from multiple locations [1]. Data replication is one of key techniques to improve data locality, increase dependability, reduce access delay, and enhance scalability for distributed systems in wide area computing environments [2]. The literature offers several papers that address this issue (e.g. [3], [4], [5], [6]). This process of replication is very much related to the problem of replica synchronization, since most of the times strict data consistency is of a paramount importance. The paper presents a novel approach to the issue of synchronization that is inseparably bound to IEBS - Intelligent ExaByte Storage. IEBS is a distributed storage system, whose structure is the basis of our mechanism, and with which our protocol is meant to be used.

2

Synchronization Problem in Distributed Systems

As mentioned before, most distributed computer systems with multiuser access, use replication as means of redundancy. Providing replication is equivalent to R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 59–67, 2008. c Springer-Verlag Berlin Heidelberg 2008 

60

B. Palacz et al.

having multiple copies of the same data stored in the system. What goes with it, is a question of replica synchronization, meaning supplying data coherency. When multiple users have access to maintaining the system data, it is possible that more than one user requires to execute an operation on the same data simultaneously. Multiple read operations do not cause a problem, since each user may acquire a different data copy. Update operations are another, much more complicated, matter. When one data copy is modified, all other copies must be synchronized before they are ever used, so that the system data stays coherent. The question is how this can be done in a system storing great amounts of data, such as IEBS? This is exactly the problem that our synchronization protocol is trying to solve.

3

Idea of Storage by IEBS

In order to understand well the idea of our approach to the problem of data synchronization, it is essential to first apprehend the general concept and structure of IEBS storage system. It is a system that is to supply distributed usable storage of ExaByte order of capacity, which would also provide an intelligent way of maintaining the retained information from many independent authorities simultaneously [7]. 3.1

Overview of System Structure

IEBS system supplies the necessary storage by the means of thousands of disk drives of approximately terabyte order of capacity each. All disks are equipped with simple processing units and Ethernet interfaces, as they work together over UDP/IP network [7]. The disks are located in racks, which, in turn, are placed in geographically distributed data centers. Each rack is managed by a device called the rack manager, which takes care of maintaining work of all its disk drives. The entire storage system is composed of several additional devices, which are necessary in order for it to function as a whole. Amongst all, these components provide easy management of great amounts of data retained in the system and high quality access control, which is extremely significant in a system of such a scale. Therefore, in context of the synchronization issue, there are three groups of devices that we will be interested in. These are: block managers, access control servers and rack managers. First of all, we will focus on the block managers, which are responsible for maintaining the entire ExaByte storage. They are a group of servers that divide the storage into logical units called blocks, serving as containers for data segments. Block managers store information about location of all objects, divided into pieces and kept in these blocks, and extract the information when needed. Secondly, we need to consider the role of access control servers, supervising the system’s access authorization. In order to do so, access control servers store information about all user accounts, access groups and access control lists. Moreover, they are also responsible for creating digital identity certificates (tickets), which are the basis of our synchronization mechanism.

IEBS Ticketing Protocol as Answer to Synchronization Issue

61

Finally, we have the rack managers, which reside in each rack of every data center of IEBS system. Aside from rack administration, their primary responsibility is taking care of aggregating communication from disk drives to block managers. This operation is essential in a system of such a scale, because otherwise block managers might be flooded with too much information flowing from thousands of disk drives simultaneously. 3.2

Replication as Problematic Feature

Since we are interested in our synchronization mechanism solution, it is essential to understand the extent of the problem in IEBS system specifically. Not only it is the matter of maintaining the data that might be accessed by multiple machines simultaneously, but also there is the issue of replica synchronization. IEBS system supports several supplementary features, which collectively constitute to the intelligence of the storage. Thanks to the built-in processing powers in disk drives, the system is capable of performing some advanced data processing scenarios, such as redundancy support on demand [7]. What it means, is that every client may request his file to be replicated a specified number of times, indicating what type of replication is to be performed. On-line replication means performing the operation at the exact moment in time, while the client is waiting for it to be finished. Off-line replication suggests that the client does not want to wait until all replicas are written, and the operation is performed later on. In both cases, disk drives communicate only between each other, passing on the replication request with no additional devices’ support needed. The problem, which here arises, is that in order to keep the system coherent, it is unacceptable to store obsolete replicas of a file. Therefore, after any changes made to any one of a file’s copies, the rest of them must be synchronized and updated, before they might be used again. Furthermore, in case if one of the disk drives crashes, the number of replicas must remain constant, thus all copies of a file, that were lost, have to be written into some other disk drive.

Fig. 1. On-line replication

Fig. 2. Off-line replication

62

4

B. Palacz et al.

Synchronization Protocol

The synchronization mechanism, included in IEBS, is based on the concept of digital identity certificates, or so-called tickets. In IT security, a ticket is a proof of access and usage rights to a particular service [8], thus it can be used as means of authentication or proof of authorization. In IEBS, this ticket definition is sustained as well.

Fig. 3. Access control and ticket creation

As mentioned before, our tickets are created by the access control servers as a response for a client’s request to perform an operation. The tickets are issued on the basis of client’s credentials and his access rights to the object that he requested to execute an operation on. Of course, if the access rights are not applicable, the operation is rejected without ticket creation. Moreover, each ticket is signed by the access control server for further verification and set out for a limited time period. In case of its expiration, it may not be used, until it is renewed within the access control server. If the ticket never expired, it could become something the client could give away without any consequences [9]. 4.1

General Operation Scenario

We have already explained the concept of tickets and responsibilities of several system devices, which are to control both data management and synchronization issues. At the moment, we will present a basic substantial situation, in which they cooperate together, in order to execute an operation requested by a client. Let us consider a simple read operation. Since the client must be authenticated, his request to execute an operation on a specified object is first sent to the access control server, where his ticket is created. The ticket strictly indicates the client’s identity, type of operation he requested to perform (in this case a read), and the object of his request (file specification). Right after its creation, the ticket is forwarded to one of the block managers, which stores in a block database all information about the previously-specified file. Since the file might have been divided into data segments, the block manager

IEBS Ticketing Protocol as Answer to Synchronization Issue

63

retrieves information about their exact location. This information takes form of a so-called block handle, containing disk drive’s identification and a block’s address on the drive. Each block handle is added to one of client’s tickets - one block handle per ticket. Since, at first, the ticket was only one, it is duplicated as many times as needed and sent back to the client. Afterwards, the client forwards his tickets directly to disk drives, indicated by the block manager in the tickets’ block handles. Thanks to the processing powers in drives, they are able to analyze the tickets, checking the client’s access rights, signature of access control server, ticket’s time limit and a list of annulled tickets, which is stored at each disk drive itself. If all these conditions are well met, the operation is executed, and the client receives his results. 4.2

Synchronization Problems

We have described the basic scenario of an operation performed in IEBS system. Certainly, in the case mentioned above, there was very little synchronization problems. However, there are several distinct situations, which are not as simple as they seem. Knowing that IEBS system supports additional advanced features, each client may want his files to be replicated several times. What will happen, if a client requests to modify a file that already had replicas? In this case, the answer is evident: replicas must be also updated and synchronized. But what if during a file’s modification, a different client requests some other operation on the same file? Should he get a reference to one of its replica? Certainly not, because this may lead to an unwanted data incoherency. Therefore the only way to maintain this is that block managers have to keep track of which copies of files are currently being used, and in what way (what operations are to be executed on them). The ultimate problem arises when we consider the same update operation to be performed on a file, but with previously-issued tickets for other operations on the same file. To clarify this, we are considering the following situation: a client requests to read a file, and thereafter another client requests an update operation on the same file. Since the tickets are set out for a limited time period, they do not have to be used exactly at a moment of their receival. Thus, if the read operation was not yet performed, it must be canceled so that the client does not receive data that is out-of-date. 4.3

Update with Replica Synchronization

As we can see, there are several significant synchronization requirements that our protocol must meet. They will all be covered while explaining an exemplary operation that includes all the problems mentioned in the previous section. We will focus on an update operation, performed on a file with replicas. The general process of the operation is basically the same as in case of a simple read operation, although it has several differences. It starts out similarly: with client authorization, ticket creation and its forwarding to one of the block managers. It is now, that our synchronization protocol must begin its action.

64

B. Palacz et al.

Fig. 4. First phase of an update operation

The block manager, seeing that the requested operation was an update, retrieves all replicas of the specified file. Each block, containing data segments of these replicas, are then marked as locked in the block database. This signifies that they may not be used again until they are properly synchronized with the copy to be modified. Therefore, the block manager will not be setting out block handles pointing to any of these blocks, until the update operation is complete. Afterwards, still before block handle creation, block manager informs all interested rack managers about annulment of tickets that were previously issued for any of the blocks, containing file’s replicas. Interested rack managers are the ones that maintain at least one disk drive, on which a data segment of a replica is stored. Of course, rack managers forward the annulment to the specified disk drive, which adds such ticket to its list of annulled tickets. Any ticket that appears on this list will not be executed, and will be returned to the client with an appropriate notification. Finally, the block handles are inserted into the tickets. They indicate the location of data segments of the only unlocked file copy. The tickets are sent back to the client, who uses them accordingly to the scenario presented before - forwards them to specified disk drives, where they are executed. When the modification is done and the client closes his file, a notification is sent by the drive to its rack manager. In the mean time, after block handle creation, the block manager had already sent information to interested rack managers about the location of replicas of each data segment being modified. This way, when a notification about end of update operation is received by a rack manager, it forwards replica information to disk drives, which store the updated copy of the file. Therefore, every disk drive that had a block, whose contents had changed during the update, knows which other blocks (on other disk drives) need to be updated. The changes made

IEBS Ticketing Protocol as Answer to Synchronization Issue

65

Fig. 5. Second phase of an update operation

are, thus, passed on from drive to drive, until all replicas are synchronized. This is when another notification is sent to the rack manager. Knowing that all data segments of a file had been updated and synchronized, rack managers notify the block managers about it, forwarding them, as well, the information about changes that were made to any of their disk drives’ blocks. At last, when the block manager recieves confirmation that the entire process of synchronization is over, it unlocks all replicas of the file that had been modified. From now on, all copies of the file may be used again, until they are locked again.

5

Related Work

In the previous section, we presented the entire process of replica synchronization, which is very much specific for IEBS storage system. Of course, several other distributed replica systems have also been developed. AFS and NFS are systems, where strict data coherency is of major importance, although they both work assuming that there is a much tighter data coupling of individual nodes than can be supported in geographically distributed environments, such as IEBS[10]. Furthermore, AFS (Andrew File System) does not support multisite replication issues[1]. One of very popular replica systems is nsync tool, which is widely used in Grid environments, such as GridLab, Globus and SRB[10]. It is implemented as an asynchronous distributed algorithm that, similarly to IEBS, does all synchronization at a block level. However, unlike IEBS, it does not synchronize all

66

B. Palacz et al.

replicas as soon as block modification is over, but it does it regularly. Each time the synchronization is performed, it first detects all files that had been modified since the last synchronization, and then it propagates the modifications[10]. Another common synchronization solution is a tool called rsync++, which is an amelioration of an open source file mirroring tool, rsync[11]. It is based on a master-slave model, where replica update information is generated on master site, then it is transported through the network to slave sites and finally it is processed on each of the slave sites[12]. What is also interesting in this solution, is that the differences between old and new copies of file are calculated locally, and only the patches are transmitted to other nodes[10].

6

Concluding Remarks

Replication is one of the key techniques to achieve better access times and reliability in data grid and storage systems[2], which are extremely significant in modern computer systems. The problem of replica synchronization is tried to be solved by various technologies, but many systems develop their own synchronization mechanisms that are specific to their structure. This is the case with Intelligent ExaByte Storage (IEBS), for which a new ticket-based synchronization protocol had been created. As IEBS is a distributed system that can be maintained by many institutions simultaneously, it generates multiple synchronization issues, which can all be resolved by our novel technology. The protocol supports real-time update of replicas, meaning that it does not allow storing obsolete data in the system. Furthermore, it provides an advanced mechanism for annulment of operations that might have produced results, which are out-of-date and a guarantee that the number of replicas stays the same at all times during the system’s performance. It is, therefore, perfect for IEBS storage system’s needs.

Acknowledgements The work described in this paper was supported in part by the European Union through the project int.eu.grid funded by the European Commission program IST-2006-031857.

References 1. Hoschek, W., Jan-Martnez, F.J., Samar, A., Stockinger, H., Stockinger, K.: Data Management in an International Data Grid Project. In: Buyya, R., Baker, M. (eds.) GRID 2000. LNCS, vol. 1971, Springer, Heidelberg (2000) 2. Qin, X., Jiang, H.: Data Grid: Supporting Data-Intensive Applications, May 07 (2003) 3. Stockinger, H.: Distributed Database Management Systems and the Data Grid. In: Eighteenth IEEE Symposium on Mass Storage Systems and Technologies (MSS 2001), January 26 (2001)

IEBS Ticketing Protocol as Answer to Synchronization Issue

67

4. Stockinger, H., Stockinger, K., Schikuta, E., Willers, I.: Towards a Cost Model for Distributed and Replicated Data Stores. In: Proceedings of the Ninth Euromicro Workshop on Parallel and Distributed Processing, pp. 461–467. IEEE Computer Society, Los Alamitos (2001) 5. Lamehamedi, H., Szymanski, B., Shentu, Z., Deelman, E.: Data Replication Strategies in Grid Environments. In: Proceedings 5th International Conference on Algorithms and Architectures for Parallel Processing, pp. 378–383. IEEE Computer Science Press, Los Alamitos (2002) 6. Slota, R., Skital, L., Nikolow, D., Kitowski, J.: Algorithms for Automatic Data Replication in Grid Environment. In: Wyrzykowski, R., Dongarra, J., Meyer, N., Wa´sniewski, J. (eds.) PPAM 2005. LNCS, vol. 3911, pp. 707–714. Springer, Heidelberg (2006) 7. Dutka, L., Palacz, B., Kitowski, J.: IEBS - Intelligent ExaByte Storage based on Grid Approach. In: CGW 2006 Proceedings (2006) 8. Mobile electronic Transactions: MeT White Paper on Mobile Ticketing Mobile electronic Transactions Ltd (2003) 9. Blumson, S.: DLPS Security Alternatives University of Michigan, http://www.umdl.umich.edu/pubs/dlpssecurity.html 10. Schutt, T., Schintke, F., Reinefeld, A.: Efficient Synchronization of Replicated Data in Distributed Systems. In: Sloot, P.M.A., Abramson, D., Bogdanov, A.V., Gorbachev, Y.E., Dongarra, J., Zomaya, A.Y. (eds.) ICCS 2003. LNCS, vol. 2657, pp. 274–283. Springer, Heidelberg (2003) 11. Dempsey, B.J., Weiss, D.: Towards an Efficient, Scalable Replication Mechanizm for the I2-DSI Project Tech. Rep. TR-1999-01, School of Information and Library Science, University of North Carolina. Chapel Hill, NC (April 1999) 12. Dempsey, B.J., Weiss, D.: On the Performance and Scalability of a Data Mirroring. Approach for I2-DSI University of North Carolina at Chapel Hill (1999); In Plank, J., Dempsey B.: Internet2 Network Storage Symposium (NetStore 1999), Seattle, WA, October 14-15 (1999)

Analysis of Distributed Packet Forwarding Strategies in Ad Hoc Networks Marcin Seredynski1 , Pascal Bouvry1 , and Mieczyslaw A. Klopotek2 1

University of Luxembourg, Faculty of Sciences, Technology and Communication 6, rue Coudenhove Kalergi, L-1359, Luxembourg, Luxembourg {marcin.seredynski, [email protected]}@uni.lu 2 Polish Academy of Sciences, Institute of Computer Science, Ordona 21, 01-237 Warsaw, Poland [email protected]

Abstract. Battery energy conservation and preventing selfish behavior are important, closely related issues in ad hoc networks. In this paper we demonstrate how using a strategy based packet forwarding approach can increase throughput in such a networks and at the same time can minimize the usage of resources of participating nodes. Each node is asked to use a strategy that defines conditions under which packets are being forwarded. Such strategy is based on the notion of trust and activity of the source node of the packet. We demonstrate that nodes applying such strategy use the network in more efficient way comparing to nodes not using such a reputation based packet forwarding. A genetic algorithm (GA) is applied to evolve good strategies, while for evaluation purposes of the strategies a game theoretical model of the ad hoc network is used. Keywords: Ad hoc networks, cooperation, game theory.

1

Introduction

A wireless mobile ad hoc network is composed of two or more devices (nodes) equipped with wireless communications and network capability [1]. Such network does not rely on any fixed architecture. Routing functionality is incorporated into mobile nodes. Devices can directly communicate with each other only when they are located in their radio range. Otherwise, intermediate nodes should be used to forward packets. As a result, nodes beside sending their own packets are also expected to forward packets on behalf of others. Topology of such network may change quickly in an unpredictable way. Most of the devices participating in the network run on batteries [2] [3] [4]. As a result the temptation to save energy and therefore act selfishly might be very high. So-called self-policing mobile ad hoc networks [5] [6] [7] [8] [9] is a common solution to this problem. In such networks nodes are equipped with a reputation management system combined with a response mechanism. Each nodes keeps its R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 68–77, 2008. c Springer-Verlag Berlin Heidelberg 2008 

Analysis of Distributed Packet Forwarding Strategies

69

own ratings of other network participants based on own experience and reputation data received from other nodes. If a packet forwarding request comes from a node with a bad reputation it is very likely to be discarded. Such an approach enforces cooperation because selfish nodes can not use the network for their own purposes unless they contribute to the packet forwarding. Energy saving can be done by either discarding packets (instead of forwarding them) or switching wireless interface into a sleep mode. Second option is by far the greatest way of saving the battery [10]. The power consumption is about 98% lower comparing to the one into the idle mode. The significantly higher idle power consumption reflects the cost of listening to the wireless channel. If a node wants actively participate to the network then its network interface should be in the idle mode, ready to receive traffic from its neighbors. Switching into a sleep node is very tempting, because such a behavior will be unnoticed by other network participants. This is due to the fact that it is not possible to distinguish between a node being in a sleep mode and a node which has temporally left the network. In order to enforce nodes to minimize the time spent in a sleep mode, their activity has to be taken into account by the reputation management system. In this paper we address the problem of the selfish behavior in self-policing ad hoc networks. Our approach aims at enforcing cooperation using packets forwarding strategy. The decision whether to forward or discard packets depends on the trust level to the source node of the packet and its activity. Nodes using such approach will spend less battery energy for sending same number of packets than those nodes not using it. We also propose a new game theoretic-based model of the ad hoc network whose goal is to evaluate strategies. A GA is used to evolve strategies. The paper is organized as follows. In the next Section, related work is discussed. Then in Section 3 we show our trust and activity evaluation mechanisms and we describe the strategy driven forwarding approach. This is followed by Section 4, where we present our game based model of ad hoc network. Next, in Section 5 we describe the evolution of strategies using GA. Simulation results are presented in Section 6. Last Section concludes the paper.

2

Related Work

A good survey of cooperation models with a game theoretical analysis can be found in [8] [11] [12]. In [3] authors present two techniques, watchdog and pathrater that aim at improving throughput of the network in the presence of selfish nodes. First, watchdog mechanism identifies selfish nodes and next, pathrater helps routing protocol to avoid these nodes. Such mechanisms do not discourage nodes from selfish behavior because selfish nodes are not excluded from the network. Authors show that in the network composed of 50 nodes with presence of 20 selfish nodes proposed mechanisms can increase the throughput by 17%. In [6] authors propose a mechanism called CONFIDANT whose goal is to make selfish behavior unattractive. Both, first and second-hand observations

70

M. Seredynski, P. Bouvry, and M.A. Klopotek

are used. Similarly to CORE, packets coming from selfish nodes will not be forwarded by normally behaving nodes. Additionally, if a selfish node starts to behave correctly for a certain amount of time it might re-integrate with the network. In [2] authors present an economic approach to the problem. Network is modelled as a market in which a virtual currency called nuglet is used. In such network nodes have to pay for the packets they want to send and are paid when they forward packets coming from other nodes.

3

Strategy Based on Trust and Activity Levels

Evaluation of trust and path rating We assume that each node uses an omni-directional antenna with the same radio range. A source routing protocol is used, which means that a list of intermediate nodes is included in the packet’s header. In our model the reputation information is gathered only by nodes directly participating in the packet forwarding. Similarly to the watchdog mechanism proposed in [3] each node monitors the behavior of the next forwarding node. Reputation data are collected in the following way: let’s suppose that the node A wants to send a packet to node E using intermediate nodes B, C, and D. If the communication is successful then node E receives the packet and all nodes participating in that forwarding process update reputation information about each other. If communication fails (for example node D decides to discard the packet) this event is recorded by the watchdog mechanism of the node C. In such a case node C forwards alert about selfish node D to the node B and then node B forwards it to the source node A. If at some point node B would like to check how trustworthy node A is (using collected reputation data concerning node B) it had to do the following: first calculate the fraction of correctly forwarded packets by the node B (forwarding rate) and then, using a trust lookup table (Figure 1a), get one of the four possible trust levels. For example, forwarding rate of 0.95 results in the trust level of 3. If a source node has more than one path available to the destination it will choose the one with the best rating which is calculated as a multiplication of all known forwarding rates of all nodes belonging to the route. An unknown node has a forwarding rate set to 0.5. Evaluation of the activity level Three activity levels are defined: low (LO), medium (ME) and high (HI). Those levels are calculated using the same reputation data as used for trust evaluation. In order to verify the activity level of a source node an intermediate node calculates the average number of all packets forwarded by all known nodes and next compares it with the number of packets forwarded by the source node. The exact activity value is assigned using an activity lookup table (Figure 1b). Coding the strategy The decision whether to forward or discard the packet is determined by the strategy represented by a binary string of the length 13. An example of a strategy is shown in Figure 1c. The exact decision is based on trust and activity levels of

Analysis of Distributed Packet Forwarding Strategies

Trust lookup table

pfA fr(B,A)= psA

fr 1 - 0.9 0.9 - 0.6 0.6 - 0.3 0.3 - 0

Activity lookup table

packets forwarded pfA > (pfAVR + pfAVR * 0.2) pfA pfAVR +/- pfAVR * 0.2 pfA < (pfAVR - pfAVR * 0.2)

TLBA 3 2 1 0

<

fr(B,A) - forwarding rate of the node A pfA - number of packets forwarded by the node A psA - number of packets sent to the node A TLBA - trust level of the node B in the node A

decision

trust 0

trust 1

trust 2

HI ME LO

decision against an unknown player

trust 3

DDD FFF DDD FDD F 1

<

b)

LO ME HI LO ME HI LO ME HI LO ME HI 0

ALA

pfA - number of packets forwarded by node A pfAVR - average number of packets forwarded by all known nodes ALA - activity level of node A

a) Trust and activity of the source node

71

2

3

4

5

6 7

8

9 10 11

c)

12

D: discard intermediate packet F: forward intermediate packet

Fig. 1. Computation of trust level (a), computation of activity level (b), coding of the strategy (c)

a source node. There are 12 possible combinations of those levels. Decisions for each case are represented by bits no. 0-11. Bit no. 12 defines behavior against an unknown node. Decision F means ”forward packet” while D stands for the opposite (”discard the packet”). For example, let’s suppose that the node B receives a packet originally coming from the node A. Assuming that node B has a trust level 3 in the node A, and nodes’ A activity is ”LO” then according to the strategy shown in Figure 1c the decision would be to forward the packet to the next hop (F , bit no. 9). In order to use more advanced and refined algorithms for trust and activity calculation several properties of the network should additionally be taken into account (like security algorithms used in the network, social relations between nodes, etc.).

4

Ad Hoc Network Game, Payoff Table, Tournament Scheme and Types of Players

Packet forwarding as Ad Hoc Network Game We define an Ad Hoc Network Game as a game in which one node (refereed also as player) is originating the packet and some other nodes have to decide whether to forward or to discard it. The number of game participants depends on the length of the path leading from the source to the destination node. The game participants the source node and all intermediate nodes. The destination node is not a part of the game. Each player is said to play his own game when being a source of a packet and is said to be a participant of other players’ game when being an intermediate node. After the game is finished all its participants receive payoffs according to the decisions they made.

72

M. Seredynski, P. Bouvry, and M.A. Klopotek source node reliability T A HI 3 ME LO HI 2 ME LO HI 1 ME LO HI 0 ME LO

T-trust F- decision forward A-activity D- decision discard Rank: forwarding preference

payoff F 14 9 5 12 8 4 10 7 3 2 1 0

D 0 3 7 1 4 8 2 5 9 10 12 14

Rank 1 4 7 2 5 8 3 6 9 10 11 12

a)

transmission payoff status 12 success (S) faillure (F) 0 b)

interface payoff mode in (per a round round) sleep 6 iddle 0 c)

Fig. 2. Payoff tables for: intermediate nodes forwarding or discarding packets (a), source nodes originating packets (b), nodes staying in a sleep mode (c)

Payoff table and fitness function The goal of payoffs is to capture essential relations between alternative decisions and their consequences. Three payoff tables for different kind of activities are defined: one for intermediate nodes forwarding packets (Figure 2a), one for source node of the packet (Figure 2b), and one for spending time in a sleep mode (Figure 2c). Payoffs received by intermediate nodes depend on their decisions (packet discarded or forwarded) and on their trust and activity levels in the source node. Generally, the higher the trust/activity levels are the higher payoff node forwarding the packet receives. High trust level in the source node means that in the past this node has already forwarded some packets for the currently forwarding node. So it is more likely that such node will be used in the future as a intermediate node. This means that forwarding for such node might be considered as an investment of trust for the future situations. On the other hand when a node decides to discard a packet it will be rewarded for saving its battery live. When defining a payoff for forwarding/distarding packets one has to decide about preference order of particular combinations of trust and activity values. Our rank of payoffs for forwarding/discarding packets is shown in Fig. 2a - column ”Rank”. The highes payoff is received for forwarding packets for a source node with trust3/activityHI, then for trust2/activityHI, next for trust1/activityHI, etc. As for the source node, the exact payoff depends only on the status of the transmission. If the packet reaches the destination then the transmission status is denoted as S (success). Otherwise (packet discarded by one of the intermediate node’s), the transmission status is denoted as F (failure). Players also receive additional payoff for time (rounds) spent in a sleep mode as a reward for saving the battery. The fitness value of each player is calculated as follows: f itness =

tps + tpf + tpd , ne

(1)

Analysis of Distributed Packet Forwarding Strategies

73

where tps, tpf, tpd are total payoffs received respectively for sending own packets, forwarding packets on behalf of others and discarding them. The ne is a number of all events (number of own packets send, number of packets forwarded and number of packets discarded). Evaluation of strategies in a tournament Each player is using a single strategy which is evaluated in a tournament that represents an ad hoc network. Several players using different strategies play number of Ad Hoc Games. Tournament is composed of R rounds. In every round each player is a source of a packet exactly once (plays its own game) and participates in the packet forwarding several times. Destination and intermediate nodes are chosen randomly. Each node might decide to spend certain number of rounds in a sleep mode and as a result become unavailable for routing purposes. But whenever it wants to send own packet it turns to iddle mode. The tournament itself can be described as follows: Tournament scheme Step 1: Specify i (source node) as i := 1, K as a number of players participating in the tournament and R as a number of rounds. Step 2: Randomly select player j (destination of the packet) and intermediate nodes. Step 3: For each available path calculate its rating (as described in Section 3) and select the path with the best reputation. Step 4: Play the Ad Hoc Network Game. Step 5: Update payoffs of the source node i and all intermediate nodes (game participants) that received the packet. Step 6: Update the reputation data among all game participants. Step 7: If i < K, then choose the next player i := i + 1 and go to the step 2. Else go to the step 8. Step 8: If r < R, then r := r + 1 and go to the step 1 (next round). Else stop the tournament. Types of players We defined five different types of players participating in the network (tournament): normal players 1 and 2 (N P 1, N P 2), low activity player (LAP ), and two types of selfish players - SP 1 and SP 2. Both types of normal players (N P 1 and N P 2) play according to their trust and activity based strategy. The only difference between them is the approach towards getting into a sleep mode. N P 2 is more focused on battery saving comparing to N P 1. He goes to the sleep mode as soon as over 40% of his packet reach the destination, while NP1 does the same at the level of 90%. When those rates fall below predefined thresholds, player is switching back to an idle mode. The remaining types of players do not take into account any information concerning a source node when forwarding packets. LAP spends 25% of the time (rounds) in a sleep mode and forwards all packets whenever being awake. Selfish players (SP 1 and SP 2) are always awake. SP 1 discard all packets, while SP 2 players forward packets with a probability of 0.5.

74

M. Seredynski, P. Bouvry, and M.A. Klopotek Table 1. Types of players participating in the network type of player forwarding strategy sleeping approach normal 1 (N P 1) based on trust/activity of src. node sleep if succ. rate > 0.9 normal 2 (N P 2) based on trust/activity of src. node sleep if succ. rate > 0.4 low activity(LAP ) always forward when awake 25 % of time (rounds) selfish 1 (SP 1) newer forward never selfish 2 (SP 2) forward with a probability of 0.5 never

All players try to send the same number of packets. Properties of each type of player are summarized in Table 1.

5

Evolution of the Behavior Using GA

Only strategies of normal players evolve, other types of players continue to use the same forwarding strategy. There are two populations of normal players denoted as P OP 1 (players N P 1) and P OP 2 (players N P 2). At the beginning randomly generated strategies are assigned to each normal player. Then players are evaluated in a tournament as described in Section 4. During the evaluation, normal players have to compete with other types of players. Afterwards, selection and reproduction operators are applied on the population of strategies used by normal players (independently for P OP 1 and P OP 2). Then N pairs of strategies are selected using a tournament selection. New strategies are obtained by applying crossover and mutation operators to each of N selected pairs. A standard one-point crossover is used. One of the two strategies created after crossover is randomly selected to the next generation. Finally, a standard uniform bit flip mutation is applied. As a result a new population of strategies of both populations is created. The process is repeated for a predefined number of times. In each generation the number of particular types of players participating to the tournament remains the same.

6

Experiments

Conditions of experiments In each experiment the players compete in a network (tournament) composed of 50 players, but numbers of players of the particular typed are different in each experiment. Settings of the tournament for each experiment are shown in Table 2. Both populations, P OP 1 and P OP 2 are composed of 50 players. The following parameters of GA are used: crossover probability - 0.9; mutation probability 0.001, number of generations - 100. Each tournament is composed of 600 rounds. Unknown nodes have a default trust value assigned to 1. All experiments were repeated 60 times and an average value was taken as a result. Experiments were performed according to the following steps: Step 1: Load SP 1, SP 2, LAP players according to values shown in Table 2. Step 2: Randomly select players from P OP 1 and P OP 2 according to values shown in Table 2) and load them to the tournament.

Analysis of Distributed Packet Forwarding Strategies

75

Table 2. Number of players in the tournament in each experiment

number of NP1 number of NP2 number of SP1 number of SP2 number of LAP

experiment 1 experiment 2 experiment 3 25 15 10 25 15 10 0 5 5 0 10 20 0 5 5

Step 3: Play the tournament with selected players as described in Section 4. Step 4: If some players from P OP 1 or P OP 2 have not yet played in the tournament then go to the step 1, otherwise go to the step 5. Step 5: Create a new P OP 1 and P OP 2 (as described in Section 5). Step 6: Repeat steps 1-5 for the predefined number of generations. Experimental results A success rate of a player is defined as a percentage of packets that reached the destination. All results were taken from last generations of each experiment (average values per type of player). All types of players were trying to send the same number of packets. The performances of each type of player in experiment 1 are shown in Table 3. One can see than in a network composed of normal players only, there was a significant difference in the success rate between two participating types of players. Although N P 1 and N P 2 where trying to send the same number of packets, N P 2 managed to successfully send only 28.6 % which was a very bad result comparing to almost 82% sent by N P 1. This was due to the fact that N P 2 players where seen by N P 1 as the ones the activity below the average. Strategies used by N P 1 preferred nodes with at least average activity level. Table 3. Performance of various types of players in experiment 1 type of player success rate number of packets forwarded sleep time (rounds) NP1 82.7 % 914 5% NP2 28.6 % 615 13 %

If N P 2 would change its sleeping conditions from 0.4 to 0.6 it would result in a success rate improvement from 28.6% to 46.8%. In the next two experiments network was composed of all types of players. The results of experiment 2 are shown in Table 4. This time the success rate of N P 2 slightly improved, while for for N P 1 it had decreased significantly (from almost 83% to 47%). In order to achieve these results both types of players had to forward the same number of packets, but N P 2 additionally spent 15% of time in a sleep mode saving battery live. In order to decide which ”sleeping” approach was better, one have to choose between sending more packets or saving more battery live. As for other types of players LAP managed to send 40% of packets (second best result). It was quite costly effort: LAP nodes had to forward on average 2400 packets (SP 1 and SP 2 forwarded only 720 packets). On the other hand such players spent

76

M. Seredynski, P. Bouvry, and M.A. Klopotek Table 4. Performance of various types of players in experiment 2 type of player success rate number of packets forwarded sleep time (rounds) NP1 47 % 720 0% NP2 34.5 % 720 15 % SP1 4.3 % 0 0% SP2 10,5 % 2839 0% LAP 40 % 2400 25 % Table 5. Performance of various types of players in the experiment 3 type of player success rate number of packets forwarded sleep time (rounds) NP1 34.5 % 1095 0% NP2 33.5 % 1005 7% SP1 7.1 % 0 0% SP2 16.8 % 3404 0% LAP 29.3 % 6191 25 %

25% in a sleep mode. The probabilistic forwarding strategy of SP 2 turned to be very unsuccessful. Those nodes forwarded the highest number of packets among all types of players and obtained a success rate of 10.5%. In the network tested in experiment 3 more SP 2 were present. The results are shown in Table 5. Success rate for N P 1 decreased again, while N P 2 saw a very slight decrease. Both types of normal players had almost the same success rate, although overall sleeping strategy of N P 2 turned to be more successful than the one of N P 1. While having almost the same success rate, N P 2 managed to spend 7% of time in a sleep mode (N P 1 did not sleep at all). As for other types of players, all of them improved their success rate except LAP . The success rate of LAP was very close to the one of normal players but for reaching this level those players had to forward on average 6191 (about 6 times more than normal players). The experiments showed that even when players using reputation based packet forwarding strategies form only 40% of the population of the network (experiment 3), they sill obtain best results in terms of success rate and battery use. Other types of players had to spent far more energy on packets forwarding in order to be able to send their own packets. Such energy was often wasted when forwarding packets for nonreciprocal players like SP 1.

7

Conclusions

Due to the limited energy of the batteries, nodes might be reluctant to share packet forwarding responsibilities. Such behavior might seriously threat the existence of the network. In this paper we demonstrated how a strategy driven packet forwarding approach based on the notion of trust and activity can increase the cooperation level in ad hoc networks. A fair contribution to the packet forwarding alongside the high activity are the only ways enabling nodes to send large

Analysis of Distributed Packet Forwarding Strategies

77

number of their own packets. However the most efficient way to send own packets required the usage of the reputation based forwarding strategies.

References 1. Ilyas, M., Mahgoub, I. (eds.): Mobile Computing Handbook. Auerbach Publications (2005) 2. Buttyan, L., Hubaux, J.P.: Nuglets: a virtual currency to stimulate cooperation in self-organized mobile ad hoc networks. Technical Report DSC/2001/001, Swiss Federal Institute of Technology (2001) 3. Marti, S., Giuli, T., Lai, K., Baker, M.: Mitigating routing misbehavior in mobile ad hoc networks. In: Proc. ACM/IEEE 6th International Conference on Mobile Computing and Networking (MobiCom 2000), pp. 255–265 (2000) 4. Michiardi, P., Molva, R.: Simulation-based analysis of security exposures in mobile ad hoc networks. In: Proc. European Wireless Conference (2002) 5. Buchegger, S., Boudec, J.Y.L.: The effect of rumor spreading in reputation systems for mobile ad-hoc networks. In: Proc. Workshop on Modeling and Optimization in Mobile, Ad Hoc and Wireless Networks (WiOpt 2003), pp. 131–140 (2000) 6. Buchegger, S., Boudec, J.Y.L.: Performance analysis of the confidant protocol. In: ACM 3rd International Symposium on Mobile Ad Hoc Networking and Computing (MobiHoc 2002), pp. 226–236 (2002) 7. Buchegger, S., Boudec, J.Y.L.: Self-policing mobile ad-hoc networks by reputation systems. IEEE Communications Magazine, Special Topic on Advances in SelfOrganizing Networks 43(7) (July 2005) 8. Giordano, S., Urpi, A.: A self-organized and cooperative ad hoc networking. In: Basagni, S., Conti, M., Giordano, S., Stojmenovic, I. (eds.) Mobile Ad Hoc Networking, Wiley-IEEE Press (2004) 9. Michiardi, P., Molva, R.: Core: A collaborative reputation mechanism to enforce node cooperation in mobile ad hoc networks. In: Proc. IFIP 6th Conference on Security Communications, and Multimedia (CMS 2002), pp. 107–121 (2002) 10. Feeney, L., Nilsson, M.: Investigating the energy consumption of a wireless network interface in an ad hoc networking environment. In: Proc. The IEEE Conference on Computer Communications (INFOCOM 2001), pp. 1548–1557 (2001) 11. Felegyhazi, M., Hubaux, J.P.: Game theory in wireless networks: A tutorial. Technical report, EPFL - Switzerland (2006) 12. MacKenzie, A., DaSilva, L.A.: Game Theory for Wireless Engineers. Morgan & Claypool Publishers (2006)

Implementation and Optimization of Dense LU Decomposition on the Stream Processor Ying Zhang, Tao Tang, Gen Li, and Xuejun Yang School of Computer, National University of Defense Technology, 410073 Changsha China {zhangying,tangtao,ligen,yangxuejun}@nudt.edu.cn

Abstract. Developing scientific computing applications on the stream processor has absorbed a lot of researchers attention. In this paper, we implement and optimize dense LU decomposition on the stream processor. Different from other existing parallel algorithms for LU decomposition, StreamLUD algorithm aims at exploiting producerconsumer locality and at overlapping chip-off memory access with kernel execution. Simulation results show that dealing with matrices of different sizes, compared with LUD of HPL on an Itanium 2 processor, StreamLUD we implement and optimize gets a speedup from 2.56 to 3.64 ultimately. Keywords: stream processor; LU decomposition; kernels; stream; producer-consumer locality; scientific computing.

1

Introduction

Of all rising architecture processors, the stream processor has absorbed a lot of scientific computing researchers attention[1,2,3,4,5,6] because of its capacity of high parallel computing, memory system, low power and low cost[7,8,9]. The stream processor is absolutely different from other conventional processors whether from programming model or from the architecture. Stream programming model organizes applications as two levels: kernel-level and stream-level[10,11,12,13]. A kernel is a computation-intensive function that

Fig. 1. Block diagram of a stream processor

Fig. 2. Stream flowing across the memory system

R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 78–88, 2008. c Springer-Verlag Berlin Heidelberg 2008 

Implementation and Optimization of Dense LU Decomposition

79

operates on sequences of records called streams. Each kernel takes streams of records as input and produces streams as output. Kernels are written using a C-like language called KernelC. The stream-level program declares the streams and defines the high-level control- and data-flow between kernels. Stream-level programs are written using a programming language extension called StreamC intermixed with C++. Stream programming model exposes three level parallelisms of the stream architecture to programmers: Instruction-Level Parallelism (ILP), Data-Level Parallelism (DLP) and Task-Level Parallelism (TLP). Both the organization of ALUs and the memory system of the stream architecture(as shown in figure 1) are different from those of conventional architecture. There are N clusters controlled by a single controller to work in SIMD mode, and every cluster consists of M ALUs(in the stream processor prototype Imagine[14,15] developed by Stanford University, N = 8 and M = 6). At the same time, the stream processor has three level memory hierarchies - local register files (LRF) near ALUs, exploiting locality in kernels, global stream register files (SRF), exploiting producer-consumer locality between kernels, and streaming memory system, exploiting global locality. Figure 2 shows a stream flows across three level memory hierarchies during the execution of a stream program. First, the stream is loaded from chip-off memory into SRF and distributed among buffers in SRF. Then it is loaded from SRF to LRF as operands of a kernel. After the kernel is finished, the input and output streams are stored back to SRF. If there is producer-consumer locality between this kernel and its later kernel, the result streams are saved in SRF. Otherwise, they are stored back to chip-off memory. Dense LU decomposition is representative of many dense linear algebra computations and a typical dense matrix factorization. The kernel is widely used in a variety of solvers and eigenvalue computations, and radar cross-section applications typically require LU decomposition of large matrices. Additionally, LU decomposition is a typical kernel of many benchmark packages, such as SPLASH and NPB, and is the based algorithm of LINPACK which is the most popular benchmark to evaluate the float-point sustained performance of supercomputers. It is a hard work to program on the stream processor for general-purpose uses, although the stream processor can provide high performance. In implementing StreamLUD (Stream LU Decomposition), how to organize data as streams is an important and difficult task. Additionally, StreamLUD not only exploits parallelism inherent as existing parallel algorithms for LU decomposition, but also aims at exploiting producer-consumer locality and at overlapping memory access with kernel execution. In this paper, we present our implementation of dense LU decomposition on the stream processor, which is aimed at media applications originally. We give the mapping method and our optimization, and run our StreamLUD on Isim[14,15], a cycle-accurate stream simulator. Compared with an Itanium 2 processor, dealing with matrices of different sizes the stream processor achieves a speedup from 2.56 to 3.64.

80

Y. Zhang et al.

The remainder of this paper describes and evaluates our implementation and optimization of dense LU decomposition on the stream architecture. In Section 2 we describe our implementation of StreamLUD on the stream processor. In section 3 we evaluate and analyze the performance of StreamLUD on Isim. In section 4 we optimize our implementation special for the stream processor, and evaluate and analyze our optimization. Section 5 discusses what must be done before the stream processor can support scientific computing applications well and section 6 makes a conclusion.

2

Dense LU Factorization

The LU factorization has the form A = P × L × U , where P is a permutation matrix, L is the lower triangular with unit diagonal elements, and U is the upper triangular. L and U are stored in A. The kernel is widely used in a variety of solvers and eigenvalue computations. Solving a system of equations requires O(n3 ) floating-point operations, more specifically, 2/3n3 + 2n2 + O(n) floatingpoint additions and multiplications. 2.1

Algorithm

Our StreamLUD is based on block-partitioned algorithm[16,17] which divides a dense n × n matrix A into an N × N array of B × B blocks, (n = N B). Blocking is performed in our algorithm to improve the temporal locality in LRF and the producer-consumer locality in SRF, and reduce the number of loop nests and the number of streams in the stream program, which consequently reduce the overheads of the stream program. The block size should be chosen to be large enough to exploit the locality in LRF and SRF, and small enough to avoid the overflow of LRF. It also should be the multiple of cluster number in the stream processor, because all streams are consumed by clusters in SIMD mode. In our experiment, we choose the block size as 8 which is the cluster number in our simulator, Isim[14,15]. The pseudo-code in figure 3(a), expressed in terms of streams and kernels, shows the most important steps in StreamLUD. The pseudo-code of kernel lu edge kc is shown in figure 3(b), and that of kernel lu body kc is shown in StreamLUD 1. For k=0 to N-1 do 2. Update l_edge_in, u_edge_ in, l_ body_ in, u_body_in, diagnal_block_ T 3. Factor edge: lu_ edge_ kc(in: u_ edge_in, l_ edge_in,diagnal_ block_ T out: u edge out,l edge out)) 4. Update the body using corresponding edge record: lu body kc(in: u_edge_out, l_edge_out,l_body_in,u_body_in, out:l_body_out,u_body_out);

lu_edge_kc 1. Factor the first block of u_edge_in and record the diagnal elements 2. For k=0 to (u_edge_length/block_size-1) do Factor the remaining elements in u_edge_in. 3. For k=0 to (l_edge_length/block_size) do Factor l_edge_in using the diagonal elements generated above.

(a) StreamLUD

(b) lu edge kc

lu_doby_kc 1. Factor l_body_in using corresponding u_edge_out and l_edge_out 2. Factor u_body_in using corresponding l_edge_out and u_edge_out

(c) lu body kc

Fig. 3. Pseudo-code of StreamLUD and its kernels

Implementation and Optimization of Dense LU Decomposition

Fig. 4. Simplified dataflow graph of a single iteration of StreamLUD

81

Fig. 5. Concrete computation of a single k iteration

figure 3(c). Figure 4 shows the dataflow graph of a single k iteration. And figure 5 shows the concrete computation corresponding to a single k iteration in the pseudo-code symbolically. In StreamLUD, we organize all computation as two kernels, lu edge kc and lu body kc, to process all input streams, as shown in figure 4. In every iteration, the first block size×block size records of u edge in is decomposed first using the first block size × block size records of u edge in and stream diagonal block T. And the diagonal elements of this block are recorded in step 1 of figure 3(b). Stream diagonal block T has the same records as the first block size×block size records of u edge in, but the record sequences of them are different. The former organizes elements of the same column one by one, then the next column, while the latter are just the contrary. Then the remaining records of u edge in and l edge in are factorized in step 2 and 3 of figure 3(b). And then stream l body in and u body in are updated by corresponding records of l edge out and u edge out, as shown in step 1 and 2 of figure 3(c). 2.2

Organizing Stream

Stream programming model exposes a lot of architecture details to programmers, such as the use of SRF, chip-off memory, and clusters. Although this improves the hardness to programme, it provides a consistent structure that is specialized for stream applications[18]. So, how streams are organized decides program behavior and performance straightly. Consequently, it is very important to organize stream efficiently. Then, how streams are organized in StreamLUD is described. Figure 6 shows stream organization of StreamLUD which decomposes a 48 × 48 matrix that is divided into 6 × 6 arrays of 8 × 8 blocks. Of the blocked arrays, the lower triangular is organized as stream l one column after anther; the upper triangular with diagonal blocks is organized as stream u one row after anther. The record length of stream l equals to the row length of the block and consequently the same row of a block lies on the same cluster; the record length of stream u equals to 1, the record sequence of blocks in stream u is row first, and consequently the same column of a block lies on the same cluster.

82

Y. Zhang et al.

Fig. 6. Simplified graph of data stream organization

Our stream organization can ease programming and improve program performance efficiently. First, when stream l edge in, u edge in, l body in, u body in and diagonal block T are updated, which are all parts of stream l or stream u, our organization makes them be derived by stream l or stream u easily. Secondly, when stream l edge in and u edge in are factorized, records that are processed by identical operations are on different clusters, which fits SIMD mode perfectly. Thirdly, when l body in and u body in are updated, the dot product of the corresponding row of l edge out and the corresponding column of u edge out should be computed. This stream organization makes rows and columns participating in the same dot product lie on the same cluster. As a result, the dot product can be done without any data collecting from other clusters, and consequently the number of inter-cluster communications is cut down efficiently.

3

Experimental Setup and Performance Evaluation

In our experiments, we use Isim[14,15], a cycle-accurate stream processor simulator supplied by Stanford University, to get the performance of StreamLUD. The baseline configuration of the simulated stream processor and memory system is detailed in table 1, and is used for all experiments unless noted otherwise. For comparison, LU decomposition from HPL is compiled by Intels IA64 compiler (with optimization option -O0, -O1, -O2 and -O3) and runs on an Itanium 2 processor whose baseline parameters are listed in table 2. And the best performance results on the Itanium 2 processor are chosen to compare with StreamLUD results. Table 3 lists the comparison of the performance of LU decomposition of different sizes matrices on Itanium 2 with on Isim and corresponding speedup yielded by Isim over Itanium 2. The speedup is more than 1, which means that for dense LU decomposition, the stream processor can get much better performance. For 64 × 64 and 96 × 96 matrices, StreamLUD is faster and faster with the increase of size. This is because input and output streams of StreamLUD are all in SRF, which means that when running, kernels gets all streams participating in computation from SRF not from chip-off memory. For 128 × 128, 160 × 160 and 192 × 192 matrices, the speedup over Itanium 2 gets smaller and smaller. This is because input and output streams of corresponding StreamLUD are larger than SRF and double buffer happens. At this time when kernel running, only a part

Implementation and Optimization of Dense LU Decomposition

Table 1. Baseline parameter of Isim Parameter Number of clusters Operating frequency Capacity of LRF Capacity of SRF Bandwidth of LRF Bandwidth of SRF Bandwidth of chip-off DRAM

8 500MHz 9.6KB 128KB 544GBps 32GBps 2.67GBps

83

Table 2. Baseline parameter of Itanium 2 processor Parameter Operating frequency Capacity of Data Cache level 1 Capacity of Instruction Cache level 1 Capacity of Data/Instruction Cache level 2 Capacity of Data/Instruction Cache level 3

1.5GHz 16KB 16KB 256KB 4MB

Table 3. Performance of LU decomposition of matrices with different sizes on Itanium and on Isim(ms) and the speedup Matrix Size Itanium 2 Isim Speedup

64 × 64 0.31171 0.014973 2.082

96 × 96 0.78165 0.032339 2.417

128 × 128 1.54545 0.068209 2.266

160 × 160 2.60274 0.120907 2.153

192 × 192 4.1098 2.81449 1.460

of some stream is in SRF. After this part has been consumed by corresponding kernel, the stream processor stalls to wait for another part of this stream that is being loaded from chip-off memory to SRF by double-buffer. When this happens, kernel stalls and double buffer withdraw StreamLUD performance. Then we take 128×128matrix as example to discuss the details of StreamLUD on Isim. Figure 7 depicts cluster and memory system occupancy. The leftmost column is cycle number. The CLUSTERS column shows which kernels were running

4.0E+05

MC Chip-off Memory Run_time

3.0E+05

2.5E+07

Chip-off Memory SRF LRF

2.0E+07 1.5E+07

2.0E+05 1.0E+05

1.0E+07 5.0E+06 0.0E+00

0.0E+00

Fig. 7. Cluster and memory system occupancy

Fig. 8. Main parts active time and program run time

Read

Write

Read-Write

Fig. 9. Traffics of chipoff memory, SRF and LRF

84

Y. Zhang et al.

during which cycles, and the MEMORY STREAM columns show the streams passing between Isim and its off-chip memory. Note that because in our algorithm, input steams of kernel lu edge kc in iteration k depend on output streams of kernel lu body kc in iteration k-1, there is little overlap between kernel execution and chip-off memory access, excluding that double buffer happens. But double buffer will generate a lot of kernel stalls. So, StreamLUD should be optimized to avoid problem above. Figure 8 shows run time of StreamLUD, chip-off memory access time and microcontroller active time which equals to the total of kernel run time. MC and chip-off memory run time weighs highly, 45% and 61% respectively. There is a little overlap between them, only 6% of total run time. Read, write and total traffics of chip-off memory, SRF and LRF are shown in figure 9. The ratio of them is 1 : 2.7 : 26 with respect to total traffic. Temporal and space locality in LRF is well exploited in our StreamLUD. But locality in SRF is employed badly. So we should optimize our StreamLUD to improve its SRF locality.

4

Implementation Optimizations

Imagine provide two methods, unroll and pipeline, to optimize kernels, and two methods, doUnroll and doSoftwarePipeline, to optimize stream-level programs[12,13]. Implementation above has been optimized by these methods. Then well depict optimizations we used to optimize StreamLUD specially. From figure 9, we can see that SRF locality of StreamLUD isnt exploited well. From figure 7 we can see that chip-off memory access seldom overlaps kernel execution. Chip-off memory access is on the critical path of the performance of StreamLUD. So, we optimize our StreamLUD in the two ways. 4.1

Exploiting Producer-Consumer Locality in SRF

There is dependency between l body out of iteration k − 1 and l edge in of iteration k, and between u body out of iteration k−1 and u edge in of iteration k. So, l edge in and u edge in of iteration k cant be loaded into SRF until l body out and u body out are saved into chip-off memory, as shown in figure 7. And consequently kernel lu edge kc can only run after l body out and u body out of iteration k − 1, and l edge in and u edge in of iteration k have been transferred between SRF and chip-off memory. As a result, there is no overlap of memory access latency between the execution of kernel lu edge kc of iteration k and kernel lu body kc of iteration k − 1. As shown in figure 5, in iteration k, the input streams, l edge in and l body in, are parts of the output stream l body out produced by iteration k − 1; the input streams, u edge in and u body in, are parts of the output stream u body out produced by iteration k − 1. If output streams of iteration k − 1 are organized as l edge in next, l body in next, u edge in next and u body in next, output streams of iteration k − 1 have not to be saved to chip-off memory, but remain in SRF as input streams of iteration k.

Implementation and Optimization of Dense LU Decomposition

85

Dataflow graph of two neighboring iterations of optimized StreamLUD is shown in figure 10. From the figure, we can see that output streams of kernel lu body kc havent to be saved to chip-off memory but saved in SRF as input streams of kernel lu edge kc of next iteration, except stream u edge in next. Because input stream diagonal block T of next iteration depends on output stream u edge in next of last iteration, stream u edge in next has to not only be saved in SRF but also be loaded to chip-off memory. The performance results of exploiting SRF locality optimization are shown in figure 11. Figure 12 shows chip-off memory traffics with and without our optimization. Exploiting SRF locality improves program performance greatly. Over Itanium 2 processor, the speedup yielded by optimized StreamLUD is from 2.33 to 3.30. This optimization not only cuts down memory access but also reduces SRF space needed by StreamLUD, because between adjacent iterations output streams of last iteration and input streams of next iteration are the same. Over unoptimized program, the speedup yielded by optimizing becomes larger and larger with the increase of size. For 64 × 64 and 96 × 96 matrices, exploiting SRF locality cuts down chip-off memory access sharply which weighs highly in total program run time. For other matrices, exploiting SRF locality not only cuts down chip-off memory access but also reduces double buffer. For 128 × 128 matrix, there is even no double buffer during StreamLUD execution. 4.2

Balancing Chip-Off Memory Access and Kernel Execution

As discussed above, while the first block size × block size records of u edge in are factorized, all records of the input stream diagonal block T are the same as the first block size × block size records of u edge in, but the record sequences of

Speedup over Itanium Speedup over Unoptimized

3.0

Without

With

4.0E+06 3.0E+06

2.5

2.0E+06

2.0

1.0E+06

1.5 1.0

0.0E+00 64

96

128

160

192

Fig. 10. Dataflow graph of two Fig. 11. Performance neighboring iterations results of exploiting SRF locality optimization

64

96

128

160

192

Fig. 12. Comparison of chip-off memory traffics with and without our optimization

86

Y. Zhang et al. Speedup over Itanium Speedup over Unoptimized

Without

With

2.0E+06 1.5E+06

3.0

1.0E+06

2.0

5.0E+05 0.0E+00

1.0 64

96

1 28

160

19 2

Fig. 13. Performance results of balancing chip-off memory access and kernel execution

64

96

128

16 0

1 92

Fig. 14. Comparison of chip-off memory traffics with and without our optimization

them are quite different. The former organizes elements of the same column one by one, then the next column; on the contrary, the latter organizes elements of the same row one by one, then the next row. In the optimization of balancing chip-off memory access and kernel execution, the input stream diagonal block T is gotten rid of. When kernel lu edge kc runs, the records of the stream diagonal block T can be supplied by getting records of the first block size × block size u edge in from other clusters via inter-cluster communication. So, by balancing chip-off memory access and kernel execution, output stream u edge in next of the last iteration neednt be saved to chip-off memory. Additionally, as we all known, communication between clusters is far faster than chip-off memory access, so this optimization can improve program performance. The results of balancing chip-off memory access and kernel execution optimization are shown in figure 13, and figure 14 shows chip-off memory traffics with and without our optimization. Balancing chip-off memory access and kernel execution improves program performance greatly. Over Itanium 2 processor, the speedup yielded by optimized method StreamLUD is from 2.56 to 3.64. Over unoptimized StreamLUD, the speedup yielded by our optimization gets larger and larger with the increase of matrix size, except for 192 × 192 matrix. This is because in unoptimized program, records of stream diagonal block T are produced by last iteration, stream u edge in next must be saved to SRF. And consequently saving stream u edge in next and loading stream diagonal block T cant be overlapped with any kernel execution. After optimized, not only kernel lu edge kc neednt stream diagonal block T any more which means chip-off memory access is cut down, but also SRF space needed by StreamLUD is reduced. For 160 × 160 matrix, there is even no double buffer during StreamLUD running. But for 192×192 matrix, double buffer happens, which withdraws program performance.

5

Discussion

StreamLUD gets good performance on the stream processor compared with LUD from HPL on Itanium 2 processor. But before scientific computing applications

Implementation and Optimization of Dense LU Decomposition

87

are dealt with on the stream processor well, some key issues should be dealt with first. – Effectively allocating SRF and utilizing SRF should be solved urgently, because how SRF is employed is the key issue to program performance. – When large matrix is dealt with, double buffer happens. So, it is meaningful to improve program performance with large input and output streams that exceed the capacity of SRF. Although Imagine supply stripmining to solve this problem, it is a semi-auto method and the effect brought by this method depends on programmers very much.

6

Conclusion

In this paper, we implement and optimize dense LU decomposition which is used widely in scientific computing applications on a stream processor that is designed for media applications. Optimizing StreamLUD program makes it get better performance on the stream processor by exploiting it producer-consumer locality and by balancing chip-off memory access and kernel execution. Measurements of StreamLUD implemented and optimized on Isim show that the architecture is successful in developing scientific applications. Dealing with different size matrices, the stream processor gets a speedup from 2.56 to 3.64 over the Itanium 2 processor. Although the stream processor gets good program performance, there is a great deal work to do to make this architecture fit scientific applications well as discussed above.

References 1. Merrimac: Stanford Streaming Supercomputer Project. Stanford University, http://merrimac.stanford.edu/ 2. Dally, W.J., Hanrahan, P., Erez, M., Knight, T.J., Labont´e, F., Ahn, J.H., Jayasena, N., Kapasi, U.J., Das, A., Gummaraju, J., et al.: Merrimac: Supercomputing with Streams. In: Proceedings of the ACM/IEEE SC 2003 Conference (SC 2003), vol. 1, pp. 58113–695 (2003) 3. Erez, M., Ahn, J.H., Jayasena, N., Knight, T.J., Das, A., Labont´e, F., Gummaraju, J., Dally, W.J., Hanrahan, P., Rosenblum, M.: Merrimac: Supercomputing with Streams. In: Proceedings of the, SIGGRAPH GP2 Workshop on General Purpose Computing on Graphics Processors (June 2004) 4. Fatica, M., Jameson, A., Alonso, J.J.: STREAMFLO: an Euler solver for streaming architectures. In: AIAA Conference (submitted) 5. Yuhua, T., Guibin, W.: Application and Study of Scientific Computing on Stream Processor. In: Proceedings of Advances on Computer Architecture (ACA 2006) (2006) 6. Yang, X.J., Du, J.: Implementation and Evaluation of Scientific Computing Programs on Imagine. In: Proceedings of Advances on Computer Architecture (ACA 2006) (2006)

88

Y. Zhang et al.

7. Khailany, B.: The Vlsi Implementation And Evaluation Of Area-And EnergyEfficient Streaming Media Processors. PhD thesis, Stanford University (2003) 8. Owens, J.: Streaming architectures and technology trends. In: International Conference on Computer Graphics and Interactive Techniques (2005) 9. Rixner, S.: Stream Processor Architecture. Kluwer Academic Publishers, Dordrecht (2002) 10. Kapasi, U.J., Rixner, S., Dally, W.J., Khailany, B., Ahn, J.H., Mattson, P., Owens, J.D.: Programmable Stream Processors. Computer 36(8), 54–62 (2003) 11. Mattson, P.R.: A programming system for the imagine media processor. PhD thesis, Stanford, CA, USA, Adviser-William J. Dally (2002) 12. Mattson, P., et al.: Imagine Programming System Developers Guide 13. Das, A., Mattson, P., Kapasi, U., Owens, J., Rixner, S., Jayasena, N.: Imagine Programming System Users Guide 2.0 (2004) 14. The Imagine Project. Stanford University, http://cva.stanford.edu/imagine/ 15. Kapasi, U.J., Dally, W.J., Rixner, S., Owens, J.D., Khailany, B.: The Imagine Stream Processor. In: Proceedings of 2002 IEEE International Conference on Computer Design, pp. 282–288 (2002) 16. Woo, S.C., Singh, J.P., Hennessy, J.L.: The Performance Advantages of Integrating Block Data Transfer in Cache-Coherent Multiprocessors 17. Woo, S.C., Ohara, M., Torrie, E., Singh, J.P., Gupta, A.: The SPLASH-2 programs: characterization and methodologicalconsiderations. In: Proceedings. 22nd Annual International Symposium on Computer Architecture, pp. 24–36 (1995) 18. Zain, U., Ola, J., Magnus, S.: Programming & Implementation of Streaming Applications. Masters thesis, Computer and Electrical Engineering Halmstad University (2005)

An Adaptive Interface for the Efficient Computation of the Discrete Sine Transform, Pedro Alonso1 , Miguel O. Bernabeu1 , and Antonio-Manuel Vidal-Maci´ a1 Universidad Polit´ecnica de Valencia, Camino de Vera s/n, 46022 Valencia, Spain {palonso,mbernabeu,avidal}@dsic.upv.es

Abstract. This paper shows an easy to use interface for applying the Discrete Sine Transform (DST) to a vector. This transform is an FFT– related routine and frequently used in many applications, such as, the translation of a class of structured matrices into another class. The ease of use of the interface has been achieved by exploiting the interesting features of the Fortran 90/95 programming language. In addition, a technique has been incorporated to resolve the performance breakdown arising when the vector size cannot be decomposed into small prime numbers. This breakdown stems from the divide–and–conquer type of algorithms used when applying the DST.

1

Introduction

This paper presents a package to make use of the type I Discrete Sine Transform (also Sine–I Transform, DST–I, or simply DST) [1] of an n–array x,   y = Sx , where S = [Sn ]jk = α sin 2πkj , (1) n+1 for k, j = 0, . . . , n − 1. If α = 1 the transformation is termed unnormalized, otherwise, if α = 2/(n + 1) the transformation is termed normalized [2]. The DST has a wide range of applicability, both in signal processing and in the numerical solution of partial differential equations. One of the most important applications is the mapping of Toeplitz class matrices into the class of Cauchy– like matrices. This mapping has been recently used to build successful parallel routines to solve linear symmetric [3] and hermitian [4] Toeplitz systems. It is well known that the DST is closely related to the Fast Fourier Transform (FFT) [1], so the computational cost of the algorithm when appling the DST to an array is O(log n) operations. Many routine libraries, using differing algorithms, are available for computing the FFT of a vector. Algorithms based on divide–and–conquer methods really obtain a cost of O(log2 n) operations if the size of the problem is a power of two (radix–2); otherwise, they produce very 



Supported by the Directorate of Research and Technology Transfer of the Valencian Regional Administration under grant number GV06/091. Supported by Spanish MCYT and FEDER under Grant TIC 2003-08238-C02-02.

R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 89–98, 2008. c Springer-Verlag Berlin Heidelberg 2008 

90

P. Alonso, M.O. Bernabeu, and A.-M. Vidal-Maci´ a

poor performance results. Other radix–p (p  = 2) can be used, but the performance inevitably breaks down in a manner inversely proportional to the size of the maximum prime number in the prime decomposition of n. Most available FFT packages do not have a specific routine to apply the DST to a n–vector, maybe because this can be obtained by applying a real–data FFT to a 2n + 2 expanded vector. However, some performance and ease of use advantages are lost with this omission. The fftpack is one of the best known packages that includes a DST routine [5]. The routine runs fast because the largest prime number in which n + 1 is decomposed is ‘smaller’ (the concept of small will be set experimentally); otherwise, the cost can be very high. To solve the problem arising when the largest prime number is itself large, a technique called Chirp–z factorization [1] as used in the FFT has been adapted to the DST case. The fftpack and the mentioned factorization technique use additional workspaces and intermediate FFT’s that make their use fairly unintuitive. Thus, a software module has been developed that makes the application of a DST transformation to a vector v as easy as an application to type dst(v); that is, in the same way as in Matlab or Octave. Our module is written in Fortran 90/95 [6] which enables routine calls to be easily made. Moreover, the package makes the best selection between the fftpack routine, or Chirp–z factorization, for use at runtime with negligible overhead. In addition, the nontrivial problem of using the package in multithreaded contexts has been addressed. The following section deals with Chirp–z factorization. In Section 3 the Fortran 90/95 module basics are outlined. The routine selection tool is explained in the following section and the installation process is shown in Section 5. Section 6 shows a real example involving use of the module under differing conditions. Finally, conclusions and further research lines are presented.

2

The Chirp–z Factorization of the DST

It is possible to convert the DST (1) into a convolution of expanded length. The central idea can be found in Bluestein (1970), and is very well explained by C. Van Loan in [1]. The exposition in [1] is a reformulation called the Chirp–z factorization of the FFT that is extended in this paper to the case of the DST. Taking Theorem 4.2.5 of [1] and changing dimensions as needed, the FFT given by matrix:   2πkji F = [F2(n+1) ]jk = exp − , 2(n + 1) √ where k, j = 0, . . . , 2n + 1 and i = −1, can be factorized as: F = ΣT Σ ,

(2)

Σ = diag (σ0 , . . . , σ2n+1 ) ,

(3)

where

An Adaptive Interface for the Efficient Computation

and T is the following Toeplitz matrix: ⎛  ⎞   σ0 σ−1 ... σ−(2n+1) ⎜ ⎟ ..   ⎜ σ ⎟ . ⎜ 1 σ0 σ−1 ⎟ ⎜ . ⎟ .. T = ⎜ ..  ⎟ , . σ1 ⎜ ⎟ ⎜ ⎟ . . .. .. ⎝  σ−1 ⎠  σ2n+1 σ1 σ0

91

(4)

where the tilde denotes the conjugate, and being:   πj 2 i σj = exp − , 2(n + 1) for j = −(2n + 1), . . . , −1, 0, 1, . . . , 2n + 1. Thus, the computation of a DST involves a convolution operation denoted by the product of T by a vector. The convolution can be evaluated using FFT techniques, given yet another computational framework. Given the blocking form of the FFT matrix exposed in [1] through theorems 4.4.1-2, the DST y of a vector x of size n can be computed by means of an FFT, ⎛ ⎞ ⎛ ⎞ y0 0 ⎜y⎟ ⎜ ⎟ ⎜ ⎟ = i F2(n+1) ⎜ x ⎟ , (5) ⎝ z0 ⎠ 2 ⎝ 0 ⎠ z −¯ x where x ¯ represents the vector x with its elements in reverse order. Scalars y0 , z0 and vector z are irrelevant entries. Applying the Chirp–z factorization for the computation of F2(n+1) in (5) yields the Chirp–z factorization for the DST. Summarizing, given a vector x of size n, the problem of computing the DST y = Sx can be solved by means of the computation of a convolution operation composed of two FFT’s and one inverse FFT of size m = 2j , where j is chosen such that m is the minimum value greater than 4n+3, being 4n+3 the minimum order of the circulant matrix containing T used in the convolution operation.

3

The DST Fortran 90/95 Module

The goal of the DST module is to facilitate for the programmer the computation of the DST of a vector. To apply the DST to a given vector v is as easy as to type dst(v), that is, as easy as in Matlab or Octave. The solution is accomplished by means of a Fortran 90/95 module. The use of Fortran 90/95 [7] makes this kind of simple interface possible as it is used in other numerical libraries such as LAPACK95 [8], which is really an interface for LAPACK [9]. The Fortran module hides all the implementation details and enables the optimal computational method. The essential point of the module is explained by the fact that it stores state properties from a call

92

P. Alonso, M.O. Bernabeu, and A.-M. Vidal-Maci´ a

to a public routine to the following call by means of static (save in Fortran) arguments. The main routine of the module is called dst, and this is the only public routine. The interface is dst(v,n_). The first argument is the array to compute the DST. The second argument is an optional integer argument representing the size of the array. If the second argument is not present, the size of the problem is taken from the array by means of the size intrinsic Fortran function. Then, an auxiliary array of size size(v)+1 is allocated because the fftpack routine requires on entry an array of n + 1 entries. Otherwise, the user can call dst with an array of size n + 1 and set the second argument to n_= n, so the routine will work on v and thereby avoid using the auxiliary array. However, the auxiliary array is allocated once because the module keeps it for reuse in further calls. One of the main advantages of this module is that the initialization step performed before the DST computation is automatically carried out only at the first call to dst –as long as the following calls to dst use the same vector size. Thus, one of the state arguments is the actual size (actual_size). The first call to dst performs a call to the initialization fftpack routine: dsinti( n, wsave ) being n the size of the DST and wsave a workspace requested by the routine. The workspace wsave is allocated at this first call and stored in a saved array for further calls as long as the size of the transformation does not change. It is clear that with the module, the workspaces are completely hidden to the user. Further calls will only make calls to the fftpack routine dsint( n, v, wsave ) to apply the DST to v. The same technique is used when the Chirp–z factorization is selected for the computation of the DST. The advantages of this initialization stage are now even clearer. The initialization stage, carried out at the first call to dst as in the other case, is made up of the following tasks: – array allocations, – computation of the size of the FFT that will be used (as shown at the end of Section 2), – initialization of the array Σ (3), – initialization of the FFT that will be used in the convolution of matrix T (4) with other arrays, Other fftpack routines such as dcffti for the initialization of a workspace, or dcfftf (forward FFT), and dcfftb (backward FFT), are used in this case. Workspaces needed for this routines are conveniently allocated, and saved for reuse in following calls. Another of the most important tasks performed by the initialization stage on the first call to dst deals with the decision to be taken about which routine to use to perform the DST: the fftpack routine (dsint); or the Chirp–z factorization based routine (ChirpDST). This decision is made by an external routine

An Adaptive Interface for the Efficient Computation

93

(fftpack or ChirpDST). This routine uses information gathered at installation time (Section 4). Once the selection is made, it is stored in a boolean state variable (use_fftpack) to be used in further calls. The following scheme of the dst routine gives a clear view of the essentials of the process. if( actual_size .ne. n ) then actual_size = n use_fftpack = (fftpack_or_ChirpDST(n) .eq. 1) if (use_fftpack) then call init_dsint(n) else call init_ChirpDST(n) end if end if if (use_fftpack) then call apply_dsint(n, v) else call apply_ChirpDST(n, v) end if The routine takes the decision of making the initialization process based on the actual size stored in the module (actual_size) and the size n of the vector received as actual argument. Obviously, calls with the same array size will result in more efficient use of the module, than calls with alternate sizes. Furthermore, the use of the module from a C code source has been considered. Arrays passed from C code to Fortran 90/95 do not have the same attributes as the Fortran ones because Fortran 90/95 cannot take the array shape from an array allocated by C. The solution consists in implementing the Fortran routine cdst(n,v), where n is the integer representing the size, and v is the array for performing the DST. The only cdst assignment is to make a call to dst. This can be called from any C code in the usual way, cdst_( n, v ) –taking into account the familiar underscore. Features found in Fortran 90/95, such as the use of hidden workspaces, have been used to create user-friendly and efficient code. However, the use of these workspaces resulted in a problem of thread–safe usage of the dst subroutine. Different calls from concurrent threads returned wrong results because they concurrently accessed the same workspace (dst is a non–reentry routine). Protecting workspaces under critical sections is not a solution, because it does not work when the routine is called from different threads with different problem sizes yielding routines which only work well with one problem size at a time. In addition, we noticed that initialized data was problem size specific, so sequences of calls with different n produced unnecessary initialization work and memory allocation/deallocation overheads (i.e. calls with alternate problem sizes n = 2000, n = 1000, n = 2000, . . . ). To ensure thread–safeness and avoid unnecessary workload we have extended the model of the Fortran 90/95 module private data presented in Section 3. Once the routine detects a change in the problem size, a new workspace for

94

P. Alonso, M.O. Bernabeu, and A.-M. Vidal-Maci´ a

the new size n is created and initialized without discarding the previous ones. That is, a new structure made of pointers to real data workspaces and simple variables is created, so future calls with size n will find the structure already initialized. This feature has been implemented by means of a linked list. Starting from an empty list, each time a new problem size is detected, a new node is created and added to the list. These nodes are identified by n. Each contains the null pointers necessary to allocate and initialize data plus a boolean variable identifying whether dsint or ChirpDST will be used for a single problem size. This improvement still does not solve the contention of concurrent data access, because the linked list is located in the heap of the process, a common memory space for all the threads. To fix this problem, every time a thread calls dst and prior to the computation itself, it allocates the necessary memory space in its own stack storage space by copying the initialized data from the heap. Note that only arrays read and written during the computations (i.e. wsave in dsint or the workspace for the convolution of matrix T in the Chirp–z factorization) are processed in this way. Arrays which are only accessed for reading (such as Σ) are used directly from the heap. Furthermore, this improvement avoids the need for the overhead introduced by locks for preventing common storage areas being accessed concurrently.

4

Selection of the Best Routine

One of the main contributions of this work consists in the technique used to select the best DST routine on runtime: the fftpack routine (dsint); or the routine based on Chirp–z factorization (ChirpDST). The cost of the computation of the fftpack routine depends on the size of the prime numbers in which n + 1 is decomposed, where n is the size of the transformation. Thus, the first approach consisted of selecting a threshold prime number, so if the maximum prime number in the decomposition of n+1 was lower than the given threshold, the fftpack routine (dsint) was chosen: otherwise the Chirp–z based routine (ChirpDST) was selected instead. However, our first approach did not work. Using the example shown in Table 1 executed on our target machine and assuming that some prime number lower and close to 127 was selected as the threshold, then, for a problem size of n = 126, ChirpDST was chosen. The selection seemed correct since ChirpDST runs 1.32 times faster than dsint for n = 126. This selection was also suitable for n = 253, that is, with a problem size for which the largest prime value was also larger than the threshold (n + 1 = 2 × 127). However, this selection became incorrect for a larger multiple of 127 (n = 380, n + 1 = 3 × 127). For this last problem size, dsint was almost twice as fast as ChirpDST. An explanation of this behavior is easy in the light of Fig. 1. The figure shows the execution time of both routines for all the different sizes in the range 1– 1024. For the sake of clarity, data is shown in a logarithmic scale of both axes. Time used by dsint is quite short along the different values. Time results from very similar values can be dramatically different (i.e. the time for size n = 1888,

An Adaptive Interface for the Efficient Computation

95

Table 1. Comparison of the fftpack and the Chirp–z factorization based routines n n + 1 prime decomposition dsint/ChirpDST (time ratio) 4n + 3 fft size

126 127 1.32 507 512

253 2 × 127 1.18 1015 1024

380 3 × 127 0.51 1523 2048

1e-02 dsint ChirpDST

sec.

1e-03

1e-04

1e-05

1e-06 1

2

4

8

16

32

64

128

256

512

1024

n+1 Fig. 1. Comparison of time between the fftpack routine (dsint) and the Chirp–z factorization technique (ChirpDST) (logarithmic scales). The sparsity in the dots for dsint shows how differing and ‘unexpected’ can be the execution time, whereas the graphic of ChirpDST shows a layout in ‘time steps’.

where n + 1 = 1889 is prime, is 25.20 ms. and the time for n = 1889, where n + 1 = 1890 = 2 × 33 × 5 × 7, is only 0.18 ms.). Table 2 shows how this problem increases with the increment of the largest prime number and explains the sparsity of dsinttime time values. In the other case, time values of ChirpDST are clustered into time ‘steps’. The DST of an n array with ChirpDST consisting of the computation of three FFT’s of size m, being m the minimum power of two greater than 4n + 3 (Table 1 also shows m). Table 3 shows similar results as Table 1 by using the prime number 1889 as the largest prime number in the decomposition of n + 1. It can be seen that different multiples of 1889 produce time results in different time ‘steps’ of ChirpDST; the ratio between both times varies from 6.31 to 0.63. The interpretation of Fig. 1 clearly means that our first approach was correct, as long as n belongs to the same time ’step’. Size n belongs to ’step x’, for x = 0, . . . , if n ∈ [2x , 2x+1 ). Time of ChirpDST is rather similar for all values

96

P. Alonso, M.O. Bernabeu, and A.-M. Vidal-Maci´ a

Table 2. Comparison of the fftpack and the Chirp–z factorization based routines n dsint/ChirpDST ( dsint time ChirpDST time ) n + 1 prime decomp. 4997 0.017 ( 000.829 48.418 ) 4998 = 2 × 3 × 72 × 17 4998 3.579 ( 173.844 48.578 ) 4999 = 4999 4999 0.008 ( 000.405 48.339 ) 5000 = 23 × 54

x inside the ’step’ because it computes three FFT’s of size m = 2x+3 . Thus, a different threshold must be chosen in each time ’step’.

5

Installation and Tuning

Our tools consist of Fortran 90/95 (Section 3), as well as other Fortran and C routines. The installation process consists of the usual compilation to build a library (libDST.a) and some header files. Following the initial installation stage, a tuning process takes place in which the threshold prime value of each ‘’step’ is computed yielding a table similar to the one in Table 4. The output of the tuning stage shows the x value defined in the previous section in the first column and the threshold in the second column (INT_MAX means that the choice will always be the dsint in the given ’step’). The tuning Table 3. Comparison of time for different ’time’ steps n 1888 3777 5666 11333 17000 prime dec. n + 1 1889 2 × 1889 3 × 1889 2 × 3 × 1889 3 × 3 × 1889 dsint (msec.) 25.2 34.5 38.7 60.8 84.9 ChirpDST (msec.) 4.0 11.1 26.1 59.3 134.6 dsint/ChirpDST 6.31 3.11 1.48 1.03 0.63 fft size 8192 (213 ) 16384 (214 ) 32768 (215 ) 65536 (216 ) 131072 (217 )

Table 4. Example of ’step’ prime thresholds obtained by the tuning process step (x) 0 1 2 3 4 5 6 7 8 9

threshold INT MAX INT MAX INT MAX INT MAX INT MAX INT MAX 89 163 193 251

An Adaptive Interface for the Efficient Computation

97

stage can take some time because different size problems must be tested in order to obtain the threshold prime. The table obtained is used to build a C header file that will be used by the selection routine (fftpack_or_ChirpDST). Later, the software is compiled again and the definitive library for the given target machine is completely installed. The dst routine will, in a negligible time, chose the best routine at its first call on runtime.

6

Example of Use

The complete module with the improvements introduced in this section has been proved to be useful in the solution of symmetric Toeplitz linear systems [3], Tx = b

T ∈ IRn×n x, b ∈ IRn .

(6)

The first stage to solve problem (6) consists of translating the problem itself to the field of Cauchy–like matrices T x = b → Cx ˆ = ˆb ,

where C = ST S, xˆ = Sx and ˆb = Sb ,

(7)

being C a symmetric Cauchy–like matrix. The transformation is carried out by means of a DST of size n, S (1). It is known that matrix C has a large sparsity that can be exploited since entries ci,j , i, j = 0, . . . , n − 1, such that i + j is odd and zero. Performing an odd-even permutation of the Cauchy–like linear system (7), two independent linear Cauchy–like systems arise of size n/2 and n/2 , respectively. Once these two reduced linear systems are solved by the corresponding two concurrent threads, each makes a DST call (size n/2 and n/2 , respectively) to undo the transformation that lets it return to the Toeplitz field and yields the solution to the original linear system (6). Through this practical example, we can see how the Fortran 90/95 module provides the programmer with an easy method to handle with the DST by making simple calls to dst and only passing the target array as argument, so freeing the programmer from worry about workspaces, data initialization, or selecting the best performance routine - as the array size is set at each call in a multi-thread context.

7

Conclusions

An initial real transformation (DST) of a future library consisting of different FFT–related routines has been developed. With this first routine, the basis of how to build all of the FFT–related routines with an easy to use interface and a performance free of drawbacks has been set. The first goal is achieved thanks to the features introduced by Fortran 90/95, one of the most used programming languages in numerical algorithms, and which even provides an easy interface

98

P. Alonso, M.O. Bernabeu, and A.-M. Vidal-Maci´ a

for traditional C language. The second objective has been achieved by means of an intelligent installation/tuning process. As numerical programmers know, a very powerful and popular package for using FFT–related functions called FFTW [10] is currently available. This package has been set as a standard because the new Intel processors are provided with the same interface as the FFTW for using native FFT–related routines. Our interface is so simple that it can be used like a layer over the FFTW function calls. Our future library will offer the option of being used over the fftpack of FFTW routines, and so in the second case, providing it with the performance of native routines.

References 1. Loan, C.V.: Computational Frameworks for the Fast Fourier Transform. SIAM Press, Philadelphia (1992) 2. Bojanczyk, A.W., Heinig, G.: Transformation techniques for toeplitz and toeplitzplus-hankel matrices part I. transformations. Technical Report 96-250, Cornell Theory Center (1996) 3. Alonso, P., Vidal, A.M.: The symmetric–toeplitz linear system problem in parallel. In: Sunderam, V.S., van Albada, G.D., Sloot, P.M.A., Dongarra, J. (eds.) ICCS 2005. LNCS, vol. 3514, pp. 220–228. Springer, Heidelberg (2005) 4. Alonso, P., Bernabeu, M.O., Vidal, A.M.: A parallel solution of hermitian toeplitz linear systems. In: Alexandrov, V.N., van Albada, G.D., Sloot, P.M.A., Dongarra, J. (eds.) ICCS 2006. LNCS, vol. 3991, pp. 348–355. Springer, Heidelberg (2006) 5. Swarztrauber, P.: Vectorizing the FFT’s. Academic Press, New York (1982) 6. Metcalf, M., Reid, J.K.: Fortran 90/95 explained, 2nd edn. Oford University Press, Inc., New York (1999) 7. Akin, E.: Object Oriented Programming Via FORTRAN 90/95. Cambridge University Press, New York (2003) 8. Barker, V.A., Blackford, L.S., Dongarra, J., Croz, J.D., Hammarling, S., Marinova, M., Wasniewski, J., Yalamov, P.: LAPACK95 Users’ Guide. SIAM: Software, Environments and Tools, SIAM, Society for Industrial and Applied Mathematics, 3600 University City Science Center, Philadelphia, PA 19104-2688, USA (2001) 9. Anderson, E., Bai, Z., Bischof, C., J., D., Dongarra, J.: LAPACK Users’ Guide LAPACK Quick Reference Guide to the Driver Routines: Release 2.0, 2nd edn. SIAM Press, Philadelphia (1995) 10. Frigo, M., Johnson, S.G.: The design and implementation of FFTW3. Proceedings of the IEEE 93, 216–231 (2005), special issue on Program Generation, Optimization, and Platform Adaptation

Incomplete WZ Factorization as an Alternative Method of Preconditioning for Solving Markov Chains Beata Bylina and Jaroslaw Bylina Department of Computer Science Institute of Mathematics Marie Curie-Sklodowska University Pl. M. Curie-Sklodowskiej 1, 20-031 Lublin, Poland [email protected], [email protected]

Abstract. The purpose of the article is to present and evaluate usefulness of a new preconditioning technique (for the Gauss-Seidel algorithm), namely incomplete WZ factorization, for iterative solving of sparse and singular linear equations systems, which arise during modeling with Markov chains. The incomplete WZ factorization proposed here will be compared with the incomplete LU factorization in respect of amount of fill-in (newly created non-zeros) and in respect of the accuracy improvement of preconditioned algorithms in relation to not preconditioned ones. In the paper, the results of some numerical experiments will be presented, which were conducted for various matrices representing Markov chains. The experiments show that the incomplete WZ factorization can be a real alternative – because it is faster than incomplete LU factorization and the fill-in generated in the process is smaller (the output matrices are sparser).

1

Introduction

During modeling a real system (as, for example, a computer system or a communications network) with Markov chain we come across a linear system (of n equations with n unknowns), when we want to find stationary probabilities of states of the modeled system [15]: QT x = 0,

n 

xi = 1,

x ≥ 0,

(1)

i=1

where the unknown x = (x1 , . . . , xn ) is a vector of probabilities of states of the modeled system after a long run (xi being the probability of the ith state) and the matrix Q = (qij ) is an infinitesimal generator (that is a transition rate matrix) of the given Markov chain (qij being a rate of a transition from the ith state to the jth state). R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 99–107, 2008. c Springer-Verlag Berlin Heidelberg 2008 

100

B. Bylina and J. Bylina

The equation (1) has an unambiguous solution in the most interesting cases (when the Markov chain is irreducible, so the rank of Q is n − 1). However, the matrix Q has some unusual properties, which require special treatment. Namely, the matrix Q is a huge, sparse, ill-conditioned matrix with a weakly dominant diagonal. For solving the system (1), one generally uses iterative [15], projection and decomposition methods. However, sometimes direct methods are used – especially when high accuracy is needed [5,15]. All the methods have their own advantages and disadvantages. Two methods of solving linear systems can be joined – namely, direct methods with iterative methods [8]. The preconditioning is an example of such a fusion. The very concept of preconditioning is almost as old as iterative methods [11]. The idea of incomplete factorization was presented by Buleev [2,3] and Varga [16]. The papers that popularized the incomplete factorizations were [12,13]. There is a need for fast, stable, scalable, easy to parallelize and generating a small fill-in preconditioners. Incomplete LU factorization has some of these features, but the paper presents another one – which, perhaps, is better. Moreover, the work on incomplete factorizations is needed for other purposes – as, for example, for block approximate inverse [1]. Preconditioning is presented in Section 2, which also describes incomplete LU and WZ factorizations and their application for preconditioning with the cost of computations needed for such a preconditioning. Section 3 includes results of a numerical experiment which consists in solving linear systems arising during Markovian modeling with the Gauss-Seidel method preconditioned with the incomplete WZ and LU factorizations. Section 3 also analyzes properties of both preconditioners and their influence on the convergence of Gauss-Seidel method. Section 4 contains some conclusion.

2

Preconditioning

Convergence rate of iterative methods depends on properties of the coefficient matrix of the linear system. The matrix Q is ill-conditioned what can make convergence of iterative methods slow. One of manners for preventing such problems is to transform the system (1) into an equivalent system (having the same solution), but with better numerical properties. Such a transformation can be done by preconditioning, that is by converting the system (1) into: M−1 QT x = 0,

n 

xi = 1,

x ≥ 0,

(2)

i=1

where the nonsingular matrix M (known as a preconditioner) approximates the matrix QT in a manner. The system (2) has the same solutions as (1) but it is (hopefully) better conditioned. Generally, computing and using a good preconditioner is an expensive task consisting of finding the matrix M and its inverse. If the preconditioning is to be used, that cost should be refunded by reduced number of iterations needed

Incomplete WZ Factorization as an Alternative Method of Preconditioning

101

to acquire required accuracy – or by using the same preconditioner for various linear systems. There are various techniques used for finding the preconditioner matrix M, none of them sufficiently universal. To verify the usability of a preconditioning technique, the only sure method is heuristics. The preconditioner matrix is usually built on the basis of the original coefficients of the matrix Q. In [1] preconditioners for Krylov subspace methods for solving large singular linear systems arising from Markov modeling are considered. Here we describe a manner of constructing preconditioners based on incomplete factorizations for stationary iterative methods. 2.1

Incomplete LU Factorization

Incomplete LU factorization (denoted ILU) is based on the well known LU fac and torization, where a lower triangular matrix (with ones on the diagonal) L  are found and where the preconditioner matrix an upper triangular matrix U U  is a kind of approximation for the matrix QT . M=L There are many variants of ILU, the most straightforward being ILU(0) [15]. In ILU(0) computations are conducted as in traditional (complete) LU factorization (that is Gaussian elimination), but new non-zero elements (lij and uij ) arising in the process are dropped if they appear in the place of a zero element in the original matrix QT . Hence, the factors together have the same number of nonzeros as the original matrix QT . Thereby, the most important problem of the factorization of sparse matrices – the fill-in (which consists in appearing non-zero elements in new matrices on the places of zero elements in the original matrix, what makes dense the output factors and and renders impossible their packed storage) – is eliminated at the expense of accuracy. After ILU(0) we have: U  + RLU , QT = L (3)  and U  are (respectively) lower and upper triangular matrices and the where L remainder matrix RLU is supposed to be small in a sense.  U,  then M−1 = U  −1 L  −1 and the equation (2) takes the shape: Let M = L n 

 −1 L  −1 QT x = 0, U

xi = 1,

x ≥ 0.

(4)

i=1

 −1 L  −1 QT . Now, the equation (4) takes the shape: Let SLU = U SLU x = 0,

n 

xi = 1,

x ≥ 0.

(5)

i=1

2.2

Incomplete WZ Factorization

The WZ factorization consists in decomposition of the given matrix (QT in the paper) into a product of two matrices: W and Z (Figure 1).

102

B. Bylina and J. Bylina

1

0

0

0

0 1 Fig. 1. The form of the output matrices in the WZ factorization (left: W; right: Z)

The WZ factorization was proposed by Evans and Hatzopoulos [10] as a manner of matrix factorization intended for SIMD machines (Single Instruction stream – Multiple Data stream in Flynn classification; such an architecture is characterized by multiple processing units). In [6,9,10,17] the WZ is developed, as well as its modifications and implementations (parallel ones, among others). Incomplete WZ factorization [4] (denoted IWZ) is based on the described  and Z  (of the form of matrices above WZ factorization, where we find matrices W   W and Z shown in Figure 1) and the product WZ is a kind of approximation for the matrix QT . Let us present the simplest form of IWZ, that is IWZ(0). In IWZ(0) computations are conducted as in complete WZ factorization, but new non-zero elements (wij and zij ) arising in the process are dropped if they appear in the place of a zero element in the original matrix QT . Hence, the factors together have the same number of non-zeros as the original matrix QT – as in ILU(0) – and we have no fill-in.  Z,  then M−1 = Z  −1 W  −1 and the equation (2) takes the shape: Let M = W n 

 −1 W  −1 QT x = 0, Z

xi = 1,

x ≥ 0.

(6)

i=1

 −1 W  −1 QT . Now, the equation (4) takes the shape: Let SW Z = Z SW Z x = 0,

n 

xi = 1,

x ≥ 0.

(7)

i=1

There is one more thing worth noting – because of singularity of the matrix QT its complete factorizations (both LU and WZ) are difficult to compute. However, factors of an incomplete factorization of a sparse matrix are singular with a zero probability, hence incomplete factorizations are easier to do.

Incomplete WZ Factorization as an Alternative Method of Preconditioning

3

103

Numerical Experiment

The experiment was conducted on a Pentium IV 2.8GHz computer, 1GB RAM, with Debian GNU/Linux operating system. The described algorithms were implemented with the language C and compiled with gcc using an optimizing option -O3. Data structures for storing all the matrices were full two-dimensional square arrays, fitting wholly in the machine’s RAM. Matrices (with IDs from 1 to 6) used in tests were generated by the paper authors on the basis of some abstract queuing models – the matrices are infinitesimal generators of Markov chains describing these models – and they are neither symmetric nor anyway structural. In Table 1 the essential characteristics of the matrices are presented (n is the number of rows/columns of the matrix, nz is the number of non-zeros in the matrix and d is an average number of non-zeros in a row/column of the matrix). Table 1. Essential characteristics of the matrices used in tests matrix ID n nz d 1 100 1190 11.9 2 1000 7744 7.7 3 1500 37955 25.3 4 1500 5873 3.9 5 3000 120590 40.2 6 3000 11636 3.9 7 2116 12376 5.9

The matrix 7 was generated from a standard two-dimensional model [7,14]. The states of such a chain are described with two numbers (u, v), u = 0, . . . , Nx , y = 0, . . . , Ny (here Nx = Ny = 45) and transitions are only allowed from (u, v) to (u , v  ) if |u − u| ≤ 1 and |v  − v| ≤ 1. There was assumed – as in [7] – that only some transition from each state are permitted. Each of these matrices was incompletely factorized with the use of algorithms ILU(0) and IWZ(0) and then the matrices SLU and SW Z are created. Table 2 presents times of ILU(0) and IWZ(0) factorization. In Table 3 numbers of non-zeros in matrices SLU and SW Z are presented. It can be noticed that the number of non-zeros in those matrices depends on sparsity of the input matrix QT . More important, the number of non-zeros in SW Z is lesser than in SLU . In the paper, the aim of incomplete factorization is its usage as preconditioning for Gauss-Seidel method. That is we use Gauss-Seidel method (but others iterative, projection etc. methods can be used as well) to solve the equation (5) when ILU(0) is used, or the equation (7) when IWZ(0) is used – instead of the original equation (1). Gauss-Seidel method is given by: x(k+1) = (D − L)−1 Ux(k) ,

(8)

104

B. Bylina and J. Bylina

or in the scalar form: i−1 (k+1) xi

=

(k+1) j=1 lij xj

+

n

(k)

j=i+1

dii

uij xj

,

for i = 1, . . . , n,

(9)

where the coefficient matrix (in the paper: QT , SLU and SW Z ) is written as D − L − U (D = (dij ) is a diagonal matrix, U = (uij ) is an upper triangular matrix with a zero diagonal, L = (lij ) is a lower triangular matrix with a zero diagonal). n Because equation (1) without the additional condition i=1 xi = 1 has an infinite number of solution vectors (and so have (5) and (7)), the output vector x have to be normalized after computations (in this case it should be divided by its first norm ||x||1 ). After that, when rankQT = n − 1 (which n is true in our cases) there will be only one solution, it will fulfil the condition i=1 xi = 1, and moreover, it will fulfill the condition x ≥ 0 (it is true when Q is an infinitesimal generator of a Markov chain). (0) (0) A vector x(0) = (xi ) with xi = 1i was chosen as an initial vector. As a measure of accuracy of the solution we chose: ε(k) = ||0 − QT x(k) ||2 .

(10)

Table 2. Comparison of times of ILU(0) and IWZ(0) factorization matrix ID IWZ(0) time [s] ILU(0) time [s] 1 0.01 0.01 2 1.17 2.18 3 4.60 8.13 4 3.02 5.80 5 30.69 57.59 6 20.43 41.30 7 9.88 15.04

Table 3. Numbers of non-zeros in matrices SLU and SW Z (distinguished IDs are for matrices which are more sparse (d ≤ 8) and where the greater improvement in sparsity is seen at the same time)

matrix ID nz(SLU ) nz(SW Z ) 1 9901 9632 2 910496 535808 3 2250000 2155092 4 200173 66250 5 9000000 8766420 6 397272 137560 7 3777061 732514

by how much percent SW Z is sparser than SLU (nz(SLU ) − nz(SW Z ))/nz(SLU ) 2.72% 41.15% 4.22% 66.90% 2.60% 65.37% 80.61%

Incomplete WZ Factorization as an Alternative Method of Preconditioning

105

Table 4. Comparison of accuracy of tested methods (distinguished IDs are for matrices which are more sparse (d ≤ 8) and where the greater improvement in accuracy is seen at the same time) matrix ID step 1 1 5 10 2 1 5 10 3 1 5 10 4 1 5 10 5 1 5 10 6 1 5 10 7 1 50 100

GS IWZ(0)+GS ILU(0)+GS 5.0e − 01 1.4e − 01 1.7e − 01 4.5e − 04 1.8e − 05 6.6e − 05 6.8e − 08 3.7e − 10 2.16e − 09 1.7e − 01 5.9e − 02 4.7e − 02 9.6e − 04 2.1e − 06 1.1e − 06 1.0e − 06 3.6e − 11 1.9e − 11 1.3e − 01 2.3e − 02 1.9e − 02 2.5e − 05 1.5e − 06 2.7e − 05 3.2e − 10 3.6e − 11 1.4e − 10 1.4e − 01 8.8e − 02 7.5e − 02 7.5e − 03 1.6e − 04 1.1e − 04 1.5e − 04 4.0e − 08 1.4e − 08 9.6e − 02 1.2e − 02 1.1e − 02 7.1e − 06 8.2e − 07 1.5e − 06 9.7e − 11 2.6e − 11 7.1e − 11 1.1e − 01 6.3e − 02 5.3e − 02 5.8e − 03 1.1e − 04 8.9e − 05 1.5e − 04 2.9e − 08 1.1e − 08 5.2e − 02 6.5e − 02 1.7e + 00 1.1e − 01 1.6e − 09 1.2e − 15 1.3e − 08 1.5e − 15 1.2e − 15

Table 4 shows values of (10) for selected steps of Gauss-Seidel method for three method: Gauss-Seidel alone (denoted GS), Gauss-Seidel preconditioned with IWZ(0) (denoted IWZ(0)+GS) and Gauss-Seidel preconditioned with ILU(0) (denoted ILU(0)+GS).

4

Conclusion

The paper presents the incomplete WZ factorization, and compares it with the incomplete LU factorization. We can see that the cost of both incomplete factorizations is O(n3 ), both the preconditioners improve significantly convergence of Gauss-Seidel method – especially for very sparse matrices (as for example the matrix 2, 4 etc. – the more sparse matrix, the better effects of preconditioning – see also [1]), the accuracy is roughly the same for both preconditioners (slightly to the advantage of IWZ(0)). So, for the accuracy sake, there is no matter which of the factorizations we choose (IWZ(0) or ILU(0)). However, from the results of the experiment we can conclude the following. 1. When we care about time of execution, we should choose IWZ(0), because it is about twice as fast as ILU(0) (it is so, because loops in IWZ(0) have

106

B. Bylina and J. Bylina

half the repetitions of loops in ILU(0), but these loops’ turns are longer and an optimizing compiler – as gcc -O3 – can more effectively use minivector properties of modern processors and multi-level organization of the memory). These results relate to authors’ own implementations of both the algorithms. 2. The number of non-zeros in SW Z is lesser than in SLU . 3. For very sparse matrices (d ≤ 8) the matrix SW Z is much more sparse than the matrix SLU and for such matrices the Gauss-Seidel method improves very much with preconditioning. 4. For the matrix 7 it turned out that it needed much more iterations to get sufficient accuracy. However, it can be observed that the presented algorithms improve the convergence of the Gauss-Seidel method. It is contrary to observations in [7] – however, the Gauss-Seidel method was not considered there (other iteration methods were, with SOR being the most similar). On the basis of the results presented it appears that the incomplete WZ factorization as a preconditioner is a good alternative for preconditioning with the LU factorization, especially for sparse matrices – because it is faster than incomplete LU factorization and the fill-in generated in the process is smaller.

Acknowledgements This work was partially supported by Marie Curie-Sklodowska University in Lublin, Poland within the project Niekompletny rozklad WZ jako uwarunkowanie wstepne (ang. preconditioning) stosowane do znajdowania wektora prawdopodobienstw stanow sieci calkowicie optycznych modelowanych lancuchami Markowa. This work was also partially supported within the project Metody i modele dla kontroli zatloczenia i oceny efektywnosci mechanizmow jakosci uslug w Internecie nastepnej generacji (N517 025 31/2997).

References 1. Benzi, M., Ucar, B.: Block Triangular preconditioners for M-matrices and Markov chains. Electronic Transactions on Numerical Analysis 26 (to appear, 2007) 2. Buleev, N.I.: A numerical method for solving two-dimensional diffusion equations. At. Energ. 6, 338 (1959) (in Russian) 3. Buleev, N.I.: A numerical method for solving two- and three-dimensional diffusion equations. Mat. Sb. 51, 227 (1960) (in Russian) 4. Bylina, B., Bylina, J.: Niekompletny rozklad WZ jako uwarunkowanie wst¸epne dla rozwi¸azywania Markowowskich modeli sieci optycznych. Sieci komputerowe, Wydawnictwa Komunikacji i L  a¸czno´sci, Warszawa, tom 1 (Nowe technologie), p. 103 (2007) (in Polish) 5. Bylina, B., Bylina, J.: The Vectorized and Parallelized Solving of Markovian Models for Optical Networks. In: Bubak, M., van Albada, G.D., Sloot, P.M.A., Dongarra, J. (eds.) ICCS 2004. LNCS, vol. 3037, p. 578. Springer, Heidelberg (2004) 6. Chandra Sekhara Rao, S.: Existence and uniqueness of WZ factorization. Parallel Computing 23, 1129 (1997)

Incomplete WZ Factorization as an Alternative Method of Preconditioning

107

7. Dayar, T., Stewart, W.J.: Comparison of Partitioning Techniques for Two-Level Iterative Solvers on Large, Sparse Markov Chains. SIAM Journal on Scientific Computing 21, 1691 (2000) 8. Duff, I.S.: Combining direct and iterative methods for the solution of large systems in different application areas. Technical Report RAL-TR-2004-033 9. Evans, D.J., Barulli, M.: BSP linear solver for dense matrices. Parallel Computing 24, 777 (1998) 10. Evans, D.J., Hatzopoulos, M.: The parallel solution of linear system. Int. J. Comp. Math. 7, 227 (1979) ¨ 11. Jacobi, C.G.J.: Uber eine neue Aufi¨ osungsart der bei der Methode der kleinsten Quadrate vorkommenden linearen Gleichungen. Aston. Nachrichten 22, 297 (1845) 12. Kershaw, D.S.: The incomplete Cholesky conjugate gradient method for the iterative solution of systems of linear equations. J. Comput. Phys. 26, 43 (1978) 13. Meijerink, J.A., van der Vorst, H.A.: An iterative solution method for linear systems of which the coefficient matrix is a symmetric M-matrix. Math. Comput. 31, 148 (1977) 14. Pollett, P.K., Stewart, D.E.: An Efficient Procedure for Computing QuasiStationary Distributions of Markov Chains with Sparse Transition Structure. Advances in Applied Probability 26, 68 (1994) 15. Stewart, W.: Introduction to the Numerical Solution of Markov Chains. Princeton University Press, Chichester, West Sussex (1994) 16. Varga, R.S.: Factorizations and normalized iterative methods. In: Langer, R.E. (ed.) Boundary Problems in Differential Equations, Univ. Wisconsin Press, Madison (1960) 17. Yalamov, P., Evans, D.J.: The WZ matrix factorization method. Parallel Computing 21, 1111 (1995)

A Block-Based Parallel Adaptive Scheme for Solving the 4D Vlasov Equation Olivier Hoenen and Eric Violard Laboratoire LSIIT - ICPS, UMR CNRS 7005 Universit´e Louis Pasteur, Strasbourg {hoenen,violard}@icps.u-strasbg.fr

Abstract. We present a parallel algorithm for solving the 4D Vlasov equation. Our algorithm is designed for distributed memory architectures. It uses an adaptive numerical method which reduces computational cost. This adaptive method is a semi-Lagrangian scheme based on hierarchical finite elements. It involves a local interpolation operator. Our algorithm handles both irregular data dependencies and the big amount of data by distributing data into blocks. Performance measurements on a PC cluster’s confirm the pertinence of our approach. This work is a part of the CALVI project1 . Keywords: Parallel numerics, 4D Vlasov equation, Adaptive method.

1

Introduction

The Vlasov equation can describe the evolution in time of charged particles under the effects of electro-magnetic fields. It is used to model important phenomena in plasma physics such as controlled thermonuclear fusion. This equation is defined in the phase space, i.e., the position and velocity space which has 6 dimensions in the real case (one dimension of velocity for each dimension of position). Methods discretizing the Vlasov equation on a mesh of phase space has been proposed to get an accurate description of the physics. Due to the high number of dimensions of the equation domain, solving this equation with such a numerical method yields a very large computational problem. In order to reduce this computational cost, adaptive methods have been developed. Some of these methods are based on the semi-Lagrangian schemes [1,2]. We developed such an adaptive method and its parallel implementation for the 2D Vlasov equation [3]. But a higher number of dimensions brings new challenges. On an other hand, some parallel implementations of 4D adaptive Vlasov solver exist but they are essentially designed for shared memory architectures. Therefore new parallel adaptive schemes have to be developed for distributed memory machines.  1

Supported by a grant from Alsace Region. CALVI is a french INRIA project devoted to the numerical simulation of problems in Plasma Physics and beams propagation.

R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 108–117, 2008. c Springer-Verlag Berlin Heidelberg 2008 

A Block-Based Parallel Adaptive Scheme

109

This paper presents an appropriate block-based semi-Lagrangian scheme and its parallelization. This scheme is based on a hierarchical finite element decomposition [4] which involves a local interpolation operator. This method yields an efficient parallelization based on data distribution into blocks. The paper is organized as follows. Section 2 presents our adaptive semiLagrangian scheme. Section 3 presents in details its parallel implementation. Section 4 shows our experimental results before concluding.

2

An Adaptive Numerical Scheme

We consider the four dimensional Vlasov equation ∂f ∂f ∂f +v + E(t, x) = 0. ∂t ∂x ∂v

(1)

whose unknown f (t, (x, v)) represents the distribution of particles at time t, where x = (x, y) ∈ R2 and v = (vx , vy ) ∈ R2 are coordinates of a point in the phase space, and E(t, x) is the, so called, self consistent electrostatic field, which is generated by particles. The equation is coupled with the Poisson’s equation which gives the electric field E. Our resolution scheme is based on the splitting of the equation (introduced in [5]) into a succession of three transport equations of the form ∂f ∂f + U (t, z) =0. ∂t ∂z

(2)

whose resolution, so called advection in z (where z stands for phase space coordinates), is performed by using a semi-Lagrangian scheme. This scheme uses the property of conservation of the unknown along the characteristic curves, i.e., n+1 Property 1. f (tn+1 , (x, v)) = f (tn , AΔt − tn is the time z (x, v)) where Δt = t Δt step and Az , so called advection operator, is a one-to-one correspondence between points of the phase space (see [6] for more details).

Our numerical method therefore boils down to perform three successive advections on a phase space mesh : one in x, one in y and one in v. These advections are defined by the advection operators: AΔt x (x, v) = ((x − vx Δt , y) , v), AΔt y (x, v) = ((x , y − vy Δt) , v), AΔt v (x, v) = (x , v − E(t, x)Δt) .

(3)

We use a dyadic structured adaptive mesh, i.e., a mesh whose each cell belongs to an uniform grid of 2j cells per dimension. The integer j is called the level of the cell. In this 4-dimensional dyadic mesh, each cell of level j, say α, is a 4-cube and can be refined into 24 smaller cells of level j + 1. These smaller cells are called the daughters of α and, by analogy, cell α is called their mother. We will denote J, the finest level and j0 , the coarsest level of our mesh cells.

110

O. Hoenen and E. Violard

At any time step tn = nΔt, the solution f is represented by a dyadic mesh Mn and a function F n which gives the value of f at every point (x, v) corresponding to a node of Mn . Each cell has 34 equally spaced nodes and we use a biquadratic Lagrange interpolation to reconstruct the value of f at any point (x, v) within a cell. Let us now present one advection in z, i.e., our numerical scheme for solving one transport equation. It gives the new representation (Mn+1 , F n+1 ) from a known old one (Mn , F n ). It consists in the following procedure, for every cell α of the uniform grid of level j0 and denoting Mn+1 the part of Mn+1 which is α contained in α: 1. Mesh prediction: Let us note jβ , the level of any cell β; Cβ , the point of the phase space corresponding to the center of β; β  , the cell of Mn  which contains the advected point AΔt z (Cβ ) and jβ  , the level of β . Then n+1 recursively refine each cell β of Mα such that jβ  ≥ jβ . 2. Computation of values: For every node of Mn+1 α , let N be its corresponding point in the phase space and α , the cell of Mn which contains the n+1 advected point AΔt (N ) to the z (N ). By conservation property 1, set F Δt n interpolated value at point Az (N ) by using the values of F at nodes of α . 3. Mesh compression: For every group of 24 daughter cells in Mn+1 α , compare every value at their nodes with the value obtained by interpolation by using the values at mother’s nodes. If the L1-norm of the difference is lower than a given threshold , then replace the daughter cells with their mother into Mn+1 α . Figure 1 shows the time loop of our resolution scheme. Notice that for sake of conciseness, the diagnostic step has not been reported. As the advection in v step uses the electric field E, it is preceded by the computation of E. In order to compute E, the Poisson’s equation is discretized onto an uniform grid of the position space (x) with 2J+1 +1 points per dimension. The computation of E then consists in a summation of values onto the 2D position space. Mn,Fn

advection in x

advection in y

comput. of E

advection in v

Mn+1,Fn+1

Fig. 1. The time loop of our resolution scheme

3

Parallelization

In our numerical scheme an advection consists in applying the same treatment to every parts Mn+1 of the dyadic mesh in any order. In the following we call α block all the data used to describe such a part. Our parallel implementation is based on the distribution of blocks amongst processors. We first present the data structure which will represent the dyadic mesh. We propose a partitioning of the mesh which will determine how blocks are distributed. Then we show how to optimize communications.

A Block-Based Parallel Adaptive Scheme

3.1

111

Data Structure

Data structure has a great impact on the efficiency of an adaptive method implementation. In our case, due to the domain high number of dimensions and the big amount of data it represents, we search a good tradeoff between data access cost and memory usage. In our algorithm, the access to any value is defined by the advection operator that uses an absolute location within the phase space. Therefore, random access is critical to our algorithm (as shown in [7]). Thus, we use arrays to store values. Moreover, we use pointers to handle mesh sparsity and we use dynamically-allocated arrays to minimize memory usage. Our data structure is composed with four-dimensional arrays and has two level. The first level is made of two arrays: one [2j0 +1 ]4 -sized array of double which stores the values at nodes of the cell of level j0 and one [2j0 ]4 -sized array of pointers whose elements are in one-to-one correspondence with cells of the uniform grid of level j0 . Each pointer either is NULL meaning that the corresponding cell belongs to the mesh, or points to an array of second level. An array of second level stores the values of all the nodes which are located within the corresponding part of the mesh. The undefined value is stored in the array when the corresponding node does not exist in the mesh. The second level is thus made of some [2J−j0 +1 ]4 -sized arrays of double. Moreover since the mesh adapts at each time step, the arrays of second level are dynamically-allocated arrays. A data block either is stored into the array of first level if it describes a single cell of level j0 , or is stored into an array of second level. 3.2

Mesh Partitioning

A good mesh partitioning is one which both reduces the cost of communications, and balances the computational load [8]. In order to meet the first issue, we use the following properties of our numerical method : – During any advection in v, the treatment of any block, say B, only requires data within blocks having the same coordinates in position space as B. Symmetrically, during any advection in x, the treatment of any block B only requires data within blocks having the same coordinates in velocity space as B. – Any v-advection exhibits irregular and unpredictable data dependencies because these dependencies are defined by the self consistent electrostatic field. On an other hand, the data dependencies in any x-advection are predictable as they are only defined by v and dt. According to these properties, if we partition the mesh along the position dimensions, then the advections in v will induce no communication. Moreover, the communications which will occur during the advections in x are predictable and linear. Hence, we choose to reduce the partitioning problem to partitioning the 2D uniform grid of level j0 of the position space. This means that each partition is defined by an 2D area made of some cells of this uniform grid. The partition contains all the mesh cells of the 4D phase space whose projection onto the

112

O. Hoenen and E. Violard

position space is included in this area. As said previously the induced communications are linear, therefore we only consider connex areas to form our partitions. More precisely, we choose to define our partitions from some rectangular areas, which simplifies the communications scheme. Now to achieve the second requirement that is load balancing we build approximately balanced areas for any given test case. Notice that our approach in this work is to perform a static partitioning. Future work will be directed at modifying partitions dynamically to follow the evolution in time of the physics more closely. As said previously, each area will be a 2D rectangular one. Numerous techniques tends to build such areas by minimizing the ratio between height and width and reduce the amount of border elements in each direction that usually induces communications (see [9] as an example). Besides these techniques, our partitioning uses some knowledge on the considered test case. More precisely, it uses a bounding box, say B, which approximates the shape of the beam of particles. It is assumed that most of computational load is carried by the mesh elements contained within bounding box B. The other parameters used in the building of our partitions are P , the number of processors and j0 , the level of the uniform grid. Notice that level j0 defines the coarse grid and determines the number of discretisation points per surface unit of the physics domain. We therefore consider it as a parameter and not as an unknown because we do not want to influence the solution accuracy. The partitioning problem then reduces to find P rectangles composed of coarse cells such that these rectangles do not overlap and every rectangle approximatively covers the same extent of surface of B. Our partitioning problem is an optimization one which is minimizing the greatest area amongst the P surface extents of B which are covered by each of the P rectangles. This optimization problem can be expressed as an 0-1 integer linear programming problem. It is thus in general NP-hard, and as such, it is considered unlikely that there exists an efficient algorithm for solving it. Instead, we propose an heuristic which consists in recursively splitting the bounding box area into 2 approximatively equal parts. It is thus assumed that the number of processors is a power of 2. 3.3

Obtaining Regular Communications

The communications are generated by the data dependences which are induced by the advection operator: the treatment of a node corresponding to the point (x, y, vx , vy ) of the phase space requires the values of the cell containing the advected point. As we partition the mesh along the (x, y)-axis (Cf. section 3.2), the communications only occur during a x- or y-advection and the data dependences are defined by the linear advection operators x  → x − vx Δt and y  → y − vy Δt. Figure 2 (left) shows these data dependences for the x-advection in the Cartesian (x, vx )-plane : the data on the oblique line are needed to compute the data on the vertical line. Figure 2 (right) shows which data blocks are required to compute the data blocks on a column. In this particular case, we observe that the computation of any data blocks only requires the data within two blocks : the block itself and a neighboring block at the left or at the right depending on

A Block-Based Parallel Adaptive Scheme

113

x-vx.dt x vxmax

0

vxmin xmin

x

x-vx.dt

xmax

Fig. 2. Data dependencies and blocks required for updating one column of blocks

the vx sign. It is a suitable situation where the communication volume is minimum. This situation occurs when the test case parameters satisfy a particular condition. Our algorithm works on the assumption that this condition holds. This condition can be expressed by the following equation xmax − xmin ≥ max( |vxmin | , |vxmax | )Δt . 2j0 3.4

(4)

Communication Overlapping

Conceptually, in the application of partitioning, each partition is assigned to a processor. In this section, we consider a lower level of abstraction and discuss how data have to be effectively distributed across processors in order to reach the best performance. To allow easy access to boundary data on neighboring processors, the data structure is extended to include ghost cells on each processor. Ghost cells are remote data blocks replicated in local memory to reduce access time (see [10]). The data structure have allocated memory for some boundary blocks that can be filled in with data from the adjoining processors. This is illustrated in figure 3. In the following, the region assigned to a processor that is formed from the partition extended with replicated blocks will be refered to as local region. Each advection in our numerical scheme then consists in an “update operation” which computes data within the local region and fills in some of replicated blocks with updated data.

p1

p0

p2 y

x

Fig. 3. Extension of global ordinary data structure to local regions, i.e., data structures with replicated blocks (for 3 processors)

114

O. Hoenen and E. Violard

Several algorithms for this update operation can be investigated. Their efficiency mostly depends on the characteristics of the computer system and on the size or dimensionality of the problem (see [11] for more details). Our implementation is described in algorithm 1. This algorithm tends to maximize the overlapping of communication and computation. Algorithm 1. Update of one local region Input: A, one local region Output: B, the updated region begin init recv of replicated blocks of B from adjoining processors foreach border blocks of B do compute it from blocks of A init its send to the neighboring processor end compute inner blocks of B (from inner and border blocks of A) wait all send/recv end

We use two data structures we call A and B. Data structure A stores the data before the operation and B stores the resulting data. Since our numerical scheme consists in a succession of advections, the output data is used as the input by the next operation. The role of each structure changes each time the operation is performed. This change of role is achieved by pointer assignments. In addition, the local region is virtually split into three disjunctive parts : the replicated blocks, the border blocks and inner blocks. These parts are defined by the direction (x, y or v = (vx , vy )) of the next advection. They are shown on figure 4. The algorithm works on the assumption that all required data of A are initially available. In particular, it means that the replicated blocks have been filled in with data at the first time step. The algorithm first initiates the non-blocking receive of all these blocks in B. Then, it computes each border block of B and initiates its send as soon as it is computed. Communications are overlapped with computation of inner blocks and next border blocks of B.

replicated block border block inner block

(a)

(b)

(c)

Fig. 4. The local region p2 in Fig.3 split into three kinds of blocks: (a) when the next update is a x-advection, (b) when the next update is a y-advection, (c) when the next update is a v-advection (no replicated or border block)

A Block-Based Parallel Adaptive Scheme

4

115

Implementation and Performance Measurements

Our code has been written in C and use the portable implementation MPICH [12]. In order to reduce memory copies, we defined two MPI data-types that describe the storage of data within the cells which are received or sent. A difficulty here is that data blocks may have two different memory mapping because a data block describes one single or more cells, as said in section 3.1. Because of the adaptive mesh, it is not possible to predict which kind of blocks will be sent or received. Therefore we use two MPI data-types : MPI Block1 and MPI Block2, one for each block kind. These two types are identical except that the extent of type MPI Block2 is greater than the extent of type MPI Block1 in memory. Each of these two types is a structure with the same fields and the same offset for each of these fields. We use type MPI Blocks2 to initiate a block receive. Therefore data are received in place whatever the kind the block has. Last, a call to function MPI Get count is used to determine the block type. For our experiments, we use a PC cluster’s equipped with 17 dual core processors with 2 GB of RAM each. Processors are Athlon 4800+ cadenced at 2.4 GHz and connected through an Gigabit Ethernet. Each core is seen as an independent MPI processor unit. Our test case is the uniform magnetic focusing of a semi-Gaussian beam of protons. The emittance of this proton beam is 2.36×10−5 π m rad, its current is equal to 100 mA, and its energy is 5 M eV . The initialization parameters are computed by solving the envelope equation of the equivalent KV beam (see [13]). These physical parameters give a tune depression of 0.7. The length of the period is equal to S = 26.6292 m. We perform 75 time steps (with Δt = 0.0841 s). The particles beam is given by a semi-Gaussian distribution function  v 2 +v 2 1 (− x 2 y ) if x2 + y 2 < 6 2e 8π f0 (x, v) = . (5) 0 else where x and v live in [−6, 6]2 . The number of points is equal to 128 in each direction of position space and 64 in each direction of velocity space. We use j0 = 3 and J = 5. Figure 5 shows projection of the distribution function. Impact of load balancing. We compare two partitioning of the mesh. The first one uses our heuristics to build balanced areas with a bounding box set

Fig. 5. Representation of the x − y projection (left) and x − vx projection (right) of the distribution function at time step 20

116

O. Hoenen and E. Violard

to the square [−3.5, 3.5]2. The second one splits the domain into equally sized areas such that the difference between width and height is minimum. Figure 6 (right) shows the wall-clock time of the code in each of these two cases. Since the x-projection of the distribution function is round in shape and forms a disk of center (0, 0), the second partitioning gives well-balanced areas for 4 processors. Therefore our heuristics cannot enhance performance in that case. Otherwise simulations are between 15% and 30% faster with our heuristics. 02:00

32 equi part box part

linear balanced imbalanced

90.8%

01:45

01:30

speedup

time hh:mm

01:15

01:00

16 93.1%

00:45

47%

00:30 58.5% 8

00:15

95.9% 79%

00:00

4 4

8

16 nb procs

32

4

8

16

32

nb procs

Fig. 6. Performances of our code on a PC cluster

Performance and speedup. We observe that the performance improves until 32 processors. But the speedup is under-linear due to load imbalance arising from the beam evolution in time. Figure 6 (left) shows the speedup and efficiency obtained when the workload is well-balanced throughout the execution (setting the compression threshold  = 0) in comparison with an unbalanced execution. Note that the version of the code is not fully optimized, particularly the interpolation operation cost can be reduce. We observe a slight loss of efficiency. It is not due to communications because they are totally overlapped by computation: we checked that in this experiment all messages are received before the wait barrier is reached. Thus, it can be explained by the computation of field E that is sequential.

5

Conclusion

In this paper, we proposed a new adaptive numerical scheme and its parallel implementation for solving the 4D Vlasov equation. On the contrary to the previous method [14], each part of the predicted mesh can be computed independently of each other, at each time step. Thus, our parallel implementation based on mesh partitioning induces low overhead. Moreover, we made some assumptions on the physical parameters in order to ensure a regular communication pattern.

A Block-Based Parallel Adaptive Scheme

117

Experiments on a PC cluster shown a good scalability provided the computational workload is well balanced throughout the whole simulation. They shown that a static partitioning alone is not enough to obtain the good scalability and performance even in the advantageous case of the focusing beam. Future works then consist in integrating to the code a mechanism that updates partitioning in order to follow the evolution of the physics. Then we intend to compare the scalability of our solver for distributed memory machines with the codes which target shared memory machines.

References 1. Gutnic, M., Haefele, M., Latu, G.: A parallel Vlasov solver using a wavelet based adaptive mesh refinement. In: 7th Workshop on High Perf. Scientific and Engineering Computing (ICPP 2005), pp. 181–188 (2005) 2. Sonnendr¨ ucker, E., Filbet, F., Friedman, A., Oudet, E., Vay, J.L.: Vlasov simulation of beams with a moving grid. Comput. Phys. Commun. 164, 390 (2004) 3. Hoenen, O., Mehrenberger, M., Violard, E.: Parallelization of an adaptive Vlasov solver. In: Kranzlm¨ uller, D., Kacsuk, P., Dongarra, J. (eds.) EuroPVM/MPI 2004. LNCS, vol. 3241, pp. 430–435. Springer, Heidelberg (2004) 4. Campos-Pinto, M., Mehrenberger, M.: Adaptive numerical resolution of the Vlasov equation. In: Numerical Methods for Hyperbolic and Kinetic Problems, CEMRACS, pp. 43–58 (2004) 5. Cheng, C.Z., Knorr, G.: The integration of the Vlasov equation in configuration space. J. of Comput. Phys. 22, 330–351 (1976) 6. Sonnendr¨ ucker, E., et al.: The semi-Lagrangian method for the numerical resolution of the Vlasov equation. J. Comput. Phys. 149, 201–220 (1999) 7. Hoenen, O., Violard, E.: An efficient data structure for an adaptive Vlasov solver. Technical Report 06-02, LSIIT–ICPS (February 2006) 8. Boukerche, A., Tropper, C.: A static partitioning and mapping algorithm for conservative parallel simulations. In: 8th workshop on Parallel and Distributed Simulation (PADS 1994), pp. 164–172 (1994) 9. Berger, M.J., Bokhari, S.H.: A partitioning strategy for nonuniform problems on multiprocessors. IEEE Transactions on Computers 36(5), 570–580 (1987) 10. Ding, C., He, Y.: A ghost cell expansion method for reducing communications in solving PDE problems. In: Supercomputing 2001 (CDROM) (2001) 11. Palmer, B., Nieplocha, J.: Efficient algorithms for ghost cell updates on two classes of mpp architectures. In: Parallel and Distributed Computing and Systems (PDCS 2002), pp. 192–197 (2002) 12. Gropp, W., et al.: A high-performance, portable implementation of the MPI message passing interface standard. Parallel Computing 22(6), 789–828 (1996) 13. Filbet, F., Sonnendr¨ ucker, E.: Modeling and numerical simulation of space charge dominated beams in the paraxial approximation. Technical Report RR-5547, INRIA Lorraine (2005) 14. Mehrenberger, M., et al.: A parallel adaptive Vlasov solver based on hierarical finite element interpolation. Nucl. Inst. and Meth. in Phys. Res. 558(A), 188–191 (2006)

On Optimal Strategies of Russia’s Behavior on the International Market for Emissions Permits Alexey Kadiyev, Vyacheslav Maksimov, and Valeriy Rozenberg Institute of Mathematics and Mechanics, Ural Branch of RAS, 16 S.Kovalevskoi Str., 620219 Ekaterinburg, Russia {kadiev,maksimov,rozen}@imm.uran.ru http://www.imm.uran.ru

Abstract. A simplified dynamical model of the international market for greenhouse gases emissions permits is considered. A procedure for constructing optimal strategies of Russia’s behavior is suggested. A possibility of obtaining algorithm’s input data from integrated assessment models is discussed. A specific numerical analysis is performed.

1

Introduction

The global climate change is one of the most important problems in the modern world. The driving forces of this process are not completely studied yet, and its ecological, social, and economic consequences are rather disputable and complicated from the analytical viewpoint. However, the experts are in agreement that the dramatic climate change observed in the last century is explained to some extent by the increase of the atmospheric concentration of greenhouse gases (GHG), first of all, CO2 , due to man’s impact that is characterized by the essential increase of fossil fuel consumption. One of the efforts of the international community to control environmental impacts is the Kyoto Protocol developed by the United Nations Framework Convention on Climate Change (UNFCCC) and passed in December 1997. As is known, after the waivers of ratification of the Kyoto Protocol by the main countries producing GHG emissions, USA and China, the future of the Protocol directly depended on the position of Russia. However, even after the President of the Russian Federation signed the Federal Law “On the ratification of the Kyoto Protocol...” on 4 November 2004, the debate in Russia about future costs and benefits of being a Party to the Kyoto Protocol has continued with hardly mitigated intensity since many statements of the Protocol and mechanisms of its application have an ambiguous value in the context of developing economy of Russia. In the discussion, arguments of proponents and opponents of Russia’s participation in the Kyoto Protocol are 

The work was supported by the International Institute for Applied Systems Analysis (Laxenburg, Austria), by the Russian Foundation for Basic Research (project no. 06-01-00359), and by the Programs of Basic Research of the Russian Academy of Sciences no. 16 “Environment and climate change: natural catastrophes” and no. 22 “Control processes”.

R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 118–126, 2008. c Springer-Verlag Berlin Heidelberg 2008 

On Optimal Strategies of Russia’s Behavior on the International Market

119

often based on results of application of different mathematical models, mainly, of two types: (i) integrated models for evaluating regional and global effects of GHG reduction policies; (ii) optimization models. In the present paper, a model-oriented approach to constructing optimal strategies of Russia’s behavior on the international market for emissions permits is applied. This market is one of the Kyoto flexible mechanisms. It is regulated by the Marrakech Accords (passed in 2001) and was intensively discussed at the first and second Meetings of the Parties to the Kyoto Protocol (November 2005, Montreal, Canada; November 2006, Nairobi, Kenya). A simple model is characterized by Russia’s monopoly on the trade with the Annex B countries and by an opportunity of banking permits and optimizing their sale over time. Note that, due to the collapse of the industrial sectors in 1990s, Russia actually does not need to reduce emissions for selling permits, since the amount of Russian so called “hot air” is large enough. Therefore, Russia’s monopoly is considered as the first approximation to a more complicated multi-pole market. The model uses a demand function describing the market price for emissions permits in the Annex B countries, a cost function for emissions abatement in Russia, and a temporal dynamics of “hot air”. To obtain specific dependencies, different integrated assessment models are applied.

2

The Simple Model of Dynamics of the Stock of Permits

To describe the process of emissions permits banking with an opportunity at every time moment to sell some amount on the market and/or to increase the stock by emissions abatement, we use a dynamical controlled system. Let x(t) be the stock of permits that are banked at time t; h(t) be the “hot air” available for sale (this function is actually regulated by the Protocol and assumed to be known); q(t) be the emissions abatement; u(t) be the amount of permits supplied for sale. The last two functions are control parameters. It is natural to equate the rate of the stock of permits, x(t), ˙ with the difference of two values, h(t)+q(t) and u(t). This results in the following differential equation: x(t) ˙ = h(t) + q(t) − u(t),

t ∈ [t0 , T ].

(1)

We assume that the initial time is a moment t0 when the stock of permits equals zero, i.e., x(t0 ) = 0. (2) It is evident that the “hot air” h(t), the emissions abatement q(t), and the amount of permits u(t) supplied for sale can not exceed some definite values; this can be expressed as the following constraints: a1 (t) ≤ q(t) ≤ b1 (t),

a2 (t) ≤ u(t) ≤ b2 (t),

a1 (t) ≥ 0,

a2 (t) ≥ 0, x(t) ≥ 0.

a3 (t) ≤ h(t) ≤ b3 (t),

(3)

a3 (t) ≥ 0, (4)

120

A. Kadiyev, V. Maksimov, and V. Rozenberg

Note that constraints (3) provide fulfillment of the inequality x(t) ≤ K, where K is a constant, which can be explicitly written. Therefore, condition (4) is naturally replaced by the condition 0 ≤ x(t) ≤ K.

(5)

We assume that all the scalar functions from the right-hand part of (1) belong to the space L2 ([t0 , T ]; R), functions ai (·), bi (·), i = 1, 2, 3, are continuous. In what follows, we consider any function h(·) as a known one. Functions q(·) and u(·) satisfying relations (3) and providing fulfillment of inequality (5) are called admissible controls. The set of all admissible controls is denoted by the symbol U∗ . A solution of equation (1) is a Caratheodory solution and belongs to the space of absolutely continuous functions A([t0 , T ]; R). A solution corresponding to a pair of admissible controls (q(·), u(·)) ∈ U∗ is denoted by x(·; q(·), u(·)).

3

Statement of Optimal Control Problems

Let us formulate a problem of optimal control for system (1)–(3), (5). Problem P1. It is required to find functions q∗ (·) and u∗ (·) solving the extremal problem max F (u, q), (6) u,q

T β(τ )[P (u(τ ))u(τ ) − C(q(τ ))q(τ )] dτ + β(T )π(T )x(T ),

F (u, q) =

(7)

t0

T (h(τ ) + q(τ ) − u(τ )) dτ,

x(T ) =

(8)

t0

and satisfying constraints (3) and (5). Here β(t) is the discount rate; P (u(t)) is the price of permits, which, as a rule, is inversely proportional to the amount of permits on the market; C(q(t)) is the cost function for marginal abatement, which, as a rule, is directly proportional to the level of abatement (it is determined by a so called regional MAC curve); π(t) is the expected price of permits. Thus, the integral term characterizes the total income from operations on the market minus abatement costs, whereas the terminal one represents the cost of all emissions permits banked till the moment T ; both with discounting. Optimization problems of a similar type have been investigated by many authors, see, for example, [1, 2]. Note that in the model described above the idea of possible banking of permits that can be profitable, firstly, due to the growth of demand and, consequently, of the market price for emissions permits in the future and, secondly, for decreasing own abatement costs (in a remote perspective) is realized.

On Optimal Strategies of Russia’s Behavior on the International Market

121

We assume that the functions P (·) and C(·) are such that the functional F (u, q) is strongly convex with respect to u and q. Since equality (8) is valid, functional (7) can be rewritten in the form: T [(β(τ )P (u(τ )) − β(T )π(T ))u(τ ) − (β(τ )C(q(τ )) − β(T )π(T ))q(τ )+

F (u, q) = t0

+ β(T )π(T )h(τ )] dτ.

(9)

Let us formulate an auxiliary problem of optimal control. Problem P2. It is required to find functions qα (·) and uα (·) solving the extremal problem max Fα (u, q), (10) u,q

T [(β(τ )P (u(τ )) − β(T )π(T ))u(τ ) − (β(τ )C(q(τ )) − β(T )π(T ))q(τ )+

Fα (u, q) = t0

+ β(T )π(T )h(τ ) − αx2 (τ )] dτ,

(11)

τ (h(ξ) + q(ξ) − u(ξ)) dξ,

x(τ ) = t0

and satisfying constraints (3) and (5). Here α > 0 is a small parameter. Note that, in virtue of strong convexity of functionals (9) and (11) with respect to u and q, and convexity, boundedness, and closedness (in L2 ([t0 , T ]; R) × L2 ([t0 , T ]; R) ) of the set U∗ , Problems P1, P2 have unique solutions, (q∗ (·), u∗ (·)) ∈ U∗ and (qα (·), uα (·)) ∈ U∗ , respectively. Theorem 1. Let for any α > 0 functions qα (·) and uα (·) solve Problem P2. Then, for the functional sequence (qα (·), uα (·)), the following convergence is valid as α → 0: (qα (·), uα (·)) → (q∗ (·), u∗ (·)) weakly in L2 ([t0 , T ]; R) × L2 ([t0 , T ]; R),

(12)

where (q∗ (·), u∗ (·)) is the unique solution of Problem P1. Taking into account convergence (12), we solve auxiliary Problem P2 instead of Problem P1. The former, being a problem of optimal control under phase constraints, needs special solving methods. The algorithm used in the present work is described in [3]. It consists in reduction of solving the problem with phase constraints to solving a sequence of classical optimal control problems, for example, by means of Pontryagin’s maximum principle [4].

122

4

A. Kadiyev, V. Maksimov, and V. Rozenberg

Results of Numerical Modeling

In the numerical experiments, system (1) was considered on the time interval [2010, 2030], under the assumption that the Kyoto mechanisms are applicable on the whole interval (a so called “Kyoto Forever” scenario), i.e., under the assumption that the emissions levels (regulated by the Protocol) for the Annex B countries and the international market for emissions permits are preserved. The unit of the stock of permits x(t) was megaton of carbon equivalent (1 MtC); respectively, the controls u(t) and q(t) as well as the function h(t) were measured in MtC per year. The dynamics of carbon dioxide CO2 was studied. All prices were given in USD. As a forecast of the dependence of the market price for emissions permits on the amount of permits supplied for sale, under the conditions of Russia’s monopoly (actually, as an estimate of the demand for permits in the Annex B countries), the demand function P (u(·)) from model GEMINI-E3 was chosen, see Fig. 1a. This model is a general equilibrium model of the world economy [5]. The cost function C(q(·)) for the marginal abatement depending on the level of abatement (a so called regional MAC curve) was taken from the same model, see Fig. 1b. The linear interpolation was used between the pictured curves.

Fig. 1. Input data: (a) law of demand; (b) MAC curve of Russia. Functions for 2010, 2020, and 2030 are presented.

The constraints on the controls u(t) and q(t) were chosen as constants: a1 (t) = a2 (t) = 0, b1 (t) = b2 (t) = 250. We considered the process without discounting (β = 1), the parameter α (see (11)) was equal to 0.01, the expected price of permits at the terminal time π(T ) was equal to 0. The main goal of the experiment was studying the dependence of the optimal dynamics of control parameters u(t) and q(t), the stock of permits x(t), and the income obtained by Russia from operations on the market for permits on the amount of “hot air”, additionally (to the abatement) available for sale, i.e., on the function h(t). Actually, the value of the function h(t) is the difference between emissions at the moment t and the known emissions level of 1990 (the Kyoto level for Russia, 646 MtC). Then, using different scenarios of the economic

On Optimal Strategies of Russia’s Behavior on the International Market

123

Table 1. Estimates for the temporal dynamics of Russian “hot air”, MtC per year time 2010 2015 2020 2025 2030

(1) 155 114 69 33 -1

(2) 169 110 52 -13 -85

(3) 155 125 93 60 25

(4) 132 78 19 -19 -59

(5) 85 67 57 28 -3

(6) 186 105 41 16 6

(7) 300 245 199 163 136

Remarks. 1. Variants (1)–(7) correspond to the following forecasts: (1) — the reference scenario of the International Energy Outlook 2006 [6]; (2) — the forecast of the Energy Research Institute of RAS [7]; (3) — the reference scenario of the Fourth National Communication of the RF [8]; (4) — the innovation-active scenario of the Fourth National Communication of the RF [8]; (5) — the simulation results by MERGE [9]; (6) — the simulation results by EPPA [10]; (7) — the simulation results by GEMINI-E3 [5]. 2. Between the points listed in Table 1, the linear interpolation was used. 3. Negative values were replaced by zeros.

development of Russia and applying different models forecasting the dynamics of CO2 emissions, we obtained several scenarios of the dynamics of h(t), see Table 1. In addition to variants (1)–(7), two “extremal” cases with constant functions h(t) were computed: variant (0), where h(t) = 0, and variant (8), where h(t) = 163. The temporal dynamics of the main output parameters was presented in Figs. 2–3. Note that, due to the method’s error, there is a sense to limit the analysis by 2028. It is evident that the income from permits sale (see extremal problem (10)– (11)) should take a minimal (comparing with other variants) value in variant (0);

Fig. 2. The temporal dynamics of (a) the amount of permits supplied for sale, u(t); (b) the emissions abatement, q(t). Variants (0)–(8).

124

A. Kadiyev, V. Maksimov, and V. Rozenberg

this fact is confirmed by simulations. It turns out that variant (8) provides a maximum possible income over different functions h(t) (in the case when the remaining parameters of the problem are fixed); the same result is obtained in variant (7). The maximality of the income in these variants is explained by the zero optimal value of q(t) (see Fig. 2b). Note that the least integer providing the maximum above was taken as the constant value of h(t) in variant (8). As is seen in Fig. 2a, the amount of permits supplied by Russia for sale on the international market is varied in 2010 from 73 MtC up to 127 MtC, in 2020 from 119 MtC up to 167 MtC, in 2028 from 142 MtC up to 187 MtC (in variant (0) and in variants (7), (8), respectively). In all the variants, the amount of permits supplied for sale increases with time, and the growth rate is approximately the same (varies from 2.0% up to 3.4% per year). On the contrary, the emissions abatement is rather stable with time in all the variants (see Fig. 2b; the most considerable growth is observed in variant (0)). The share of emissions reduction in the amount of permits supplied for sale is changed from evident 0% in variants (7), (8) up to 94% in 2010, 65% in 2020, and 59% in 2028 in variant (4) (with the exception of variant (0), when this index is not informative). In all the variants, it is inexpedient to use the disposable “hot air” at a time; the maximal banking of permits (the abatement is also taken into account) with the purpose of future income increase is observed in variant (6): 160 MtC in 2010 (with the exception of variants (7), (8), when this index is not informative). Note that the banking becomes possible due to the intertemporal optimization (on the whole time interval). As to the market price of permits (see Fig. 3a), it rises from 116 USD/MtC in 2010 up to 229 USD/MtC in 2028 in variants (7), (8) (the minimal prices) and from 171 USD/MtC in 2010 up to 287 USD/MtC in 2028 in variant (0) (the maximal prices), in all the variants rather slowly (from 2.9% up to 3.9% per year) increasing with time. For the comparative analysis of modeling results in variants (0)–(8), the histogram (Fig. 3b), where the maximum possible income (for the whole time

Fig. 3. Modeling results: (a) the temporal dynamics of the price of permits; (b) the Russian income from permits sale (in % from the maximum possible income in the model). Variants (0)–(8).

On Optimal Strategies of Russia’s Behavior on the International Market

125

interval) is taken as 100%, is constructed. Analyzing the histogram, we conclude that the maximal income loss over forecasts (1)–(7) is 10.6% (in variant (4)), whereas the maximum possible loss is 20.5%. The average (over variants (1)– (7)) loss is rather small (6.2%). Hence, we can deduce that domestic resources of Russia (namely, an opportunity of relatively cheap (especially comparing with countries of European Union and Japan) emissions reduction in Russia due to incomplete realization of the energy effectiveness and energy saving potential) provide a considerable income from permits sale even in the case of unfavorable situation with “hot air”. It turns out that the dependence of the value of this income on the function h(t) is not so essential, as one can suppose, when analyzing mathematical model (1)–(11).

5

Concluding Remarks

It should be noted that there is a high level of uncertainty in the specification of parameters of the model in question. In the paper, several scenarios forecasting the temporal dynamics of “hot air” were studied. It is reasonable to consider the analysis of the dependence of optimal strategies of Russia’s behavior on the international market for permits on variation of different model parameters (in particular, the functions presented in Fig. 1) as one of the basic perspective directions in modeling.

References 1. Grimm, B., Pickl, S., Reed, A.: Management and optimization of environmental data within emissions trading markets - VEREGISTER and TEMPI. In: Antes, R., Hansj¨ urgens, B., Letmathe, P. (eds.) Emissions Trading and Business, pp. 165–176. Physica, Heidelberg (2006) 2. Bernard, A., Haurie, A., Vielle, M., Viguier, L.: A two-level dynamic game of carbon emissions trading between Russia, China, and Annex B countries. Swiss National Centre of Competence, NCCR-WP4 Working paper 11, University of Geneva (September 2002) 3. Kryazhimskii, A., Ruszczy´ nski, A.: Constraint aggregation in infinite-dimensional spaces and applications. IIASA Interim Report 97-051, Laxenburg, Austria (1997) 4. Pontryagin, L.S., Boltyanskii, V.G., Gamkrelidze, R.V., Mishchenko, E.F.: Mathematical theory of optimal processes. Moscow, Nauka (1969) (in Russian) 5. Bernard, A., Reilly, J., Vielle, M., Viguier, L.: Russia’s role in the Kyoto Protocol. In: Proceedings of the Annual Meeting of the International Energy Workshop jointly organized by EMF/IEA/IIASA, Stanford University, USA, June 18-20 (2002) 6. International Energy Outlook 2006. Energy Information Administration, www.eia.doe.gov/oiaf/ieo/index.html 7. Makarov, A., Likhachev, V.: Mitigation Measures and Policies from Russian Perspective. In: Workshop on Post-2012 Climate policies, IIASA, Laxenburg, Austria, June 13 (Preprint, 2006) 8. The Fourth National Communication of the Russian Federation. Israel, Y.A., Nakhutin, A.I., Semenov, S.M., et al. (eds.), ANO Meteoagentstvo Rosgidrometa, Moscow (2006)

126

A. Kadiyev, V. Maksimov, and V. Rozenberg

9. Manne, A., Mendelson, R., Richels, R.: MERGE - a Model for Evaluating Regional and Global Effects of GHG reduction policies. Energy Policy 23(1), 17–34 (1995) 10. Babiker, M.H., Reilly, J.M., Mayer, M., Eckaus, R.S., Sue Wing, I., Hyman, R.C.: The MIT Emissions Prediction and Policy Analysis (EPPA) Model: revisions, sensitivities, and comparisons of results. MIT Joint Program on the Science and Policy of Global Change, report 71, Cambridge, MA, USA (2000)

Message-Passing Two Steps Least Square Algorithms for Simultaneous Equations Models Jose-Juan L´opez-Esp´ın1 and Domingo Gim´enez2 1

2

Departamento de Estad´ıstica, Matem´ aticas e Inform´ atica Universidad Miguel Hern´ andez, 03202, Elche, Spain [email protected] Universidad de Murcia, Departamento de Inform´ atica y Sistemas 30071 Murcia, Spain [email protected]

Abstract. The solution of Simultaneous Equations Models in high performance systems is analyzed. Message-passing algorithms for the Twostage Least Squares method are developed. Algorithms are studied theoretically and experimentally. The algorithms make extensive use of basic libraries like BLAS, LAPACK, and ScaLAPACK to obtain efficient and portable versions.

1

Introduction

The solution of Simultaneous Equations Models in high performance systems is studied. These models can be solved through a variety of methods. In [5], a study of Indirect Least Square (ILS) and Two Step Least Square (2SLS) was presented. The cost of using ILS is lower than using 2SLS but ILS can only be used in exactly identified equations. 2SLS can be used in exactly identified equations and in over identified equations, and for this reason 2SLS must be improved. In this paper three different versions of the 2SLS algorithm are presented. The first is a basic algorithm which will be improved in the second and the third versions. In the first version, the structure of the parallel 2SLS algorithm is stated. In the other versions, the same structure is followed but matrix decompositions are used to obtain lower costs. A number of variables will be used to analyze the algorithms. The system will be in structural form, and each equation will have one endogenous variable (called main endogenous) as a function of the other variables [3]. The variables which appear in the algorithms are: – The number of data per equation, d. We suppose d has the same value for each equation. – The number of endogenous variables in the system, N . We suppose it coincides with the number of equations in the systems. Each endogenous variable is the main one in one equation. – The number of predetermined variables in the system, K. R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 127–136, 2008. c Springer-Verlag Berlin Heidelberg 2008 

128

J.-J. L´ opez-Esp´ın and D. Gim´enez

– The number of endogenous variables in equation i, ni . – The number of predetermined variables in equation i, ki . – X is the predetermined matrix. It has d × K dimension and contains the data of predetermined variables. – Y is the endogenous matrix. It has d × N dimension and contains the data of endogenous variables. There are three types of equations: non identified (ni − 1 > K − ki ), with no solution; exactly identified (ni − 1 = K − ki ), for which it is possible to use ILS or 2SLS; and over identified (ni − 1 < K − ki ), where it is necessary to use 2SLS. In an equation, the problem of the correlation between random and endogenous variables is avoided by substituting the original variable for a new variable called proxy which has been calculated previously.

2

Parallel 2SLS Algorithms

Parallel algorithms of 2SLS techniques for distributed memory have been developed. Parallelization can be made at different levels: in the basic matrix operations using PBLAS and ScaLAPACK, by dividing the work in the loops among the processors in the system, etc. The parallelization has been made each time at the highest possible level. It is assumed that X and Y are distributed between all processors in ScaLAPACK style. When the algorithm finishes, the solution is also distributed. It is also assumed that all the processors have the complete structure of the system at the beginning of the algorithm, and it is not necessary to receive data from other processors. In the theoretical study, the typical model of the communications is used [4]. It is assumed that ts is the start-up time in point-to-point communications and tsb in broadcast, and tw and twb are the word-sending time in a point-to-point and a broadcast communication. In some parts of the algorithms, a matrix is distributed in all the processors in ScaLAPACK style, and it is necessary to send the complete matrix to all processors. MPI library can be used here (MPI Alltoall routine). The cost of this routine is assumed as TA2A (T, p) = (p−1)ts +T tw , where T is the total data to send and p is the number of processors. ts and tw depend on the hardware system. OLS technique is used in the algorithms studied in this work. The computation in OLS consists of (X t X)−1 X t Y . The resulting matrix is the estimator of the ˆ in the equation: coefficients β (β) Yt = β1 X1,t + . . . + βn Xn,t + ut

(1)

and Yˆ = X βˆ is calculated to obtain an estimation of Y .The cost of OLS is: TOLS (N, d, K, δest ) =

2 3 K + 2K 2 (d + N ) + 2N Kd + δest 2N Kd 3

where δest is 1 if Yˆ is requested in the algorithm, and 0 in the other case.

(2)

Message-Passing Two Steps Least Square Algorithms

129

When parallel multiplications and inverse are used in (X t X)−1 X t Y , the cost of OLS (TOLSp (N, d, K, δest )) is the cost of the sequential case divided by the number of processors, plus a communication cost of lower order. 2.1

Parallel Two-Stage Least Squares (First Version)

Since the solution of a system by 2SLS requires the use of OLS in each equation with proxys, predetermined and endogenous variables, it is necessary to send all the proxys and part of the system structure to each processor. Algorithm 1 shows the scheme of 2SLS when solving a complete system. Matrix Xe is formed by the endogenous variables (which now take the values of the proxys) of the equation, excluding the main endogenous and the predetermined variables. Thus, matrix Xe has size d × (ni − 1 + ki ). initially, in 2SLS all the proxys are calculated. It could be that several of the N proxys will not be used, but since they can not be identified and the matrices are distributed at the beginning of the algorithm, it is better to calculate all of them. The cost is TOLSp (N, d, K, δest = 1). When OLS ends, Yˆ is distributed in all the processors in ScaLAPACK style and it is necessary to send the complete matrix to all the processors. The cost of sending Yˆ is TA2A (dN, p). Algorithm 1. Scheme of the parallel 2SLS algorithm 1: 2: 3: 4: 5: 6: 7: 8:

Yˆ = X(X t X)−1 X t Y Distribute Yˆ to all the processors IN PARALLEL Each processor q DO for j=1... Np do i = q + (j − 1)p Compute X(Xet Xe )−1 Xet yi end for END PARALLEL

To simplify the problem it is assumed that the structure of all the equations is similar, and the cost of solving the first Np equations is similar to that of solving any other Np equations. Thus the cost is: T2SLSp,1ver (N, d, K) = TOLSp (N, d, K, δest = 1) + TA2A (dN, p)+ N  p    + maxq=1...p TOLS (1, d, kq+(j−1)p + nq+(j−1)p − 1, δest = 0) ≈ j=1 N

≈ TOLSp (N, d, K, δest = 1) + TA2A (dN, p) +

p 

(TOLS (1, d, ki + ni − 1, δest = 0))

i=1

(3) 2.2

Second Version of 2SLS (QR Decomposition)

An algorithm using the Householder decomposition is studied. QR decomposition has been used before in other econometric techniques with good results

130

J.-J. L´ opez-Esp´ın and D. Gim´enez

[1,2,7]. The exogenous matrix X is decomposed as QR and the new matrices are used. Given the exogenous matrix X (with d × K dimension), there is an orthogonal matrix Q (with d × d dimension) and atriangular matrix R (with  R1 K ×K d × K dimension), with X = QR. R has the form , where 0 (d − K) × K R1 is an upper triangular matrix. Partitioning Q in (Q1 |Q2 ), where Q1 is d × K, we obtain:   R1 X = QR = (Q1 |Q2 ) = Q1 R1 (4) 0 An expression for Yˆ is: Yˆ = XΠ = QRR1−1 Qt1 Y = Q



Id 0

 Qt1 Y = Q1 Qt1 Y

(5)

Using this expression it is easy to prove: Yˆ t Yˆ = Y t Q1 Qt1 Q1 Qt1 Y = Y t Q1 Qt1 Y = Y t Yˆ

(6)

Because R1 is an upper triangular matrix, it is less costly to calculate its inverse. LAPACK can be used here and also in the QR decomposition. Routines used in the experiments in section 3 are: dtrtri for the inverse; dgeqrf for QR decomposition; and dorgqr to calculate Q. The cost of dgeqrf is 23 K 2 (3d − K) and the cost of dorgqr is 23 K 2 (3d− K). The three versions of the 2SLS algorithm have been compared using only one processor (table 3). A parallel version can be developed by substituting the LAPACK functions (dgeqrf , dorgqr and dtrtri) for their equivalent ones in ScaLAPACK (pdgeqrf , pdorgqr and pdtrtri). The scheme of a new OLS using QR decomposition is shown in algorithm 2. Algorithm 2. Scheme of OLS using QR decomposition (OLS2ver ) 1: 2: 3: 4: 5: 6:

Obtain Q1 and R1 {cost → 43 K 2 (3d − K)} Compute R1−1 {cost → 13 K 3 } coef = R1−1 Qt1 yi {cost → 2K(K + d)} if estimation = TRUE then est = Q1 Qt1 yi {cost → 2dK} end if

Algorithm 2 is used by each processor in each equation solved by 2SLS. When the OLS is used in this case, the X matrix is formed by the proxy variables and the predetermined variables of the equation, and yi is the main endogenous variable of the i-th equation. The cost of OLS using the QR decomposition is: TOLS2ver (d, K, δest ) = 4K 2 d − K 3 + 2K(K + d) + δest 2dK

(7)

Message-Passing Two Steps Least Square Algorithms

2.3

131

Third Version of 2SLS (Inverse Decomposition)

It is possible to obtain better results by using a special decomposition of a matrix in 2SLS, according to the theorem below. Theorem 1. For symmetrical matrices A and D, if all necessary inverses exist, then  −1  −1    A B A 0 −E (8) = + (D − B t A−1 B)−1 (−E t , Id) Bt D 0 0 Id where E = A−1 B Proof. see Appendix A of [6]. Let us suppose that an equation is solved using 2LSL. Since the proxys have already been calculated, the first step is to substitute the original endogenous variables in the equation by the proxys: yj = α0 + γ1 xj1 + . . . + γk xjk + α1 yˆj1 + . . . + αm yˆjm + 

(9)

The set of proxys will be denoted by Yˆ1 and the set of predetermined variables by X1 . The matrix used in subsequent calls to OLS is [X1 Yˆ1 ] (dimension d × (m + k)). Then a call to OLS is made to calculate ([X1 Yˆ1 ]t [X1 Yˆ1 ])−1 [X1 Yˆ1 ]t yj . Using the previous theorem, the inverse of the matrix can be calculated as follows:  t  −1  X t X X t Yˆ −1  (X t X )−1 0  X1 t ˆt 1 1 X1 Y1 = ˆ 1t 1 ˆ 1t ˆ1 = + 0 0 Yˆ1t Y1 X1 Y1 Y1 

−(X1t X1 )−1 X1t Yˆ1 Id



(Yˆ1t Yˆ1 − Y1t X1 (X1t X1 )−1 X1t Yˆ1 )−1 (−Y1t X1 (X1t X1 )−1 , Id)

(10) where a lot of data can be taken from the matrix calculated at the beginning of the algorithm. The operations necessary in equation 10 are shown bellow (some of then were obtained in previous computations and others are performed at this point): – X1t X1 is taken from X t X, which was calculated at the beginning. – (X1 X1 )−1 is calculated (cost 23 k 3 ). – X1t Yˆ1 is taken from X t Y , which was calculated at the beginning because X t Yˆ = X t XΠ = X t X(X t X)−1 X t Y = X t Y . – (X1t X1 )−1 X1t Yˆ1 is calculated (cost 2k 2 m). – Yˆ1t X1 (X1t X1 )−1 X1t Yˆ1 is calculated (cost 2m2 k). – Yˆ1t Yˆ1 is taken from Yˆ t Yˆ , which was calculated at the beginning. – (Yˆ1t Yˆ1 − Y1t X1 (X1t X1 )−1 X1t Yˆ1 )−1 is calculated (cost 23 m3 ). – Multiplication of the first and the second matrices in equation 10 has cost 2m2 k, because the identity matrix is not computed.

132

J.-J. L´ opez-Esp´ın and D. Gim´enez

– Multiplication of the previous matrix and the third matrix in equation 10 has cost 2m2 k because it also includes the identity matrix. The cost of additions is not considered, and the total cost is 23 m3 + 23 k 3 + 6m2 k + 2k 2 m. Matrix [X1 Yˆ1 ]t yj must be also calculated. X1t yj can be taken from X t Y , which was calculated at the beginning. Yˆ1t yj can be taken from Yˆ t Yˆ (Yˆ t Yˆ = Yˆ t Y , equation 6). Finally, the multiplication of the two matrices is made and the total cost is TOLS3ver = 23 (m3 + k 3 ) + 2(m + k)2 + 6m2 k + 2k 2 m when only one equation is solved. This can be compared with the cost of the OLS which was used in the first version of 2SLS for an equation with m endogenous variables, k predetermined variables and sample size d, TOLS1ver = TOLS (1, d, m + k, δest = 0) = 23 (m + k)3 + 2(m + k)2 d + 2(m + k)2 + 2(m + k)d. Since TOLS3ver does not depend on d, the cost of solving the equations (without considering the matrix operations in lines 1, 2 and 3 of algorithm 4) is independent of the sample size d. It is possible to deduce that TOLS1ver > TOLS3ver . Because 23 m3 + 23 k 3 ≤ 2 2 3 3 3 2 2 2 2 3 (m+k) = 3 (m +k +3m k+3mk ), it is sufficient to see that 6m k+2k m < 2 t −1 2(m + k) d + 2(m + k)d. (X X) must be calculated and it is necessary that K ≤ d (X has d × K dimension), and 0 < m + k ≤ K ≤ d because the equation is identified. From the previous inequality, it is deduced that (m+k)3 < d(m+k)(m+k+1) and 3m2 k +k 2 m ≤ (m+k)3 . Then 3m2 k +k 2 m < (m+k)3 < d(m+k)(m+k +1) and TOLS3ver < TOLS1ver . Of course, there are several extra operations (Yˆ Yˆ in line 3 of algorithm 4 must be computed) at the beginning of the algorithm. These extra operations have a cost of 2N 2 d. Experimental results confirm that the third version is faster than the first version. On comparing expressions 7 and 11, it is possible to deduce that TOLS2ver > TOLS3ver . Because 2SLS is used in over identified equations, K + 1 > ni + ki > 0, and then 3K 3 ≥ 3(ni + ki )3 > 23 (n3i + ki3 ) + 6n3i ki + 2ni ki3 . Furthermore 4K 2 d − K 3 ≥ 3K 3 and 2K(K + d) ≥ 4K 2 > 2(ni + ki )2 , because K ≤ d (X has d × K dimension). From the previous inequalities, it is deduced that TOLS3ver < TOLS2ver . When ki > 0 and ni > 1, the equation can be solved by OLS3ver . When ki = 0 or ni = 1, which is very improbable, OLS can be used with a reduced cost because matrix Yˆ t Yˆ has been calculated before. Algorithm 3 shows the scheme of OLS using the inverse decomposition (equation 10). The cost of OLS using the inverse decomposition is: TOLS3ver (ki , ni ) = 23 (ki3 + n3i ) + 2(ni + ki )2 + 6n2i ki + 2ki2 ni

(11)

A scheme of the third version of 2SLS is shown in algorithm 4. A call to OLS3ver is made sequentially in each equation and the scheme studied above is used.

Message-Passing Two Steps Least Square Algorithms

133

Algorithm 3. Scheme of the OLS algorithm using inverse decomposition (OLS3ver ) 1: Compute (X1 X1 )−1 (X1t X1 )−1 X1t Yˆ1 Y1t X1 (X1t X1 )−1 X1t Yˆ1 (Yˆ1t Yˆ1 − Y1t X1 (X1t X1 )−1 X1t Yˆ1 )−1  t −1 X X1 X1t Yˆ1 2: Compute ˆ 1t {using equation 10} Y1 X1 Yˆ1t Yˆ1 3: Compute ([X1 Yˆ1 ]t [X1 Yˆ1 ])−1 [X1 Yˆ1 ]t yj

Algorithm 4. Scheme of the parallel 2SLS algorithm 3rd version 1: 2: 3: 4: 5: 6: 7: 8: 9:

Yˆ = X(X t X)−1 (X t Y ) {saving Π, X t X and X t Y } Yˆ t Yˆ {parallel multiplications} Distribute Π, X t X, X t Y, Yˆ , Yˆ Yˆ to all the processors IN PARALLEL Each processor q DO for j=1... Np do i = q + (j − 1)p OLS3ver (yi ,Yˆ ,X,X t X,X t Y , Yˆ Yˆ ) end for END PARALLEL

The total cost of the third version of 2SLS is: T2SLSp,3ver (N, d, K) = 2 TOLSp (N,⎧ d, K, δest = 1) + 2Np d + TA2A (K 2 + N 2 + 2KN + dN, p)+ ⎫ N ⎨ ⎬ p 2 3   3 2 2 2 + maxq=1...p (k + n ) + 2(n + k ) + 6n k + 2k n ≈ t t t t t t t t 3 ⎩ ⎭ j = 1 t = q + (j − 1)p

≈ TOLSp (N, d, K, δest = 1) +

2N 2 d p

+ TA2A (K 2 + N 2 + 2KN + dN, p)+

N p

 2 i=1

3 3 (ki

+ n3i ) + 2(ni + ki )2 + 6n2i ki + 2ki2 ni



(12)

3

Experimental Results

Experimental results have been obtained in: – Kefren: A cluster of 20 biprocessors Pentium Xeon 2 Ghz interconnected by a SCI net with a Bull 2D topology in a mesh of 4 × 5. Each node has 1 Gigabyte RAM.

134

J.-J. L´ opez-Esp´ın and D. Gim´enez

Table 1. Execution time (in seconds) of the third version of the 2SLS algorithm in Kefren, with N =1000, K=400 and varying the sample size (d) in one processor d:

500 1000 1500 2000 time % time % time % time % total time 197,43 202,78 203,35 228,08 Yˆ , Yˆ t Yˆ 2,05 1,04 3,40 1,68 4,99 2,45 6,29 2,76

Table 2. Execution time (in seconds) and speed-up of the third version of the 2SLS algorithm in Marenostrum, when varying the number of endogenous variables (N ), the sample size (d) and the number of processors N: 500 1000 1500 2000 2500 d: 500 500 1000 1000 1500 proc. time Sp time Sp time Sp time Sp time Sp 1 21,20 352,22 2014,73 7005,57 18471,71 4 6,26 3,39 91,55 3,85 528,08 3,82 1820,22 3,85 4930,59 3,75 8 3,49 6,07 48,23 7,30 274,49 7,34 944,70 7,42 2536,07 7,28 16 2,12 10,00 26,97 13,06 148,08 13,61 1425,33 4,92 1348,91 13,69 32 1,36 15,55 15,69 22,45 83,77 24,05 273,55 25,61 677,84 27,25 64 1,12 18,96 10,38 33,94 48,42 41,61 150,91 46,42 393,16 46,98

Table 3. Execution time (in seconds) and speed-up of the second and third versions of the 2SLS algorithm with respect to the first version, with one processor in Kefren, when varying the number of endogenous variables (N ), the sample size (d), and the number of processors N: d: proc. 1st ver 2rd ver 3nd ver

500 1000 1500 2000 2500 500 500 1000 1000 1500 time Sp time Sp time Sp time Sp time Sp 72,82 790,93 7031,96 19337,92 97874,21 74,32 0,98 549,07 1,44 4643,24 1,51 9676,49 2,00 29830,90 3,28 12,91 5,64 198,16 3,99 1225,33 5,74 4192,10 4,61 10217,02 9,58

– Marenostrum: A supercomputer based on PowerPC processors, BladeCenter architecture, a Linux system and a Myrinet interconnection. The main characteristics are: 10240 IBM Power PC 970MP processors at 2.3 GHz (2560 JS21 blades), 20 TB of main memory, 280 + 90 TB of disk storage and a peak Performance of 94,21 Teraflops. Marenostrum is the most powerful supercomputer in Europe and the fifth in the world, according to the latest TOP500 list. As has been shown, the theoretical cost depends on a large number of parameters. To simplify the experiments, these have been done with different numbers of endogenous variables (N ), and the number of exogenous variables has been

Message-Passing Two Steps Least Square Algorithms

135

Fig. 1. Quotient between theoretical and experimental execution times of the second version of 2SLS, varying the number of processors

taken as 40% of N . The Simultaneous Equations Systems used in the experiments have been created randomly. In all the experiments, the system to solve is in P0 , and the solutions are in P0 when the algorithm finishes. Table 1 shows the cost of the third version of the 2SLS algorithm when d is varied and the other parameters are fixed. It can be seen that the total cost depends slightly on d. The cost of calculating Yˆ and Yˆ t Yˆ (lines 1 and 2 of algorithm 4) increases with d to a high degree, and this cost and the communications influence the total cost by about 2 per cent. Furthermore, the cost of calculating Yˆ and Yˆ t Yˆ is negligible compared with the total cost. Thus, it is important to parallelize the solution of the equations (lines 5 to 8 of algorithm 4). The third version of the 2SLS algorithm is experimentally analyzed (table 2). It can be seen that the speed-up is good and it is better when N , K and d are increased. Figure 1 shows that the model of the execution time (equation 4) approaches well the experimental time. The table shows the quotient for the second 2SLS version. Table 3 shows a comparison between the first, second and third versions of the algorithm with one processor. It can be seen that the third algorithm is the best.

4

Conclusions

Algorithms for the solution of Simultaneous Equations Models with Two-stage Least Square methods have been developed for message-passing systems. Simultaneous Equations Models appear in different scientific fields, and because the size of the applications varies greatly, it is interesting to have tools for the solution of small and large systems. Routines in LAPACK and ScaLAPACK style

136

J.-J. L´ opez-Esp´ın and D. Gim´enez

have been developed. The parallel versions of the algorithms have proved to be scalable, and thus appropriate to afford solutions for huge systems, like those in the simulation of national or world economy.

Acknowledgment This work has been funded in part by the Spanish MCYT and FEDER under Grant TIC2003-08238-C02-02 and Fundaci´ on S´eneca 02973/PI/05. The author gratefully acknowledges the computer resources, technical expertise and assistance provided by the Barcelona Supercomputing Center - Centro Nacional de Supercomputaci´ on (Marenostrum) and the Networking Group of the Polytechnic University of Valencia (Kefren).

References 1. Foschi, P., Kontoghiorghes, E.J.: Estimation of VAR Models: Computational Aspects. Computational Economics 21(1), 3–22 (2003) 2. Gatu, C., Kontoghiorghes, E.J.: Parallel algorithms for computing all possible subset regression models using the QR decomposition. Parallel Computing 29(4), 505–521 (2003) 3. Greene, W.H.: Econometric Analysis, 3rd edn. Prentice-Hall, Englewood Cliffs (1998) 4. Kumar, V., Grama, A., Gupta, A., Karypis, G.: Introduction to Parallel Computing. Design and Analysis of Algorithms. The Benjamin Cumming Publishing Company (1994) 5. L´ opez-Esp´ın, J.J., Gim´enez, D.: Solution of Simultaneous Equations Models in high performance systems. In: PARA 2006 (June 2006) 6. Mardia, K.V., Kent, J.T., Bibby, J.M.: Multivariate Analysis. Academic Press, London (1979) 7. Yanev, P., Foschi, P., Kontoghiorghes, E.J.: Algorithms for Computing the QR Decomposition of a Set of Matrices with Common Columns. Algorithmica 39(1), 83–93 (2004)

Parallel Implementation of Cholesky LLT -Algorithm in FPGA-Based Processor Oleg Maslennikow1, Volodymyr Lepekha2 , Anatoli Sergiyenko2, Adam Tomas3 , and Roman Wyrzykowski3 1

Technical University of Koszalin, ul. Sniadeckich 2, 75-453 Koszalin, Poland [email protected] 2 National Technical University of Ukraine, pr.Peremogy 37, 03056 Kiev, Ukraine 3 Czestochowa University of Technology, ul. Dabrowskiego 69, 42-200 Czestochowa, Poland

Abstract. The fixed-size processor array architecture, which is intended for realization of matrix LLT -decomposition based on Cholesky algorithm, is proposed. In order to implement this architecture in modern FPGA devices, the arithmetic unit (AU) operating in the rational fraction arithmetic is designed. The AU is intended for configuring in the Xilinx Virtex4 FPGAs, and its hardware complexity is much less than the complexity of similar AUs operating with floating-point numbers.

1

Introduction

The majority of modern HPC clusters are constructed from high-end PCs, significantly reducing procurement costs. The double precision floating-point calculations became the standard method for the linear algebra problem solving. But the floating-point super-scalar microprocessor units show their low loadings due to the often pipeline stalls and small data memory bandwidth. This disadvantage is slightly compensated by cash memory volume increase and high processor clock frequency. Many classes of applications demonstrate significant performance improvements when implemented in a field programmable gate array (FPGA). Although FPGA’s clock rate rarely exceeds one tenth that of a PC, applications which exhibit massive parallelism are typically speeded-up by FPGAs. Therefore many efforts were made to introduce floating-point arithmetic units (AUs) configured in FPGA in academic institutions [1], [2], [3], [4], [5], as well as in industry [6], [7], [8], [9]. Some authors prove that FPGAs can provide the excellent peak performance of such AUs [1],[2], [8]. But it is shown [9] that FPGAs could not be prevalent in this area, because they do not provide a good performance/cost ratio for double precision floating-point calculations. To minimize the drawbacks of the floating-point FPGA implementation a set of limitations is used. The standardized underflow calculations and rounding methods are usually not implemented because they are both hardware and time R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 137–147, 2008. c Springer-Verlag Berlin Heidelberg 2008 

138

O. Maslennikow et al.

consumable. Addition, multiplication, division and square root operations are often implemented in the separate AUs, because when they are combined in a single AU, then the hardware volume is increased substantially [7]. Only the multiplication with addition is combined in a single AU due to the fact that it is a natural operation in matrix calculation [2]. FPGAs give the opportunity to configure the system architecture, which is adapted to the application field. Therefore in papers [11],[12], floating-point AUs are used, whose data bit width depends on the needed calculation precision. Moreover, it is proposed [12] to implement general calculations in AUs with small bit width, and to use the double precision AUs to derive precise results in the final iteration. Since the complexity of the floating-point multiplier and divider is proportional to the square of operand bit width, using the logarithmic number system can have some advantages [13]. Rational numbers are the real numbers, which are derived as solutions of linear equations or integer polynomial divisions. Any rational number can be represented by a fraction, which consists of an integer numerator and integer denominator [17]. Therefore, representation of data by rational fraction numbers is a natural solution for the linear algebra problem solving in FPGAs. In particular, the rational fraction number system is proposed [14] for the Toeplitz matrix problem solving. Then, this system was successfully used to implement the conjugate gradient method. Cholesky matrix factorization algorithm is widely used in many applications like modeling and least square problems, due to its comparatively low computational complexity and convergence properties. In this paper, we propose a processor array architecture and its AU operating in the rational fraction arithmetic (RFA) which are intended for the Cholesky algorithm implementation. This AU is adapted to configuring in Xilinx Virtex 4 FPGA platform, and its hardware complexity is much less than the complexity of similar AUs operating with floating-point numbers.

2

Cholesky Algorithm and Processor Array Structures for Its Implementation

LLT -decomposition is [16], [19]-[22] an efficient alternative to classical Gaussian elimination in case of solution of dense linear systems Ax = b,

(1)

when A is an N × N symmetric positively definite matrix. Cholesky algorithm is usually used for the LLT -decomposition of such matrices A in such a way that L LT = A,

(2)

where L is a lower N × N triangular matrix. This algorithm can be represented by the following construction:

Parallel Implementation of Cholesky LLT -Algorithm

139

for i = 1 to N do begin l(i, i)= SQRT(a(i, i)); for j = i+1 to N do begin l(j, i)= a(j, i)/l(i, i); for k = i+1 to j do a(j, k) = a(j, k) - a(j,i)*a(k, i); end end Note that the input matrix A = A1 = aij (where i = 1, ..., N ; j = 1, ..., i) is recursively modified during N computation steps in order to obtain the lower triangular matrix L = AN+1 . Moreover, in the i-th algorithm step (i = 1, ..., N ) the elements of the i-th column of the matrix L are computed, as well as the modification of the k-th column of the matrix Ai is carried out (k = i + 1, i + 2, ..., N ). The main advantage of Cholesky algorithm is its low computational complexity (for example, two times lower than Gauss elimination [16]). Moreover, this algorithm is numerically stable without any pivoting procedures and is more suitable for the parallel implementation. The basic dependence graph GB of the Cholesky algorithm is a threedimensional one, and is shown in the left part of Fig.1 for N = 4. It has been constructed in accordance with the method for deriving dependence graph of regular algorithm proposed in [18]. Nodes of GB are distributed in nodes of the threedimensional lattice Q = {K = (i, j, k) : 1 ≤ i ≤ N, (i+1) ≤ j ≤ N, k ≤ j} which form a pyramid. The height of the lattice is N units (or layers), where the i-th layer (i = 1, 2, ..., N ) corresponds to computation of elements lji , where j = i + 1, ..., N , and to transformation of others columns of the matrix A, from (i + 1)-th to N -th. The right part of Fig.1 represents an example of the processor array structure S1, obtained after space-time mapping of the graph GB using the method [19]. This architecture consists of N identical processor units (PUs), where the i-th PU performs the i-th algorithm step. Therefore all PUs must perform both types of the algorithm operations: multiplication with subtraction and division. Because a new matrix A can be processed as soon as the input of the previous matrix A is completed, the above scheme is characterized by the pipelining period of t = N 2 steps and the processor utilization η → 1/6. Note that the obtained value of the processor utilization can be improved in a processor array architecture which consists of a given number h of processor elements. In order to design such a processor array structure, where value h depends on the target FPGA device, the locally parallel globally sequential (LPGS) method of the dependence graph decomposition is used [19]. In accordance with this method, all nodes within one subgraph are processed concurrently, while all subgraphs are processed sequentially. As a result all intermediate data which correspond to data dependencies between subgraphs can be stored in FIFO buffer or random access memory (RAM) block outside the processor array.

140

O. Maslennikow et al.

Fig. 1. Dependence graph of Cholesky algorithm and one of possible processor array structures

Starting with the graph GB, we try to decompose it into a set of s =]N/h[ subgraphs having the same (or nearly same) topology, where ]e[ denotes the nearest integer equal to or greater then e. As evident from Fig.1, this can be done only if we cut the graph GB using a set of planes parallel to i-axis. These lines decompose GB into s regular subgraphs with h layers each. After mapping each resulting g-th subgraph (g = 1, 2, ..., s) into corresponding architecture, the fixed-size processor array structure S2 is obtained (see Fig.2). In this array all PUs are of the same type, and must implement addition, multiplication, division and square root. The total execution time T of the Cholesky algorithm realization in this array is nearly equal to T =

S 

(N − g · h + h)2 = s · N 2 /3 + N

(3)

g=1

time steps, and the asymptotic processor utilization η → 1/2 when N >> h. Note that this array processor is characterized by the highest value of utilization η and lowest time T among known architectures [19]-[22]. The analysis of the processing units of any modern architecture shows that, firstly, all operations are implemented in the pipelined mode. Secondly, different

Parallel Implementation of Cholesky LLT -Algorithm

141

Fig. 2. The designed fixed-size processor array structure S2

operations like addition, division, and square root have different latent period of their calculations. Therefore, to keep the synchronous mode of calculations, the synchronization period of Cholesky algorithm must be equal to the period of the longest operation; here it is the square root operation. As a result, the above estimations of PUs loadings can be decreased dramatically. This fact forced the effort to find out an another form of Cholesky algorithm, in which square root operations are separated from others. Table 1 shows the calculations in the original Cholesky algorithm for N = 4. Let us consider calculation of the variable a∗22 = a22 − a21 · a21 /(l11 · l11 ). In this expression, value l11 is equal to the square root of a11 , i.e. the datum is recalculated as (l11 · l11 ) = a11 . Thus, here the unnecessary calculations are made, which introduce additional errors. Note that similar calculations are made for elements a∗32 , ..., a∗44 . In order to eliminate unnecessary computations, we introduce extra variables mji in the original Cholesky algorithm, where mji = aji /aii (for example, m21 = a21 /a11 ) and change the standard order of computation. The modified algorithm is represented in Table 2. The analysis of the modified algorithm shows that for the i-th PU its arithmetic unit must perform only two types of operations: division and multiplication with subtraction. Note that the new order allows us to group all square root operations in the last (N -th) algorithm step, and therefore to move these operations to the last processor element of the array, denoted as SP in Fig.3. Such modifications cause to further reduction of the processor array hardware overhead. The only disadvantage is increase in memory volume and bandwidth of communication channels necessary to store and transfer multipliers mij .

Fig. 3. Modified processor array structure

142

O. Maslennikow et al.

Table 1. Standard order of computations in Cholesky LLT -decomposition algorithm (N = 4) Step 1 √ l11 = a11 l21 = a21 /l11 l31 = a31 /l11 l41 = a41 /l11 a∗22 = a22 − l21 · l21 a∗32 = a32 − l31 · l21 a∗42 = a42 − l41 · l21 a∗33 = a33 − l31 · l31 a∗43 = a43 − l41 · l31 a∗44 = a44 − l41 · l41

Step 4 Step 2 Step 3  l22 = a∗22 l33 = a∗∗ l44 = a∗∗∗ 33 44 l32 = a∗32 /l22 l43 = a∗∗ 43 /l33 ∗∗ l42 = a∗42 /l22 a∗∗∗ 44 = a44 − l43 · l43 ∗∗ ∗ a33 = a33 − l32 · l32 ∗ a∗∗ 43 = a43 − l42 · l32 ∗ a∗∗ = a 44 44 − l42 · l42

Table 2. Cholesky algorithm with reordered computations (N = 4) Step 1 m21 = a21 /a11 m31 = a31 /a11 m41 = a41 /a11 a∗22 = a22 − a21 · m21 a∗32 = a32 − a31 · m21 a∗42 = a42 − a41 · m21 a∗33 = a33 − a31 · m31 a∗43 = a43 − a41 · m31 a∗44 = a44 − a41 · m41

Step 2 Step 3 ∗∗ m32 = a∗32 /a∗22 m43 = a∗∗ 43 /a33 ∗∗ m42 = a∗42 /a∗22 a∗∗∗ = a − a∗∗ 44 44 43 · m43 ∗∗ ∗ ∗ a33 = a33 − a32 · m32 ∗ ∗ a∗∗ 43 = a43 − a42 · m32 ∗∗ ∗ a44 = a44 − a∗42 · m42

l11 l22 l33 l44 l21 l31 l41 l32 l42 l43

Step 4 √ = a11 = a∗22 = a∗∗ 33 = a∗∗∗ 44 = m21 · l11 = m31 · l11 = m41 · l11 = m32 · l22 = m42 · l22 = m43 · l33

The modified Cholesky algorithm can be represented as follows: for i = 1 to N-1 do begin for j = i+1 to N do begin m(j, i) = a(j, i)/a(i, i); for k = i+1 to j do a(j, k) = a(j, k) - a(j,i)*m(k, i); end end for i = 1 to N do l(i, i) = SQRT(a(i, i)); (4) for i= 1 to N do for j = i+1 to N do l(j, i) = m(j, i)*l(i, i); It is obvious from this description that square root calculations can be implemented separately, in the pipelined mode.

Parallel Implementation of Cholesky LLT -Algorithm

3

143

Rational Fraction Arithmetic Unit

In this section, we discuss how to implement the obtained processor array architecture in the Xilinx Virtex4 FPGAs. The main quality criteria of the design process are both reducing the PU arithmetic unit hardware and maximizing the clock. The designed AU must perform all types of arithmetic operations (square roots, divisions and multiplications with subtractions) in the pipelined mode. The authors propose to use the rational fraction AUs in the designed processor array, in order to utilize effectively the FPGA resources. Pipelined implementation of a given arithmetic operation is effective if AU consecutively performs several identical operations of this type. It is clear from Table 2 that only the division result may initiate the inner loop of the first loop nest. Therefore it is preferable to implement the division operation and subtraction with multiplication with an equal calculation period. Moreover, it would be better to redirect the division result m(j, i) just to the input of the subtraction with multiplication hardware, not to store it in the memory. As a result, AU performs multiplication with or without addition (subtraction) P = Z × X + Y , division operation P = Z/X, and square root P = SQRT (X), where X, Y , Z and P are rational fraction numbers. In case of multiplication with addition, AU calculates the following expression: Pn Zn Xn Yn (Zn · Xn ) · Yd + (Zd · Xd ) · Yn = · + = Pd Zd Xd Yd (Zd · Xd ) · Yd

(4)

Note that in case of a division operation P = Z/X, operands Z and X are substituted to each other and operand Y is equal to zero. Newton-Rafson method allows to calculate the expression P = SQRT (X) based on a given approximate value P1 of the result P . Then this value is modified while performing K iterations of the following form: P k+1 = (P k + X/P k )/2,

(5)

where k = 1, 2, ..., K. This expression differs from the multiplication with addition only in substituting numerator and denominator of the multiplier to each other, and in a single bit shift right of the resulting numerator. Therefore, only slight modifications are required to perform the square root iteration on the AU designed for multiplication with addition, and division. In the proposed AU structure, the values of numerator Pn1 and denominator 1 Pd of P1 are obtained from the read-only memory (ROM) modules. Their volume is very small, and after performing K = 4 iterations the needed result precision is provided. The AU structure is shown in Fig.4, where indices n and d denote the numerator and denominator of the corresponding fraction number. This AU consists of five multiply units MPUs, one adder SM and two normalization blocks NM1, NM2. These blocks shift left both numerator and denominator of the operation result to the equal bit number to prevent the significant bits loss after

144

O. Maslennikow et al.

the multiplication result truncation. Input multiplexers MX serve to exchange the numerator and denominator of an operand Z, to feed back the intermediate results P i during the square root calculation, and the division result before implementing the inner loop in the modified Cholesky algorithm. The designed rational fraction AU has been implemented in the Xilinx Virtex4 XC2VP4 device. This device has built-in DSP48 blocks based on 18-bit multiply units with 48-bit accumulators and 18 kilobit dual port BlockRAM blocks. The fraction bit width was selected equal to 35. This bit width is enough to process matrices of size of up to thousands of columns with the enough precision, as it was shown in [15]. A single 35-bit multiplier is implemented on four DSP48 blocks without additional logic. In Table 3 the performance of designed AU is compared with the known double precision floating point AUs, which have been implemented in similar FPGA devices. To compare the efficiency of different variants, we consider a system corresponding to the structure S3 from Fig.3; it can be configured in a single FPGA device. Note that the Xilinx XC4VSX55 device contains 512 DSP48 modules and 24648 CLB slices. This comparison shows that the proposed AU has higher throughput and minimized configurable hardware volume, which is in 1,7-9 times less than in other AUs, and provides much higher throughput thanks to its small hardware volume in CLB slices.

Fig. 4. Structure of the proposed rational fraction AU

Parallel Implementation of Cholesky LLT -Algorithm

145

Table 3. Comparison of the designed AU with double precision floating point AUs (* division is not implemented; ** floating point divider was added to the PU) AU parameter Hardware volume: slices multiplier units Pipeline stages (latent delay) for addition and multiplication for division for square root Maximum clock frequency PU number in XC4VSX55 device Throughput of the system in XC4VSX55, Mflops

Proposed AU 811 20

AU in [7] 7311 9

AU in [2]* 1419 9

9 9 45 175 25

12+10 56 56 187 5

5+8 200 5**

4300

900

1000

In order to estimate the precision of computations in the designed rational fraction AU, the calculation error δ for elements of the resulting matrix L has been determined as:   N i   δ = [ (ai,j − qi,j )2 ]/(0, 5 · N 2 ) (6) i=1 j=1

where qij are elements of the matrix Q = L · LT . We have computed the values δ for different data bit widths of the AU. Computation has been performed for input matrices A(N, N ) with N = 10...120. The obtained results are presented in Fig.5 for N = 20. It follows from Fig.5 that the error δ is nearly equal to δ = 5 ∗ 10−9 when the bit width is equal to 35. Moreover, experimental results show that the error δ slowly increases (up to 2.5 times) with increasing of the matrix size N from 10 to 120. Therefore, the designed 35-bit rational fraction AU can be used to compute the LLT decomposition of the matrices of size up to 1000 columns with enough precision.

Fig. 5. Computational errors of the proposed rational fraction AU

146

4

O. Maslennikow et al.

Conclusions

Cholesky algorithm for the LLT -decomposition of symmetric matrices is characterized by minimized computations and inherent parallelism at the cost of a set of division and square root operations. The last properties make it unattractive to implement it in the FPGA-based parallel systems, which perform the double precision floating-point calculations. In this paper, the arithmetic unit structure is proposed, which calculates division and square root as well as multiplication with addition on the base of the rational fraction data representation, and can be effectively configured in modern FPGAs. The advantages of using the rational fractions in the modern FPGA implementation are: small hardware volume, high throughput, possibility to regulate the precision and hardware volume by selecting the data width. The VHDL modeling has shown the possibility of using such a data representation in solving systems of linear equations by different methods, with a sufficiently reduced hardware complexity of AUs comparing to similar AUs operating with float-point numbers, without decreasing AU performance.

Acknowledgment This work was supported by the Polish Ministry of Science and Higher Education under grant N515 002 32/0176.

References 1. Underwood, K.D., Hemmert, K.S.: Closing the Gap: CPU and FPGA Trends in Sustained Floating Point BLAS Performance. In: Proc. IEEE Symp. Field Programmable Custom Computing Machines, FCCM-(2004) 2. Dou, Y., Vassiliadis, S., Kuzmanov, G.K., Gaydadjiev, G.N.: 64-bit Floating Point FPGA Matrix Multiplication. In: ACM/SIGDA 13-th Int. Symp. on Field Programmable Gate Arrays, FPGA-2005, pp. 86–95 (2005) 3. Johnson, J., Nagvajara, P., Nwankpa, C.: High-Performance Linear Algebra Processor using FPGA. In: Proc. High Performance Embedded Computing, HPEC 2003 (2003) 4. El-Kurdi, Y., Gross, W.J., Giannacopoulos, D.: Sparse Matrix-Vector Multiplication for Finite Element Method Matrices on FPGAs. In: Proc. 14th IEEE Symp. on Field-Programmable Custom Computing Machines FCCM 2006 (2006) 5. Beauchamp, M.J., Hauck, S., Underwood, K.D., Hemmert, K.S.: Embedded Floating-Point Units in FPGAs. In: Proc. ACM Int. Symp. on Field Programmable Gate Arrays, Monterey, CA (2006) 6. Durbano, J.P., Ortiz, F.E., Humphrey, J.R., Prather, D.W.: FPGA-Based Acceleration of the 3D Finite-Difference Time-Domain Method. In: Proc. 12-th IEEE Symp. on Field-Programmable Custom Computing Machines FCCM 2004 (2004) 7. Xilinx Floating-point Operators v2.0 DS335, Xilinx (2006), http://www.xilinx.com 8. Storaasli, O.: Scientific Applications on a NASA Reconfigurable Hypercomputer. In: Proc. 5-th MAPLD Conf. (2002)

Parallel Implementation of Cholesky LLT -Algorithm

147

9. Strenski, D.: Computational Bottlenecks and Hardware Decisions for FPGAs. FPGA and Structured ASIC Journal (14), 1–20 (2006) 10. Craven, S., Athanas, P.: Examining the Viability of FPGA Supercomputing. EURASIP Journal on Embedded Systems (2) (2007) 11. Lienhart, G., Kugel, A., M¨ anner, R.: Using floating-point arithmetic on FPGAs to accelerate scientific n-body simulations. In: Proc. 10-th Ann. IEEE Symp. on Field-Programmable Custom Computing Machines, FCCM 2002, p. 182 (2002) 12. Strzodka, R., Goddeke, D.: Pipelined Mixed Precision Algorithms on FPGAs for Fast and Accurate PDE. Solvers from Low Precision Components. In: Proc. FieldProgrammable Custom Computing Machines (2006) 13. Matousek, R., Tichy, M., Phol, Z., Kadlec, J., Softley, C., Coleman, N.: Logarithmic number systems and floating-point arithmetics on FPGA. In: Proc. 12-th Int. Conf. on Field Programmable Logic and Applications, London, pp. 627–636 (2002) 14. Maslennikow, O., Shevtshenko, J., Sergyienko, A.: Configurable Microprocessor Array for DSP applications. In: Wyrzykowski, R., Dongarra, J., Paprzycki, M., Wa´sniewski, J. (eds.) PPAM 2004. LNCS, vol. 3019, pp. 36–41. Springer, Heidelberg (2004) 15. Maslennikow, O., Sergyienko, A., Lepekha, V.: FPGA Implementation of the Conjugate Gradient Method. In: Wyrzykowski, R., Dongarra, J., Meyer, N., Wa´sniewski, J. (eds.) PPAM 2005. LNCS, vol. 3911, pp. 526–533. Springer, Heidelberg (2006) 16. Golub, G.G., Van Loan, C.F.: Matrix Computations, 2nd edn., p. 642. J. Hopkins Univ. Press, Baltimore (1989) 17. Irvin, M.J., Smith, D.R.: A rational arithmetic processor. In: Proc. 5-th Symp. Comput. Arithmetic, pp. 241–244 (1981) 18. Wyrzykowski, R., Kanevski, J., Maslennikova, N., Maslennikov, O., Ovramenko, S.: Formalized Construction Method of Array Functional Graphs for Regular Algorithms. In: Engineering Simulation, vol. 14, pp. 217–232. Gordon and Breach Science Publishers, England (1997) 19. Kung, S.Y.: VLSI Array Processors. Prentice-Hall, Englewood Cliffs (1988) 20. Moreno, J.H., Lang, T.: Matrix Computation on Systolic-Type Arrays. Kluwer Acad. Publ., Boston (1992) 21. Quinton, P., Robert, Y.: Systolic Algorithms and Architectures. Prentice-Hall, Engl. Cliffs (1991) 22. Cosnard, M., Trystram, D.: Parallel Algorithms and Architectures. Int. Thomson Computer Press, Boston (1995)

Dimensional Analysis Applied to a Parallel QR Algorithm Robert W. Numrich Minnesota Supercomputing Institute University of Minnesota Minneapolis, MN 55455 USA

Abstract. We apply dimensional analysis to a formula for execution time for a QR algorithm from a paper by Henry and van de Geijn. We define a single efficiency surface that reduces performance analysis for this algorithm to an exercise in differential geometry. As the problem size and the number of processors change, different machines move along different paths on the surface determined by two computational forces specific to each machine. We show that computational force, also called computational intensity, is a unifying concept for understanding the performance of parallel numerical algorithms. Keywords: Scalability, performance analysis, computational intensity, computational force, parallel numerical algorithms, dimensional analysis.

1

Introduction

In a paper on the scalability of a parallel QR algorithm, Henry and van de Geijn [8] report the formula, t = (Cn + 20n2 /p + 40n)γ + (n/r + 2n/h)α + (3n + 4n2 /hp)β ,

(1.1)

for execution time as a function of problem size n and the number of processors p. The details of the algorithm are not important for the purpose of this paper except to say that, according to Henry and van de Geijn, the QR algorithm is notoriously difficult to program such that it scales to large numbers of processors. In a later paper, Henry, Watkins and Dongarra [9] report another algorithm, which scales better but has a more complicated timing formula. For simplicity of presentation, we use the older formula to illustrate the principles of dimensional analysis as they apply to performance analysis for parallel algorithms. In a system of units based on length, energy and time, the quantities involved in timing formula (1.1) have the matrix of dimensions, l 0 e 0 t0 t γ α β n p C r h L(word) 1 0 0 0 0 0 −1 0 0 0 0 0 E(flop) 0 1 0 0 −1 0 0 0 0 0 0 0 T (s) 0 0 11 1 1 1 00 00 0 R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 148–155, 2008. c Springer-Verlag Berlin Heidelberg 2008 

(1.2)

Dimensional Analysis Applied to a Parallel QR Algorithm

149

where the unit of length is the eight-byte word, l0 = 1 word, the unit of energy (work) is the eight-byte floating-point operation, e0 = 1 flop, and the unit of time is one second, t0 = 1 s. The quantity t is the execution time; the quantity γ is the reciprocal of the computational power of a single processor; α is a startup time; β is the reciprocal of bandwidth; n is the matrix size; and p is the number of processors. The three parameters C, h, and r are specific to the details of the QR algorithm. The block size h is set to the value h = n/p; the “bundling factor” r is set to the value r = 2; and the number of operations C for a Householder transformation is an undefined small number, which we take to be C = 5.

2

Dimensional Analysis

Dimensional analysis is based on the algebraic properties of the matrix of dimensions [2,4]. From the matrix (1.2), it is clear that formula (1.1) is not dimensionally consistent. This formula, along with many others like it, lacks a clear distinction between the unit of length and the unit of work. The problem size, which is dimensionless, enters into the calculation of both the number of operations performed and the amount of data moved. The reader is expected to infer units of work or units of length based on various powers of the problem size. The work done in this QR algorithm, for example, corresponds to the first factor of the first term of formula (1.1), w = (Cn + 20n2 /p + 40n)e0 .

(2.1)

The first factor of the second term counts the number of startup penalties incurred, s = (n/r + 2n/h) , (2.2) and is dimensionless. The first factor in the third term, l = (3n + 4n2 /hp)l0 ,

(2.3)

equals the number of words moved. To make the formula dimensionally consistent, therefore, we need to rewrite it as, t = wγ + sα + lβ .

(2.4)

Execution time, then, is a function of six dependent variables, t = t(w, l, s, α, β, γ) .

(2.5)

We have found in previous work [15,16] that a system of units based on length, force and time provides a unifying framework for performance analysis. Computational force is measured in the unit, f0 = e0 /l0 = 1 flop/word.

(2.6)

It is also called computational intensity [6,10,11,12,13] and is recognized as an important quantity in performance analysis. In a system of measurement with

150

R.W. Numrich

length, force and time as primary units, the quantities involved in relationship (2.5) have the matrix of dimensions, l0 f0 t0 t w l s α β γ L(word) 1 0 0 0 1 1 0 0 −1 −1 F (flop/word) 0 1 0 0 1 0 0 0 0 −1 T (s) 0 0 11 000 1 1 1

(2.7)

The basic assumption of dimensional analysis is that relationship (2.5) is independent of the system of units we use [1,2,5,14,15,16]. Under this assumption, we may pick three scaling parameters, αL , αF , αT , one for each primary unit, and scale each quantity, according to its entries in the matrix of dimensions, such that the relationship still holds, −1 −1 αT t = t(αL αF w, αL l, s, αT α, α−1 L αT β, αL αF αT γ) .

(2.8)

We can pick these scale factors in such a way to establish a new system of units where the number of parameters reduces from six to three and all quantities are scaled to dimensionless ratios. Indeed, if we pick them such that αL αF w = 1 ,

αL l = 1 ,

−1 α−1 L αF αT γ = 1 ,

(2.9)

we can solve for the three parameters to find αL = 1/l ,

αF = l/w ,

αT = 1/(γw) .

(2.10)

Relationship (2.8) scaled by these parameters becomes the new relationship, t/(γw) = t(1, 1, s, α/(γw), lβ/(γw), 1) ,

(2.11)

involving only three quantities rather than six. We recognize the new unit of time, t∗ = 1/αT , as the minimum time to complete the computation with the value, t∗ = γw ,

(2.12)

from (2.10). The left side of equation (2.11) is, therefore, the reciprocal of the efficiency, t/t∗ = 1/e . (2.13) Scaling both sides of (2.4) using the parameters from (2.10), we obtain the efficiency function, e = [1 + sα/(γw) + lβ/(γw)]−1 .

3

(2.14)

Computational Forces and the Efficiency Surface

The efficiency formula (2.14) depends on ratios of computational forces. To recognize this fact, we first observe that the efficiency formula assumes the simple form, 1 e= , (3.1) 1+u+v

Dimensional Analysis Applied to a Parallel QR Algorithm

151

in terms of the two variables, u = sα/(γw) ,

v = lβ/(γw) .

(3.2)

Each of these variables is the ratio of opposing computational forces. Using the quantities defined by equations (2.1)-(2.2) and the values h = n/p, r = 2, we find the first variable, φH u(n, p) = S 1 , (3.3) φ1 (n, p) is the ratio of a hardware force, φH 1 = (α/l0 )/γ ,

(3.4)

n(20n/p + 40 + C) f0 . n/2 + 2p

(3.5)

to a software force, φS1 =

In the same way, the second variable is the ratio of two other opposing forces, v(n, p) =

φH 2 , φS2 (n, p)

(3.6)

with a second hardware force, φH 2 = β/γ ,

(3.7)

and a second software force, φS2 (n, p) =

20n/p + 40 + C f0 . 7

(3.8)

The two hardware forces are independent of the problem size and the number of processors. The first one is determined by the ratio of the startup time to the reciprocal of the computational power, and the second one is determined by the ratio of the reciprocal of the bandwidth to the reciprocal of the computational power. The two software forces, on the other hand, are functions of problem size and the number of processors, independent of the hardware. They are determined by this particular version of the QR algorithm. To help visualize how efficiency changes as a function of problem size and the number of processors, we define curvilinear coordinates on the efficiency surface. For fixed problem size, n = n∗ , we use the number of processors as a parameter to define the curvilinear coordinates, u∗ (p) = u(n∗ , p) ,

v ∗ (p) = v(n∗ , p) .

(3.9)

As the number of processors changes, these coordinates define paths along the surface determined by the equation, e∗ (p) =

1 1+

u∗ (p)

+ v ∗ (p)

.

(3.10)

152

R.W. Numrich

These paths correspond to strong scaling where the problem size remains fixed and the efficiency decreases as the number of processors increases. Different machines follow different paths determined by their particular values of the hardH ware forces φH 1 and φ2 . Similarly, for fixed number of processors, p = p∗ , the curvilinear coordinates, u∗ (n) = u(n, p∗ ) ,

v∗ (n) = v(n, p∗ ) ,

(3.11)

define paths along the surface, e∗ (n) =

1 , 1 + u∗ (n) + v∗ (n)

(3.12)

as the problem size changes. These paths correspond to weak scaling where the problem size increases to obtain higher efficiency for a fixed number of processors.

4

Discussion

The relative sizes of the opposing hardware and software forces determine the efficiency of the algorithm. From (3.1) it is clear that high efficiency requires small values for the variables u and v. Machines with large hardware forces require large software forces to obtain the same efficiency. The following table, α β −1 γ −1 φH 1 φH 2

Machine 1 ca. 1990 Machine 2 ca. 2007 10−4 s 10−6 s 6 10 word/s 109 word/s 6 10 flop/s 109 flop/s 2 10 flop/word 103 flop/word 1 flop/word 1 flop/word

(4.1)

H shows how the hardware forces φH 1 and φ2 have changed over approximately the last two decades. We expect modern machines to be less effecient than older machines for the same size problem. For this algorithm, equal efficiency requires higher software forces obtained by increasing the problem size or by decreasing the number of processors. Figure 1 shows two paths along the efficiency surface, one for each machine in the table. The number of processors is fixed at p = 512, and each point on a path corresponds to a different problem size calculated from the curvilinear coordinates u∗ (n) and v∗ (n) from equation (3.11). For large enough problem size, both machines approach perfect efficiency. But they follow different paths at different rates along the surface to reach the summit. The modern machine is clearly less efficient than the earlier machine. Our approach to performance analysis [14,15,16] is quite different from the more familiar approaches that have appeared in the literature. Given formula (3.1), our first impulse is to plot the surface, or contours of the surface, as a

Dimensional Analysis Applied to a Parallel QR Algorithm

153

1 0.9 0.8 0.7

e

0.6 0.5 0.4 0.3 0.2

0 10

0.1 0 0

20 30 0.05

0.1

40 0.15

0.2

50

u

v

Fig. 1. Two paths along the efficiency surface for fixed number of processors, p = 512. The problem size increases from n = 500 at low efficiency in increments of 100. In the limit of very large problem size, both machines approach perfect efficiency. They approach the limit at different rates along different paths on the surface. The higher path on the surface corresponds to Machine 1; the lower path to Machine 2.

function of the problem size and the number of processors. For example, Henry and van de Geijn [8] display the results in the first three columns of the following table, p n efficiency(measured) efficiency(modeled) 4 2000 0.63 0.76 8 2800 0.65 0.69 16 4000 0.63 0.62 32 4800 0.53 0.39 48 6000 0.48 0.45 64 8000 0.47 0.45 96 9600 0.39 0.39

(4.2)

measured on an Intel Paragon XP/S Model 140 parallel system. Unfortunately, they do not specify the parameters α, β, and γ that enter into their timing formula (1.1). Nor, for that matter, do they specify the value of the constant C, which we have arbitrarily taken as C = 5. This lack of information makes it difficult to compare their measured results with their analytic results. It is especially difficult, after ten to fifteen years, to determine what exactly the Intel Paragon system was and what the particular parameters for the machine might have been. As our best guess, we have used the values α = 30 microseconds and β −1 = 175 Mbyte/s from Bolen et al. [3]. From reference [7] we find that the frequency of the machine was ν = 200 MHz and that it could perform one eight-byte floating-point operation per clock cycle. We therefore picked the value γ −1 = 200 Mflop/s. These values correspond to forces φH 1 = 6000 flop/word and φH = 9.1 flop/word. 2 The fourth column in Table (4.2) shows the efficiencies calculated using these parameters in formula (3.1), and the solid line in Figure 2 shows the path across

154

R.W. Numrich

1 0.9 0.8 0.7

e

0.6 0.5 0.4 0.3 0.2 0.1 0

0.04 1

0.03 2

0.02 3

0.01 4

0 v

u

Fig. 2. The path followed across the efficiency surface for the selected values of p and n chosen by Henry and van de Geijn [8]. The solid line is the path on the surface predicted by the analytic formula (3.1). The bullets are the measured efficiencies. Some of the measured values lie below the surface and are not visible from this perspective.

the efficiency surface followed by these values. The bullets mark the measured efficiencies from column three of the table. Some of the measured values lie below the surface and are not visible from this viewing angle. Agreement of the measured values with the predicted values is reasonably good, especially given that we had to guess the values for the parameters that enter the analytic formula.

5

Summary

We have shown that for a particular QR algorithm there exists one efficiency surface for all machines. Each machine follows its own path along the surface H depending on two hardware forces φH 1 and φ2 . Two machines with the same hardware forces are self-similar and scale the same way even though the particular values for their startup time α, their bandwidth β −1 , and their computational power γ −1 may be quite different. For a fixed problem size and number of processors, the distance along the geodesic between paths is a measure of the distance between machines. The two paths shown in Figure 1, for example, indicate that modern machines are a long distance from earlier machines and that they are less efficient. They travel a long way across the surface at low efficiency before mounting the ascent to the summit. The modern machine has higher values for the coordinate u∗ (n) for each point along the paths. Startup times, although they have decreased, have not kept up with higher computational power so that the hardware force φ1 , which determines the coordinate u∗ (n), has increased by a factor of ten. It is harder now to overcome long startup times than it was before. This difficulty translates directly into a harder job for the programmer who must overcome the larger

Dimensional Analysis Applied to a Parallel QR Algorithm

155

hardware force by designing a new algorithm as done, for example, by Henry, Watkins and Dongarra in a later paper [9].

Acknowledgment The United States Department of Energy supported this research by Grant No. DE-FG02-04ER25629 through the Office of Science.

References 1. Barenblatt, G.I.: Scaling. Cambridge University Press, Cambridge (2003) 2. Birkhoff, G.: Hydrodynamics: A Study in Logic, Fact and Similitude, 2nd edn. Princeton University Press, Princeton (1960) 3. Bolen, J., Davis, A., Dazey, B., Gupta, S., Henry, G., Robboy, D., Schiffler, G., Scott, D., Stallcup, M., Taraghi, A., Wheat, S., Fisk, L., Istrail, G., Jong, C., Riesen, R., Shuler, L.: Massively Parallel Distributed Computing: World’s First 281 GigaFlop Supercomputer. In: Proceedings of the Intel Supercomputer Users Group (1995), http://www.cs.utk.edu/∼ ghenry/isug.ps 4. Brand, L.: The Pi Theorem of Dimensional Analysis. Arch. Rat. Mech. Anal. 1, 35–45 (1957) 5. Bridgman, P.W.: Dimensional Analysis, 2nd edn. Yale University Press, New Haven (1931) 6. Callahan, D., Cocke, J., Kennedy, K.: Estimating interlock and improving balance for pipelined architectures. Journal of Parallel and Distributed Computing 5, 334– 358 (1988) 7. Henry, G.: (1995), http://www.cs.utk.edu/∼ ghenry/mplin.soft 8. Henry, G., van de Geijn, R.A.: Parallelizing the QR algorithm for the unsymmetric algebraic eigenvalue problem: myths and reality. SIAM Journal on Scientific Computing 17(4), 870–883 (1996) 9. Henry, G., Watkins, D., Dongarra, J.: A parallel implementation of the unsymmetric QR algorithm for distributed memory architectures. SIAM Journal on Scientific Computing 24(1), 284–311 (2002) 10. Hockney, R.W.: Performance parameters and benchmarking of supercomputers. Parallel Computing 17, 1111–1130 (1991) 11. Hockney, R.W.: The Science of Computer Benchmarking. Society for Industrial and Applied Mathematics (SIAM), Philadelphia (1996) 12. McCalpin, J.D.: Memory bandwidth and machine balance in current high performance computers, IEEE Computer Society; Technical committee on Computer Architecture Newsletter (December 1995) 13. Miles, D.: Compute intensity and the FFT. In: Proceedings Supercomputing 1993, pp. 676–684 (1993) 14. Numrich, R.W.: A note on scaling the Linpack benchmark. Journal of Parallel and Distributed Computing 67(4), 491–498 (2007) 15. Numrich, R.W.: Computer Performance Analysis and the Pi Theorem. Under review (2007) 16. Numrich, R.W.: Computational force: A unifying concept for scalability analysis. In: Proceedings of ParCo 2007 (to appear, 2007)

Sparse Matrix-Vector Multiplication - Final Solution? ˇ Ivan Simeˇ cek and Pavel Tvrd´ık Department of Computer Science and Engineering, Czech Technical University, Prague xsimecek, [email protected]

Abstract. Algorithms for the sparse matrix-vector multiplication (shortly SpM ×V ) are important building blocks in solvers of sparse systems of linear equations. Due to matrix sparsity, the memory access patterns are irregular and the utilization of a cache suffers from low spatial and temporal locality. To reduce this effect, the register blocking formats were designed. This paper introduces a new combined format, for storing sparse matrices that extends possibilities of the variable-sized register blocking format.

1

Introduction

There are several formats for storing sparse matrices. They have been designed mainly for the SpM ×V . The SpM ×V for the most common format, the compressed sparse rows (shortly CSR) format, suffers from low performance due to the indirect addressing. Several papers focussed on increasing the efficiency of the SpM ×V [1,2]. There are some formats, such as register blocking, that eliminate indirect addressing during the SpM ×V . Then, vector instructions can be used. These formats are suitable only for matrices with a known structure of nonzero elements. The overhead of a reorganization of a matrix from one format to another one is often of the order of tens of executions of a SpM ×V . So, such a reorganization pays off only if the same matrix A is multiplied with multiple different vectors, e.g., in iterative linear solvers. In this paper, we propose a new format designed mainly for the SpM ×V . We compare the performance with the CSR format and other related works. Our measurements shows that this new format can gain the significant speedup.

2 2.1

Terminology and Notation The Cache Model

The cache model we consider corresponds to the structure of L1 and L2 caches in the Intel x86 architecture. An s-way set-associative cache consists of h sets and R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 156–165, 2008. c Springer-Verlag Berlin Heidelberg 2008 

Sparse Matrix-Vector Multiplication - Final Solution?

157

one set consists of s independent lines. Let CS denote the size of the data part of a cache in bytes and LS denote the cache line size in bytes. Then CS = s · LS · h. 2.2

Definitions and Notation

In the following text, we assume that A is a real sparse matrix of order n. We will use the following definitions and notation: – nZ denotes the total number of nonzero elements in A. – A block is a small submatrix of matrix A. It can be linear (horizontal, vertical, or diagonal) or rectangular. – A partially full block is a block that contains at least one nonzero element. Its fill-in ratio, denoted by α, is the ratio between the number of nonzero elements in the block and the size of the block. To be more precise, we will sometimes call these blocks α-dense blocks. – A fully dense block is a block that contains only nonzero elements (α=1). – NB denotes the number of partially full blocks. – L denotes the total number of elements in all blocks. – SB denotes the size of the auxiliary data structure for a block (see Section 2.3). – SD denotes the size of type double in bytes. – SI denotes the size of type integer in bytes. 2.3

Common Sparse Matrix Formats

The coordinate (XY) format. In this simplest sparse format, the matrix A is represented by 3 linear arrays A, Xpos, and Y pos. Array A[1, . . . , nZ ] stores the nonzero elements of A, arrays Xpos[1, . . . , nZ ] and Y pos[1, . . . , nZ ] contain x-position and y-position, respectively, of the nonzero elements. The compressed sparse row format. It is the most common format for storing sparse matrices. A matrix A stored in the CSR format is represented by 3 linear arrays A, adr, and ci.Array A[1, . . . , nZ ] stores the nonzero elements of A, array adr[1, . . . , n] contains indexes of initial nonzero elements of rows of A, and array ci[1, . . . , nZ ] contains column indexes of nonzero elements of A. Hence, the first nonzero element of row j is stored at index adr[j] in array A. Register blocking (RB) formats. Sparse matrices often contain dense submatrices (blocks in our terminology), so various blocking formats were designed to accelerate matrix operations (mainly the SpM ×V ). Compared to the CSR format, the aim of these formats is to consume less memory and to allow better use of registers. Also, the effect of indirect addressing is to be reduced and vector (SIMD) instructions can be used. The RB formats are designed to handle randomly occurring dense blocks in a sparse matrix. Geometrical and topological properties (such as structure and size of one-subgraphs) in matrices with faulty components are studied by many

158

ˇ I. Simeˇ cek and P. Tvrd´ık

author (for example in [3]). Blocks can be linear (horizontal, vertical, diagonal) or rectangular. There are 2 variants: – Fixed-sized register blocking (shortly FRB): Elements of A are grouped into partially-full blocks of the same size. For example, in case of the rectangular FRB, each block i has size rx · ry and is described by a pair [xstart ,yistart ] i (the position of its left upper corner) and rx ·ry numbers (the values of all its elements, including zeroes). This format has been deeply studied [2,4] (including fast heuristics for finding a suboptimal sizes rx and ry of rectangular blocks) and it is discussed in Section 3.1. – Variable-sized register blocking (shortly VRB): Nonzero elements of A are grouped into blocks whose sizes and shapes can differ. For example, in case of the diagonal VRB, each block i is described by a pair [xstart ,yistart ] (the i end end position of its begin) and yi (the y-position of its end) and yi −yistart +1 numbers (the values of all its elements). The main idea of the diagonal VRB format is illustrated on Figure 1 b). It can be implemented in 2 ways: • a linked list of diagonal blocks (see Figure 1 c)); • a pair of arrays block elems[1, . . . , L] (elements of diagonal blocks) and block aux[1, . . . , NB ] (items xstart , yistart , yiend , and pointers to the first i elements of blocks into the array block elems[1, . . . , L]) (see Figure 1 d)).

c)

a)

b) d)

Fig. 1. The idea of the VRB format: a) Nonzero elements of a sparse matrix A. b) A diagonal blocking of the nonzero elements. c) Blocks are stored as linked dense arrays. d) Blocks are stored in arrays block aux and block elems.

3

Related Work and Comparison

Storing a matrix in register blocking formats is the most common technique for improving the performance of the SpM ×V . There are 2 implementations of RB formats that are relevant for the proposal of this paper: – the SPARSITY method [2,4] (and his successor OSKI), – the CARB method [5].

Sparse Matrix-Vector Multiplication - Final Solution?

3.1

159

The SPARSITY Method

The SPARSITY method is an implementation of the rectangular FRB format. The SPARSITY transformation algorithm of a sparse matrix to the rectangular FRB format can be described by the following pseudocode. The SPARSITY transformation algorithm (simplified pseudocode) for rx = 1 to max do for ry = 1 to max do find a decomposition of the matrix into rx × ry blocks such that all nonzero elements are covered and the number of blocks is minimal. predict the SpM ×V performance for that decomposition endfor endfor transform the matrix into the FRB format with the best performance. The most important feature of this format is that it can adapt to machine specific parameters due to the performance prediction. Before the transformation, machine specific parameters are measured and incorporated into performance prediction. 3.2

The CARB Method

In [5], we have proposed an implementation of the diagonal VRB format, called the CARB format. Matrix A is viewed as a set of equal-sized horizontal belts. The belt size is computed from cache hierarchy parameters. Also, a lower bound α ¯ on the fill-in ratio α is estimated by a block heuristic using similar architecture dependent parameters. The CARB transformation algorithm can be described by the following pseudocode. The CARB transformation algorithm (simplified pseudocode) for every belt do for every diagonal in the current belt do if the fill-in ratio of the current diagonal is greater than α ¯ then represent the current diagonal as a diagonal block, else represent nonzero elements in the current diagonal in the CSR format endif endfor endfor The most important features of this method are very small transformation overhead and also the possibility to combine of the CSR and the VRB formats.

160

ˇ I. Simeˇ cek and P. Tvrd´ık

? Fig. 2. The block heuristic tries to find in the CARB format a configuration of blocks with minimal space complexity

4

Design of a New Format for Sparse Matrices

In this section, we describe a new linear VRB method called SMURB (Specific Machine parameters Using Register Blocking). Firstly, we describe the new transformation algorithm and derive the supplying equations. Secondly, we will describe our simplifying assumptions for the estimation of time complexity. For the purpose of finding an efficient SMURB format, a matrix A is partitioned into a set of square regions. A region is a temporary data structure representing a submatrix of A. 4.1

The SMURB Transformation Algorithm

The SMURB transformation algorithm follows. SMURB transformation (simplified pseudocode) for every region do if the number of nonzero elements in the current region > threshold then compute arrays V erti and Hori using Eq. 1)-5) compute array C using Eq. (6) backtrace the decomposition of current region from array C else represent nonzero elements in the current region in the CSR format endif endfor Let us discuss the algorithm in more details. The matrix is decomposed into equal-sized square regions of side length γ. The choice of γ is a tradeoff between space complexity of temporary data structures (for matrix transformation) and the probability of finding suitable blocks in a region. The experiments indicate that the best choice is γ ∈ 50, 200 for most of nowadays CPU architectures. The SMURB method uses the following transformation algorithm to find linear blocks inside each region. It uses dynamic programming to estimate the

Sparse Matrix-Vector Multiplication - Final Solution?

161

space complexity using different blocking schemas. It evaluates the complexity for each element in a region so its space complexity for an γ × γ region is Θ(γ 2 ). We precompute arrays Hori (and V erti) with space complexity of horizontal (and vertical) blocks. Firstly, we define array Single(x, y) for each element of the matrix A in this way:  0 if A(x, y) = 0, Single(x, y) = min (1) SD + 2SI if A(x, y) = 0. Then we can precompute arrays V ertin (x, y), V ertib (x, y)), and V erti(x, y). All these arrays denote space complexity of elements from (x, 0) to (x, y) using vertical blocks, but – In V ertib (x, y), the element (x, y) is in a vertical block. – In V ertin (x, y), the element (x, y) is not in a vertical block. – V erti(x, y) = min(V ertin (x, y), V ertib (x, y)). These arrays can be computed as follows: V ertin (x, 0) = Single(x, 0),

(2)

V ertib (x, 0) = SB + SD , //start of vertical block

(3)

⎧ ⎪ ⎪ V ertin (x, y − 1) + Single(x, y) ⎨ //continous vertical non-block, V ertin (x, y) = min V ertib (x, y − 1) + Single(x, y) ⎪ ⎪ ⎩ //end of vertical block. ⎧ V ertin (x, y − 1) + SB + SD ⎪ ⎪ ⎨ //start of vertical block, V ertib (x, y) = min V ertib (x, y − 1) + SD ⎪ ⎪ ⎩ //continous vertical block.

(4)

(5)

And similarly, we precompute arrays Horin (x, y), Horib (x, y), and Hori(x, y) that estimate the space complexity of elements from (0, y) to (x, y) using horizontal blocks. For elements with coordinates x ≥ 1, y ≥ 1, we can compute their space complexity of elements from (0, 0) to (x, y) by this equation (for better illustration, see Figure 3). ⎧ C(x − 1, y) + V erti(x, y) ⎪ ⎪ ⎪ ⎪ //continous vertical block, ⎪ ⎪ ⎨ C(x, y − 1) + Hori(x, y) C(x, y) = min //continous horizontal block, ⎪ ⎪ ⎪ ⎪ C(x−1, y−1)+Hori(x − 1, y)+V erti(x, y − 1)+Single(x, y) ⎪ ⎪ ⎩ //continous diagonal block. (6)

162

ˇ I. Simeˇ cek and P. Tvrd´ık C(x−1,y)

Verti(x,y)

111 000 000 111 000 111 000 111 000 111 a)

C(x,y−1)

11111 00000 00000 11111 00000 11111 Hori(x,y)

11111 00000 00000 11111 00000 11111 Hori(x−1,y)

b)

111 000 000 111 000 111 000 111 000 111

C(x−1,y−1)

Verti(x,y−1)

Single(x,y)

c)

Fig. 3. Graphical representation of the computation of the space complexity

In our approach, the algorithm finds the optimal configuration of blocks with minimal space complexity. The transformation time complexity for γ ×γ is Θ(γ 2 ) and it can be too much for some types of matrices. Therefore region size γ and the constant threshold must be chosen carefully. 4.2

Estimation of the SpM ×V Time Complexity

Time complexity can be predicted similarly, just replace terms SD , SI and SB in Eq. 1-6 with TD , TI and TB . We assume the following model: – The SpM ×V using the XY format takes time T = nZ · T I , where TI is an architecture dependent constant. – The SpM ×V using the CSR format takes time T = n · Tn + nZ · Tnz , where Tn and Tnz are architecture dependent constants. – The multiplication of NB blocks (all these blocks together contain L elements) takes time T = NB · T B + L · T D , where TB and TD are architecture dependent constants. 4.3

Comparison Related Results

Let us briefly compare previous methods and SMURB format. – In SPARSITY, all nonzero elements are in blocks. In the SMURB method (similarly in the CARB format), we use the combination of the VRB and the CSR format. – In SPARSITY, only 2D rectangular blocks of the same size can occur (the FRB format, see Section 2.3). In the CARB format, only diagonal blocks are used. In the SMURB format, we use only linear (horizontal, vertical, or diagonal) blocks. Block sizes can differ (the VRB format, see Section 2.3).

Sparse Matrix-Vector Multiplication - Final Solution?

163

– Various (even architecture dependent) block heuristics can be designed. We have designed a heuristic to find blocks with minimal space complexity or time of the SpM ×V . It is adaptive to machine specific parameters. – In SPARSITY, the time overhead of the matrix transformation is very high (in the order of thousands of execution of the SpM ×V ). In the CARB format, the transformation is fast simply because the block heuristic is trivial. Since the SMURB method uses much more complex heuristics, it is applied only for regions that are ”dense enough” due to the overhead of region transformation. This approach minimizes the total overhead of the matrix transformation and the matrix transformation algorithm is relatively faster (approximately one order less than in SPARSITY). – We extend the CARB’s idea of belts to regions. The regions divide matrix A in 2 dimensions so that sizes of all regions are approximately the same. The usage of regions allows to keep as much as possible elements in the cache memory during the SpM ×V (an application of the cache blocking).

5

Evaluation of the Results

All results was measured at Pentium Celeron M420 at 1.6 GHz, 1GB@ 266 MHz, running OS Windows XP Professional with the following cache parameters: L1 cache is data cache with LS = 64, CS = 32K, s = 8, h = 64, and LRU replacement strategy. L2 cache is data cache with LS = 64, CS = 1M B, s = 8, h = 2048, and LRU strategy. Microsoft Visual Studio 2003 Intel compiler version 9.0 with switches: /O3 /Og /Oa /Oy /Ot /Qpc64 /QxB /Qipo /Qsfalign16 /Zp16 5.1

Test Data

We have used 52 real matrices from various technical areas from the MatrixMarket and Harwell sparse matrix test collection. 5.2

Evaluation of the Results

The matrix transformation performance. The SMURB matrix transformation overhead depends strongly on the constant threshold (see Section 4.1). We have chosen threshold = 250. We can conclude that the time of the transformation varies from 50 to 200 executions of the SpM ×V for this threshold for all testing matrices. The SpM ×V performance Comparison with the CSR format We compare SpM ×V performance for the CSR and for SMURB format.Speedups depend strongly on the structure of nonzero elements (it means on the application domain of the matrix). Great speedups (sometimes over

ˇ I. Simeˇ cek and P. Tvrd´ık

164

200%) were achieved for matrices of partial differential equations (PDE) origin. No speedup for matrices with the near-random structure. More exactly, a significant speedup (more than 10%) has been achieved for real matrices from areas: diffusion operators, structural engineering, simulation of solid objects. On the other hand, the matrices from the following areas turned unsuitable for our new format: economical and social models, driven cavity problems, nonlinear chemical problems, matrices with near-random structure, but no slowdowns were observed. Comparison with the SPARSITY method It is hard to compare our and SPARSITY methods. The SMURB method achieved speedup for wider range of matrices due to more universal VRB scheme. But SPARSITY achieved higher speedups for some types of matrices due to the smaller overhead of the FRB format. Both of these methods fail for matrices with (almost) random structure. Comparison with the CARB method Figure 4 shows the relative speedup of the CARB and the SMURB formats for all testing matrices. Our implementation is always faster than the CARB format due to the more complex and machine adaptive block heuristics. 2.4 SMURB CARB 2.2

Speedup over the CSR format

2 1.8 1.6 1.4 1.2 1 0.8 0.6 0.4

50

40

30

20

10

Testing matrix

Fig. 4. The relative speedup of the SpM ×V using CARB and SMURB format

Sparse Matrix-Vector Multiplication - Final Solution?

6

165

Conclusions

In this paper, we described a new format called SMURB for storing sparse matrices that combine advantages of the CSR format and the VRB format. This format uses new transformation algorithm with a small overhead. This format is also adaptive to the machine specific parameters. For many types of matrices arising from various technical disciplines, our new format gives a significant performance speedup.

Acknowledgements ˇ This research has been supported by MSMT under research program MSM6840770014.

References 1. Mellor-Crummey, J., Garvin, J.: Optimizing sparse matrix vector product computations using unroll and jam. International Journal of High Performance Computing Applications 18(2), 225–236 (2004) 2. Vuduc, R., Demmel, J.W., Yelick, K.A., Kamil, S., Nishtala, R., Lee, B.: Performance optimizations and bounds for sparse matrix-vector multiply. In: Proceedings of Supercomputing 2002, Baltimore, MD, USA (November 2002) 3. Neh´ez, M.: On Geometrical Properties of Random Tori and Random Graph Models. Journal of Electrical Engineering 51(12/s), 59–62 (2000) 4. Im, E.: Optimizing the Performance of Sparse Matrix-Vector Multiplication. Dissertation thesis, University of Carolina at Berkeley (2001) ˇ 5. Tvrd´ık, P., Simeˇ cek, I.: A new diagonal blocking format and model of cache behavior for sparse matrices. Proceedings of Parallel Processing and Applied Mathematics 12(4), 617–629 (2005)

Petascale Computing for Large-Scale Graph Problems David A. Bader College of Computing Georgia Institute of Technology Atlanta, GA 30332 USA

Abstract. Graph theoretic problems are representative of fundamental kernels in traditional and emerging computational sciences such as chemistry, biology, and medicine, as well as applications in national security. Yet they pose serious challenges for parallel machines due to non-contiguous, concurrent accesses to global data structures with low degrees of locality. Few parallel graph algorithms outperform their best sequential implementation due to long memory latencies and high synchronization costs. In this talk, we consider several graph theoretic kernels for connectivity and centrality and discuss how the features of petascale architectures will affect algorithm development, ease of programming, performance, and scalability.

1

Petascale Computing

Computational science enables us to investigate phenomena where economics or constraints preclude experimentation, evaluate complex models and manage massive data volumes, model processes across interdisciplinary boundaries, and transform business and engineering practices. Increasingly, cyberinfrastructure is required to address our national and global priorities, such as sustainability of our natural environment by reducing our carbon footprint and by decreasing our dependencies on fossil fuels, improving human health and living conditions, understanding the mechanisms of life from molecules and systems to organisms and populations, preventing the spread of disease, predicting and tracking severe weather, recovering from natural and human-caused disasters, maintaining national security, and mastering nanotechnologies. Several of our most fundamental intellectual questions also require computation, such as the formation of the universe, the evolution of life, and the properties of matter. Realizing that cyberinfrastructure is essential to research innovation and competitiveness, several nations are now in a “new arms race to build the world’s mightiest computer” (John Markoff, New York Times, August 19, 2005). These petascale computers, expected around 2008 to 2012, will perform 1015 operations per second, nearly an order of magnitude faster than today’s speediest 

This keynote talk presented joint work with Kamesh Madduri. This work was supported in part by NSF Grants CNS-0614915, CAREER CCF-0611589, and DBI0420513; and DARPA contract NBCH30390004.

R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 166–169, 2008. c Springer-Verlag Berlin Heidelberg 2008 

Petascale Computing for Large-Scale Graph Problems

167

supercomputer. In fact several nations are in a worldwide race to deliver highperformance computing systems that can achieve 10 petaflops or more within the next five years.

2

Massive Graph Theoretic Applications

The modeling and analysis of massive, dynamically evolving semantic networks raises new and challenging research problems to respond quickly to complex queries. Empirical studies on real-world systems such as the Internet, socioeconomic interactions, and biological networks have revealed that they exhibit common structural features – a low graph diameter, skewed vertex degree distribution, self-similarity, and dense sub-graphs. Analogous to the small-world (short paths) phenomenon, these real-world data sets are broadly referred to and modeled as small-world networks [1,2]. Our research highlights the design and implementation of novel high performance computing approaches to efficiently solve advanced small-world network analysis queries, enabling analysis of networks that were previously considered too large to be feasible. For tractable analysis of large-scale networks, we present SNAP(Small-world Network Analysis and Partitioning) [3], an open-source graph analysis framework. SNAPis a modular infrastructure that provides an optimized collection of algorithmic building blocks (efficient implementations of key graph-theoretic analytic approaches) to end-users. In prior work, we have designed novel parallel algorithms for several graph problems that run efficiently on shared memory systems. Our implementations of breadth-first graph traversal [4], shortest paths [5,6], spanning tree [7], minimum spanning tree, connected components [8], and other problems achieve impressive parallel speedup for arbitrary, sparse graph instances. We redesign and integrate several of our recent parallel graph algorithms into SNAP, with additional optimizations for social networks. Thus, SNAPprovides a simple and intuitive interface for the network analyst, effectively hiding the parallel programming complexity involved in low-level algorithm design from the user while providing a productive high-performance environment for complex queries. 2.1

Centrality Analysis Queries

One of the fundamental problems in network analysis is to determine the importance (or the centrality) of a given entity in a network. Some of the wellknown metrics for computing centrality are closeness, stress and betweenness [9]. Of these indices, betweenness has been extensively used in recent years for the analysis of social-interaction networks, as well as other large-scale complex networks. Some applications include lethality in biological networks, study of sexual networks and AIDS, organizational behavior, supply chain management processes, as well as identifying key actors in terrorist networks. Betweenness is also used as the primary routine in accepted social network analysis algorithms for clustering and community identification in real-world networks.

168

D.A. Bader

Betweenness is a global centrality metric that is based on shortest-path enumeration. It is compute-intensive with a quadratic time complexity in the number of vertices. We explore high performance computing techniques [10] that exploit the typical small-world graph topology to speed up exact, as well as approximate, centrality computation. We demonstrate the capability to compute betweenness on networks that are three orders of magnitude larger than ones that can be processed by state-of-the-art network analysis packages. In contrast to existing approaches, we use a global topological measure for centrality queries in a large network and also support this metric in SNAP. 2.2

Path-Based Queries

Several common analysis queries can be naturally formulated as path-based problems. For instance, while analyzing a collaboration network, we might be interested in chains of publications or patents relating the work of two researchers. In a social network, relationship attributes are frequently encapsulated in the edge type, and we may want to discover paths formed by the composition of specific relationships (for instance, subordinate of and friend of ). SNAPsupports common variants of shortest-path and flow-based query formulations. Other related advanced queries include subgraph isomorphism (finding an exact or approximate pattern in the large graph) and connection subgraphs (informally, finding a subgraph relating two entities of interest) that are extensions of simpler path-based algorithms. 2.3

Automated Community Detection

A key problem in social network analysis is that of finding communities, dense components, or detecting other latent structure. In recent work, we designed three new clustering schemes (two hierarchical agglomerative approaches, and one divisive clustering algorithm) [3] that optimize modularity, a popular clustering measure. We also conducted an extensive experimental study and demonstrated that our parallel schemes give significant running time improvements over existing modularity-based clustering heuristics. For instance, our novel divisive clustering approach based on approximate edge betweenness centrality is more than two orders of magnitude faster than the Newman-Girvan algorithm [11] on a multicore computer, while maintaining comparable clustering quality.

3

Summary

The analysis of massive graphs requires petascale computing systems. We discuss the design and implementation of efficient parallel algorithms for novel community structure identification, classical graph-theoretic kernels, topological indices that provide insight into the network structure, and preprocessing kernels for small-world graphs. Our results demonstrate that these parallel approaches are several orders of magnitude faster than competing algorithms – this enables

Petascale Computing for Large-Scale Graph Problems

169

analysis of networks that were previously considered too large to be tractable. As part of ongoing work, we are designing new small-world network analysis kernels and incorporating existing techniques into SNAP. Our current focus is on the design of novel algorithms for massive-scale small-world networks, including better techniques for centrality analysis, path-based queries, spectral partitioning, and community detection.

References 1. Watts, D., Strogatz, S.: Collective dynamics of small world networks. Nature 393, 440–442 (1998) 2. Amaral, L., Scala, A., Barth´el´emy, M., Stanley, H.: Classes of small-world networks. Proceedings of the National Academy of Sciences USA 97, 11149–11152 (2000) 3. Bader, D., Madduri, K.: SNAP, Small-world Network Analysis and Partitioning: an open-source parallel graph framework for the exploration of large-scale networks. In: Proc. Int’l Parallel and Distributed Processing Symp. (IPDPS 2008), Miami, FL (2008) 4. Bader, D., Madduri, K.: Designing multithreaded algorithms for breadth-first search and st-connectivity on the Cray MTA-2. In: Proc. 35th Int’l Conf. on Parallel Processing (ICPP), Columbus, OH, IEEE Computer Society, Los Alamitos (2006) 5. Madduri, K., Bader, D., Berry, J., Crobak, J.: An experimental study of a parallel shortest path algorithm for solving large-scale graph instances. In: Proc. The 9th Workshop on Algorithm Engineering and Experiments (ALENEX 2007), New Orleans, LA (2007) 6. Crobak, J., Berry, J., Madduri, K., Bader, D.: Advanced shortest path algorithms on a massively-multithreaded architecture. In: Proc. Workshop on Multithreaded Architectures and Applications, Long Beach, CA (2007) 7. Bader, D.A., Cong, G.: A fast, parallel spanning tree algorithm for symmetric multiprocessors (SMPs). Journal of Parallel and Distributed Computing 65, 994– 1006 (2005) 8. Bader, D., Cong, G., Feo, J.: On the architectural requirements for efficient execution of graph algorithms. In: Proc. 34th Int’l Conf. on Parallel Processing (ICPP), Oslo, Norway, IEEE Computer Society, Los Alamitos (2005) 9. Freeman, L.: A set of measures of centrality based on betweenness. Sociometry 40, 35–41 (1977) 10. Bader, D., Madduri, K.: Parallel algorithms for evaluating centrality indices in realworld networks. In: Proc. 35th Int’l Conf. on Parallel Processing (ICPP), Columbus, OH, IEEE Computer Society, Columbus, OH, IEEE Computer Society, Los Alamitos (2006) 11. Girvan, M., Newman, M.: Community structure in social and biological networks. Proceedings of the National Academy of Sciences USA 99, 7821–7826 (2002)

The Buffered Work-Pool Approach for Search-Tree Based Optimization Algorithms Faisal N. Abu-Khzam1, , Mohamad A. Rizk1 , Deema A. Abdallah1 , and Nagiza F. Samatova2,3 1

Division of Computer Science and Mathematics Lebanese American University Beirut, Lebanon [email protected] 2 Computer Science and Mathematics Division Oak Ridge National Laboratory Oak Ridge, TN, USA 3 Computer Science Department North Carolina State University Raleigh, NC, USA

Abstract. Recent advances in algorithm design have shown a growing interest in seeking exact solutions to many hard problems. This new trend has been motivated by hardness of approximation results that appeared in the last decade, and has taken a great boost by the emergence of parameterized complexity theory. Exact algorithms often follow the classical search-tree based recursive backtracking strategy. Different algorithms adopt different branching and pruning techniques in order to reduce the unavoidable exponential growth in run time. This paper is concerned with another time-saving approach by developing new methods for exploiting high-performance computational platforms. A load balancing strategy is presented that could exploit multi-core architectures, such as clusters of symmetric multiprocessors. The well-known Maximum Clique problem is used as an exemplar to illustrate the utility of our approach.

1

Introduction and Background

Computationally demanding applications remain the main source of challenge for algorithm designers and for the computer industry. The major reason for 



This research has been supported by the “Exploratory Data Intensive Computing for Complex Biological Systems” project from U.S. Department of Energy (Office of Advanced Scientific Computing Research, Office of Science). The work of N. F. Samatova was also sponsored by the Laboratory Directed Research and Development Program of Oak Ridge National Laboratory. Oak Ridge National Laboratory is managed by UT-Battelle for the LLC U.S. D.O.E. under contract no. DE-AC0500OR22725. Communicating author.

R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 170–179, 2008. c Springer-Verlag Berlin Heidelberg 2008 

The Buffered Work-Pool Approach for Search-Tree

171

this challenge stems from the fact that most real life questions are modeled by computationally intractable problems. Algorithm designers have resorted to approximation methods whenever it is acceptable to trade quality for time. However, some problems are hard to approximate with a guaranteed error bound, unless some strongly believed complexity hierarchy collapses. Another issue with approximation methods is (what we call) two-fold approximations: adopting simplifying assumptions to model a question by a certain problem and adopting an approximate solution to that problem. This kind of double inaccuracy is not desired, especially when a problem is to be solved only once in a while. For example, when data is produced over years (as in some biological experiments), one may afford spending a few days or even weeks to get an accurate answer. Determining whether an accurate solution exists that is affordable requires more scientific efforts and extensive experimental study. With the high expected increase in multiprocessor performance, a natural question to pose is whether we would eventually bridge the gap between intractability and practicality, by solving practical instances of problems that were once judged (and dismissed) as intractable. The hope for a positive question has been intensified recently when a new atmosphere that favors exact algorithms appeared in the algorithms community. This is partially attributed to the emergence of fixed-parameter tractability [1], which led to new methodologies for problem reduction and novel parameter-driven search algorithms whose exponential-time behavior is restricted to some input parameter(s). Coupling the best exact algorithm for a problem with a highly scalable parallel implementation sounds like the right approach to bridging the gap between intractability and practicality. We shall try to realize this vision by providing a special parallel technique for problems that are solved (optimally) via recursive backtracking. We use the Maximum Clique problem as an exemplar.

2

The Maximum Clique Problem

Maximum Clique (henceforth MC) is a well-known NP-complete problem [2]. Its decision version is often formulated as follows: Input: a graph G = (V, E) and a positive integer k < |V |. Question: is there a subset V  of V with |V  | ≥ k for which every pair of vertices in V  is joined by an edge in E? When posed as an optimization search problem (in which a clique of maximum size is sought), Maximum Clique finds application in a wide variety of domains. In computational biology, for example, powerful Maximum Clique tools are needed in the context of cis regulatory motif finding [3], microarray analysis [4], and the study of quantitative trait loci [5]. Moreover, recent clique finding methods have been incorporated into ClustalXP [6], the parallel version of ClustalW. There are many methods for solving the optimization version of MC. Most of these methods appeared in the context of solving the Independent Set problem [7,8,9], which is equivalent to MC: an independent set in a graph G is a clique

172

F.N. Abu-Khzam et al.

in its complement. Here, the complement of G is the graph representing the complement of the adjacency relation of G. A search-tree based clique algorithm can be viewed as a traversal of a virtual tree whose nodes are search states. Every search state consists of a set CurrentK holding the current clique, a set Clique N eighbors containing common neighbors of the elements of CurrentK, and a few parameters. The largest clique found during the search, say M axK, is kept unchanged along with its size, until a search-tree node is reached whose CurrentK is larger than M axK. We consider a particular recent MC algorithm, due to Tomita and Kameda [10], which showed a great improvement over previously known methods. We refer to this algorithm by TK in what follows. The TK algorithm employs a greedy coloring of the Clique N eighbors vertices at every state of its search. Initially, Clique N eighbors is the whole vertex set. The assigned colors are represented by positive integers in such a way that no two adjacent vertices receive the same color. Note that the maximum clique size in a graph cannot exceed the number of colors assigned by any coloring. The TK coloring method guarantees that the colored vertices are also sorted according to their color values. A vertex of color c must have at least c − 1 adjacent vertices with c − 1 different smaller colors. This has a significant impact on increasing the performance during the search. In fact, if the size of M axK is greater than the sum of |CurrentK| and the maximum assigned color, then the corresponding branch of the search is terminated (or pruned). As for the criteria for selecting a candidate vertex for branching/expansion, the last vertex in the sorted list is considered, being a vertex of maximum color value. We refer to this vertex by Candidate in the sequel. Once Candidate is determined, Clique N eighbors is updated to become the intersection between itself and the neighbors of Candidate. If the intersection set is not empty, the coloring procedure is applied to the new Clique N eighbors set and the branching function is called again. If this intersection is empty and the size of the new CurrentK is greater than the size of M axK, the latter is then replaced by CurrentK. Otherwise, the vertex is removed from the clique and another one is to be chosen. If no other candidate is found, the search backtracks.

3

The Buffered Workpool Approach

The Buffered Work-Pool approach (henceforth BWP) is a hybrid dynamic load balancing technique that combines threading and message passing between the different processors of any cluster. This is a master-worker technique that is similar to decentralized work-pool (see chapter 7 of [11]). The main objectives are to allow workers to manage their own local shared work-pools (or task-buffers), from/to which the different local threads can exchange tasks, and to let them communicate tasks through the master. The master acts like a grid middleware, whose role is to reduce the overhead of communication and synchronization. Relying on a master process in the inter-processor communication facilitates

The Buffered Work-Pool Approach for Search-Tree

173

Fig. 1. The General BWP Approach

termination detection. Figure 1 below illustrates the main functions of a general BWP computation. We describe these functions in the following subsections. We use a BWP system in a computation whenever a single task has the potential of producing a large number of other tasks. Such is the case in searchtree based exact algorithms. To explicate, note that each search-tree node (or search-state) represents a complete problem instance that can be viewed as a task by itself. All the state-related information could be encoded in the structure of its corresponding task. In the following description of the general BWP approach, and for the sake of illustration, we assume a search-tree based computation. 3.1

The Master

The master starts the computation with any problem-specific pre-processing steps. It then fills its task-buffer with a set of initial tasks and distributes them equally among the workers (in a sequential manner). Afterwards, the master’s entire task evolves around communication with the workers. If it receives a request for tasks from a worker, it answers by allotting a number of tasks to be exchanged after a master-worker agreement is settled. Similarly, if the worker sends a request for sharing part of its load with other workers, the master tries to accommodate part of that worker’s task-buffer in order to delegate it later to other workers that are requesting tasks.

174

3.2

F.N. Abu-Khzam et al.

The Worker

After receiving the initial set of tasks, each worker sets off one thread to handle the communication with the master and initiates a pre-defined number of search threads that are in charge of expanding/branching the tasks from its task-buffer. The addition of tasks to the task-buffer takes place when all of the following conditions are satisfied: – The number of sub-tasks that are generated from the task being expanded is greater than a predefined parameter called Add Threshold. – The current search tree level (or, in general, a computation stage) is a multiple of a user-defined parameter Level Threshold. This allows the user to determine how coarse/fine (s)he wants the computation to be. – The worker’s buffer is not full and not locked by another thread. If any of the above mentioned conditions fails, the search thread expands its tasks recursively (it proceeds as in the sequential version). 3.3

Master/Worker Communication

During the master-worker communication, we distinguish various types of messages whose role is identified by special message tags. Some messages are problem specific (such as updating the maximum clique size), while others are used to exchange tasks and to settle agreements. An agreement is needed to determine the number of tasks to be exchanged. The worker can issue two types of requests. A Tasks Request is used to request tasks when the number of tasks in its buffer drops below the pre-defined parameter Worker Starving Level. The master sends back the current number of tasks in its buffer. If the master’s answer is greater than zero, then the worker responds back with the number of empty slots below its buffer’s Hungry Level. A Delegation Request is used by the worker to delegate some tasks when its buffer is full. In this case, the master sends back the number of empty slots in its task-buffer. If the master’s answer is positive, the worker sends the number of tasks that are above its Hungry Level. If the worker’s task-buffer was no longer full when it receives the master’s response, it sends back a zero answer. The master sends a Normal Request or an Urgent Request message when the number of tasks in its buffer drops below the normal (hungry) or urgent (starving) level, respectively. In case of a Normal Request, the worker’s reply message has a Normal Request Answer tag and contains the number of tasks in its buffer that are above its Worker Starving Level. An Urgent Request Answer tag is used in response to an Urgent Request message from the master, in which case the worker sends a pre-defined percentage of the number of tasks in its buffer. The master’s requests are sent only to those workers that have tasks and have answered previous requests. Similarly, the worker’s communication thread does not send a Task Request message unless the master has available tasks and has previously answered the worker’s requests.

The Buffered Work-Pool Approach for Search-Tree

175

The communication algorithms are shown below. (Note that New Message() is true if the messages buffer has a new message.) Master-Communication-Function Begin While (Terminate==FALSE) if(number of tasks ≤ Master Hungry Level) for each worker do if(worker is not starving and has answered previous requests) ISend(Normal Request, worker, number of tasks); if(number of tasks ≤ Master Starving Level) for each worker do if(worker is not starving and has answered previous requests) ISend(Urgent Request, worker, number of tasks); if(New Message()) if(new message tag == Normal Request Answer) Receive Normal Request Answer(); if(new message tag == Urgent Request Answer) Receive Urgent Request Answer(); if(new message tag == Tasks Request) Send Tasks Request Answer(); if(new message tag == Delegation Request) Send Delegation Request Answer(); End

Worker-Communication Function Begin While (Terminate==FALSE) if (number of tasks ≤ Worker Starving Level) if(master has tasks and has answered previous requests) ISend(Tasks Request, master, number of tasks); if (task-buffer is full) if (master has answered previous requests) ISend(Delegation Request, master, number of tasks); if(New Message()) if(new message tag==Normal Request) Send Normal Request Answer(); if(new message tag==Urgent Request Answer) Send Urgent Request Answer(); if(new message tag==Tasks Request Answer) Receive Tasks Request Answer(); if(new message tag==Delegation Request Answer) Receive Delegation Request Answer(); if(new message tag==Termination) Terminate = TRUE End

176

F.N. Abu-Khzam et al.

Note that the master and the workers always send the number of tasks they own, even if no tasks are to be exchanged. This is used, in the above protocol, as an estimate of the ability of the worker/master to delegate tasks. Blocking messages are used when the master and a worker are settling on an agreement or exchanging tasks1 . However, negative answers can use non-blocking send routines, because they will not be followed by exchange of tasks. 3.4

Termination Detection

The master detects termination and broadcasts a termination signal to all workers only when the following conditions are satisfied. – All workers and the master have zero tasks in their local task buffers; – No worker has an active search thread; – No messages are in transmission. When a worker has an empty task-buffer and receives an Urgent Request message with a zero value (i.e. the number of tasks in the master’s buffer is zero), it does any of the following: – If all of its search threads are idle and the master has answered all its previous Tasks Request messages, it sends back a negative value for its number of tasks, signaling that it is ready to terminate. – If any of its search threads is active, or it has an unanswered Tasks Request message, the worker replies with a zero number of tasks. On the other hand, when the master has an empty task-buffer and receives a negative value in all the Urgent Request Answer messages, it broadcasts a termination signal. 3.5

Avoiding Deadlocks

While settling an agreement with a certain worker X, the master uses blocking send/receive messages. To avoid having a deadlock, the master probes its message buffer, looking for any blocking message from X. This takes place after a blocking send message and before its corresponding receive message. If a blocking message from X arrives whose tag is different from the master’s blocking send, then a deadlock may occur. In this case, the master serves the worker’s message before it goes back into its blocking receive state.

4

A Buffered WorkPool Algorithm for Maximum Clique

In the BWP version of the TK algorithm, the computation process begins at the master with the pre-processing steps that include coloring and sorting of vertices. 1

Note that non-blocking send/receive messages allow the sender/receiver to proceed in its computation instead of sitting idle waiting for a respond to their message.

The Buffered Work-Pool Approach for Search-Tree

177

The master starts the search phase and creates a number of tasks by imitating the sequential branching in a breadth-first manner until it has a number of visited search-tree nodes that is equal to the number of processors available. Every task would have its own CurrentK clique. Each worker, consequently, initiates a communication thread and a user-predefined number of search threads. 4.1

The Task Structure

The task structure is made up of a linear array called task. The size of task is n + 5, where n is the number of vertices in the graph. The first n slots contain integer values representing the states of the vertices. If a vertex v is active (a member of Candidate), task[v] is the index of v in the sorted list. Otherwise, task[v] is either −1 or −2 depending on whether v is in CurrentK or just deleted. The remaining slots of task contain the candidate vertex and its color, the size of CurrentK, the number of vertices and edges in the subgraph induced by the Candidate set. 4.2

Parallel Branching

When the branching function is called on a task chosen by its corresponding search thread, it starts by checking whether the maximum clique size is smaller than the sum of its candidate vertex color and its CurrentK size. If so, the candidate vertex is added to CurrentK whose size is thus incremented by one. Then the intersection set between the neighbors of Candidate and the current Clique N eighbors set is determined in order to extract a new set of active neighbors of the clique. As in the sequential algorithm, if the intersection is empty and |CurrentK| > M axK, then CurrentK is sent to the master in order to replace M axK, and the master broadcasts |CurrentK| as the new maximum clique size found so far. If |CurrentK| is not greater than M axK, then Candidate is removed from the clique and another one is chosen for a similar procedure. On the other hand, if the intersection set is non-empty, the greedy coloring is re-applied to Clique N eighbors to produce a new ordering by color, as in the sequential version. A new list of tasks is produced in the same breadth-first manner used by the master. The current thread will proceed in handling the branching as in the sequential version, unless all the conditions for adding tasks hold (see section 3.2), in which case the newly generated tasks are added to the local task-buffer.

5

Preliminary Experimental Results

We have implemented the sequential TK algorithm, together with our parallel version. The two versions were tested on graphs from the DIMACS Benchmark (http://dimacs.rutgers.edu/Challenges). For the preliminary experiments we report below, we chose three graphs on which the sequential TK is known to take hours. These graphs are given the names: brock800 1, brock800 4 and p hat10002. Figure 2 shows the average speedups obtained over all the different runs

178

F.N. Abu-Khzam et al.

Fig. 2. Sample Experimental Results

(on the three files) after adjusting the parameters. The experiments were conducted on a cluster of dual-core processors at Oak Ridge National Lab. Each node consists of a Dual Intel 3.4GHz Xeon EM64T processor with 4GB of memory and dual Gigabit Ethernet Interconnects. As for the number of threads per machine, we tried different values and we found that using five threads per machine gave the best performance on the used dual-core processors. Finally, note that our user-defined parameters play a major role in the efficiency of the produced code. More experiments are needed on randomly generated graphs and known benchmarks in order to determine the relationships between the structure of input graphs (such as density, chromatic number, and degree distribution) and the various parameters.

6

Concluding Remarks

We presented a dynamic load balancing technique that targets clusters of sharedmemory multiprocessors. The Buffered Work-Pool approach is suitable for highly demanding computations where each task assigned to a process has the potential of producing a large number of sub-tasks. Experiments were presented that showed work in progress. Further experiments are needed to assess the BWP approach on large platforms and on other problems. The BWP method applies well to many other combinatorial problems. We are currently using it in designing algorithms for many classical problems like SAT, Vertex Cover and Maximal Cliques enumeration.

References 1. Downey, R.G., Fellows, M.R.: Parameterized Complexity. Springer, Heidelberg (1999) 2. Garey, M.R., Johnson, D.S.: Computers and Intractability. W.H. Freeman, New York (1979)

The Buffered Work-Pool Approach for Search-Tree

179

3. Leung, H., Chin, F.: An Efficient algorithm for String Motif Discovery. In: Proceedings, Asia-Pacific Bioinformatics Conference (APBC), pp. 79–88 (2006) 4. Ji, Y., Stormo, G.: Clustering binary fingerprint vectors with missing values for DNA array data analysis. Journal of Computational Biology 11(5), 887–901 (2004) 5. Chesler, E., Lu, L., Shou, S., Qu, Y., Gu, J., Wang, J., Hsu, H., Mountz, J., Baldwin, N., Langston, M., Hogenesch, J., Threadgill, D., Manly, K., Williams, R.: Complex Trait Analysis of Gene Expression Uncovers Polygenic and Pleiotropic Networks that Modulate Nervous System Function. Nature Genetics 37(3), 233– 242 (2005) 6. Abu-Khzam, F.N., Cheetham, J., Dehne, F., Langston, M.A., Pitre, S., RauChaplin, A., Shanbhag, P., Taillon, P.J.: ClustalXP, http://134.117.206.42:8000 7. Fomin, F.V., Grandoni, F., Kratsch, D.: Measure and conquer: A Simple O(20.288n ) Independent Set Algorithm. In: SODA 2006: Proceedings of the Seventeenth Annual ACM-SIAM Symposium on Discrete Algorithms, Miami, Florida, pp. 18–25 (2004) 8. Jian, T.: An O(20.304n ) Algorithm for Solving Maximum Independent Set Problems. IEEE Transactions on Computers 35(9), 847–851 (1986) 9. Robson, J.M.: Finding a maximum independent set in time O(2n/4 ). Universite Bordeaux I, LaBRI, Techincal Report 1251-01 (2001) 10. Tomita, E., Kameda, T.: An efficient branch-and-bound algorithm for finding a maximum clique with computational experiments. Journal of Global Optimization 37, 95–111 (2007) 11. Wilkinson, B., Allen, M.: Parallel Programming: Techniques and Applications Using Networked Workstations and Parallel Computers, 2nd edn. Prentice-Hall, Inc., Upper Saddle River (2004)

Parallel Scatter Search Algorithm for the Flow Shop Sequencing Problem Wojciech Bo˙zejko1 and Mieczyslaw Wodecki2 1

Institute of Computer Engineering, Control and Robotics, Wroclaw University of Technology Janiszewskiego 11-17, 50-372 Wroclaw, Poland [email protected] 2 Institute of Computer Science, University of Wroclaw Joliot-Curie 15, 50-383 Wroclaw, Poland [email protected]

Abstract. In the paper we consider strongly NP-hard flow shop problem with the criterion of minimization of the sum of job’s finishing times. We present the parallel algorithm based on the scatter search method. Obtained results are compared to the best known from the literature. Superlinear speedup has been observed in the parallel calculations. Keywords: metaheuristics, scatter search, flow shop problem.

1

Introduction

We take into consideration the permutation flow shop scheduling problem described as follows. A number of jobs are to be processed on a number of machines. Each job must go through all the machines in exactly the same order and the job order must be the same on every machine. Each machine can process at most one job at any point of time and each job may be processed on at most one machine at any time. The objective is to find a schedule that minimizes the sum of job’s completion times. The problem is indicated by F ||Csum . There are plenty of good heuristic algorithms for solving F ||Cmax flow shop problem, with the objective of minimizing maximal job’s completion times. For the sake of special properties (blocks of critical path, [4]) it is recognized as an easier one than a problem with objective Csum . Unfortunately, there are not any similar properties (which can speedup computations) for the F ||Csum flow shop problem. Constructive algorithms (LIT and SPD from [11], NSPD [7]) have low efficiency and can only be applied to a limited range. There is hybrid algorithm in [9], consisting of elements of tabu search, simulated annealing and path relinking methods. The results of this algorithm, applied to Taillard benchmark tests [10], are the best known ones in the literature nowadays. The big disadvantage of the algorithm is its time-consumption. Parallel computing is the way to speed it up. This work is the continuation of author’s research on constructing efficient parallel algorithms to solve hard combinatorial problems ([1,2,3,12]). Further, we present a parallel algorithm based on scatter search method which not only speeds up the computations, but also improves the quality of the results. R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 180–188, 2008. c Springer-Verlag Berlin Heidelberg 2008 

Parallel Scatter Search Algorithm for the Flow Shop Sequencing Problem

2

181

Problem Definition and Notation

The flow shop problem can be defined as follows: there is a set of n jobs J={1,2,. . . ,n} and a set of m machines M ={1,2,. . . ,m}. Job j ∈ J consists of a sequence of m operations Oj1 , Oj2 ,. . . ,Ojm . Operation Ojk corresponds to the processing of job j on machine k during an uninterrupted processing time pjk . We want to find a schedule so that a sum of job’s completion times is minimal. Let π =(π(1), π(1),. . . ,π(n)) be a permutation of jobs {1,2,. . . ,n} and let Π be the set of all permutations. Each permutation π ∈ Π defines a processing order of jobs on each machine. We wish to find a permutation π ∗ ∈ Π that: Csum (π ∗ ) = min Csum (π), where Csum (π) = π∈Π

n 

Ci,m (π),

i=1

where Ci,j (π) is the time required to complete job i on the machine j in the processing order given by the permutation π. The completion time of job π(j) on machine k can be found by using the recursive formula: Cπ(j)k = max {Cπ(j−1)k , Cπ(j)k−1 } + pπ(j)k , where π(0)=0, C0k =0, k=1,2,...,m , Cj0 =0, j =1,2,...,n. Such a problem belongs to the strongly NP-hard class.

3

Scatter Search Method

The main idea of the scatter search method is presented in [6]. The algorithm is based on the idea of evaluation of the so-called starting solutions set. In the classic version a linear combination of the starting solution is used to construct a new solution. In case of a permutational representation of the solution using linear combination of permutations gives us an object which is not a permutation. Therefore, in this paper a path relinking procedure is used to construct a path from one solution of the starting set to another solution from this set. The best element of such a path is chosen as a candidate to add to the starting solution set. Algorithm 1. Scatter search for i := 1 to iter do Step 1. Generate a set of unrepeated starting solutions S, |S| = n. Step 2. For randomly chosen n/2 pair from the S apply path relinking procedure to generate a set S  - of n/2 solutions which lies on paths. Step 3. Apply local search procedure to improve value of the cost function of solutions from the set S  .

182

W. Bo˙zejko and M. Wodecki

Step 4. Add solutions from the set S  to the set S. Leave in the set S at most n solutions by deleting the worst and repeated solutions. Step 5. if |S| < n then Add new random solutions to the set S such, that elements in the set S does not duplicate and |S| = n. end for.

4

Path Relinking

The base of the path relinking procedure, which connects two solutions π1 , π2 ∈ Π, is a multi-step crossover fusion (MSXF) described by Reeves and Yamada [9]. Its idea is based on a stochastic local search, starting from π1 solution, to find a new good solution where the other solution π1 is used as a reference point. The neighborhood N (π) of the permutation (individual) π is defined as a set of new permutations that can be achieved from π by exactly one adjacent pairwise exchange operator which exchanges the positions of two adjacent jobs of a problem’s solution connected with permutation π. The distance measure d(π,σ) is defined as a number of adjacent pairwise exchanges needed to transform permutation π into permutation σ. Such a measure is known as Kendall’s τ measure. Algorithm 2. Path-relinking procedure Let π 1 , π 2 be reference solutions. Set x = q = π1 ; repeat For each member yi ∈ N (π), calculate d(yi , π 2 ); Sort yi ∈ N (π) in ascending order of d(yi , π 2 ); repeat Select yi from N (π) with a probability inversely proportional to the index i; Calculate Csum (yi ); Accept yi with probability 1 if Csum (yi ) ≤ Csum (x), and with probability PT (yi ) = exp((Csum (x) − Csum (yi )) / T ) otherwise (T is temperature); Change the index of yi from i to n and the indices of yk , k = i+1,...,n from k to k−1; until yi is accepted; x ← yi ; if Csum (x) < Csum (q) then q ← x; until some termination condition is satisfied ; return q { q is the best solutions lying on the path from π1 to π2 } The condition of termination consisted in exceeding 100 iterations by the path relinking procedure.

Parallel Scatter Search Algorithm for the Flow Shop Sequencing Problem

5

183

Parallel Scatter Search Algorithm

The parallel algorithm was projected to execute on the cluster of 152 dual-core Intel Xeon 2.4 GHz processors connected by Gigabit Ethernet with 3Com SuperStack 3870 swiches installed in the Wrocaw Center of Networking and Supercomputing. This supercomputer has got a distributed memory, where each processor has its local 4 GB memory. Taking into consideration this type of architecture we choose a client-server model for the scatter search algorithm proposed here, where calculations of path-relinking procedures are executed by processors on local data and communication takes place rarely to create a common set of new starting solutions. The process of communication and evaluation of the starting solutions set S is controlled by processor number 0. We call this model global. For comparison a model without communication was also implemented in which an independent scatter search threads are executed in parallel. The result of such an algorithm is the best solution from solutions generated by all the searching threads. We call this model independent. Algorithms were implemented in C++ language using MPI (mpich 1.2.7) library and executed under the OpenPBS batching system which measures times of processor’s usage. Algorithm 3. Parallel scatter search algorithm for the SIMD model without shared memory parfor p := 1 to number of processors do for i := 1 to iter do Step 1. if (p = 0) then {only procesor number 0} Generate a set of unrepeated starting solutions S, |S| = n. Broadcast a set S among all the processors. else {other processors} Receive from the procesor 0 a set of starting solutions S. end if; Step 2. For randomly chosen n/2 pair from the S apply path relinking procedure to generate a set S  - of n/2 solutions which lies on paths. Step 3. Apply local search procedure to improve value of the cost function of solutions from the set S  . Step 4. if (p = 0) then Send solutions from the set S  to procesor 0 else {only processor number 0} Receive sets S  from other processors and add its elements to the set S Step 5. Leave in the set S at most n solutions by deleting the worst and repeated solutions.

184

W. Bo˙zejko and M. Wodecki

if |S| < n then Add a new random solutions to the set S such, that elements in the set S does not duplicate and |S| = n. end if; end if; end for; end parfor.

6

Computer Simulations

Tests were based on 50 instances with 100,. . . ,500 operations (n × m=20×5, 20×10, 20×20, 50×5, 50×10) due to Taillard [10], taken from the OR-Library [8]. The results were compared to the best known, taken from [9]. For each version of the scatter search algorithm (global or independent), following metrics were calculated: – ARPD - Average Percentage Relative Deviation to the benchmark’s cost function value from [9], – ttotal (in seconds) – real time of executing the algorithm for 50 benchmark instances from [10], – tcpu (in seconds) – the sum of time’s consuming on all processors for 50 benchmark instances from [10]. Table 1. Average percentage deviations ARPD (independent model - no communication). The sum of iterations number for all processors is 1600. n×m 20x5 20x10 20x20 50x5 50x10 average

1 (iter =1600) 0.007 0.000 0.000 1.024 1.060 0.418

Processors 2 (iter = 800) 4 (iter = 400) 0.021 0.065 0.012 0.010 0.013 0.047 1.093 1.364 1.312 1.425 0.490 0.582

8 (iter = 200) 0.111 0.024 0.046 1.662 1.821 0.733

Tables 1, 2, 3 and 4 presents results of computations of the scatter search method for the number of iterations (as a sum of iterations on all the processors) equals to 1600. The cost of computations, understanding as a sum of time-consuming an all the processors, is about 7 hours for the all 50 benchmark instances of the flow shop problem (Table 2, 4). The best results (average percentage deviations to the best known solutions) has the 2-processors version of the global model of the scatter search algorithm (with communication), see Figure 1. Because the timeconsuming on all the processors is a little bit longer than the time of the sequential version we can say that the speedup of this version of the algorithm if almost-linear

Parallel Scatter Search Algorithm for the Flow Shop Sequencing Problem

185

Table 2. Times of execution (for all 50 instances, independent model, iter = 1600) Cluster of Xeon 3000 2.4 GHz processors ttotal(hours:min:sec) tcpu (hours:min:sec) 7:13:30 7:13:13 3:34:08 7:04:44 1:46:05 6:58:43 0:53:33 6:57:44

Processors 1 2 4 8

Table 3. Average percentage deviations ARPD (global model - with communication). The sum of iterations number for all processors is 1600. n×m 20x5 20x10 20x20 50x5 50x10 average

1 (iter =1600) 0.21 0.037 0.008 0.917 1.171 0.431

Processors 2 (iter = 800) 4 (iter = 400) 0.020 0.007 0.006 0.004 0.000 0.004 0.762 0.978 0.860 1.126 0.330 0.423

8 (iter = 200) 0.077 0.013 0.015 1.208 1.448 0.552

Table 4. Times of execution (for all 50 instances, global model, iter = 1600) Cluster of Xeon 3000 2.4 GHz processors ttotal(hours:min:sec) tcpu (hours:min:sec) 7:26:00 7:25:51 3:52:36 7:17:39 2:14:02 7:04:07 1:24:31 7:06:52

Processors 1 2 4 8

Table 5. Average percentage deviations ARPD (independent model - no communication). The sum of iterations’s number for all processors is 16000. n×m 20x5 20x10 20x20 50x5 50x10 average

1 (iter =16000) 0.000 0.000 0.000 0.904 0.913 0.363

Processors 2 (iter = 8000) 4 (iter = 4000) 0.007 0.000 0.000 0.000 0.000 0.000 1.037 0.906 0.986 1.033 0.406 0.388

8 (iter = 2000) 0.006 0.000 0.000 0.903 0.989 0.380

(or even super-linear, because sequential algorithms, for both: independent and global models, have worse results of the ARPD). The situation is more clear for the number of iterations equals to 16000, Tables 5, 6, 7 and 8. The cost of computations, a sum of time-consuming an all

186

W. Bo˙zejko and M. Wodecki

Table 6. Times of execution (for all 50 instances, independent model, iter = 16000) Cluster of Xeon 3000 2.4 GHz processors ttotal(hours:min:sec) tcpu (hours:min:sec) 75:27:40 75:25:48 37:40:08 75:02:51 18:38:23 74:10:18 9:06:24 72:19:26

Processors 1 2 4 8

Table 7. Average percentage deviations ARPD (global model - with communication). The sum of iterations’s number for all processors is 16000. n×m 20x5 20x10 20x20 50x5 50x10 average

1 (iter =16000) 0.000 0.000 0.000 0.993 1.103 0.419

Processors 2 (iter = 8000) 4 (iter = 4000) 0.000 0.000 0.000 0.000 0.000 0.000 0.677 0.537 0.648 0.474 0.265 0.202

8 (iter = 2000) 0.008 0.004 0.000 0.449 0.404 0.173

Table 8. Times of execution (for all 50 instances, global model, iter = 16000) Processors 1 2 4 8

Cluster of Xeon 3000 2.4 GHz processors ttotal(hours:min:sec) tcpu (hours:min:sec) 75:23:43 75:20:42 41:19:51 77:57:57 23:28:19 75:46:07 14:30:03 74:38:51

the processors, is about 75 hours for the all 50 benchmark instances of the flow shop problem (Table 6, 8). The best results are achieved for the 8-processors version of the global model version of scatter search and they are 58.6% better than the results of sequential global scatter search algorithm, and 52.3% better than the results of sequential independent model of scatter search algorithm (see Figure 2). The time-consuming on all 8 processors is shorter than the time of the both sequential version. We can say that the speedup of 8-processors global version of the scatter search algorithm is superlinear: better results are achieved with the lower cost of computations. This anomaly can be understood as the situation where the sequential algorithm executes its search threads such that there is a possibility to chose a better path of the solutions space trespass, which the parallel algorithm do. As we can say in Table 5 such a situations takes place only for the global model of the scatter search algorithms – independent searches are not so effective. The advantage of the global model of calculations over the independent searches is specially visible for the large instance of the flow shop problem - for

Parallel Scatter Search Algorithm for the Flow Shop Sequencing Problem

1

2

0,552

0,423

0,582

0,733

global model

0,33

0,49

0,431

1 0,9 0,8 0,7 0,6 0,5 0,4 0,3 0,2 0,1 0

0,418

ARPD

independent model

187

4

8

number of processors

Fig. 1. Average percentage deviations ARPD of the global and independent scatter search algorithms for the number of iterations iter = 1600 for all 50 instances from OR-Library [8]

1

2

global model

4

0,173

0,202

0,38

0,388

0,265

0,406

0,419

1 0,9 0,8 0,7 0,6 0,5 0,4 0,3 0,2 0,1 0

0,363

ARPD

independent model

8

number of processors

Fig. 2. Average percentage deviations ARPD of the global and independent scatter search algorithms for the number of iterations iter = 16000 for all 50 instances from OR-Library [8]

n = 50, m = 5, 10. The ARPD is about 50% better for the 8-processors implementation comparing to 1-processor version, for the same number of iterations calculated as a sum of iterations executed on all processors.

188

7

W. Bo˙zejko and M. Wodecki

Conclusions

We have discussed a new approach to the permutation flow shop scheduling based on parallel scatter search algorithm. The advantage is especially visible for large problems. As compared to the sequential algorithm, parallelization increases the quality of obtained solutions keeping comparable costs of computations.

Acknowledgements The calculations were done in the Wroclaw Centre of Networking and Supercomputing.

References 1. Bo˙zejko, W., Wodecki, M.: Solving the flow shop problem by parallel tabu search. In: Proceedings of PARELEC 2004, pp. 189–194. IEEE Computer Society, Los Alamitos (2004) 2. Bo˙zejko, W., Wodecki, M.: Parallel genetic algorithm for the flow shop scheduling problem. In: Wyrzykowski, R., Dongarra, J., Paprzycki, M., Wa´sniewski, J. (eds.) PPAM 2004. LNCS, vol. 3019, pp. 566–571. Springer, Heidelberg (2004) 3. Bo˙zejko, W., Wodecki, M.: A fast parallel dynasearch algorithm for some scheduling problems. In: Proceedings of PARELEC 2006, pp. 275–280. IEEE Computer Society, Los Alamitos (2006) 4. Grabowski, J., Pempera, J.: New block properties for the permutation flow-shop problem with application in TS. Journal of Operational Research Society 52, 210– 220 (2001) 5. Holland, J.H.: Adaptation in natural and artificial systems: An introductory analysis with applications to biology, control, and artificial intelligence. University of Michigan Press (1975) 6. James, T., Rego, C., Glover, F.: Sequential and Parallel Path-Relinking Algorithms for the Quadratic Assignment Problem. IEEE Intelligent Systems 20(4), 58–65 (2005) 7. Liu, J.: A new heuristic algorithm for csum flowshop scheduling problems, Personal Communication (1997) 8. OR-Library: http://people.brunel.ac.uk/∼ mastjjb/jeb/info.html 9. Reeves, C.R., Yamada, T.: Solving the Csum Permutation Flowshop Scheduling Problem by Genetic Local Search. In: IEEE International Conference on Evolutionary Computation, pp. 230–234 (1998) 10. Taillard, E.: Benchmarks for basic scheduling problems. European Journal of Operational Research 64, 278–285 (1993) 11. Wang, C., Chu, C.: Proth J., Heuristic approaches for n/m/F/ΣCi scheduling problems. European Journal of Operational Research, 636–644 (1997) 12. Wodecki, M., Bo˙zejko, W.: Solving the flow shop problem by parallel simulated annealing. In: Wyrzykowski, R., Dongarra, J., Paprzycki, M., Wa´sniewski, J. (eds.) PPAM 2001. LNCS, vol. 2328, pp. 236–247. Springer, Heidelberg (2002)

Theoretical and Practical Issues of Parallel Simulated Annealing Agnieszka Debudaj-Grabysz1 and Zbigniew J. Czech1,2 1

Silesia University of Technology, Institute of Computer Science Akademicka 16, 44-100 Gliwice, Poland {agrabysz,zczech}@polsl.pl 2 Silesia University, Sosnowiec, Poland

Abstract. Several parallel simulated annealing algorithms with different co-operation schemes are considered. The theoretical analysis of speedups of the algorithms is presented. The outcome of the theoretical analysis was verified by practical experiments whose aim was to investigate the influence of the co-operation of parallel simulated annealing processes on the quality of results. The experiments were performed assuming a constant cost of parallel computations, i.e., searching for solutions was conducted with a given number of processors for a specified period of time. For the experiments a suite of benchmarking tests for the vehicle routing problem with time windows was used. Keywords: Simulated annealing, parallel processing, co-operation of processes, vehicle routing problem with time windows.

1

Introduction

The paper presents several co-operation schemes for parallel algorithms of simulated annealing (SA) which is a popular heuristic method of optimization. The algorithms with periodical and hybrid communication are compared with a no communication algorithm. The distinctive feature of the algorithm with hybrid communication is that it is intended to be executed on clusters of shared-memory nodes (SMP), combining the benefits of both shared and distributed memory systems. The theoretical analysis of speedups of the algorithms is presented. It considers the computational complexity of SA trials under the assumption that the algorithms stop after having performed a specified number of trials. The outcome of the theoretical analysis was verified by practical experiments whose aim was to investigate the influence of the co-operation of parallel SA processes on the quality of results. The experiments were performed assuming a constant cost of parallel computations, i.e. searching for solutions was conducted with a given number of processors for a specified period of time. Such an approach guarantees linear speedup which might be desirable in practical applications. For the experiments a suite of benchmarking tests for the vehicle routing problem with time windows (VRPTW) was used. R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 189–198, 2008. c Springer-Verlag Berlin Heidelberg 2008 

190

A. Debudaj-Grabysz and Z.J. Czech

Simulated annealing is a heuristic method of optimization employed in the cases when the solution space is too large to be exhaustively explored within a reasonable amount of time. The VRPTW is an example of such a problem. Other examples are: school bus routing, newspaper and mail distribution, delivery of goods to department stores etc. The optimization of routing lowers distribution costs, whereas parallelization makes possible to find better routes if the computation time is limited. The SA bibliography focuses mainly on the sequential versions of the algorithm [2,15]. However parallel versions are also investigated, since the sequential method is considered to converge slowly as compared with other heuristics [16]. In [1,3,11,12,13] some recommendations for parallelization of SA are given. The VRPTW, formulated by Solomon [14] who also proposed a suite of benchmarking tests, has a rich bibliography [16]. The research whose results are presented in this paper is a continuation of our efforts described in [8,9,10,5,6], where parallel SA algorithms to solve the VRPTW are discussed. The VRPTW is a bicriterion optimization problem. Solving it consists in finding a solution with minimum number of routes (first optimization criterion) and then minimizing the total travel distance travelled by vehicles (second optimization criterion). This paper focuses on the first optimization goal, i.e. on minimizing the number of solution routes. Section 2 presents the theoretical foundation of sequential and parallel SA algorithms. Section 3 describes a few variants of the co-operation of processes in parallel simulated annealing. Section 4 is devoted to the theoretical analysis of the algorithms. The results of the experiments are described in section 5. The last section contains conclusions.

2

Sequential and Parallel Simulated Annealing

Simulated annealing searches for the optimal state, i.e. the state which minimizes (or maximizes) the cost function. This is achieved by comparing the current solution with a random solution taken from a specific neighborhood. If a neighbor solution has lower cost than the current solution then it is accepted. Worse solutions can also be accepted with some probability, what prevents the algorithm from being prematurely stuck in local optima. The probability of accepting a worse solution decreases over the process of annealing, and it is governed by a control parameter called temperature. An outline of the SA algorithm is presented in Figure 1. A single execution of the innermost loop is called a trial. The final solution which is returned is the best one ever found. Simulated annealing can be modelled in terms of Markov chains. Namely, the algorithm is considered to generate a sequence of Markov chains, where each chain consists of trials executed in the same temperature. Since in SA each new state is potentially a modification of the previous state, the process is often considered as inherently sequential. However, a few strategies to develop parallel SA were proposed. In our implementations the trials are executed in parallel by a number of processors. We assumed that the length of

Theoretical and Practical Issues of Parallel Simulated Annealing

01 02 03 04 05 06 07 08 09 10 11 12

191

S ← GetInitialSolution(); T ← InitialTemperature; for i ← 1 to NumberOfTemperatureReduction do for j ← 1 to ChainLength do S  ← GetSolutionFromNeighborhood(); ΔC ← CostFunction(S  ) − CostFunction(S); if (ΔC < 0 or AcceptWithProbabilityP(ΔC, T )) then S ← S; {the trial is accepted} end if; end for; T ← λT ; {with λ < 1} end for; Fig. 1. Simulated annealing algorithm

Markov chains was fixed in such a way that the number of trials executed by all processors within each chain is the same as in the sequential algorithm.

3

Co-operation of Processes in Parallel Simulated Annealing

No communication algorithm (NC). The main assumptions for the no communication algorithm were formulated in [2], where the division algorithm was proposed. The method uses all available processors to run many sequential algorithms, where the original chain is split into sub-chains of ChainLength (see Figure 1) divided by the number of processors. At the end, the best solution found is selected as the final one. The use of a large number of processors can result in excessive shortening of the sub-chains length what in turn may negatively affect the quality of results. Periodical communication algorithm (PC). The idea of periodically interacting searches was fully developed in [13]. As in the NC algorithm the length of the sub-chains is decreased. Additionally, processes communicate after performing a part of a chain called a segment, and the best solution is selected and mandated for all processes. In the PC algorithm the segment length is defined by the number of temperature drops. As suggested in [13] to prevent search paths from being trapped in local minima areas as a result of communication, the period of the information exchange needs to be carefully selected. Hybrid communication algorithm (HC). The details of the method were described in [9]. The implementation is intended to run on clusters of SMP nodes, so the parallelization is accomplished using two levels. Intensively communicating operations are moved to the inner level where a shared memory environment is used. The remaining, comparatively rare, communication – the outer level – takes place in a distributed memory environment.

192

A. Debudaj-Grabysz and Z.J. Czech

Outer-level parallelization. Each Markov chain of SA optimization is divided into sub-chains. Their length is equal to the length of the original chain divided by the number of sub-chains. The main idea is to assign a separate sub-chain to a cluster node to allow nodes to generate different sub-chains simultaneously. In this way the computational effort of generating a Markov chain is distributed among the available nodes. Having generated the first Markov chain, the generation of the next chains is performed with no communication among nodes. Each node takes the outcome of the last trial of the preceding sub-chain as the starting point for the subsequent sub-chain. At the end, the best solution found is produced as the final one. It is worth mentioning the sub-chains length is shortened by the number of nodes instead of the number of processors. That results in longer sub-chains as compared with the NC and PC algorithms. Inner-level parallelization. Within a node a few threads communicate with each other while building a single sub-chain of the length determined at the outer level. The idea of parallelization consists in dividing the total number of trials in each sub-chain into rather small sets of trials. Each thread performs its part of the set independently of the other threads. Having completed a set, the master thread selects a solution among all solutions which have been accepted and the remaining ones are discarded. The selected solution is made common for all threads and it becomes the starting point for further computation.

4

Theoretical Analysis of Algorithms

The speedups of the algorithms are established through the analysis of the computational complexity of trials. It is assumed that the algorithms stop when a defined number of sub-chains are completed. The results of the analysis are presented in Table 1, where the parameters denote: p — number of processors, L — length of the Markov chain for the sequential algorithm (see Figure 1), ω — period of communication (in the PC algorithm only), λ — intensity of the Poisson’s process of generation of trials, to be identified experimentally, β — coefficient determining the duration of a broadcast type inter-node communication, to be identified experimentally, M — number of Markov chains for the corresponding sequential algorithm, d — size of the set of trials (in the inner level of the HC algorithm), t1 — duration of the sequential part executed on the inner-level of parallelization, to be identified experimentally. The details of the derivation of formulas showed in Table 1 can be found in [7]. Note that the formulas differ only in the denominators. By the definition the denominator is an inverse of the parallel efficiency η −1 . For arbitrarily and experimentally defined parameters the parallel efficiency as a function of the number of processors is presented in Figure 2. In terms of efficiency the NC algorithm seems to be the best. The HC algorithm is the worst although its

Theoretical and Practical Issues of Parallel Simulated Annealing

193

Table 1. Theoretical speedups Algorithm

Speedup 

PC 1+



NC 1+

HC

1 p(p−1)2 2βpλ + Lω (2p−1)Lω

p log2 p

1 p(p−1)2 (2p−1)LM

p βpλ

+ LM log2

1 βpλ p 1+ √d−1 +λt1 + LM log2 d 2d−1

p

efficiency has a striking feature of being relatively independent from the number of processors. One could conclude that only the trivial parallelizations of the NC algorithm are recommended. Nevertheless, analysing the chart the following caveats should not be ignored: – In a real life application a user can demand the linear speedup, or he/she can set a time limit for the execution of the algorithm. In other words, one might need to stop the algorithm after a specified period of execution time. – The way of exploration of the search space is not taken into account. – The influence of the co-operation of processes is not considered.

10

Ș-1

8 6 4 2 0 1

10

100

1000

10000

number of proces s ors P C1

P C10

NC

HC2

HC4

Fig. 2. Parallel efficiency vs. number of processors (PC1, PC10 – the PC algorithm with the period of communication 1 and 10; NC, HC2, HC4 – the NC and HC algorithms with 2 and 4 parallel threads)

194

A. Debudaj-Grabysz and Z.J. Czech

The first note deserves explanation. As assumed at the beginning of this section, the chart concerns executions with no time limits (instead, the number of sub-chains to be generated is defined). In [7] the following theorem is proved: Theorem 1. The deterioration of the solution quality of the parallel algorithm with the time limit is a non-increasing function of the efficiency of the parallel algorithm without the time limit. The deterioration of the solution quality is measured by the difference between the expected solutions obtained by the parallel and sequential algorithms (the result of the sequential algorithm is taken as a reference). The theorem says that lower theoretical efficiency indicates a possibly worse quality of the solution achieved if the time limit is imposed. In the context of the investigated algorithms running with the time limit one may infer from the chart that it is necessary to take into account the number of available processors. More specifically, for a small number of processors one may expect that the HC and NC algorithms shall yield the worst and best results, respectively. On the other hand, for a large number of processors the HC algorithm is expected to show its superiority. We believe, however, that these theoretical conclusions should be verified in practice, what is a subject of the next section.

5

Experimental Results

In the vehicle routing problem with time windows it is assumed that there is a warehouse “centrally” located to the customers. There is a road between each pair of customers and between each customer and the warehouse. The objective is to supply goods to all customers at the minimum cost. A solution with fewer routes (first goal of optimization) is better than a solution with a smaller total distance travelled (second goal of optimization). With each customer as well as with the warehouse a time window is associated. Each customer has its own demand for goods and should be visited only once. Every route must start and terminate at the warehouse and should preserve the maximum vehicle capacity. The sequential algorithm proposed in [4] was the starting point for parallelization. The experiments were carried out on the NEC Xeon EM64T Cluster installed at the High Performance Computing Center in Stuttgart. Additionally, the NEC TX-7 (ccNUMA) system was used. Due to the lack of access to a genuine SMP cluster with 4 CPUs per node, the use of 4 threads per node was emulated (the NEC Xeon EM64T Clusters consisted of 2 CPU nodes). The experiments were conducted on 5 tests from Solomon’s benchmarking suite [14]: R108, R111, RC105, RC106 and RC108. It was shown in [5] that all the tests differ from each other substantially in terms of difficulty. This difficulty was measured by the factor P r1 – the probability that after an execution of the co-operating searches algorithm (CS, see [5] for details) a solution with the

Theoretical and Practical Issues of Parallel Simulated Annealing

195

minimum number of routes is found. In the current research we measured the same factor for algorithms PC, NC and HC. However: – the following numbers of processors: 4, 8, 16, 20, 32, 40, 60, 80, 100, 120 were used instead of 5, 10, 15, 20, – the limit on the execution time instead on the number of performed cooling stages was imposed. It should be also stressed that the goal of parallelization in the PC, NC and HC algorithms is to achieve a high speedup with as small deterioration of solution quality as possible. Table 2. Solomon’s tests ranked by the factor P r1 obtained for the CS algorithm Test

RC106 RC105 R111 R108 RC108

Algorithm CS

PC1

PC10

NC

HC2

HC4

0.21 0.35 0.66 0.94 1.00

0.17 0.32 0.47 0.46 0.96

0.13 0.60 0.30 0.56 0.86

0.07 0.68 0.23 0.82 1.00

0.12 0.71 0.31 0.88 1.00

0.16 0.73 0.48 0.92 1.00

Table 2 shows the results of the experiments for the selected Solomon’s tests ranked by the factor P r1 obtained by executing the CS algorithm. Our results confirm that the tests differ from each other substantially in terms of difficulty. It can be seen that in every case an algorithm with co-operating processes, either the PC or HC, outperforms the NC algorithm. Let us compare the results obtained for growing numbers of processors (Figure 3). For the hardest test RC106 the significant loss of the quality of results is observed as the number of processors increases. The co-operation of processes alleviates this loss, as in every case the results of the NC algorithm are worse as compared to other algorithms. Generally, the results of the PC1 algorithm are the best, although the results of the HC4 algorithm can also be described as satisfactory. Considering the test RC105, the algorithms with periodic communication do not perform well. For every number of processors the NC algorithm generates results of better quality. All versions of the HC algorithm perform better than the versions of the PC. If the number of processors exceeds 40, the HC versions outperform the NC algorithm as well. Note that except of 4 processors the HC4 algorithm produces the results which are the closest to solutions generated by the sequential algorithm (number of processors = 1). The positive influence of the co-operation of processes can be observed for the R111 test. The HC4 algorithm gives the best results and the NC the worst. The former algorithm gives also the best results for the test R108, although the NC algorithm outperforms both versions of the PC algorithm. In case of the test RC108 it is easy to find a solution with the minimum number of routes.

196

A. Debudaj-Grabysz and Z.J. Czech

RC106

no. of solutions

100 80 60 40 20 0 1 P C1

P C10

NC

4

8

HC2

HC4

16

20

32

40

60

80

100

120

60

80

100

120

60

80

100

120

60

80

100

120

60

80

100

120

number of processors

RC105

no. of solutions

100 80 60 40 20 0 1 P C1

P C10

NC

4

8

HC2

HC4

16

20

32

40

number of processors

R111

no. of solutions

100 80 60 40 20 0 1 P C1

P C10

NC

4

8

HC2

HC4

16

20

32

40

number of processors

R108

no. of solutions

100 80 60 40 20 0 1 P C1

P C10

NC

4

8

HC2

HC4

16

20

32

40

number of processors

RC108

no. of solutions

100 80 60 40 20 0 1 P C1

P C10

NC

4

8

HC2

HC4

16

20

32

40

number of processors

Fig. 3. Comparison of quality of results

Using the HC algorithms this can be done even for more than 400 processors (what is not presented in the figure). The quality of results generated by the NC algorithm decreases when the number of processors exceeds 120. Solutions of the PC algorithms lose their quality much earlier.

Theoretical and Practical Issues of Parallel Simulated Annealing

6

197

Conclusions

The theoretical analysis shows that the efficiency of the NC algorithm is the best among the algorithms under consideration. The experiments indicate that the cooperation of processes (not present in the NC algorithm) can improve the quality of solutions. The influence of periodic communication (PC) strongly depends on a test. Although for the test R111 the results of the PC1 algorithm were very good, for the test R108 they were the worst. The periodical co-operation of processes does not always compensate for shorter annealing sub-chains executed by processes. In the HC algorithms the inner level communication enables to extend the length of the sub-chains while preserving the same number of trials executed within a cooling stage. That is why the quality of results of these algorithms are generally better than the results of the NC algorithm. The HC algorithms can be also described as ,,balanced”, since the quality of results is satisfactory for all investigated tests. Summing up, the HC algorithms are the advisable choice for solving the VRPTW by parallel simulated annealing.

Acknowledgement The authors thank Rolf Rabenseifner of the High Performance Computing Center in Stuttgart for his contribution to this work. The research was supported by the HPC-Europa project (contract No RII3-CT-2003-506079), the Minister of Education and Science of Poland grants 3 T11F 004 29 and BK-239/RAu2/2006. Computing time was provided within the framework of the HLRS-NEC cooperation and by the following computing centers: Academic Computer Centre in Gda´ nsk TASK, Academic Computer Centre CYFRONET AGH, Krak´ ow (computing grant 027/2004), Pozna´ n Supercomputing and Networking Center, Interdisciplinary Centre for Mathematical and Computational Modelling, Warsaw University (computing grant G27-9), Wroclaw Centre for Networking and Supercomputing (computing grant 04/97).

References 1. Aarts, E., de Bont, F., Habers, J., van Laarhoven, P.: Parallel implementations of the statistical cooling algorithm. Integration, the VLSI journal, 209–238 (1986) 2. Aarts, E., Korst, J.: Simulated Annealing and Boltzman Machines. John Wiley & Sons, Chichester (1989) 3. Azencott, R. (ed.): Simulated Annealing Parallelization Techniques. John Wiley & Sons, New York (1992) 4. Czarnas, P.: Traveling Salesman Problem With Time Windows. Solution by Simulated Annealing. MSc thesis (in Polish), Uniwersytet Wroclawski, Wroclaw (2001) 5. Czech, Z.J.: Speeding up sequential simulated annealing by parallelization. In: Proc. of the International Symposium on Parallel Computing in Electrical Engineering (PARELEC 2006), Bialystok, pp. 349–356 (2006) 6. Czech, Z.J.: Co-operation of processes in parallel simulated annealing. In: Proc. of the 18th IASTED International Conference on Parallel and Distributed Computing and Systems, Dallas, Texas, pp. 401–406 (2006)

198

A. Debudaj-Grabysz and Z.J. Czech

7. Debudaj-Grabysz, A.: Parallel simulated annealing algorithms. PhD thesis (in Polish), Silesian University of Technology, Gliwice (2007) 8. Debudaj-Grabysz, A., Czech, Z.J.: A concurrent implementation of simulated annealing and its application to the VRPTW optimization problem. In: Juhasz, Z., Kacsuk, P., Kranzlmuller, D. (eds.) Distributed and Parallel Systems. Cluster and Grid Computing. Kluwer International Series in Engineering and Computer Science, vol. 777, pp. 201–209 (2004) 9. Debudaj-Grabysz, A., Rabenseifner, R.: Nesting OpenMP in MPI to implement a hybrid communication method of parallel simulated annealing on a cluster of SMP nodes. In: Di Martino, B., Kranzlm¨ uller, D., Dongarra, J. (eds.) EuroPVM/MPI 2005. LNCS, vol. 3666, pp. 18–27. Springer, Heidelberg (2005) 10. Debudaj-Grabysz, A., Rabenseifner, R.: Load balanced parallel simulated annealing on a cluster of SMP nodes. In: Nagel, W.E., Walter, W.V., Lehner, W. (eds.) Euro-Par 2006. LNCS, vol. 4128, pp. 1075–1084. Springer, Heidelberg (2006) 11. Greening, D.R.: Parallel Simulated Annealing Techniques. Physica D 42, 293–306 (1990) 12. Lee, F.A.: Parallel Simulated Annealing on a Message-Passing Multi-Computer. PhD thesis, Utah State University (1995) 13. Lee, K.–G., Lee, S.–Y.: Synchronous and Asynchronous Parallel Simulated Annealing with Multiple Markov Chains. IEEE Transactions on Parallel and Distributed Systems 7(10), 993–1008 (1996) 14. Solomon, M.: Algorithms for the vehicle routing and scheduling problem with time windows constraints. Operation Research 35, 254–265 (1987), http://w.cba.neu.edu/∼ msolomon/problems.htm 15. Salamon, P., Sibani, P., Frost, R.: Facts, Conjectures and Improvements for Simulated Annealing. SIAM, Philadelphia (2002) 16. Tan, K.C., Lee, L.H., Zhu, Q.L., Ou, K.: Heuristic methods for vehicle routing problem with time windows. In: Artificial Intelligent in Engineering, pp. 281–295. Elsevier, Amsterdam (2001)

Modified R-MVB Tree and BTV Algorithm Used in a Distributed Spatio-temporal Data Warehouse Marcin Gorawski and Michal Gorawski Silesian University of Technology, Institute of Computer Science, Akademicka 16, 44-100 Gliwice, Poland {Marcin.Gorawski, Michal.Gorawski}@polsl.pl

Abstract. Structural and software modifications of MVB-tree (reverse pointers, aggregations) in exchange for higher space consumption enable answering the timestamp and time aggregated queries in a fast and easy way . Software extensions are new algorithms that accelerate query processing. This paper contains a brief description of temporal data and ways of handling them in modified R-MVB tree and presents distributed system in which above-mentioned index was tested along with a load balancing algorithm used in this solution. Keywords: spatio-temporal indexing, R-MVB, load balancing.

1

Introduction

Standard (relational) database systems accelerate query performance basing on one-dimensional indexes from a B-tree family. Those indexes were optimized for one-dimensional processing where objects are identified by a single alphanumerical value. Actual research is concentrated near: (a) spatial (multidimensional) indices, where the key is an alphanumerical values vector and (b) temporal indices, where object parameters are characterized by the keys that differs in time. In the case where one-dimensional indexes are used for spatio-temporal queries, it is very hard to create good query execution plan, because separate columns are described by separate indexes, (selectivity on every column depends not on the query range but only on data). Extensive research of dedicated indexes has been conducted for many years. Thanks to that, many different access methods were discovered. This paper focuses on one of the temporal indexes, namely the MVB-tree (Multi-version B-tree) [1,2,3,4] and its usage in a distributed spatio-temporal telemetric data warehouse also presented in the paper. There were many modifications made to the original MVB-tree algorithm including its connection with an R-tree index. The newly obtained structure is called a modified R-MVB tree. The modifications made to the original MVB-tree were essential for efficient algorithm use in the B-DSTDW(t) system (section 4) that is the main subject of our research. R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 199–208, 2008. c Springer-Verlag Berlin Heidelberg 2008 

200

M. Gorawski and M. Gorawski

The paper structure can be described as follows: in section 2 spatial and temporal data characteristic is given, section 3 describes an MVB-tree and its modifications, section 4 presents a B-DSTDW(t) system, an on-line balancing algorithm named BTV [11] used in this system and a spatio-temporal version of MVB-tree, which is a modified R-MVB tree. In section 5 results of the tests performed in a B-DSTDW(t) system are shown, section 6 summarizes the paper and section 7 gives a quick glance on authors current research.

2

Spatial and Temporal Data Characteristic

Spatial data are multidimensional ones, defined as a number of unique key parts. Data connected to objects with different coordinates do not maintain linear order and are characterized with dominating dissimilarity object schedule. Effective access to large number of spatial data is assured by its indexing. Multidimensional character of spatial data makes use of indexing methods for one-dimensional indexes (for example B-tree family) impossible. In the case of one-dimensional indexes usage obtaining a good plan of query execution in spatio-temporal solutions is extremely hard, when several indexes are placed upon next columns (selectivity by each of the columns depends not only on query range but also on the data itself.) Spatial index is a structure, which groups spatial data in a manner, which speeds up spatial queries execution. After analysis of telemetric data character, decision was made to expand spatial indexing issue by a temporal dimension. This allowed efficient handling not only spatial queries, but also spatio-temporal queries. Such approach of including temporal dimension in Spatial Data Warehouse (SDW) enabled the creation of Spatio-Temporal Data Warehouse (STDW), and with usage of telemetric data and data distribution along several system nodes, the obtained system can be called Distributed (Parallel) Spatio-Temporal Telemetric Data Warehouse (D(P)STDW(t)) [5,6,7]. In the following section the idea of the MVB-tree is presented. It is ideal for servicing temporal data, and its variation, along with modification made, structure is a efficient solution to temporal data indexing problem in the Balanced Distributed Spatio-Temporal Telemetric Data Warehouse (B-DSTDW(t)) system. Its variation, the R-MVB tree is presented as a complex solution for spatio-temporal data servicing.

3

MVB-Tree and Its Modifications

The original MVB-tree algorithm is presented in [2,8]. During original MVB-tree usage certain drawbacks have been observed. Below-mentioned modifications make MVB-tree better suited for our needs presented in section 4. The MVBtree is a modification of a B-tree that was prepared for temporal (multi-version) data. In other words, it is a B-tree prepared for storing temporal data [9]. The main goal was creation of an index that will have the same performance for multiversion data as systems for single-version (that is static) data. We are talking about asymptotic optimal solution. Authors of [2] proved that MVB-tree fulfills

Modified R-MVB Tree and BTV Algorithm

201

those criteria. Looking deeply, the MVB-tree is not a ’tree’ but a hierarchical structure based on a directed list of roots. Figure 1 shows outline, which should give better view into this definition. Nevertheless, according to convention, this structure will still be called ’a tree’.

5-22

0-5

38-*

22-37

Fig. 1. MVB-tree structure outline

The tree structure consists of the following elements: – Tree root descriptor - object that contains information about the time range in which the root is valid and the root identificator. – List of tree descriptors. Descriptors for all roots in the tree are connected in a list (or other structures) according to growing, separable life spans. – Middle nodes and leaves Nodes in MVB-tree are divided into two groups, middle (directory) nodes and leaves. This division is similar for that made in every balanced hierarchical index. Every entry in a node has its life span (that is, validity time range), in which information contained in the entry is valid. Middle nodes are redirecting the search to lower levels; they use key ranges in their subtrees (minKey/maxKey). Leaves contain individual data objects connected to specific keys. ROOT 1

ROOT 2

ROOT 3

[- *, 6 )

[6 ,9 )

2 2 3 5

2 3 7 9

[3 ,6 ) [1 ,3 ) [2 ,*) [4 ,5 )

RO OT 4 [9 ,1 1 )

[6 ,*) [2 ,*) [7 ,*) [8 ,*)

[11 , *)

2 3 9 9

2 / 3 [ 2 ,1 1 ) A 9 /1 1 [7 ,1 1 ) B 2 /9 [1 1 ,*) C

A 2 [6 ,*) 3 [2 ,*)

B 7 [7 ,1 0 ) 9 [8 ,*) 1 1 [9 ,1 1 )

[6 ,* ) [2 ,* ) [8 ,1 2 ) [1 2 ,*)

C 2 [6 ,*) 3 [2 ,*) 9 [8 ,*)

Fig. 2. An example of MVB-tree

Figure 2, shows an MVB-tree after the following series of operations: Add(2), Add(3), Add(2), Add(5), Del(5), Add(2), Add(7), Add(9), Add(11), Del(7), Del(11), Add(9). Adding a non-existing object result in its creation. Adding an existing object results in creation of a new version for this object. Structural modifications, which are presented in the following section, are autonomous. Implementations can use both of them, one or none without impact on correctness of the solution. Extending MVB-tree with those modifications is relatively simple, but results in one inconvenience which is higher disc space consumption [3].

202

3.1

M. Gorawski and M. Gorawski

Reverse Pointers

MVB-tree structure was originally designed for timestamp queries processing. the first proposed extension introduces the ability to overcome this limitation by applying a convenient mechanism for a time-range query processing. Our proposal is to keep for every entry the reverse pointer pointing to the previous version of the given object. These pointers are stored in one-directional list in every object. Storing the pointers is very simple (when inserting an object, it is checked if a previous version of a given object exists (it is checked whether it’is an update or ’pure’ insertion)). If the above is true, in a new entry a reverse pointer to the leaf containing the previous version is activated. The approach has at least one significant drawback (which depends on the characteristics of the system employing the MVB-tree). Namely, if at some point in time, deletion of an object was performed, then the reverse pointers list is finished. Further inserted object will not be connected to the deleted version. If it is required then an additional effort during insertion must be taken (only for objects which appear to be new) by searching the latest active version of the object. Such an operation can be time-consuming. Other solution is to use a separate B+ tree structure to keep the last object versions. The second solution has two advantages: (1) it can be used for fast processing of queries concerning present time, (2) keeping all reverse pointers is very fast and simple. In our implementation, the problem has marginal significance, because for the implemented system there is no deletion operation after which insertion for an object is performed. Figure 3. shows an MVB-tree with reverse pointers. The reverse pointers list is presented for object with ID=2. As we can see, regardless of entry duplication (for example entry 2[6,*) for ROOT2 and ROOT3) this approach points directly to the previous version of the object. It is possible to get a full object A history with max H+UPD(A) node accesses, where H is the MVB-tree max height and UPD(A) is the number of updates for object A. Some new ideas are tested targeting the reverse pointers disc usage reduction. Probably only a few reverse pointers for the entire leaf are required, and every entry can be assigned to a specific entry in the table.

ROOT 1

ROOT

2

ROOT 3

[ - *, 6)

[6 ,9 )

[9 ,* )

2 [3,6) 2 [1,3) 3 [2,*) 5 [4,5)

2 [6,*) 3 [2,*) 7 [7,*) 9 [8,*)

2/3 [2,*) A 9/11 [7,*) B

A 2 [6,*) 3 [2,*)

B 7 [7,*) 9 [8,*) 11 [9,*)

Fig. 3. Reverse pointers in MVB-tree structure

Modified R-MVB Tree and BTV Algorithm

3.2

203

Temporal Aggregates

There are applications in which the access to detailed data is not necessary, sometimes even undesirable. In data warehousing and data processing, the emphasis is put on fast aggregates and summaries retrieval, rather than on retrieving detailed results for specific objects. Dependencies between keys (if any exists) are not visible at the tree level, so one way to use the aggregates is the time aggregation approach. The approach assumes aggregating values for every object during its development. The MVB-tree does not provide a way to answer such a query, and without reverse pointers, evaluating such a query is very difficult. The simplest example of the approach application can be a system storing values from water meters. Every meter has its ID, and values are updated by sending a changed value since the previous measurement. In the basic version of the system, such a query is quite difficult to process, however by setting additional aggregate field named SUM it becomes easy. The detailed data are not lost, so it is still possible to get the detailed answer. Another measured feature can be the number of updates counted since the first object registration. The feature provides knowledge about objects change rate. These aggregations can be used not only for retrieving measurements for entire history. By submitting two queries concerning two extreme points in time range, it is possible to easily retrieve an answer for such a range query. Figure 4 shows the structure of a MVB-tree extended with aggregations. The tree keeps the last measure update as a value, SUM is a sum of values (the current value of the meter) and COUNT contains the number of updates (the number of versions in the history). ROO T 1

ROO T 2

ROOT 3

[-*, 6)

[6,9)

2 2 3 5

2 3 7 9

[3,6):4:2:5 [1,3):1:1:1 [2,*):3:1:3 [4,5):2:1:2

[6,*):-2:3:3 [2,*):3:1:3 [7,*):7:1:7 [8,*):6:1:6

A 2 [6,*):-2:3:3 3 [2,*):3:1:3

RO OT 4 [9,11)

2/3 [2,11) A 9/11 [7,11) B 2/9 [11,*) C

B 7 [7,10) ):7:1:7 9 [8,*):6:1:6 11 [9,11)5:1:5

[11, *)

2 3 9 9

[6,*) :-2:3:3 [2,*):3:1:3 [8,12) :6:1:6 [12,*) :-6:2:0

C 2 [6,*):-2:3:3 3 [2,*):3:1:3 9 [8,*):6:1:6

Fig. 4. Example of MVB-tree with aggregation fields in leaves

4

MVB Structure Usage in Distributed Spatio-temporal Telemetric Data Warehouse (B-DSTDW(t)) - The BTV Balancing Algorithm

Load balancing in spatio-temporal data warehouses systems is used for minimizing average response time. Average response time is defined as an average interval

204

M. Gorawski and M. Gorawski

between the moment when a query had been send to DSTDW, and the moment when a response had been acquired. Acceleration of response time is acquired by adequate data distribution between DSTDW system nodes (servers) and making operations on them as parallel as possible [10]. In the case of B-DSTDW systems based on computers with the same parameters, balancing comes down to equal loading of every node (here the node performance is the bottleneck). However when a B-DSTDW system is based on computers with various performance characteristics, adequate load balancing algorithm should be used for balancing such a system. Moreover, adequate data allocation schema must be used, to ensure proper participation of every node in the process of response generation. The algorithm of direct balancing based on threshold values (BTV) applied to BDSTDW system uses vertical data partitioning basing on ranged values of data partitioning algorithm, where relation is divided into partitions, with partition boundaries in [11]. The node administers partition attributes value’s range , for all . When , it is assumed that administers empty partition Nodes administrating adjacent ranges are called neighbors. denotes node load, defined as a number of tuples stored in . It is assumed that coordinator (in PSDW it is the managing module, which performs distributed data loading) has access to information of partition ranges and it points adequate nodes needed byl query, insert and delete operations. After every insert or delete operation, BTV algorithm is activated. The BTV algorithm bases on two fundamental operations: – When a certain node becomes responsible for too big data set, it moves part of the data to least loaded neighbor, attempting to balance the load. Such operation shall be called NBRADJUST – In the case when node ( ) is greatly overloaded comparing to least loaded node but loads in neighboring nodes does not differ enough REORDER operation is performed. The original online balancing described in paper [10] based on an algorithm which was designed for peer-to-peer systems. For efficient usage of this algorithm in B-DSTDW(t) system (BTV), certain modifications had to be applied: – node data load analysis is related only to the amount of data stored in the node nodes, servers performance is not taken into consideration, – data rows loaded to the data warehouse one by one cause a considerable elongation of data loading process, – an assumed way of calculating successive threshold values is adjusted for systems consisting of a large number of nodes (original definition of threshold values allows considerable system imbalance ratios, which is not acceptable in DSDW(t) system). – the algorithm allows for existence of only one partitioning attribute (in PSDW(t) system data partitioning bases on 2 attributes - X,Y coordinates of meters location (source of the measures). In the BTV algorithm those limitations were removed, by the following modifications: – the introduction of new methods of calculating the imbalance ratio for the system that take the nodes performance characteristics into consideration,

Modified R-MVB Tree and BTV Algorithm

205

– the insertion of tuples in packets of size determined by system administrator, – the creation of one partitioning attribute for measurements table based on X, Y coordinates of meter location 4.1

Balanced Distributed Spatio-temporal Telemetric Data Warehouse (B-DSTDW(t)) System

Regulation of energy sector in European Union created a new market - media (electricity, gas, heat and water) recipient market. Main problem which emerges is automated reading of hundred of thousands recipients meters and critically fast analysis of terabyte sets of meters data. In the case of electricity suppliers reading, request prognosis and decision-making subjects to energy balance closure regime (e.g. in 30 minutes time). Condition for achieving this goal is usage of Integrated Meter Reading (IMR) and Distributed Spatial Telemetric Data Warehouse (DSDW(t)) technology 5. IMR system sends data from media meters to database system via cellular telephony network (GPRS technology). IMR is a transactional system, which services four types of meters: electrical energy, water, gas and heat Data stored in the telemetric server are in raw state and needs to be adequately formatted. DSDW(t) system gathers data from telemetric servers in extraction process via network with TCP/IP protocol.In distributed DSTDW(t) system during extraction process additional partitioning is performed followed by data loading into system nodes. Lo c a l ra dio c om m u nica tio n

B -D ST D W (t)

G S M N etw o rk

T C P /IP

GSM

E T L p ro ce ss – d ata p artitio nin g an d lo ad ba lan cin g

D ata lo ad ing

AIUT G SM Telem etric Server

D S TDW S erv ers

C lien t – se rver R M I com m un ication

S p a tio te m p o ra l q u e rie s

D S TDW C lien t

10

In te grated M eter R ead in g S ystem (IM R )

12

33

34 1

31 21 15

22

14 30

Fig. 5. IMR/DSTDW(t) system architecture

Existing research on DSDW systems and spatio-temporal DW systems (STDW) concentrated on architecture indexed with aR-tree [5]. In current research we test new structures capable of servicing spatio-temporal queries. One of proposed structures is the below-presented R-MVB index, which combines both MVB-tree and R-tree. 4.2

Modified R-MVB Structure

Modified R-MVB index 6 consists of two indexes - R-tree and modified MVBtree. The first index handles responses to spatial queries (list of spatial objects in

206

M. Gorawski and M. Gorawski Hashtable pointing adequate MVB-tree Hashtable pointing adequate R-tree 1 D A T A S E T

1

R-tree R-tree

MVB-tree

2 Spatial result + temporal query

R-tree

3

MVB-tree

4 .

MVB-tree

.

Fig. 6. Modified R-MVB-tree

a range query). The second index calculates aggregates for certain spatial object in certain time. R-tree is optimized for range-spatial search. Such combination ensures efficient support of spatio-temporal queries. In this structure data loading is independent of data characteristic. This means that no knowledge is needed while to build the index.

5

R-MVB Structure Tests

Test environment consisted of three B-DSTDW(t) server nodes and two BDSTDW(t) client nodes. The below-mentioned tests shows results of modified RMVB tree in comparison with other indexes (STCAT) [3] used in B-DSTDW(t).

Index building time [s]

800 700 600 500

STCAT

400

R-MVB

300 200 100 0 120000

350000

610000

1300000

2000000

Data set's size

Fig. 7. Indexes’ building time

The Above figure shows that modified R-MVB tree has great response time to spatio-temporal queries, however its building time 7 is much longer comparing to the other indexes (in this case STCAT index tests results are shown). This gives R-MVB certain disadvantage, but still, thanks to its ability for storing record’s complex history and superior response time it is still worth of using. The STCAT index is a good alternative to the modified R-MVB tree, however as it is shown in 8 the response time of the STCAT index increases with data set size whereas R-MVB tree has constant response time regardless of the data set size.

Response time to spatio-temporal queries [ms]

Modified R-MVB Tree and BTV Algorithm

207

1000 800 600

STCAT

400

R-MVB

200 0 120000

350000

610000

1300000

2000000

Data set's size

Fig. 8. Spatio-temporal queries response time

6

Conclusion

In this paper, the modifications of the MVB-tree are described thoroughly, along with its spatio temporal version namely R-MVB index. This structure has a quite valuable asset in terms of B-DSTDW(t) system effectiveness increase. The previous SDW(t) systems based on an aR-tree supported only spatial queries, while modified R-MVB allows efficient responses not only to spatial queries but also to spatio-temporal queries. Certain disadvantages such as long building time and large hard disk usage shows that there is still the need for new indexing structures. Currently new indexes are being upgraded and developed,, however, the modified R-MVB is still the best index available in this term.

7

Future Works

Apart from new indexing structures current research focuses also on a new GDWSA(t) system [12,13](Grid Data Warehouse with Software Agents system for telemetric data). The grid is a form of a distributed computation, that coordinates and shares: calculations, applications, data, memory and network resources. This is obtained through dynamically changing and geographically distributed organization. The grid technology can be the key to solve highly complexed computation problems. The Software Agent’s methodology bases on the grid technology. Single node has the ability to configure, connect to a remote catalogue and directly to other Agents. Tasks are divided and executed parallelly on many workstations. This results in considerable speed-up. Built-in functions, that measures efficiency of a local node allows whole system performance evaluation. The system is adapted to a multi-user mode, and a good balancing mechanism provides equal use of all system nodes. It is essential for a full usage of system capabilities in an unknown environment. The balancing goal is to omit slower nodes and to send more tasks to the fastest nodes. To achieve this goal a new genetic balancing algorithm [14] designed especially for GDWSA(t) system is currently being implemented.

208

M. Gorawski and M. Gorawski

References 1. Manolopoulos, Y., Nanopoulos, A., Papadopoulos, A.N., Theodoridis, Y.: R- Trees have grown everywhere. In: Proceedings of the 23th VLDB Conference, Greece, pp. 13–14 (1997) 2. Tao, Y., Papadias, D.: Range aggregate processing In spatial databases. IEEE Transactions on Knowledge and Data Engineering (TKDE) 16(12), 1555–1570 (2004) 3. Gorawski, M., Faruga, M.: STAH-tree, Hybrid Index for Spatio Temporal Aggregation. In: 9th International Conference on Enterprise Information System (ICEIS), Portugal (2007) 4. Papadias, D., Kalnis, P., Zhang, J., Tao, Y.: APN 2001. LNCS. Springer, Heidelberg (2001) 5. You, B., Lee, D., Eo, S., Lee, J., Bae, H.: Hybrid Index for Spatio-temporat OLAP operations. In: Proceedings of the ADVIS Conference, Izmir, Turkey (2006) 6. Gorawski, M., Malczok, R.: Materialized aR-tree in Distributed Spatial Data Warehouse. An International Journal Intelligent Data Analysis 10(4), 361–377 (2006) 7. Gorawski, M., Malczok, R.: On Efficient Storing and Processing of Long Aggregate Lists. In: Tjoa, A.M., Trujillo, J. (eds.) DaWaK 2005. LNCS, vol. 3589, pp. 190– 199. Springer, Heidelberg (2005) 8. Gorawski, M., Chechelski, R.: Parallel Telemetric Data Warehouse Balancing Algorithm. In: 5th International Conference on Intelligent Systems Design and Applications, Wrocaw, Poland, September 8-10, 2005, pp. 387–392. IEEE CS, Los Alamitos (2005) 9. Tao, Y., Papadias, D.: Spatial Queries In Dynamic Environment. ACM Transactions on Database Systems 28(2) (June 2003) 10. Becker, B., Gschwind, S., Ohler, T., Seeger, B., Windmayer, P.: An asymptotically optima multiversion B-Tree. VLDB Journal 5(4), 264–275 (1996) 11. Gorawski, M., Kamiski, M.: On-Line Balancing of Horizontally-Range- Partitioned Data in Distributed Spatial Telemetric Data Warehouse. In: 3rd International Workshop on Grid and Peer-to-Peer Computing Impacts on Large Scale Heterogeneous Distributed Database Systems (GLOBE 2006), DEXA 2006, September 4 8, 2006, pp. 273–277. IEEE CS, Los Alamitos (2006) 12. Ganesan, P., Bawa, M., Garcia-Molina, H.: Online Balancing of Range-Partitioned Data with Applications to Peer-to-Peer Systems. Stanford University, Stanford 94305 13. Gorawski, M., Bakowski, S., Gorawski, M.: The Software Agents in a Database Grid. In: 2nd Workshop on Large Scale Computations on Grids (LaSCoG 2006) XXII Autumn Meeting of Polish Information Processing Society, Wisa, Poland, November 6 - 10 2006, pp. 259–266. IMCSIT (2006) 14. Polat, F., Alhajj, R.: A multi-agent tuple-space based problem solving framework, Middle East Technical University, 06531 Ankara, Turkey (1998)

Towards Stream Data Parallel Processing in Spatial Aggregating Index Marcin Gorawski and Rafal Malczok Silesian University of Technology, Institute of Computer Science, Akademicka 16, 44-100 Gliwice, Poland {Marcin.Gorawski, Rafal.Malczok}@polsl.pl

Abstract. Data processing computer systems store and process large volumes of data. The volumes tend to grow very quickly, especially in data warehouse systems. A few years ago data warehouses were used only for supporting strictly business decisions but nowadays they find their application in many domains of everyday life. New and very demanding field is stream data warehousing. Car traffic monitoring, cell phones tracking or utilities meters integrated reading systems generate stream data. In a stream data warehouse the ETL process is a continuous one. Stream data processing poses many new challenges to memory management and data processing algorithms. The most important aspects concern efficiency and scalability of the designed solutions. In this paper we present an example of a stream data warehouse and then, basing on the presented example and our previous work results, we discuss a solution for stream data parallel processing. We also show, how to integrate the presented solution with a spatial aggregating index. Keywords: stream processing, stream data warehouse, parallel algorithms.

1

Introduction

Not so long ago data warehouses were used only for supporting decision making processes in strictly business systems. Nowadays, when more and more human activities produce huge volumes of data, data warehouse systems find their application in many domains of everyday life. Recently, stream data and stream data warehouses have become an active field of research. There are many examples of systems generating stream data: car traffic and utilities consumption monitoring or cell phones tracking. Stream data warehouse designers must face many problems which do not exist in standard data warehouses. The ETL process is not executed periodically in batch mode but runs continuously, transforming and loading data generated by data sources. The data must be stored and efficiently processed to provide up-to-date data-mining results. Processing large amounts of data is highly time and resource consuming process. To make the data processing efficient and scalable parallel or distributed computing is often applied. R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 209–218, 2008. c Springer-Verlag Berlin Heidelberg 2008 

210

M. Gorawski and R. Malczok

In [1] we presented Materialized Aggregates List (MAL) – a solution that is able to process long lists of aggregates. We applied multithreading and partial aggregates materialization to make the solution applicable for large volumes of data. In this paper we first show how to use MAL to process stream data and then we address the problem of integrating MAL with a spatial aggregating index. The final outcome of our research is a spatial aggregating index which not only can answer a range query, but also is able to paralelly build and integrate streams of aggregates. The remaining part of the paper is organized as follows: in the next section we first present an example of a stream data warehouse and then, basing on results of our previous work, we discuss the details of a solution for stream data processing. Section 3 presents the streams merging operation and addresses the memory management problems. In section 4 we present efficiency tests results. Finally in section 5 we conclude the paper and present our future plans.

2

Stream Data Warehouse

We started our research from describing an example motivating our research – a stream data warehouse system. The system monitors consumption of utilities such as water, natural gas and electrical energy. The meters are located in some region where a telemetric installation exists. The meters send the readings via radio waves to collecting points called nodes. The nodes, using standard TCP/IP network, transfers the readings to telemetric servers. The ETL process gathers the data from telemetric servers. The operation of reading the utility consumption in a meter can be executed according to two different models. The first model assumes that there is a signal that triggers the meter to send the reading to the collecting point. In the second model no signal is required and the meters send the readings at some time intervals. The intervals depend on the telemetric system configuration. In the discussed example the second model is applied. Next to the meter readings the data warehouse stores also additional information: the geographical location of the meters and collecting points, weather conditions in the region encompassed by the telemetric installation and a brief description of the meters users. The information is used when the utilities consumption is being analyzed. Stream Source Definition. Considering the telemetric system operation we assume that every single meter is an independent source generating an endless stream of readings. A single reading generated by a meter contains a unique identifier of a meter that generated the reading, a precise timestamp defining the date and time of reading generation and a list of values storing the meter’s counters values for the zones monitored by the meter. As a data stream we understand an endless sequence of elements of a given type. The elements exists in a stream in a well-defined moment of time. For a stream there is no concept of a stream end. As as a stream beginning we consider the first element generated by the source of the stream (utilities meter).

Towards Stream Data Parallel Processing in Spatial Aggregating Index

211

The intensity of a stream (the number of elements occurring in a stream in a given time unit) depends on the telemetric system configuration parameters defining how often the meters send the readings to the collecting point. In figure 1 we present an example of a stream generated by an utility meter (depicted as a triangle). The stream elements (readings) are marked with dots. 2.1

Materialized Aggregate List

It is very common for data warehouses to process aggregated data instead of raw data. There are two main reasons for this: (1) in most cases end users are more interested in general trends existing in data than in detailed analyses results, and (2) aggregating data reduces data volumes sizes and, as a consequence, shortens the processing times. To efficiently process stream data, raw data streams are aggregated as well. Aggregates are calculated for some time interval called an aggregate time window Taw . In the case of utilities consumption the values stored in the aggregate can be interpreted as approximated medium consumption in some time period (aggregate time window). Figure 1 presents aggregate time window beginning at the moment tb1 and ending at the moment tb2 . Aggregate values are calculated in the following way: 1. The meter readings in the moment tb1 are approximated using linear interpolation (vn denotes a meter reading in the moment tn ). For a scenario presented in figure 1 the meter reading in the moment tb1 is calcu−tn−1 lated as: vb1 = vn−1 + ttb1 (vn − vn−1 ). Similarly, for the moment tb2 : n −tn−1 tb2 −tn+2 vb2 = vn+2 + tn+3 −tn+2 (vn+3 − vn+2 ) 2. Aggregate value for a given time window is calculated as a product of subtracting approximated readings in aggregate window time moments: vb2 −vb1 .

aggn+2

Taw meter kn-1

aggregate stream

kn

aggn-1

aggn

tn-1

tn tb1

TIMESTAMP value1 value2

kn+2 kn+3

aggn+1

tn+2 tn+3 tb2

aggn+2

valuen t

Fig. 1. Aggregates stream is generated basing on a readings data stream

The outcome of a raw data stream aggregation is a stream of aggregates calculated for a given aggregate time window (user-defined). Figure 1 shows that the number of elements in the aggregates stream is significantly less than the number of elements in the raw data stream – the raw data stream is reduced. The wider the aggregate time window, the less elements in the resulting aggregate stream.

212

M. Gorawski and R. Malczok

The process of raw data stream processing and aggregates calculation is very time-intensive, especially when we want to process a few streams in the same time. Materialized Aggregate List can be applied for creating and processing aggregates streams. A single information stored in MAL is an aggregate A. The aggregate comprises of a timestamp T S and a set of values VA = {Vi } (A = [T S, VA ]). Every element Vi ∈ VA is of a defined type tVi (integer or floating point number). MAL bases its operation on the concept of list and iterator. Mechanisms implemented in the list allows parallel generation and optional materialization of the calculated aggregates. Iterators are used for browsing the generated data and communicating with the list. Aggregates created by the list are stored in a static table located in the iterator. The table is a set of logical parts called pages. Each page stores some number of aggregates; the number is equal for each page. The fact that the table is logically divided into pages is used when the table is being filled with aggregates. In [1] we presented and compared three multithread page-filling algorithms. Static iterator tables coupled with multi-thread algorithms allow efficient processing data streams of any length without imposing memory limitations. MAL supports also the materialization mechanism which significantly speeds-up the process of aggregates recreation. The list can be used as a tool for retrieving aggregates directly from a database (eg. for single telemetric object) and also as a component of indexing structure nodes (the aggregates are created basing on the data from lower levels). The MAL can work in one of the following aggregate-creating modes: 1. In the first mode MAL creates aggregates retrieving data directly from a database. This mode is used when MAL processes data of a single object (a single telemetric meter). 2. In the second mode MAL also creates aggregates retrieving data from a database, but the aggregates are created for more than one object. This mode finds its application when MAL manages aggregates of a node located on the lowest level of the indexing structure. The objects, for which the MAL creates aggregates are appropriately marked and their readings are aggregated when MAL creates aggregates for the node. 3. In the third mode MAL creates aggregates basing on the aggregates of other MALs. This mode is used when MAL is a component of a node located on intermediate level of the indexing structure.

3

Managing Aggregates Streams in Spatial Aggregating Index

The basic functionality of the presented stream data warehouse system is to provide an answer to a range query. A range query is defined as a set of regions R encompassing one or more utilities meters. The answer generated by the system consist of two parts. The first part contains information about the number of

Towards Stream Data Parallel Processing in Spatial Aggregating Index

213

various kinds of meters located in the query region. The second part is a merged stream (or streams) of aggregated readings coming from the meters encompassed by the query region. There are many spatial indexes which can answer any region query. The best known and most popular are indexes based on the R-Tree [2] spatial index. The main idea behind the R-Tree index is to use a hierarchical indexing structure where index nodes on the higher levels encompasses regions of the nodes on the lower levels of the index. A detailed review of spatial indexing structures based on R-Tree index can be found in [3]. To calculate the second part of the answer we need to apply an aggregating spatial index which in index nodes stores aggregates concerning objects located in the nodes regions. The first proposed solution to this problem was aR-Tree [4] (aggregation R-Tree). aR-Tree index nodes located on the higher levels of the hierarchy stores the number of objects in the nodes located on the lower levels of the hierarchy. The functionality of this solution can be easily extended by adding any kind of aggregated information stored in the nodes. The idea of partial aggregates was used in a solution presented in [5], where the authors want to use it for creating a spatio-temporal index. Most of the solutions presented in the literature assume that the size of the aggregated data is known and small enough to be stored in main computer memory. In the case of the stream data such an assumption cannot be made. In this paper we focus on the problems resulting from integrating MAL in a spatial aggregating index nodes. We show the algorithm we use to make the resulting solution efficient and scalable. 3.1

Aggregates Streams Merging

When MAL is used in aggregating spatial index the lists stored in upper level nodes merges aggregates streams created by the lower level nodes. Prior to defining details of the streams merging operation we need to define the requirements which must be satisfied to add two aggregates. Each aggregate has a timestamp and a list of values. The length of the values list is not limited and can be adopted to system requirements. Aggregate type is defined by the cardinality of the VA list and the types of elements of VA . Two aggregates are of equal type if their values lists are of equal cardinality and values located under the same indexes are of the same type tV . Two aggregates can be added if an only if they have equal timestamps and they are of equal type. The result of aggregates adding operation is an aggregate having timestamp equal timestamps of the added aggregates and resulting aggregate values list is created by adding values of the added aggregates (see fig. 2). Aggregates streams merging operation merges two or more aggregates streams creating one aggregates stream. The merging operation is performed by adding aggregates with equivalent timestamps. For aggregates streams merging there is a special moment in time when the operation begins. All merged streams must contain all aggregates used during adding operation. The merging operation cannot be performed for streams where some required aggregates are missing.

214

M. Gorawski and R. Malczok

2007-01-01 00:30 2007-01-01 01:00 23.345 3.2 10.7 4.05

Si

Sk 2007-01-01 00:30 2007-01-01 01:00 31.475 3.7 16.97 4.29

2007-01-01 00:30 2007-01-01 01:00 8.13 0.5 6.27 0.24

Sj

ts

t

Fig. 2. Example of aggregates adding and aggregates streams merging. Two streams Si and Sj are merged into one resulting stream Sk .

3.2

Query Answering and Memory Management

MAL iterator uses a static table for browsing and managing aggregate stream. All tables used by the system are stored in form of a resource pool. It is possible for an external process to use MAL iterator only if there is at least one free table in the pool. Defining the number of tables stored in the pool one can very easily control the amount of memory consumed by the part of system responsible for processing data streams. In most cases the number of MALs involved in answer stream generation process is significantly greater than the number of available tables. To make the process of answer stream generation efficient we defined an algorithm that assigns the available tables to appropriately chosen MALs. Finding Nodes Answering the Query. For every window from the query region set the indexing structure is recursively browsed in order to find the nodes which provides answer to the query. The browsing starts from the tree root and proceeds towards tree leaves. The tree browsing algorithm checks the relation of the window region O and the node region N . The node is skipped if its region share no part with the window region (O ∩ N = ∅). If the window region entirely encompasses the node region (O ∩ N = N ), the node is added to F AN set (Full Access Nodes – nodes which entirely participates in answer stream generation with all theirs aggregates). The last case is when a window region and node region share a part (O ∩ N = O ). The algorithm performs a recursive call to lower structure levels, passing parameter O as an aggregate window. When traversing to lower tree levels it is possible that the algorithm reaches the node on the lowest hierarchy level. In this case, the algorithm executes a query searching for encompassed objects. A set of found objects is marked with a letter M . The objects in the M set are marked with a so called query mark which is then used when the answer stream is being generated. Sorting Elements Answering the Query. The process of finding nodes and single objects involved in query answer stream generation creates two sets: F AN and M . The first operation executed before iterator tables assignment is to check the M set if it contains all elements encompassed by some lowest level node. Such a scenario may occur when the query contains many small windows of which

Towards Stream Data Parallel Processing in Spatial Aggregating Index

215

none encompasses the entire node region, but merging a few windows results in a window which region encompasses the entire node region. If such windows are found then the single objects encompassed by those windows are removed from the M set, and a node which region is encompassed by the merged window region is added to the F AN set. An example can be found in figure 3. M set

O1

O2 node E region

O5 O4 O3 O7

O6

O1 O3

O2

O4

O5 O6

queries regions

FAN set

O7 O8

F

O8 O10 O9

E H

F

G

A B C D E

Fig. 3. An example showing a few query windows, which, when merged, encompass the entire node region. The node E region encompasses objects O1 − O7 . There are three windows which, when merged, entirely encompass node E region. The objects O1 − O7 are removed from the M set, the node E is added to the F AN set.

In the next step the nodes creating the F AN set are sorted according to the following criteria: 1. The number of materialized data available for a given node. The more materialized data, the faster a stream is recreated and the higher the position the node. Only the materialized data that can be used during answer stream generation are taken into account. 2. Generated stream materialization possibility. One of the data warehouse operating parameter is the border level, starting from which the streams generated by the nodes are materialized. For level set to 0 all streams generated by the nodes are materialized. For level set to -1 not only nodes streams are materialized, but also streams generated by single objects. The materialization possibility is a binary criteria: a node has it or has not. The node with the materialization possibility is always placed on higher level than a node which stream will not be materialized. 3. The the amount of objects encompassed by the node’s region. The more encompassed objects, the higher the position of the node. Alike the elements in F AN set sorted are the elements of M set. In the case of M set, the only sorting criteria is the amount of available materialized data which can be used during answer stream generation. Assigning Iterator Tables. After sorting F AN and M sets the algorithm starts the tables assignment process. Let P denotes the iterator table pool, and |P | the number of tables available in the pool. If ((|F AN | = 1 and |M | = 0) or

216

M. Gorawski and R. Malczok

(|F AN | = 0 and |M | = 1)) (the answer is a single stream generated by a node or a single object) then one table is taken from the pool and it is assigned to an element generating the answer stream. The condition that must be satisfied is that |P | ≥ 1 (there is at least one table in the pool). In the other case (the answer stream consists of more than one stream) the algorithm requires an additional structure called GCE (Global Collecting Element). The GCE element is of type MAL and it is used for merging streams generated by other elements (nodes and objects). The GCE element requires one iterator table. If, after assigning a table to the GCE element, there are some free tables left in the pool they are assigned to other elements involved in the answer stream generating process. The tables are assigned first to the elements in F AN set (according to the order set by the sorting operation). Then, if there are still some free tables, they are assigned to single objects from M set (also in appropriate order). A few tables assigning scenarios are possible: – |P | ≥ |F AN | + |M | – every element of F AN and M is assigned a separate table, – |P | ≥ |F AN | and |P | < |F AN |+|M | – every element of F AN is assigned a separate table, some objects streams of M are merged at creation stage, – |P | < |F AN | – all streams of the elements from M set and some streams of the elements from F AN set are merged into one stream during the process of aggregates creation. A stream can be materialized only if a generating element is assigned a separate iterator table. If a stream of element aggregates is at early generation stage merged with streams of other elements the matarialization is not performed because it cannot be explicitly determined for which element the stream was generated. In general, partial answer streams, for example streams merged in GCE, are not materialized. The stream generated by GCE is not materialized either because it changes with every single query. If the system materialized every indirect stream the amount of materialized data would grow very fast. Figure 4 presents an example of tables assignment algorithm operation. In this example the number of tables in the pool is 4. Answer stream generating elements are: nodes 4, 6 and 7 (F AN set) and objects a, b and c (M set). The elements were sorted into an order presented in the params table (GCE element). Because (|F AN | + |M | > 1) we need to reserve one iterator table for the GCE element. The remaining tables are assigned to nodes 7, 4 and merged elements: node 6 and single objects a, b and c. If the materialization level is ≤ 0 the streams generated by nodes 7 and 4 will be materialized.

4

Test Results

In this section we present experimental test results. We implemented the presented solution in Java. For the tests we used a machine equipped with Intel Core 2 T7200 processor, 2 GB of RAM and 100 GB HDD 7200 rpm. The software environment was: Windows XP, Sun Java 1.5 and Oracle 9i database. The F AN and M sets sorting algorithm has guaranteed log(n) performance.

Towards Stream Data Parallel Processing in Spatial Aggregating Index

217

GCE params

1

7

2

4

3

6 6

7

8

4

5 b

c

MAL2

+ +

answer stream

a b

a

MAL1

MAL3

c

Fig. 4. Operation of the algorithm assigning iterator tables to answer stream generating nodes and objects. Elements are stored in the params table of the GCE element.

The test model contained 1000 meters. Every meter generated a reading in a time period from 30 to 60 minutes. The aggregation window width was 120 minutes. Database contained readings from a period of one year (over 12 million readings). We tested the system by first defining a set of query windows and then calculating the answer stream. The generated stream was browsed without aggregates processing. The times presented in the charts are sums of answer calculation and stream browsing times. In all tested cases the answer calculation time was negligibly small when compared to the stream browsing time. The measured times concerned answer stream generation and browsing for a query encompassing 100 objects. In order to investigate various query defining possibilities we separated two different query variants: 1. a query encompassing 2 index nodes (each node encompassed 25 objects) and 50 single objects (marked as 2*25+50), 2. a query encompassing 3 index nodes and 25 single objects (marked as 3*25+25), Figure 5 shows answer stream generation times for the above described queries for the first run (no materialized data) and the second run (materialized data available). We can observe that stream materialization has a great positive influence on the query answering process. The chart shows also that the stream browsing time depends on the number of MALs used for answer stream generation. The less number of MALs used the shorter the stream browsing times during the first run. On the other hand, even thought the first run times slightly increase with the number of MALs, during the second run the times significantly decrease (thanks to materialization). The second investigated aspect concerned system scalability. Figure 6 presents the answer generation and stream browsing times for various numbers of objects encompassed by the query. Experiments show that the stream browsing time depends linearly on the number of encompassed objects and aggregation period length. Thanks to the linear dependency, knowing the number of objects and the aggregation period length, we can easily predict the answer stream generation time.

218

M. Gorawski and R. Malczok 2*25+50 (first run) 2*25+50 (second run)

1200

3*25+25 (first run) 3*25+25 (second run)

1000

1600

first run

second run

1400 1200

800

1000

time [s]

time [s]

600 400

800 600 400

200

200 0

0 2

5

15

30

number of available iterator tables

25

50

75

100

125

150

175

200

number of encompassed elements

Fig. 5. Answer calculation times for vari- Fig. 6. Answer calculation time as a funcous number of available iterator tables tion of the number of elements

5

Conclusions and Future Plans

In this paper we presented continuation of our research on efficient stream data processing. We integrated a previously presented solution with a spatial aggregating index and provided a memory managing algorithm. The algorithm, using a set of criteria, sorts all the elements involved in answer stream calculation process and assigns available iterator tables. The most important criteria is the amount of available materialized data that can be used in answer stream generation process. The presented tests results show that increasing number of encompassed objects and extending the aggregation period length results in linear growth of answer stream calculation time. In the nearest future we want to verify the necessity of the materialization process. We suppose that if some of the streams generated by the lowest level nodes are not materialized it will cause no harm to answer generation time but will significantly reduce the number of materialized data stored in the database.

References 1. Gorawski, M., Malczok, R.: Multi-thread Processing of Long Aggregates Lists. In: Wyrzykowski, R., Dongarra, J., Meyer, N., Wa´sniewski, J. (eds.) PPAM 2005. LNCS, vol. 3911, pp. 59–66. Springer, Heidelberg (2006) 2. Guttman, A.: R-trees: a dynamic index structure for spatial searching. In: Proceedings of the SIGMOD Conference, Boston, MA, June 1984, pp. 47–57 (1984) 3. Manolopoulos, Y., Nanopoulos, A., Papadopoulos, A.N., Theodoridis, Y.: R-trees: Theory and Applications. Springer, Heidelberg (2005) 4. Papadias, D., Kalnis, P., Zhang, J., Tao, Y.: APN 2001. LNCS. Spinger, Heidelberg (2001) 5. You, B., Lee, D., Eo, S., Lee, J., Bae, H.: Hybrid Index for Spatio-temporat OLAP operations. In: Proceedings of the ADVIS Conference, Izmir, Turkey (2006)

On Parallel Generation of Partial Derangements, Derangements and Permutations Zbigniew Kokosi´ nski Cracow University of Technology, Faculty of Electrical and Computer Eng., ul. Warszawska 24, 31-155 Krak´ ow, Poland [email protected]

Abstract. The concept of a partial derangement is introduced and a versatile representation of partial derangements is proposed with permutations and derangements as special cases. The representation is derived from a representation of permutations by iterative decomposition of symmetric permutation group Sn into cosets. New algorithms are proposed for generation of partial set derangements in t. The control sequences produced by the generation algorithms appear either in lexicographic or reverse lexicographic order while the output sequences representing partial derangements are obtained from the control sequences in corresponding linear orders. A parallel hardware implementation of the generator of partial derangements is described. Keywords: derangement, partial derangement, permutation, derangement generation, permutation generation.

1

Introduction

Combinatorial generation is one of basic problems in computer science [11,17]. Combinatorial objects are involved as test or problem instances in numerous important application areas. A great many generation algorithms has been developed for such combinatorial objects like n–tuples, combinations, permutations, numerical and set partitions, trees, graphs etc. The focus of this paper is on generation of important classes of n–permutations with a forbidden set of k constant points, 0 ≤ k ≤ n. One specific class of permutations with no constant points (no 1–cycles) are derangements. Combinatorial properties of derangements are described in depth in [4,6]. Several methods for generation of all set derangements sequentially or in parallel linear array model are published in the literature [1,2,3,5,14]. In the present paper a new concept of a partial derangement is introduced and a versatile representation of partial derangements is proposed with permutations and derangements as special cases. No generation algorithm for such a class of combinatorial objects is known so far. The representation of partial derangements is derived from a representation of permutations by iterative decomposition of symmetric permutation group Sn into cosets [12]. Some particular properties of partial derangements as permutations are also established. R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 219–228, 2008. c Springer-Verlag Berlin Heidelberg 2008 

220

Z. Kokosi´ nski

The algorithms for generation of partial set derangements are developed in the parallel counter model. The control sequences are produced in O(1) average time per generated object. The output sequences are then obtained from the control sequences in O(n) time. Following [12,13] we propose a parallel hardware implementation of the generation algorithms with the help of a cellular permutation array that makes the generation process time efficient. The representation and some properties of permutations and partial derangements are described in section 2. In the next section two generation algorithms are presented. Finally, a hardware implementation of the generator of partial derangements is described in detail in section 4. The last section contains concluding remarks.

2

Representation of Partial Derangements

The representation of permutations is derived from an iterative decomposition of symmetric permutation group Sn into cosets [12]. In this representation set permutations are described by integer sequences called choice functions of indexed families of sets. Let us introduce at first the two coset representations of permutations. Let < Ai >i∈I denote an indexed family of sets Ai = A, where: A = I ={1, . . . , n}. Any mapping f which ”chooses” one element from each set A1 , . . . , An is called a choice function (or a system of representatives, or a transversal) of the family < Ai >i∈I . If for every i  = j a suplementary condition: ai  = aj is satisfied then any choice function α =< ai >i∈I that belongs to the indexed family < Ai >i∈I is called n–permutation of the set A. Set of all such choice functions represents the set of all permutations of the n–element set. Let us denote any permutation π of n–element set A = {1, . . . , n} by the sequence < π(1), π(2), . . . , π(n) >. The set of all permutations of A is called the symmetric group Sn . Theorem 1. Let < Pi >i∈I be indexed family of sets Pi ⊆ A, where Pi = {1, . . . , i}. Any choice function α =< pi >i∈I , that belongs to Carthesian product ×i∈I Pi represents a permutation of A. Proof. In [7] Hall and Paige have proposed two partitions of the symmetric group Sn on the finite set A = {1, . . . , n} into right and left cosets of Sn−1 in Sn : Sn = Sn−1 τn1 + Sn−1 τn2 + . . . + Sn−1 τnn

(1)

τnn Sn−1

(2)

Sn =

τn1 Sn−1

+

τn2 Sn−1

+ ...+

where + denotes the union of disjoint sets and τij denotes the transposition (ij) (in particular τii is the identity permutation). The complete iterative decomposition of Sn into right cosets resulting from the equation (1) is given below: Sn = Sn−1 τn1 + Sn−1 τn2 + . . . + Sn−1 τnn n−1 1 2 Sn−1 = Sn−2 τn−1 + Sn−2 τn−1 + . . . + Sn−2 τn−1

On Parallel Generation of Partial Derangements

.......................................................................

221

(3)

S3 = S2 τ31 + S2 τ32 + S2 τ33 S2 = {τ21 , τ22 } In a similar way the complete iterative decomposition of Sn into left cosets is obtained from (2):

Sn−1

Sn = τn1 Sn−1 + τn2 Sn−1 + . . . + τnn Sn−1 n−1 1 2 = τn−1 Sn−2 + τn−1 Sn−2 + . . . + τn−1 Sn−2

....................................................................... S3 = τ31 S2 + τ32 S2 + τ33 S2

(4)

S2 = {τ21 , τ22 } Any set Pi = {τi1 , . . . , τij , . . . , τii } , for i ∈ I = {1, . . . , n} is a system of representatives of right cosets (left cosets) and is called the complete right (left) transversal of Si in Si−1 . Moreover, |Pi | = i and Pk ∩Pl = ∅, for every k = l, k, l ∈ I. By substituting τij = j, we receive : Pi = {1, . . . , i}. Depending on the decomposition scheme any choice function α =< pi >i∈I , belonging to Carthesian product ×i∈I Pi correspondes to one of the two sequences: π1 =< π1 (1), . . . , π1 (n) > or π2 =< π2 (1), ..., π2 (n) >. The sequences π1 and π2 are obtained by performing on the elements of the set {1, . . . , n} the transposition sequence τ1p1 , τ2p2 , . . . , τnpn (according to the decomposition scheme (1)) or τnpn , . . . , τ2p2 , τ1p1 (according to the decomposition scheme (2)), respectively. The element τ1p1 that denotes the identity transposition may be omitted. Exemplary sequences π1 =< π1 (1), . . . , π1 (n) >, π2 =< π2 (1), . . . , π2 (n) > are represented by the choice functions α in lexicographic and reverse lexicographic orders, respectively, as shown for n = 4 in Tables 1 and 2. Let us now define permutations with forbidden positions, partial derangenments and derangements. Definition 1. A permutation π of n–element set A = {1, . . . , n} with a forbidden position i is the sequence < π(1), π(2), . . . , π(n) >, where π(i)  = i, for some 1 ≤ i ≤ n. Lemma 1. Let < Pi >i∈I be an indexed family of sets Pi ⊆ A, where Pi = {1, . . . , i}, 1 ≤ i < n − 1, and Pn = Pn−1 . Any choice function α =< pi >i∈I , that belongs to Carthesian product ×i∈I Pi represents a permutation of A with a forbidden position i if and only if (p(i)  = i) ∨ [(p(i) = i) ⇒ ∃j : (i < j ≤ n) ∧ p(j) = i].

(5)

Proof. There are to cases to be considered. 1. If p(i)  = i then p(i) = j, j < i, and the transposition (ij) is performed. Hence, i is not a fixed point in the permutation π. 2. If p(i) = i then the identity transposition (ii) is performed. If no other transposition (ji) exists, (i < j ≤ n), then i is a fixed point in the permutation π.

222

Z. Kokosi´ nski

Definition 2. Any n–permutation with k forbiden positions i  = π(i), 0 ≤ k ≤ n, 1 ≤ i ≤ n, is called a partial derangement δ(n, k) and the set of all forbidden positions is called the forbidden set F. Theorem 2. Let < Di >i∈I be an indexed family of sets Di ⊆ A, where Di = Pi , 1 ≤ i < n, and F = {f1 , f2 , . . . , fk }, 0 ≤ k ≤ n, be the forbidden set. Any choice function δ(n, k) =< di >i∈I , that belongs to Carthesian product ×i∈I Di represents a partial derangement of A, i.e. an n–permutation with k forbidden positions from the set F if and only if ∀d(i) ∈ F : (d(i)  = i) ∨ [(d(i) = i) ⇒ ∃j : (i < j ≤ n) ∧ d(j) = i].

(6)

Proof. The proof follows directly from the Lemma 1. There are two special cases: 1. iff k = 0 and F = ∅ then we have a permutation α = δ(n, 0); 2. iff k = n and F = {1, 2, . . . , n} = A then we have a derangement δ(n, n). Otherwise, for k ∈ / {0, n}, we have a proper partial derangement δ(n, k). Definition 3. Any permutation with k forbiden positions i  = π(i), 1 ≤ i ≤ k < n, is called a prefix partial derangement δp (n, k) with F = {1, . . . , k}. Definition 4. Any permutation with k forbiden positions i  = π(i), 1 < k < n, n − k + 1 ≤ i ≤ n, is called a suffix partial derangement δs (n, k) with F = {n − k + 1, . . . , n}. Lemma 2. The number of partial derangements P D(n, k), partial prefix derangements P Dp (n, k) and partial suffix derangements P Ds (n, k) is given by P D(n, k) = P Dp (n, k) = P Ds (n, k) = D(n) +

k    k D(n − i), i i=1

(7)

where D(i)=PD(i,i) denotes the number of derangements of i–element set. Proof. Proof is obvious. D(n) n→∞ P (n)

It is well known that lim

= e−1 [4]. Therefore, 1 > lim

n→∞

P D(n,k) P (n)

> e−1 .

The expected number of fixed points in an element δ(n, 0) is 1 [4]. Since the set of derangements δ(n, n) has no constant points, the average number of fixed P (n e ∼ points in permutations δ(n, k), 0 ≤ k < n, is lim P (n)−D(n) = e−1 = 1.582. n→∞

3

The Generation Algorithms

In [15] the authors have invented a derangement generation algorithm by interchanging at most four elements, with roughly 2.980 elements exchanged in average. The first Gray code and a constant average time generation algorithm is described in [3]. In this algorithm the next derangement is obtained from its

On Parallel Generation of Partial Derangements

223

predecessor by one or two transpositions or a rotation of three elements. It can be used also for generation of some classes of partial derangements. Our generation methods are much simpler then the above mentioned solutions. The algorithms PDGENLEX and PDGENREVLEX are shown in Figures 1 and 2. The control sequences are produced by the algorithms in lexicographical or reverse lexicographical order, respectively, and the corresponding partial derangements appear in one of the two linear orders depending on the decomposition scheme used. Input : n – size of the set, k – the number of forbidden positions in partial derangements, D – the forbidden set. Output: Table DA with the consecutive partial derangements. Method: The first function α in the table PD is obtained in step 1. The generation method is based on a counting process in the table PD. In step 4.3 the output is produced if the condition (6) is fulfilled. Computations run until the last c.f. α is generated. /1-2 initialization phase/ 1. for i:=1 to n do PD[i]:=I[i]:=1; MAX[i]:=i+1; 2. i:=n; 3. output(PD); 4. repeat 4.1. PD[i]:=(PD[i]+1) mod MAX[i]; 4.2. if PD[i]=0 then 4.2.1. repeat 4.2.1.1. PD[i]:=PD[i]+1; 4.2.1.2. i:=i—1; 4.2.1.3. PD[i]:=(PD[i]+1) mod MAX[i]; until PD[i] = 0; 4.2.2. i:=n; 4.3. if condition (6) is satisfied then output(PD); until MAX—PD=I; output(PD) /conversion and output/ 1. case of decomposition scheme left coset: permute the set DA={1, ... , n} according to transpositions in PD; right coset: permute the set DA={1, ... , n} according to transpositions in PD; 2. output DA; Fig. 1. The algorithm PDGENLEX

Since condition (6) can be examined sequentially in O(kn) time and computing the next DA takes O(n) time the ovarall complexity of the algorithm PDGENLEX is O(D(n)kn). In a parallel implementation described in section 4 computing of PD and DA in parallel takes O(1) and O(n) time, respectively, and the condition (6) is examined in parallel in O(n) time with a very low constant factor. Although not optimal in the sense of asymptotic time and space complexity, the parallel hardware generator provides fast generation of derangements in roughly 2.718

224

Z. Kokosi´ nski

Table 1. Sequences of partial (n,k)–derangements generated by algorithm PDGENLEX (n=4, k=0,2,4) No. α = δ(4, 0) δ(4, 4) δp (4, 2) F=∅ F={1,2,3,4} F={1,2} 1 1111 1111 1111 2 1112 1112 1112 3 1113 1113 1113 4 1114 1114 5 1121 1121 1121 6 1122 1122 1122 7 1123 1123 1123 8 1124 1124 9 1131 1131 10 1 1 3 2 1132 11 1 1 3 3 1133 1133 12 1 1 3 4 1134 13 1 2 1 1 14 1 2 1 2 1212 1212 15 1 2 1 3 16 1 2 1 4 17 1 2 2 1 1221 1221 18 1 2 2 2 19 1 2 2 3 20 1 2 2 4 21 1 2 3 1 22 1 2 3 2 23 1 2 3 3 24 1 2 3 4

δs (4, 2) F={3,4} 1111 1112 1113

δ(4, 2) F={1,3} 1111 1112 1113 1114 1121 1121 1122 1122 1123 1123 1124

1133

1133

1211 1212 1213

1 1 1 1 1

1221 1222 1223

1233

2 2 2 2 2

1 1 1 1 2

1 2 3 4 1

δ(4, 2) π1 F={2,4} 1111 234 1112 431 1113 241 231 1121 342 1122 314 1123 412 312 1131 413 1132 243 1133 214 213 324 1212 341 421 321 1221 432 1222 134 1223 142 132 423 1232 143 124 123

π2 1 2 3 4 1 2 3 4 2 1 3 4 1 2 3 4 1 2 3 4 1 2 3 4

4 3 3 3 4 2 2 2 2 4 2 2 4 3 3 3 4 1 1 1 4 1 1 1

1 4 1 1 3 4 3 3 4 1 1 1 2 4 2 2 3 4 3 3 2 4 2 2

2 2 4 2 1 1 4 1 3 3 4 3 1 1 4 1 2 2 4 2 3 3 4 3

3 1 2 4 2 3 1 4 1 2 3 4 3 2 1 4 1 3 2 4 1 2 3 4

steps (clock periods) per object in average, and partial derangements in even better rate. Given constant size of the generator producing output derangements DA takes a constant time per object. Both VLSI and FPGA implementations are feasible. Two exemplary permutation sequences π1 =< π1 (1), . . . , π1 (n) > and π2 =< π2 (1), . . . , π2 (n) > (represented by choice functions α for n = 4) generated by the algorithm PDGENLEX are shown in Table 1. Input : n – size of the set, k – the number of forbidden positions in partial derangements, D – the forbidden set. Output: Table DA with the consecutive partial derangements. Method: The first function α in the table PD is obtained in step 1. The generation method is based on a counting process in the table PD. In step 4.3 the output is produced if the condition (6) is fulfilled. Computations run until the last c.f. α is generated.

On Parallel Generation of Partial Derangements

1. 2. 3. 4.

225

/1-2 initialization phase/ for i:=1 to n do PD[i]:=I[i]:=1; MAX[i]:=i+1; i:=2; output (PD); repeat 4.1. PD[i]:=(PD[i]+1) mod MAX[i]; 4.2. if PD[i]=0 then 4.2.1. repeat 4.2.1.1. PD[i]:=PD[i]+1; 4.2.1.2. i:=i+1; 4.2.1.3. PD[i]:=(PD[i]+1) mod MAX[i]; until PD[i] = 0; 4.2.2. i:=2; 4.3. if condition (6) is satisfied then output(PD); until MAX—PD=I;

output(PD) /conversion and output/ 1. case of decomposition scheme left coset: permute the set DA={1, ... , n} according to transpositions in PD; right coset: permute the set DA={1, ... , n} according to transpositions in PD; 2. output DA; Fig. 2. The algorithm PDGENREVLEX

Computational complexity of the algorithm PDGENREVLEX is the same as PDGENLEX. Two exemplary permutation sequences π1 =< π1 (1), . . . , π1 (n) > and π2 =< π2 (1), . . . , π2 (n) > (represented by choice functions α for n = 4) generated by the algorithm PDGENREVLEX are shown in Table 2.

4

A Parallel Hardware Implementation

The circuit described in this section can be used for generation of partial derangements in parallel processing systems [9]. For the hardware implementation the algorithm PDGENREVLEX has been selected. Both decomposition schemes (3) and (4) have been shown to be group–theoretic representations of triangular permutation networks [16]. The triangular permutation network and its control circuit (complex parallel counter) are the main components of the hardware generator. The hardware complexity of the generator is O(n2 ), and the network propagation delay is O(n). For practical applications the networks size is limited and the propagation delay can be considered constant. The triangular permutation network is built of two–state cells (2–permuters) [12,13,16]. Each cell requires a separate control signal. The control circuit is organized in the following way. With every ith column of the triangular network (1 ≤ i ≤ n) the ith ring counter is associated, with the initial state from the ”1– out–of–i” code. All column counters form the parallel counter with n! different

226

Z. Kokosi´ nski

Table 2. Sequences of partial (n,k)–derangements generated by algorithm PDGENREVLEX (n=4, k=0,2,4) No. α = δ(4, 0) δ(4, 4) δp (4, 2) F=∅ F={1,2,3,4} F={1,2} 1 1111 1111 1111 2 1211 3 1121 1121 1121 4 1221 1221 1221 5 1131 1131 6 1231 7 1112 1112 1112 8 1212 1212 1212 9 1122 1122 1122 10 1 2 2 2 11 1 1 3 2 1132 12 1 2 3 2 13 1 1 1 3 1113 1113 14 1 2 1 3 15 1 1 2 3 1123 1123 16 1 2 2 3 17 1 1 3 3 1133 1133 18 1 2 3 3 19 1 1 1 4 1114 20 1 2 1 4 21 1 1 2 4 1124 22 1 2 2 4 23 1 1 3 4 1134 24 1 2 3 4

δs (4, 2) F={3,4} 1111 1211 1121 1221

δ(4, 2) F={1,3} 1111 1211 1121 1221

1 1 1 1

1 2 1 2

1 1 2 2

2 2 2 2

1112 1212 1122

1 1 1 1 1 1

1 2 1 2 1 2

1 1 2 2 3 3

3 3 3 3 3 3

1113 1213 1123 1133 1114 1214 1124

δ(4, 2) F={2,4} 1111 2 3 1121 3 1221 4 1131 4 4 1112 4 1212 3 1122 3 1222 1 1132 2 1232 1 1113 2 4 1123 4 1223 1 1133 2 1 2 3 3 1 2 1

π1 3 2 4 3 1 2 3 4 1 3 4 4 4 2 1 4 1 2 3 2 1 3 1 2

4 4 2 2 3 3 1 1 4 4 3 3 1 1 2 2 4 4 1 1 2 2 3 3

π2 1 1 1 1 2 1 2 2 2 2 1 2 3 3 3 3 3 3 4 4 4 4 4 4

4 4 4 4 2 4 3 3 2 1 4 1 3 3 2 1 2 1 3 3 2 1 2 1

1 2 3 3 4 2 4 4 4 4 1 4 1 2 3 3 1 2 1 2 3 3 1 2

2 1 1 2 3 3 2 1 1 2 3 3 4 4 4 4 4 4 2 1 1 2 3 3

3 3 2 1 1 1 1 2 3 3 2 2 2 1 1 2 3 3 4 4 4 4 4 4

states. Clock enable signal for the ith ring couter is a product of carry signals (overflows) from all ring counters preceding it. The asynchronous setup of each ring counter and global reset for all ring counters is provided. If the jth bit of the ith ring counter bji = 1, for 1 ≤ j ≤ (i − 1), then in the ith column of the network only one cell denoted by C[i, j] is activated to perform the corresponding transposition τij . If bii = 1, 1 ≤ i ≤ n, then all cells in the ith column are in the ”identity” state. After setting the initial state of the network the control circuit generates consecutive states of network in a constant time (one clock period) and the permutation network generates subsequent configurations representing permutations. Valid partial derangements are detected by a logic function V checking if the condition (6) is satisfied: V =

n−1  i=1

F V (i)vi =

n−1  i=1

F V (i)(bii +

n  j=i+1

bij ),

(8)

On Parallel Generation of Partial Derangements

227

where: F V – is a binary forbidden set vector: F V (i) = 1 iff position i is forbidden, otherwise F V (i) = 0; vi – is the function detecting if the condition (5) for the forbiden position i is satisfied; in fact, for the technological reason, vi should be rewritten to the form: vi = (bii + (((. . . (bii+1 + bii+2 ) + . . .) + bin−1 ) + bin ).

(9)

The above logic functions can be computed in O(n) time what matches the network propagation delay. Because the size of the network is limited and the constant factor hidden in the function O(n) is very low, for most applications we may assume that consecutive network configurations are generated in constant time. It is expected that generation of a single partial derangement takes in average at most e clock periods, when the whole set of permutations is generated. The proposed hardware generator of partial derangements for n=32 has been implemented in Xilinx Spartan 3 FPGA. For the project we have used VHDL, Xilinx Foundation ISE 8.2i software and a Digilent board with XC3S200-4ft256 programmable device.

5

Conclusions

The concept of partial derangements can gain much attention when application problems involve irregular permutation problems with partially forbidden positions. In that case known methods for generation of permutations and derangements are not directly applicable. The correctness of both generation algorithms has been verified by computer simulation. In addition the algorithm PDGENREVLEX has been implemented in VHDL and tested in a real FPGA device. It is not likely that the hardware generator will be used for enumeration of full sets of partial derangements for large values of n. However, fast generation of derangements, partial derangements and permutations can be of practical value for many specific tasks involving generation of subsequences of such objects.

Acknowledgements This work was supported by the research grant No. E–3/112/BW/07 from Cracow University Technology.

References 1. Akl, S.G.: A new algorithm for generating derangements. BIT 20, 2–7 (1980) 2. Akl, S.G., Calvert, J.M., Stojmenovic, I.: Systolic generation of derangements. In: Proc. Int. Workshop on Algorithms and Parallel VLSI Architectures II, pp. 59–70. Elsevier, Amsterdam (1992)

228

Z. Kokosi´ nski

3. Baril, J.–L., Vajnovszki, V.: Gray code for derangements. Discrete Applied Mathematics 140, 207–221 (2004) 4. Erickson, M.J.: Introduction to Combinatorics, vol. 83, pp. 119–120. Wiley Interscience, Chichester (1996) 5. Gupta, P., Bhattacharjee, G.P.: A parallel derangement generation algorithm. BIT 29, 14–22 (1989) 6. Graham, R.L., Knuth, D.E., Patashnik, O.: Concrete Mathematics, 2nd edn., pp. 194–195. Addison-Wesley Publishing Company, Reading (1994) 7. Hall, M., Paige, J.L.: Complete mapping of finite groups. Pacific J. Math., 541–549 (1955) 8. Hassani, M.: Derangements and applications. Journal of Integer Sequences 6, 1–8 (2003) 9. Kapralski, A.: Sequential and parallel processing in depth search machines. World Scientific, Singapore (1994) 10. Kim, D., Zeng, J.: A new decomposition of derangements. J. Comb. Theory, Ser. A 96, 192–198 (2001) 11. Knuth, D.: The art of computer programming, Pre–fascicles of ch. 7.2. Addison– Wesley, Reading (2001–2004) 12. Kokosi´ nski, Z.: On generation of permutations through decomposition of symmetric groups into cosets. BIT 30, 583–591 (1990) 13. Kokosi´ nski, Z.: Circuits generating combinatorial configurations for sequential and parallel computer systems. Cracow University of Technology, Cracow, Poland, Monograph 160 (1993) 14. Korsh, J.F., LaFolette, P.: Constant time generation of derangements. IPL 1990, pp. 181–186 (2004) 15. Mikawa, K., Semba, I.: Generating derangements by interchanging at most four elements. Systems and Computers in Japan 35(12), 25–31 (2004), available online from: www.interscience.wiley.com 16. Oru¸c, A.Y., Oru¸c, A.M.: Programming cellutar permutation networks through decomposition of symmetric groups. IEEE Trans. on Computers 36, 802–809 (1987) 17. Ruskey, F.: Combinatorial generation, Working version (1j–CSC 425/520) (2003), available at: www.edu/∼ruskey/book.pdf 18. Sedgewick, R.: Permutation generation methods. Computing Survey 9, 137–164 (1977)

Parallel Simulated Annealing Algorithm for Graph Coloring Problem Szymon Łukasik1 , Zbigniew Kokosiński2, and Grzegorz Świętoń2 1

2

Systems Research Institute, Polish Academy of Sciences, ul. Newelska 6, 01-447 Warsaw, Poland [email protected] Department of Automatic Control, Cracow University of Technology ul. Warszawska 24, 31-155 Cracow, Poland

Abstract. The paper describes an application of Parallel Simulated Annealing (PSA) for solving one of the most studied NP-hard optimization problems: Graph Coloring Problem (GCP). Synchronous master-slave model with periodic solution update is being used. The paper contains description of the method, recommendations for optimal parameters settings and summary of results obtained during algorithm’s evaluation. A comparison of our novel approach to a PGA metaheuristic proposed in the literature is given. Finally, directions for further work in the subject are suggested. Keywords: graph coloring, parallel simulated annealing, parallel metaheuristic.

1

Introduction

Let G = (V, E) be a given graph, where V is a set of |V | = n vertices and E set of |E| = m graph edges. Graph Coloring Problem (GCP) [1,2] is defined as a task of finding an assignment of k colors to vertices c : V → {1, . . . , k} , k ≤ n , such that there is no conflict of colors between adjacent vertices, i.e. ∀(u, v) ∈ E : c(u) = c(v) and number of colors k used is minimal (such k is called the graph chromatic number χ(G)). A number of GCP variants is used for testing printed circuits, frequency assignment in telecommunication, job scheduling and other combinatorial optimization tasks. The problem is known to be NP–hard [3]. Intensive studies of the problem resulted in a large number of approximate and exact solving methods. GCP was the subject of Second DIMACS Challenge [4] held in 1993 and Computational Symposium on Graph Coloring and Generalizations organized in 2002. The graph instances [5] and reported research results are frequently used in development of new coloring algorithms and for reference purposes. Most algorithms designed for GCP are iterative heuristics [11], such as genetic algorithms [6], simulated annealing [7,8], tabu or local search techniques [9], minimizing selected cost functions. At the time of this writing, the only parallel metaheuristic for GCP is parallel genetic algorithm [10,12,13,14,15]. R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 229–238, 2008. c Springer-Verlag Berlin Heidelberg 2008 

230

S. Łukasik, Z. Kokosiński, and G. Świętoń

The purpose of the paper is to present a new algorithm capable of solving GCP, developed on the basis of Parallel Simulated Annealing (PSA) method [16]. Recent years brought a rapid development of PSA techniques. Classical Simulated Annealing [17] was transformed into parallel processing environment in various ways: most popular approaches involve parallel moves, where single Markov chain is being evaluated by multiple processing units calculating possible moves from one state to another. The other method uses multiple threads for computing independent chains of solutions and exchanging the obtained results on a regular basis. Broad studies of both techniques can be found in [18,19]. The PSA scheme for GCP, proposed by the authors, includes above strategies with the rate of current solution update used as a distinctive control parameter. The paper is organized as follows. Next section is devoted to the description of the proposed PSA algorithm. Besides its general structure, details of cooling schedule and cost function being used as well as neighborhood solution generation procedure are given. Subsequent part of the paper presents results of algorithm’s experimental evaluation. Final part of the contribution gives general comments on the performance of PSA algorithm, and possible directions for future work in the subject.

2

Parallel Simulated Annealing

PSA algorithm for GCP introduced in this paper uses multiple processors working concurrently on individual chains and agreeing about current solutions at fixed iteration intervals. The aim of the routine is to minimize chosen cost function, with storing the best solution found. The coordination of the algorithm is performed in master–slave model – one of processing units is responsible for collecting solutions, choosing the current one and distributing it among slave units. The exchange interval ei is a parameter which decides which PSA scheme is being used. Setting ei = 1 is equivalent to producing single chain of solutions using multiple moves strategy. Increasing the interval ei leads to creating semiindependent chains on all slave processors starting at each of concurrent rounds with the same established solution. Setting ei to infinity results in performing independent simulated annealing runs. The general scheme of the algorithm is presented below: Parallel Simulated Annealing for Graph Coloring Problem if proc=master best_cost:=infinity; if proc=slave // generate initial solution randomly at each slave solution[proc]:=Generate_Initial_Solution(); for iter:=1 to iter_no do if proc=slave // each slave generates a new solution neighbor_solution[proc]:=Generate_Neighbor_Sol(solution[proc]);

PSA Algorithm for Graph Coloring Problem

231

// and accepts it as a current one according to SA methodology solution[proc]:=Anneal(neighbor_solution[proc],solution[proc],T); // all solutions are then gathered at master Gather_at_master(solution[proc]); if proc=master current_cost:=infinity; // find solution with minimum cost and set it as a current one for j:=1 to slaves_no do if Cost(solution[j])<current_cost current_solution:=solution[j]; current_cost:=Cost(solution[j]); // update best solution found (if applicable) if current_cost
Detailed information about our Simulated Annealing algorithm like cooling scheme, representation of the solution, method for generation of neighborhood and cost calculation will be given in following subsections. 2.1

Cooling Schedule

Choosing proper temperature schedule is crucial for algorithm based on Simulated Annealing methodology since it influence the acceptance probability of positive transitions (i.e. when a cost difference Δcost,i between generated neighbor and initial solution is positive) given by Metropolis [20]: P (Δcost,i ) = e

−

Δcost,i Ti

.

(1)

As a result of intensive studies in this area multiple cooling strategies were developed [21]. The cooling schedule used here is the exponential one: Ti+1 = αTi ,

(2)

where α is the cooling rate (usually set at 0.80–0.99 level [22]) for each cooling step. Every SA step consists of Mi iterations. For the exponential schedule following holds: Mi+1 = βMi . (3)

232

S. Łukasik, Z. Kokosiński, and G. Świętoń

In order to extend gradually SA runs at lower temperature levels constant β is chosen usually from the range [1.01, 1.20]. In addition to proper cooling schedule one has to choose correct initial temperature T0 . The authors used the most common method that involves calculating average cost difference Δcost,0 from a set of pilot runs consisting of positive transitions from an initial state. Preliminary temperature assuring desired initial acceptance probability P (Δcost,0 ) can be calculated afterwards from equation: T0 = −

Δcost,0 . ln P (Δcost,0 )

(4)

The alternative approach could follow a universal method for initial temperature selection introduced in [23]. 2.2

Solution Representation and Neighborhood Generation

A graph coloring c is represented by a sequence of natural numbers c =< c[1], . . . , c[n] >, c[i] ∈ {1, . . . , k}, which is equivalent to set partition representation with exactly k non–empty blocks. A rule for generation of a neighbor solution can be selected out of a wide range of existing methods [24]. For the purpose of the presented algorithm the following form of restricted 1-exchange neighborhood is used: Restricted 1-exchange neighborhood for Graph Coloring Problem // check for vertices with color conflicts conflict_vertices:=Find_Conflicting_Vertices(c); if sizeof(conflict_vertices)>0 do // if conflicts were found choose a random conflicting vertex vertex_to_change:=random(conflict_vertices); // and replace its color randomly with one of the colors {1, ... ,k+1} c[vertex_to_change]:=random(k+1); else // if no conflicts are found choose random vertex vertex_to_change:=random(n); // and replace its color randomly with one of the colors {1, ... ,k} c[vertex_to_change]:=random(k); return c;

2.3

Cost Assessment

As a quality measure of a selected coloring c the following cost function was used [13]:  f (c) = q(u, v) + d + k , (5) (u,v)∈E

where q – is a penalty function: q(u, v) =



2 when c(u) = c(v) 0 otherwise

(6)

PSA Algorithm for Graph Coloring Problem

d – is a coefficient for solution with conflicts:   1 when (u,v)∈E q(u, v) > 0  d= 0 when (u,v)∈E q(u, v) = 0

233

(7)

and k – is the number of colors used.

3

Experimental Evaluation

For testing purposes an implementation of the algorithm based on Message Passing Interface was prepared. All experiments with simulated parallelism were R carried out on Intel XeonTM machine. As test instances standard DIMACS graphs, obtained from [5], were used. For experiments following values of SA control parameters were chosen: α = 0.95 and β = 1.05. Initial temperature was determined from a pilot run consisting of 1% (relative to overall iteration number) positive transitions. The termination condition was either achieving the optimal solution or the required number of iterations. Due to space limitations only most representative results are presented in the paper. The full set of simulation data can be found on the first author’s web site (http://www.pk.edu.pl/~szymonl). 3.1

SA Parameters Settings

At first, the optimal values of SA parameters were under investigation. Essential results of those experiments are gathered in Table 1. Opening set of 500 runs with final temperature Tf = 0.1, iter_no = 10000, randomly generated initial solution with k = χ(G) and the initial probability Table 1. Influence of Simulated Annealing parameters on algorithm’s performance Graph P (Δcost,0 ) Tf k0 G(V,E) Description Results Best P Results Best Tf Results Best k0 anna, χ(G) = 11 best f(c) 11.12 70% 11.00 0.04 · T0 11.12 χ(G) |V | = 138 avg. f(c) 11.32 11.14 11.17 |E| = 493 σf (c) 0.29 0.21 0.05 queen8_8, χ(G) = 9 best f(c) 11.33 60% 10.60 0.06 · T0 11.33 χ(G) |V | = 64 avg. f(c) 11.46 11.84 11.41 |E| = 728 σf (c) 0.10 1.26 0.05 mulsol.i.4, χ(G) = 31 best f(c) 38.23 80% 31.03 0.2 · T0 37.63 χ(G) − 5 |V | = 197 avg. f(c) 38.66 33.65 38.22 |E| = 3925 σf (c) 0.46 1.64 0.36 myciel7, χ(G) = 8 best f(c) 12.95 60% 8.00 0.2 · T0 11.38 χ(G) − 5 |V | = 191 avg. f(c) 13.22 9.47 12.93 |E| = 2360 σf (c) 0.34 1.60 0.96

234

S. Łukasik, Z. Kokosiński, and G. Świętoń

changing within the range 10%–90% proved that the best algorithm’s performance, measured primarily by minimum average cost function (the second criterion was the iteration number), is achieved for high initial probabilities with an optimum found at about 60%–80%. It was observed, however, that the exact choice of P (Δcost,0 ) in this range is not very significant for the overall algorithm’s performance. Obtained results confirmed a hypothesis that for more complex problems it is advisable to use higher values of initial temperature. In the next experiment the optimal final temperature (relative to T0 ) was under examination. For fixed P (Δcost,0 ) = 70%, iter_no = 10000 and k = χ(G) the best results were obtained for Tf ∈ [0.01, 0.2]·T0 . Again, higher solution quality for more complex graph instances was achieved with increased temperature ratios. The influence of initial number of colors k0 on the solution quality was also determined experimentally. The range [χ(G) − 5, χ(G) + 5] was under consideration with P (Δcost,0 ) = 70%, Tf = 0.05·T0 and iter_no = 10000. It was observed that using initial color number slightly different than chromatic number do not affect significantly the algorithm’s performance. For some graph instances it is even recommended to start with colorings with k0 lower than χ(G). In the end it should be noted that above presented statements are to be treated as overall guidelines for SA parameters settings obtained from a relatively small set of graphs. The exact values for those parameters depend largely on the considered class of graph instances. 3.2

Influence of Parallelization Schemes

The second stage of the computing experiments involved examination of algorithm’s performance with different parallelization schemes and comparison with results obtained with sequential Simulated Annealing algorithm. For PSA the configurations with ei = {1, 2, 4, 6, 8, 10, ∞} and slaves number from 2 to 18, were tested with various graph instances. To examine the effect of parallelization on the processing time the same number of iterations iter_no = 100000 was set for both sequential SA and PSA algorithms (in PSA each slave performs only iter_no/slaves_no iterations). For the temperature schedule following settings were applied: P (Δcost,0 ) = 70% and Tf = 0.05 · T0 . Obtained results include mean values of the cost function, the number of conflict–free/optimal solutions, the number of iterations needed to find an optimal coloring (if applicable), and algorithm’s execution time t[s] (until best solution has been found). The summary of the results is presented in Table 2. Best and worst parallel configurations, in terms of average f (c) and processing time, with the obtained results are reported. As a reference average performance of the algorithm is given as well. PSA clearly outperforms the sequential Simulated Annealing in terms of computation time. Moreover, applying parallelization improves the quality of the obtained solution. It can be seen though that it is important to select a proper configuration of the PSA algorithm to achieve its high efficiency. For most problem instances it is advisable to use multiple moves strategy with

PSA Algorithm for Graph Coloring Problem

235

Table 2. Experimental evaluation of the PSA algorithm for GCP Graph G(V,E) games120 χ(G) = 9 |V | = 120 |E| = 638 anna χ(G) = 11 |V | = 138 |E| = 493 myciel7 χ(G) = 8 |V | = 191 |E| = 2360 miles500 χ(G) = 20 |V | = 128 |E| = 1170 mulsol.i.4 χ(G) = 31 |V | = 197 |E| = 3925 queen8_8 χ(G) = 9 |V | = 64 |E| = 728 le450_15b χ(G) = 15 |V | = 450 |E| = 8169

Description avg. f(c) c.–f. c /opt. c avg. iter. /opt. c avg. t[s] /best c avg. f(c) c.–f. c /opt. c avg. iter. /opt. c avg. t[s] /best c avg. f(c) c.–f. c /opt. c avg. iter. /opt. c avg. t[s] /best c avg. f(c) c.–f. c /opt. c avg. iter. /opt. c avg. t[s] /best c avg. f(c) c.–f. c /opt. c avg. iter. /opt. c avg. t[s] /best c avg. f(c) c.–f. c /opt. c avg. iter. /opt. c avg. t[s] /best c avg. f(c) c.–f. c /opt. c avg. iter. /opt. c avg. t[s] /best c

SA PSA Results PSA Config. Results Best Worst Average Best Worst 9 9 9 9 7 slaves 18 slaves 100/100 100/100 100/100 100/100 ei = 1 ei = ∞ 477 78 258 176 0.72 0.05 0.40 0.14 11 11 11.32 11.02 4 slaves 18 slaves 100/100 100/100 100/72 100/98 ei = 1 ei = 1 5821 199 1177 462 1.31 0.08 1.73 0.31 8 8 8.66 8.05 3 slaves 18 slaves 100/100 100/100 100/43 100/95 ei = 1 ei = 1 7376 797 1524 1539 1.85 0.20 1.83 1.03 20 20 20.1 20.01 6 slaves 17 slaves 100/100 100/100 100/90 100/98 ei = 1 e1 = 1 38001 544 422 2842 4.71 0.21 0.58 1.06 31.04 31.19 38.24 34.74 2 slaves 17 slaves 100/96 100/81 100/0 100/1 ei = ∞ ei = 1 19007 15908 - 13451 4.30 1.83 2.64 2.08 9.97 9.81 10.05 9.97 5 slaves 16 slaves 100/3 100/19 100/0 100/3 ei = 1 ei = ∞ 66488 8831 8820 1.66 0.64 3.82 1.65 18.58 17.39 21.79 18.71 9 slaves 18 slaves 100/0 100/0 100/0 100/0 ei = 1 ei = ∞ 42.88 3.54 6.47 4.99

optimal, relatively small, number of slaves. There exists one exception to the presented statement - for the class of mulsol.i graphs significantly better results were obtained with fully independent SA runs. The worst results of using Parallel Simulated Annealing were obtained when a high number of slaves was involved in the computations and parallelization scheme was far from the optimal one. 3.3

Comparison with Parallel Genetic Algorithm

The last stage of the testing procedure involved comparison of time efficiency of the PSA algorithm and Parallel Genetic Algorithm introduced in [12]. The implementation of the PGA for GCP used in [13] was applied. Both algorithms were executed on the same machine for selected DIMACS graph instances and computation time needed to find optimal coloring was reported. PGA was

236

S. Łukasik, Z. Kokosiński, and G. Świętoń

executed with 3 islands, subpopulations consisting of 60 individuals, migration rate 5, migration size 5 with the best individuals being distributed, initial number of colors 4 and operators: CEX crossover (with 0.6 probability), First–Fit mutation (with 0.1 probability). For PSA 3 slaves were used, P (Δcost,0 ) = 70%, Tf = 0.05 · T0 , iter_no = 300000 (for instance mulsol.i.1 to find optimal solution runs of length 1500000 iterations were needed). For most instances multiple moves strategy was applied. One exception was mulsol.i class of graphs where, according to earlier observations, independent SA runs were executed. Results of the experiments, enclosed in Table 3, clearly demonstrate that PSA performance is comparable to the one achieved by the PGA. For some graph instances, like book graphs and miles500, the proposed algorithm was found to be superior. On the other hand, there exist a group of problems relatively easy to solve by PGA and, at the same time, difficult to solve by PSA (like mulsol.i.1). Table 3. Comparison of time efficiency of PGA and PSA algorithms applied for GCP Graph G(V,E) anna, χ(G) = 11 |V | = 138, |E| = 493 myciel7, χ(G) = 8 |V | = 191, |E| = 2360 miles500, χ(G) = 20 |V | = 128, |E| = 1170

4

t[s] Graph PSA PGA G(V,E) 0.23 0.35 mulsol.i.4, χ(G) = 31 |V | = 185, |E| = 3946 0.34 0.25 mulsol.i.1, χ(G) = 49 |V | = 197, |E| = 3925 0.48 18.0 games120, χ(G) = 9 |V | = 120, |E| = 638

t[s] PSA PGA 2.89 1.99 14.9 4.47 0.20 0.34

Conclusion

In the paper a new Parallel Simulated Annealing algorithm for GCP was introduced and evaluated. First experiments revealed that its performance depends on choosing a cooling schedule and generation of the initial coloring suitable to the considered problem. Some general guidelines were derived for the algorithm’s settings that ensure a better solution quality. Further research in the subject could concern adaptive cooling schedules and generation of initial solution by means of an approximate method. Choosing an optimal number of processing units and parallelization scheme for the PSA was also under consideration. We found that problem specification essentially influence the proper choice of these elements. However, it can be stated as a general remark that the highest efficiency of the master–slave PSA algorithm is achieved for optimal, relatively small number of slaves. During the performance evaluation the PSA algorithm was proved to be an effective tool for solving Graph Coloring Problem. The experiments showed that it achieves a similar performance level as PGA. The comparison results of both methods showed that none of them is superior. It encourages efforts for development of a new hybrid metaheuristics which would benefit from advantages of

PSA Algorithm for Graph Coloring Problem

237

both PGA and PSA approaches. The overall concept of a hybrid algorithm could implement the idea presented in [25] and include some other improvements of the standard PSA scheme as proposed in [26].

References 1. Kubale, M. (ed.): Graph colorings. American Mathematical Society (2004) 2. Jensen, T.R., Toft, B.: Graph coloring problems. Wiley Interscience, New York (1995) 3. Garey, R., Johnson, D.S.: Computers and intractability. A guide to the theory of NP–completeness. W. H. Freeman, New York (1979) 4. Johnson, D.S., Trick, M.A. (eds.): Cliques, Coloring and Satisfiability. DIMACS Series in Discrete Mathematics and Theoretical Science, vol. 26 (1996) 5. Graph coloring instances: http://mat.gsia.cmu.edu/COLOR/instances.html 6. Fleurent, C., Ferland, J.A.: Genetic and Hybrid Algorithms for Graph Coloring. Annals of Operations Research 63, 437–461 (1996) 7. Chams, M., Hertz, A., de Werra, D.: Some Experiments with Simulated Annealing for Coloring Graphs. European Journal of Operational Research 32, 260–266 (1987) 8. Johnson, D.S., Aragon, C.R., McGeoch, L., Schevon, C.: Optimization by Simulated Annealing: An Experimental Evaluation; Part II, Graph Coloring and Number Partitioning. Operations Research 39(3), 378–406 (1991) 9. Galinier, P., Hertz, A.: A Survey of Local Search Methods for Graph Coloring. Computers & Operations Research 33, 2547–2562 (2006) 10. Alba, E. (ed.): Parallel Metaheuristics. John Wiley & Sons, New York (2005) 11. Szyfelbein, D.: Metaheuristics in graph coloring. In: Kubale, M. (ed.) Discrete optimization. Models and methods for graph coloring, WNT, Warszawa, pp. 26–52 (2002) (in Polish) 12. Kokosiński, Z., Kołodziej, M., Kwarciany, K.: Parallel genetic algorithm for graph coloring problem. In: Bubak, M., van Albada, G.D., Sloot, P.M.A., Dongarra, J. (eds.) ICCS 2004. LNCS, vol. 3036, pp. 215–222. Springer, Heidelberg (2004) 13. Kokosiński, Z., Kwarciany, K., Kołodziej, M.: Efficient Graph Coloring with Parallel Genetic Algorithms. Computing and Informatics 24(2), 109–121 (2005) 14. Łukasik, S.: Parallel Genetic Algorithms using Message Passing Paradigm for Graph Coloring Problem. Journal of Electrical Engineering 56(12s), 123–126 (2005) 15. Kokosiński, Z., Kwarciany, K.: On sum coloring of graphs with parallel genetic algorithms. In: Beliczynski, B., Dzielinski, A., Iwanowski, M., Ribeiro, B. (eds.) ICANNGA 2007. LNCS, vol. 4431, pp. 211–219. Springer, Heidelberg (2007) 16. Azencott, R. (ed.): Simulated Annealing: Parallelization Techniques. John Wiley & Sons, New York (1992) 17. Kirkpatrick, S., Gelatt, C.D., Vecchi, M.: Optimization by Simulated Annealing. Science 220, 671–680 (1983) 18. Diekmann, R., Lüling, R., Simon, J.: Problem Independent Distributed Simulated Annealing and its Applications. Working Paper, University of Paderborn, Germany 19. Lee, S.-Y., Lee, K.G.: Synchronous and Asynchronous Parallel Simulated Annealing with Multiple Markov Chains. IEEE Transactions on Parallel and Distributed Systems 7(10), 993–1008 (1996) 20. Metropolis, N.A., Rosenbluth, A., Teller, A., Teller, E.: Equation of State Calculations by Fast Computing Machines. Journal of Chemical Physics 21, 1087–1092 (1953)

238

S. Łukasik, Z. Kokosiński, and G. Świętoń

21. Ingber, A.L.: Simulated Annealing: Practice versus Theory. Journal of Mathematical Computation Modelling 18(11), 29–57 (1993) 22. Sait, S.M., Youssef, H.: Iterative computer algorithms with applications in engineering. IEEE Computer Society, Los Alamitos (1999) 23. Ben-Ameur, W.: Computing the Initial Temperature of Simulated Annealing. Computational Optimization and Applications 29, 369–385 (2004) 24. Avanthay, C., Hertz, A., Zufferey, N.: A Variable Neighborhood Search for Graph Coloring. European Journal of Operational Research 151, 379–388 (2003) 25. Tomoyuki, H., Mitsunori, M., Ogura, M.: Parallel Simulated Annealing using Genetic Crossover. Science and Engineering Review of Doshisha University 41(2), 130–138 (2000) 26. Bożejko, W., Wodecki, M.: The New Concepts in Parallel Simulated Annealing Method. In: Rutkowski, L., Siekmann, J.H., Tadeusiewicz, R., Zadeh, L.A. (eds.) ICAISC 2004. LNCS (LNAI), vol. 3070, pp. 853–859. Springer, Heidelberg (2004)

Parallel Algorithm to Find Minimum Vertex Guard Set in a Triangulated Irregular Network Masoud Taghinezhad Omran Sharif University of Technology [email protected]

Abstract. This paper presents a new serial algorithm for selecting a nearly minimum number of vertex-guards so that all parts of a geographical surface modeled by a TIN (Triangulated Irregular Networks) is covered. Our algorithm selects fewer guards than the best existing algorithms on the average. Based on this approach, a new coarse-grain parallel algorithm for this problem is proposed. It has been showed that the upper bound for total number of guards, selected by this algorithm, is  2n  where n is number of vertices in the TIN. Average case analysis 3 and implementation results show that in real TINs even fewer than  n2  guards (proved upper bound of needed guards in worse-case) are selected by our serial and parallel algorithms. Keywords: Parallel Computing, Art Gallery Problem, Guarding Problem, MPI, Graph Partitioning, Approximation Algorithms.

1

Introduction

Finding the minimum number of light sources to illuminate a polygon is NPHard [6]. However, it has been shown that illuminating an n-sided simple polygon needs at most  n3  light sources, and this number is sometimes necessary [5,8]. For different kinds of guarding problem in 2 dimension, several algorithms have been proposed [12,16,17]. The problem has also been considered for 2 12 -dimensional surfaces [7]. Bose et al. [3] showed that the upper bound for number of guards, needed to cover the entire surface of an n-vertex terrain when it is triangulated and guards are placed on vertices, is  n2  and in some situations this number of guards is necessary. Triangulated Irregular Network is a good model to represent a geographical terrain and guarding such terrain has important applications. The amount of data for such problem is huge and thus the use of parallel processing is justifiable. In this paper, a new form of guarding problem in TINs is proposed that behaves better on the average compared to the best exiting algorithm. Based on this approach, a parallel algorithm for this problem is proposed. The parallel algorithm has been implemented on a 24-node PC-cluster running MPI. The experimental data will be presented in this paper. R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 239–248, 2008. c Springer-Verlag Berlin Heidelberg 2008 

240

2

M. Taghinezhad Omran

Guarding 3D Terrain

The problem of guarding a polyhedral terrain is to find the smallest set of points on the terrain such that every point on the terrain is visible from at least one point of the set. What we mean by terrain is usually a geographic surface in 3D space that no two different points of it with the same x and y values, has the same value for z where < x, y, z > is the Cartesian Coordinate of points. These surfaces are called 2 12 -dimensional surfaces. To become processable by computer, representing data of the terrain should be presented in a discrete form. Therefore, the 3D region is approximated by small triangles which is called Triangulated Irregular Network (TIN). To reduce the complexity in the guarding problem, the height of nodes is ignored. As a result, each guard, placed on a vertex of the TIN, can only see its direct adjacent triangles. The Upper bound of needed guards in this case is similar to the case that the height is considered and equal to  n2 .

3

Serial Algorithms for Guarding TINs

For guarding problem on TINs , two different approximation algorithms were posed that we will briefly go through them here. Guarding Algorithm Using Graph Coloring: It has been shown that every planar graph is five colorable. According to this theorem, it is possible to color the entire vertices of a planar graph using five colors in linear time. Algorithm: Guarding the TIN using 5-coloring 1. Color the TIN graph using five colors. 2. Among five assigned colors select three least used colors and place a guard on vertices, colored by one of these colors. Placing guards as mentioned above ensures that all triangles will have at least one guard on their vertices at the end. It has been shown that needed time for both phases is O(n). This algorithm uses fewer than  3n 5  guards for an n-vertex graph. More details of this algorithm is available in [3]. Worse-Case-Optimal Guarding Algorithm: In every TIN, the upper bound for total number of guards, needed in worse case, is  n2 . Bose et al. posed a worsecase-optimal algorithm which selects at most  n2  guards [2]. This algorithm is based on finding maximum matching on TINs for which a complex linear time algorithm exists [1]. Algorithm: Worse-Case-Optimal Guarding Algorithm 1. Find the dual graph G∗ related to the TIN graph G. 2. Calculate maximum matching M ∗ in G∗ graph. 3. Construct G by eliminating edges dual to M ∗ from graph G. As G is a bipartite graph, it is possible to partition it into two separate sets of nodes; A and B. 4. Between A and B, select smaller set as set of guards.

Parallel Algorithm to Find Minimum Vertex Guard Set

241

Each phase of this algorithm has linear time and it is clear that the upper bound of total guard number, selected by this algorithm, is  n2 .

4

New Serial Algorithm for Guarding TINs

Average case analysis and implementation results show that both serial guarding algorithms, mentioned in previous section, select near n2 guards in real TINs. Even in conditions that fewer than  n2  guards are sufficient for guarding a TIN, in contrast to their upper bound, usually the first algorithm selects fewer guards than the worse-case-optimal algorithm. Hence, existing serial guarding algorithms are not such efficient in real TINs. We suggest a new guarding algorithm that can select fewer guards than existing algorithms on the average. In new guarding algorithm, the main concept is selecting a vertex at each step and putting guard on it. By placing a guard on a vertex, all triangles around the selected vertex are covered and it is possible to remove them form the TIN. Each vertex that has no triangle around it is removed from the TIN. Algorithm: New Serial Guarding Algorithm 1. In each step, if there exist a vertex with one or two surrounding triangles in the TIN then (a) In the case that A is a vertex having one triangle around it, and B and C are its adjacent vertices, if B and C has the same number of surrounding triangles, select one of them as a vertex guard. Otherwise, between B and C select the vertex that has more surrounding triangles as a vertex guard. Remove the selected vertex along with its surrounding triangles from the TIN. (b) In the case that A is a vertex having two surrounding triangles, and B, C, and D are its adjacent vertices, among B, C, and D select the vertex, adjacent to both triangles around A, as a vertex guard and remove it along with its surrounding triangles from the TIN. 2. If there is no vertex with one or two surrounding triangles in the TIN, select a vertex that has at least four surrounding triangles and remove it from the TIN along with its surrounding triangles. Removing the vertex and triangles should not create hole in the TIN. Continue from the first phase. To prove that the new algorithm can guard the whole TIN properly, we should show that it continues until all triangles are removed from the TIN. First, we should show that when there exist no vertex, satisfying conditions of phase one, it is possible to find a vertex having conditions of phase two. Also, we should show that when there exist no vertex, satisfying conditions of phase two, the algorithm can continue to the end through phase one. In following theorems we show that these conditions are satisfied. Theorem 1. While there exist any vertex in the TIN, having at least four surrounding triangles, it is possible to remove a vertex with at least four surrounding triangles in a way that no hole is created in the TIN.

242

M. Taghinezhad Omran

Fig. 1. Different cases of the boundary node (j) and the interior node (i)

Fig. 2. Case that the boundary node (j) has three surrounding triangles

Proof. We suppose that there exist one vertex or more in the TIN that have at least four surrounding triangles and it is not possible to remove a vertex with at least four surrounding triangles in a way that no hole is created in the TIN. Consider the case that a boundary vertex (j) is connected to an interior vertex (i). Different cases are as follows: If vertex j has at least four surrounding triangles, then it can be removed from the TIN and it is a contradiction to our first supposition.(Figure 1 A) If vertex j has three surrounding triangles (Figure 1 B) and if vertex i or k has more than three triangles around it, by removing each of them the boundary remains still connected. So, it is a contradiction. So, i and k should have less than four surrounding triangles. as a result a type of TIN is produced that we call it B-type TIN (Figure 2 A). The produced TIN can extend only through vertices m and n (Figure 2 B). In the case that vertex j has two triangles around it (Figure 1 C), if vertex i has more than three surrounding triangles, again it is a contradiction to our first supposition. If vertex i has less than three surrounding triangles, as i is an interior vertex, it should have exactly three surrounding triangles. We refer to this kind of TIN as A-type TIN (Figure 3 A). Also, B-type TIN can be produced in condition that n and m have three surrounding triangles two of them in common (Figure 3 B). The produced TIN can extend only through n and m (Figure 3 C). As a result, if we can not remove a vertex with at least four surrounding triangle then there exist no such vertex in the TIN which is a contradiction to our first supposition.  Proposition 1. In triangulated simple polygons that each vertex has fewer than four surrounding triangles, at least three vertices have fewer than three (1 or 2) surrounding triangles.

Parallel Algorithm to Find Minimum Vertex Guard Set

243

Fig. 3. Case that the boundary node (j) has two surrounding triangles

Theorem 2. When there exist only vertices with fewer than four surrounding triangles in the TIN, new guarding algorithm can continue until all triangles are removed. Proof. As a result of ”Theorem 1”, after removing vertices with more than three surrounding triangles, only some A and B-type TINs and some simple polygons remain in the TIN. These items may have one common point with each other but they can not form a hole in the TIN because in both phases of the algorithm we prevented creating holes. If Pi be each A or B-type TIN or simple polygon, considering that there is no hole in the TIN, there is a chain of Pi s in which exist at least two ending heads. We consider one of these heads and name it Ph . Each Pi has at least two vertices with fewer than three surrounding triangles (According to the shape of A and B-type TINs and from ”Proposition 1”). So, Ph has at least a vertex with fewer than three surrounding triangles. This ensures that the algorithm can begin to remove a vertex. On the other hand, by removing a vertex as mentioned in the algorithm, either the whole Ph is removed or a simple polygon remains. In both cases the algorithm can continue by next Pi which is the new ending head of the chain. Consequently, the alghorithm can continue until the whole TIN is Removed.  Theorem 3. The presented guarding algorithm completely covers the TIN by selected guards. Proof. As a result of ”Theorem 1 and 2” our new guarding algorithm can correctly cover the whole TIN with assigned guards.  Theorem 4. New guarding algorithm selects at most  2n 3  guards to cover the whole TIN. Proof. We consider three cases: 1) If a vertex with one surrounding triangle exists and it is a single triangle on the plane, by removing a vertex, totally three vertices and one triangle are removed from the TIN. Also, when adjacent vertices have more than one surrounding triangle, at least two vertices and two triangles are removed. 2) When a vertex with two surrounding triangles exists, by removing respective vertex, at least two triangles and two vertices are removed from the TIN.

244

M. Taghinezhad Omran

3) When a vertex is removed according to phase two of the algorithm, at least one vertex and four surrounding triangles are removed from the TIN. Consider following assumptions: i: Number of vertices that by removing them, four or more triangles are removed. j: Number of vertices that by removing them, one triangle is removed. k: Number of vertices that by removing them, two or three triangles are removed. We know that in every TIN ”T = 2n−t−2”, where T is total number of triangles, t is number of boundary vertices, and n is total number of vertices. Then: i + 3j + 2k ≤ n ⇒ 2i + 6j + 4k ≤ 2n 4i + j + 2k ≤ 2n − t − 2 ⇒ 4i + j + 2k ≤ 2n

(1) (2)

(1) + (2) ⇒ 6i + 7j + 6k ≤ 4n ⇒ 6i + 6j + 6k ≤ 4n ⇒ i + j + k ≤

2n 3

One guard is assigned for each i, j, and k kind of vertex. So, the new algorithm selects fewer than  2n  3  guards in every TIN. The new algorithm is a linear time algorithm. This can be inferred directly from the algorithm. Each vertex is visited once either at the beginning and constructing the data-structure or in the middle of the algorithm when removing other vertices. Also, a vertex is removed either because of putting guard on it or by putting guard on one of its neighbor vertices. So, each vertex is inserted to and removed from the data-structure only once. By using DCEL scheme for representing the TIN the algorithm defines vertex guards in linear time. Experimental results show that our new serial guarding algorithm can select fewer guards than other existing algorithms on the average. We have executed our implementation on different TINs and in all cases we received better results than old guarding algorithms. For example, we tested our algorithm on different TINs, needed fewer than 2n 5 guards to be covered. While other algorithms selected near n guards, our new algorithm assigned about 2n 2 5 guards to cover TINs. Implementations show that the new algorithm runs most of the times in phase one because when there exists a vertex satisfying conditions of phase one, by removing proper vertex from the TIN, it is very probable that other vertices with one or two surrounding triangles appear in the TIN. As a result, nearly in most steps of the algorithm at least two vertices are removed from the TIN. Also, there are many cases that more than two vertices are removed in each step of the algorithm (e.g. single A-type TIN on the plane). This ensures that fewer than  n2  guards will be selected by this algorithm on the average.

5

Parallel Algorithm for Guarding TINs

One of the most important applications of guarding problem is finding guards on geographic regions which have been modeled by TINs. In such problem, the size of data is so huge that processing by single processor has some difficulties.

Parallel Algorithm to Find Minimum Vertex Guard Set

245

Based on our new serial algorithm, we have presented a new coarse-grained parallel algorithm which can identify guards in less time than the serial algorithm and can remove processing difficulties that emerge when using single processor. The new parallel algorithm is a coarse-grained algorithm in which our new serial algorithm is used on each processor parallel to other processors. To prevent conflicts between assigned guards of each processor, we have used a preprocessing phase to separate partitions of the TIN. so, there will be no need for extra work at the end of the algorithm for merging the results. Algorithm: New Parallel Guarding Algorithm 1. Partitioning phase. One of the processors (e.g. processor P 0) partitions the TIN graph into P (number of processors) parts. In this partitioning scheme, a set of connected vertices is selected as graph partitioner nodes and no hole should be created in the TIN. 2. Preprocessing Phase. Processor P 0 removes each partitioner vertex satisfying two following conditions in each step until no vertex of this kind exists in the TIN: 1) Removing the vertex does not create hole in the plane. 2) It causes at least four triangles to be removed form the TIN. 3. Parallel Phase. Processor P 0 sends remaining data of triangles to relevant processors. Each processor finds guards in its part using new serial guarding algorithm and sends back the result to processor P 0. 4. Merging Phase. Processor P 0 puts received data from each processor together to obtain the total result. After phase two of the algorithm, each processor selects guards according to our new serial guarding algorithm on remaining data. In ”Theorem 3” it has been showed that the new serial algorithm covers the whole TIN correctly by selected guards. So, ”Theorem 5” can be posed as follows: Theorem 5. The new parallel algorithm correctly covers the whole TIN by selected guards.  Now, we show that the upper bound for selected guards by parallel algorithm is the same as the new serial algorithm and equal to  2n 3 . Proposition 2. No two directly adjacent interior vertices of a TIN, has exactly three surrounding triangles and at least one of them should have more than three surrounding triangles. Theorem 6. For every partitioning path in the graph of the TIN that begins from a boundary node and ends with another boundary node, it is possible to remove vertices of the path that have at least four surrounding triangles by beginning from a boundary node until reaching to the other side boundary node. It can be done in a way that does not create hole in the graph of the TIN. Proof. Suppose that i is a boundary node of the TIN and j is an interior node connected to it both on the partitioning path. Following cases are possible:

246

M. Taghinezhad Omran

Fig. 4. Different cases of a boundary node (i) connected to an interior node (j)

Fig. 5. Different types of partitioning path

In the case that vertex i has more than three surrounding triangles, by its removing, no hole appears in the TIN (Figure 4 A). If vertex i has exactly three surrounding triangles (Figure 4 B) and j has at least four surrounding triangles, by removing j, more than three triangles are removed and the boundary remains connected. Otherwise, if vertex j has three surrounding triangles, partitioning path can extend through m or n. If it extends through the boundary vertex m, different conditions should be verified from this vertex. Otherwise, if j is connected to the interior node n, according to ”Proposition 2”, vertex n should have more than three surrounding triangles. So, by removing it the boundary remains connected. When two triangles are around vertex i (Figure 4 C), if vertex j has more than three surrounding triangles, its removing causes more than three triangles to be removed without creating a hole in the TIN. In the case that j has three surrounding triangles, partitioning path should continue through boundary vertex m or n. So, different conditions should be verified from these nodes. By considering different cases mentioned above, in each step, it is possible to remove a vertex with at least four surrounding triangles in the partitioner path without creating hole in the TIN until reaching to the other side boundary node.  Theorem 7. After removing vertices with at least four surrounding triangles along the partitioning path, P (number of processors) sub-graphs have at most P common points and at most three vertices can take part in each common point. Proof. Partitioning path can have one of following forms: 1) Partitioning path begins from and ends to a boundary vertex (Figure 5 A). 2) Partitioning path begins from a boundary vertex and ends to a vertex in the partitioning path (Figure 5 B).

Parallel Algorithm to Find Minimum Vertex Guard Set

247

Fig. 6. Two sub-graphs having common point

3) Partitioning path begins from and ends to a vertex in partitioning path (Figure 5 C). At the end points of partitioning path it is possible that some vertices with fewer than four surrounding triangles remain from the partitioning path. As we always begin from a boundary vertex at most P common points will be created among sub-graphs after removing qualified vertices from the TIN. Now, we show that in each common point at most three vertices can participate. We suppose that two sub-graphs have more than three common vertices (Figure 6). By supposing that i is a boundary node at the end of partitioning path, no other vertex near i in the path can be a boundary vertex because there would be a shorter partitioning path which we assumed does not exist. So, vertices j and m should be interior nodes (Figure 6). This is not possible because according to ”Proposition 2” one of these vertices should have at least four surrounding triangles and it is a contradiction. So, sub-graphs can have at most three common vertices at their common point.  By paying attention to our new serial and parallel algorithms, it is clear that removed vertex in phase two of the parallel algorithm is similar to the vertex, removed in phase two of the serial algorithm. So, we can consider vertices, removed in phase two of the parallel algorithm, as a part of those, removed in phase two of the serial algorithm. Remaining part of the parallel algorithm is similar to its serial counterpart done in separate sections. We can also ignore common points and extra guards they add sometimes because according to ”Theorem 7”, their sum has constant value. Now, we can present the following theorem. Theorem 8. The upper bound for selected guards by new parallel algorithm is  2n  3  in all TINs. We have implemented our new parallel algorithm using MPI (Message Passing Interface) environment. In MPI, communication between processors is done by sending messages through an interconnected network. More details on MPI and programming using MPI can be obtained from [13,15]. The parallel algorithm has been implemented on a 24-node PC-cluster running MPI. Each computer had 2.4 MHz P4 processor and 512 MB of physical memory. Different graph partitioning schemes exist [9] and among them we used METIS [11] mesh partitioner to partition the input TIN. ”Figure 7” shows the speedup, obtained by multiple runs on different number of processors.

248

M. Taghinezhad Omran

Fig. 7. The speedup obtained by the new parallel algorithm

References 1. Biedl, T., Bose, P., Demaine, E., Lubiw, A.: Efficient algorithms for Petersen’s Matching Theorem. In: Proc. of Symp. on Discrete Algorithms, pp. 130–139 (1999) 2. Bose, P., Kirkpatrick, D., Li, Z.: Worse-Case-Optimal Algorithms for Guarding Planar Graphs and Polyhedral Surfaces (1999) 3. Bose, P., Shermer, T., Toussaint, G., Zhu, B.: Guarding polyhedral terrains. Computational Geometry - Theory and Applications 7(3), 173–185 (1997) 4. Chiba, N., Nishizeki, T., Saito, N.: A linear five coloring algorithm for planar graphs. Journal of Algorithms (1981) 5. Chv´ atal, V.: A combinatorial theorem in plane geometry. Journal of Comb. Theory Ser. B 18, 39–41 (1975) 6. Cole, R., Sharir, M.: Visibility Problems for Polyhedral terrains. Journal of Symbolic Computation 7, 11–30 (1989) 7. de Floriani, L., Falcidieno, B., Pienovi, C., Allen, D., Nagy, G.: A visibility-based model for terrain features. In: Proc. 2nd International Symposium on Spatial Data Handling, pp. 235–250 (1986) 8. Fisk, S.: A short proof of chv´ atal’s watchman theorem. Journal of Comb. Theory Ser. B 24, 374 (1978) 9. Fj¨ allst¨ orm, P.O.: Algorithms for Graph Partitioning: A Survey, Link¨ oping Electronic Articles in Computer and Information Science (1998) 10. Heath, M.T., Ranade, A., Schreiber, R.S.: Algorithms for parallel processing. Springer, Heidelberg (1998) 11. Karypis, G., Kumar, V.: METIS: A Software Package for Partitioning Unstructured Graphs, Partitioning Meshes, and Computing Fill-Reducing Orderings of Sparse Matrices Version 4.0, University of Minnesota, September 20 (1998) 12. O’Rouke, j.: Art Gallery Theorems and Algorithms. Oxford University Press, Oxford (1987) 13. Pacheco, P.S.: Parallel Programming with MPI. Morgan Kaufmann, San Francisco (1997) 14. Parhami, B.: Introduction to Parallel Processing: Algorithms and Architectures. Springer, Heidelberg (1999) 15. Quinn, M.J.: Parallel Programming in C with MPI and OpenMP. McGraw-Hill Professional, New York (2004) 16. Shermer, T.: Recent results in art galleries. Proc. IEEE, Special Issue on Comp. Geom. (1992) 17. Urrutia, J.: Art Gallery and illumination Problems. In: Handbook on Comp. Geom., Elsevier, Amsterdam (2000)

JaCk-SAT: A New Parallel Scheme to Solve the Satisfiability Problem (SAT) Based on Join-and-Check Daniel Singer and Anthony Monnet LITA - EA 3097 Universit´e Paul Verlaine - Metz, ˆIle du Saulcy, 57 045 Metz cedex, France [email protected], [email protected]

Abstract. This paper presents and investigates for the first time a new trail for parallel solving of the Satisfiability problem based on a simple and efficient structural decomposition heuristic. A new Joining and model Checking scheme (JaCk-SAT) is introduced. The main goal of this methodology is to recursively cut the variable-set in two subsets of about equal size. On the one hand, in contrast with recent propositions [12,16] for sequential resolution, we do not use sophisticated hypergraph decomposition techniques such as Tree Decomposition that are very likely infeasible. On the other hand, in contrast with all the actual propositions [27] for parallel resolution, we make use of a structural decomposition (of the problem) instead of a search space one. The very first preliminary results of this new approach are presented.

1

Introduction

The propositional Satisfiability problem (SAT) is certainly the most studied one in computer science since it was the first problem proven to be NP-complete by S. Cook in 1971. Nowadays, the Satisfiability problem evidences great practical importance in a wide range of disciplines, including hardware verification, artificial intelligence, cryptography and it is especially important in the area of Electronic Design Automation (EDA). There is increasing demand for high performance SAT-solving algorithms in industry to solve huge and harder problems that show an exponential explosion of the search space. Unfortunately, most state-of-the-art solvers are sequential and fewer are parallel. In our appreciation, finding “good decompositions” for parallel resolution is the real actual challenge for researchers to handle SAT applications. There is still a gap between our actual ability to handle instances with hundreds of variables and real ones with hundreds of thousands of variables! This article introduces a new possibility to decompose a SAT problem with a simple and efficient methodology that does not need a complex hypergraph decomposition. It should be noted that this general framework can also be applied to Constraint Satisfaction Problems (CSP). R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 249–258, 2008. c Springer-Verlag Berlin Heidelberg 2008 

250

D. Singer and A. Monnet

The remainder of this paper is organized as follows. Section 2 briefly introduces the related work and motivations for this new approach. Section 3 presents the main steps of the JaCk-SAT scheme based on Joining and model-Checking. Section 4 briefly describes the main proposed methods to parallelize the core sequential algorithms and it gives the verry first experimental results of this new track for parallel resolution of SAT. The last Section 5 summerizes the whole proposition and points to some perspectives.

2

Related Work and Motivations

Decomposition methods have been applied in a variety of combinatorial problems such as CSP, especially for parallel resolution [14], but very few work exist in this direction for SAT. [2] used decompositions based on some approximation of the most constrained subproblems (those with the least number of solutions). Complex algorithms to find Minimum Vertex Separators or 0-1 fractional programming optimization decompose a formula into a tree of partitions (subproblems). Then a DPLL algorithm1 runs with a branching heuristic based on this decompostion solving first the most constrained subformulas. At last it performs a compilation procedure for each subproblem and joins the solutions to answer the initial problem. Unfortunately, we are not aware of any experimental results of these propositions. [6,15,16] study the applicability of Tree Decomposition method to SAT but only in the sequential case. Tree decomposition [7] is a graph theoretic concept which captures the topological structure of the formula represented as a hypergraph. Formulas that have bounded treewidth can be checked for satisfiability in linear time [8]. [16] recognizes the actual non-feasibility of the Tree Decomposition method for SAT but [6] considers its integration to modern DPLL solvers mainly to guide the variable ordering process. It is noticeable that it does not respond as well in terms of runtime. [12,20] use also hypergraph decomposition methods to obtain balanced clause-set bipartition or unbalanced tripartition but only to guide the SAT solver inside the subproblems. A state-of-the-art hypergraph partition tool is used and preliminary results are presented. More recently [5] presents a connected components decomposition but only to improve sequential performance. Certainly because of the number of limitations of the structural approach all the actual propositions for parallel resolution of SAT use a Search Space Decomposition scheme. This simple methodology permits to conduct independent searches that may cooperate (see [27] for a recent review of this approach). Here, we introduce a new methodology based on a different but feasible Structural Decomposition of the underlying problem graph. It permits to obtain much simpler subproblems that can be independently solved but with an additional Joining and Model Checking phase.

1

Davis Putnam Logemann Loveland procedure [9].

JaCk-SAT: A New Parallel Scheme to Solve the Satisfiability Problem

3

251

The New Decomposition Scheme

Let V = {v1 , v1 , . . . vn } be a set of n boolean variables. A (partial) truth assignment τ for V is a (partial) function: V → {T rue, F alse}. Corresponding to each variable v are two literals: v and ¬v called positive and negative literals. A clause C is a set of literals interpreted as a disjunction. A formula F is a set of clauses interpreted as a Conjunctive Normal Form (CNF) of a formula of the propositional calculus. A truth assignment τ satisfies a formula F (τ is a solution) iff it satisfies every clause in F . A truth assignment τ satisfies a clause C iff it satisfies at least one literal in C. Definition 1. The Satisfiability Problem (SAT): -Input: A set of Boolean variables V and a set of clauses C over V. -Output: Yes (gives a satisfying truth assignment τ for C if it exists) or No. Here we present the new Joining and model Checking scheme (JaCk-SAT) for parallel resolution of the SATisfiability problem2 . The main goal of this methodology is to divide the variable-set V (with |V| = n) in two subsets V1 , V2 of about equal size ( |V1 |  |V2 |  n/2). In contrast with the recent propositions [12,16] (for sequential resolution) we do not use sophisticated hypergraph decompositions such as Tree Decomposition that are very likely infeasible or not yet convincing (“they are far too expansive” [15]). The two fundamental differences are the following ones. First, remembering the inherent exponential behaviour of SAT depends on n, we do not focus our search for clause-sets decompositions but for variable-sets ones. Secondly, we do not impose a bipartition of the clause-set, that induces in general a large joining variable-set, but we define a tripartition of the clause-set. A recursive use of this scheme permits to obtain more than two subproblems. One advantage of this new scheme is allowing a great flexibility to fit the target parallel architecture: one multi-processors (cores) machine, a cluster of dozen of machines or a grid with thousands of machines. Let P = (V, C) be the initial SAT problem, the decomposition step builds the two subproblems P1 = (V1 , C1 ), P2 = (V2 , C2 ) and a residual clause-set C3 . V = V1 ∪ V2 , but we do not have V = V1 ⊕ V2 , meaning that V1 ∩ V2 = ∅. We shall denote by Vjs = V1 ∩V2 , the Join variable-set and it is not necessarily a cutset of V. C1 = {C ∈ C/∀x ∈ C, x ∈ V1 } , C2 = {C ∈ C/∀x ∈ C, x ∈ V2 } , C3 = C\(C1 ∪ C2 ) ⊆ {C ∈ C/∃x ∈ C, x ∈ V1 and ∃y ∈ C, y ∈ V2 }. S (resp. S1 , S2 ) represents the possibly empty set of solutions of P (resp. P1 , P2 ). This decomposition results in searching independently for all the solutions of both subproblems P1 and P2 , trying to join them to obtain global solutions of P thus checking for satisfiability of the residual clauses C3 . The major drawbacks of this method to be overcome are two-fold. First, the search for all the solutions of n/2 variables problems compared to the search for one solution of a n variables problem. Next, the additional Join-and-Check step that is polynomial in the number of solutions but may be exponential in term of n. 2

In fact it solves #SAT finding all solutions of the problem.

252

D. Singer and A. Monnet

1. Decompose(P, P1 , P2 , Vjs , C3 ) 2. //SearchAllSol((P1 , S1 ), (P2 , S2 )) 3. //Join-and-Check(S1 , S2 , Vjs , S, C3 ) Fig. 1. Main Algorithm of JaCk-SAT

3.1

The Decomposition Step

Extracting an approximation of the most constrained subproblem P1 = (V1 , C1 ) with |V1 | ∼ n/2 is the main goal for this decomposition step. This results in the non-symetric property of our decomposition in P1 and P2 but it is justified by the possibility to early find the formula unsatisfiability and by the next Joining step complexity depending on the number of solutions of the problems. We have experimented different heuristics to obtain the most constrained subproblem P1 with |V1 | ∼ n/2 variables and to control the different key parameters that are |Vjs |, |C3 |, |S1 | and |S2 |. Moreover applying recursively this decomposition scheme gives the possibility to fit the desired parallel granularity. A recursive depth of 1 produces 2 subproblems P1 and P2 , and a depth equals 2 applies this algorithm to P1 and P2 giving 4 subproblems . . . Heuristics may be variable or literal oriented and also may be statically or dynamically defined. Each heuristic consists in the five following steps (see Fig.2 below). The main idea is to start (step.1) with the selection of (n*mos) variablesliterals called kernel variables-literals in order to initialize V1 . A simplified example is presented in Sect. 3.3 and we had a first experimental study of the different parameters of the sequential algorithm in [28]. 1. Select (statically or dynamically) the (n ∗ mos) kernel variables (or literals) with maximal number of occurrences to initialize V1 2. Complete (statically or dynamically) V1 until |V1 | = n ∗ (1 + js)/2 with variables (or literals) with maximal number of co-occurrences with V1 (and then C1 ) 3. Initialize V2 with V\V1 4. Complete (statically or dynamically) V2 until |V2 | = n ∗ (1 + js)/2 with variables (or literals) with maximal number of co-occurrences with V2 to obtain V2 (then C2 ) and these variables are those of Vjs 5. C3 is C\(C1 ∪ C2 ) Fig. 2. Decomposition Algorithm

3.2

The Parallel Join-and-Check Step

Since the invention of relational database systems, tremendous effort has been undertaken in order to develop efficient join algorithms. Starting from a simple nested-loop, introduction of sorting, hashing and index structures are the main improvements (see [22]). The key challenge of this approach is clearly the

JaCk-SAT: A New Parallel Scheme to Solve the Satisfiability Problem

253

memory explosion obtained for S1 and S2 to be joined. It is important to note that parallelization of the resolution with //join results in dividing this explosion as much as possible thus minimizing this major obstacle. After trying a number of versions without success because of the inherent time and space limitations of the sequential prototyping study presented in [28], we developed an algorithm and data structures largely inspired by the work of M. Bamha on Parallel Join Algorithms in [3]. In order to obtain a better overall performance the step 3 of the main algorithm (Fig.1) proceeds at the same time the Joining and Checking steps. Thus S1 and S2 are filtered by the residual clause-set C3 to be satisfied. First of all to put it more precisely, the variables of Vjs shall be considered as a single compacted attribute and each assignment of these variables will be one single vector (in the C ++ terminology). At this point, it is important to note that (to our best knowedge) there is no join algorithm dedicated to boolean value attributes and that the reason why we had to make this encoding step. Let Solution-Set be an object containing the solution-vector sol and an index Index that associates to each possible value of the joining attribute (Vjs ) the corresponding set of solution numbers. This index is systematically updated each time a new solution is added to Solution-Set. Before the Join-and-Check step begins we get the solution-sets S1 and S2 from previous step.2. Below in Fig. 3 is a simplified version of //Join-and-Check Algorithm that shall be illustrated by the example in the next Sect. 3.3. We will not develop here all the specific parallel methodology and we refer to [23] for more details. 1. Compute locally the histograms hist1 and hist2 for S1 and S2 respectively from (hash)Index, thus compute their intersecting joining histogram: joinHist 2. Compute sols1Constraints (resp. sols2Constraints) the set of clauses from C3 restricted to variables of V1 (resp. V2 ) /* a joining solution of sol1 ∈ S1 and sol2 ∈ S2 satisfies C3 iff ∀i ∈ [1...|C3 |] /* sol1 satisfies sols1Constraints[i] or sol2 satisfies sols2Constraints[i] 3. forall// joinV alue ∈ joinHist do forall index i ∈ S1 .Index[joinV alue] do constr ← ∅ /* a set of integers for k from 1 to |C3 | do If Checking (S1 [i] , sols1Constraints[k]) = False Then constr ← constr +k forall index j ∈ S2 .Index[joinV alue] do If ∀k ∈ constr, Checking (S2 [j] , sols2Constraints[k]) =True Then S ← S + S1 [i]  S2 [j] Fig. 3. //Join-and-Check Algorithm

254

D. Singer and A. Monnet

3.3

A Simplified Example

CNF problem P with : |V| = n = 6 ⎧ c1 ⎪ ⎪ ⎪ ⎪ ⎪ c2 ⎪ ⎪ ⎪ ⎨c 3 ⎪ c4 ⎪ ⎪ ⎪ ⎪ ⎪ c5 ⎪ ⎪ ⎩ c6

|C| = 12

and

: v1 ∨ ¬v2 ∨ ¬v4

c7 : ¬v1 ∨ v2 ∨ ¬v3 ∨ v4 ∨ ¬v5 ∨ v6

: v3 ∨ ¬v6 : ¬v2 ∨ v4 ∨ v6

c8 : v1 ∨ ¬v3 c9 : v3 ∨ ¬v5 ∨ ¬v6

: v2 ∨ ¬v4 ∨ ¬v5 ∨ ¬v6 : v3 ∨ v4 ∨ v5

c10 : ¬v1 ∨ v6 c11 : ¬v4 ∨ ¬v6

: v1 ∨ v5

c12 : ¬v3 ∨ v5

The Decomposition Step parameters are : static and variable oriented heuristic, mos = 33% (|Vjs | = 2) and js = 33% (|V1 | = |V2 | = 4).

1.Selection of kernel variables:

variable v1 v2 v3 v4 v5 v6

5 4 6 6 6 7

occurrences (c1 , c6 , c7 , c8 , c10 ) (c1 , c3 , c4 , c7 ) (c2 , c5 , c7 , c8 , c9 , c12 ) (c1 , c3 , c4 , c5 , c6 , c11 ) (c4 , c5 , c6 , c7 , c9 , c12 ) (c2 , c3 , c4 , c7 , c9 , c10 , c11 )

V1 ← {v3 , v6 } variable v1 v2 v4 v5

2.Completion of V1 :

co-occurrences with v3 and/or v6 3 (c7 , c8 , c10 ) 3 (c3 , c4 , c7 ) 5 (c3 , c4 , c5 , c6 , c11 ) 5 (c4 , c5 , c7 , c9 , c12 )

V1 ← V1 ∪ {v4 , v5 } = {v3 , v4 , v5 , v6 } C1 ← {c | ∀v ∈ c, v ∈ V1 } = {c2 , c5 , c9 , c11 , c12 } 3.Initialization of V2 :

V2 ← V\V1 = {v1 , v2 }

4.Completion of V2 :

variable v3 v4 v5 v6

co-occurrences with v1 and/or v2 2 (c7 , c8 ) 4 (c1 , c3 , c4 , c7 ) 3 (c4 , c6 , c7 ) 4 (c3 , c4 , c7 , c10 )

V2 ← V2 ∪ {v4 , v6 } = {v1 , v2 , v4 , v6 } C2 ← {c | ∀v ∈ c, v ∈ V2 } = {c1 , c3 , c10 , c11 } Vjs ← V1 ∩ V2 = {v4 , v6 } 5.The residual clause-set:

C3 ← C\(C1 ∪ C2 ) = {c4 , c6 , c7 , c8 }

JaCk-SAT: A New Parallel Scheme to Solve the Satisfiability Problem

255

//Solving of P1

S1 (v3 , v4 , v5 , v6 ) join value (v4 , v6 ) sol11 (1, 0, 1, 0) (0,0) sol12 (1, 0, 1, 1) (0,1) sol13 (1, 1, 1, 0) (1,0) sol14 (0, 0, 1, 0) (0,0) sol15 (0, 1, 0, 0) (1,0) sol16 (0, 1, 1, 0) (1,0)

hist1 (v4 , , v6 ) Index (0, 0) 2 {1,4} (0, 1) 1 {2} (1, 0) 3 {3,5,6} (1, 1) 0 {}

//Solving of P2

S2 (v1 , v2 , v4 , v6 ) join value (v4 , v6 ) sol21 (0, 0, 0, 0) (0,0) sol22 (0, 0, 0, 1) (0,1) sol23 (0, 0, 1, 0) (1,0) sol24 (1, 0, 0, 1) (0,1) sol25 (0, 1, 0, 1) (0,1) sol26 (1, 1, 0, 1) (0,1)

hist2 (v4 , , v6 ) Index (0, 0) 1 {1} (0, 1) 4 {2,4,5,6} (1, 0) 1 {3} (1, 1) 0 {}

join value (v4 , v6 ) solution (v1 , v2 , v3 , v4 , v5 , v6 ) verifies C3 ? (0,0) sol11 1 sol21 (0, 0, 1, 0, 1, 0) no (c8 ) (0,0) sol14 1 sol21 (0, 0, 0, 0, 1, 0) yes (0,1) sol12 1 sol22 (0, 0, 1, 0, 1, 1) no (c8 ) (0,1) sol12 1 sol24 (1, 0, 1, 0, 1, 1) yes //Join-and-check (0,1) sol12 1 sol25 (0, 1, 1, 0, 1, 1) no (c8 ) (0,1) sol12 1 sol26 (1, 1, 1, 0, 1, 1) yes (1,0) sol13 1 sol23 (0, 0, 1, 1, 1, 0) no (c8 ) (1,0) sol15 1 sol23 (0, 0, 0, 1, 0, 0) no (c6 ) (1,0) sol16 1 sol23 (0, 0, 0, 1, 1, 0) yes

Conclusion: the problem is satisfiable with 4 distinct satisfying assignments.

4

Parallel Resolution of SAT: Preliminary Results

In this section are presented the first experimental results of this new decomposition scheme to solve in parallel the Satisfiability problem. Our main objective here is only to show the general approach feasibility and to point out challenging efforts that have to be done to deal in the future with problems having thousands of variables. One major advantage of this approach is that it does not depend on the solver, and we will mention below some experience with two of state-of-the-art solvers (for allsolutions) kcnfs [11] and relsat [4]. Different decomposition strategies in step.1 of the main algorithm (Fig. 2) can be characterized by the variable selection mode: dynamic or static and variableoriented or literal-oriented. Different variants have been defined and tested for each mode. The Join-Set size parameter (js) is crucial to obtain a “good decoposition” and this task is not easy as the results show a great variability of behaviour depending on the problem structure and size, the decomposition strategy and solver.

256

D. Singer and A. Monnet

We present below some results for well-known benchmark problems used by the SAT community that can be found in SATLIB [1]. uuf75 and dubois30 are hard unstisfiable instances. hole9 is a structured unsatisfiable problem from the pigeon-hole series and pret60 is a SAT encoding of a graph-colouring problem from the Pretolani-series. Early results found a dynamic and variable oriented strategy (DynVar) to be the best one. Note the very interesting result it obtained for the pret60 instances that has seemingly discovered a separating-variable set (Vjs ) of the underlying graph to be coloured? For both instances hole9 and uuf75 that have more variables the memory explosion obtained for |S1 | and |S2 | led to unmanageably large set of solutions explaining the poor results. In the uuf75 instance we have |S1 | = 393, 926, |S2| = 31, 774, 621 and 11, 044, 382 solutions have been joinedand-checked! The decomposition depth and solvers study Table 1 shows comparative results for both solvers kcnfs and relsat for different depth of decomposition as it was argued in Section3.1. The direct column corresponds to the direct solving by the sequential solver on a single processor. The depth1 seq and depth2 seq columns correspond to sequential solving after a decomposition in 2 or 4 subproblems and the depth1 par and depth2 par columns correspond to parallel solving after the same decompositions. The parallel solving with a depth i decomposition corresponds to the running with 2i processors. The parallel algorithms are implemented in C ++ (except the solvers) under (LAM)MPI and run on a linux cluster of 4 PCs with 512Mo of RAM at 2.4GHz and a 100Mbit Ethernet internal network. Table 1. Solving of various problems using a DynVar strategy with various depths (join size: 20% except for uuf75-073: 30%, mos: 10%) prob pret60 25 pret60 40 pret60 60 pret60 75 dubois30 hole9 uuf75-073

vars clauses solver direct depth1 seqdepth1 pardepth2 seq depth2 par kcnfs 3.56 1.56 1.22 0.24 0.16 60 160 relsat 0.0 2.94 0.74 0.24 0.16 kcnfs 3.58 1.62 1.22 0.24 0.16 60 160 relsat 0.0 3.02 0.72 0.24 0.16 kcnfs 3.58 1.59 1.21 0.25 0.16 60 160 relsat 0.0 2.95 0.76 0.25 0.15 kcnfs 3.52 1.61 1.2 0.24 0.15 60 160 relsat 0.0 2.95 0.72 0.26 0.15 kcnfs 2,748.880 88.03 52.3 7.24 2.62 54 140 relsat 0.0 111.89 28.97 7.94 2.6 kcnfs 4.13 553.53 541.49 513.94 271.19 90 415 relsat 14.0 1,167.78 500.54 524.28 268.05 kcnfs 0.0 310.72 862.95 601.47 455.55 75 325 relsat 0.0 3,671.05  1,373.31 602.26

JaCk-SAT: A New Parallel Scheme to Solve the Satisfiability Problem

5

257

Conclusion and Future Work

This paper presents and investigates for the first time a new track for parallel solving of the Satisfiability problem based on a simple and efficient structural decomposition. It defines a general Join-and-Chek schema and it brings forward preliminary propositions for a parallel prototype. Moreover this new general approach can also be applied to Constraint Satisfaction Problems (CSP). Early results of an experimental evaluation show the feasibility and interest of this approach making sense for a deeper study on scalability. This also makes clear that a number of substantial improvements should be made, the first being to define better solvers that efficently generate all solutions as mentioned in [19]. Then using and developing better algorithms and data structures for the crucial join step especially in a parallel-and-distributed execution framework should improve significantly. And last but not the least we hope that this work provides enough motivation for other researchers to follow this new direction.

Acknowledgments The authors would like to thank M. Bamha from Orl´eans University (LIFO) for a number of usefull discussions on the Joining algorithms that are intensively studied in the Data Base area. We also want to thank G. Dequen from Amiens University (LARIA) for making kcnfs to solve all solutions.

References 1. http://www.cs.ubc.ca/∼ hoos/SATLIB/ 2. Amir, E., McIllraith, S.: Solving Satisfiability using Decomposition and the Most Constrained Subproblem. In: IJCAI 2001 Workshop on Distributed Constraint Reasoning, online http://www.cs.berkeley.edu/∼ eyal/paper.html 3. Bamha, M.: An optimal skew-insensitive join and multi-join algorithm for distributed architecture. In: Andersen, K.V., Debenham, J., Wagner, R. (eds.) DEXA 2005. LNCS, vol. 3588, pp. 616–625. Springer, Heidelberg (2005) 4. Bayardo Jr., R.J., Pehousek, J.D.: Counting Models Using Connected Components. In: Proc. 17th. Nat. Conf. on AI (AAAI 2000) (2000) 5. Biere, A., Sinz, C.: Decomposing Sat Problems into Connected Components. Journal on Satisfiability, Boolean Modeling and Computation 2, 201–208 (2006) 6. Bjesse, P., Kukula, J., Damiano, R., Stanion, T., Zhu, Y.: Diagnosis with Tree Decomposition. In: Giunchiglia, E., Tacchella, A. (eds.) SAT 2003. LNCS, vol. 2919, Springer, Heidelberg (2004) 7. Bodlaender, H.: A Tourist Guide through Treewidth. Acta Cybernetica 11 (1993) 8. Darwiche, A.: Compiling knowledge into decomposable negation normal form. In: Proc. of 15th. IJCAI, pp. 284–289 (1999) 9. Davis, M., Logeman, G., Loveland, D.: A machine program for Theorem Proving. Com. of the ACM 5(7) (1962) 10. del Val, A.: Simplifying Binary Propositional Theories into Connected Components Twice as Fast. In: Nieuwenhuis, R., Voronkov, A. (eds.) LPAR 2001. LNCS (LNAI), vol. 2250, pp. 389–403. Springer, Heidelberg (2001)

258

D. Singer and A. Monnet

11. Dequen, G., Dubois, O.: kcnf s: an efficient solver for random k-SAT formulae. In: Giunchiglia, E., Tacchella, A. (eds.) SAT 2003. LNCS, vol. 2919, pp. 486–501. Springer, Heidelberg (2004) 12. Durairaj, V., Kalla, P.: Exploiting Hypergraph Partitioning for Efficient Boolean Satisfiability. In: Proc. 9th. IEEE Int. High Level Design Validation and Test Workshp, pp. 141–146 (2004) 13. Gummadi, R., Narayanaswamy, N.S., Venkatakrishnan, R.: Algorithms for Satisfiability Using Independent Sets of Variables. In: H. Hoos, H., Mitchell, D.G. (eds.) SAT 2004. LNCS, vol. 3542, p. 2005. Springer, Heidelberg (2005) 14. Habbas, Z., Krajecki, M., Singer, D.: Decomposition Techniques for Parallel Resolution of Constraint Satisfaction Problems in Shared Memory: a Comparative Study. Int. Jour. of Computational Science and Engineering (IJCSE) 1(2-4), 192– 206 (2005) 15. Herwig, P.R.: Decomposing satisfiability problems. Master’s Thesis, Delft University of Technology (2006) 16. Heule, M., Kullmann, O.: Decomposing clause-sets: Integrating DLL algorithms, tree decompositions and hypergraph cuts for variable and clause-based graph representations of CNF’s. TR. CSR 2-2006, Univ. Wales Swansea (2006) 17. Hicks, I.V., Koster, A., Kolotoglu, E.: Branch and Tree Decomposition Techniques for Discrete Optimization. TutORials in O.R., Informs (2005) 18. Karypis, G.: Multilevel Hypergraph partitionning. In: Cong, J., Shinnerl, J. (eds.) Multilevel Optimization Methods for VLSI, ch.6, Kluwer Ac. Pub, Dordrecht (2002) 19. Khurshid, S., Marinov, D., Shlyakhter, I., Jackson, D.: A Case for Efficient Solution Enumeration. In: Giunchiglia, E., Tacchella, A. (eds.) SAT 2003. LNCS, vol. 2919, Springer, Heidelberg (2004) 20. Li, W., Van Beek, P.: Guiding Real-World Sat Solving with Dynamic Hypergraph Separator Decomposition. In: Proc. 16th. IEEE Int. Conf. on Tools for AI (ICTAI 2004) (2004) 21. Marques Silva, J., Oliveira, A.: Improving Satisfiability Algorithms with Dominance and Partitioning. In: Int. Workshop on Logic Synthesis (IWLS) (1997) 22. Mishra, P., Eich, M.H.: Join Processing in Relational Databases. ACM Computing Surveys 24(1), 63–113 (1992) 23. Monnet, A.: JaCk-SAT: un nouvel algorithme de r´esolution parall´ele du probl´eme SAT par d´ecomposition structurelle. Master Thesis (in french), Universit´e Paul Verlaine-Metz (2007) 24. Park, T.J., Van Gelder, A.: Partitioning Methods for Satisfiabilty Testing on Large Formulas. Information and Computation 162, 179–184 (2000) 25. Parkes, A.J.: Exploiting Solutions Clusters for Coarse-Grained Distributed Search. In: Workshop on Dist. Constraints Reasoning, IJCAI 2001 (2001) 26. Schloegel, K., Karypis, G., Kumar, V.: Graph Partitioning for High Performance Scientific Simulations. In: Dongarra, J., et al. (eds.) CRPC Parallel Computing Handbook, Morgan Kauffmann, San Francisco (2000) 27. Singer, D.: Parallel Resolution of the Satisfiability Problem: a Survey. In: Talbi, E.G. (ed.) Parallel Combinatorial Optimization, Wiley and Sons, Chichester (2006) 28. Singer, D., Monnet, A.: A New Decomposition Scheme for Parallel Resolution of the Satisfiability Problem (SAT). In: 14th. RCRA 2007, Roma (2007) 29. Singer, D., Vagner, A.: Parallel Resolution of the Satisfiability Problem with OpenMP and MPI. In: Wyrzykowski, R., Dongarra, J., Meyer, N., Wa´sniewski, J. (eds.) PPAM 2005. LNCS, vol. 3911, Springer, Heidelberg (2006) 30. Snir, M., Otto, S.W., Huss-Lederman, S., Walker, D.W., Dongarra, J.: MPI: the complete reference. The MIT Press, Cambridge (1996)

Designing Service-Based Resource Management Tools for a Healthy Grid Ecosystem ¨ Erik Elmroth, Francisco Hern´ andez, Johan Tordsson, and Per-Olov Ostberg Dept. of Computing Science and HPC2N Ume˚ a University, SE-901 87 Ume˚ a, Sweden {elmroth,hernandf,tordsson,p-o}@cs.umu.se

Abstract. We present an approach for development of Grid resource management tools, where we put into practice internationally established high-level views of future Grid architectures. The approach addresses fundamental Grid challenges and strives towards a future vision of the Grid where capabilities are made available as independent and dynamically assembled utilities, enabling run-time changes in the structure, behavior, and location of software. The presentation is made in terms of design heuristics, design patterns, and quality attributes, and is centered around the key concepts of co-existence, composability, adoptability, adaptability, changeability, and interoperability. The practical realization of the approach is illustrated by five case studies (recently developed Grid tools) high-lighting the most distinct aspects of these key concepts for each tool. The approach contributes to a healthy Grid ecosystem that promotes a natural selection of “surviving” components through competition, innovation, evolution, and diversity. In conclusion, this environment facilitates the use and composition of components on a per-component basis.

1

Introduction

In recent years, the vision of the Grid as the general-purpose, service-oriented infrastructure for provisioning of computing, data, and information capabilities has started to materialize in the convergence of Grid and Web services technologies. Ultimately, we envision a Grid with open and standardized interfaces and protocols, where independent Grids can interoperate, virtual organizations co-exist, and capabilities be made available as independent utilities. However, there is still a fundamental gap between the technology used in major production Grids and recent technology developed by the Grid research community. While current research directions focus on user-centric and serviceoriented infrastructure design for scenarios with millions of self-organizing nodes, current production Grids are often more monolithic systems with stronger intercomponent dependencies. 

This research was conducted using the resources of the High Performance Computing Center North (HPC2N). Financial support has been provided by The Swedish Research Council (VR) under contract 621-2005-3667.

R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 259–270, 2008. c Springer-Verlag Berlin Heidelberg 2008 

260

E. Elmroth et al.

We present an approach to Grid infrastructure component development, where internationally established high-level views of future Grid architectures are put into practice. Our approach addresses the future vision of the Grid, while enabling easy integration into current production Grids. We illustrate the feasibility of our approach by presenting five case studies. The outline of the rest of the paper is as follows. Section 2 gives further background information, including our vision of the Grid, a characterization of competitive factors for Grid software, and a brief review of internationally established conceptual views of future Grid architectures. Section 3 presents our approach to Grid infrastructure development, which complies with these views. The realization of this approach for specific components is illustrated in Section 4, with a brief presentation of five tools recently developed within the Grid Infrastructure Research & Development (GIRD) project [26]. These are Grid tools or toolkits for resource brokering [9,10,11], job management [7], workflow execution [8], accounting [16,24], and Grid-wide fairshare scheduling [6].

2

Background and Motivation

Our approach to Grid infrastructure development is driven by the need and opportunity for a general-purpose infrastructure. This infrastructure should facilitate flexible and transparent access to distributed resources, dynamic composition of applications, management of complex processes and workflows, and operation across geographical and organizational boundaries. Our vision is that of a large evolving system, realized as a Service-Oriented Architecture (SOA) that enables provisioning of computing, data, and information capabilities as utility-like services serving business, academia, and individuals. From this point of departure, we elaborate on fundamental challenges that need to be addressed to realize this vision. 2.1

Facts of Life in Grid Environments

The operational context of a Grid environment is harsh, with heterogeneity in resource hardware, software, ownerships, and policies. The Grid is distributed and decentralized by nature, and any single point of control is impossible not only for scalability reasons but also since resources are owned by different organizations. Furthermore, as resource availability varies, resources may at any time join or leave the Grid. Information about the set of currently available resources and their status will always to some extent be incomplete or outdated. Actors have different incentives to join the Grid, resulting in asymmetric resource sharing relationships. Trust is also asymmetric, which in scenarios with cross trust-domain orchestration of multiple resources that interact beyond the client-server model, gives rise to complex security challenges. Demand for resources typically exceed supply, with contention for resources between users as a consequence. The Grid user community at large is disparate in requirements and knowledge, necessitating the development of wide ranges of

Designing Service-Based Resource Management Tools

261

user interfaces and access mechanisms. All these complicating factors add up to an environment where errors are rule rather than exception. 2.2

A General-Purpose Grid Ecosystem

Recently, a number of organizations have expressed views on how to realize a single and fully open architecture for the future Grid. To a large extent, these expressions conform to a single view of a highly dynamic service-oriented infrastructure for general-purpose use. One such view proposes the model of a healthy ecosystem of Grid components [25], where components occupy niches in the ecosystem and are designed for component-by-component selection by developers, administrators, and endusers. Components are developed by the Grid community at large and offer sensible functionality, available for easy integration in high-level tools or other software. In the long run, competition, innovation, evolution, and diversity lead to natural selection of “surviving” components, whereas other components eventually fade out or evolve into different niches. European organizations, such as the Next Generation Grids expert group [12] and NESSI [23], have focused on a common architectural view for Grid infrastructure, possibly with a more emphasized business focus compared to previous efforts. Among their recommendations is a strong focus on SOAs where services can be dynamically assembled, thus enabling run-time changes in the structure, behavior, and location of software. The view of services as utilities includes directly and immediately usable services with established functionality, performance, and dependability. This vision goes beyond that of a prescribed layered architecture by proposing a multi-dimensional mesh of concepts, applying the same mechanisms along each dimension across the traditional layers. In common for these views are, for example, a focus on composable components rather than monolithic Grid-wide systems, as well as a general-purpose infrastructure rather than application- or community-specific systems. Examples of usage range from business and academic applications to individual’s use of the Grid. These visions also address some common issues in current production Grid infrastructures, such as interoperability and portability problems between different Grids, as well as limited software reuse. Before detailing our approach to Grid software design, which complies with the views presented above, we elaborate on key factors for software success in the Grid ecosystem. 2.3

Competitive Factors for Software in the Grid Ecosystem

In addition to component-specific functional requirements, which obviously differ for different types of components, we identify a set of general quality attributes (also known as non-functional requirements) that successful software components should comply with. The success metrics considered here are the amount of users and the sustainability of software. In order to attract the largest possible user community, usability aspects such as availability, ease of installation, understandability, and quality of documentation and support are important. With the dynamic and changing nature of

262

E. Elmroth et al.

Grid environments, flexibility and the ability to adapt and evolve is vital for the survival of a software component. Competitive factors for survival include changeability, adaptability, portability, interoperability, and integrability. These factors, along with mechanisms used to improve software quality with respect to them, are further discussed in Section 3. Other criteria, relating to sustainability, include the track record of both components and developers as well as the general reputation of the latter in the user community. Quality attributes such as efficiency (with emphasis on scalability), reliability, and security also affect the software success rate in the Grid ecosystem. These attributes are however not further discussed herein.

3

Grid Ecosystem Software Development

In this section we present our approach to building software well-adjusted to the Grid ecosystem. The presentation is structured into five groups of software design heuristics, design patterns, and quality attributes that are central to our approach. All definitions are adapted to the Grid ecosystem environment, but are derived from, and conform to, the ISO/IEC 9126-1 standard [20]. 3.1

Co-existence – Grid Ecosystem Awareness

Co-existence is defined as the ability of software to co-exist with other independent softwares in a shared resource environment. The behavior of a component well adjusted to the Grid ecosystem is characterized by non-intrusiveness, respect for niche boundaries, replaceability, and avoidance of resource overconsumption. When developing new Grid components, we identify the purpose and boundaries of the corresponding niches in order to ensure the components’ place and role in the ecosystem. By stressing non-intrusiveness in the design, we strive to ensure that new components do not alter, hinder, or in any other way affect the function of other components in the system. While the introduction of new software into an established ecosystem may, through fair competition, reshape, create, or eliminate niches, it is still important for the software to be able to cooperate and interact with neighboring components. By the principle of decentralization, it is crucial to avoid making assumptions of omniscient nature and not to rely on global information or control in the Grid. By designing components for a user-centric view of systems, resources, component capabilities, and interfaces, we emphasize decentralization and facilitate component co-existence and usability. 3.2

Composability – Software Reuse in the Grid Ecosystem

Composability is defined as the capability of software to be used both as individual components and as building blocks in other systems. As systems may themselves be part of larger systems, or make use of other systems’ components, composability becomes a measure of usefulness at different levels of system

Designing Service-Based Resource Management Tools

263

design. Below, we present some design heuristics that we make use of in order to improve software composability. By designing components and component interactions in terms of interfaces rather than functionality, we promote the creation of components with welldefined responsibilities and provision for module encapsulation and interface abstraction. We strive to develop simple, single-purpose components achieving a distinct separation of concerns and a clear view of service architectures. Implementation of such components is faster and less error-prone than more complex designs. Autonomous components with minimized external dependencies make composed systems more fault tolerant as their distributed failure models become simpler. Key to designing composable software is to provision for software reuse rather than reinvention. Our approach, leading to generic and composable tools well adjusted to the Grid ecosystem, encourages a model of software reuse where users of components take what they need and leave the rest. Being decentralized and distributed by nature, SOAs have several properties that facilitate the development of composable software. 3.3

Adoptability – Grid Ecosystem Component Usability

Adoptability is a broad concept enveloping aspects such as end-user usability, ease of integration, ease of installation and administration, level of portability, and software maintainability. These are key factors for determining deployment rate and niche impact of a software. As high software usability can both reduce end-user training time and increase productivity, it has significant impact on the adoptability of software. We strive for ease of system installation, administration, and integration (e.g., with other tools or Grid middlewares), and hence reduce the overhead imposed by using the software as stand-alone components, end-user tools, or building blocks in other systems. Key adoptability factors include quality of documentation and client APIs, as well as the degree of openness, complexity, transparency and intrusiveness of the system. Moreover, high portability and ease of migration can be deciding factors for system adoptability. 3.4

Adaptability and Changeability – Surviving Evolution

Adaptability, the ability to adapt to new or different environments, can be a key factor for improving system sustainability. Changeability, the ability for software to be changed to provide modified behavior and meet new requirements, greatly affects system adaptability. By providing mechanisms to modify component behavior via configuration modules, we strive to simplify component integration and provide flexibility in, and ease of, customization and deployment. Furthermore, we find that the use of policy plug-in modules which can be provided and dynamically updated by third parties are efficient for making systems adaptable to changes in operational

264

E. Elmroth et al.

contexts. By separating policy from mechanism, we facilitate for developers to use system components in other ways than originally anticipated and software reuse can thus be increased. 3.5

Interoperability – Interaction within the Grid Ecosystem

Interoperability is the ability of software to interact with other systems. Our approach includes three different techniques for making our components available, making them able to access other Grid resources, and making other resources able to access our components, respectively. Integration of our components typically only requires the use of one or two of these techniques. Whenever feasible, we leverage established and emerging Web and Grid services standards for interfaces, data formats, and architectures. Generally, we formulate integration points as interfaces expressing required functionality rather than reflecting internal component architecture. Our components are normally made available as Grid services, following these general principles. For our components to access resources running different middlewares, we combine the use of customization points and design patterns such as Adapter and Chain of Responsibility [15]. Whenever possible, we strive to embed the customization points in our components, simplifying component integration with one or more middlewares. In order to make existing Grid softwares able to access our components, we strive to make external integration points as few, small, and well-defined as possible, as these modifications need to be applied to external softwares.

4

Case Studies

We illustrate our approach to software development by brief presentations of five tools or toolkits recently developed in the GIRD project [26]. The presentations describe the overall tool functionality and high-light the most significant characteristics related to the topics discussed in Section 3. All tools are built to operate in a decentralized Grid environment with no single point of control. They are furthermore designed to be non-intrusive and can coexist with alternative mechanisms. To enhance adoptability of the tools, user guides, administrator manuals, developer APIs, and component source code are made available online [26]. As these adoptability measures are common for all projects, the adoptability characteristics are left out of the individual project presentations. The use of SOAs and Web services naturally fulfills many of the composability requirements outlined in Section 3. The Web service toolkit used is the Globus Toolkit 4 (GT4) Java WS Core, which provides an implementation of the Web Services Resource Framework (WSRF). Notably, the fact that our tools are made available as GT4-based Web services should not be interpreted as been built primarily for use in GT4-based Grids. On the contrary, their design is focused on generality and ease of middleware integration.

Designing Service-Based Resource Management Tools

4.1

265

Job Submission Service (JSS)

The JSS is a feature-rich, standards-based service for cross-middleware job submission, providing support, e.g., for advance reservations and co-allocation. The service implements a decentralized brokering policy, striving to optimize the job performance for individual users by minimizing the response time for each submitted job. In order to do this, the broker makes an a priori estimation of the whole, or parts of, the Total Time to Delivery (TTD) for all resources of interest before making the resource selection [9,10,11]. Co-existence: The non-intrusive decentralized resource broker handles each job isolated from the jobs of other users. It can provide quality of service to end-users despite the existence of competing job submission tools. Composability: The JSS is composed of several modules, each performing a welldefined task in the job submission process, e.g., resource discovery, reservation negotiation, resource selection, and data transfer. Changeability and adaptability: Users of the JSS can specify additional information in job request messages to customize and fine-tune the resource selection process. Developers can replace the resource brokering algorithms with alternative implementations. Interoperability: The architecture of the JSS is based on (emerging) standards such as JSDL, WSRF, WS-Agreement, and GLUE. It also includes customization points, enabling the use of non-standard job description formats, Grid information systems, and job submission mechanisms. The latter two can be interfaced despite differences in data formats and protocols. By these mechanisms, the JSS can transparently submit jobs to and from GT4, NorduGrid/ARC, and LCG/gLite. 4.2

Grid Job Management Framework (GJMF)

The GJMF [7] is a framework for efficient and reliable processing of Grid jobs. It offers transparent submission, control, and management of jobs and groups of jobs on different middlewares. Co-existence: The user-centric GJMF design provides a view of exclusive access to each service and enforces a user-level isolation which prohibits access to other users’ information. All services in the framework assume shared access to Grid resources. The resource brokering is performed without use of global information, and includes back-off behaviors for Grid congestion control on all levels of job submission. Composability: Orchestration of services with coherent interfaces provides transparent access to all capabilities offered by the framework. The functionality for job group management, job management, brokering, Grid information system access, job control, and log access are separated into autonomous services. Changeability and adaptability: Configurable policy plug-ins in multiple locations allow customization of congestion control, failure handling, progress monitoring,

266

E. Elmroth et al.

service interaction, and job (group) prioritizing mechanisms. Dynamic service orchestration and fault tolerance is provided by each service being capable of using multiple service instances. For example, the job management service is capable of using several services for brokering and job submission, automatically switching to alternatives upon failures. Interoperability: The use of standardized interfaces such as JSDL as job description format, OGSA BES for job execution, and OGSA RSS for resource selection improves interoperability and replaceability. 4.3

Grid Workflow Execution Engine (GWEE)

The GWEE [8] is a light-weight and generic workflow execution engine that facilitates the development of application-oriented end-user workflow tools. The engine is light-weight in that it focuses only on workflow execution and the corresponding state management. This project builds on experiences gained while developing the Grid Automation and Generative Environment (GAUGE) [19,17]. Co-existence: The engine operates in the narrow niche of workflow execution. Instead of attempting to replace other workflow tools, the GWEE provides a means for accessing advanced capabilities offered by multiple Grid middlewares. The engine can process multiple workflows concurrently without them interfering with each other. Furthermore, the engine can be shared among multiple users, but only the creator of a workflow instance can monitor and control that workflow. Composability: The main responsibilities of the engine, managing task dependencies, processing tasks on Grid resources, and managing workflow state, are performed by separate modules. Adaptability and Changeability: Workflow clients can monitor executing workflows both by synchronous status requests and by asynchronous notifications. Different granularities of notifications are provided to support specific client requirements – from a single message upon workflow completion to detailed updates for each task state change. Interoperability: The GWEE is made highly interoperable with different middlewares and workflow clients through the use of two types of plug-ins. Currently, it provides middleware plug-ins for execution of computational tasks in GT4 and in the GJMF, as well as GridFTP file transfers. It also provides plug-ins for transforming workflow languages into its native language, as currently has been done for the Karajan language. The Chain of Responsibility design pattern allows concurrent usage of multiple implementations of a particular plug-in. 4.4

SweGrid Accounting System (SGAS)

SGAS allocates Grid capacity between user groups by coordinated enforcement of Grid-wide usage limits [24,16]. It employs a credit-based allocation model where Grid capacity is granted to projects via Grid-wide quota allowances. The

Designing Service-Based Resource Management Tools

267

Grid resources collectively enforce these allowances in a soft, real-time manner. The main SGAS components are a Bank, a logging service (LUTS), and a quota-aware authorization tool (JARM), the latter to be integrated on each Grid resource. Co-existence: SGAS is built as stand-alone Grid services with minimal dependencies on other software. Normal usage is not only non-intrusive to other software but also to usage policies, as resource owners retain ultimate control over local resource policies, such as strictness of quota enforcement. Composability: There is a distinct separation of concerns between the Bank and the LUTS, for managing usage quotas and logging usage data, respectively. They can each be used independently. Changeability and adaptability: The Bank can be used to account for any type of resource consumption and with any price-setting mechanism, as it is independent of the mapping to the abstract “Grid credit” unit used. The Bank can also be changed from managing pre-allocations to accumulating costs for later billing. The JARM provides customization points for calculating usage costs based on different pricing models. The tuning of the quota enforcement strictness is facilitated by a dedicated customization point. Interoperability: The JARM has plug-in points for middleware-specific adapter code, facilitating integration with different middleware platforms, scheduling systems, and data formats. The middleware integration is done via a SOAP message interceptor in GT4 GRAM and via an authorization plug-in script in the NorduGrid/ARC GridManager. The LUTS data is stored in the OGF Usage Record format. 4.5

Grid-Wide Fairshare Scheduling System (FSGrid)

FSGrid is a Grid-wide fairshare scheduling system that provides three-party QoS support (user, resource-owner, VO-authority) for enforcement of locally and globally scoped share policies [6]. The system allows local resource capacity as well as global Grid capacity to be logically divided among different groups of users. The policy model is hierarchical and sub-policy definition can be delegated so that, e.g., a VO can partition its share among its projects, which in turn can divide their shares among users. Co-existence: The main objective of FSGrid is to facilitate for distributed resources to collaboratively schedule jobs for Grid-wide fairness. FSGrid is nonintrusive in the sense that resource owners retain ultimate control of how to perform the scheduling on their local resources. Composability: FSGrid includes two stand-alone components with clearly separated concerns for maintaining a policy tree and to log usage data, respectively. In fact, the logging component in current use is the LUTS originally developed for SGAS, illustrating the potential for reuse of that component.

268

E. Elmroth et al.

Changeability and adaptability: A customizable policy engine is used to calculate priority factors based on a runtime policy tree with information about resource pre-allocations and previous usage. The priority calculation can be customized, e.g., in terms of length, granularity, and rate of aging of usage history. The administration of the policy tree is flexible as sub-policy definition can be delegated to, e.g., VOs and projects. Interoperability: Besides the integration of the LUTS (see Section 4.4), FSGrid includes a single external point of integration, as a fair-share priority factor call-out to FSGrid has to be integrated in the local scheduler on each resource.

5

Related Work

Despite the large amount of Grid related projects to date, just a few of these have shared their experiences regarding software design and development approaches. Some of these projects have focused on software architecture. In a survey by Filkenstein et al. [13], existing data-Grids are compared in terms of their architectures, functional requirements, and quality attributes. Cakic et al. [2] describe a Grid architectural style and a light-weight methodology for constructing Grids. Their work is based on a set of general functional requirements and quality attributes that derives an architectural style that includes information, control, and execution. Mattmann et al. [22] analyze software engineering challenges for large-scale scientific applications, and propose a general reference architecture that can be instantiated and adapted for specific application domains. We agree on the benefits obtained with a general architecture for Grid components to be instantiated for specific projects, however, our focus is on the inner workings of the components making up the architecture. The idea of software that evolves due to unforeseen changes in the environment also appears in the literature. In the work by Smith et al. [3], the way software is modified over time is compared with Darwinian evolution. In this work, the authors discuss the best-of-breed approach, where an organization collects and assembles the most suitable software component from each niche. The authors also construct a taxonomy of the “species” of enterprise software. A main difference between this work and our contribution is that our work focuses on software design criteria. Other high-level visions of Grid computing include that of interacting autonomous software agents [14]. One of the characteristics of this vision is that software engineering techniques employed for software agents can be reused with little or no effort if the agents encompasses the service’s vision [21]. A different view on agent-based software development for the Grid is that of evolution based on competition between resource brokering agents [4]. These projects differ from our contribution as our tools have a stricter focus on functionality (being welladjusted to their respective niches). Finally, it is also important to notice that there are a number of tools that simplify the development of Grid software. These tools facilitate, for example, implementation [18], unit testing [5], and automatic integration [1].

Designing Service-Based Resource Management Tools

6

269

Concluding Remarks

We explore the concept of the Grid ecosystem, with well-defined niches of functionality and natural selection (based on competition, innovation, evolution, and diversity) of software components within the respective niches. The Grid ecosystem facilitates the use and composition of components on a per-component basis. We discuss fundamental requirements for software to be well-adjusted to this environment and propose an approach to software development that complies with these requirements. The feasibility of our approach is demonstrated by five case studies. Future directions for this work include further exploration of processes and practices for development of Grid software.

Acknowledgements We acknowledge Magnus Eriksson for valuable feedback on software engineering standardization matters.

References 1. B´egin, M.-E., Diez-Andino, G., Di Meglio, A., Ferro, E., Ronchieri, E., Selmi, M., Zurek, M.: Build, configuration, integration and testing tools for large software projects: ETICS. In: Guelfi, N., Buchs, D. (eds.) RISE 2006. LNCS, vol. 4401, pp. 81–97. Springer, Heidelberg (2007) 2. Cakic, J., Paige, R.F.: Origins of the Grid architectural style. In: Engineering of Complex Computer Systems. 11th IEEE Int. Conference, IECCS 2006, pp. 227– 235. IEEE CS Press, Los Alamitos (2006) 3. Smith David, J., McCarthy, W.E., Sommer, B.S.: Agility – the key to survival of the fittest in the software market. Commun. ACM 46(5), 65–69 (2003) 4. Dimou, C., Mitkas, P.A.: An agent-based metacomputing ecosystem (October 2007), visited, http://issel.ee.auth.gr/ktree/Documents/RootFolder/ISSEL/Publications/ BiogridAnAgent-basedMetacomputingEcosystem.pdf 5. Duarte, A., Cirne, W., Brasileiro, F., Machado, P.: GridUnit: software testing on the Grid. In: Anderson, K.M. (ed.) Software Engineering. 28th Int. Conference, ICSE 2006, pp. 779–782. ACM Press, New York (2006) 6. Elmroth, E., Gardfj¨ all, P.: Design and evaluation of a decentralized system for Grid-wide fairshare scheduling. In: Stockinger, H., et al. (eds.) First International Conference on e-Science and Grid Computing, pp. 221–229. IEEE CS Press, Los Alamitos (2005) ¨ 7. Elmroth, E., Gardfj¨ all, P., Norberg, A., Tordsson, J., Ostberg, P.-O.: Designing general, composable, and middleware-independent Grid infrastructure tools for multitiered job management. In: Priol, T., Vaneschi, M. (eds.) Towards Next Generation Grids, pp. 175–184. Springer-, Heidelberg (2007) 8. Elmroth, E., Hern´ andez, F., Tordsson, J.: A light-weight Grid workflow execution engine enabling client and middleware independence. In: Wyrzykowski, R., et al. (eds.) Parallel Processing and Applied Mathematics. 7th Int. Conference, PPAM 2007. Lecture notes in Computer Science, Springer, Heidelberg (2007)

270

E. Elmroth et al.

9. Elmroth, E., Tordsson, J.: An interoperable, standards-based Grid resource broker and job submission service. In: Stockinger, H., et al. (eds.) First International Conference on e-Science and Grid Computing, pp. 212–220. IEEE CS Press, Los Alamitos (2005) 10. Elmroth, E., Tordsson, J.: A standards-based Grid resource brokering service supporting advance reservations, coallocation and cross-Grid interoperability. Concurrency and Computation: Practice and Experience (submitted to, 2006) 11. Elmroth, E., Tordsson, J.: A Grid resource brokering algorithms enabling advance reservations and resource selection based on performance predictions. In: Future Generation Computer Systems. The International Journal of Grid Computing: Theory, Methods and Applications (to appear, 2008) 12. Expert Group on Next Generation Grids 3 (NGG3). Future for European Grids: Grids and service oriented knowledge utilities. Vision and research directions 2010 and beyond (2006), visited October 2007, ftp://ftp.cordis.lu/pub/ist/docs/grids/ngg3 eg final.pdf 13. Finkelstein, A., Gryce, C., Lewis-Bowen, J.: Relating requirements and architectures: a study of data-grids. J. Grid Computing 2(3), 207–222 (2004) 14. Foster, I., Jennings, N.R., Kesselman, C.: Brain meets brawn: why Grid and agents need each other. In: Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, vol. 1, pp. 8–15. IEEE CS Press, Los Alamitos (2004) 15. Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design Patterns. In: Elements of Reusable Object-Oriented Software, Addison-Wesley, Reading (1995) 16. Gardfj¨ all, P., Elmroth, E., Johnsson, L., Mulmo, O., Sandholm, T.: Scalable Gridwide capacity allocation with the SweGrid Accounting System (SGAS). Concurrency and Computation: Practice and Experience (accepted, 2007) 17. Guan, Z., Hern´ andez, F., Bangalore, P., Gray, J., Skjellum, A., Velusamy, V., Liu, Y.: Grid-Flow: a Grid-enabled scientific workflow system with a petri-net-based interface. Concurrency Computat.: Pract. Exper. 18(10), 1115–1140 (2006) 18. Hastings, S., Oster, S., Langella, S., Ervin, D., Kurc, T., Saltz, J.: Introduce: an open source toolkit for rapid development of strongly typed Grid services. J. Grid Computing 5(4), 407–427 (2007) 19. Hern´ andez, F., Bangalore, P., Gray, J., Guan, Z., Reilly, K.: GAUGE: Grid Automation and Generative Environment. Concurrency Computat.: Pract. Exper. 18(10), 1293–1316 (2006) 20. ISO/IEC. Software engineering - Product quality - Part 1: Quality model. International standard ISO/IEC 9126-1 (2001) 21. Leong, P., Miao, C., Lee, B.-S.: Agent oriented software engineering for Grid computing. In: Cluster Computing and the Grid. 6th IEEE Int. Symposium, CCGRID 2006, IEEE CS Press, Los Alamitos (2006) 22. Mattmann, C.A., Crichton, D.J., Medvidovic, N., Hughes, S.: A software architecture-based framework for highly distributed and data intensive scientific applications. In: Anderson, K.M. (ed.) Software Engineering. 28th Int. Conference, ICSE 2006, pp. 721–730. ACM Press, New York (2006) 23. Networked European Software and Services Initiative (NESSI), visited October 2007, http://www.nessi-europe.com 24. Sandholm, T., Gardfj¨ all, P., Elmroth, E., Johnsson, L., Mulmo, O.: A serviceoriented approach to enforce Grid resource allocations. International Journal of Cooperative Information Systems 15(3), 439–459 (2006) 25. The Globus Project. An “ecosystem” of Grid components, visited October 2007, http://www.globus.org/grid software/ecology.php 26. The Grid Infrastructure Research & Development (GIRD) project. Ume˚ a University, Sweden (visited October 2007), http://www.gird.se

BC-MPI: Running an MPI Application on Multiple Clusters with BeesyCluster Connectivity Paweł Czarnul Faculty of Electronics, Telecommunications and Informatics Gdansk University of Technology, Poland [email protected] http://fox.eti.pg.gda.pl/∼pczarnul

Abstract. A new software package BC-MPI which allows an MPI application to run on several clusters with various MPI implementations is presented. It uses vendor MPI implementations for communication inside clusters and exploits the multithreaded MPI THREAD MULTIPLE mode for handling inter-cluster communication in additional threads of the MPI application. Furthermore, a BC-MPI application can be automatically compiled and started by the BeesyCluster middleware. The latter allows users to manage and use cluster accounts via a single BeesyCluster account and WWW or Web Services. The middleware connects to clusters via SSH and does not require any software installation on the clusters. Results of various latency and bandwidth tests for intra and inter-cluster communication are presented for BC-MPI using OpenMPI and LAM/MPI and Infiniband or TCP within clusters. Keywords: WAN-aware MPI, threads and MPI, grid middleware.

1 Introduction In recent years we have observed a growth of interest in both highly parallel software solutions and bridging clusters using grid technology. The first included progress towards thread-safe MPI implementations (OpenMPI), focusing on transparent checkpointing for MPI applications (LAM/MPI/BLCR or MPICH-V), failure-proof parallel algorithms in view of large parallel machines like IBM’s BlueGene, mixed shared memory and MPI programming etc. The latter resulted in releases of high-level grid systems like CrossGrid ([1]), CLUSTERIX ([2]) and also MPI implementations for WANs like MPICH-G2 ([3]) based on grid middleware Globus Toolkit ([4]). Still shortcomings can be identified especially regarding: – difficulty of installation, complex configuration and version compatibility of grid middlewares, – limited availability of high level grid systems for a broader community, – a usually complex process for middleware configurations, setup, distribution of account credentials, etc. 

Partially covered by MNiI grant No N516 035 31/3499.

R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 271–280, 2008. c Springer-Verlag Berlin Heidelberg 2008 

272

P. Czarnul

2 Related Work and Motivations The author proposes a software package, BC-MPI, that allows running an MPI application over several distributed clusters, using different MPI implementations on various clusters if needed. The package consists of a library, a compilation script replacing mpicc to preprocess and replace MPI * calls with ones shipped with BC-MPI as well as (optional) TCP forwarders for inter-cluster communication. There are MPI implementations available allowing an MPI application to span over several clusters. Examples include MPICH-G2 ([3]) using Globus ([4]) for job control (startup, monitoring, termination) and TCP messaging ([5]). PACX-MPI ([6]) and LAM/ MPI ([7]) can use Globus to couple remote clusters and start the application. Interoperable MPI ([8]) defines a protocol for communication between MPI implementations. Compared to other MPIs for WANs like MPICH-G2 or PACX-MPI, benefits of the proposed solution include: 1. BC-MPI is designed to exploit multithreading (MPI THREAD MULTIPLE model, [9]) as additional threads are used for handling inter-cluster communication in MPI processes. In BC-MPI each MPI process is multithreaded and one per cluster serves as a proxy for inter-cluster communication unlike in PACX-MPI – where additional processes acting as cluster proxies are created. This is discussed in paragraph 3. 2. BC-MPI can use any MPI implementations on cluster sides. This allows to exploit additional features of the particular MPI application, if necessary. 3. ability to start the BC-MPI application using the BeesyCluster middleware. Although the package is also meant as a standalone software with manual startup on remote clusters (much like PACX-MPI in [6]), it can benefit from automatic and secure startup via BeesyCluster ([10]) which is a J2EE application and middleware to clusters exposing WWW and Web Service interfaces. Contrary to using Globus, this approach does not require installation of any parts of the middleware on remote clusters since BeesyCluster accesses user accounts on such clusters via the Jsch SSH library. As such, it only requires that the BeesyCluster user provides system logins/passwords to accounts on remote clusters they would like to use through BeesyCluster. This allows to start an MPI application on several clusters via a single account in BeesyCluster to which several registered user accounts from clusters are mapped. Complex middlewares like Globus are more prone to modifications while plain Web Services are well established standards. [5] lists specific Globus versions MPICH-G2 does not operate with.

3 Architecture and Design of the Proposed Solution 3.1 Architecture of a BC-MPI Application The architecture of the BC-MPI application is depicted in Figure 1. BC-MPI uses multithreaded MPI processes one of which is a proxy for a cluster rather than distinguished processes like e.g. in PACX-MPI ([6]). If the cluster configuration requires access via a dedicated node then a TCP forwarder process can be launched there. In the case MPI is used to forward/receive from the proxy, communication links are equivalent to

BC-MPI: Running an MPI Application on Multiple Clusters

273

PACX-MPI. However, inter-proxy communication is potentially faster in BC-MPI since it requires only TCP communication and not MPI-TCP-MPI like in PACX-MPI. Application threads use MPI for communication within one cluster. BC-MPI does not require any changes in the MPI application source code.

app thread

recv thread

BC−MPI

send thread

BC−MPI

send thread

BC−MPI

recv thread

BC−MPI

app thread

send thread

BC−MPI

recv thread

BC−MPI

app thread

send thread

BC−MPI

recv thread

BC−MPI

app thread

optional TCP forwarder

TCP

vendor MPI

TCP or MPI

Fig. 1. Architecture of BC-MPI Application

3.2 BeesyCluster as a Middleware for BC-MPI BeesyCluster, installed at Academic Computer Center in Gdansk, Poland (at https:// beesycluster2.eti.pg.gda.pl/ek/Main), can be seen as an access portal/middleware to clusters/supercomputers/PCs with WWW (Figures 2, 3) and Web Service ([10]) interfaces. The user can access and use many accounts on various clusters through one account in BeesyCluster (single sign-on). Users can run any commands on clusters, edit, copy files and directories between clusters, queue or run tasks interactively, publish actions such as running a parallel or sequential application (run interactively or queued on clusters) or editing a file as services visible to other users via WWW or Web Services. For the use of services, users-providers earn points which can be spent on running services published by others. Services can also be offered free of charge and can be combined into workflows as presented by the author in [11]. BeesyCluster only presumes that user accounts on clusters are accessible via SSH and does not require installation of any software to run or publish services. Further information can be found in [10] or at the aforementioned web site. In such an environment, in the context of a BC-MPI application the user can: 1. Register existing accounts in a BeesyCluster account in seconds without assistance of remote clusters’ administrators. 2. Launch a WAN-aware MPI application (using BC-MPI) on these clusters via BeesyCluster. BeesyCluster can be used to upload application sources to target clusters, compilation and starting processes of the application on the clusters. In particular, BC-MPI’s MPI Init() can use BeesyCluster’s Web Services to launch processes of the application on other clusters. BeesyCluster’s Web Services can be called in a secure way using SOAP/HTTPS and use client authentication/authorization. Specifically, BeesyCluster’s Web Services require an initial call to method

274

P. Czarnul

Fig. 2. BeesyCluster’s File Manager in Web Browser

Fig. 3. Task’s Results in BeesyCluster

String[] auth = port.logIn(new String[] "", "<password>","loginAgentID","signerID") to log in and then allow to call any of the following ([10]): runCommand(auth,cluster,command)) for running a command, enqueueJob (auth,cluster,jobPath,minCPU,maxCPU, resultPath,email) for queueing a task using a queueing system on cluster (queueing details handled transparently by BeesyCluster), retrFile(auth,cluster,remoteFileName,localFileName) for retrieving a file from cluster, sendFile(auth,cluster,remoteFileName,localFileName) for sending a file to cluster. The latter can be used to download sources of a BC-MPI application from one cluster, upload to others, the first one to compile the source code. Then processes on other clusters could be started using one of the first two services. Figure 4 shows the startup sequence (download/upload ommitted). One process of an MPI application will act as a proxy waiting for socket connections and forwarding traffic to other clusters. Communication between clusters in a BC-MPI application will use TCP. 3. Optionally consume services made available by other BeesyCluster users from their accounts if proper rights have been set by the owners. This potentially extends the MPI application with ability to run external functionality using SOAP/HTTPS.

4 Multithreaded Implementation Calling MPI functions in threads other than the main one for forwarding to or receiving from the proxy process for inter-cluster communication requires proper threading support from the MPI implementation (possibly different in various clusters). Thus, for performance tests in this paper, in terms of multithreading the author tested three versions of the code: 1. OpenMPI in the MPI THREAD MULTIPLE mode, 2. LAM/MPI in the MPI THREAD SERIALIZED mode with the BC-MPI code including special synchronization to avoid deadlocks but only for the exemplary code,

BC-MPI: Running an MPI Application on Multiple Clusters

1. start app on fox (WWW)

275

BeesyCluster server(s) Sun

BeesyCluster’s client 1. start app on fox (SSH)

3. start app (SSH) 2. start app on parowiec and holk (Web Service) parowiec

holk 288 IA−64 procs Infiniband

fox 16 Xeon procs Fast Ethernet BC−MPI communication (TCP)

Fig. 4. Start-up of a BC-MPI Application using BeesyCluster

3. BC-MPI’s TCP – fully multithreaded (can be used with any MPI implementation) using sockets for forwarding/receiving to/from proxy for inter-cluster communication. MPI is still used for communication within one cluster. The following are used in BC-MPI, apart from the main MPI application threads: 1. Receive threads – used to listen to both internal communication (MPI or TCP) from MPI processes and from external clusters (only in the proxy process) via TCP. 2. Sending thread – forwarding to the proxy process or TCP communication to another cluster is done in another thread. This allows the client-side to continue and implement e.g. MPI I*send modes as well as continue receiving when forwarding data in proxy processes. Data is set in proper structures and a call to pthread cond signal instructs the sending thread to flush the send buffer. BC-MPI currently incorporates the following optimizations: receiving to user buffer when a matching receive has already been posted – if a message has not already arrived, an MPI Recv() call inserts a receive request (with the destination buffer specified by the user) into a table and waits on a condition; when the message arrives, it is received into the user buffer, partitioning and forwarding packets of long messages – the proxy process receives the message it should forward in packets; the received packet is forwarded immediately in another thread while receiving of next packets continues.

5 Performance Tests For performance tests, an MPI example for benchmarking point-to-point performance from [12] was used but modified to test latency/bandwidth between any MPI ranks. For each test described below a pair of clusters was used to benchmark point-to-point times

276

P. Czarnul

between MPI processes in separate clusters. Cluster holk features 288 Itanium2 processors with Infiniband, parowiec contains 16 Pentium Xeon processors with Fast Ethernet, fox is an AthlonXP PC node, all running Linux. We use a single MPI application compiled with BC-MPI with processes of ranks 0 to 3 on one cluster and rank 4 (and 5 in the last case) on the second cluster. Note the inter-departmental link between cluster parowiec and fox is of 10Mbit/s speed only. 5.1 Inter-cluster Communication and Communication within a Node in Cluster Figure 5 shows latency times and Figure 6 bandwidth for communication between ranks 1 and 4 of the MPI application. Communication occurs in configuration: 1 (parowiec) MPI (shared memory or TCP) - 0 (proxy process on parowiec) - inter-cluster interdepartmental TCP (10 Mbit/s) - 4 (proxy process on fox)). Processes 0 and 1 run on the 750

Intercluster Latency [us]

700

intercluster communication (TCP) 0-4

650

1-OMPI shm (THREAD_MULTI PLE) -0 – TCP -4

600 550

1-LAMMPI shm (THREAD_SERIA LIZED)-0–TCP-4

500 450

1-OMPI tcp -0 – TCP -4

400 350

1-LAMMPI tcp -0 – TCP -4

Message size [bytes]

300 0

32

64

96

128

160

192

224

256

Fig. 5. Latency for Inter-cluster Communication + Communication in a Node in Cluster 1.1 1.075

Intercluster Bandwidth [MB/s}

intercluster communication (TCP) 0-4

1.05

1-OMPI shm (THREAD_MULTI PLE) -0 – TCP -4

1.025 1

1-LAMMPI shm (THREAD_SERIA LIZED)-0–TCP-4

0.975

1-OMPI tcp -0 – TCP -4

0.95

1-LAMMPI tcp -0 – TCP -4

0.925

Message Size [bytes]

0.9 0

4096

8192

12288

16384

Fig. 6. Bandwidth for Inter-cluster Communication + Communication in a Node in Cluster

BC-MPI: Running an MPI Application on Multiple Clusters

277

same node. For communication between 0 and 1 both shared memory and TCP modes in MPI were used for comparison. It can be seen that LAM/MPI slightly outperforms OpenMPI (as also reported in [13]). The times are obviously dominated by the inter-cluster communication times. 5.2 Inter-cluster Communication and Communication between Nodes in Cluster Figure 7 shows latency times and Figure 8 bandwidth for communication between ranks 1 and 4. Communication occurs in configuration: 1 (parowiec) - MPI (TCP) or BCMPI’s TCP - 0 (proxy process on parowiec) - inter-cluster inter-departmental TCP (10 Mbit/s) - 4 (proxy process on fox)). Processes 0 and 1 run on separate nodes in cluster parowiec using Fast Ethernet. For communication between 0 and 1 both vendor MPI and BC-MPI’s TCP were used for comparison. It can be seen that as in Figure 5 750

Intercluster Latency [us]

700 650

1-OMPI tcp (THREAD_MULTI PLE) -0 – TCP -4

600 550 500

1-LAMMPI tcp (THREAD_SERIA LIZED)-0–TCP-4

450

1-BC-MPI's TCP -0 – TCP -4

Message size [bytes]

400 0

32

64

96

128

160

192

224

256

Fig. 7. Latency for Inter-cluster Communication + Communication between Nodes in Cluster 1 0.95 0.9 0.85 0.8 0.75 0.7 0.65 0.6 0.55 0.5

Intercluster Bandwidth [MB/s] 1-OMPI tcp (THREAD_MULTI PLE) -0 – TCP -4 1-LAMMPI tcp (THREAD_SERIA LIZED)-0–TCP-4 1-BC-MPI's TCP -0 – TCP -4

0

4096

8192

Message Size [bytes] 12288

16384

Fig. 8. Bandwidth for Inter-cluster Communication + Comm. between Nodes in Cluster

278

P. Czarnul

(communication between 0 and 1 on a single node) LAM/MPI offers slightly better latency times than OpenMPI (for TCP). Furthermore, BC-MPI’s TCP implementation is slightly faster than the code using LAM/MPI with MPI THREAD SERIALIZED. This may be in part due to the synchronization added to BC-MPI in the latter case so that LAM/MPI can be called in many threads without deadlocks. This synchronization is specific and would work only for the MPI application used for tests. OpenMPI was tested with MPI THREAD MULTIPLE which allows to call MPI in many threads without deadlocks. 5.3 Data Partitioning and Forwarding For larger messages the author tested the impact of partitioning messages and immediate forwarding of received data in rank 0 while continuing receiving. Figure 9 shows results for the configuration as in the previous case. It indicates that for large messages the latency of sending the message of size 8MBytes between processes 1 and 0 (around 0.7s between separate nodes in cluster parowiec) can be practically hidden. 12 forwarding whole messages

10 8

Intercluster Communication Times [s]

partitioning when forwarding

6 4 2

Message size [Kbytes]

0 0

1024 2048 3072 4096 5120 6144 7168 8192

Fig. 9. Latency for Inter-cluster Communication with and without Message Partitioning

5.4 Testing Infiniband vs. TCP and an TCP Forwarding In this configuration communication occurs between processes 1 and 5 in configuration: 1 (parowiec) - BC-MPI’s TCP or MPI (TCP) - 0 (proxy process on parowiec) inter-cluster TCP - additional TCP forwarder on access node karawela - inter-cluster TCP - 4 (proxy process on holk) - BC-MPI’s TCP or MPI (Infiniband) - 5 (holk). 1 and 0 as well as 4 and 5 run on separate nodes in clusters parowiec and holk respectively. Figures 10 and 11 show differences in communcation times for both small and large message sizes when using Infiniband within holk compared to TCP (Ethernet).

BC-MPI: Running an MPI Application on Multiple Clusters 1200

Intercluster Latency [us] Infiniband vs TCP in holk

1100 1000 900

279

1–BC-MPI's TCP–0–TCP–4– BC-MPI's TCP– 1–MPI(TCP)–0– TCP–4–MPI (IB)–5

800 700

Linear regression, 1– BC-MPI's TCP–

600 500 400

Linear regression, 1– MPI(TCP)–0–

Message Size [bytes]

300 0

512

1024

1536

2048

Fig. 10. Inter-cluster Communication Latency /w TCP vs Infiniband on holk: Short Messages 0.8

Intercluster Communication Times [s] Infiniband vs TCP in holk

0.7 0.6 0.5

1–BC-MPI's TCP–0–TCP– 4–BC-MPI's

0.4 0.3 0.2 0.1

Message Size [Kbytes]

1–MPI(TCP)– 0–TCP–4– MPI (IB)–5

0.0 0

512

1024 1536 2048

2560 3072 3584

4096

Fig. 11. Inter-cluster Communication Times with TCP vs Infiniband on holk

6 Summary and Future Work In this paper, a new software package for bridging MPI applications on clusters was presented. While it can be used as a standalone software with MPI implementations, it can benefit from the BeesyCluster middleware for automatic startup of the MPI application spanning clusters as well as spawning BeesyCluster services in parallel. Its architecture can exploit multithreaded features of MPI implementations e.g. MPI THREAD MULTIPLE of OpenMPI. Various performance tests of point-to-point communication between clusters were presented using BC-MPI’s TCP, LAM/MPI and OpenMPI. Future work includes further implementation of BC-MPI including collective operations, performance comparisons with other systems like MPICH-G2, PACX-MPI, LAM/MPI etc. as well as tests of TCP communication with encryption. The author will also focus on building an ontology for HPC computing and using it for intelligent searching of HPC services within BeesyCluster.

280

P. Czarnul

Acknowledgments Calculations were carried out at the Academic Computer Center in Gdansk, Poland.

References 1. Official Crossgrid Information Portal. supported by Grant No. IST-2001-32243 of the European Commission, http://www.crossgrid.org/main.html 2. CLUSTERIX: The National Linux Cluster, http://clusterix.pcz.pl 3. Karonis, N., Toonen, B., Foster, I.: MPICH-G2: A Grid-Enabled Implementation of the Message Passing Interface. Journal of Parallel and Distributed Computing (JPDC) 63(5), 551– 563 (2003) 4. Sotomayor, B.: The Globus Toolkit 4 Programmer’s Tutorial (November 2005), http://www.casasotomayor.net/gt4-tutorial/ 5. Karonis, N., Toonen, B.: MPICH-G2, http://www3.niu.edu/mpi/ 6. Keller, R., Mller, M.: The Grid-Computing library PACX-MPI: Extending MPI for Computational Grids, www.hlrs.de/organization/amt/projects/pacx-mpi/ 7. LAM/MPI Parallel Computing, http://www.lam-mpi.org/ 8. National Institute of Standards and Technology: Interoperable MPI, http://impi.nist.gov/ 9. Lusk, E., et al.: MPI-2: Extensions to the Message-Passing Interface: MPI and Threads, http://www-unix.mcs.anl.gov/mpi/mpi-standard/mpi-report-2.0/ node162.htm#Node162 10. Czarnul, P., Bajor, M., Fraczak, M., Banaszczyk, A., Fiszer, M., Ramczykowska, K.: Remote Task Submission and Publishing in BeesyCluster: Security and Efficiency of Web Service Interface. In: Wyrzykowski, R., Dongarra, J., Meyer, N., Wa´sniewski, J. (eds.) PPAM 2005. LNCS, vol. 3911, Springer, Heidelberg (2006) 11. Czarnul, P.: Integration of Compute-Intensive Tasks into Scientific Workflows in BeesyCluster. In: Alexandrov, V.N., van Albada, G.D., Sloot, P.M.A., Dongarra, J. (eds.) ICCS 2006. LNCS, vol. 3993, Springer, Heidelberg (2006) 12. Gropp, W., Lusk, E.: MPI Tutorial: Benchmarking point to point performance, http://www-unix.mcs.anl.gov/mpi/tutorial/mpiexmpl/src3/ pingpong/C/main.html 13. Barrett, B.: Open MPI User’s Mailing List Archives, http://www.openmpi.org/community/lists/users/2006/04/1076.php

Managing Distributed Architecture with Extended WS-CDL Konrad Dusza and Henryk Krawczyk Gdansk University of Technology, Narutowicza 11/12, 80-952 Gdansk, Poland [email protected], [email protected]

Abstract. The paper presents problems with WS-CDL in providing a central point of service layer management in a distributed system architecture. A proposal of CDLExt a model extension to WS-CDL addressing its drawbacks in services management is given. Features of CDLExt model include: describing dependencies between services and other IT artifacts and QoS attributes introduced by these dependencies. Furthermore, a toolset supporting service layer management with the new model is presented. Future directions for CDLExt development are also discussed. Keywords: WS-CDL, choreography, Web services, QoS.

1

Introduction

One of the latest trends in distributed systems architecture is service orientation. Widely known as SOA[1], it delivers a promise for more scalable, composable and business-oriented software. In order to support new architectural paradigm, two complementary approaches to supporting service layer management have been developed: choreography and orchestration. Orchestration can be defined as programming in-the-large in WSBPEL[2] or other programming languages in order to compose web services according to modeled business processes. On the other hand, choreography focuses on describing the observable behavior of the system, from a distant, not being a part of the system, perspective. At present, the most recognized approach for describing web services choreography is the Web Services Choreography Description Language (WS-CDL) developed by W3C[3]. Although, WS-CDL provides a set of constructs that seem sufficient for choreography description, its scope does not cover certain aspects of service layer management including: describing QoS characteristics of service interactions and modeling dependencies between services and other components existing in a organizations IT infrastructure. The purpose of this paper is to present CDLExt, a model proposal for service layer management based on WS-CDL extended with additional elements describing important service layer characteristics. The design of WS-CDL extension is based on WS-CDL feature set analysis from a service architect point of view. Features introduced by CDLExt are to enable service architects to achieve better R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 281–290, 2008. c Springer-Verlag Berlin Heidelberg 2008 

282

K. Dusza and H. Krawczyk

control of the service layer by supporting a more detailed view at IT infrastructure in service oriented organization than WS-CDL itself. Ideas and scenarios are presented in this paper within a context of an organization which is transitioning its distributed, legacy architecture towards a service-oriented one. Moreover, we assume that the organization maintains description of current state of service layer in form of a WS-CDL described choreography. By a mature SOA we understand the definition provided by the OASIS SOA Technical Committee[1]. The remainder of this paper is organized as follows. The next section presents an overview of concepts and issues related to service layer management. Section 3 gives more in-depth analysis of areas which are not covered by WS-CDL and their importance in service layer management. Sections 4 and 5 cover details on the CDLExt and tool supporting this solution. Section 6 discusses future plans and areas for CDLExt improvement. Final section of the paper presents conclusion.

2

Managing the Service Layer

Service orientation, meant as the encapsulation of organizations business-related functionalities in form of services, implies logical and often physical distribution of system architecture components. Effective SOA requires high-level management in order to ensure fulfilling service contracts negotiated between the organization and its services consumers. Service layer management involves various tasks including: creating, modifying and removing services from the layer, monitoring service activities and risks of failing a services contract, service-enabling of existing legacy components, etc. Many of those tasks are supported in a variety of ways by a plethora of tools and standards available. Moreover, even the initial version of web services standards were designed with support for service management in mind. One of the first standards related to web services was WSDL[4] the language for specification of functional capabilities of services. Later, the concept of service registries that would help discovering needed services was introduced in form of UDDI[5]. UDDI-compliant registries can store references to services deployed in distributed locations, granting authorized parties access to service registry data via a single point of entry. With the advancement of web services development, the concept of services composability was reintroduced in form of business process modeling and mapping of business processes onto web service invocations. Orchestration engines delivered the functionality of interpreting such scripted web services invocations and support for some non-functional aspects of service layer including: logging, authorization and authentication management, etc. The extension to the concept of orchestration engine - the Enterprise Service Bus[6] (ESB) provided a single point of entry into the IT infrastructure by integrating various technologies of IT infrastructure (CORBA[7], Web Services, etc.). Each of the aforementioned concepts, addresses certain aspects of service layer management. However, solutions they provide are either fragmentary (like service listing in form of registry) or too detailed and orchestration-centered

Managing Distributed Architecture with Extended WS-CDL

283

(e.g. ESB). Lack of a generalized, top-level insight into service layer state hinders reliable evaluation of organizations capability of fulfilling negotiated service contracts. On the other hand, the capabilities of WS-CDL, which include describing both the static side of service layer (e.g. enumeration of services, providing services description, etc.) and the dynamic side (describing expected observable behavior on a service layer) gave us reason to pick WS-CDL as the foundation for further research on a lightweight and complete solution for service layer management. WS-CDL capabilities and potential areas for extending its service layer model are discussed in the next section.

3

Reasons for Extending WS-CDL

The most common scenarios with regard to building a service layer in an organization can be described with keywords: create, compose and use. Create scenario represents encapsulation of a business-relevant functionality in form of a service. Compose scenario, is a special case of create that involves combining existing services into a service offering new functionality from the business point of view. Use scenario describes the customer side of a create/compose situation adapting the organization to consuming a service provided by an external provider. Apart from implementation and deployment, all the aforementioned situations usually involve: formalization of service contract agreed between the consumer and the provider, identification of service dependencies and arranging the service layer so that service invocations are possible, effective, reliable and service contract compliant. To conduct all aforementioned tasks, a top-level view of the organizations service layer is needed by a service architect (the one in charge of the service layer). Main purpose of WS-CDL is to provide means to describe the sequence and conditions in which the messages are exchanged[8], with regard to Web Services. This is to be achieved from a perspective of a third-party observer that does not interfere in any of the message exchanges. Although, WS-CDL and WSDL offer a wide array of elements for modeling services functionalities and their interactions, they do not provide sufficient support for other activities related to service layer management. Examples of such important features are given in the following paragraphs. Firstly, effective management of service layer being the front line for providing business functionalities needs proper identification and verification of services dependencies due to their impact on organizations ability to fulfill service contracts. For instance, it could happen that two independently designed and implemented component, i.e. a service and a legacy component, access the same database in a manner that introduces race condition (see Fig. 1). If such case, one of the components is unable to work properly, what will affect the organizations workflow. Moreover, such problem could be discovered after production deployment of the service, and therefore introduces severe risk for the organization violation of service contract.

284

K. Dusza and H. Krawczyk

Secondly, if SOA is to be the core for the organizations business, it is necessary that both functional and the non-functional side of a service contract are properly served. To define the former, the WSDL standard has been developed, and currently it is has reached an acceptable degree of maturity. Even though the latter, named quality of service (QoS), gained recognition in the web services world [9] [10] and was discussed within W3C [11], it has not become a part of any service contract-related specification. As an example for QoS importance in service contract we could use a modified scenario from the previous paragraph (see Fig. 1). Having resolved the race condition possibility (i.e. through transactions), system architect could face a situation, in which the legacy component overloads the database with queries, so that the service is unable to respond to customers with response time above the minimum described in service contract. Once again a problem with fulfilling the contract by the service provider has arisen. In our opinion, separation of service layer from its dependencies in

Fig. 1. Example of access conflict that could disrupt a service execution

existing IT architecture and QoS aspects makes it difficult for service architect to manage the layer in an effective way. Whereas WS-CDL provides sufficient means to model interactions, it should be reinforced with additional constructs to support other service layer management tasks in an effective way.

4

Model of the CDLExt Extension

Analysis of problems related to service layer management described in section 3 resulted in the CDLExt model presented in Fig. 2. The choreography of a service layer is represented by the Package element and its subelements, according to the WS-CDL specification. The Artifacts element represents the IT infrastructure

Managing Distributed Architecture with Extended WS-CDL

285

Fig. 2. CDLExt model structure. The package element is the WS-CDL root, the Artifacts element is the root of the extension.

layer of an organization. In order to allow different types of IT infrastructure artifacts to be modeled, an abstract Artifact element and its derivative types were created (Business/Database/Service/Other). Distinction between various types of IT artifacts enables enriching each of the artifact types with typespecific attributes. For example, in the CDLExt model, the Service artifact was extended with the RoleRef attribute which can contain a reference to a WS-CDL RoleType element instance. The Dependency element serves the purpose of dependency modeling, whereas the Load element enables modeling the load which is introduced by a particular dependency. Among various QoS characteristics, load introduced by dependencies was chosen as the most important QoS aspect regarding the ability of an organization to fulfill a service contract. A detailed description of CDLExt elements with shapes related to dependency modeling is given in Table 1. Notably, the CDLExt syntax has been described in an XSD schema, which can be used for validation of a CDLExt document.

5

CDLExt Toolset

In order for the proposed extension to be effective and truly supporting service architects tasks of managing the organizations service layer, a special toolset has been developed. In the following subsections, three main development steps are presented. 5.1

Design

Toolset design was based on the Model-View-Controller (MVC) pattern, with a miscellaneous tool package added to the toolset. The general architecture of the CDLExt toolset is shown in Fig. 3. The naming convention on the diagram is metamodel, as it represents the model for CDLExt instances models of service layer.

286

K. Dusza and H. Krawczyk

Table 1. Description of CDLExt elements and attributes Element

Attributes

CDLExt Package

root element as in ws-cdl package root element for containing the choreograelement phy definition root element for extension part ID, name abstraction representing artifacts in IT architecture Artifact-derived database tier artifact Artifact-derived business tier artifact

Artifacts Artifact Database Business

Description

Shape -

cuboid round. rect. Other Artifact-derived other kind of IT artifact rect. Service Artifact-derived, rol- service layer artifact, allows referencing a circle eRef role from ws-cdl part Dependency ID, to dependency between artifacts, the artifact arrow containing the dependency is depenendent on the artifact pointed by to attribute Load amount, timeunit, QoS related element data on the load rect. type that the containing dependency element linked enforces on the dependent on artifact to dep.

Metamodel structure core part of CDLExt support. It represents the implementation of CDLExt metamodel including syntax and semantics rules of WS-CDL and the extension part as described in section 4 (see Table 1). Metamodel logic logic of model editing. It is based on the strategy/policy and command patterns[12], an enable undo/redo for operations on a CDLExt instance, validation of editing operations, etc. Examples of operation on the model include adding a valid element to the CDLExt instance (e.g. WS-CDL RoleType), removing an element, etc. Metamodel editing graphical editors for editing different parts of a CDLExt model (e.g. modeling dependencies and QoS, modeling static aspects of WS-CDL, etc.). Support tools supporting tasks related to web service management but unrelated to CDLExt code generation, editors for web services-related languages, etc. The choice of shapes representing model elements in the toolsets graphical editors (like circle for services, etc.) is arbitrary. Shapes relevant to dependency modeling are presented in Table 1. To represent static choreography elements from WS-CDL, a similar notation was chosen (ie. rectangles and lines). However, if any of graphical notation elements are proven unclear or more appropriate shapes are found, the graphical notation will be subject to change.

Managing Distributed Architecture with Extended WS-CDL

287

Fig. 3. CDLExt toolset architecture diagram. Main elements are shown within the dashed rectangle labeled MVC, auxiliary tools like syntax coloring are symbolized by the Support element.

5.2

Implementation

The aforementioned toolset has been implemented using Eclipse 3.3 and supporting technologies: Eclipse Modeling Framework[13], Graphical Modeling Framework[14] and other minor Eclipse Foundations projects, which EMF and GMF depend on. The most important part of toolset development was to implement the CDLExt metamodel described in section 4 (metamodel structure) with use of the Eclipse Modeling Framework. The EMF implementation of the CDLExt metamodel (in EMF such metamodel is called ecore) should reflect the syntax and semantics of WS-CDL and the extension. In order to achieve that, the WSCDLs XSD schema had to be enriched with additional EMF attributes according semantic rules described in the WS-CDL specification (for instance: WS-CDL RoleRef element allowing pointing to only to existing Role elements). Next steps of implementation focused on fine-tuning of EMF and GMF-generated editors (metamodel logic and metamodel editing) and enriching them with additional features (support). Current version of the CDLExt toolset (0.8) delivers the following functionalities: CDLExt document editing, graphical modeling of artifacts and dependencies, graphical modeling of static WS-CDL part (roles, participants, relationship types, etc.), validation of CDLExt documents and others. At present, toolset development is focused on improving the editors look and feel (colors, icons, etc.) and visualization of activity-related WS-CDL parts in form similar to UML sequence diagrams. 5.3

Testing

The purpose of testing process was to evaluate whether CDLExt addresses problems discussed in section 3, to verify the usability of the metamodel and its toolset and to discover areas for further improvements in CDLExt and its toolset.

288

K. Dusza and H. Krawczyk

Fig. 4. One of scenarios used for testing of the CDLExt metamodel. Customer represents a service consumer, whereas other entities (Store, Warehouse, Pricing) represent services, with Store being a composite one.

We have tested the CDLExt toolset in five service layer scenarios. Each scenario included creation of at least 1 composite service (2 or more services composed) or 3 simple services. One of test scenarios is presented in Fig. 4. The service layer configuration description capabilities (enumeration of services and their interactions) provided by WS-CDL have proven sufficient for describing different service interactions scenarios, including request, request/response and notification cases. CDLExt features, i.e. IT artifacts and dependency modeling were found useful in early discovery of risk for service contract fulfillment introduced by services concurrent access to a shared resource. Furthermore, tests confirmed that given a general view provided by the toolset, a service architect can elaborate more realistic and therefore less risky to fulfill service contracts. Without visualization of service dependencies, it is very hard to evaluate, whether a service with given set of parameters can be delivered under specified contract and budget. Is such case, dependency visualization and modeling is in fact the recognition of service contracts risks. However, the existing QoS modeling capabilities provided unsatisfactory QoS description capabilities for 1 scenario customer demanded a high load level for 8 hours a day and low load level for the rest of the day. Modeling such distribution is impossible with present, flat average characteristic.

Managing Distributed Architecture with Extended WS-CDL

289

The most important areas for further extension in the metamodel are: extending the dependency QoS modeling capabilities to enable more complex load distribution modeling, and revision of WS-CDL interaction description constructs, including verification of channel variables usability for service layer management tasks. Future improvements of the toolset should follow changes made to the metamodel.

6

Future Plans

Establishment of the core part of CDLExt and the toolset provided a starting point for future extensions and improvements. These features can be categorized in the following way. Automatic verification of new constructs against choreography implementing formal verification of programmed services (e.g. WSBPEL) against the choreography by projecting them on a common process algebra like FSP[15] or other specifically designed for that purpose[16] and verifying whether the programmed service composition fits organizations interactions model described in the choreography. Such feature would increase the value of a properly maintained choreography in the organization by ensuring early recognition of choreography breach by new services (create/compose/use scenarios). Extending graphical editing capabilities for the CDLExt metamodel providing a graphical editor for modeling interactions of WS-CDL would certainly ease WS-CDL editing tasks, and therefore make use of CDLExt more efficient and productive. Extending QoS features of the CDLExt metamodel complementing the CDLExt with additional elements for more adequate modeling of the IT infrastructure characteristics that have influence on the organizations capability of fulfilling the non-functional aspects of service contracts e.g. reliability. For example, adding the higher estimate of downtime per year attribute (in seconds) to the Artifact element to model availability of an IT artifact, which could be used to calculate a service availability for a contract basing on its dependencies availability values.

7

Summary

We have proposed a WS-CDL based solution for top-level service layer management. In our opinion, effective transition of IT infrastructure to SOA, will be impossible without providing a single point of insight into current service layer state and related architectural dependencies. The functionalities of service registries should be extended for supporting listing of services, their interactions (wires), QoS capabilities and their dependencies. Otherwise, transition to SOA will be fragmentary and not triggering its full potential risking breaching of service contracts and reducing services business value. The proposed CDLExt model is the first step to eliminate such discrepancy problems.

290

K. Dusza and H. Krawczyk

References 1. OASIS SOA Reference Model TC: OASIS Reference Model for Service Oriented Architecture V 1.0, http://www.oasis-open.org/committees/download.php/19679/soa-rm-cs.pdf 2. OASIS WSBPEL Technical Committee: Web Services Business Process Execution Language Specification. http://www.oasis-open.org/committees/download.php/18714/ wsbpel-specification-draft-May17.htm 3. W3C: Web Services Choreography Description Language Version 1.0. http://www.w3.org/TR/2005/CR-ws-cdl-10-20051109/ 4. W3C: Web Services Description Language, http://www.w3.org/TR/wsdl 5. UDDI Spec Technical Committee: Universal Description Discovery and Integration (UDDI), http://uddi.org/pubs/uddi v3.htm 6. Keen M., Bishop S., Hopkins A., Milinski S., Nott C., Robinson R., Adams J., Verschueren P., Acharya A.: Patterns: Implementing an SOA using an Enterprise Service Bus. IBM Redbook (2004), http://www.redbooks.ibm.com/abstracts/SG246346.html?Open 7. Object Management Group: Common Object Request Broker Architecture (CORBA), http://www.omg.org/technology/documents/formal/corba iiop.htm 8. W3C: WS Choreography Model Overview, http://www.w3.org/TR/2004/WD-ws-chor-model-20040324/ 9. Menasce, D.A.: Composing Web Services: A QoS View. IEEE Internet Computing, XI-XII, IEEE (2004) 10. Mani A., Nagarajan A.: Understanding quality of service for Web services. IBM developerWorks (2002), http://www-128.ibm.com/developerworks/library/ws-quality.html 11. W3C Working Group: QoS for Web Services: Requirements and Possible Approaches (November 25, 2003), http://www.w3c.or.kr/kr-office/TR/2003/ws-qos/ 12. Gamma, E., Helm, R., Johnson, R., Vlissides, J.M.: Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, Reading (1995) 13. Budinsky, F., Steinberg, D., Merks, E., Ellersick, R., Grose, T.J.: Eclipse Modeling Framework. Addison-Wesley, Reading (2003) 14. Eclipse Graphical Modeling Framework project page, http://www.eclipse.org/gmf/ 15. Foster, H., Uchitel, S., Magee, J., Kramer, J.: Model-Based Analysis of Obligations in Web Service Choreography. In: Advanced International Conference on Telecommunications and International Conference on Internet and Web Applications and Services (AICT-ICIW 2006), p. 149 (2006) 16. Busi, N., Gorrieri, R., Guidi, C., Lucchi, R., Zavattaro, G.: Choreography and Orchestration: a synergic approach for system design. In: Benatallah, B., Casati, F., Traverso, P. (eds.) ICSOC 2005. LNCS, vol. 3826, pp. 228–240. Springer, Heidelberg (2005)

REVENTS: Facilitating Event-Driven Distributed HPC Applications Dawid Kurzyniec, Vaidy Sunderam, and Magdalena Sławi´nska Dept. of Math and Computer Science, Emory University 400 Dowman Drive, Atlanta, GA 30322, USA {dawidk,vss,magg}@mathcs.emory.edu

Abstract. Modern scientific applications that need to share geographically scattered resources and dynamically adapt to changes in the environment pose challenges to traditional parallel and distributed programming paradigms. Distributed component frameworks attempt to address the demands of contemporary HPC applications by enabling coarse-grained decomposition and loose coupling. Nonetheless, components usually communicate via synchronous RPC, which is not suitable for interactive applications. This paper introduces a novel distributed event notification system, called REVENTS, which enables both synchronous and decentralized asynchronous component interactions. The REVENTS system is based on a topic-list publisher-subscriber model. It integrates and enhances common technologies for messaging, events, and group communication. The article introduces the REVENTS API, its reference implementation, and its application in the H2O metacomputing framework. Presented experimental results confirm REVENTS’ usability in distributed HPC scenarios. Keywords: distributed event system, middleware, loosely coupled system.

1 Introduction Seamless resource aggregation across administrative domains is one of the objectives of grid computing. However, metacomputing over geographically distributed machines introduces numerous challenges related to resource and network heterogeneity, unreliability, and dynamicity. In grid environments, resources appear and disappear dynamically, load changes unpredictably, and nodes exhibit non-uniform performance characteristics. Additionally, scientific applications are becoming increasingly more sophisticated. Unlike traditional, monolithic, uni-modal parallel codes executed in batch mode on supercomputing clusters, modern applications often consist of multiple coupled simulation kernels, and support interactive steering and visualization. Traditional parallel programming paradigms are not designed to cope with these new levels of complexity. For instance, the Single-Program-Multiple-Data (SPMD) model, commonly used in conjunction with the message passing communication paradigm on 

Research, publication, and presentation supported in part by DOE grant DE-FG0206ER25729, NSF grant CNS-720761, and a faculty travel grant from The Institute of Comparative and International Studies, Emory University.

R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 291–301, 2008. c Springer-Verlag Berlin Heidelberg 2008 

292

D. Kurzyniec, V. Sunderam, and M. Sławi´nska

massively parallel machines, assumes a static collection of uniform and reliable computing nodes linked with a low-latency interconnect. Performance of SPMD applications degrades dramatically when these assumptions are not met, as is the case with grid environments. Support for interaction and coupling of simulation kernels are difficult to implement within SPMD. Clearly, new programming approaches are needed to allow modern scientific applications to take advantage of the cumulative distributed computational power of grid environments. Distributed component frameworks have been proposed as a deployment technology that might be better suited to address these emerging application requirements. The component-based approach facilitates coarse-grained, heterogeneous application decomposition, which maps naturally to widely distributed systems. While individual components may still internally encapsulate parallel solvers requiring tightly-coupled, co-located resources, they externally present coarse-grained and opaque interfaces and service endpoints. In this model, applications are constructed by deploying necessary components and linking their communication interfaces. Existing distributed component frameworks typically use a variant of Remote Procedure Calls (RPC) or Remote Method Invocations (RMI) paradigms to implement component interactions. RPC (or RMI) is appropriate for controlling the standard, sequential execution flow. However, interactive applications that need to respond to asynchronous events, such as user requests or resource availability changes, may be better served by a distributed event notification paradigm. This paper describes REVENTS – an API and a reference implementation of a distributed event notification middleware system based on the topic-list publishersubscriber model. REVENTS combines features provided individually by various event, messaging, and group communication technologies [1,2,3,4,5], augmenting and extending them as necessary to apply in the context of distributed event notification. The REVENTS API is partially based on Java Messaging Service API (JMS) [1], a popular messaging technology for enterprise middleware. We give an overview of the REVENTS API and its unique features, discuss its use case in the H2O framework [6], and briefly evaluate its performance characteristics, assessing its applicability for scientific applications.

2 Related Work Event notification is a communication paradigm commonly used in conjunction with remote procedure calls in distributed component- and service-oriented systems [3,5,7,8,9]. This section discusses REVENTS in the context of the most important related projects. ECho [10] is an event-based middleware library, providing flexible event typing functionality and efficient binary transport. It supports group communication by allowing multiple (distributed) sources to publish events to a common channel, and uses dynamic code generation to propagate subscription filters from event sinks to event sources. In comparison, REVENTS is agnostic with respect to the communication protocol, enabling interoperability with third-party event systems. It refrains from supporting group communication facilities at the low level, stressing full decentralization. It utilizes SQL-based expression syntax for specifying event filters, avoiding security and

REVENTS: Facilitating Event-Driven Distributed HPC Applications

293

logistical complications related to dynamic code generation. Finally, it supports initial state notification for late joining subscribers, which is not addressed by ECho. Java Message Service (JMS) [1] is a messaging standard for JEE applications to create, send, receive, and read messages. Although REVENTS derives many abstractions from JMS, including topic-based publish-subscribe APIs, message types and metadata, filter expressions, and expiration semantics, it targets different application scenarios. Contrary to JMS, REVENTS is an asymetric API, distinguishing event sources from event consumers. It does not require deployment of dedicated middleware infrastructure except on the directly involved (event source and event sink) machines. It provides APIs for dynamic topic creation and deletion, subscription listing and notification. It supports features specific to event (rather than message) delivery, such as initial state notification, and event overriding. WS-Messenger [8] is an event broker for grid systems, based on Web Services technologies. It delegates the event delivery semantics and implementation to the underlying messaging middleware (e.g. JMS). WS-Messenger is intended for handling infrastructure events rather than being used as an application runtime library.

3 REVENTS Overview The REVENTS library provides an API and a reference implementation for distributed event notification based on the publisher-subscriber model with a topics list, as shown in Figure 1. In this model, events are generated by publishers asFig. 1. The architecture of REVENTS sociated with specific topics, and remote subscribers register subscriptions on their topics of interest. A subscription may be considered a proxy for its remote subscriber; events are propagated through the subscription to the subscriber through an event channel providing transport and session control mechanisms. Channel Implementation. Asynchronous messaging middleware systems usually mandate deployment of designated, centralized services for storing queued messages enroute from the publisher to the subscriber. Such an approach enables loose coupling, allowing message exchange between parties that are not simultaneously online. In contrast, event channels in REVENTS link clients and servers directly. Publishers insert events to the channel’s server-side queue, and subscribers pull them from the clientside queue, as shown in Figure 1. The channel is responsible for transferring events between queues, necessitating that both parties are simultaneously online. This design choice has the advantage of decentralization, enabling ad-hoc application composition without the need for deployment of additional dedicated services. If needed though, decoupling can be enabled through a dedicated proxy.

294

D. Kurzyniec, V. Sunderam, and M. Sławi´nska

Event Structure. Each event contains (1) a header, consisting of common fields such as an expiration date, priority, etc., (2) a set of properties (metadata) which can be used to further filter out the events using selectors (described below), and (3) a body (payload) depending on the event type. Access to header fields is provided via dedicated methods in the base Event interface, and to properties – via appropriate getter and setter methods. Access to the body is provided via subinterfaces, separately for each event type. REVENTS supports two groups of events: control and ordinary. The former are generated automatically by the system and inform the subscriber about important metaoccurrences such as deletion of a publication topic or disconnection of an event channel. The latter are created by the application at the server side via event factories, and published via topic publishers. Header fields are initialized upon publication. In particular, a publication ID, unique within the topic publisher, is assigned to every event. Published events become read-only and remain in read-only mode at the subscriber side; however, the event payload may be reused by cloning the event before publishing. Hierarchical Topic List. In REVENTS, topics are organized into a tree structure, allowing certain operations to be applied to subtrees rather than single nodes. Publishers have a scope attribute, which can be either SELF or SUBTREE. Whereas events generated by a SELF-scoped publisher events are delivered to subscriptions linked to that publisher’s topic only, SUBTREE-scoped publisher events are delivered to all subscriptions on the subtree. For instance, an event published on the root of the topic tree by a SUBTREEscoped publisher will be broadcasted to all subscriptions. By symmetry, a subscription is also associated with the scope attribute. A SELF-scoped subscription receives only the events published specifically on its topic. A SUBTREE-scoped subscription additionally receives events posted on subtopics of its main topic. For example, a SUBTREE subscription on the root node receives all posted events. As these examples illustrate, scoping enables and promotes natural grouping of event topics, allowing event broadcasting and monitoring within hierarchical sub-groups. Additionally, the API provides chroot-like mechanisms for projecting subtrees as if they were separate topic trees, enabling multiplexing of virtual topic trees through a single event channel. Priorities, Selector Filters, Expiration, and Overriding. Event publishers have associated priorities, assigned to them upon creation. Events from a single publisher, or from multiple publishers of the same priority, are always delivered to clients in their publication order. Events from publishers of different priorities, on the other hand, may be subject to en-route reordering, applied at server- and client-side queues. The exact ordering of such event streams is non-deterministic and may be non-uniform between subscribers, as it depends on timing of subscribers consuming the events versus the publishers producing them. It is guaranteed, however, that the only departures from the FIFO order stem from pulling higher-priority events ahead of lower-priority ones. Subscribers indicate their notification interests by creating subscriptions on selected topics. REVENTS allows the subscription scope to be refined via event selectors. The selectors are expressions (based on SQL expression syntax, borrowed from JMS [1]) applied at the publisher’s side and evaluated against the event’s header and metadata in order to decide if it is a match for the subscriber’s interest. Events pending delivery may have limited periods of validity. They may become outdated either upon a predefined timeout or when more recent events obsolete them.

REVENTS: Facilitating Event-Driven Distributed HPC Applications

295

Canonical example scenarios include progress- or status-tracking mechanisms, in which each event often carries complete status information. In these cases, pending events may be safely dropped when a new event is enqueued. REVENTS supports event overriding by allowing publishers to assign events with override selectors upon publication. The event’s override selector is evaluated when the event is inserted into a queue, and specifies criteria for events (originating from the same publisher) that should be dropped from the queue. Additionally to override selectors, events may be associated with an expiration timeout, which limits the total time the event is allowed to spend in queues.1 Late Subscription and Welcome Events. Events are often used for state change notification. In simple cases, each event fully and independently describes the complete state (e.g., absolute mouse pointer position, or percentage of completion of an activity). However, when the state is complex, the change event is often reduced to a mere notification, requiring the subscriber to issue an explicit query in order to obtain the updated state. This approach is appropriate in non-distributed event frameworks such as GUI subsystems. In distributed settings, in order to avoid frequent additional remote calls, it is common to embed complete state change information within each event. It allows the current state at any time to be fully determined by (and reconstructible from) an initial state followed by the sequence of state change events. As long as the changes are small and localized, this approach saves bandwidth while minimizing the number of RPC calls. In addition to reliable event delivery, however, this model also requires that the subscriber knows the initial state preceding the first change event it receives. Since subscriptions are created dynamically, a newly created subscriber may have missed some number of events, say n; therefore, the initial state from its point of view is the state between events n (which it missed) and n+1 (which it is going to receive). The REVENTS API supports generating and propagating such initial state events (called welcome events). When a new subscription is created, a request is issued to all relevant publishers asking them to generate a welcome event dedicated to the new subscription. The library guarantees that the welcome event is the first that the subscriber receives from a given publisher, and that the correct FIFO semantics are preserved for subsequently received events. Pluggable Transport Layer. In the grid computing arena, event notification technologies tend to focus on communication protocols and interoperability [7,8]. REVENTS exemplifies a complementary approach, concentrating on APIs and semantics of event subscription and delivery. In fact, REVENTS is protocol-agnostic. The APIs for the client and for the server are defined separately. The library isolates the transport layer from server- and client-side event handling. The transport layer is responsible merely for implementing channel functionality, which includes (1) transferring events from server-side queues to client-side queues and (2) propagating subscription-related requests from clients to servers. Thus, in principle, interoperability of REVENTS with third-party distributed event systems could be achieved by implementing an appropriate transport layer to couple them at the communication protocol level. 1

The expiration timeout does not account for the communication overhead, i.e., time needed to transfer events between server- and client-side queues, as the API has no means of measuring it.

296

D. Kurzyniec, V. Sunderam, and M. Sławi´nska

The default transport implementation in REVENTS is based on RMIX [11,12] – a flexible multiprotocol RMI library. Subscription-related requests are sent from clients to servers as RMI calls. To transfer events from servers to clients, the default implementation uses RMI-based blocking polling. The client repeatedly sends RMI requests to fetch all available events from the server-side queue and append them to the client-side queue. If the server-side queue is empty upon the invocation, the call blocks, waiting (with a timeout) for the next event to be published. Server-side blocking eliminates spin-waiting, while timeouts allow clients to distinguish a no-events condition from a silent network failure. If the client does not receive any response within triple the timeout duration, it assumes disconnection and signals it to the application. In case no events are generated for a prolonged period, the resulting timeouts can be viewed as heartbeats sent from the server and acknowledged by the client through subsequent poll requests. By using RMIX as the event transport layer, REVENTS can take advantage of its multi-protocol nature and configurable communication stacks, enabling configuration of the communication channel. Events may be encoded using various protocols such as SOAP, ONC RPC, or JRMP, and transmitted over a variety of transport layers, including SSL and JXTA. Moreover, when used in an RMIX-based distributed system (like H2O described in Section 4), REVENTS can leverage the existing communication infrastructure (including facilities for authentication, authorization, encryption, compression, or firewall tunneling) without introducing new semantics at the communication level.

4 Use Case: The H2O Distributed Platform H2O is a lightweight, componentoriented resource sharing framework. In H2O, resource providers execute lightweight Java-based containers, called kernels, providing a hosting environment for remotely deployable components, called pluglets. Similar to Java servlets, pluglets execute in the server context; however, they may be dynamically plugged in by external entities such as clients, the Fig. 2. H2O collaborative metacomputing model provider, or third-party resellers. Subsequently, the deployed pluglets can be utilized and assembled by users. Role separation between container providers, software deployers, and users distinguishes H2O among other component frameworks, enabling resource sharing. Providers can supply raw resources (CPU, storage) that are further reconfigured by external deployers or clients. As illustrated in Figure 2, providers share resources autonomously and independently. Aggregation is

REVENTS: Facilitating Event-Driven Distributed HPC Applications

297

assumed by specialized DVM-enabling pluglets that communicate in order to implement an abstraction of a Distributed Virtual Machine (DVM). Pluglets interact using two basic communication paradigms: RMI, including asynchronous and one-way RMI (RMIX), and distributed event notification (REVENTS). Kernel Events. The H2O API allows clients to register event listeners to monitor kernel, session and pluglet state and to track pluglet loading progress. The kernel can generate the following events: – NewSessionEvent: fired when a new session is created at the kernel (e.g. when a new remote user logs in), given that the client is authorized to see the new session. – SessionStateEvent: fired when the session state changes (e.g. when a remote user logs out). The event carries information about the old and the new state. – DeployEvent: fired when a new pluglet is deployed. – PlugletStateEvent: fired when the pluglet state changes. The event carries information about the old and new states. Two subclasses supply more information in specific cases: (1) PlugletLoadedEvent, fired after the main pluglet class is loaded into the kernel JVM, and (2) PlugletFailedEvent, fired when the pluglet fails to initialize or start due to an exception. – JarDownloadingEvent: fired during pluglet loading, when a new chunk of data has been read by the class loader. This event can be used to track the progress of pluglet class loading. (It is particularly useful for pluglets deployed from the network). H2O uses REVENTS to deliver kernel events to remote clients. The kernel uses a single topic tree for all event-handling purposes, exploiting its hierarchical structure. Kernel-related events (NewSessionEvent and DeployEvent) are scoped within the /system/kernel/ topic, while session- and pluglet-related events (SessionStateEvent, PlugletStateEvent, JarDownloadingEvent) are scoped within /system/session/{ID}/ and / system/pluglet/{ID}/ subtopics, respectively. Each H2O session is associated with a dedicated event channel through which the events are propagated. H2O relies on REVENTS transport facilities (Section 3), to monitor session liveness. From the kernel’s perspective, if the client does not poll for events for a predefined timeout, which is equivalent to the lack of a heartbeat acknowledgment, the client disconnection is concluded; the kernel then closes the session and releases all its resources. From the client’s perspective, if the REVENTS transport reports a broken channel (due to the lack of response from the server, equivalent to three missing heartbeats), the H2O library assumes disconnection, invalidates all the client API objects associated with it, and attempts a best-effort, asynchronous session close request to the server. Pluglet Events. Additionally to propagating kernel events, H2O enables pluglets to publish events of their own. Each pluglet is assigned a subtree within the main topic tree, anchored at /user/pluglets/{ID}/, serving as the application-level event scope. Pluglet events, along with kernel events, are tunneled through the client session’s event channel. From the API point of view, however, pluglets and their clients are provided with dedicated channel abstractions. A pluglet can obtain an event provider API object allowing it to create subtopics and publishers and to publish events, as well as the current session’s event channel server, allowing the pluglet to register event subscriptions in response to client’s RMI requests. At the client side, the API provides the application

298

D. Kurzyniec, V. Sunderam, and M. Sławi´nska

with the virtual event channel instance through which the subscribers can be managed. These virtual channels are mapped onto the underlying session channel through multiplexing mechanisms described in Section 3.

5 Performance Evaluation We have conducted a series of experiments in order to assess the performance of REVENTS. We focused on measuring throughput, in terms of number of events generated and delivered per second, under various load conditions. All experiments were conducted on a network of 32 Sun Blade 2500 machines, each with two 1.2 GHz CPUs and 2 GB RAM, interconnected via a Gigabit Ethernet switch. In the experimental setup, one of the machines was used as the event publisher, while a varying number of remaining machines were event subscribers. We ran four separate experiments, each for a different event type: (1) a basic text event, containing a single string, (2) a map event, containing 10 key-value pairs, (3) a medium size bytes event containing a 1 KB byte array, and (4) a large size bytes event containing a 1 MB byte array. Within each experiment, and for each count of subscribers, a series of 5-second event sequences were generated with increasing requested throughput, until we observed saturation indicating that further throughput increase was not possible. We report two quantities: the cumulative sustained throughput, representing the number of events per second that the system is able to deliver to subscribers, and the burst throughput, indicating how fast can the publisher generate events, even if they cannot be processed by subscribers at that rate. The justification for these two quantities is that while the former shows how much processing capacity the system has at a steady rate, the latter indicates the potential for generating short bursts of events that can be enqueued at a faster rate without causing the publisher to block. Both quantities represent the total number of events generated per the time interval multiplied by the number of subscribers, thus representing the count of events per second that have been actually pushed through all event queues at the server side. Given that representation, in an ideal case of a perfectly scalable system, the reported throughput would remain constant irrespective of the number of subscribers. The experimental results are shown in Figure 3. For single-string events (Fig. 3a), the maximum sustained throughput of about 45000 events/s is obtained for a subscriber count of 5; each subscriber observes a throughput of about 9000 events/s in this case. As the number of subscribers increases beyond 5, sustained throughput becomes limited by burst throughput, indicating CPU saturation at the server side. For map events (Fig. 3b), which are still relatively small yet somewhat more complex, the maximum sustained throughput of about 22000 events/s is achieved for a subscriber count in the 5-9 range, while the server-side CPU saturation can be observed for 7 and more subscribers. For 1 KB events (Fig. 3c), characterized by moderate size and structural simplicity, further flattening of the throughput curve and shift of the server-side CPU saturation point can be observed. The maximum sustained throughput of about 13000 events/s (equivalent to 13 MB/s of payload transmitted over the network) is achieved for a subscriber count between 2 and 17, while the server saturates at 13 subscribers. In all three tests, the sustained throughput at a load level of 30 subscribers is approximately equal to 8000

REVENTS: Facilitating Event-Driven Distributed HPC Applications

(a) Text event containing a single string

(b) Map event with 10 key-value pairs

(c) Bytes event, 1 KB payload

(d) Bytes event, 1 MB payload

299

Fig. 3. Maximum throughput measured for various event types, as a function of subscriber count. The throughput is normalized with respect to subscriber count (i.e. shows events that were actually enqueued rather than published), so that in the ideal case, a constant line would be expected.

events/s, corresponding to about 260 events/s perceived by each subscriber. Finally, for 1 MB events (Fig. 3d), the sustained throughput quickly reaches 25 events/s and remains in the 25-30 range, corresponding to 25-30 MB/s of payload transmitted over the network. The communication capacity is limited by the networking layer in this case; we were unable to measure the maximum burst throughput as it exceeded limits imposed by available memory. These results suggest that REVENTS can offer satisfactory performance on generalpurpose networks.2 Using moderately powerful hardware, it can achieve network utilization at the level of 7-30 MB/s, which significantly exceeds the capacities of typical wide-area networks for which it is primarily designed.

6 Conclusions and Future Work This paper describes REVENTS – an experimental distributed event notification middleware system. REVENTS derives many of its features from popular event, messaging, and group communication libraries, and enhances them with hierarchical topic 2

REVENTS does not aspire to be a replacement for MPI group communication primitives that offer greater scalability by exploiting static network topologies.

300

D. Kurzyniec, V. Sunderam, and M. Sławi´nska

lists, event publisher prioritization, and support for late subscriptions, providing flexible and comprehensive mechanisms to facilitate development of modern interactive distributed applications. By separating the transport layer from event handling, and focusing on the API and event subscription and delivery semantics, REVENTS can take advantage of popular communication protocols, enabling interoperability with thirdparty distributed event systems. REVENTS has been used to implement notification facilities in the H2O distributed component framework, both for internal infrastructure events and user-defined application-level events. The H2O distribution [13] includes further usability examples. The conducted experiments show that REVENTS achieves satisfactory performance in general-purpose network settings for which it is designed. Our current work concentrates on examining the usability of REVENTS within the context of adaptive applications and resource discovery in dynamically changing and collaborative grid environments.

References 1. Hapner, M., Sharma, R., Burridge, R., Fialli, J., Haase, K.: Java Message Service API tutorial and reference: messaging for the J2EE platform. Addison-Wesley Longman Publishing Co., Inc, Boston (2002) 2. Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design Patterns: Elements of Reusable Object-Oriented Software. Professional Computing Series. Addison-Wesley, Reading (1995) 3. Microsoft Corporation: .NET framework developer’s guide: Handling and raising events, http://msdn.microsoft.com/library/default.aspurl=/library/ cpguide/html/cpconEvents.asp 4. Ban, B.: JGroups – a toolkit for reliable multicast communication, http://www.jgroups.org/ 5. Goetz, B.: Java theory and practice: Be a good (event) listener. IBM developerWorks, http://www-128.ibm.com/developerworks/java/library/ j-jtp07265/index.html 6. Kurzyniec, D., Wrzosek, T., Drzewiecki, D., Sunderam, V.: Towards self-organizing distributed computing frameworks: The H2O approach. Parallel Processing Letters 13(2), 273– 290 (2003) 7. Huang, Y., Gannon, D.: A comparative study of Web Services-based event notification specifications. In: ICPPW 2006: Proceedings of the 2006 International Conference Workshops on Parallel Processing, Washington, DC, USA, pp. 7–14. IEEE Computer Society Press, Los Alamitos (2006), Available at http://doi.ieeecomputersociety.org/10.1109/ICPPW.2006.5 8. Huang, Y., Slominski, A., Herath, C., Gannon, D.: WS-Messenger: A Web Services-based messaging system for service-oriented grid computing. In: CCGRID 2006: Proceedings of the Sixth IEEE International Symposium on Cluster Computing and the Grid, Washington, DC, USA, pp. 166–173. IEEE Computer Society Press, Los Alamitos (2006), Available at: http://doi.ieeecomputersociety.org/10.1109/CCGRID.2006.109 9. van Hoof, J.: How EDA extends SOA and why it is important. Thoughts on Service Oriented Architecture and Event-Driven Architecture (blog) (November 2006), http://soa-eda.blogspot.com/2006/11/ how-eda-extends-soa-and-why-it-is.html 10. Eisenhauer, G., Bustamante, F.E., Schwan, K.: Event services in high performance systems. Cluster Computing: The Journal of Networks, Software Tools, and Applications 4(3), 243– 252 (2001)

REVENTS: Facilitating Event-Driven Distributed HPC Applications

301

11. Kurzyniec, D., Wrzosek, T., Sunderam, V., Slomi´nski, A.: RMIX: A multiprotocol RMI framework for java. In: Proc. of the International Parallel and Distributed Processing Symposium (IPDPS 2003), Nice, France, pp. 140–146. IEEE Computer Society, Los Alamitos (2003) 12. Wrzosek, T., Kurzyniec, D., Sunderam, V.S.: Performance and client heterogeneity in service-based metacomputing. In: Proc. of Heterogeneous Computing Workshop, in conjunction with IPDPS 2004, Santa Fe, New Mexico, USA, IEEE Computer Society, Los Alamitos (2004) 13. DCL: H2O Home Page http://www.mathcs.emory.edu/dcl/h2o/

Empowering Automatic Semantic Annotation in Grid ˇ Michal Laclav´ık, Marek Ciglan, Martin Seleng, and Ladislav Hluch´ y Institute of Informatics, Slovak Academy of Sciences, D´ ubravsk´ a cesta 9, 845 07 Bratislava, Slovakia [email protected] http://ikt.ui.sav.sk/

Abstract. Nowadays, capturing the knowledge in ontological structures is one of the primary focuses of the semantic web research. To exploit the knowledge from the vast quantity of existing unstructured texts available in natural languages in ontologies, tools for automatic semantic annotation (ASA) are heavily needed. In this paper, we present the ASA tool Ontea and empowering of the method by Grid technology for performance increase, which help us in delivering formalized semantic data in shorter time. We have adjusted Ontea annotation algorithm to be executable in the distributed grid environment. We also give performance evaluation of Ontea algorithm and experimental results from cluster and grid implementation. Keywords: semantic annotation, grid, Ontea.

1

Introduction

Automated annotation of the Web documents is a key challenge of the Semantic Web [1] effort. Web documents are structured but their structure is understandable only for humans, which is the major problem of the Semantic Web. In this paper, we present a tool for automatic semantic annotation named Ontea. We describe the methods implemented in the tool and present the results of experimental evaluation. Although Ontea’s results are promising, we face the problem of the performance, as the process of Ontea’s annotation is rather time consuming. This led us to the adoption of the computational grid technology, which allows us to reduce the delivery time of annotation results. The distributed version of Ontea is described in the paper and experimental evaluation of the time savings is presented. 1.1

Related Work

Annotation solutions can be divided into manual and semi-automatic methods. This different strategy depends on a use of the annotation. There is number of annotation tools and approaches such as CREAM [2] or Magpie [3] which follow the idea to provide users with useful visual tools for manual annotation, R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 302–311, 2008. c Springer-Verlag Berlin Heidelberg 2008 

Empowering Automatic Semantic Annotation in Grid

303

web page navigation, reading semantic tags and browsing [4] or provide infrastructure and protocols for manual stamping documents with semantic tags such as Annotea[5], Ruby[6] or RDF annotation[7]. Semi-automatic solutions focus on creating semantic metadata for further computer processing, using semantic data in knowledge management [8] or in information extraction application. Semi-automatic approaches are based on natural language processing [9] [10], a document structure analysis [11] or learning requiring training sets or supervision [12]. Moreover, other annotation approaches exist, e.g. KIM [14] which uses information extraction, or pattern-based semi-automatic solutions such as PANKOW and C-PANKOW [13], using Google API for automatic annotation. One of relevant automatic semantic annotation solution and the only one which runs on distributed architecture is SemTag [15]. It uses Seeker [15] information retrieval platform to support annotation tasks. SemTag annotates web pages using Stanford TAP ontology [16]. 1.2

Ontea

One of pattern based solutions is Ontea [17] [18] working on text, in particular domain described by domain ontology and using regular expression patterns for semi-automatic semantic annotation. Ontea detects or create ontology elements/individuals within the existing application/domain ontology model according to defined patterns. Several cross application patterns are defined but in order to achieve good results, new patterns need to be defined for each application. Main difference e.g. with SemTag is that solution can also create, not only detect ontology individuals within defined ontology model. This functionality makes Ontea algorithm slower than SemTag [15], but suitable for wider scope of applications where semantics of documents is needed. Ontea is similar to C-PANKOW [13] annotation with better annotation results and better performance. By the Ontea annotation engine we want to achieve the following objectives: – Detecting/Creating Meta data from Text – Preparing improved structured data for later computer processing – Structured data based on an application ontology model 1.3

Use of Information Retrieval Techniques

Ontea uses also RFTS (Rich full-text search) [19] - the tool for document indexing and document search. Ontea uses RFTS functionality when creating a new ontology individual for evaluating the relevance of the newly created instances. Ontea identifies part of text related to semantic context and match the subsequent sequence of characters to create an instance of the concept. Let us denote the sequence of words related to semantic context by C and word sequence identified as a candidate instance as I. We evaluate the relevance of the new instance by computing the ration of the close occurrence of C and I and occurrence of I in the whole collection of documents: close occurrence(C, I) / occurrence(I).

304

M. Laclav´ık et al.

The Ontea annotation method can be also used with Lucene [20] information retrieval library. When connected with RFTS indexing, Ontea asks for relevance based on words distance. When connecting with Lucene, Ontea asks for percentage of occurrence of matched regular expression pattern to detected element represented by word. Example can be Google, Inc. matched by pattern for company search: [\s]+([-A-Za-z0-9][ ]*[A-Za-z0-9]*),[ ]*Inc[\.\s]+, where relevance is computed as Google, Inc. occurrence divided by Google occurrence. RFTS indexing tool also supports this type of queries, however Lucene can achieve better performance. Use of Lucene or RFTS is related to case Ontea creation IR in evaluation.

2

Evaluation

In this chapter we describe evaluation of Ontea annotation success rate as well as performance evaluation on a single machine. Performance evaluation on the grid infrastructure is not yet available but will be at time of the conference. We also describe a test set of documents. 2.1

Test Set of Documents

As reference test data, we used 500 job offers downloaded from web using wrapper which prepared us some structured data. This was converted to a defined ontology, manually checked and edited according to 500 html documents representing reference job offers. Ontea processed reference html documents using the reference ontology resulting in new ontology metadata consisting of 500 job offers, which were automatically compared with reference, manually checked job offers ontology metadata. 2.2

Target Ontological Concepts for Identification

In this test, Ontea used simple regular expressions matching from 1 to 4 words starting with a capital letter. This experiment is referred to as Ontea in next chapter. In the second case we used domain specific regular expressions which identified locations and company names in text of job offers and Ontea also created individuals in knowledge base, while in the first case Ontea did not create extra new property individuals only searched for relevant individuals in knowledge base. This second case is referred to as Ontea creation. The third case used also previously described RFTS indexing tool or Lucene to find out if it is feasible to create a new individual using relevance techniques described earlier. This case is referred to as Ontea creation IR. To sum up, we conducted our experiments in 3 cases: – Ontea: searching relevant concepts in knowledge base (KB) according to generic patterns – Ontea creation: creating new individuals of concrete application specific objects found in text

Empowering Automatic Semantic Annotation in Grid

305

– Ontea creation IR: Similar as previous with the feedback of RFTS or Lucene to get relevance computed above word occurrence. Individuals were created only when relevance was above defined threshold which was set up to 10% We used following regular expressions: – Generic expression matching one or more words in text. This was used only to search concepts in KB ([A-Z][-A-Za-z0-9]+[\s]+ [-a-zA-Z]+) – Identifying geographical location in text and if not found in KB individual was created Location:[\s]*([A-Z][-a-zA-Z]+[ ]*[A-Za-z0-9]*) used for English [0-9]{3}[ ]*[0-9]{2}[ ]+([A-Z][ˆ\s,\.]+[ ]*[ˆ0-9\s,\.]*)[ ]*[0-9\n,]+ used for Slovak text where settlement name is usually next to ZIP code – Identifying company in the text, this was used also with other abbreviations such as ”Ltd” or ”a.s.”, ”s.r.o.” for the Slovak language [\s]+([-A-Za-z0-9][ ]*[A-Za-z0-9]*),[ ]*Inc[\.\s]+ for English [\s]+([A-Z][ˆ\s,\.]+[ ]*[ˆ\s,\.]*[ ]*[ˆ\s,\.]*)[, ]*s\.r\.o\.[\s]+ used for Slovak texts 2.3

Success Rate of the Ontea Algorithm

In this chapter we discuss the algorithm evaluation and success rate. To evaluate success of annotation, we used the standard recall, precision and F1 measures. Recall is defined as the ratio of correct positive predictions made by the system and the total number of positive examples. Precision is defined as the ratio of correct positive predictions made by the system and the total number of positive predictions made by the system. Recall and precision measures reflect the different aspects of annotation performance. Usually, if one of the two measures is increasing, the other will decrease. These measures were first used to measure IR (Information retrieval) system by Cleverdon [21]. To obtain a better measure to describe performance, we use the F1 measure (first introduced by van Rijsbergen [22]) which combines precision and recall measures, with equal importance, into a single parameter for optimization. F1 measure is weighted average of the precision and recall measures. 2.4

Experimental Results of Annotation Success Rate

Experimental results using precession, recall and F1 -measures are in a table bellow. In the table we compare our results with other semantic annotation approaches and we also list some advantages and disadvantages. The column relevance is in case of Ontea F1 -measures but in case of other methods it can be evaluated by other techniques and usually it is not common. For example for C-PANKOW, relevance is referred as recall. Rows relevant to our annotation approach are in grey color, where we show success rate of three evaluation cases mentioned in the previous chapter.

306

M. Laclav´ık et al. Table 1. Annotation experimental results Method regular expresions, search in knowledge base (KB)

Ontea

disambiguatiy check, SemTag searching in KB regular expresions (RE), Ontea creation creation of individuals in KB RE, creation of individuals in Ontea creation KB + RFTS or Lucene IR relevance document structure Wrapper PANKOW C-PANKOW Hahn et al. Evans Human

pattern matching POS taging and pattern matching Qtag library semantic and syntactic analysis clustering manual annotation

relevance precision recall % % % 71

64

high

high

41

28

62

53

high

high

59 74 76 41 high

high

Disadvantages

high recall, lower 83 precesion works only for TAP KB aplication specific 81 patterns are needed low precision some good results are 79 killed by relevance identification zero success with unknown structure low success rate suitable only for English, 74 slow algorithm works only for English not Slovak low success rate problem with creation of high individuals duplicities, inacuracy

Advantages high succes rate, generic solution, solved duplicity problem, fast algorithm fast and generic solution support Slovak language

disambiguities are found and not annotated, good results high success with known structure generic solution generic solution

high recall and precesion

The row Ontea creation IR case is the most important considering evaluation where we combined information retrieval (IR) and annotation techniques. By using this combination we could eliminate some not correctly annotated results. For example by using [Cc]ompany[:\s]*([A-Z][-A-Za-z0-9][ ]*[A-Za-z0-9]* regular expression in the second case we have created and identified companies such as This position or International company which were identified as not relevant in the third case with use of IR. Similarly Ontea creation identified also companies as Microsoft or Oracle which is correct and in combination with IR eliminated. This issue decreases recall and increases precession. Here it seems that IR case is not successful but opposite is true because in many texts Microsoft is identified as products e.g. Microsoft Office and if we take more text to annotate it is better not to annotate Microsoft as a company and decrease recall. If we would annotate Microsoft as a company in other texts, used in context of Microsoft Office we will decrease precision of annotation. It means it is very powerful to use presented annotation technique in combination with indexing in applications where precession need to be high. 2.5

Performance Evaluation on Single Machine

In this section we will evaluate performance of our solution on the same document set as described above. Annotation was executed on Centrino notebook 1.66 GHz, 1GB RAM, Linux OS. The algorithm is written in Java using Sesame [23] semantic engine. An average size of documents from test set was from 2000 to 4000 bytes. We will measure the average time of document annotating and duration of the single ontology query to knowledge base. This depends on the ontology and knowledge base size but in our case this can be omitted. In the future evaluation on a larger set of document we will need to take this into account, since query duration will increase by each individual added to the ontology.

Empowering Automatic Semantic Annotation in Grid

307

First we run the test on the empty ontology with only regular expressions which can create individuals (see Ontea creation case). If the individual returned by the regular expression exists in the ontology system, continue to the next regular expression result (query) but if the result do not exist in the ontology, a new individual will be created by the system. Stats after annotating of 500 documents by the system are in the following table Table 2. Performance in Ontea Creation case Total time duration

220 s

Average Total Average Total Average time queries executed individuals created duration per executed queries per created individuals document document per document 439 ms 1533 3 1241 2.482

Next test has been run on the results ontology from previous case. Now we search for ontology individuals within knowledge base and not creating them (see Ontea case) Table 3. Performance in Ontea case Total time duration

2399 s

Average Total Average Total Average time queries executed expressions found duration per executed queries per found expression document document per document 4797 ms 103570 207 4984 10

In the tables above we can see performance on document set on one machine. Both tables describe annotation tasks, which can be put to Grid environment. Annotation of one document takes approximately 5 seconds.

3

Putting Ontea into Grid

While Ontea yields interesting results, its annotation method is rather time consuming and annotating large number of documents or periodical re-annotating of updated document collection is highly impractical when performing the computation on a single server. The process of Ontea computation is easy parallelized and parallel threads can run independently with only two synchronization steps needed. This allows us to use distributed computing technology to speed up annotation process for large document collections. In this section, we describe the technique we use for Ontea annotation method distribution, exploiting existing grid computing infrastructure.

308

3.1

M. Laclav´ık et al.

Annotation Process Distribution

Ontea semantic annotation is performed in the following stages: 1. In the first stage, the instances of ontological concepts are created in the input text collection based on regular patterns matching. (case Ontea Creation) This stage will produce OWL ontology files which need to be integrated on a central machine. 2. After integration, instances created in the first stage are evaluated by computing their relevance using IR techniques. The instances with relevance value above given threshold are identified as relevant and filled in result domain ontology OWL file. (case Ontea creation IR) 3. Domain ontology OWL file is used in the third stage of the process for searching annotation tags within annotated text similarly to step one but using general keyword matching patterns. This results to executing more ontology queries and thus consuming more time. (case Ontea in evaluation) 4. Last stage integrated produced semantic metadata to one knowledge base represented by OWL file. The distribution of the computation is straightforward, it is sufficient to split the document collection and run the annotation process in a distributed manner, where different computation jobs process distinct subsets of the collection. The distribution is possible for the first and third stage of the annotation process, the second and forth stage is dedicated to integration of the jobs results and is performed on a single node. In the first step, ontology individuals are created. They are created with unique RDF ID based on the detected text and timestamp (e.g. http://...inst#region San Francisco 1179239158496). In case of distribution of annotation algorithm, same individuals can be created on different nodes. We had to change ID creation to be same at all nodes, due to final integration. Thus timestamp value was replaced by hash of detected text and ontology class URI. 3.2

Data Management Aspects

Following data have to be managed for running Ontea text annotation: – executables and libraries – document collection for text annotations – domain ontology OWL files In the first stage, the jobs for pattern matching are submitted to the grid. The subset of document collections must be replicated from central storage to grid sites where the jobs will be executed. After all of the first stage jobs are finished, the results are transferred to a central server, where the integration takes place. To reduce transfer of collection data in the infrastructure, it would be advantageous to submit the third phase jobs to the same grid sites where the first stage jobs were executed. However, limiting sites for third phase jobs only to

Empowering Automatic Semantic Annotation in Grid

309

those sites where document collection subsets are replicated might be impractical when those sites are heavily loaded and the jobs would be queued for a long time. We address this problem by using job management routine which submits the jobs only to given subset of grid sites, after defined timeout it cancels those that are still in scheduled state and resubmits the jobs without restriction to the grid sites. In addition, the OWL ontology files have to be replicated to the execution sites of first and third annotation stage jobs. 3.3

Evaluation Environment

We have developed a distributed version of Ontea using the grid infrastructure and the approach was evaluated in the Gilda testbed [26] training and test grid infrastructure of EGEE [27] project. The grid testbed used for running Ontea is powered by gLite middleware and gridified version of Ontea is tailored to this grid middleware. Worker nodes were Intel(R) Pentium(R) 4 CPU 2.40GHz PCs. As the sites in Gilda testbed are usually low loaded, we have not experience any wait times in the batch system queues for sub-jobs of Ontea computation. We have experienced an average 10 minutes overhead from the grid middleware (the overhead includes the procedure of submitting job via resource broker, scheduling job from resource broker to the execution site, job processing in local batch system, transfer and unpacking of the input data set, notification of resource broker about the state of processing and retrieval of the results from the worker node after the processing is done.) Table 4. Performance on the Grid Total time Average Total Grid Results Number of duration time time overhead integration documents on Grid duration per on single overhead document node 173 min 8.1 sec 678 min 10 min 25 min 5000

The evaluation was performed on the set of 5000 documents, split to five groups of approximately same size; five jobs were submitted to the grid infrastructure, each for one input data set. The same setting was used for 10 experiment runs. Average result delivery time for the whole input set was 173 minutes, with approximately 10 minutes grid middleware overhead and 25 minutes for results integration. The single machine processing of the whole input set took 678 minutes. The annotation of one document took approximately 8 seconds. This is longer than evaluation on a single machine due to a slower machine in testbed as well as changes in the code when allowing it for a grid. The utilization of the grid infrastructure for Ontea data processing brings us near linear speed-up of the results delivery, especially if used in real application where higher number of documents is annotated.

310

4

M. Laclav´ık et al.

Conclusion and Future Work

In this paper we discussed how automatic semantic annotation solution Ontea can benefit from Grid environment to achieve faster results of semantic annotation. The Ontea method is quite a powerful method [17] with acceptable success rate and performance suitable for knowledge management applications or applications in organizations where semantics of documents is needed [8]. If applied on bigger document bases, performance is not acceptable and need to be improved. To our best knowledge, none of annotation solutions yet benefit of Grid infrastructure and only SemTag [15] annotation used distributed architecture to perform annotation tasks. We have shown that semantic annotation can be gridified and achieve its results much faster. Semantic web research can be successful only when number of annotated web documents will reach critical mass, we believe this can happen only when annotation solution will benefit from parallel and distributed computing which is widely used in the information retrieval field. In our future work we would like to parallelize Ontea algorithm using MapReduce algorithm [24] and its open source implementation Hadoop [25]. Acknowledgments. This work is supported by projects NAZOU SPVV 1025/2004, VEGA 2/6103/6, VEGA 2/7098/27, EGEE-II EU 6FP RI-031688 and RAPORT APVT-51-024604.

References 1. Berners-Lee, T., Hendler, J., Lassila, O.: Semantic web. Scientific American 1, 68–88 (2000) 2. Handschuh, S., Staab, S.: Authoring and annotation of web pages in cream. In: WWW 2002: Proceedings of the 11th international conference on World Wide Web, pp. 462–473. ACM Press, New York (2002) 3. Domingue, J., Dzbor, M.: Magpie: supporting browsing and navigation on the semantic web. In: IUI 2004: Proceedings of the 9th international conference on Intelligent user interface, pp. 191–197. ACM Press, New York (2004) 4. Uren, V., Motta, E., Dzbor, M., Cimiano, P.: Browsing for information by highlighting automatically generated annotations: a user study and evaluation. In: K-CAP 2005: Proceedings of the 3rd international conference on Knowledge capture, pp. 75–82. ACM Press, New York (2005) 5. Annotea Project (2001), http://www.w3.org/2001/Annotea/ 6. W3C, Ruby Annotation (2001), http://www.w3.org/TR/ruby/ 7. The Institute for Learning and Research Technology, RDF Annotations (2001), http://ilrt.org/discovery/2001/04/annotations/ 8. Uren, V., Cimiano, P., Iria, J., Handschuh, S., Vargas-Vera, M., Motta, E., Ciravegna, F.: Semantic annotation for knowledge management: Requirements and a survey of the state of the art. Journal of Web Semantics: Science, Services and Agents on the World Wide Web 4(1), 14–28 (2005) 9. Madche, A., Staab, S.: Ontology learning for the semantic web. IEEE Intelligent Systems 16(2), 72–79 (2001)

Empowering Automatic Semantic Annotation in Grid

311

10. Charniak, E., Berland, M.: Finding parts in very large corpora. In: G´ omez-P´erez, A., Benjamins, V.R. (eds.) EKAW 2002. LNCS (LNAI), vol. 2473, pp. 358–372. Springer, Heidelberg (2002) 11. Glover, E., Tsioutsiouliklis, K., Lawrence, S., Pennock, D., Flake, G.: Using web structure for classifying and describing web pages. In: Proceedings of the Eleventh International Conference on World Wide Web, pp. 562–569. ACM Press, New York (2002) 12. Reeve, L., Han, H.: Survey of semantic annotation platforms. In: SAC 2005: Proceedings of the 2005 ACM symposium on Applied computing, pp. 1634–1638. ACM Press, New York (2005) 13. Cimiano, P., Ladwig, G., Staab, S.: Gimme the context: context-driven automatic semantic annotation with c-pankow. In: WWW 2005: Proceedings of the 14th international conference on World Wide Web, pp. 332–341. ACM Press, New York (2005) 14. Kiryakov, A., Popov, B., Terziev, I., Manov, D., Ognyanoff, D.: Semantic Annotation, Indexing, and Retrieval. Elsevier’s Journal of Web Semantics 2(1) (2005), http://www.ontotext.com/kim/semanticannotation.html 15. Dill, S., Eiron, N., et al.: A Case for Automated Large-Scale Semantic Annotation. Journal of Web Semantics (2003) 16. Guha, R., McCool, R.: Tap: Towards a web of data, http://tap.stanford.edu/ 17. Laclavik, M., Seleng, M., Gatial, E., Balogh, Z., Hluchy, L.: Ontology based Text Annotation OnTeA; Information Modelling and Knowledge Bases XVIII. In: Duzi, M., Jaakkola, H., Kiyoki, Y., Kangassalo, H. (eds.) Frontiers in Artificial Intelligence and Applications, vol. 154, pp. 311–315. IOS Press, Amsterdam (2007) 18. NAZOU Project Website (2006), http://nazou.fiit.stuba.sk/ 19. Ciglan, M.: Documents Content Indexing for Supporting Knowledge Acquisition Tools. In: Navrat, P., et al. (eds.) Tools for Acquisition, Organisation and Presenting of Information and Knowledge, pp. 49–63. Vydavatelstvo STU, Bratislava (2006) ISBN 80-227-2468-8 20. Hatcher, E., Gospodnetic, O.: Lucene in Action, Manning (2005) ISBN: 1932394281 21. Cleverdon, C.W., Mills, J., Keen, E.M.: Factors determining the performance of indexing systems, vol. 1-2. College of Aeronautics, Cranfield, U.K (1966) 22. Van Rijsbergen, C.J.: Information Retrieval. Butterworth-Heinemann, Newton, MA (1979) 23. OpenRDF.org, Sesame RDF Database (2006), http://www.openrdf.org/ 24. Dean, J., Ghemawat, S.: MapReduce: Simplified Data Processing on Large Clusters. Google, Inc. In: OSDI 2004, San Francisco, CA (2004) 25. Wiki, L.: HadoopMapReduce (2007), http://wiki.apache.org/lucene-hadoop/HadoopMapReduce 26. GILDA, Grid Infn Laboratory for Dissemination Activities (2007), https://gilda.ct.infn.it/ 27. EGEE, Enabling Grids for E-sciencE (2007), http://www.eu-egee.org/

Fault Tolerant Record Placement for Decentralized SDDS LH* Grzegorz L  ukawski and Krzysztof Sapiecha Department of Computer Science, Kielce University of Technology, Poland {g.lukawski,k.sapiecha}@tu.kielce.pl

Abstract. Scalable Distributed Data Structures (SDDS) is a file of records distributed to servers of a multicomputer. Faults of record placement in SDDS file may lead the file to crash. In this paper a fault model for decentralized SDDS LH* (DSDDS LH*) is given and, for such a model, a fault tolerant record placement architecture is presented. Client message forwarding is supplemented with client message backwarding. Split token (used in DSDDS LH* for controlling the file scalability) is supplemented with token tracing protocol and so called token level. These additional mechanisms allow for regeneration of DSDDS LH* file damaged by transient control faults.

1

Introduction

The cheapest way to High Performance Computing (HPC) is multicomputing [1]. For example, a multicomputer may be built from desktop or server PCs, connected through fast Ethernet and supervised by Linux-based operating system. Scalable Distributed Data Structures (SDDS) [2] consists of two components dynamically spread across a multicomputer: records belonging to a file and a mechanism controlling record placement in file space. Record placement mechanism is spread between SDDS servers and their clients. In a multicomputer there is a lot of CPUs, memories and another hardware and software components cooperating with each other. Hence, there is a lot of sources for operational faults that may disturb expansion of SDDS file. Faulttolerant record placement architectures for centralized SDDS LH* were presented and evaluated in [4,5]. In this paper fault-tolerant record placement architecture for decentralized SDDS LH* (DSDDS LH*) is introduced. In section 2 a brief outline of DSDDS LH* architecture is presented. Possible DSDDS LH* failures are analyzed in section 3. In section 4 fault tolerant record placement architecture for DSDDS LH* is presented. The paper ends with conclusions.

2

Decentralized Scalable Distributed Data Structures with LH*

The least SDDS component is a record. Each record is equipped with an unique key. Records with keys are stored in buckets. Each bucket’s capacity is limited. R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 312–320, 2008. c Springer-Verlag Berlin Heidelberg 2008 

Fault Tolerant Record Placement for Decentralized SDDS LH*

313

If a bucket’s load reaches some critical level, it performs a split. A new bucket is created then and a half of data from the splitting bucket is moved into a new one. A client is another SDDS file component. It is a front-end for accessing data stored in SDDS file and may be a part of an application. There may be one or more clients operating the file simultaneously. The client may be equipped with so called file image used for bucket addressing. Such file image not always reflects actual file state, so client may commit addressing error. Incorrectly addressed bucket forwards such message to the correct one, and sends Image Adjustment Message (IAM) to the client, updating his file image, so he will never commit the same addressing error again [2]. All the SDDS file components are connected through a network. Usually, one multicomputer node maintains single SDDS bucket or a client, but there may be more components maintained by single node. In extreme situation all SDDS buckets, clients and additional components may be run on single PC. For record addressing, modified Linear Hashing (LH*) [2] is used. For distributing records between buckets LH* uses hashing function, such as simple division modulo N: hi (C) = C mod 2i (1) where C is a record key and i is a file/bucket level. Because the number of buckets in DSDDS file must not be a power of 2, for bucket and record addressing two hashing functions with levels i and i + 1 are used. Value i is called file level, each bucket has its own bucket level denoted j (Fig. 1). After successful bucket split, its level j increases. LH* bucket splits must be performed in specific order. So called split token is used for split synchronization. Only bucket actually holding a token may create a new bucket instance and split. After successful split, bucket number m sends the split token to bucket number m : m = (m + 1) mod 2j

(2)

where: j denotes splitting bucket level. Fig. 1 shows an example of decentralized LH* file evolution. At the beginning there is only one bucket with number 0 (Fig. 1a), as the hashing function uses this bucket’s level (j = 0) for record addressing, it accepts all records. The lonely bucket overloads and because it possesses the split token, then split is initiated. About half of its capacity is moved to the new bucket number 1 (Fig. 1b), bucket 0 keeps the split token. After more successful insert operations, bucket 0 overloads again, bucket number 2 is initiated (Fig. 1c). If the number of buckets is a power of two (Fig. 1a, b, d), only one bucket level for all active buckets is applied, two consecutive levels are used otherwise (Fig. 1c).

3

Decentralized SDDS LH* Fault Model

As far as record placement is concerned faulty behaviors of both the client and the server (the bucket) were considered.

314

G. L  ukawski and K. Sapiecha

Fig. 1. Example DSDDS LH* file evolution (bucket capacity 4 records)

3.1

Client

The DSDDS file client does not pose any real problem for correct DSDDS file operation. If a bucket’s file image or addressing mechanism is damaged, the client may send his queries to invalid buckets. In such a situation, three scenarios may take place: – A query is send beyond the file space (to nodes where no buckets are stored at all). – A query is send to a bucket having a number too small. In this case the message will be properly forwarded to the correct bucket, after at most two additional send operations, and the client will receive Image Adjustment Message [2]. – A query is send to a bucket having a number too big. In such a situation the client’s query may not reach the correct bucket, as the basic SDDS LH* architecture is not aware of such possibility [2]. 3.2

Bucket

Damaged bucket obviously means loss of data, but the split token may be in danger, too. Bucket recovery algorithm defined in [3] for data fault-tolerant architectures could be applied in case of transient or permanent faults concerning one or more buckets. However, these architectures were developed for centralized LH* and no procedure concerning token recovery was given. Split token failures may cause what follows: – No split token in the whole SDDS file and the file evolution is stuck, as no bucket is able to perform a split. – Multiple split tokens in the file what may happen after failed bucket recovery (if the recovery procedure generated additional unnecessary token(s)) or

Fault Tolerant Record Placement for Decentralized SDDS LH*

315

directly after bucket or message damage. The SDDS file structure may be in real danger. Invalid splits may be performed leading to invalid file structure, addressing problems and/or data loss. – Wandering token if one or many buckets are damaged (e.g. invalid address tables) and the token is send between buckets with no respect to LH* rules. This means that invalid splits may take place like as for multiple tokens.

4

Fault Tolerant Record Placement for DSDDS LH*

For SDDS file analysis under faulty conditions, specialized software implemented fault injector called SDDSim was developed [6]. Simulation experiments proved high vulnerability of DSDDS LH* on operational faults [4,5]. Additional mechanisms for the client and the bucket (split token) are necessary for correct DSDDS LH* file operation. They concern so called backwarding, token tracing protocol and token level. 4.1

Backwarding

In case of original SDDS LH* an expansion of SDDS file causes that a message forwarding algorithm is necessary [2]. However, a failure could cause that a client query may be sent incorrectly not only to lower addressed bucked but to higher addressed one, too. Therefore, client message forwarding should be supplemented with client message backwarding. Such additional mechanism is contained in Algorithm 1 (where a is the number of destination bucket according to clients file image, and a is the number of actual bucket). The last line of the algorithm causes that in case of the message send incorrectly to a bucket having a number too big such query will reach the correct bucket at last. Algorithm 1. Client message forwarding and backwarding a ← hj (c); if a =  a then a ← hj−1 (c); if a < a and a < a then a ← a ; end if if a < a and a < a then { Backwarding phase } a ← a ; end if end if

4.2

Token Tracing Protocol

To ensure that the token will not be lost in case of bucket failure, the bucket actually holding the split token is periodically checked by its previous holder. Token tracing protocol allows to recover the split token in its correct place

316

G. L  ukawski and K. Sapiecha

without the danger of file evolution stuck [4]. Algorithm 2 is performed by bucket k, previously holding the split token1 . Bucket k should also run this algorithm after data fault recovery, and if actual token holder is bucket k + 1. Moreover, bucket k + 1 should return its current level jk+1 along with request reply message. Supplied with such information, bucket k may verify k + 1 state comparing bucket levels: – Bucket k + 1 holds the token and jk+1 = jk − 1, or jk+1 = jk (if k + 1 = 0) correct file state, k + 1 is valid token holder; – Bucket k + 1 has no token and jk+1 = jk , or jk+1 = jk + 1 (if k + 1 = 0) correct file state; k + 1 has performed a split and the token was send to the next bucket; – Bucket k + 1 has no token and jk+1 = jk − 1, or jk+1 = jk (if k + 1 = 0) split token is lost, another one should be generated and send to bucket k + 1; Algorithm 2. Token tracing protocol – bucket k (previous token holder) waits for predefined time-out; – k sends to bucket k + 1 query requesting information about current token state; if k gets positive answer then k + 1 is in good condition and still holds the token, restart; else if k gets negative answer then k + 1 no longer possesses the token as it was send to k + 2; k + 1 will run token tracing protocol with k + 2, end of algorithm. else if k gets no answer then { k + 1 is probably damaged, do one of the following steps: } – transient fault – send another token to k + 1, restart; – persistent fault – skip bucket k + 1, send a new token to k+2, restart and start guarding bucket k + 2; – persistent fault – recover damaged bucket with number k + 1, restart; end if

Besides the token loss, invalid file state with multiple split token may also be detected. If the bucket k + 1 being guarded holds the split token and jk+1 = jk , or jk+1 > jk (if k + 1 = 0), invalid extra token is detected. Moreover, if bucket k detecting extra token in k + 1 has its own split token, at least one extra invalid token circulates at the file. After such detection there is possibility to remove invalid token, but it is not recommended. Because in decentralized LH* file there is no single centralized component monitoring bucket state (just like the Split Coordinator in centralized LH* [2]), such action may be dangerous and may lead to further file damage. 1

This procedure may be applied if SDDS file consists of at least three active buckets. Otherwise previous and current token holder is the same bucket (Fig. 1a, b).

Fault Tolerant Record Placement for Decentralized SDDS LH*

4.3

317

Token Level

The message used for transferring the split token between buckets, may be supplemented with token level, an additional information concerning levels of splitting buckets. Token with level h, permits a bucket to split, but also defines the resulting bucket level j after successful split. Token level is increased every time the split token visits bucket number 0. In practice, the token level is equal to bucket level after successful split. A bucket may perform a split only if: – For bucket 0: its current level j is lower or equal than token level h. – For bucket  = 0: its current level j is lower than token level h. Fig. 2 shows an example of LH* file evolution, using split token supplemented with token level. Bucket 1 holds correct split token allowing this bucket to reach level 2 (Fig. 2a). After overload, bucket 1 splits and sends the split token to bucket number 0, where token level h is increased to 3 (Fig. 2b). Consecutive buckets split reaching level 3 according to token level h (Fig. 2c, d).

Fig. 2. DSDDS LH* file with token level

To make SDDS LH* file tolerant to token control faults, every bucket verifies the token status after receiving token message from another bucket (Algorithm 3). Multiple tokens, even valid, are ignored. A bucket accepts split token, only if acceptance rules shown above are fulfilled. Otherwise the token is instantly sent to the next bucket. If such invalid token is received by bucket number 0, its level is increased before sending to another bucket, what introduces unique possibility of file structure regeneration (Fig. 3). The split token itself becomes critical file component, but it may be easily made fault tolerant with one of well known error detecting and/or correcting codes.

318

G. L  ukawski and K. Sapiecha

Algorithm 3. Token verification {Split token with level h was received by bucket m with level j;} if m has no token then { Ignoring multiple tokens } if m = 0 then { Bucket number 0 } if h ≥ j then Accept token; else Send token with level h + 1 to the next bucket; end if else if m > 0 then { Bucket number > 0 } if h > j then Accept token; else Send token with level h to the next bucket; end if end if end if

Fig. 3. SDDS LH* file with token level, extra valid token and successful file regeneration

An example of LH* file evolution with extra valid token is shown on Fig. 3. Suppose that in correct LH* file (Fig. 3a), where token with level h = 2 was currently used, another valid token with level h = 3 was generated (Fig. 3b) as an effect of transient fault. In case of overload bucket 2 splits, because j2 = 2. A new bucket 6 is initialized and the token goes to bucket 0 (Fig. 3c). Because the extra token was verified as valid, bucket 0 also splits and a new bucket 4 is initialized (Fig. 3d). Bucket 1, already holding a valid token, drops extra token obtained from bucked 0.

Fault Tolerant Record Placement for Decentralized SDDS LH*

319

The extra token disappears but the file structure is still invalid. In case of bucket 1 overload, another split is performed and bucket 3 is initialized (Fig. 3e). The split token goes back to bucket 0 and this time it is verified as invalid. According to Algorithm 3 the level of such a token is increased and it is instantly send to the next bucket (Fig. 3f). Then, bucket 1 performs a split, bucket 5 is created and the file structure is valid again. However, the split token is in wrong bucket (Fig. 3g). Bucket 2 verifies the token as invalid and it is instantly send to bucket number 3. As a result, the token is verified valid at last and the file structure is successfully regenerated (Fig. 3h). Unfortunately, during file evolution shown on Fig. 3 many operations requested by clients were not finished and many messages were lost (due to addressing problems occurring while the file structure was invalid). Many different scenarios are possible, most of them should lead to correct file regeneration.

5

Example DSDDS LH* File Regeneration

Fig. 4 shows sample experimental results coming from SDDSim. Vertical axis shows how much of the network throughput is taken by messages of given type. In case of the number of buckets, vertical axis shows maximum of 128 bucket supported by SDDSim. Horizontal axis represents the progress (in time), expressed as the total number of messages sent through the simulated network. An additional valid token was injected into fault tolerant DSDDS LH* file (Fig. 4 – point A on the horizontal axis). Many insert operations were not finished then, and many lost messages (sent beyond the file space) were detected. The structure of the file was temporarily damaged. Next, thanks to the token tracing

Fig. 4. SDDS LH* file with token level, extra valid token and successful file regeneration

320

G. L  ukawski and K. Sapiecha

and token level mechanisms, the file was successfully regenerated and after some splits additional token was dropped (Fig. 4 – point B on the horizontal axis).

6

Conclusions

Simulation experiments proved high vulnerability of DSDDS LH* on operational faults [4,5]. To make DSDDS LH* tolerant to data faults the bucket recovery algorithm defined in [3] could be applied. Backwarding, token tracing protocol and token level were introduced here to tolerate variety of faults of LH* control. These extra architectural mechanisms were experimentally evaluated with the help of SDDSim software fault injector. The faults were tolerated. DSDDS LH* file structure was successfully regenerated after serious split token control faults. Token tracing protocol requires some additional circuitry (bucket timers) and additional messages to be send through a network. The number of extra messages strongly depends on specific implementation details (network connection speed, time-out chosen, etc.). In practice, efficiency decrease caused by additional messages used for token tracing is negligible. For token level checking no extra messages are needed at all. Token level value, even with additional redundant data, requires a few bytes at most. Expanding token transfer message with such extra data costs nothing, and benefit from using such extended split token may be priceless.

References 1. Dongarra, J., Sterling, T., Simon, H., Strohmaier, E.: High-Performance Computing: Clusters, Constellations, MPPs, and Future Directions. IEEE Computing in Science and Engineering (March 2005) 2. Litwin, W., Neimat, M.-A., Schneider, D.: LH*: A Scalable Distributed Data Structure. ACM Transactions on Database Systems ACM-TODS (December 1996) 3. Litwin, W., Neimat, M.-A.: High-Availability LH* Schemes with Mirroring. In: Intl. Conf. on Coope. Inf. Syst. COOPIS 1996, Brussels (1996) 4. Sapiecha, K., L  ukawski, G.: Fault-tolerant Control for Scalable Distributed Data Structures. Annales Universitatis Mariae Curie-Skodowska, Informatica (2005) 5. Sapiecha, K., L  ukawski, G.: Fault-tolerant Protocols for Scalable Distributed Data Structures. In: Wyrzykowski, R., Dongarra, J., Meyer, N., Wa´sniewski, J. (eds.) PPAM 2005. LNCS, vol. 3911, Springer, Heidelberg (2006) 6. L  ukawski, G., Sapiecha, K.: Software Functional Fault Injector for SDDS. GIEdition. In: ARCS 2006. Lecture Notes in Informatics (LNI) (2006)

Grid Services for HSM Systems Monitoring Darin Nikolow1 , Renata Slota1 , and Jacek Kitowski1,2 1

2

Institute of Computer Science, AGH-UST, al. Mickiewicza 30, 30-059, Krak´ ow, Poland Academic Computer Center CYFRONET-AGH, ul. Nawojki 11, 30-950 Krak´ ow, Poland {darin,rena,kito}@agh.edu.pl

Abstract. As HSM systems are commonly used in data intensive grids to economically store huge amounts of data a need for monitoring them arises. This paper describes an approach to create a common model for describing the state of an HSM system. An implementation of a grid service for HSM monitoring and test results are presented.

1

Introduction

Data intensive grid applications often require access to huge amount of data. Storage systems dealing with such amounts of data can either use expensive high performance disk arrays or use the less expensive tape storage managed by HSM software. HSM systems are used in the case when the most important is the ability to store huge amounts of data at low cost and when the access time to data is not a primary issue. The access time to the data stored on HSM systems can vary a lot depending on the localization of the data and the current state of the HSM system. For example if the data is cached to disk the access time will be short, but if the data is only on tape then the access time will be longer. For some applications and middleware functionalities it is desirable to have some apriori information about the access time of the data we are just going to access. Optimization of data access for replicated data sets is one of the examples of such functionality [1]. In this case the middleware layer functionality should be able to choose the most appropriate replica for downloading. In order to have the ability to estimate access time for HSM systems a monitoring service running along with HSM system is needed. The monitoring service should gather information about the state of the HSM system and provide it to requesters. There are many different HSM systems available having quite different architectures, APIs or command line interfaces. Some of the most popular HSM systems are: DiskXtender [6], CASTOR [7], HPSS [8], FSE [9], TSM [10]. In order to be more feasible the service should provide the monitoring data via a common well defined interface. The structure of these data should also be well defined and should be able to describe the state of any HSM system using a state model of general HSM system having all the HSM systems functionalities and peculiarities. R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 321–330, 2008. c Springer-Verlag Berlin Heidelberg 2008 

322

D. Nikolow, R. Slota, and J. Kitowski

Grid services are based on the OGSA standard [11] and can provide a method for interoperability between heterogeneous computer systems. The grid services can be implemented as statefull services using WSRF [12] and thus provide more flexibility than the traditional web services. For example the data describing a storage system can be kept in the internal resources of the grid service instead of digging it out from the storage system each time the service is called, thus allowing for lower system response times. This article describes the attempt to design and implement a set of grid services for efficient monitoring of storage systems with special respect to HSM systems. The rest of the paper is organized as follows: The following section presents the state of the art. Next section section presents architecture of monitoring and estimation system for storage systems. The forth section describes a proposition of monitoring model of general HSM system. An implementation of monitoring utility as a grid service is presented in the fifth section. The sixth section presents test results and the last section concludes the paper.

2

State of the Art

Replication is a common method used to increase the performance of data access or to increase the data availability or the level of protection [2,3]. Utilities for monitoring and access time estimation are needed in such replication systems to make proper decisions where to create new replica or which one to choose for data access. Data access time time estimation for tertiary storage devices has been studied in [4]. In our previous work a system for estimating access time for the DiskXtender HSM system has been proposed [5] and tested [18]. Our experience with data access cost estimation for the CASTOR HSM system has been presented in [17]. Shriver et. al. in [15] describes several performance analysis techniques for storage systems. The GGF (Global Grid Forum) Performance Working Group has defined the GMA (Grid Monitoring Architecture) intended to provide a minimal specification supporting infrastructure monitoring and allowing interoperability [13]. The Grid Resource Monitoring (GridRM) project [14] is aimed to provide a generic open-source resource monitoring architecture, designed specifically for the Grid. The DiskXtender HSM software provides an administration graphical interface which can monitor many aspects of the system interactively [6], but its not grid-ready and of course can support only one HSM system.

3

Architecture of Distributed Monitoring and Estimation System for Storage Systems

Our vision of a system for monitoring and data access time estimation of storage resources in grid environment is graphically presented in Fig. 1. The idea is that every storage node is being equipped with a Storage System monitoring grid

Grid Services for HSM Systems Monitoring

323

SN SN

HSM DiskXtender HSM CASTOR

SN

SSmon

Disk Array

SSmon SSmon SN Local filesystem

CN

Network

SSmon

SSest

SSest

2 3

4 1

Client

Replication service Legend: CN − Computing Node SN − Storage Node SSmon − Storage System monitoring service SSest − Storage System estimation service

Fig. 1. Architecture of distributed monitoring and estimation system for storage systems

service called SSmon. The implementation of this service is dependent on the underlying storage system, because different storage system and especially HSM systems have quite different graphical user interfaces (GUI), command line interfaces (CLI) or application programming interfaces (API). On the other hand the services for different storage systems provide the same interface for the other components of the grid installation. SSmon provides information about the state of the monitored storage system. The state is defined as a set of values of parameters concerning performance, capacity and behavior of the storage system. The storage system estimation service, SSest, can use the current state of the storage system provided by SSmon to estimate the access time of data residing on the given storage system. The implementation of the SSest grid service can differ depending on the adopted approach (statistical, rule-based, simulated). SSest provides information about the expected latency and transfer time of a given data access request. This information can be used by client to optimize the data access. One typical client will be the replication service responsible for choosing the best replica for downloading or choosing the best location for uploading a new replica. A typical information flow when requesting a data access estimation is depicted with dash lines (see Fig.1). The client calls one of the available SSest grid services giving the URI of file of interest. The estimation service then makes calls to the appropriate SSmon and gathers the necessary data to calculate the estimated access time. When the results are ready they are returned to the client.

324

4

D. Nikolow, R. Slota, and J. Kitowski

Monitoring Model of General HSM System

Generally HSM systems consist of tertiary storage hardware (typically tape libraries) attached to one or more servers on which HSM software is running. Additional disk array volumes are often attached to the server acting as a cache for the data kept on the tertiary storage. The HSM storage space is usually accessed via traditional filesystem access methods or via special API provided by the vendor. The requests are served according to the implemented queuing strategy which is generally FIFO. Table 1. Static parameters describing HSM system Parameter name HSMVendorString

Type Unit Description

String n/a A string identifying the HSM software vendor, version etc. LibVendorString String n/a A string identifying the automated media library - vendor, model, firmware version. DriveVendorString String n/a A string identifying the drive hardware vendor, model, firmware version. TotalCapacity float TB The total capacity of the HSM system - it can be the capacity in the license or the capacity of the tertiary storage hardware. MountDir String n/a The mount point of the HSM filesystem. nrOfLibraries int n/a Number of automated media libraries attached to the HSM systems. nrOfDrvInLib int n/a Number of drives in each library. nrOfSlotsInLib int n/a Number of slots in each library. TypeOfDrvInLib String n/a Type of drives in each library. avgMountTime float s Average time for the robot to move a medium from a slot to a drive. avgDismountTime float s Average time for the robot to move a medium from a drive to a slot. avgLoadTime float s Average time for the drive to bring the medium online. avgUnloadTime float s Average time for the drive to eject a medium. avgPosTime float s Average time to position a head. avgDiskCacheTransferRate float MB/s Average transfer rate for the disk cache. maxNatDrvTransferRate float MB/s Maximal native transfer rate of a drive. mediumNativeCapacity int GB The native capacity of a medium.

The proposed model consists of a set of parameters which can be used to describe the current state of a given HSM system. The model is general and should be suitable for describing any HSM system. The parameters describing the HSM model can be divided into two classes:

Grid Services for HSM Systems Monitoring

325

– static parameters which don’t change often, these are usually configuration parameters, – dynamic parameters changing frequently. The static parameters describing our model are listed in Table 1. These parameters should be set once when the service is started for the first time. They give a general view of the HSM system capabilities, like the overall capacity, disk cache capacity, the number of libraries and drives. Some performance characteristic can also be found here like the average mount and dismount times, average load and unload times and the average position time. Table 2. Dynamic parameters describing HSM system Parameter name FreeCapacity MaxTransferRate

Type

Unit Description

float float

GB Free storage space in the HSM system. MB/s Maximal transfer rate of HSM system for local file transfers - the data consumer process is run on the same machine where the process of the HSM system responsible for the transfer of files is running. This means that when the client is remote the actual transfer rate could be less or equal to this value depending on the network bandwidth. CurTransferRate float MB/s The last measured transfer rate for local transfers. HSMLoad int n/a The number of requests being served. HSMQueue aggregate n/a List of requests being served. Each item consists of the following attributes describing a request: tapeID, start block, end block. HSMDriveState aggregate n/a Describes the state of each drive: usage (empty, idle, read, write), block position, tapeID.

The dynamic parameters are listed in Table 2. These parameters describe the current state of HSM system and its actual performance characteristics. Some of the parameters are aggregates which actually means array of structures (using C jargon). Table 3. File and media related parameters describing HSM system Parameter Type Unit Description name HSMFileInfo aggregate n/a Describes the state of a given file (file name, isCached, tapeID, start block, end block, size). HSMTapeInfo aggregate n/a Describes the state of a given medium tapeID, mediaType, usage, lastMountTime, blockSize, capacity, capacityUsage, copyNumber).

326

D. Nikolow, R. Slota, and J. Kitowski

There are also dynamic parameters (not listed in Table 2) related to a given file or medium, which should be provided only on request because of their large total size. These parameters are kept in the internal data base of the HSM system. These parameters are listed in Table 3.

5

Implementation of HSM Monitoring Grid Service

As part of our research a grid service for HSM monitoring has been implemented. The grid service uses the model described in the previous section in order to supply (via well defined interface) information about the state of the HSM system being monitored. The service is implemented in the Java language and uses the WSRF technology provided with the Globus toolkit version 4. The class diagram of the service is presented in Fig. 2. In the current version the service provides the following methods: – getStorageSystemInfo() - this method provides general information about the storage system like type of storage, identification of vendor, model and version, storage capacity, performance characteristics, available data (or file) access methods. – getHSMFileInfo() - this method provides information about the requested file. This information includes the availability of the file (in cache or on removable medium), location of the file data in the medium (medium id, start block, end block). – getHSMState() - provides information about the state of the HSM system. The state is described by the queue of files being requested and the state of the removable media drives. The state of the drive is specified by the medium being in the drive and its current block position. – getLibraryInfo() - provides information about the automated media libraries attached to the HSM system. The information includes the inventory (which media belong to the library), number and type of drives, performance times of mount and dismount operations. – getHSMTapeMap() - provides information about the media like media type, block size used, capacity, time of last mount. A simple client with GUI has also been implemented to allow testing of the service.

6

Test Results

The HSM monitoring grid service has been tested on the following system configuration: – HP D class server with 2 processors, 640MB RAM with HPUX 11.11 – tape library ATL 7100 with DLT7000 tape drives – DiskXtender HSM software with 10GB disk cache Two kinds of tests are being performed: time performance tests and tests of influence on the HSM system performance. Below the procedure of performing these tests and the obtained results are presented.

Grid Services for HSM Systems Monitoring

327

Fig. 2. Class diagram of HSM monitoring grid service

6.1

Performance Tests

During these tests the time of executing the service methods is being measured. Each method is being called 10 times and the response wall-clock time is registered. For the getHSMFileInfo() method a random file is chosen from the files residing on the HSM system. The Table 4 presents the obtained data. The second column presents the initial response time measured when a method has been called for the first time after starting the service. The third column presents the average response time of the second and next calls. We can see that the initial call for the getStorageSystemInfo() method lasts about half minute but the next calls last less than a second. This is because of using service resources to keep the static information, which is gathered only once when the method is Table 4. Execution time performance test for DiskXtender Service method getStorageSystemInfo() getHSMState() getLibraryInfo() getHSMTapeMap() getHSMFileInfo()

Response time [s] Initial Average 30.80 0.85 0.79 0.60 1.44 1.05 6.15 2.54 2.93 1.08

328

D. Nikolow, R. Slota, and J. Kitowski

called for the first time. Similar effect can be observed for the getHSMTapeMap() and getHSMFileInfo() methods, but the speedup is not that impressive. 6.2

Influence Test

This test for DiskXtender is conducted as follows: 1. A random list of files being in the disk cache is created. The total size is about 500MB and the file sizes were from 5MB to 100MB. 2. The DiskXtender filesystem is unmounted and mounted again to flush the filesystem buffers. 3. The files from the list are read sequentially and the time of the whole operation registered. This time is further referred as tnogs 4. The DiskXtender filesystem is remounted again. 5. The HSM monitoring grid service is started. 6. A client generating a pattern of typical requests is started with time interval between requests being set to 5 seconds. 7. The files from the list are read sequentially and the time of the whole operation registered. This time is further referred as tgs The above procedure is repeated 10 times to obtain average values. Table 5. Service influence test for DiskXtender nr tnogs [s] tgs [s] 1 2 3 4 5 6 7 8 9 10 Avg

82,13 81,29 83,29 83,34 81,23 78,56 78,36 77,43 77,48 77,4 80,05

87,5 83,5 90,55 97,6 87,04 82,83 84,06 92,55 98,75 83,74 88,81

Overhead Transfer rate [MB/s] abs [s] rel [%] nogs gs 5,37 6,54 6,09 5,71 2,21 2,72 6,15 5,99 7,26 8,72 6,00 5,52 14,26 17,11 6,00 5,12 5,81 7,15 6,16 5,74 4,27 5,44 6,36 6,04 5,7 7,27 6,38 5,95 15,12 19,53 6,46 5,40 21,27 27,45 6,45 5,06 6,34 8,19 6,46 5,97 8,76 11,01 6,25 5,65

The results are presented in Table 5. We can see that the running service causes about 10% performance degradation of the DiskXtender HSM system.

7

Conclusions

In this paper a common model describing the state of a general HSM system has been proposed. The model is able to describe any HSM system. As part of our research an HSM monitoring grid service has been implemented. The test

Grid Services for HSM Systems Monitoring

329

results showed moderate overhead for rather heavy loading of the monitoring service. Our future work will be focused on adding support for the FSE [9] and TSM [10] HSM systems. Performance tests using modern computer architectures and Linux operating system will be conducted in order to study the influence of different system components on the performance. We also plan to make certain modification to our implementation in order to make it compliant with GMA.

Acknowledgments The work described in this paper was supported by AGH-UST grant. Thanks ˙ also go to the implementation team: Daniel Zmuda, Dariusz Antolak, Krzysztof Trubalski, L  ukasz Wiktor and Wojciech W¸atroba.

References 1. Slota, R., Nikolow, D., Skital, L., Kitowski, J.: Optimizing of Data Access Using Replication Technique. In: Bubak, M., Turala, M., Wiatr, K. (eds.) Proceedings of Cracow Grid Workshop - CGW 2004, Krak, ACC-Cyfronet UST, December 13-15 2004, pp. 215–220 (2004) 2. Kunszt, P., Laure, E., Stockinger, H., Stockinger, K.: File-based replica management. Future Generation Computer Systems 21, 115–123 (2005) 3. Lamehamedi, H., Szymanski, B., Deelman, E.: Data Replication Strategies in Grid Environments. In: Proceedings of the 5th International Conference on Algorithms and Architectures for Parallel Processing, ICA3PP 2002, Beijing, China, October 2002, pp. 378–383. IEEE Computer Science Press, Los Alamitos (2002) 4. Sandst˚ a, O., Midstraum, R.: Low-Cost Access Time Model for Serpentine Tape Drives. In: Proc. of 16th IEEE Symp. on Mass Storage Systems, the 7th NASA Goddard Conf. on Mass Storage Systems and Technologies, San Diego, California, USA, March 1999, pp. 116–127 (1999) 5. Nikolow, D., Slota, R., Kitowski, J.: Data Access Time Estimation for HSM Systems in Grid Environment. In: Proc. of the Cracow Grid Workshop, Cracow, Poland, December 11-14, pp. 209–216. ACK Cyfronet-AGH, Cracow (2002) 6. Legato Systems, Inc. - DiskXtender Unix/Linux, http://www.legato.com/products/diskxtender/diskxtenderunix.cfm 7. The CASTOR Project, http://castor.web.cern.ch/castor 8. HPSS - High Performance Storage System, http://www.hpss-collaboration.org/ 9. HP StorageWorks File System Extender Software - Overview & Features, http://h18006.www1.hp.com/products/storageworks/fse/index.html 10. IBM Tivoli Storage Manager, http://www-306.ibm.com/software/tivoli/products/storage-mgr/ 11. Towards Open Grid Services Architecture, http://www.globus.org/ogsa/ 12. Globus: WSRF - The WS-Resource Framework, http://www.globus.org/wsrf/ 13. Grid Monitoring Architecture, http://www-didc.lbl.gov/GGF-PERF/GMA-WG/ 14. GridRM: Resource monitoring for the Grid, http://gridrm.org/ 15. Shriver, E., Hillyer, B.K., Silberschatz, A.: Performance Analysis of Storage Systems. In: Reiser, M., Haring, G., Lindemann, C. (eds.) Dagstuhl Seminar 1997. LNCS, vol. 1769, pp. 33–51. Springer, Heidelberg (2000)

330

D. Nikolow, R. Slota, and J. Kitowski

16. Slota, R., Skital, L., Nikolow, D., Kitowski, J.: Algorithms for Automatic Data Replication in Grid Environment. In: Wyrzykowski, R., Dongarra, J., Meyer, N., Wa´sniewski, J. (eds.) PPAM 2005. LNCS, vol. 3911, pp. 707–714. Springer, Heidelberg (2006) 17. Kuta, M., Nikolow, D., Slota, R., Kitowski, J.: Data Access Time Estimation for the CASTOR HSM System. In: Wyrzykowski, R., Dongarra, J., Meyer, N., Wa´sniewski, J. (eds.) PPAM 2005. LNCS, vol. 3911, pp. 148–155. Springer, Heidelberg (2006) 18. Nikolow, D., Slota, R., Kitowski, J.: Gray Box Based Data Access Time Estimation for Tertiary Storage in Grid Environment. In: Wyrzykowski, R., Dongarra, J., Paprzycki, M., Wa´sniewski, J. (eds.) PPAM 2004. LNCS, vol. 3019, pp. 182–188. Springer, Heidelberg (2004)

The Vine Toolkit: A Java Framework for Developing Grid Applications Michael Russell, Piotr Dziubecki, Piotr Grabowski, Michal Krysin´ski, Tomasz Kuczy´ nski, Dawid Szjenfeld, Dominik Tarnawczyk, Gosia Wolniewicz, and Jaroslaw Nabrzyski Pozna´ n Supercomputing and Networking Center, Pozna´ n, Poland {russell,piotr.dziubecki,piotrg,mich,docentt, dejw,dominikt,gosiaw,naber}@man.poznan.pl http://www.man.poznan.pl

Abstract. The Vine Toolkit is a modular, extensible Java library that offers developers an easy-to-use, high-level API for Grid-enabling applications. It supports a wide-variety of application environments and Grid middleware platforms. Keywords: Java, Grid, framework, application, web, portal, service.

1

Introduction

The Grid [1] landscape has changed quite a bit since the early days of the Globus Toolkit [2]. Where the Globus Toolkit was the de facto Grid middleware platform on the open-source market, and remains so in the United States, in Europe and elsewhere there has been a proliferation of Grid platforms and middleware technologies in the last 7 years. While UNICORE [3] has been around for nearly as long as the Globus Project, EGEE’s gLite [4] platform, GRIA [5], CROWN [6] and other platforms have now acquired substantial user bases. Moreover, many auxiliary services are available today that address particular problem areas, such as GridSAM [7] for job submission, OGSA-DAI [8] for federated heterogeneous database access, the Storage Resource Broker (SRB) [9] for managing files logically across distributed filesystems and the Virtual Organization Management Service (VOMS) [10] for managing VOs. Some of these auxiliary services are designed to build upon or integrate with particular platforms, others may be designed to be platform agnostic but support only a limited set of middleware for production use. Deciding on which technologies to adopt is not trivial. Considerable effort is needed to install, configure, maintain and support Grid infrastructures, thus care must be taken to choose a platform that will meet an organization’s needs for years to come. Even when the motivation to select a platform is pre-determined, such as when there is a need to interoperate with other collaborating institutes R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 331–340, 2008. c Springer-Verlag Berlin Heidelberg 2008 

332

M. Russell et al.

that have existing infrastructures, building applications and services on top of just one platform is usually not trivial either. There are several risks to consider when doing so. – Portability: Writing to one platform may negate the use of another sometime in the future. The growing support for industry-wide standards that expose relevant features to applications may alleviate problems one day, but that day has not yet arrived. – Maintenance: The rate at which a particular platform evolves may result in hundreds hours of work spent on maintaining software. This is more so for applications that are meant to be used by the general community and thus need to be supported for multiple versions of particular middleware services. – Functionality: Not all Grid platforms are created equal. All have their strengths and weaknesses. It may turn out during the course of implementing a solution one finds the desired functionality does not yet exist, isn’t fully supported, or requires many “work-arounds” to achieve at the application level. 1.1

Standards and Their Importance to Applications

Today, most Grid platforms are Web-service based. In practice, this essentially means there now exists a standard way for issuing calls to services. This is not enough for true interoperability between infrastructures and therefore does not solve the application portability problem. Semantic Grid [11] approaches, or some other methodology involving meta-data to describe resources, may offer a means to dynamically discover how to use a particular service for a particular purpose. However, even here, higher-level standards will be needed for describing types of services. The authors of the Vine Toolkit can testify that the need for standards that encapsulate the major areas of functionality addressed by most Grid platforms is real. Though the Vine Toolkit, as will be discussed later, hides the problem of developing applications for different Grid middleware behind an application programmer interface (API), this has only put stress on the Vine Project, and other software projects that provide similar solutions, to maintain support for all those middleware. For example, consider the task of maintaining job submission support for 4 different evolving middleware platforms, as the Vine Project has done, as opposed to having to support job submission to only one specification. Even if that specification is changing over time, the workload is considerably reduced (and more so if user authentication and data staging are likewise standardized!). The Open Grid Forum (OGF) adopted Job Description Specification Language (JSDL) 1.0 [13] and the evolving Basic Execution System (BES) [14] specifications, though still somewhat immature, are steps in the right direction. Another approach to standardization lead within OGF is the Simple API for Grid Applications (SAGA) [15]. The SAGA Core Working Group (SAGACORE-WG) [16] has produced a nearly completed language-independent specification that describes what many application experts and Grid experts perceive

The Vine Toolkit: A Java Framework for Developing Grid Applications

333

to be the base and most relevant capabilities of Grid platforms to expose to application programmers. The main idea is to solve the portability problem at the API level as opposed to the infrastructure level. The SAGA specification outlines programming interfaces, entities and basic behavior that should be implemented for any programming language“binding” written for the specification. It also outlines how support for third party libraries and services should be plugged into a SAGA implementation. Some members of the SAGA effort are actively involved in developing language bindings and reference implementations of those bindings. Nearly completed SAGA-C++ and SAGA-Java binding and implementation efforts already exist with support for various Grid platforms. 1.2

Our Position

The Vine Project strongly agrees with the SAGA approach but also stresses the need for middleware-level standards for reasons cited above and feels these two approaches are, in fact, complimentary. The Vine Project intends to work with SAGA peers to implement the evolving Java SAGA 1.0, by “wrapping” the Vine Toolkit, and will encourage application programmers to consider using SAGA for basic Grid programing needs instead of the Vine API directly. However, the authors of the Vine Toolkit contend the Vine API, as it is applied to the Java programming language and target applications for Java, is more robust and extensible. Additionally, Vine is more than just a Grid API, it provides several areas of functionality that facilitate proper integration with web applications and services.

2

The Vine Toolkit

The Vine Toolkit, like the upcoming SAGA Java reference implementation or the Intel Grid Programing Environment [20], is a Java-based framework that offers developers an easy-to-use, high-level API for Grid-enabling applications. Vine began as an evolution of the GridSphere GridPortlets Project [22]. GridSphere [21] is a Portlet 1.0 [23] compliant portal framework for building portal websites and hosting collections of portlet web applications. GridPortlets was designed to provide a framework for using the Grid in portlet applications. Like the evolving Java SAGA 1.0 specification, GridPortlets is supported by a “pluggable” architecture for implementing its API to support different middleware. While its API is generally perceived to be very useful, GridPortlets is also recognized for several shortcomings. In practice, GridPortlets has only ever been distributed with support for Globus Toolkit 2 and 4 because extending it to support other middleware is overly complicated. Moreover, it is dependent on elements of the GridSphere Portal Framework and needs to be managed at runtime by a Portlet 1.0 API compliant container. Therefore, GridPortlets cannot be used or even tested outside of a web application server. Vine was designed from the outset to address these issues and more. Now in its alpha stages and available for public use, Vine includes an API and model

334

M. Russell et al.

of the Grid that greatly improves upon the original GridPortlets API while retaining its best features. More importantly, Vine is a general library that can be deployed for use in desktop, Java Web Start [24], Java Servlet 2.3 [25] and Java Portlet 1.0 environments with very little effort on behalf of the application programmer. This is because Vine provides an easy-to-use, extensible build system for deploying, configuring and bundling Vine for use with many different types of applications. However, taking into account the explicit need of web application developers, Vine is designed to provide all the necessary entry points for “attaching” Vine to web portals, web services and other “container” environments. Several of these entry points are discussed below. – User account management: Vine provides the ability to plug into the user account management mechanisms of its container environment, as well as to manage user accounts in standalone applications and services. – User registration with third-party libraries and services: Vine simplifies the process of generating credentials, registering those credentials with Grid middleware or creating accounts on remote systems with provider defined registration modules. This process can be attached to Vine’s account management system. – Single sign-on: Vine defines its own extensible authentication module system that can also be plugged into any container that supports extensions to its sign-on mechanisms, say during user login to a portal. – Authorization: The resources Vine makes accessible to end-users is completely configurable. Authorization in Vine can be linked to its container as well as to third-party authorization services. – User session management: In the Vine Toolkit, all user activity is managed within sessions. Vine sessions can naturally be attached to environments that user support session management. – Per-request behavior: Vine defines what is called a service context. Service contexts have a well-defined lifecycle. When applied to web applications, service contexts can be attached to HTTP requests, for example. When a request has completed, its associated service context will be destroyed, therefore cleaning up resources. This is useful in the context of connections to remote services and databases. – Startup and shutdown behavior: When Vine is first invoked by its container, it can execute one or more startup services to, say, begin polling remote information services or monitor remote activity. Similarly, one can define shutdown services for use with Vine. – Multi-threaded behavior: Vine inherently supports multi-thread applications and contains built-in mechanisms for performing tasks in separately managed threads. – Persistence: No extra work is required by the application programmer to manage persistence of information registered with Vine or tasks performed with it.

The Vine Toolkit: A Java Framework for Developing Grid Applications

2.1

335

Vine Deliverables

Vine is not developed in a vacuum, but rather to meet the specific needs of the projects that fund it, in addition to the needs of the community-at-large. Begun in the HPC-Europa Project [17] at the Pozna´ n Supercomputing and Networking Center, Vine is currently funded by the OMII-Europe Project [18] and BEinGRID [19]. Both of these projects are driven by the need for more common components on the market that promote interoperability and portability across different Grid middleware. OMII-Europe places particular emphasis on interoperability through Web and Grid standards adopted by the OGF and elsewhere. In fact, OMII-Europe and its partner institutes lead many of the OGF standards efforts currently underway. Moreover, OMII-Europe sponsors re-engineering activities in the US, Europa and China to enable today’s leading Grid platforms to support relevant standards. BEinGRID places particular emphasis on the need for common components that meet the needs of industry as well as support the middleware and services that have established user communities, including but not limited to: – – – – – – –

gLite Globus Toolkit GRIA OGSA-DAI SRB UNICORE VOMS

The Vine Project actively supports the technologies cited above to “Gridenable” web applications through generic, reusable Web 2.0 [26] user interfaces for use with GridSphere and other Portlet 1.0 compliant containers.

3

Vine Architecture

The Vine Toolkit consists of a core project that defines a base API and programming model upon which sub projects are built. Each sub project addresses a particular problem area (Fig. 2). Some, like the Grid Vine, build upon core Vine to define more general concepts and extensible elements. Others, like the Globus Toolkit 4 Vine, are concerned with adding support for particular third party libraries and services . Each project conforms to a particular file structure that defines how source code is built as well as how third party libraries and configuration files are packaged and deployed. Users can select the specific Vine projects they require for their applications. Naturally, there are dependencies between certain projects that must also be taken into account (Fig. 3). When Vine is deployed, Vine’s build system will deploy and package only those source files that are relevant to the target environment. Source files that are included in the main source tree for each project are deployed to all types of target environments, while source files contained in a project’s web source tree

336

M. Russell et al.

Fig. 1. Vine can be deployed as one more portlet web applications to a servlet container, such as Apache Tomcat, for use with the GridSphere Portal Framework

are deployed for use only with web applications (Fig. 4). Typically, web source trees include user interfaces developed in one or more web UI frameworks, such as the Google Web Toolkit [27] or Adobe Flex [28] , as well as Java servlets, portlets and any web services that are intended for use by that project. Project

Purpose

Grid Vine

Provides Grid API and base classes

Group Vine

Provides Groupware API and base classes

gLite 3 Vine

Adds gLite 3 client support to the Vine Toolkit

GT 4 Vine

Adds GT4 client support to the Vine Toolkit

UNICORE 6 Vine

Adds UNICORE 6 client support to the Vine Toolkit

VOMS Vine

Adds VOMS security support to the Vine Toolkit

BES Vine

Adds BES job submission support to the Vine Toolkit

SRB Vine

Adds SRB file management support to the Vine Toolkit

RUS Vine

Adds RUS information gathering support to the Vine Toolkit

Fig. 2. Some of the projects that are actively supported in the Vine Toolkit

3.1

Key Concepts

In addition to the base session management, persistence management and other mechanisms described above, the Root Vine defines several key concepts upon which all other Vine projects are built.

The Vine Toolkit: A Java Framework for Developing Grid Applications

337

Fig. 3. Dependencies between Vine projects

Fig. 4. How files are organized in the Vine Toolkit

Resources are perhaps the most important concepts modeled in the Vine Toolkit. Vine defines a resource as anything that can be utilized. A computer, an application, a software library, a person, these are all resources in Vine. In fact, in Vine, resources define the application just as they define grids. At their most basic level, grids are collections of resources with policies describing how to use those resources. Vine extends the resource concept to include tasks. 3.2

Advanced Concepts

More advanced interfaces and classes are provided in the Group Vine and the Grid Vine (Fig. 7). Group Vine defines the user account management and registration concepts described earlier. Grid Vine defines concepts specifically related to supporting grid application development. In addition to modeling basic concepts such as computers, software and services, Grid Vine models information services, file management services and job management services. Projects that extend Grid Vine implement part or all of the Grid Vine API. The Storage Resource Broker Vine, for example, implements the file aspects of the Grid Vine API.

338

M. Russell et al.

Fig. 5. Core interfaces defined in Vine

Fig. 6. Main task interfaces in the Vine Toolkit

Fig. 7. Group Vine and Grid Vine add base functionality to the Vine Toolkit

The Vine Toolkit: A Java Framework for Developing Grid Applications

4

339

Next Steps

Vine 1.0 Alpha has recently been released and been put in the hands of early adopters. Plans are underway for a second release before the end of 2007. The current focus is on solidifying the information gathering features of the Vine Toolkit as well as adding new Web 2.0 user interfaces to the web source trees of projects. The long term future for Vine is wide-open. In addition to adding support for more Grid middleware and platforms, Vine can be used to build higher-level web services, applied to P2P applications and much more. The only limits are those set by the projects that fund its development. Acknowledgments. The authors would like to extend our sincerest gratitude to Roman Wyrzykowski and of course to the Pozna´ n Supercomputing and Networking Center. We would also like to thank the OMII-Europe Project, BEinGRID, [29], HPC-Europa and other European Union funded organizations that have made this work possible.

References 1. Foster, I.: What is the Grid? A Three Point Checklist. GRIDToday, July 20 (2002) 2. Foster, I.: Globus Toolkit Version 4: Software for Service-Oriented Systems. In: Jin, H., Reed, D., Jiang, W. (eds.) NPC 2005. LNCS, vol. 3779, pp. 2–13. Springer, Heidelberg (2005) 3. Erwin, D.W., Snelling, D.F.: UNICORE: A Grid Computing Environment. In: Sakellariou, R., Keane, J.A., Gurd, J.R., Freeman, L. (eds.) Euro-Par 2001. LNCS, vol. 2150, Springer, Heidelberg (2001) 4. Laure, E., et al.: Programming the Grid with gLite. Computational Methods In Science And Technology 12(1), 33–45 (2006) 5. Surridge, M., Taylor, S., De Roure, D., Zaluska, E.: Experiences with GRIA Industrial Applications on a Web Services Grid. In: First Int. Conf. e-Science and Grid Computing Proceedings (2005) 6. Sun, H., Zhu, Y., Hu, J.H., Liu, Y., Li, J.: Early Experience of Remote and Hot Service Deployment with Trustworthiness in CROWN Grid. In: Cao, J., Nejdl, W., Xu, M. (eds.) APPT 2005. LNCS, vol. 3756, Springer, Heidelberg (2005) 7. Lee, W., McGough, A.S., Darlington, J.: Performance Evaluation of the GridSAM Job Submission and Monitoring System. In: UK e-Science All Hands Meeting 2005, Proceedings (2005) 8. Antonioletti, M., et al.: The design and implementation of Grid database services in OGSA-DAI. Concurrency and Computation: Practice and Experience 17(2-4), 357–376 (2005) 9. Rajasekar, A., et al.: Storage Resource Broker - Managing Distributed Data in a Grid. Computer Society of India Journal, Special Issue on SAN (2003) 10. Alfieri, R., et al.: From gridmap-file to VOMS: managing authorization in a Grid environment. Future Generation Computer Systems 21(4), 549–558 (2005) 11. De Roure, D., Jennings, N.R., Shadbolt, N.R.: The Semantic Grid: Past, Present, and Future. Proceedings of the IEEE 93(3), 669–681 (2005) 12. Open Grid Forum, http://www.ogf.org

340

M. Russell et al.

13. Job Submission Description Language (JSDL) Specification, Version 1.0. Global Grid Forum, GFD.56 (November 2005), http://www.ogf.org/documents/GFD.56.pdf 14. OGSA Basic Execution Service 1.0. Open Grid Forum, GFD.108 (July 2006), http://forge.gridforum.org/projects/ogsa-bes-wg 15. Goodale, T., et al.: SAGA: A Simple API for Grid Applications. High-level application programming on the Grid. Computational Methods in Science and Technology 12(1), 7–20 (2006) 16. Simple API for Grid Applications Core Working Group (SAGA-CORE-WG), http://www.ogf.org 17. HPC-Europa Project, http://www.hpc-europa.org 18. OMII-Europe Project, http://www.omii-europe.org 19. BEinGRID Project: http://www.beingrid.eu 20. Intel Grid Programming Environment, An Overview, http://gpe4gtk.sourceforge.net/GPE-Whitepaper.pdf 21. Novotny, J., Russell, M., Wehrens, O.: GridSphere: a portal framework for building collaborations. Concurrency and Computation: Practice and Experience 16 (5) 503, 503–513 (2004) 22. Russell, M., Novotny, J., Wehrens, O.: The Grid Portlets Web Application: A Grid Portal Framework. In: Wyrzykowski, R., Dongarra, J., Meyer, N., Wa´sniewski, J. (eds.) PPAM 2005. LNCS, vol. 3911, pp. 691–698. Springer, Heidelberg (2006) 23. Java Portlet 1.0 API, http://jcp.org/aboutJava/communityprocess/final/jsr168/ 24. Java Web Start, http://java.sun.com/products/javawebstart/ 25. Java Servlet 2.3 API: http://jcp.org/en/jsr/detail?id=53 26. Miller, P.: Web 2.0: Building the New Library, http://www.ariadne.ac.uk/issue45/miller/ 27. Google Web Toolkit, http://code.google.com/webtoolkit/ 28. Adobe Flex: http://www.adobe.com/products/flex/ 29. Gridipedia, http://www.gridipedia.eu

Enhancing Productivity in High Performance Computing through Systematic Conditioning Magdalena Sławi´nska, Jarosław Sławi´nski, and Vaidy Sunderam Dept. of Math and Computer Science, Emory University 400 Dowman Drive, Atlanta, GA 30322, USA {magg,jaross,vss}@mathcs.emory.edu

Abstract. In order to take full advantage of high-end computing platforms, scientific applications often require modifications to source codes, and to their build systems that generate executable files. The ever-increasing emphasis on productivity in HPC makes the efficiency of the build process an extremely important issue. Our work is motivated by the need for a systematic approach to the HPC life-cycle that encompasses build and porting tasks. In this paper we briefly present the design of the Harness Workbench Toolkit (HWT) that addresses the porting issue through a user-assisted conversion process. We briefly describe our adaptation capability model that includes routine conversions (string substitutions) as well as more advanced transformations such as 32-64 bit changes. Next, we show and discuss results of an experiment with a production source code (the CPMD application) that examines the effort of adapting the baseline code (the Linux distribution) to specific high-end machines (IBM SP4, Cray X1E, Cray XT3/4) in terms of the number of necessary conversions. Based on the conversion capability model, we have implemented conversion assistant modules that were used in the experiment. The experimental results are promising and demonstrate that our approach takes a step towards improving the overall productivity of scientific applications on highend machines. Keywords: porting parallel programs, productivity, legacy codes, conditioning, automatic code transformations, software tools.

1 Introduction High performance computing is well-established as a mainstream methodology for conducting science. The efficacy of applications, and the resulting scientific advances, are directly related to “productivity” in HPC – traditionally measured by raw performance, problem-size, and machine efficiency. This and related notions of productivity form the focus of several recent efforts, including the DARPA High Productivity Computing Systems (HPCS) [1] program and Cray’s adaptive supercomputing [2] vision. Whereas the HPCS initiative focuses on developing a new generation of economically viable systems that follow the rate of underlying technology improvement in delivering increased value 

Research, publication, and presentation supported in part by DOE grant DE-FG0206ER25729, NSF grant CNS-720761, and a faculty travel grant from The Institute of Comparative and International Studies, Emory University.

R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 341–350, 2008. c Springer-Verlag Berlin Heidelberg 2008 

342

M. Sławi´nska, J. Sławi´nski, and V. Sunderam

to end-users [3], Cray aims to provide the next productivity breakthrough by developing a single adaptable HPC platform that integrates multiple processing technologies. However, HPC productivity is also greatly influenced by several other less tangible factors, the most significant among them being the preparatory stages of application execution. Thus another strategy to improve HPC productivity is to reduce the effort involved in scientific application development. Research shows that in terms of productivity, the build process, i.e., compilation, linking, installation, deployment, and staging, may consume up to 30% of the development effort [4]. This large effort results mainly from the variety of software packages, program-building tools, and hardware architectures together termed the build problem [5]. Adaptation and deployment of scientific codes on high-performance platforms is particularly challenging due to new and unique architectures and the legacy nature of applications that have evolved over several decades with continual tweaking. Modern scientific codes are usually developed on common Unix-like architectures and need to be ported to run on specific HPC machines that are controlled by specialized (often lightweight) kernels. In order to obtain efficient executables, necessary modifications depend on vendor’s compilers and libraries, and may be relatively easily defined such as function renaming (e.g. POSIX functions), removing unsupported system calls, references to signals, taking care of pointer-sizes, modifications to handle the absence of the TCP protocol (sockets), etc. In addition to more or less straightforward source code changes, more challenging modifications such as fundamental algorithm reimplementation (e.g. vectorization), changing communication patterns to reflect specific hardware might be necessary. Our project aims to partially relieve site administrators, porting specialists, and computational scientists from the burdens of the build process by providing a systematic, integrated, and streamlined approach, in contrast to current ad-hoc practices. In order to accomplish this we propose a toolkit that facilitates conditioning, i.e., adaptation, of HPC applications to target platforms. In this paper we focus on the source code application adaptation aspect. Our methodology is based on a toolkit-assisted approach that can facilitate routine tasks. Discussions with application scientists and experts at Oak Ridge National Laboratory (ORNL), summarized in Section 3, show that conditioning is often tedious and requires cross-domain expertise. Furthermore, there is an acute lack of dedicated toolkits that can support this large and complex process. Although it is likely that conditioning HPC applications cannot be fully automated and requires human intervention, results of the conducted research and experiments show that certain patterns can be observed. In Section 4 we define the adaptation capability model based on the analysis of a few ORNL application source codes and demonstrate the productivity improvement with developed conversion assistants (Section 5). Finally, Section 6 concludes the paper.

2 Related Work As mentioned, execution of scientific codes on HPC platforms requires dealing with adaptation of scientific source codes, the build system and conditioning the target environment. There are many ways of tackling different aspects of these phases. One

Enhancing Productivity in High Performance Computing

343

way to approach conditioning issues is to learn from HPC community experiences that often are documented as guidelines or study cases. For instance, HPC vendors or compiler providers support their users with porting guidelines, compiler optimization flags, optimized libraries [6,7], or even with documentation on how to build popular scientific applications [8]. Clearly, this approach, albeit valuable, is very general, and in a specific situation a well-documented use case may be more desirable. Recent porting cases regard technologies that have gained popularity such as multi-core processing or reconfigurable supercomputers based on Field Programmable Gate Array (FPGA) technology. They report on portable frameworks [9,10], guidelines for specific applications [11,12,13], or algorithms [14,15]. Although addressing multi-core or highperformance reconfigurable computing platform porting issues is not our project main focus, our methodology can be applied to facilitate conditioning on such machines. We propose to encapsulate system- and application- specific knowledge into reusable, community-shared descriptions called profiles. Profiles instruct our conditioning assistant modules to retrieve appropriate parameters for conditioning tasks such as environment variable settings, compilation suites (compilers, libraries, optimization flags, etc) or code-snippets. Another way to deal with some conditioning issues is to utilize standards. For instance, portable run-time environments such as the Open Run-Time Environment (ORTE) [16] provide a unified interface to interprocess communication, resource discovery and allocation, and process launching across heterogeneous platforms. Our project intends to benefit from the unification provided by ORTE, e.g. with respect to application launching. Another example is notable GNU Autotools that helps overcome the build problem by standardizing the compilation, linking, and installation process. However, GNU Autotools does not fully address all the aspects characteristic to the HPC domain such as cross-compilation, use of restricted microkernels, and tuning to the hardware environment. In addition, it introduces its own compatibility issues such as the requirement for compatible versions at the user’s and developer’s side [17]. Our approach supports different build systems (Makefile-based as well as GNU Autotoolsbased) by providing a unified interface and encapsulating specific features into profiles. Providing tools that facilitate porting or building processes takes conditioning a step further. For instance, Environment Modules facilitates management of configuration information regarding software packages and libraries installed on a given computer system [18]. Although, in the past there have been efforts to develop tools for automatic porting [19], currently a supportive approach is preferred due to application complexity. The most advanced software solutions provide integrated programming environments that facilitate conditioning [20,21]. However, they are usually restricted to popular operating systems or very specific architectures. Instead of developing a new IDE for conditioning, our project aims to provide uniform access to build tools through a tool-virtualization methodology and by storing expert knowledge regarding specific tool configuration requirements, parameters, or options into profiles. This knowledge would be described once and repeatedly applied. In the HPC arena, Eclipse PTP [22] provides an interesting option for GUI-oriented development and run-time interface to heterogeneous systems. Our project can be perceived as complementary to Eclipse PTP in terms of supplying conditioning Eclipse plugins.

344

M. Sławi´nska, J. Sławi´nski, and V. Sunderam

3 A New Approach to Conditioning The traditional conditioning model is presented in Figure 1. End-users connect from their client systems to designated front-end nodes through remote (secure) connections.

Fig. 1. Traditional and proposed conditioning model

The security policy depends on the high-end system and may vary, e.g., static passwords and private-key authentication methods or one-time passwords. In fact, the manner of the user-supercomputer interaction is forced by the front-end node environment. Fortunately, often front-end nodes are controlled by the Linux operating system albeit usually modified by the vendor. Despite high Linux popularity and taking into account that scientists often need to utilize a few computational centers, adjusting to a new environment can be inconvenient for researchers and may distract them from their actual work especially in the context of the build problem. Analysis of current practices at large scientific computing establishments (e.g. DOE labs) suggests that application building is usually accomplished via GNU Autotools and Makefile-based systems, sometimes through proprietary scripts, and occasionally without any provided build system. But even when standard tools are employed, user modifications are often needed due to, e.g., cross-compilation issues. These factors contribute to lost productivity and worse, inconsistent or even incorrect results due to misuse of compiler flags or wrong library versions. Moreover, executing applications on different high-end platforms usually involves different preparatory steps that lead to further inefficiencies or inaccuracies. In order to address the above issues, we propose a shift in the conditioning model, as depicted in Figure 1. In our model the user works locally, instead of working directly

Enhancing Productivity in High Performance Computing

345

on a remote front-end node. We propose the concept of the build-tool virtualization to provide a common and unified interface to conditioning tools. In our approach the end-users interact with the Harness Workbench Toolkit (HWT) and issue a virtualized build command. The building assistant (Figure 2) orchestrates and runs, based on information stored in declarative profiles, relevant plugin modules (e.g. for environment preconditioning, staging, and compiling), the command that is passed to the execution assistant. The execution assistant (also steered by profiles) generates and executes actual target-specific commands.

Fig. 2. Harness Workbench Toolkit

The HWT architecture is presented in Figure 2. The HWT consists of three layers designed to be pluggable to support toolkit extensibility. The HWT behavior is configurable and tunable through declarative profiles that incorporate expert knowledge. Profiles, described in more detail in our previous paper [23], embody target platform specific knowledge, and may inherit from or override other descriptions. In addition, dynamic recursive resolution allows profile elements to cross-refer, and thus enables switching among predefined settings to select an appropriate suite of, e.g., compiler flags provided by a vendor. The other very important aspect of conditioning, as mentioned earlier, is adaptation of the application source code to the target platform. In the HWT, the porting assistant layer is dedicated to source code adaptation tasks. Its merits are based on the adaptation (conversion) capability model that aims to systematize required and commonly performed application source code conversions.

346

M. Sławi´nska, J. Sławi´nski, and V. Sunderam

4 Adaptation Capability Model Porting can be defined as a set of tasks required to launch and correctly execute the application on a target machine. Whereas it usually involves source code modifications, the application semantics needs to be preserved. In particular, the uniqueness of high-end machines and pursuit of peak performance makes this process exceptionally challenging in the HPC domain. We propose a toolkit-assisted approach that can facilitate routine tasks. In order to determine the routine activities that porting specialists deal with we have examined eight scientific codes in ORNL production use from a wide spectrum of computational science (chemistry, biology, fusion, computer science, and climate). We focused on those applications with available baseline and ported source codes, relevant to ORNL computing systems (Cray X1E (vector machine), Cray XT3/4, IBM SP4 (PowerPC processors)). A PC Linux distribution of a scientific application served as a baseline code. The results of our analysis are presented in the form of porting conversion categories shown in Figure 3.

Fig. 3. Conversion capability model

In general we can distinguish between two main code conversion categories, namely automatic and manual. The former refers to the set of conversions that to some extent may be automated, although user steering and input is still necessary. The simplest automatic conversions are substitutions that play a similar role to name refactoring (e.g. adding the prefix PXF for POSIX functions on Cray machines, or identifier mangling conventions). More advanced conversions concern pattern mapping. As an illustration, consider the different parameter passing conventions, e.g., the PGI Fortran compiler CALL FREE(PTR) and the IBM Fortran compiler CALL FREE(%VAL(PTR)), or time functions that may differ in sematics on various high-end machines. The other example relates to library incompatibilities such as a new version of the same library that has not been ported to a machine yet (e.g. FFTW 3.x  → 2.x), or highly vendor-optimized

Enhancing Productivity in High Performance Computing

347

library counterparts. In addition to converting patterns into patterns, there are conversions that track dependencies outside the matched patterns. For instance, consider the code adaptation from the 32-bit to the 64-bit compilation model for Fortran on Cray X1E. It involves tracking variables’ declarations and existance of the compilation parameter ’-sdefault64’ in the build system. The other very important example of a tracking template conversion is loop unrolling. This technique (i.e., loop unrolling) attemps to optimize loop execution time (e.g. by utilizing the maximum number of available CPU registers) and is commonly used for fast computations. Apart from mapping conversions there are cases where a given HPC system does not support certain features such as signals, threads, some system calls, sockets, or a synthetic file system (i.e., /proc). Detection conversions intend to deal with such situations and inform the user about non-portabilities. In general, detections trigger manual code adaptations that usually require expert knowledge of the hardware (architecture), system software in terms of compiler switches, usage of relevant libraries or versions, and application algorithms. For instance, in order to utilize a streaming feature such as SSE or 3DNow, the algorithm must be implemented in an assembler code. Another example concerns code vectorization to fully exploit vector processors, manual loop unrolling, or performance optimization and tuning. Based on the described conversion capability model, we have developed conversion assistant modules for the HWT porting assistant layer in order to perform the porting experiment with production scientific code.

5 CPMD Experiment We have implemented examples of substitution, template, and detection conversions of the conversion capability model (Figure 3) as appropriate porting assistant modules (Figure 2). The goal of this experiment was to examine the number of modifications of the baseline source code that can be supported by our conversion modules in comparison to the total number of necessary modifications to successfully build and execute an application on a given high-end machine. As target platforms we chose HPC systems relevant to ORNL, i.e., IBM SP4, Cray X1E, and Cray XT3/4. The most convenient from the analysis standpoint would be a comparison of original source codes with their ported counterparts. Unfortunately, obtaining original source codes may encounter considerable obstacles due to their constant adaptation over time. To overcome this difficulty, a preprocessor can be used to generate architecture-specific versions of application source codes. Therefore, we chose the CPMD application [24] since it supports hundreds of configuration sets including architectures of our interest. We assumed the baseline code is the CPMD PC Linux distribution. CPMD is a molecular dynamics scientific code consisting of about 700 files implemented in Fortran (and an occasional C file). The CPMD build system is based on a shell script that generates the appropriate Makefile file. The results of our experiment are presented in Table 1. The Cray X1E turned out to be the most challenging of all of the examined highend machines in the context of the number of required source code modifications. This includes manual as well as automatic conversions and results from the vector

348

M. Sławi´nska, J. Sławi´nski, and V. Sunderam Table 1. CPMD experimental results Conversion Specific conversion Target HPC system category IBM SP4 Cray X1E Cray XT3/4 manual changing FFT → ESSL FFT 12 7 N/A optimization & tuning 4 95 9 unknown 2 0 7 Total 18 102 16 automatic substitution 11 11 0 detection 8 21 6 basic template 4 1 2 tracking template 1 43 18 Total 24 76 26

architecture of Cray X1E and the non-vectorized CPMD baseline code. The CPMD application contains the intrinsic FFT library. However, in case of IBM SP4 and Cray X1E the vendor provides the optimized version of FFT. Therefore, we distinguished a named optimization subgroup in Table 1. Other optimization and tuning conversions we identified concern manual loop unrolling (often combined with pragmas to enable aggressive code optimization such as for indicating which variables can be shared), changing the 32-bit to 64-bit compilation model, and setting zeros to the memory (on IBM SP4 due to performance reasons). Finally, there are cases that require expert knowledge for their explanation and that is why we classified them into the ’unknown’ category. With regard to automatic conversions we did not identify any substitutions for Cray XT3/4. This results from the existence of the same compiler for Cray XT3/4 and for Linux and appropriate configuration files in the CPMD distribution, so there is no need for substitutions. In general, the number of required substitutions measures the portability of a given routine. The number of substitutions would be greater in the case of changing calculation precision. This usually implies exchanging the function suite and in consequence, often many substitutions and tracking conversions (the number of arguments and their types often needs to be adjusted to counterpart function calls). For instance, changing calculation precision for the CPMD code in regard to the BLAST library requires exchanging a suite of 58 function names. Not surprisingly, due to its vector nature, Cray X1E leads in the detection conversion group that included detecting of unsupported signals and synthetic file system. Results shown in Table 1 indicate that tracking conversions are dominant (for Cray X1E and Cray XT3/4) over the basic templates and suggests that more advanced conversions may reduce the porting effort to a greater degree. The obtained results demonstrate that our methodology is promising and may even contribute much more to porting applications not as well prepared for this process as CPMD (CPMD is portability-oriented, e.g. porting-sensitive routines or functions such as malloc() or open() are wrapped into proprietary functions). The outcome of this experiment shows that although manual conversions are much more cumbersome, require substantial effort and knowledge, and cannot be eliminated, we identified conversions that can be supported by a tool, and in this regard improve productivity of scientists involved in the porting process.

Enhancing Productivity in High Performance Computing

349

6 Conclusions and Future Work Despite maturing in many ways, high-end computing systems continue to require substantial effort in terms of application and environment adaptation to execute a scientific code on a target HPC platform. Conditioning needs will intensify in the future as new technologies emerge and gain popularity (e.g. multicore processors, reconfigurable supercomputers). Utilization of legacy codes is inevitable due to their tested reliability and demonstrated performance. In this paper we introduce the Harness Workbench Toolkit that supports conditioning of the environment and adaptation of source code to a particular high-end machine. We focus on the porting aspect of the conditioning, namely the HWT porting assistant layer. We describe the conversion capability model that classifies source code adaptation conversions. Our experiment with developed conversion assistants performed on the production code (the CPMD application) demonstrates that our approach is feasible and may improve productivity of computational scientists by limiting their involvement only to those porting tasks that cannot be supported by software (i.e., optimization and tuning). In addition, the results of the CPMD experiment show that conversion assistants can be parametrized according to a machine architecture. Therefore, our future work will concentrate on including architecture-specific information into profiles. For instance, the set of detection conversion assistants for a given high-end machine would be determined by the machine’s profile. We also plan to develop conversion assistants as lexical analyzers. Currently, they are implemented as regular expressions and were sufficient for the experiment. In addition, we intend to perform similar experiments with scientific applications implemented in C and mixed (Fortran/C) programming languages.

References 1. Feldman, M.: HPC, Thy Name is Productivity. HPCwire (March 2007), http://www.hpcwire.com/ 2. Snell, A., Wu, J., Willard, C.G., Joseph, E.: Bridging the Capability Gap: Cray Pursues Adaptive Supercomputing Vision. White Paper (February 2007), http://www.cray.com/downloads/IDC-AdaptiveSC.pdf 3. Kepner, J.: HPC Productivity: An Overarching View. The International Journal of High Performance Computing Applications 18(4), 393–397 (2004) 4. Kumfert, G.K., Epperly, T.G.W.: Software in the DOE: The hidden overhead of the build. Technical Report UCRL-ID-147343, Lawrence Livermore National Laboratory (2002) 5. Dubois, P.F., Kumfert, G.K., Epperly, T.G.W.: Why Johnny can’t build. Computing in Science and Engineering 5(5), 83–88 (2003) 6. Cray, Inc.: Craydoc (2007), http://docs.cray.com/ 7. Sun Microsystems, Inc.: Porting UNIX Applications to the Solaris Operating Environment (1999), http://developers.sun.com/solaris/articles/portingUNIXapps.html 8. PathScale, LLC: Building popular codes with PathScale (August 2007), https://www.pathscale.com/building code/index.html 9. Saunders, R., Jeffery, C., Jones, D.T.: A Portable Framework for High-Speed Parallel Producer/Consumers on Real CMP, SMT and SMP Architectures. In: Proc. 21st Intl. Parallel and Distr. Processing Symp. (IPDPS 2007), Long Beach, CA (March 2007)

350

M. Sławi´nska, J. Sławi´nski, and V. Sunderam

10. Bader, D., Kanade, V., Madduri, K.: SWARM: A Parallel Programming Framework fro Multicore Processors. In: Proc. 21st Intl. Parallel and Distr. Processing Symp (IPDPS 2007), Long Beach, CA (March 2007) 11. Olivier, S., Prins, J., Derby, J., Vu, K.: Porting the GROMACS Molecular Dynamics Code to the Cell Processor. In: Proc. 21st Intl. Parallel and Distr. Processing Symp (IPDPS 2007), Long Beach, CA (March 2007) 12. Petrini, F., Fossum, G., Fernandez, J., Varbanescu, A.L., Kistler, M., Perrone, M.: Multicore Suprises: Lessons Learned from Optimizing Sweep3D on the Cell Broadband Engine. In: Proc. 21st Intl. Parallel and Distr. Processing Symp. (IPDPS 2007), Long Beach, CA (March 2007) 13. Kindratenko, V., Pointer, D.: A Case Study in Porting a Production Scientific Supercomputing Application to a Reconfigurable Computer. In: IEEE Symp. Field-Programmable Custom Computing Machines (FCCM 2006), pp. 13–22. IEEE CS Press, Los Alamitos (2006) 14. Villa, O., Scarpazza, D.P., Petrini, F., Peinador, J.F.: Challenges in Mapping Graph Exploration Algorithms on Advanced Multi-core Processors. In: Proc. 21st Intl. Parallel and Distr. Processing Symp (IPDPS 2007), Long Beach, CA (March 2007) 15. Brunner, R., Kindratenko, V., Myers, A.: Developing and Deploying Advanced Algorithms to Novel Supercomputing Hardware. In: NASA Science Technology Conference – NSTC 2007 (2007) 16. Castain, R.H., Woodall, T.S., Daniel, D.J., Squyres, J.M., Barrett, B., Fagg, G.E.: The open run-time environment (openrte): A transparent multi-cluster environment for highperformance computing. In: Proc. 12th European PVM/MPI Users Group Meeting, Sorrento, Italy (September 2005) 17. Doar, M.B.: 5. In: Practical Development Environments, O’Reilly, Sebastopol (2005) 18. Furlani, J.L., Osel, P.W.: Environment Modules Project (2005), http://modules.sourceforge.net/ 19. Muppidi, S., Krawetz, N., Beedubail, G., Marti, W., Pooch, U.: Distributed computing environment (DCE) porting tool. In: Proceedings of the IFIP/IEEE International Conference on Distributed Platforms: Client/Server and Beyond: DCE, CORBA, ODP and Advanced Distributed Applications, pp. 115–129 (1996) 20. Sun Microsystems, I.: Sun Studio 12 C, C++ & Fortran Compilers and Tools (2007), http://developers.sun.com/sunstudio/ 21. SRC Computers, I.: SRC’s Carte Programming Environment (2007), http://www.srccomp.com/SoftwareElements.htm 22. The Eclipse Foundation: Parallel Tools Platform (2007), http://www.eclipse.org/ptp 23. Sławi´nska, M., Sławi´nski, J., Kurzyniec, D., Sunderam, V.: Enhancing Portability of HPC Applications across High-end Computing Platforms. In: Proc. 21st Intl. Parallel and Distr. Processing Symp (IPDPS 2007), Long Beach, CA (March 2007) 24. CPMD Consortium: Car-Parrinello Molecular Dynamics – CPMD 3.11 (March 2006), http://www.cpmd.org/

A Formal Model of Multi-agent Computations Maciej Smolka Institute of Computer Science, Jagiellonian University, Krak´ ow, Poland [email protected]

Abstract. The paper contains an extension of a formal model of multiagent computing system developed in previous publications towards considering a more general system state. We provide also some deeper details of the model in the case of a homogeneous hardware environment. The model provides us with a precise definition of the optimal task scheduling together with results on the existence and characterization of optimal scheduling strategies.

1

Introduction

The application of the multi-agent paradigm together with local, diffusion-based agent scheduling strategies provide us with a relatively simple decentralized management of large-scale distributed computations. Multi-agent systems based on the idea of task diffusion (cf. [1]) dedicated to large-scale computations have been developed for the last several years [2]. A formal model for such systems was introduced in [3] and has been further developed in [4,5,6]. It provides us with a precise definition of the optimal task scheduling as well as results on the existence of optimal scheduling strategies and optimality conditions. In this paper the model is extended to consider a more general system state. We provide also more detailed modelling of a simple but realistic case of a homogeneous network of identical machines. Finally results on the existence and characterization of optimal scheduling strategies are presented.

2

Architecture of a Computing MAS

The principles of the MAS architecture which form the basis for our mathematical model were defined in [2,7] and has been further developed in [8,9]. The full description as well as the details of realization and test results can be found therein. Here let us recall only some crucial points. The suggested architecture of the system is composed of a computational environment (MAS platform) and a computing application being a collection of mobile agents called Smart Solid Agents (SSA). The computational environment is a triple (N, BH , perf ), where: N = {P1 , . . . , PN } , where Pi is a Virtual Computational Node (VCN). Each VCN can maintain a number of agents. R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 351–360, 2008. c Springer-Verlag Berlin Heidelberg 2008 

352

M. Smolka

BH is the connection topology BH = {N1 , . . . , NN }, Ni ⊂ N is an immediate neighborhood of Pi (including Pi as well). perf = {perf1 , . . . , perfN }, perfi : R+ → R+ is a family of functions where perfi describes relative performance of VCN Pi with respect to the total memory i request Mtotal of all agents allocated at the node. The MAS platform is responsible for maintaining the basic functionalities of the computing agents. Namely it delivers the information about the local load concentration Lj and Qj (see (2) and (3) below), it performs agent destruction, partitioning and migration between neighboring VCN’s and finally it supports the transparent communication among agents. We shall denote an SSA by Ai where index i stands for an unambiguous agent identifier. Each Ai contains its computational task and all data necessary for its computations. Every agent is also equipped with a shell which is responsible for the agent logic. At any time Ai is able to denominate the pair (Ei , Mi ) where Ei is the estimated remaining computation time measured in units common for all agents of an application and Mi is the agent’s RAM requirement in bytes. An agent may undertake autonomously one of the following actions: continue executing its internal task, migrate to a neighboring VCN or decide to be partitioned, which results in creating two child agents {Aij = (Tij , Sij )}, j = 1, 2. We assume that in the case of the agent partitioning the following conditions hold: Ei > Eij , Mi > Mij , j = 1, 2. The parent SSA disappears after such a partition. A computing application may be characterized by the triple (At , Gt , Scht ), t ∈ [0, +∞) where At is the set of application agents active at the time t, Gt is the tree representing the history ofagents’ partitioning until t. All agents active t till t constitute the set of its nodes s=0 At , while the edges link parent agents to their children. All information on how to rebuild Gt is spread among all agents such that each of them knows only its neighbors in the tree. {Scht }t∈[0,+∞) is the family of functions such that Scht : At → N is the current schedule of application agents among the MAS platform servers. The function is defined by the sets ωj containing indices of agents allocated on each Pj ∈ N. Every ωj is locally stored and managed by Pj . Each server Pj ∈ N asks periodically all local agents (allocated on Pj ) for their requirements and computes the local load concentration Lj =

j Etotal j perfj (Mtotal )

j where Etotal =



j Ei and Mtotal =

i∈ωj



Mi

(1)

i∈ωj

Then Pj communicates with neighboring servers and establishes   k k Lj = (Lk , Etotal , Mtotal , perfk ) : Pk ∈ Nj

(2)

as well as the set of node indices Qj = {k = j : Pk ∈ Nj , Lj − Lk > 0} .

(3)

A Formal Model of Multi-agent Computations

353

The mentioned papers as well as [4] describe migration and partitioning strategies which make use of the above-defined quantities. These strategies are related to the physical phenomenon of the molecular diffusion in crystals.

3

System State Revisited

Next let us recall the features of a mathematical model of multi-agent computations, which is based on the presented architectural principles. It has already been presented in [3,4,5,6]. Here we shall propose a generalization of the model, namely we shall consider a more general form of the system state. Let us recall that we have considered the set of all possible application’s agents A. Because of problems with the determination of such a set the crucial idea is to avoid considering the evolution of a single agent and to study the dynamics of a whole computing application instead. We observe the system state in discrete time moments. Let us recall the notion of the vector weight of an agent, which is the mapping w : N × A −→ RM + . Note that otherwise than in our previous papers we allow M to be greater than 2. We assume that the dependency of the total weight of child agents after partition upon their parent’s weight before partition is well-known and linear, i.e. there is a matrix P ∈ RM×M such that + in the case of partition A → {A1 , A2 } we have wt+1 (A1 ) + wt+1 (A2 ) = Pwt (A).

(4)

This time we relax the assumption of componentwise dependency. If this is the case P is diagonal. Next let us recall the notion of the total weight of all agents allocated on a virtual node P at any time t, i.e.  Wt (P ) = Scht (A)=P wt (A) This notion is crucial in our search for a global description of the system dynamics since it is a global quantity describing the state of the system in such a way that is appropriate for our purposes. As for the components of w it is straightforward to choose Ei and Mi defined i i in Sec. 2 as we did in previous papers. Then obviously Etotal and Mtotal are the components of W . In general it may be convenient to find other state variables i i (see Sec. 5). In any case both Etotal and Mtotal should remain observables of our system. In the sequel we shall assume that the number of virtual nodes N = N is fixed. Let us introduce the notation Wtj = Wt (Pj ) for j = 1, . . . , N . Then Wt may be interpreted as a vector from RMN or, if it is + more convenient, as a nonnegative M × N matrix. The equality Wt = 0 means that at the time t there is no computational activity. Thus 0 is the target of the application’s evolution: once our computing MAS reaches this state we want it to stay there forever.

354

4

M. Smolka

Equations of Evolution

As said before we shall consider Wt as a state of the computing application. Now we shall formulate the equations of evolution of Wt . Since they contain a stochastic term (see below), Wt turns out to be a discrete stochastic process. The following state equations are an adaptation of the equations presented in [3] or [5] to the generalized state. First consider three simple cases. ’Established’ evolution (when there are neither partitions nor migrations). Then we assume that the state equation has the form Wt+1 = F (Wt , ξt )

(5)

where F is a given mapping and (ξt )t=0,1,... is a given sequence of random variables representing the background load influence. We assume that ξt are mutually independent, identically distributed and have a common finite set of values Ξ, which is justified in many natural situations [6]. Since we want Wt to stay at 0 (i.e. we want 0 to be an absorbing state of Wt ), we need to assume that for every t F (0, ξt ) = 0 (6) with probability 1. Partition at node j. Then we have   j j Wt+1 = I − diag(ujj (W )) Wtj + P diag(ujj t t t (Wt )) Wt i Wt+1

for i = j

= Wti

(7)

MN where components of ujj → [0, 1]M are the proportions of the weight t : R+ components of splitting agents to the corresponding components of the total weight of all agents at node j at time t. By diag(v) we denote the square matrix obtained by putting the elements of the vector v on the diagonal and 0’s outside the diagonal. Migration from j to k. In this case the state equations have the form ⎧  j jk ⎪ ⎪ W = I − diag(u (W )) Wtj t t ⎨ t+1 j k (8) Wt+1 = Wtk + diag(ujk t (Wt ))Wt ⎪ ⎪ ⎩W i i =W for i ∈ {j, k} t+1

ujk t

ujj t .

t

with analogous to In reality all three cases usually appear simultaneously. Therefore the final state equations shall be a combination of them which will reduce to a particular ’simple’ case when there are no activities in the system related to the other cases. Here we present such a combination, namely we propose the following form of the state equations    ji j i i t , ξt ), P diag(uii Wt+1 = g i F i (W (W )) W , diag(u (W )) W t t t t t t j=i (9) i Wi =W 0

A Formal Model of Multi-agent Computations

355

 ti = I − N diag(uik  is a given for i = 1, . . . , N , where W (W )) Wti and W t t k=1 initial state. Remark 1. In our previous papers we considered only g(s, p, m) = s + p + m. A more general assumption on g could be g(s, 0, 0) = s, g(0, p, 0) = p, g(0, 0, m) = m. It follows that Wt is a controlled stochastic process with a control strategy π = (ut )t∈N ,

ut : RMN −→ U. +

(10)

The control set U contains such elements α from [0, 1]M(N ×N ) that satisfy at least the following conditions for m = 1, . . . , M . ji αij m · αm = 0 for i = j,

iN αi1 m + · · · + αm ≤ 1 for i = 1, . . . , N .

(11)

In fact quite often the conditions imposed on U shall be more restrictive (see next section). The first equation in (11) can be interpreted in the following way: at a given time migrations between two nodes may happen in only one direction. The second equality says that the number of agents leaving a node must not exceed the number of agents present at the node just before the migration. Remark 2. It is easy to see that the control set U defined by the conditions (11) is compact (and so are of course its closed subsets). In the most general case one might want to take the whole RMN for the state + space of the stochastic process Wt . But, on the other hand, in the most common situation its components represent some resources which are naturally bounded and quantized (see [5, Sec. 2]) for an analysis of a special case). Therefore we shall assume that the state space is a finite subset of NMN containing 0. Let us call the elements of this finite set si , i.e. S = {s0 = 0, s1 , . . . , sK }. This analysis of the state space has some consequences. First of all we have to assume that F and g have values in S. Likewise, the condition (11) is not sufficient for the equations (9) to make sense. Namely we have to assume that for any t ∈ N and W ∈S ut (W ) ∈ UW = {α ∈ U : G(W, α, ξ) ∈ S for ξ ∈ Ξ} with Gi (W, α, ξ) = g i (F i ((I −

N k=1

diag(αik ))W i , ξ), P diag(αii ) W i ,  ji j j=i diag(α ) W ) (12)

denoting the right hand side of (9). The above equality implies that G(W, 0, ξ) = F (W, ξ) ∈ S for any W and ξ, which means that 0 ∈ UW , therefore UW is nonempty for every W ∈ S. Another consequence of (12) is that G(0, α, ξ) = F (0, ξ) = 0, i.e. 0 remains an absorbing state of Wt even if we apply some agent operations. Remark 3. Given (9) it is easy to see that Wt is a controlled Markov chain with transition probabilities pij (α) = Pr(G(si , α, ξ0 ) = sj ) for i, j = 0, . . . , K, α ∈ Usi . The transition matrix for the control u is P (u) = [pij (u(si ))]i,j=0,...,K .

356

5

M. Smolka

Analysis of a Special Case

In this section we shall present more details on the state variables and the state equations in a simple but nontrivial special case. Let us assume that we use a homogeneous computational environment, i.e. VCN’s are deployed on different identical physical nodes. Furthermore assume that the evolution of Ei for each agent when there are no migrations, partitions or delays related to a higher background load is like on Fig. 1, i.e. linearly decreasing from the time of the agent’s creation ts until the end of computations. For each agent the maximal ¯ of Ei may be different. Assume also that Mi is equal to a positive convalue E stant throughout the agent’s life and before its creation and after its destruction Mi is equal to 0 (Fig. 1). Of course not all computational agents behave in the E

M

¯ E

¯ M

ts

¯ ts + E

t

ts

¯ ts + E

t

Fig. 1. An agent’s remaining time of computations (in common units, left picture) and memory requirement (in bytes, right picture)

presented way (e.g. HGS agents [9] do not), but there is an important class of agents related to CAE computations (e.g. SBS linear solver agents [7]) whose evolution can be modeled as above. In order to find appropriate state variables we divide the agents into generations according to their place in the partition history, i.e. the generation 0 consists of agents never split and the last generation (with number G) contains agents too small to be partitioned. We assume that within each generation all agents have the same memory requirement. Then the memory requirement becomes a strictly decreasing function of a generation M : {0, . . . , G} −→ N. Dej note by Nk,t the number of agents in the generation k at the node j at the time j t and by Ek,t the total remaining time (cf. (1)) of all agents in the generation k at the node j at the time t. Using these notations we define the state variable  T j j j j Wtj = E0,t , . . . , EG,t , N0,t , . . . , NG,t G j (W T denotes the transposed matrix for W ). Note that both Etotal = k=0 Ekj  G j and Mtotal = k=0 M (k)Nkj are observables of our system, however they are not state variables. Next let us consider the evolution of Wt . To this end let us assume that during the partition an agent is split into two equal children in such a way that both

A Formal Model of Multi-agent Computations

357

get a half of the parent’s remaining time of computations. Then the equations of the partition for one agent belonging to the generation k (k < G) have the form  j j j j Ek,t+1 = Ek,t −e Nk,t+1 = Nk,t −1 j j j j Ek+1,t+1 = Ek+1,t +e Nk+1,t+1 = Nk+1,t +2 where e is the splitting agent’s remaining time. Thus the partition matrix has the following form   B 0 P= 0 2B where B is the matrix which has 1’s right under the diagonal and 0’s elsewhere. It means that the system evolution follows (7). Similarly it is easy to see that (8) describes the dynamics of migration in this situation as well. Therefore let us concentrate on the ’established’ case. Then the evolution equation of variables E is the following j j j j Ek,t+1 = Ek,t − Nk,t + Dk,t j j where Dk,t ∈ {0, . . . , Nk,t } is a random number of agents delayed due to the high background load. In order to drop the dependency of the set of values of D on j j j the current state we shall rewrite it in the following way Dk,t = [Nk,t ξk,t ] where j [x] stands for the whole part of x and ξk,t is a sequence of random variables with values in [0, 1]. The equations of evolution of variables N are slightly more complicated, namely we have ⎧ j j j j ⎪ if Ek,t ≥ 2Nk,t ⎨Nk,t − Dk,t j j j j j j j  Nk,t+1 = Ek,t − Nk,t − D (13) if Nk,t < Ek,t < 2Nk,t k,t ⎪ ⎩ j j j Ek,t+1 if Ek,t = Nk,t

 j ∈ {0, . . . , N j −Dj −1} is another random variable. It represents the where D k,t k,t k,t uncertainty about whether agents within a generation are at the same point of  j , the better balanced the generation. computations: the smaller the value of D k,t  j = 0 means that all agents within the generation j are exactly The equality D k,t at the same point. The main reason of a probable poor balancing are agentswanderers, which do not perform any computations and migrate over the network instead for some time, and finally end up at the original node in a very early stage of computations in comparison to agents which have stayed ’home’ and  j enables us to consider such situations in a probabilistic way. worked hard. D k,t  j = [(N j − Dj )ζ j ] where ζ j is a sequence of random As before we put D k,t k,t k,t k,t k,t variables with values in [0, 1). Thus the interpretation of the equations (13) in the case of the perfect balancing of computations within generations is the following. The first means that when there is sufficiently much work to do the number of agents is constant. The second and third mean that at the end of computations we have a number of agents and each of them is to finish its task in a time unit. To sum up, in the presented case we have obtained the state equations, which have the form (5). This shows what F may look like. In general we need to consider migrations and partitions as well, ending up with equations like (9).

358

6

M. Smolka

Optimal Scheduling Problem

Now let us recall the definition of the optimal scheduling for a computing MAS. We shall follow the steps of [3] and put our problem into the stochastic optimal control framework. The general form of the cost functional for controlled Markov chains of our type is (cf. [10]) ∞ V (π; s) = E[ t=0 k(Wt , ut (Wt ))] (14) where π is a control strategy (10) and s is the initial state of Wt , i.e. W0 = s. Since 0 is an absorbing state we shall always assume that remaining at 0 has no cost, i.e. k(0, ·) = 0. This condition guarantees that the overall cost can be finite.  Let us define the following set of admissible control strategies U = π : ut (W ) ∈ UW , t ∈ N . Now we are in position to formulate the optimal scheduling  we look for a control strategy problem. Namely given an initial configuration W ∗ π ∈ U such that    ) = min V (π; W ) : V (π ∗ ; W π ∈ U, Wt is a solution of (9) . (15) Consequently, an optimal scheduling for Wt is a control strategy π ∗ realizing the minimum in (15). Changing the cost functional we obtain various criteria for the optimality of the scheduling. Let us present here some cost functionals of type (14) which are appropriate for multi-agent computations. The first one is the expected total time of computations   VT (π; s) = E inf{t ≥ 0 : Wt = 0} − 1 . (16) In other words, in this case a scheduling is optimal if it is expected to finish the computations in the shortest time. The second functional promotes the good mean load balancing over time. It has the following form   N ∞ N i 2 i VL (π; s) = E (L − L ) , Lt = N1 (17) t t t=0 i=1 i=1 Lt where Lit is the load concentration (1) at the i-th VCN at the time t. These i i quantities are well defined because we have assumed that Etotal and Mtotal are observables of our system. Both the above examples do not contain an explicit dependency on the control. Generalizing it a little allows us to penalize migrations. Namely take ϕ : S → R+ , a ≥ 0 and μij m : [0, 1] → R+ nondecreasing and such that μij (0) = 0, and put    M  ∞ ij ij VM (π; s) = E . (18) t=0 ϕ(Wt ) + a m=1 i=j μm (um,t (Wt )) In the above expression ϕ can have the form of the term under the estimated value in (16) or (17), μij allows us to penalize the distance between i-th and j-th VCN and a is a tuning factor (the greater is the value of a, the greater is the influence of the ’migration’ term).

A Formal Model of Multi-agent Computations

7

359

Existence and Characterization of Optimal Strategies

Let us consider first the existence of solutions for problem (15). To this end let us denote by R(u) = [pij (u(si ))]i,j=1,...,K the ’probably not absorbing’ part of the transition matrix for a control u. The following proposition is the main existence result. Proposition 1. Assume that (R1) RK (u) is a contraction for every u such that u(s) ∈ Us or (R2) Rn (u) is a contraction for some n ≥ 1 and u as above but additionally k(sj , α) ≥ ε > 0 for j = 0, α ∈ Usj . Then there exists the unique optimal solution of (15). Proof. It is sufficient to notice that Us are finite so they are compact and k(sj , ·) is obviously continuous. It means that the assumptions (A1)–(A3) from [10, Chap. 4] hold. Thus the thesis is a straightforward consequence of [10, Theorem 4.2]. Corollary 1. Consider our example cost functionals VT , VL and VM . 1. Problem (15) for VT has the unique solution provided (R2) holds. 2. Problem (15) for VL has the unique solution provided (R1) holds. 3. Existence of the solution for (15) for VM depends on the assumption on ϕ. If the latter is separated from 0 we need (R2) otherwise (R1). Now we shall present some Bellman-type optimality conditions for (15), which are another consequence of [10, Theorem 4.2] and its proof. We shall formulate them in the following proposition. Proposition 2. Assume (R1) or (R2). Then the optimal solution of (15) is a stationary strategy π ∗ = u∞ = (u, u, . . . ) and it is the unique solution of the equation   K ∗ V (π ∗ ; s) = minα∈Us (19) j=1 pij (α)V (π ; sj ) + k(s, α) . The solution of (19) exists and is the optimal solution of (15). The simple but important consequence of this proposition is that in order to find the optimal scheduling we need to consider only stationary strategies. Note that diffusion strategies defined in [8,4] are stationary, thus we can try to verify their optimality (or quasi-optimality) by means of a variant of the equation (19). On the other hand this equation also allows us to compute the optimal strategy by means of some iterative procedures like Gauss-Seidel [10].

8

Conclusions

The presented MAS architecture accompanied by diffusion-based agent scheduling strategies make a convenient and efficient environment for large-scale distributed computations. The presented mathematical model based on the

360

M. Smolka

stochastic optimal control theory provides us with a new precise definition of optimal scheduling in such an environment. It also enables us to obtain some results on the existence of optimal scheduling strategies (Proposition 1 and Corollary 1) as well as the optimality conditions (Proposition 2). These results show in particular that any optimal scheduling must belong to the class of stationary strategies, which have been utilized during tests [8,9]. In this paper the formal model has been extended to allow more general system state. This in turn has enabled us to provide a more detailed description of the generalized model in an important special case (Sec. 5) which form the basis for experiments designed to verify the model. Such experiments are being undertaken with results soon to come.

References 1. Luque, E., Ripoll, A., Cort´es, A., Margalef, T.: A distributed diffusion method for dynamic load balancing on parallel computers. In: Proceedings of EUROMICRO Workshop on Parallel and Distributed Processing, San Remo, Italy, pp. 43–50. IEEE Computer Society Press, Los Alamitos (1995) 2. Uhruski, P., Grochowski, M., Schaefer, R.: Multi-agent computing system in a heterogeneous network. In: Proceedings of the International Conference on Parallel Computing in Electrical Engineering (PARELEC 2002), Warsaw, Poland, pp. 233– 238. IEEE Computer Society Press, Los Alamitos (2002) 3. Smolka, M., Grochowski, M., Uhruski, P., Schaefer, R.: The dynamics of computing agent systems. In: Sunderam, V.S., van Albada, G.D., Sloot, P.M.A., Dongarra, J. (eds.) ICCS 2005. LNCS, vol. 3516, pp. 727–734. Springer, Heidelberg (2005) 4. Grochowski, M., Smolka, M., Schaefer, R.: Architectural principles and scheduling strategies for computing agent systems. Fundamenta Informaticae 71(1), 15–26 (2006) 5. Smolka, M.: Optimal scheduling problem for computing agent systems. Inteligencia Artificial 9(28), 101–106 (2005) 6. Smolka, M., Schaefer, R.: Computing MAS dynamics considering the background load. In: Alexandrov, V.N., van Albada, G.D., Sloot, P.M.A., Dongarra, J. (eds.) ICCS 2006. LNCS, vol. 3993, pp. 799–806. Springer, Heidelberg (2006) 7. Grochowski, M., Schaefer, R., Uhruski, P.: An agent-based approach to a hard computing system — Smart Solid. In: Proceedings of the International Conference on Parallel Computing in Electrical Engineering (PARELEC 2002), Warsaw, Poland, pp. 253–258. IEEE Computer Society Press, Los Alamitos (2002) 8. Grochowski, M., Schaefer, R., Uhruski, P.: Diffusion based scheduling in the agentoriented computing systems. In: Wyrzykowski, R., Dongarra, J., Paprzycki, M., Wa´sniewski, J. (eds.) PPAM 2004. LNCS, vol. 3019, pp. 97–104. Springer, Heidelberg (2004) 9. Momot, J., Kosacki, K., Grochowski, M., Uhruski, P., Schaefer, R.: Multi-agent system for irregular parallel genetic computations. In: Bubak, M., van Albada, G.D., Sloot, P.M.A., Dongarra, J. (eds.) ICCS 2004. LNCS, vol. 3038, pp. 623– 630. Springer, Heidelberg (2004) 10. Kushner, H.: Introduction to Stochastic Control. Holt, Rinehart and Winston (1971)

An Approach to Distributed Fault Injection Experiments Janusz Sosnowski, Andrzej Tymoczko, and Piotr Gawkowski Institute of Computer Science, Warsaw University of Technology, ul. Nowowiejska 15/19, Warsaw 00-665, Poland [email protected]

Abstract. Software implemented fault injection technique is gaining much interest in evaluating system dependability. For complex software applications fault injection experiments take a lot of time. In the paper we present an innovative approach to fault injection by performing it in LAN distributed environment. The paper presents the system architecture, task scheduling and the analysis of its effectiveness. The presented considerations are illustrated with practical results. Keywords: distributed systems, processor intensive workloads, task scheduling, load balancing, performance evaluation.

1

Introduction

Fault injection experiments are widely used to check system dependability (susceptibility to faults) [1,2,3]. We have developed fault injectors which simulate faults in the real IBM PC system by disturbing the states of processor registers and memory cells during the execution of the analysed program [4,5]. In the performed experiments we generate in an automatic way many faults and analyse their effects (e.g. incorrect results, system exceptions, time-outs, and correct results). For every injected fault (test) the program has to be executed, so usually experiments need long simulation time. To speed up the experiments we have developed DInjector system, which manages and distributes fault injection tasks in LAN environment. In the literature only single node fault injectors were described, so we are pioneers in distributed fault injection. Developing DInjector we assumed the possibility of performing experiments by many users in parallel with normal workload in LAN. From the variety of parallel computation models (e.g. [6,9,11]) the farm processing is relatively close to the specificity of fault injection experiments. Nevertheless its adaptation to heterogeneous environment, multiaccess and performance enhancements were challenging tasks. This required some studies on task decomposition, load balancing in nodes, task scheduling, etc. In the literature these issues are usually analysed separately and for abstract assumptions (e.g. [8,9,10,12]). In our project we consider them in a comprehensive way taking into account some specific problems (neglected in the literature), in particular: automatic adaptation to system configuration changes, dealing with various erroneous situations caused by injected faults as well as faults generated by other LAN users. Another challenge R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 361–370, 2008. c Springer-Verlag Berlin Heidelberg 2008 

362

J. Sosnowski, A. Tymoczko, and P. Gawkowski

was verification of the developed system, its effectiveness and performance [7]. For this purpose we have embedded some original monitoring modules. Resuming the paper presents a case study of a sophisticated system, which integrates various technologies and is used in practice. In section 2 we give an outline of the system architecture. Schemes of distributing simulation tasks and experiment management are presented in section 3. Section 4 describes the methodology of evaluating system effectiveness and performance (illustrated with practical results).

2

System Architecture

Fault injection experiments consist of several tasks: ME - extraction of program modules for the analysis, GR - collecting reference information (golden run log GRL) from non disturbed application execution, MS - selection of fault injection moments, EX - fault injection tests. The user specifies experiment configuration, the number, localisation and classes of injected faults, etc. [4,5]. Basing on these specifications delivered by the users the DInjector system creates and distributes various tasks for the computer nodes, monitors their execution and collects the results. These processes are realised by the following functional objects: – User interface - assures user interaction with the system, implemented in the JSP (JavaServer Pages) technology running on Tomcat HTTP server. – Database - collects the information provided by the user and the DInjector system. It co-operates with data warehouse (based on MS SQL) comprising aggregation and data mining capabilities [6,10,11]. – DICoordinator - responsible for the distribution and management of computational tasks assigned to a subnetwork of homogeneous workstations. Coordinators are implemented in C++ (system services in Win32 or daemons in Linux environment). Communication channels use CORBA technology, which simplifies many low-level aspects. – Fault injection cores (FICs) - services or daemons (implemented in C++.) realising all computational tasks (ME, GR etc.) in the workstations (managed by the corresponding instance of the DICoordinator). All tasks of performed experiments are written into the DInjector database. The sequence of tasks corresponds to the experiment model and is guaranteed by the user interface (WWW application) and database transactions. The scheduling of the tasks (section 3) takes into account assigned user credits and priority as well as node loading. All system components (databases, coordinators and simulation services) are independent and can work on different computers as well as on the same workstation. This allows scaling the system and using different workstation subnetworks as a part of DInjector. There is no limit for the number of coordinators in the system so many subnetworks (Linux or Win32 like) can be connected with coordinators to serve the system. The subsystem composed of the DICoordinator and related to it processing nodes (DINodes) is called DICluster. Different DIClusters cooperate with the same database. The DICoordinator is composed of three

An Approach to Distributed Fault Injection Experiments

363

co-working but independent parts (they can be spread within one subnetwork of homogeneous workstations): – Distribution coordinator (DIManager) selects tasks, assigns them to nodes, manages and monitors their execution at the workstations. Moreover, it controls resource and merging servers (defined below) and assigns nodes to them, – Resource server (DIRServer) is responsible for distribution of needed files (e.g. GRL, application code and environment) across the network of processing nodes (DINodes), – Merging coordinator (DIMerger) collects results from DINodes, writes them into the database and signals the termination of tasks to the DIManager. These three components are compiled within one application (they can be switched on or off). DIManager comprises two subcomponents: one used for the preliminary phases (ME, GR, MS) and the second one responsible for the testing phase (EX). This functional decomposition assures good system scalability and eliminates possible bottlenecks of the system (e.g. installation of multiple resource servers within a network). Increased number of processing nodes results in higher speed of results generation. In a single instantiation of the system we can create several DIMergers operating on different machines and writing results in parallel to several databases. Fault injection subsystems can use the same resource server (or several of them). One DIManager should be associated with at least one DIMerger and DIRServer. One DIMerger is associated with only one DIManager. One DIRServer can be associated with many DIManagers. Hence, two DICoordinators can share one DIRServer. Each DIManager has its own group of associated processing nodes. In the case of homogeneous fault injectors one DIManager can control preliminary phases and the second one the test phase (EX). The preliminary phases of the experiment i.e. ME, GR and MS are executed by simple assignment of each task of this type to a single processing node. In the case of tests (phase EX) they are partitioned into subsets which can be assigned to different nodes. Each subset of tests is denoted as task TS which is composed of so called parcels (subtask PC). A parcel may comprise one or more tests. Each correct execution of the task (ME, GR, MS or TS) is signalled by the fault injection core (FIC in DINode) to DIMerger. The node signals its readiness for further processing to DIManager and in the meantime DIMerger initiates writing the results to the database. The termination of this process is signalled to DIManager. System operation is described in section 3.

3

System Operation and Task Scheduling

In the DInjector system we have two levels of task identification. The higher level relates to tasks defined by the user (ME, GR, MS, EX, PC). At the lower level we have so called monitored tasks which are operational units of DICluster. These tasks are correlated with appropriate resource and operation to be executed. They control the allocation of the resources as well as deallocation in the

364

J. Sosnowski, A. Tymoczko, and P. Gawkowski

case of unsuccessful operation. Within the DIManager we have the task monitor which checks the time-out for the initiated task. Expired time-out restores allocated resources. Each monitored task handles the following events: task correct termination, error during task execution (task cancelling) and task time-out expiration. The most important monitored tasks are: allocation task and task status monitoring task. The allocation task relates to processing node and tasks performed by this node. Termination and closing a task moves the node from the set of busy nodes to the set of free nodes. Time-out will result in deleting the node from the set of registered nodes. The identifiers of monitored tasks are assigned by DIManager. The experiment is distributed through the coordinators by dividing the set of all tests. A single test requires the specification of the fault (location and type) and its injection time instant (fault triggering). The Cartesian product of these two spaces defines the test space for the experiment. To improve the processing efficiency, tests are grouped into parcels. The parcel is a unit treated in a transactional manner (if all tests in the parcel are executed the whole parcel is committed). The coordinator locks in the database a set of parcels (for a specified period of time) and sends it to the available free node (within its network). The results of the tests are successively sent back to the merging coordinator, which collects single test results and writes the results of the whole parcel into the database committing the transaction. The node keeps the application environment for the further experiments as long as there are more tasks of the analyzed application. Such caching minimises the communication overhead while setting the node for a task. Every simulation node registers itself to the local coordinator and signals its readiness for serving. The whole system sets-up automatically - nodes register to coordinators in their networks, coordinators connect to the system database. Two kinds of time-outs are defined - long and short. The first one is applied to task parcels at the level of coordinators. If the result is not committed after the long period of time the task parcel assignment is cancelled and restarted. Short timeouts are used to monitor the task activity at the node level by the management coordinator. In the case of a single task time-out at the node level, the node is forced to cancel its execution. Similarly, in the case of a node or coordinator failure tasks are rescheduled to other available nodes in the system. In the case of a short break of the communication between the database and coordinators or coordinators and nodes, all the tasks assigned to unavailable stations after the time-outs will be unlocked and available for execution by other nodes. The workstations can be turned-on or turned-off independently (dynamic configuration). Tests (EX task) are executed by distributing parcels (with test specifications) to the nodes. All tests related to the analyzed application are defined by triggering moments and fault locations. A single test is defined by a pair . For such a specification the fault is simulated in the node by generating a fault mask (e.g. at random) and then disturbing an appropriate bit (or bits) in accordance with the mask specification and fault type (e.g. bit flip) [5]. In an experiment we usually simulate thousands or

An Approach to Distributed Fault Injection Experiments

365

millions of faults. So, an important issue is appropriate coding of test specifications for the whole space of specified tests defined by the generated fault triggering moments (e.g. in a pseudorandom way in the dynamic image of the application code) and specified locations. So, DIManager comprises both lists of specifications. Than test parcels are created and numbered according to the following principle: each parcel is assumed to comprise the same number of tests (NTP), all tests are numbered in such a way that we take the first triggering moment and define subsequent tests taking into account subsequent locations from the location list. Then we take the second triggering moment and assign subsequent test for all the locations etc. The parcel 0 comprises the first NTP tests from the list, parcel 1 the second portion on NPT tests, etc. DIManager sends to the nodes only the set of the numbers of test parcels, the set of related fault triggering moments, fault locations and the size of the parcel. The fault triggering moments are assigned to the appropriate code addresses during the selection phase (MS task). The list of locations comprises pairs , where the location cardinality defines the number of requested fault injections. Such a compressed list of locations is revealed in the node to specify individual fault injections. Hence the total test specification data sent in the network of nodes is of the complexity O(M + N*L), where M is the number of all triggering moments, N is the number of used processing nodes and L is the number of all considered different locations (without repetitions). We have admitted simultaneous access to the system by many users. The preliminary tasks (ME, GR, MS) are selected by the related executor in two phases: at first we select the user than its tasks. Selection of the user will be described later on (fair access is assured). User task are selected in the sequence ME, GR and MS, this is a natural sequence which assures the fastest interaction with the user. The preliminary phase executor performs the following operations in a loop: reservation of free nodes, selection of tasks for execution, assignment of the tasks to the nodes (not used nodes are returned as free nodes), task delegation to the nodes. In the experiment phase the appropriate executor selects at first the user from the set of the users with ready experiments for the execution. For the selected user we select the experiment with highest priority and for such experiment test parcels are defined and distributed to the nodes. Let us assume that we have n parcels waiting for the execution and we have m idle nodes ready for operation, other registered nodes (N - m) are used for other tasks. The executor will try to assign to each free node a unique parcel if the number of idle nodes is sufficient. If the number of free nodes is lower than n, then we assign n*m/N parcels to each free node. User scheduling is based on the pre-assigned usage share (quota) e.g. for 3 users we can have usage scores 50, 20 and 10 (total 80), which results in system usage percentage 50/80 = 62.5%, 20/80 = 25% and 10/80 = 12.5%. The real usage share may fluctuate to some extent. For example after the execution of some task we may have the real execution time for the tasks of the considered users 200s, 150 s and 50s. This leads to the real usage 50%, 37.5% and 12.5%, which is different from

366

J. Sosnowski, A. Tymoczko, and P. Gawkowski Users

cycle N

Id U

M

Δ

R

cycle N + 1 Δ cycle N + 2 Δ cycle N + 3 M

R

M

R

M

R

1 40

440s 44.0

440s 40.7

440s 38.2 +20s 460s 37.6

2 20

200s 20.0 +30s 230s 21.3

230s 20.0 +20s 250s 20.5

3 20

190s 19.0 +20s 210s 19.4 +40s 250s 21.7

4 14

130s 13.0 +5s

5

30s

5

3.0 +10s 40s

3.7

+20s 60s

6 1 10s 1.0 +7s  100 1000s 100.0 920s 85.0

11s

250s 20.5

5.2

60s

4.9

1.0* +20s 31s

2.5

991s 86.0

1051s 86.0

Fig. 1. Illustration of the task scheduling algorithm

the targeted one. The first user has some deficiency, the second one some surplus (excess) in the usage as compared with the targeted one. The used scheduling algorithm converges to assure the real usage close to the pre-assigned target. For this purpose for each new task assignment cycle we take into account parameter: scaled real usage = (usage score)*(real usage percentage)/(targeted usage percentage). So for the considered example we will get 50*50%/62.5%=40, 20*37.5%/25% = 30 and 10*12.5%/12.5% =10. The total scaled real usage is 40 + 30 + 10 = 80 (the targeted total usage is also 80). In the initial phase of the scheduling algorithm we have so called initial working list comprising users id’s and the corresponding assigned target usage scores. In fig. 1 we have 6 users with assigned scores: 40, 20, 20, 14, 5, and 1. In the N-th cycle of user selection we take into account the targeted usage score (U), the cumulated (from the beginning) processing time (M) of the specified user and the scaled real usage score (R). If in the row there is no entry it means that the related user has no tasks ready for execution. Between subsequent cycles we have the specification of the user additional processing time (Δ). This value is added to the processing time (M) in the N-th cycle and gives the usage time in cycle N+1. Users 4 and 6 finished their tasks and are not included in cycle N + 1. A new task of user 6 appears before cycle N + 2. In italic we specify parameters of users that were absent in the previous cycle (lack of ready tasks). Here we assume artificially the value M such that assures R=U (lack of history). This is denoted with * (user 6, cycle N+2). In cycle N all users have tasks for the execution. The users 2, 3, 4, 5 and 6 have no usage surplus (i.e. U ≥ R, bolded figures), so in this order they are included in the newly created list. In the N+1 cycle only users 1, 2, 3 and 5 have tasks for the execution. Among them 3 and 5 are selected due to usage deficiency. In cycle N+2 a new user reappears (6). In this cycle users 1, 2 and 6 will be selected. In cycle N + 3 users 1 and 5 will be selected. Hence, resuming we have obtained the following list of selected users in subsequent cycles: {2,3,4,5,6}, {3,5}, {1,2,6}, {1,5}. The effectiveness of the algorithm was verified in simulation experiments. For tasks with uniform

An Approach to Distributed Fault Injection Experiments

367

random distribution of execution times from the range <50s, 150s> the usage scores converged to the targeted ones after 20-30 scheduling cycles.

4

Testing and Evaluating System Performance

The developed system has been systematically tested in LAN environment covering several laboratories comprising various workstations. To simplify and automate configuration of nodes in the network we have developed DIDeployer and DIDeployManager applications. DIDeployer has to be installed once on every workstation previewed to be used in experiments. The system installer accesses DIDeployManager and selects appropriate text file comprising the list of used node addresses. This activates communication with DIDeployers in the specified nodes. Then the installer selects appropriate fault injection core (FIC) and initiates broadcasting it to the nodes. DIDeployer deinstalls the previous version of FIC, creates new file structures, installs and initiates the new FIC version. DIDeployManager provides also other services such as reading log files from the nodes, interrupting node operation, sending new configuration file. Testing the whole system needed some supplementary tools collecting operational logs in every node. In the developed logging system we can specify the required level of data collection (more or less detailed). Data is stored by invoking logging function by the application components. For the highest accuracy of logging we have observed 1GB of data generated by the DICoordinator logging subsystem per 24 hours (for 20 operating nodes). Each node (FIC application) generated 20-30 MB of logs per 24 hours. The high accuracy logging was needed in the first phases of the system testing. Later we selected rough logging. The lowest level of logging is restricted to error messages, decisions on selecting tasks for execution and moments of their termination. In this case the log volume increase for DICoordinator is about several MB per 24 hours. Higher levels of logging include issued SQL commands, image of task queues and many other specific events. Analysing events in the log we can identify incorrect system operation or proof its correctness. The system has been verified for many test scenarios involving various applications for fault injection and several users. For the executed tests we verified not only the final task results but also analysed the system operation and its performance. During the experiment we can monitor its progress, in particular the number of executed parcels, number of processed parcels, number of incorrectly executed parcels, distribution of executed tasks or parcels within nodes, task queues, etc. Moreover, we can visualise times of registering tasks in co-ordinators, time of initiating tasks execution, time to the execution deadline, time of staying in specified system states, execution time of messages, etc. The rich monitoring capabilities allowed us to eliminate system bottlenecks and optimise its performance. In the first phase of experiments many parcels were denoted as incorrect or many timeouts were generated. The first situation related mostly to existing bugs in the fault injector (systematically revealed and corrected). Timeouts in a large extent appeared during student laboratories and mostly resulted from

368

J. Sosnowski, A. Tymoczko, and P. Gawkowski

manual computer resetting. In the case of clean system shutdown in the node the operating system signalled this fact to the application. The application signalled this to DIManager, which delegated the remaining tasks to other nodes (instead of trying to repeat them in the reset node). The performed experiments proved the functionality of the system and its effectiveness. We illustrate this in tab. 1, which gives the experiment execution time of three applications (Qsort, LZW compression and HTML conversion). Parameter N specifies the number of computer nodes used for fault injection. All nodes were used only for the simulation experiment (not loaded with other applications). The results prove linear scalability, this effect was observed also for other applications. Some time fluctuation is related to system background processes, hence for LZW we have got higher speedup (S) than 10 for 10 nodes. It is worth noting that the distribution of test parcels processed by the nodes fluctuates due to some differences in the execution time of disturbed application, generated time-outs and automatic load balancing of nodes by the system. Tab. 2 shows some statistics for HTML conversion: the mean number of processed parcels (M) in nodes, standard deviation (D), as well as difference between maximal and minimal numbers (Δ). The presented results relate to 33%, 60% and 100% test progress respectively (100% means execution of all previewed tests). Table 1. Experiment (phase EX) execution time (in hours and minutes) and relative speed-up (S) N

Qsort - sorting

LZW compression

HTML conversion

1 10 28

30 h 15 min (S = 1) 3 h 05 min (S = 9.8) 1h 05 min (S = 27.9)

4 h 36 min (S = 1) 27 min (S = 10.2) 10.5 min (S = 26.2)

112 h 05 min (S = 1) 11 h 10 min (S = 9.96) 4 h 03 min (S = 27.8)

The impact of the assumed time-out limit (To) on the experiment execution time (TE) is not negligible. This is illustrated in tab.2 for HTML application (columns To and TE relate to complete tests - 100%). Here we show experiment execution time TE (in minutes) in function of the assumed time-out (To) in the nodes (for 28 nodes). We give the execution time for all tests and the execution time related to tests which generated time-outs (in brackets). For subsequent values of To: 1, 5 and 10 seconds the time-out tests contributed to the experiment execution time (TE) 19%, 55% and 71%, respectively. This is the reason for increasing TE. The experiments involved fault injections of single bit flip faults in the processor EAX register (pseudorandomely in time and space). The timeout should be selected carefully for the analysed application. Too long time-out slows down the experiment execution. On the other hand too short timeout may disturb obtaining valid test results. Optimising performance of the system we analysed the impact of the parcel size and distribution of system components in the network nodes. For the 100Mb/s transmission speed in the network the optimal parcel size is 10 tests.

An Approach to Distributed Fault Injection Experiments

369

Table 2. Statistics of experiment progress (phase EX) for HTML conversion Test

Processed parcels

Execution time

progress

M

D

Δ

To [s]

TE [min]

33% 60% 100%

100 149 297

6.22 6.84 8.56

23 26 39

1 5 10

65 (18) 135 (89) 234 (178)

Longer parcels may result in more losses due to unsuccessful simulation (and the need of repetitions of the whole parcel). To eliminate the bottleneck of the access to the database it is recommended to install DICoordinator and the MS SQL database server on separate computers. Performing tests for 3 applications Qsort, LZW compression and HTML conversion (over 2 million tests) on 26 nodes we obtained 17% speedup in this configuration (as compare with a single computer comprising DICoordinator and the database). In the optimisation process we used not only the event logging program but also standard performance monitor, which allowed us to trace transmissions, resource bottlenecks, system errors, etc. We also analysed the impact of other jobs executed in the computing nodes. Practically student laboratories on programming had negligible impact on simulation speed. Using special programs simulating CPU and disc loading showed that the high CPU loading results in a fair access to CPU processing power (close to 50% for simulation and 50% for the loading application), disc loading (continuous copying of files) had low impact (about 5% slow down of the simulation processes).

5

Conclusion

The main benefit of distributing fault injection experiments is speeding up their execution and performing them in a well-controlled environment. Moreover, it creates the possibility of multi-access as well as better usage of available computer resources. The developed system adapts to unpredictable fluctuations of the available computation power. Designing the system we optimised task management, data formats and communication overhead. High system effectiveness was proved in many experiments. Fault injection experiments fit well to distributed environment, howeve their specificity needed resolving some special problems. In particular the performed tests may result in node crashes. On the other hand running the experiments in parallel with other tasks (e.g. student laboratories) created some risks with node reloading, restarting, operating system changes etc. So it was required to embed special mechanisms dealing with these problems. This was quite challenging problem which required extensive studies of various situations covering different simulation tasks schemes, node loading and other users interactions. Hence, in the testing and validation phases we used specially developed monitoring mechanisms

370

J. Sosnowski, A. Tymoczko, and P. Gawkowski

related to collecting various task activity reports as well as some standard performance parameters. The developed distributed manager is quite universal and we plan to use it for other fault injectors, which are being developed. Moreover, it will be directly connected to our new data warehouse to facilitate exploring test results [4,13].

References 1. Arlat, J., et al.: Comparison of physical and software implemented fault injection technique. IEEE Trans. on Computers 52(8), 1115–1133 (2003) 2. Benso, A., Prinetto, P.: Fault injection techniques and tools for embedded systems reliability evaluation. Kluwer Academic Publishers, Dordrecht (2003) 3. Gawkowski, P., Sosnowski, J., Radko, B.: Analyzing the effectiveness of fault hardening procedures. In: Proc. of the 11th IEEE Int’l On-Line Testing Symp., pp. 14–19 (2005) 4. Gawkowski, P., Sosnowski, J.: Experiences with software implemented fault injection. In: Int. Conf. on Architecture of Computing Systems, Workshop Proceedings, pp. 73–80. VDE Verlag GMBH (2007) 5. Sosnowski, J., Gawkowski, P., Lesiak, A.: Software implemented fault inserters. In: Proc. of IFAC PDS 2003 Workshop, Pergamon, pp. 293–298 (2003) 6. Tanenbaum,, van Steen, M.: Distributed systems, principles and paradigms. Prentice Hall, Englewood Cliffs (2002) 7. Herruzo, E., et al.: Distributed Architecture system for computing performance testing. In: Wyrzykowski, R., Dongarra, J., Meyer, N., Wa´sniewski, J. (eds.) PPAM 2005. LNCS, vol. 3911, pp. 140–147. Springer, Heidelberg (2006) 8. Bonorden, O., et al.: Load balancing strategies in a web computing environment. In: Wyrzykowski, R., Dongarra, J., Meyer, N., Wa´sniewski, J. (eds.) PPAM 2005. LNCS, vol. 3911, pp. 839–846. Springer, Heidelberg (2006) 9. Wagner, A.S., et al.: Performnace models for the processor farm paradigm. IEEE Trans. on Parallel and Distributed Systems 5(5), 475–486 (1997) 10. Xiao, L., et al.: Dynamic cluster resource allocations for jobs with known and unknown memory demands. IEEE Trans. on Parallel and Distributed Systems 13(3), 223–240 (2002) 11. Bosque, J.L., Pastor, L.: A parallel computational model for heterogeneous clusters. IEEE Trans. on Parallel and Distributed Systems 17(12), 1390–1400 (2006) 12. Dhakal, S., et al.: Dynamic load balancing in distributed systems in the presence of delays. IEEE Trans. on Parallel and Distributed Systems 18(4), 485–497 (2007) 13. Sosnowski, J., et al.: Enhancing fault injection testbench. In: Proc. of Int. Conf. on Depenability of Comp. Systems, DepCoS, pp. 76–83 (2006)

Parallel Solution of Nonlinear Parabolic Problems on Logically Rectangular Grids Andrs Arrar´ as, Laura Portero, and Juan Carlos Jorge Departamento de Ingenier´ıa Matem´ atica e Inform´ atica Edificio Las Encinas, Universidad P´ ublica de Navarra Campus de Arrosad´ıa s/n, 31006 Pamplona (Spain) {andres.arraras,laura.portero,jcjorge}@unavarra.es

Abstract. This work deals with the efficient numerical solution of nonlinear transient flow problems posed on two-dimensional porous media of general geometry. We first consider a spatial semidiscretization of such problems by using a cell-centered finite difference scheme on a logically rectangular grid. The resulting nonlinear stiff initial-value problems are then integrated in time by means of a fractional step method, combined with a decomposition of the flow domain into a set of overlapping subdomains and a linearization procedure which involves suitable Taylor expansions. The proposed algorithm reduces the original problem to the solution of several linear systems per time step. Moreover, each one of such systems can be directly decomposed into a set of uncoupled linear subsystems which can be solved in parallel. A numerical example illustrates the unconditionally convergent behaviour of the method in the last section of the paper. Keywords: Domain Decomposition, Fractional Step Method, Linearly Implicit Method, Logically Rectangular Grid, Nonlinear Parabolic Problem, Support-Operator Method.

1

Introduction

Darcian water flow through non-swelling isothermal soils has been shown to obey Richards’ equation [5,2]. A simplified version of such equation, together with suitable initial and boundary conditions, gives rise to nonlinear parabolic initial-boundary value problems of the following form: Find ψ : Ω × [0, T ] → R such that ⎧ ∂ψ(x, t) ⎪ ⎪ = div(K(ψ) grad ψ(x, t)) + g(ψ) + f (x, t), (x, t) ∈ Ω × (0, T ], ⎪ ⎨ ∂t ψ(x, 0) = ψ0 (x), x ∈ Ω, ⎪ ⎪ ⎪ ⎩ ψ(x, t) = ψD (x, t), (x, t) ∈ ∂Ω × (0, T ], (1) 

This research is partially supported by the Spanish Ministry of Science and Education under Research Project MTM2004-05221 and FPU Grant AP2003-2621 and by Government of Navarre under Research Project CTP-05/R-8.

R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 371–380, 2008. c Springer-Verlag Berlin Heidelberg 2008 

372

A. Arrar´ as, L. Portero, and J.C. Jorge

where ψ(x, t) denotes the pressure head, K(ψ) is a symmetric positive-definite tensor of the form  11  K (ψ) K 12 (ψ) K(ψ) = , K 12 (ψ) K 22 (ψ) which represents nonlinear hydraulic conductivity, g(ψ) is a smooth nonlinear function which may describe, for instance, root water uptake in soil profiles (cf. [10]) and f (x, t) is a source/sink term. The flow domain is assumed to be a bounded open set Ω ⊆ R2 with boundary ∂Ω, where we have considered Dirichlet boundary conditions ψD (x, t). This paper is devoted to the design of an efficient numerical algorithm for solving problems of type (1). The construction of such scheme is carried out by using a two-stage discretization process (space/time), described in sections 2 and 3. The last section includes a numerical experiment which illustrates the main advantages of the proposed method.

2

Spatial Semidiscretization

The semidiscrete scheme related to (1) is obtained by using a finite difference spatial discretization based on the support-operator method. Such technique, initially developed in [6] and subsequently discussed in [7], provides a methodology for constructing discrete analogs of invariant first-order differential operators which appear in (1) (i.e. divergence and gradient). The standard support-operator method was designed for that case in which the self-adjoint operator div(K(x) grad ψ) is linear. However, the nonlinear nature of the conductivity tensor K(ψ) involved in Richards’ equation makes it necessary to combine the original technique with a bivariate interpolation method. This section describes the general basis of the discretization scheme, introducing the specifics of our proposal which permit to solve problem (1). Let us first discretize Ω by means of a logically rectangular grid Ωh , where h denotes the spatial mesh size. The structure of such grid is indexed as follows: if Nx and Ny are positive integers, then the (i, j)-node is given by the coordinates (˜ xi,j , y˜i,j ), for i ∈ {1, 2, . . . , Nx } and j ∈ {1, 2, . . . , Ny }. Moreover, the quadrangle defined by the nodes (i, j), (i + 1, j), (i, j + 1) and (i + 1, j + 1) is called the (i, j)-cell and its center is given by the coordinates (xi,j , yi,j ), which can be obtained as xi,j = 0.25 (˜ xi,j + x ˜i+1,j + x ˜i,j+1 + x ˜i+1,j+1 ), yi,j = 0.25 (˜ yi,j + y˜i+1,j + y˜i,j+1 + y˜i+1,j+1 ), for i ∈ {1, 2, . . . , Nx − 1} and j ∈ {1, 2, . . . , Ny − 1}. Within this framework, the support-operator method considers cell-centered approximations for scalar functions ψ(x, t), g(ψ) and f (x, t) denoted by ψh (t), gh (ψh ) and fh (t), respectively. On the other hand, vector functions w(x, t) ≡ ˜ h (t) ≡ (w (wx (x, t), wy (x, t)) are nodally discretized by means of w ˜hx (t), w ˜hy (t)).

Parallel Solution of Nonlinear Parabolic Problems

373

As described in [9], it is natural to use the divergence as the first-order prime operator for this approximation scheme. Based on the invariant definition of the divergence, we can derive a discrete analog divh of div such that divh :

V˜h × V˜h ˜h v

→ →

Vh ˜h, divh v

˜ h ≡ (˜ where v vhx , v˜hy ). Both V˜h and Vh are finite-dimensional spaces of discrete functions defined on the nodes and the cell-centers of Ωh , respectively. Hence, ˜ h (t) the expression of the discrete divergence acting on a semidiscrete vector w has the following form ˜ h (t))i,j = (divh w

1 (((w˜hx )i+1,j+1 − (w ˜hx )i,j )(˜ yi,j+1 − y˜i+1,j ) 2σi,j −((w ˜hx )i,j+1 − (w ˜hx )i+1,j )(˜ yi+1,j+1 − y˜i,j ) −((w ˜hy )i+1,j+1 − (w ˜hy )i,j )(˜ xi,j+1 − x ˜i+1,j )

(2)

−((w ˜hy )i,j+1 − (w ˜hy )i+1,j )(˜ xi+1,j+1 − x ˜i,j )), for i ∈ {1, 2, . . . , Nx − 1} and j ∈ {1, 2, . . . , Ny − 1}, where σi,j is the area of the (i, j)-cell and (w ˜hz )i,j denotes the ((i − 1)(Ny − 1) + j)-th component of w ˜hz (t) z which approximates w (xi,j , yi,j , t) for z = x, y. Considering a discrete version of Gauss’ theorem, together with equation (2), we shall construct the derived operator gr adh as the discrete analog of grad as follows gr adh : Vh → V˜h × V˜h uh → gr adh uh . The components of the discrete gradient acting on the semidiscrete function ψh (t) can be obtained as (gr adxh ψh (t))i,j =

1 ((˜ yi,j+1 − y˜i+1,j ) (ψh )i,j + (˜ yi−1,j − y˜i,j+1 ) (ψh )i−1,j 2ηi,j +(˜ yi+1,j − y˜i,j−1 ) (ψh )i,j−1 + (˜ yi,j−1 − y˜i−1,j ) (ψh )i−1,j−1 ),

(gr adyh ψh (t))i,j =

1 ((˜ xi,j+1 − x ˜i+1,j ) (ψh )i,j + (˜ xi−1,j − x˜i,j+1 ) (ψh )i−1,j 2ηi,j +(˜ xi+1,j − x˜i,j−1 ) (ψh )i,j−1 + (˜ xi,j−1 − x ˜i−1,j ) (ψh )i−1,j−1 ),

(3) for the internal values i ∈ {2, . . . , Nx − 1} and j ∈ {2, . . . , Ny − 1}, where ηi,j = 0.25 (σi,j +σi−1,j +σi,j−1 +σi−1,j−1 ) and (ψh )i,j denotes the ((i−1)(Ny −1)+j)th component of ψh (t) which approximates ψ(xi,j , yi,j , t). It is easy to see that the previous equations can be extended to the boundaries if we introduce the following fictitious nodes x ˜i,0 = x˜i,1 , x˜0,j = x˜1,j ,

y˜i,0 = y˜i,1 , y˜0,j = y˜1,j ,

x˜i,Ny +1 = x˜i,Ny , x ˜Nx +1,j = x˜Nx ,j ,

y˜i,Ny +1 = y˜i,Ny , y˜Nx +1,j = y˜Nx ,j ,

374

A. Arrar´ as, L. Portero, and J.C. Jorge

for i ∈ {1, 2, . . . , Nx } and j ∈ {1, 2, . . . , Ny }, as well as the evaluations of the Dirichlet boundary condition ψD (x, t) at the centers of the boundary segments, i.e. (ψh )0,j = ψD (ˆ x1,j , yˆ1,j , t),

(ψh )Nx ,j = ψD (ˆ xNx ,j , yˆNx ,j , t),

where x ˆ1,j = 0.5 (˜ x1,j + x ˜1,j+1 ), yˆ1,j = 0.5 (˜ y1,j + y˜1,j+1 ), x ˆNx ,j = 0.5 (˜ xNx ,j + x ˜Nx ,j+1 ) and yˆNx ,j = 0.5 (˜ yNx,j + y˜Nx ,j+1 ), for j ∈ {1, 2, . . . , Ny − 1}, and (ψh )i,0 = ψD (ˆ xi,1 , yˆi,1 , t),

(ψh )i,Ny = ψD (ˆ xi,Ny , yˆi,Ny , t),

where xˆi,1 = 0.5 (˜ xi,1 + x˜i+1,1 ), yˆi,1 = 0.5 (˜ yi,1 + y˜i+1,1 ), x ˆi,Ny = 0.5 (˜ xi,Ny + x ˜i+1,Ny ) and yˆi,Ny = 0.5 (˜ yi,Ny + y˜i+1,Ny ), for i ∈ {1, 2, . . . , Nx − 1}. Let us next proceed to explain the spatial discretization for tensor K(ψ). In the linear case studied in [9] (i.e. when K ≡ K(x) does not depend on ψ), the discrete equations (3) together with the nodal evaluations of the components of ˜ 11 )i,j , (K ˜ 12 )i,j and (K ˜ 22 )i,j , lead us to K, denoted by (K h h h ⎛ ⎞ x 11 ˜ )i,j (gr ˜ 12 )i,j (gr (K adh ψh (t))i,j + (K adyh ψh (t))i,j h h ˜ h gr ⎠. (K adh ψh (t))i,j = ⎝ ˜ 12 ˜ 22 )i,j (gr (K )i,j (gr adx ψh (t))i,j + (K ady ψh (t))i,j h

h

h

h

(4) Now, using (2)-(4), it is straightforward to obtain the discrete linear opera˜ h gr ˜ h gr tor divh (K adh ). In this case, the local stencil for (divh (K adh ψh ))i,j involves the cell-centered approximations (ψh )i−1,j−1 , (ψh )i,j−1 , (ψh )i+1,j−1 , (ψh )i−1,j , (ψh )i,j , (ψh )i+1,j , (ψh )i−1,j+1 , (ψh )i,j+1 and (ψh )i+1,j+1 , as well as the evaluations of the components of K at the nodes (i, j), (i + 1, j), (i, j + 1) and (i + 1, j + 1). In the nonlinear case, the nodal discretization of K(ψ) grad ψ is similar to the one given by (4). However, as the conductiviy tensor depends on the unknown ψ, its discrete analog will involve approximations of such unknown at the grid nodes. Let us denote these approximations by ψ˜h . Now, combining such discretization with equations (2) and (3), we shall obtain an approxima˜ h (ψ˜h ) gr tion for div(K(ψ) grad ψ). The local stencil for (divh (K adh ψh ))i,j involves, as in the linear case, the cell-centered approximations (ψh )i−1,j−1 , (ψh )i,j−1 , (ψh )i+1,j−1 , (ψh )i−1,j , (ψh )i,j , (ψh )i+1,j , (ψh )i−1,j+1 , (ψh )i,j+1 and (ψh )i+1,j+1 ; moreover, it includes the nodal approximations given by (ψ˜h )i,j , (ψ˜h )i+1,j , (ψ˜h )i,j+1 and (ψ˜h )i+1,j+1 . Such values will be obtained by means of a bivariate interpolation method as linear combinations of the nine values of ψh (t) at the cell centers, i.e. (ψ˜h )i,j =

1

ci,j k, (ψh )i+k,j+ ,

k,=−1 1

(ψ˜h )i,j+1 =

k,=−1

ci,j+1 (ψh )i+k,j+ , k,

(ψ˜h )i+1,j =

1

ci+1,j (ψh )i+k,j+ , k,

k,=−1 1

(ψ˜h )i+1,j+1 =

ci+1,j+1 (ψh )i+k,j+ . k,

k,=−1

Figure 1 shows the structure of the local nine-cell stencil corresponding to ˜ h (ψh ) gr (divh (K adh ψh ))i,j .

Parallel Solution of Nonlinear Parabolic Problems

(ψh )i−1,j+1

(ψh )i,j+1

(ψh )i+1,j+1

(i, j + 1)

(i + 1, j + 1)

(ψh )i,j

(ψh )i−1,j

(ψh )i+1,j (i + 1, j)

(i, j)

(ψh )i−1,j−1

375

(ψh )i+1,j−1

(ψh )i,j−1

˜ h (ψh ) gr Fig. 1. Nine-cell stencil for (divh (K adh ψh ))i,j

Using this discretization for the diffusion term, we can approach the original problem by solving a nonlinear stiff initial-value problem of the form: Find ψh : [0, T ] → Vh such that ⎧ ⎨ dψh (t) = div (K ˜ h (ψh ) gr adh ψh (t)) + gh (ψh ) + fh (t), t ∈ (0, T ], h dt (5) ⎩ ψh (0) = rh (ψ0 ) = ψ0h , where rh denotes the restriction to the cell centers of Ωh .

3

Time Integration

Let us consider Ω decomposed into the union of m overlapping subdomains, where each one of them consists of a certain number of disjoint connected components, i.e. Ω=

m 

Ωi , where Ωi =

i=1

mi 

Ωij such that Ωij ∩ Ωik = ∅ if j = k.

j=1

Next, we define a smooth partition of unity consisting of m functions {ρi (x)}m i=1 , where each function ρi : Ω → [0, 1] is defined as follows ⎧ 0, if x ∈ Ω \ Ωi , ⎪ ⎪ ⎪ ⎪ m ⎪  ⎪ ⎪ ⎨ hi (x), if x ∈ (Ωi ∩ Ωj ), ρi (x) = j=1 ⎪ ⎪ j =i m ⎪  ⎪ ⎪ ⎪ 1, if x ∈ Ω \ (Ωi ∩ Ωj ), i ⎪ ⎩ j=1 j =i

for 0 ≤ hi (x) ≤ 1 and

m

i=1

hi (x) = 1 ∀ x ∈

m  j=1 j =i

(Ωi ∩ Ωj ).

376

A. Arrar´ as, L. Portero, and J.C. Jorge

By using this partition of unity, we shall define the following splittings for the ˜ h (·) gr nonlinear discrete operator Ah (·) ≡ divh (K adh ·) and the semidiscrete function fh (t) (cf. [3])

Ah (·) =

m

Ai,h (·),

i=1

fh (t) =

m

fi,h (t).

(6)

i=1

˜ i,h (·) gr ˜ i,h (ψh ) gr Here, we denote Ai,h (·) ≡ divh (K adh ·), considering K adh ψh as the discretization of Ki (x, ψ) grad ψ, where Ki (x, ψ) ≡ ρi (x)K(ψ). On the other hand, fi,h (t) ≡ rh (ρi (x) f (x, t)). Considering the splittings given by (6), the variant of the fractional implicit Euler method with m internal stages introduced in [1] reduces the nonlinear stiff problem (5) to the following set of nonlinear systems (one per internal stage) ⎧ ψh,0 = ψ0h , ⎪ ⎪ ⎪ ⎪ k ⎪

⎪   ⎪  k ⎪ ⎪ ψ = ψ + τ A,h (ψh,n ) + f,h (tn+1 ) + τ gh (ψh,n ), h,n ⎪ h,n ⎨ =1

for k ∈ {1, 2, . . . , m}, ⎪ ⎪ ⎪ ⎪ ⎪ m ⎪ ⎪ ψh,n+1 = ψh,n , ⎪ ⎪ ⎪ ⎩ for n ∈ {0, 1, . . . , NT },

(7)

where NT ≡ [T /τ ] − 1. The discrete solution ψh,n+1 approximates ψh (tn+1 ), where tn+1 = (n + 1) τ and τ denotes the constant time step. The choice of the fractional implicit Euler scheme is motivated by the fact that this method is stable even when combined with an operator splitting which considers an arbitrary number of terms that do not necessarily commute (cf. [4]). This is the case for the discrete operators Ai,h (·) involved in (6). Note that (7) also entails an explicit treatment of the nonlinear discrete function gh (ψh,n ).  In order to linearize (7), we approximate A,h (ψh,n ) by the two first terms of its Taylor expansion around ψh,n , i.e.  A,h (ψh,n )

 dA,h (ψh )   ≈ A,h (ψh,n ) + (ψh,n − ψh,n ).  dψh ψh =ψh,n

(8)

If we denote by (ψh,n )i,j the ((i − 1)(Ny − 1) + j)-th component of ψh,n , for i ∈ {1, 2, . . . , Nx − 1} and j ∈ {1, 2, . . . , Ny − 1}, then (A,h (ψh,n ))i,j depends nonlinearly on nine unknowns: (ψh,n )i−1,j−1 , (ψh,n )i,j−1 , (ψh,n )i+1,j−1 , (ψh,n )i−1,j , (ψh,n )i,j , (ψh,n )i+1,j , (ψh,n )i−1,j+1 , (ψh,n )i,j+1 and (ψh,n )i+1,j+1 . Therefore, the ((i − 1)(Ny − 1) + j)-th row of the Jacobian matrix J,h (ψh,n ) ≡ dA,h (ψh )/dψh |ψh =ψh,n will contain nine non-zero elements representing the derivatives of (A,h (ψh,n ))i,j with respect to each one of the previous unknowns.

Parallel Solution of Nonlinear Parabolic Problems

377

Inserting (8) into (7), we obtain the totally discrete scheme for problem (1) ⎧ ˇ ⎪ ⎪ ψh,0 = ψ0h , ⎪ ⎪ k−1 ⎪

 ⎪  ⎪  ⎪ ˇh,n ) ψˇk = ψˇh,n +τ I − τ J ( ψ A,h (ψˇh,n )+J,h (ψˇh,n )(ψˇh,n − ψˇh,n ) ⎪ k,h h,n ⎪ ⎪ ⎪ =1 ⎪   ⎪ ⎨ +f,h (tn+1 ))+τ Ak,h (ψˇh,n ) − Jk,h (ψˇh,n ) ψˇh,n +fk,h (tn+1 ) +τ gh (ψˇh,n ), ⎪ ⎪ ⎪ for k ∈ {1, 2, . . . , m}, ⎪ ⎪ ⎪ ⎪ ⎪ m ⎪ ⎪ ψˇh,n+1 = ψˇh,n , ⎪ ⎪ ⎪ ⎪ ⎩ for n ∈ {0, 1, . . . , NT }, (9) where ψˇh,n+1 is an approximation of ψh (tn+1 ) which preserves the same order of accuracy as ψh,n+1 . Note that the domain decomposition splitting chosen for Ah (·) makes each internal stage consist of a linear system which involves the unknowns lying just on one of the subdomains {Ωi }m i=1 . Moreover, since each subdomain Ωi comprises mi disjoint connected components, this system can be immediately decomposed into mi uncoupled subsystems which allow a straightforward parallelization. For those points lying outside subdomain Ωi , we have that Ji,h (ψˇh,n ) ≡ 0 (remember that ρi (x) = 0 if x ∈ Ω \ Ωi ) and so, in this case, the solution of the i-th internal stage in (9) simply requires an explicit evaluation of the right-hand side. As a difference with respect to classical domain decomposition methods, artificial boundary conditions are not required on each subdomain and, hence, no Schwarz iterative procedures are needed in the computation. Finally, the uncoupled linear subsystems arising at each internal stage are solved by the Gauss-Seidel method. In the case when K(ψ) is a diagonal tensor and Ω is a rectangular domain discretized by means of a rectangular grid Ωh , we can derive efficient algorithms for the solution of (1) by combining a finite difference spatial discretization with the time integration procedure described in this section. In such a case, we have two options for decomposing operator Ah (·): the domain decomposition splitting explained before or a classical alternating direction splitting (cf. [1]). In the latter type, we obtain an essentially one-dimensional linear system at each internal stage which is tridiagonal and can be easily decomposed into several subsystems whose solution may be parallelized.

4

Numerical Experiment

In this section, we test the behaviour of the numerical algorithm on a set of pseudo-random logically rectangular grids. A similar test is shown in [8] for a classical implicit Euler scheme, combined with the support-operator technique, in the solution of linear parabolic problems.

A. Arrar´ as, L. Portero, and J.C. Jorge 1

1

0.9

0.9

0.8

0.8

0.8

0.7

0.7

0.7

0.6

0.6

0.6

0.5

0.5

0.5

y

1 0.9

y

y

378

0.4

0.4

0.4

0.3

0.3

0.3

0.2

0.2

0.2

0.1

0.1

0

0

0

0.2

0.4

0.6

0.8

x

(a) N = 17

1

0.1 0

0.2

0.4

0.6

0.8

x

(b) N = 33

1

0

0

0.2

0.4

0.6

0.8

1

x

(c) N = 65

Fig. 2. Pseudo-random logically rectangular grids described in the numerical experiment

Let us consider an equation of type (1) posed on Ω × (0, T ] ≡ {x = (x, y) ∈ R2 : 0 < x < 1, 0 < y < 1} × (0, 0.01]. The hydraulic conductivity K(ψ) is a full nonlinear tensor defined as K(ψ) = Q(θ) D(ψ) Q(θ)T , where Q(θ) is a 2 × 2 rotation matrix with angle θ = π/4 and D(ψ) is a 2 × 2 diagonal matrix whose diagonal entries are 1 + ψ 2 and 1 + 8ψ 2 . The nonlinear function is chosen to be g(ψ) = 1/(1 + ψ 3 ), whereas the source/sink term f (x, y, t) and both initial and Dirichlet boundary conditions are defined in such a way that 2 ψ(x, y, t) = e−2π t sin(πx) sin(πy) is the exact solution of the problem. The spatial semidiscretization is based on the finite difference method described in section 2. The flow domain Ω is first discretized by means of a pseudo-random logically rectangular grid Ωh ≡ {(xi,j , yi,j )}N i,j=1 with coordinates xi,j = (i − 1) h − 0.25 h + 0.5 h Rx and yi,j = (j − 1) h − 0.25 h + 0.5 h Ry , where h = 1/(N − 1) and Rx , Ry are random numbers generated on the interval (0, 1). Figure 2(a) shows an example of such type of grids for N = 17. In order to study the asymptotic behaviour of the error, we successively refine the original pseudo-random grid by using the following procedure: starting from a given grid, we add the lines which connect, on each cell, the centers of the opposite sides. Figures 2(b) and 2(c) show the first two refinements for the grid displayed on Figure 2(a). Let us now consider a decomposition of Ω into m = 4 overlapping subdomains {Ωi }m i=1 , each of which consists of mi = 4 disjoint  1connected   1 components,  for 3 i ∈ {1, 2, 3, 4}. In particular, if we denote I ≡ 0, + d ∪ − d, + d and 1 4 2 4     I2 ≡ 14 − d, 12 + d ∪ 34 − d, 1 , the four subdomains are given by Ω1 ≡ I1 × I1 , Ω2 ≡ I2 × I1 , Ω3 ≡ I1 × I2 and Ω4 ≡ I2 × I2 . Note that the width of the overlapping regions is 2d, where d is chosen to be 1/16. Next, we define a smooth partition of unity consisting of four functions {ρi (x)}4i=1 which are related to the previous domain decomposition. For that purpose, we start by introducing  ⎞ ⎛ −1 exp x−x 0 +d ⎠, h(x, x0 , d) = exp ⎝d exp(1/d) log(2) x − x0 − d

Parallel Solution of Nonlinear Parabolic Problems

379

Table 1. Global errors and numerical orders of convergence for N = 129 τ

τ0 = 10−3

τ0 /2

τ0 /4

τ0 /8

τ0 /16

τ0 /32

EN,τ pN,τ

3.703 E-2 0.5230

2.577 E-2 0.6759

1.613 E-2 0.7766

9.416 E-3 0.8439

5.246 E-3 0.8919

2.827 E-3 −

Table 2. Global errors and numerical orders of convergence for τ = 10−7 N EN,τ pN,τ

17

33

65

129

257

5.312 E-3 1.4667

1.922 E-3 2.0340

4.693 E-4 2.0077

1.167 E-4 2.0122

2.893 E-5 −

which is subsequently used to ⎧ 1, ⎪ ⎪ ⎪ ⎪ ⎨ 0, i1 (x) = ⎪ h(x, α, d), ⎪ ⎪ ⎪ ⎩ 1 − h(x, 12 , d),

define the functions     if x ∈ 0, 14 − d ∪ 12 + d, 34 − d ,     if x ∈ 14 + d, 12 − d ∪ 34 + d, 1 , if x ∈ [α − d, α + d] with α = 14 , 34 ,   if x ∈ 12 − d, 12 + d

and i2 (x) = 1 − i1 (x). Finally, if we consider suitable products of i1 (x) and i2 (x), we can construct the following non-negative C ∞ -functions: ρ1 (x, y) = i1 (x) i1 (y), ρ2 (x, y) = i2 (x) i1 (y), ρ3 (x, y) = i1 (x) i2 (y) and ρ4 (x, y) = i2 (x) i2 (y). In order to obtain the totally discrete scheme (9), we use the fractional step method given by (7), with four internal stages (m = 4), in combination with the linearization procedure described in (8). The solution of our numerical scheme provides vectors ψh,n ∈ R(N −1)×(N −1) , for n = 1, 2, . . . , NT + 1, whose components are approximations to the exact solution ψ(x, tn ) at the cell centers of Ωh . Owing to the domain decomposition splitting considered for Ah (·), the linear system obtained at each internal stage reduces to a set of four smaller uncoupled subsystems which can be easily solved in parallel. Therefore, the number of unknowns involved in the computation will decrease from (N − 1) × (N − 1) to a number between n1 × n1 and n2 × n2 , where n1 = (N − 1)(1/4 + d) and n2 = (N −1)(1/4+2d). From a practical point of view, as the amount of available processors increases, each subdomain can be decomposed into a greater number of disjoint components in order to reduce the actual execution time. Finally, we include two tables which contain the global errors, EN,τ (upper row), and numerical orders of convergence, pN,τ (lower row), obtained for different values of N and τ when using the maximum norm in time and the L2 -norm in space, i.e. · L∞(0,T ;L2 (Ω)) . The method shows unconditional convergence of first order in time (see Table 1) and second order in space (see Table 2).

380

A. Arrar´ as, L. Portero, and J.C. Jorge

References 1. Arrar´ as, A., Jorge, J.C.: An alternating-direction finite difference method for threedimensional flow in unsaturated porous media. Mathematical Modelling and Analysis, 57–64 (2005) 2. Celia, M.A., Bouloutas, E.T., Zarba, R.L.: A general mass-conservative numerical solution for the unsaturated flow equation. Water Resour. Res. 26, 1483–1496 (1990) 3. Portero, L., Bujanda, B., Jorge, J.C.: A combined fractional step domain decomposition method for the numerical integration of parabolic problems. In: Wyrzykowski, R., Dongarra, J., Paprzycki, M., Wa´sniewski, J. (eds.) PPAM 2004. LNCS, vol. 3019, pp. 1034–1041. Springer, Heidelberg (2004) 4. Portero, L.: Fractional Step Runge-Kutta Methods for Multidimensional Evolutionary Problems with Time-Dependent Coefficients and Boundary Data. Ph.D. Thesis, Universidad P´ ublica de Navarra, Pamplona (2007) 5. Richards, L.A.: Capillary conduction of liquids through porous mediums. Physics 1, 318–333 (1931) 6. Samarski˘ı, A., Tishkin, V., Favorski˘ı, A., Shashkov, M.: Operational finitedifference schemes. Differ. Equ. 17, 854–862 (1981) 7. Shashkov, M.: Conservative Finite-Difference Methods on General Grids. CRC Press, Boca Raton (1996) 8. Shashkov, M., Steinberg, S.: Solving diffusion equations with rough coefficients in rough grids. J. Comput. Phys. 129, 383–405 (1996) 9. Shashkov, M., Steinberg, S.: The numerical solution of diffusion problems in strongly heterogeneous non-isotropic materials. J. Comput. Phys. 132, 130–148 (1997) ˘ 10. Simunek, J., Hopmans, J.W., Vrugt, J.A., van Wijt, M.T.: One-, two- and threedimensional root water uptake functions for transient modeling. Water Resour. Res. 37, 2457–2470 (2001)

Provenance Tracking in the ViroLab Virtual Laboratory Bartosz Bali´s1,2 , Marian Bubak1,2, and Jakub Wach2 1

Institute of Computer Science, AGH, Poland {balis,bubak}@agh.edu.pl, [email protected] 2 Academic Computer Centre – CYFRONET, Poland

Abstract. Provenance describes the process which led to the creation of a piece of data. Tracking provenance of experiment results is essential in modern environments which support conducting of in silico experiments. We present a provenance tracking approach developed as part of the virtual laboratory of the ViroLab project. The applied provenance solution is motivated by the Semantic Grid vision as an infrastructure for e-Science. Provenance data is represented in XML and modeled as ontologies described in the OWL knowledge representation language. The provenance tracking system, PROToS, has been designed and implemented to address important stages of the knowledge management lifecycle. Keywords: e-Science, Grid, ontology, provenance, ViroLab.

1 Introduction The term ‘e-Science’ [6] was introduced by John Taylor to denote a new type of scientific research based on the collaboration within a number of scientific areas, enabled by a next generation infrastructure. The infrastructure in question is usually identified with Grid systems which offer at least two benefits important for loosely-coupled crossinstitution research and collaboration: the virtualization and sharing of resources [7], and building of virtual organizations [3]. Several initiatives have emerged to develop existing Grid infrastructures towards higher-level functionalities useful for scientists. Examples of such initiatives are myGrid [9], Taverna [8] and Triana [12]. Sometimes the term virtual laboratory is used to denote an environment that supports conducting in silico experiments. Provenance of a piece of data is defined as a process that led to that data [5], also known as a derivation path of a piece of data. Provenance is essential for a scientist as it provides the information on how the particular result was obtained which might be equally important as the result itself. The scientist would use the provenance data for various purposes, for example to assess the quality of the result, to repeat whole or part of the experiment, to validate the results, etc. [4]. The goal of ViroLab1 [11] is to set up a virtual organization and develop a virtual laboratory2 for infectious diseases in order to support medical doctors and scientists in 

1 2

This work is supported by the European Union through the IST-027446 project ViroLab and by the Foundation for Polish Science within the Domestic Grant for Young Scientists. http://www.virolab.org http://virolab.cyfronet.pl

R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 381–390, 2008. c Springer-Verlag Berlin Heidelberg 2008 

382

B. Bali´s, M. Bubak, and J. Wach

their daily work. The primary scenario that drives ViroLab is the HIV drug resistance. Medical doctors are provided with a decision support system which helps them rank drugs for particular patients. Virologists can use data integrated from a number of institutions, and various tools, virtualized and perhaps composed into scripts (similar to workflows), to perform data mining, statistical analysis, as well as molecular dynamics and cellular automata simulations. This paper presents the provenance tracking approach in the ViroLab Project. We describe our motivation (Section 2), the provenance model in ViroLab (Section 3), and the architecture of our provenance tracking system – PROToS (Section 4). Section 5 presents the implementation of PROToS in more detail. Section 6 contains an overview of related work.

2 Motivation Our motivation in designing the provenance solution was not only to support recording and browsing of provenance logs but also to enable complex queries over provenance data that can help find interesting properties. Examples of useful queries are as follows: – Which rule sets were used most frequently to obtain drug rankings? – Return all experiments performed by John Doe within last two weeks. – Return all input sequences of experiments of type Drug Ranking Support whose input parameter threshold was between 0.03 and 0.04. Our work is also driven by the vision of semantic grid as a future infrastructure for e-Science, as presented in [10]. In this article, three types of services – data services, information services, and knowledge services – are introduced, where data is understood as an uninterpreted sequence of bits (e.g. an integer value), information is data associated with meaning (e.g. a temperature), while knowledge is understood as “information applied to achieve a goal, solve a problem or enact a decision”. Provenance is important part of an e-Science infrastructure providing both information and knowledge services. However, an adequate knowledge management has several aspects concerning various stages of knowledge lifecycle: acquisition, modelling, retrieval, reuse, publishing and maintenance [10]. Consequently, our aim is to design provenance data model and representation as well as the provenance tracking system with those requirements in mind. With the provenance service built as a knowledge service, its use can go beyond the simple usage as a personal scientist’s logbook documenting the experiments. Sufficiently rich information stored in provenance records can enable other uses. For example, records of multipe invocations of particular services equipped with timing information can easily be used for instance-based learning to optimize future resource brokering or scheduling decisions. In the future, we anticipate the integration with the ViroLab’s brokering/scheduling component for this purpose.

3 Applied Provenance Model Fig. 1 presents a simplified architecture of the ViroLab’s virtual laboratory. The virtual laboratory provides an Experiment Planning Environment (EPE) which allows to develop experiment scripts which can be subsequently executed for different input data.

Provenance Tracking in the ViroLab Virtual Laboratory

383

Both data and computation are virtualized as so-called Grid Objects. A unified data layer (accessible through Data Access Client, DAC) provides data integration over multiple data sources. Grid Objects represent services with one or more methods which can be programatically accessed from scripts in a uniform way, regardless of the underlying technology (web services, components, jobs). Information about Grid Objects is stored in a registry (Grid Resources Registry, GRR). Grid objects are invoked by the Computation Access component. Currently several tools, such as HIV sequencing, HIV subtyping or HIV drug ranking, are virtualized within the virtual labolatory and available as Grid Objects. The enactment engine (GSEngine) orchestrates the execution of scripts. Most virtual laboratory components feed a monitoring system (GEMINI [1]) with monitoring events which are aggregated, translated to ontological models, and published into the provenance tracking system, PROToS.

Experiment Planning Environment (EPE)

Look up Grid Objects

Grid Resources Registry (GRR) Read Grid Object properties Computation Access (Invoker & Optimizer)

Execute Experiment Execute remote processing Events

Instantiate and invoke GObjs

Computing Resource

Events

Monitoring System (GEMINI)

Events

Events

Experiment Information store Store information

GSEngine (enactment engine)

Provenance Tracking System (PROToS)

Data request

Publish information

Data Access Client (DAC)

Monitoring data (events)

(translated to ontological representation)

Information Aggregator

Fig. 1. Architecture of the virtual laboratory in ViroLab

Provenance in ViroLab is process-oriented, i.e. complete traces of experiments conducted in the virtual laboratory are captured and stored in a repository. Provenance data is generated as a set of events generated in distributed, instrumented components of the virtual laboratory runtime services and application services. The events are collected, correlated and aggregated, and then translated into an ontology representation. Provenance data is represented as XML documents in native XML database. The advantages of using XML to represent data exchanged across services was pointed out many times. XML is self-describing, interoperable, reusable, flexible and extensible [2]. Our model of provenance data is based on a comprehensive ontology-based description, modeled in the knowledge representation language, OWL3 . The basic conceptualization of the provenance is based on a generic experiment ontology supplied with provenancerelated concepts. Part of this ontology is depicted in Fig. 2. Its main concept is an 3

OWL Web Ontology Language Reference, http://www.w3.org/TR/owl-ref

384

B. Bali´s, M. Bubak, and J. Wach

Experiment which is composed of Execution Stages (invocations of Grid Operations or Data Access calls). Virtualized computation is represented as GridObjects which have one or more GridOperations. Physical computing resources involved in the execution are represented by other concepts, such as Hosts and Containers. Many important information pieces are modeled as concepts’ properties. Those generic concepts are connected with domain ontologies which, in our case, are ontologies describing applications and data of the ViroLab virtual laboratory. Concepts shown in the diagram – V iroLabEvent and V iroLabDataEntity are top level concepts of domain ontologies which enhance semantic descriptions of applications (a ViroLab event represents a well defined stage of an application, e.g. ‘computation of drug rankings’), and data, virological and medical in our case.

Fig. 2. Conceptualization of provenance data

In a virtual laboratory, we deal with a number of systems which collect and share information, and are maintained for a long time. In such an environment, data schemas might easily evolve in time. The use of self-describing and interoperable data representation enriched with semantic information enables an easy or even seamless and automatic integration of services in such cases. The benefits to model provenance as semantic, ontology-based data are as follows: 1. The ontology will enable the creation of a semantic description of each experiment performed in the virtual laboratory. This description can reflect different levels of detail and different points of view on the experiment.

Provenance Tracking in the ViroLab Virtual Laboratory

385

2. The generic ontology can be extended with domain ontologies which reflect the conceptual model of a particular research domain (e.g. bioinformatics) as well as a particular application (e.g. drug resistance). In this way the basic experiment description can be easily extended with more detailed, domain-specific information. 3. The ontologies denote the space of provenance event types that are relevant for an experiment. Thus, they can be used for automatic generation of instrumentationhelper tools that generate event representations. They also suggest which actors should be instrumented to obtain a particular piece of information. 4. Ontologies enhance the usability of provenance data in both manual and automatic manner. First, they enable construction of powerful and user-friendly query tools which operate on the level of ontology concepts and relationships instead of those of the particular data model. This is very helpful in constructing complex queries over provenance and actual data, ultimately aimed at extraction of knowledge for researchers and clinicians, who are not familiar with the technical details of underlying data representations. Second, a repository of experiment traces with well-defined semantics could be used for machine learning purposes by virtual laboratory runtime services, e.g. to improve scheduling and brokering decisions.

4 Provenance Tracking System – PROToS The Provenance TRacking System (PROToS) is developed on the basis of the provenance model presented in Section 3. Fig. 3 presents a simplified component diagram of PROToS and its context environment. The main PROToS components are as follows: – PROToS Core provides core functionalities – provenance data gathering (consumer interface) and retrieval (producer interface). Incoming information, in the form of events will be provided by the Monitoring System. Provenance data could be queried in one of supported languages (XQuery, RDQL, SPARQL, etc.) by any component, such as a QUaTRO tool, Application Optimizer or an application-specific portlet

Query Translation Tools (QUaTRO)

Data Access Service <<external>>

Ontology Store Producer Interface

PROToS Core

Provenance Data Repository

Consumer Interface

Events

Monitoring System

Fig. 3. Structure of PROToS – PROvenance Tracking System in ViroLab

386

B. Bali´s, M. Bubak, and J. Wach

(part of the ViroLab portal). External interfaces, exposed by core, are provided as Stateless Web Services, to enable a high level of interoperability with other system components. Internally, components are interconnected by RMI (Remote Method Invocation) middleware because of performance and security reasons. – Query Translation Tools (QUaTRO). This component consists of a set of tools to enable VL end-users to define and build provenance queries in an automated, easy way, without touching technical details. To achieve this goal, QUaTRO tools will make use of semantic descriptions existing in the VL – data, applications and experiments. Also, a simple Data Mining and Natural Language Processing will be employed by more sophisticated tools. This component is crucial in order to enable provenance use for a typical user who does not have any knowledge about used ontologies, provenance data structure or query languages such as XQuery or SQL. – Provenance Data Repository (PDR) stores provenance data and used ontologies (data, application, experiment), in the XML-RDF representation of the ontology language, OWL. Storage is hierarchical to enlarge storage capacity, and distributed to achieve high query performance. PDR also takes care of balancing provenance data in a way that ensures high performance and high storage capacity. – Ontology Store contains all Domain Ontologies used by Core and QUaTRO components, namely experiment, application and data ontologies. Currently, the Ontology Store is implemented as Sesame4 and accessible by a programmatic (Java) interface and http protocol.

5 Implementation of PROToS From the technical point of view, PROToS is organized as a set of applications, possibly and preferably deployed on different nodes. The PROToS core is deployed as Java Web Application (WAR), built on the Spring framework5 and requires a servlet container, such as Apache Tomcat or Jetty to operate. We have chosen Spring because it offers full support for such technologies as JMX, RMI and XFire-based Web Services. It is also a very well supported industry standard. Internally, the core runs an eXist database6 instance for Domain Ontologies storage purposes. It is necessary to provide such functionalities as quick validation of incoming provenance events and maintenance of PDR. The communication between PDR elements and PROToS core is implemented with Java RMI because of performance and security reasons. The PDR itself could be composed of many Storage Nodes (SNs) – groups of physical nodes that store and manage one or more domain ontologies with an associated set of individuals (called register). Each Storage Node has one central node (called Storage Super Node) to store the domain ontology and provide other functionalities such as group management/configuration and ontology reasoning with Pellet7 or other, Jena8 -compliant reasoner. It performs also other work associated with the ontology 4 5

6 7 8

Sesame Ontology Store, http://www.openrdf.org Spring framework (Dependency Injection container), http://www.springframework.org eXist XML native DB, http://exist.sourceforge.net Pellet OWL reasoner, http://www.mindswap.org/2003/pellet Jena Semantic Framework, http://jena.sourceforge.net

Provenance Tracking in the ViroLab Virtual Laboratory

387

PROToS Storage Node

Storage Node

Storage Peer

Storage Peer

PROToS CORE Storage Super Node

Storage Super Node

Storage Peer Storage Peer

Storage Peer Storage Peer

Fig. 4. PROToS example deployment diagram

processing, such as the validation of individual consistency and correctness with respect to a Domain Ontology. What is more, also queries in language other than XQuery will have to be handled by the ontology-core, running on the SN central node. We have chosen Jena as our framework for ontology processing. It includes its own SPARQL query language implementation, called ARQ with a query engine and a rule-based reasoner. Jena allows other reasoners to be deployed and provides us with many features required for our ontology-core. Jena also natively supports reading and storing OWL in XML format, which is crucial as we have chosen an XML database for storage. The rest of physical nodes will be used as Storage Peers – components that run an eXist instance with the sole purpose of storing part of the individuals register. We plan to divide the register into more than one Storage Peer, because individuals registries tend to grow to very large sizes, especially with ontology reasoning enabled. Currently, we plan to provide at least two implementations of the Storage Peer component. The first one, simple with embedded Jetty container and the second one, fully configurable and implemented with PicoContainers9 . Pico was chosen, because it is much more lightweight than Spring-based, and we believe that all node resources available should be devoted to storage functionality. Hierarchical, multi-node PDR architecture described above was developed to meet performance and storage requirements of PROToS. The investigation of the typical provenance use in VL applications convinced us that most queries will pertain to a single Domain Ontology. In our schema, such queries will be handled by one Storage Node, so that no merging or post-processing of the results will be needed. Fig. 4 presents an example core and PDR deployment. As mentioned before, the PROToS core provides maintenance of PDR. This task includes ontology-based routing mechanisms and algorithms for distributing and balancing ontology data on Storage Nodes. Other important roles of the core include choosing on which Storage Nodes a new query should be executed, performing the query and integrating results from all SNs involved. The PROToS core runtime 9

Picocontainer (Dependency Injection container), http://www.picocontainer.org

388

B. Bali´s, M. Bubak, and J. Wach

configuration allows for adding and removing Storage Nodes as well as Storage Peers, managing Domain Ontologies and rearranging the PDR structure. All configuration options/methods available both for core and Storage Nodes are remotely available via JMX. This is very important since performance greatly depends on such configuration options as reasoning enabled/disabled or reasoner used. JMX is highly standardized and provides off-the-box tools for building modular, dynamic solutions for managing systems such as PROToS. The use of this technology saved us much time and made our system cleaner and more robust.

6 Related Work A number of provenance tracking systems have been built. Provenance approach presented in [14] is based on recording all data sets (serialized as XML), and their transformations. Provenance data is stored as a relational database in a separate repository. In myGrid, workflow templates, data, and metadata including provenance records are stored in a common information repository [16]. Provenance records are generated based on the events from workflow enactment engines. In a postprocessing phase, provenance logs are annotated with ontological concepts taken from semantic description of services involved in a workflow [16]. The Karma provenance framework [13] was designed for data-centric workflows in a SOA, and collects three forms of provenance: workflow-describing services called, process-describing activities that took place in single service invocation and datadescribing services that produced/used a piece of data (across all recorded workflows). Virtual Data System allows to define and use procedures that perform data analysis resulting in new, derived data. Authors of [15] define provenance of a data object as a functional procedure that was used to produce and can be used to reproduce it. Focusing on above definition, they modeled a logical virtual data schema for provenance recording that allows to store information needed for a complete provenance record. For storing the information about operations arguments, such as type, VDS model uses simply string labels. Datasets for workflows are also stored by names (strings). Described solutions have certain limitations. Only myGrid is seriously focused on semantics of provenance records but only limited to annotations. The schema concept of [15] is similar to our generic experiment ontology, but because of the focus on VDS definition, much more limited. Example provenance queries presented in this work are fully supported by the PROToS provenance model. However, coupled with specific application ontologies, PROToS will support more complicated queries, similar to mining over provenance data. We use semantic model of data used in experiments which features mappings to database schema. That enables ontology-guided construction of queries combined with transparent browsing of databases themselves, if necessary.

7 Summary and Future Work We have presented PROToS – PROvenance Tracking System developed in the ViroLab project. Currently, the first prototype of PROToS has been released. It supports main functionalities, the storage of provenance events collected from Monitoring as well as

Provenance Tracking in the ViroLab Virtual Laboratory

389

XQuery-based queries. PDR is limited to one Storage Node, with basic ontology reasoning enabled. The QUaTRO component consists currently of a single tool which enables end-user oriented, GUI-based ontology-driven construction of queries which are subsequently transformed into XQuery. Future work includes the development of main ontologies for primary application scenarios in ViroLab, and extensions to the PROToS architecture and functionalities. PDR will be extended into a multi-node, hierarchical storage. This is a major task, because enabling PROToS for new VL applications will generate tremendous amounts of data which will certainly exceed the capability of a centralized storage. An important challenge is to address the inevitable evolution of VL Domain Ontologies – not only aplication, but also data DOs. Obviously, the user does not want to lose the whole registry of individuals when the domain ontologies are changed. Long term goals involve extending the PROToS ontology reasoning capabilities. As mentioned before, the current, Jena-based implementation requires all individuals to be loaded into the main memory. This of course will not be possible in a production environment with millions of individuals from one Domain Ontology. To overcome this limitation, we are closely evaluating such solutions as distributed reasoning with e-connections and automatic loading of individuals on demand from data base. Apart from PROToS core and PDR, we plan to develop Language-to-Query Translation tool as a part of our QUaTRO tools suite. This tool will enable a more sophisticated approach to the problem of user-provenance interaction. In our concept, it would analyze queries expressed by the user in a natural language to produce queries in XQuery. We are going to use text mining techniques and algorithms to achieve this. Of course, this tool will also use VL Domain Ontologies to speed-up the process of query analysis. Acknowledgements. We thank Peter M. A. Sloot, Maciej Malawski, Tomek Gubala and participants of the ViroLab Meeting in Dagstuhl in May 2007 for valuable discussions.

References 1. Balis, B., Bubak, M., Dziwisz, J., Truong, H.-L., Fahringer, T.: Integrated Monitoring Framework for Grid Infrastructure and Applications. In: Innovation and the Knowledge Economy. Issues, Applications, Case Studies, Ljubljana, Slovenia, October 2005, pp. 269–276. IOS Press, Amsterdam (2005) 2. Bean, J.: XML for Data Architects: Designing for Reuse and Integration. Morgan Kaufmann, San Francisco (2003) 3. Foster, I., Kesselman, C., Tuecke, S.: The anatomy of the grid: Enabling scalable virtual organizations. International Journal of High Perform. Comput. Applications 15(3), 200–222 (2001) 4. Goble, C.: Position statement: Musings on provenance, workflow and (semantic web) annotations for bioinformatics. In: Data provenance/derivation workshop (October 2002) 5. Groth, P., Jiang, S., Miles, S., Munroe, S., Tan, V., Tsasakou, S., Moreau, L.: D3.1.1: An Architecture for Provenance Systems. Technical report, University of Southampton (2006) 6. Hey, A.J.G., Trefethen, A.E.: The UK e-Science Core Programme and the Grid. Future Generation Computer Systems 18(8), 1017–1031 (2002) 7. Nemeth, Z., Sunderam, V.: Virtualization in grids: A semantical approach. In: Rana, O.F., Cunha, J.C. (eds.) Grid Computing: Software Environments and Tools, Springer, Heidelberg (2006)

390

B. Bali´s, M. Bubak, and J. Wach

8. Oinn, T., Li, P., Kell, D.B., Goble, C., Goderis, A., Greenwood, M., Hull, D., Stevens, R., Turi, D., Zhao, J.: Taverna / myGrid: aligning a workflow system with the life sciences community. In: Taylor, I., Deelman, E., Gannon, D., Shields, M. (eds.) Workflows for e-Science, pp. 300–319. Springer, New York (2007) 9. Robinson, A., Stevens, R.D., Goble, C.A.: myGrid: Personalised Bioinformatics on the Information Grid. In: Proc. 11th International Conference on Intelligent Systems in Molecular Biology (June 2003) 10. de Roure, D., Jennings, N.R., Shadbolt, N.: The Semantic Grid: a Future e-Science Infrastructure, Grid Computing–Making the Global Infrastructure a Reality, pp. 437–470. Wiley, Chichester (2003) 11. Sloot, P.M., Tirado-Ramos, A., Altintas, I., Bubak, M., Boucher, C.: From Molecule to Man: Decision Support in Individualized E-Health. Computer 39(11), 40–46 (2006) 12. Taylor, I., Shields, M., Wang, I., Harrison, A.: The Triana Workflow Environment: Architecture and Applications. In: Taylor, I., Deelman, E., Gannon, D., Shields, M. (eds.) Workflows for e-Science, pp. 320–339. Springer, New York (2007) 13. Simmhan, Y.L., Plale, B., Gannon, D., Marru, S.: Performance Evaluation of the Karma Provenance Framework for Scientific Workflows. In: Moreau, L., Foster, I. (eds.) IPAW 2006. LNCS, vol. 4145, Springer, Heidelberg (2006) 14. Szomszor, M., Moreau, L.: Recording and reasoning over data provenance in web and grid services. In: Meersman, R., Tari, Z., Schmidt, D.C. (eds.) CoopIS 2003, DOA 2003, and ODBASE 2003. LNCS, vol. 2888, pp. 603–620. Springer, Heidelberg (2003) 15. Zhao, Y., Wilde, M., Foster, I.: Applying the Virtual Data Provenance Model. In: Moreau, L., Foster, I. (eds.) IPAW 2006. LNCS, vol. 4145, Springer, Heidelberg (2006) 16. Zhao, J., Goble, C., Stevens, R.: Semantically linking and browsing provenance logs for escience. In: Proc. of the 1st International Conference on Semantics of a Networked World, Paris, France. LNCS, Springer, Heidelberg (2004)

Efficiency of Interactive Terrain Visualization with a PC-Cluster Dariusz Dalecki, Jacek Lebied´z, Krzysztof Mieloszyk, and Bogdan Wiszniewski Gda´ nsk University of Technology Faculty of Electronics, Telecommunications and Informatics ul. Narutowicza 11/12, 80-952 Gda´ nsk {jacekl,krzymi,bowisz}@eti.pg.gda.pl

Abstract. The paper presents final results of the project aimed at interactive visualisation of a terrain based on real spatial data, which was one of the tasks of the CLUSTERIX project. Practical aspects of various performance issues are addressed in particular to verify adequacy of using PC-clusters in graphical applications. Keywords: mesh simplification, dynamic 3D scene, parallel sector simplification.

1

Introduction

Interactive visualization of real terrain based on spatial data have been one of the topics investigated in the recently concluded CLUSTERIX (National Cluster of Linux Systems) Project [6]. In particular, the research addressed two problems that have been identified during development of one of this project’s applications, Vis3D, for visualizing real terrain with a PC-cluster1 : – information represented by spatial data may be incomplete, inconsistent, or in incompatible format for high performance computing, and – data must be processed on time to enable real-time interaction of observers with a dynamic 3D scene. The former problem was investigated by authors and results published in their previous PPAM2005 paper [3]. This paper presents results of the research on the latter problem, namely assessment of major performance characteristics of the Vis3D application.

2

Terrain Visualization

Reliable visualization of a real terrain requires large volumes of spatial data. Publically available data (’hgt’ files) of a digital Earth terrain model, provided by 1

This work has been supported in part by the Polish Ministry of Science and Information Society Technologies under grant 6T11 2003C/06098.

R. Wyrzykowski et al. (Eds.): PPAM 2007, LNCS 4967, pp. 391–399, 2008. c Springer-Verlag Berlin Heidelberg 2008 

392

D. Dalecki et al.

the Shuttle Radar Topography Mission (SRTMP) project [7] include hipsometric values measured for each three longitude and latitude seconds. SRTM data are grouped in sectors (tiles) of the size corresponding to 1×1 longitude and latitude degrees. For the Gdansk region (see Figure 1) it takes 1442401 (1201×1201) 16bit values representing 2880000 triangles covering the 61 km × 110 km area of of 7150 km2 . The number of triangles estimated here corresponds more or less to the resolution of contemporary images. It is worth mentioning that in order to visualize a scene from higher altitudes many sectors, each one of nearly three million triangles, may be needed. Confronting such a huge number of triangles with often limited capabilities of graphical cards indicates the need to simplify a visualized scene, especially that limited resolution of displays would not allow to visualize details of distant sectors. 2.1

Triangle Mesh Simplification

For a moving observer a perceived level of detail changes for different sectors of the visualized terrain. Details deteriorate when close sectors are left behind, and new details emerge when getting closer to sectors that lied far before. For each observation point certain “subareas” of various level of detail may be distinguished. They take a form of concentric rings surrounding the observer. Since it is easier to represent terrain as rectangular sectors of a uniform distribution of visible detail, rings are represented most often as a composition of such sectors. Note in Figure 1a that sectors of the considered Gdansk area ehxibit various level of detail measured with a number of triangles. They are shaded from light gray corresponding to over 100 triangles down to dark grey corresponding to no more than 20 triangles. Mesh simplification may be implemented in a couple of ways. For example the Real-time Optimally Adapting Meshes (ROAM) algorithm [4] processes a regular mesh of triangles by splitting and merging pairs of triangles, as shown in a)

b)

Fig. 1. Intensity of computations in sectors around a moving observer: (a) a cluster view (b) an aerial view

Efficiency of Interactive Terrain Visualization with a PC-Cluster a)

393

b) split

vsplit

TR

TL

fn1

T

T0

TB

TB1 TB0

fn1

T1

fn0

fn3 Vs

Vu fn3 fL fR

fn2

fn0

Vt

fn2

ecol

merge

Fig. 2. Operations on a triangle mesh: (a) ROAM, (b) VDPM

Figure 2a. Alternatively View-Dependent Progressive Mesh (VDPM) method [1] is aimed at an irregular mesh of triangles, by performing on triangle nodes ecol (collecting) and vsplit (splitting) operations, shown in Figure 2b. Vis3D application uses a yet another approach, known as Sequential Greedy Insertion (SGI) [5] (see Figure 3).

+

+

Fig. 3. Greedy insertion of nodes

The algorithm starts from a maximally simplified mesh (two triangles per one sector) and gradually inserts nodes into the mesh at places where the greatest error is observed in each step. An inserted node implies reorganization of triangles around it, because if it happens to fall inside a circle circumscribing any existing triangle that triangle is eliminated from the mesh. The “hole” is filled next with triangles determined by the new node and pairs of neighboring nodes that lie on the hole’s border. 2.2

Parallel Simplification of Sectors

Simplification of data representing a wide geographical area involves a rather large number of loosely-coupled parallel tasks computing respective sectors. Therefore it is quite natural to parallelize simplification of sectors, provided that simplification tasks are physically distributed [2]. It may be expected that

394

D. Dalecki et al.

the total scene computation time will shorten proportionally to the number of available processing nodes. Partitioning of the visualized area into sectors, eg. 1 km × 1 km, allows for equal distribution of data among processing nodes. This may however limit simplification of sectors observed from a further distance, since the simplest sector may not include less than two triangles. Therefore distant sectors should be grouped into larger units, called hipersectors – to be simplified by one task. See for example varying sizes of sectors in Figure 1a. Assigning sectors to the respective tasks (or computing nodes) could be done in many ways. The simplest way is to assume a uniform “complexity” of the visualized geographical area. In such a case tasks will receive sector/hypersector data distributed equally with regard to the number of sectors. Sector simplification algorithm would then take into account only a position of the observer reported by the visualization station. This solution, illustrated in Figure 4a, allows connecting many visualization stations to the application to view a dynamic scene in a panoramic fashion. a)

b) cluster

cluster geospatial data

geospatial data

sector data process 1

process 2

observer’s position

sector data

MPI

...

process N

TCP/IP multicast

graphical workstation

server

client 1

observer’s position

client 2

...

client N

TCP/IP multicast

graphical workstation

Fig. 4. Vis3D architecture: (a) without supervision (b) with supervision

Various test of the Vis3D application indicated that sometimes data may be lost when sent simultaneously to the graphical station for a final rendering of a scene view. To eliminate this phenomenon the notion of a token was introduced. Only a current owner of the token can send out data upon completion of its sector simplification. In oder to guarantee the fair use of the token an additional token server was introduced. It acts as an arbiter resolving conflicting request from tasks simplifying sectors of the scene. Communication between simplifying tasks and the token server was implemented with MPI (see Figure 4b). The server also takes responsibility of a dynamic mapping of sectors on the processing nodes.

Efficiency of Interactive Terrain Visualization with a PC-Cluster

3

395

Performance Testing Experiments

Experiments were run on a PC cluster (Intel Itanium II 1,2 G Hz, 16 B RAM) in different configurations ranging from 2 up to 32 nodes, and in a standard lab of two to four PCs (Intel Pentium IV 3,6 G Hz, 1B RAM) with Fast Ethernet LAN. In each case the graphical station was a Dell Latitude D50 laptop with a specially written terrain visualisation program, which besides scene rendering was capable of measuring and displaying various parameters characterizing simplification of the scene. Some of them may be seen in Figure 5 as horizontal bars representing numbers of sectors received for visualization from each client. Their similar length (as shown in Figure 5b) indicates a uniform distribution of sector computation load among processing nodes. As mentioned before, the number of processed sectors equals the number of processes simplifying the mesh. The visualization program provides also an insight into the structure of the simplified terrain (see Figure 1b). a)

b)

Fig. 5. Visualized terrain: (a) observer’s view, (b) sectors of the area

During the tests of various configurations involving the cluster and LAN based versions of the Vis3D application hipsometric data of an 65 km × 110 km area around Gdansk were used. This terrain has been found representative for the tests, as it included flat surfaces to the North (the Bay of Gdansk), slightly elevated coastal areas in the center, and moraine hills surrounding the bay area from the South West (see Figure 5b). Measurements included the total number of triangles in a scene, their reduction rate, scene generation time, average scene update time, maximum allowed speed of the observer and an average single node computation time in three scenarios – an observer moving along the same route at 1 km, 4 km and 10 km over the area shown in Figure 5. For each of these altitudes simplification of sectors required respectively 60, 174 and 286 tasks (one per sector).

396

3.1

D. Dalecki et al.

Cluster Based Visualization

Basic performance characteristics collected during the experiments with a cluster based Vis3D application included among others: – scene generation time, defined as a time from sending a request from the graphical workstation to receiving all required sectors for rendering. Each sector was calculated from scratch up to the detail level implied by its distance from the initial position of the observer (Figure 6a); – average scene update time, i.e. an average time required to compute a new level of detail of a sector changing its distance from the observer (Figure 6b); – maximum speed of the observer that cannot be exceeded due to delays in computing new level of detail of changing sectors (Figure 6c); – average single node computation time, indicating scalability of the Vis3D application (Figure 6d). Experimental configurations of cluster nodes were ordered in Figure 6 according to the increasing degree of parallelism, taking into account their double-core processors, i.e. from 2×1 (two nodes with a single task each), through 2×2 (two nodes with two tasks), up to a 16×2 configuration. It can be seen from Figure 6a that computations with a lower degree of parallelism require considerably more time to compute the initial scene view, however configuration with four nodes is already sufficient and further increase of computational power is not needed. One exception is for the highest altitude of 10 km, since the level of detail even for the closest sectors does not matter much, so they may be practically calculated on a single/double processor machines. While computation of the initial scene view may be demanding for weaker configurations, scene updates are much faster (tenths of seconds as shown in Figure 6b) – especially for higher altitudes, as at and over 4 km the terrain significantly flattens. However, performance of updates for the lower altitudes becomes less predictable, as the terrain shape may change rapidly due to the abrupt maneuvers of the observer. Computations of particular sectors may dominate individual nodes and slow down completion of the entire scene update. Note the “saw” like shape of this characteristic for altitudes of 1 km in Figure 6b. Certainly its shape will vary significantly with regard to the exact route taken by the observer, its speed and orientation, as well as the shape of the terrain below. This observation may be reinforced by analyzing the maximum allowable speeds in Figure 6c. At the lowest altitude of 1 km the speed varies for the same reason as mentioned before, while for the higher altitudes it does not depend on the applied degree of parallelism. Surprisingly, the maximum speed at 10 km is lower than at 4 km, what may be explained by the square increase of the number of visible sectors to compute with the increase of altitude vs. the only linear decrease of the number of details in each sector. Anyway, the basic requirement for the Vis3D system to allow for the whole range of subsonic speeds [3] is met, as in any case maximum speeds are well in the upper 500-1000 km range.

Efficiency of Interactive Terrain Visualization with a PC-Cluster b) 30

0.6

25

0.5 seconds

seconds

a)

397

20 15

0.4 0.3

10

0.2

5

0.1

0

2x1

2x2

4x1

4x2

8x1

8x2

16x1

0.0

16x2

2x1

2x2

4x1

nodes x tasks

4x2

8x1

8x2

16x1

16x2

8x2

16x1 16x2

nodes x tasks

c)

d) 4000 3500 3000

25 20

2000

seconds

km/h

2500

1500 1000

10 5

500 0

15

0 2x1

2x2

4x1

4x2

8x1

8x2

16x1

16x2

1 km, 60 sectors

2x1

2x2

4x1

4x2

8x1

nodes x tasks

nodes x tasks

4 km, 174 sectors

10 km, 286 sectors

Fig. 6. Performance characteristics of cluster based Vis3D: (a) scene generation time, (b) scene update time, (c) max observer’s speed, (d) single node computation time

Finally, the task mapping algorithm in Vis3D is scalable, since the average single node computation time decreases counter-proportionally to the increase of the number of nodes for each altitude, as shown in Figure 6d. 3.2

LAN Based Visualization

Computational power of processors of the PC-cluster used in the experiments was estimated at the level of nearly 4000 BogoMIPS (number of null instructions performed in 1 second). Given the almost doubled power of processors in the PC lab (the LAN based configuration of Vis3D) compared to processors in cluster nodes it was expected that cluster based configuration of Vis3D will perform worse. Surprisingly, the latter performed better, probably owing to the architecture of the Ithanium II processor (64-bit, double core) used by the cluster and 16 times larger RAM. For comparison, some measured results of the total computation time of all sectors in a scene (over 1 million triangles) are shown in Figure 7.

398

D. Dalecki et al. 160 140 120

seconds

100

1km, 60 sectors, LAN

80 10km, 286 sectors, LAN 60 1km, 60 sectors, cluster

40 20

10km, 286 sectors, cluster

0 2x1

2x2

4x1

4x2

nodes x tasks

Fig. 7. Computation time of all sectors

The total time shown in Figure 7 was measured as a sum of local execution times of all task involved in computing the initial scene view. In other words, it equals a time required to calculate the scene with only a single processor in a sequential fashion. It can be seen that LAN configuration processors were not able to cope well with more than 1 task per node for relatively difficult “low altitude” sectors. Sector computations can stabilize only for less detailed “high altitude” sectors (see 10 km, 258 sectors, LAN based configuration in Figure 7). On the other hand, performance of the cluster based configuration with this regard remains stable for all altitudes and gets better with the increase of the altitude. We skip other characteristics of the LAN based configuration, as they have similar shapes to their cluster based counterparts shown in Figure 6. One difference is that the respective curves are placed higher, as generally the LAN based configuration performed worse. In particular, the average single node computation time characteristic indicated almost linear scalability of the Vis3D application in a LAN based configuration, as in its cluster based counterpart.

4

Conclusions

Spatial data used for performance testing of the Vis3D application were optimized before the experiments. This was possible because terrain data remained static during execution of all scenarios. Experiments with unoptimized data were carried out in parallel to the ones described earlier in the paper. Their results, when compared to the “optimized” case, indicate that all scenarios became to perform very well at a rather low degree of parallelism (typically four nodes were enough) only for optimized data. By the way, four (or more) processing nodes are a quite common feature of upper class graphical workstations, having typically 8-16 processors. In that sense, using clusters for realistic terrain visualization might be considered “using a steam hammer to kill an ant”. However, results obtained for the latter case (unoptimized data) indicate that simplification of sectors in real-time may consume considerably more computational power offered by a PC-cluster. While for the optimized hipsometric data

Efficiency of Interactive Terrain Visualization with a PC-Cluster

399

of the Gdansk area about one million triangles could be effectively processed by no more than four processors (see Figure 6a), unoptimized data involved 2-3 times greater numbers of triangles, depending on the particular altitude, and required considerably more nodes to achieve the same rate of reduction as for the optimized case. For example, altitudes of 1 and 4 km required respectively over 32 and 16 nodes to observe the same effect as in Figure 6a for four nodes processing optimized data. This is an important result for interactive visualization of dynamic data (acquired on-line) that cannot be optimized prior to the interactive visualization session. These may include data modelling of meteorogical phenomena, such as weather fronts or building-up clouds, dynamic surfaces, like sea waves, or physical processes, like smoke, forest fires, etc. Such dynamic data could be produced by specialized simulators used in future interactive visualization exercises with the Vis3D application. Further research on this topic, inspired by the CLUSTERIX project is currently on the way.

References 1. Hoppe, H.: Smooth View-Dependent Level-of-Detail Control and its Application to Terrain Rendering. research.microsoft.com/hoppe/svdlod.pdf 2. Lebied´z, J., Mieloszyk, K.: Real Terrain Visualisation on the Basis of GIS Data. In: Bolc, L., Michalewicz, Z., Nishida, T. (eds.) IMTCI 2004. LNCS (LNAI), vol. 3490, pp. 40–49. Springer, Heidelberg (2005) 3. Lebied´z, J., Mieloszyk, K., Wiszniewski, B.: Real Terrain Visualisation with a Distributed PC-Cluster. In: Wyrzykowski, R., Dongarra, J., Meyer, N., Wa´sniewski, J. (eds.) PPAM 2005. LNCS, vol. 3911, pp. 349–356. Springer, Heidelberg (2006) 4. Duchaineau, M., Wolinsky, M., Sigeti, D.E., Miller, M.C.: Aldrich Ch., and Mineev-Weinstein M.B. ROAMing Terrain: Real-time Optimally Adapting Meshes, http://www.llnl.gov/graphics/ROAM 5. Garland, M., Hecbert, P.S.: Fast Polygonal Approximatio of Terrains and Height Fields. gpraphics.cs.uiuc.edu/garland/CMU/scape/scape.pdf 6. Wyrzykowski, R., Meyer, N., Stroi´ nski, M.: Clusterix: National Cluster of Linux Systems (2002), http://www.clusterix.pl 7. Shuttle Radar Topography Mission. NASA, http://www2.jpl.nasa.gov/srtm

Implementing Commodity Flow in an Agent-Based Model E-Commerce System Maria Ganzha1,2 , Maciej Gawinecki2 , Pawel Kobzdej2 , Marcin Paprzycki2,3, and Tomasz Serzysko4 1

2

Elblag University of Humanities and Economy, Elblag, Poland Systems Research Institute Polish Academy of Science, Warsaw, Poland 3 Warsaw Management Academy, Poland 4 Warsaw University of Technology, Poland

Abstract. In our work we are developing a complete agent-based e-commerce system. Thus far we have been focusing on interactions between clients and shops (C2B relationships). In this work we discuss how the proposed system can be augmented with a logistic subsystem and discuss specifics of its implementation.

1

Introduction

In our work, we are developing a complete model agent-based e-commerce system [7, 3]. Thus far we have considered interactions between clients and shops (C2B relationships) and assumed that in each shop there exists a warehouse where products, to be sold through pri