Tests and Proofs - TAP 2011 - PDF Free Download

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris...

Author: Martin Gogolla | Burkhart Wolff

25 downloads 2073 Views 3MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Report copyright / DMCA form

DOWNLOAD PDF

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen TU Dortmund University, Germany Madhu Sudan Microsoft Research, Cambridge, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max Planck Institute for Informatics, Saarbruecken, Germany

6706

Martin Gogolla Burkhart Wolff (Eds.)

Tests and Proofs 5th International Conference, TAP 2011 Zurich, Switzerland, June 30 – July 1, 2011 Proceedings

13

Volume Editors Martin Gogolla Universität Bremen Fachbereich Mathematik/Informatik Bibliothekstr. 1, 28334 Bremen, Germany E-mail: [email protected] Burkhart Wolff Université Paris-Sud 11 Parc Club Orsay Université ZAC des vignes, 4, rue Jacques Monod, 91893 Orsay Cedex, France E-mail: [email protected]

ISSN 0302-9743 e-ISSN 1611-3349 ISBN 978-3-642-21767-8 e-ISBN 978-3-642-21768-5 DOI 10.1007/978-3-642-21768-5 Springer Heidelberg Dordrecht London New York Library of Congress Control Number: Applied for CR Subject Classiﬁcation (1998): D.2.4, D.2, D.1, D.3, F.3, F.4.1 LNCS Sublibrary: SL 2 – Programming and Software Engineering

© Springer-Verlag Berlin Heidelberg 2011 This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, speciﬁcally the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microﬁlms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a speciﬁc statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typesetting: Camera-ready by author, data conversion by Scientiﬁc Publishing Services, Chennai, India Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)

Preface

This volume contains the research papers and invited papers presented at the 5th International Conference on Tests and Proofs (TAP 2011) held from June 30 to July 1, 2011 in Z¨ urich, Switzerland. TAP 2011 was the ﬁfth event in a series of conferences devoted to the convergence of tests and proofs. It combines ideas from both areas for the advancement of software quality. To prove the correctness of a program is to demonstrate, through impeccable mathematical techniques, that it has no bugs; to test a program is to run it with the expectation of discovering bugs. On the surface, the two techniques seem contradictory: if you have proved your program, it is fruitless to comb it for bugs; and if you are testing it, that is surely a sign that you have given up hope of proving its correctness. Accordingly, proofs and tests have, since the onset of software engineering research, been pursued by distinct communities using rather diﬀerent techniques and tools. And yet the development of both approaches leads to the discovery of common issues and to the realization that each may need the other. The emergence of model checking has been one of the ﬁrst signs that contradiction may yield to complementarity. Further evidence is given by test data generation techniques from models or programs which boil down to constraint resolution techniques for relatively large formulae; the advent of powerful SAT and SMT solvers have therefore powered new testing techniques. Finally, since formal, proof-based veriﬁcation is costly, testing invariants and background theories can be helpful to detect errors early and to improve cost eﬀectiveness. Summing up, in the past few years an increasing number of research eﬀorts have encountered the need for combining proofs and tests, dropping earlier dogmatic views of incompatibility and taking instead the best of what each of these software engineering domains has to oﬀer. The ﬁrst TAP conference (held at ETH Z¨ urich in February 2007) was an eﬀort to provide a forum for the cross-fertilization of ideas and approaches from the testing and proving communities. For the 2008 edition we found the Monash University Prato Centre near Florence to be an ideal place providing a stimulating environment. The third TAP was again held at ETH Z¨ urich in July 2009. Since 2010, TAP has been co-located with TOOLS, and its instance for 2010 therefore took place at the School of Informatics (E.T.S. de Ingenieria Informatica) of the University of Malaga, while TOOLS 2011 took place at ETH Z¨ urich again. We wish to sincerely thank all authors who submitted their work for consideration. We received 27 submissions from which we ﬁnally accepted 12, after a formal refereeing process requiring at least three reviews. While the TAP community is still relatively young, these ﬁgures give evidence that TAP has been consolidated as a scientiﬁc event.

VI

Preface

We would like to thank the Program Committee members as well as the additional referees for their energy and their professional work in the review and selection process. Their names are listed on the following pages. The discussions during the paper selection were vital and constructive. We received positive feedback from the submitting authors about the review quality. TAP 2011 was decorated by two distinguished invited speakers: Marie-Claude Gaudel and Patrice Godefroid. Both are internationally well-accepted scientists and experts in the fundamentals and applications of testing and proving techniques. Our thanks go to both of them. It was a team eﬀort that made the conference so successful. We are grateful to the TAP Conference Chairs Yuri Gurevich and Betrand Meyer for their support. And we particularly thank the organizers of the Tools Federated Conferences, Christian Estler, Yu Pei and Claudia G¨ unthart from ETH Z¨ urich, our Publication Chair Gordon Fraser from the University of Saarbr¨ ucken and our Web Chair Lars Hamann from the University of Bremen for their hard work and their support in making the conference a success. In addition, we gratefully acknowledge the generous support of the ETH Z¨ urich, who ﬁnanced the invited speakers. April 2011

Martin Gogolla Burkhart Wolﬀ

Conference Organization

Conference Chairs Yuri Gurevich Bertrand Meyer

Microsoft Research, USA ETH Z¨ urich, Switzerland

Program Chairs Martin Gogolla Burkhart Wolﬀ

University of Bremen, Germany University Paris-Sud (Orsay), France

Proceedings Chair Gordon Fraser

Saarland University, Germany

Web Chair Lars Hamann

University of Bremen, Germany

Program Committee Nazareno Aguirre Bernhard K. Aichernig Paul Ammann Benoit Baudry Dirk Beyer Nikolaj Bjorner Achim D. Brucker Koen Claessen Robert Claris´o Marco Comini Catherine Dubois Gordon Fraser Carlo Alberto Furia Angelo Gargantini Martin Gogolla Arnaud Gotlieb Reiner H¨ahnle Bart Jacobs Thierry J´eron Gregory Kapfhammer

Universidad Nacional de Rio Cuarto, Argentina TU Graz, Austria George Mason University, USA INRIA, France University of Passau, Germany Microsoft Research, USA SAP Research, Germany Chalmers University of Technology, Sweden Universitat Oberta de Catalunya, Spain Universit` a di Udine, Italy ENSIIE-CEDRIC, France Saarland University, Germany ETH Z¨ urich, Switzerland University of Bergamo, Italy University of Bremen, Germany INRIA, France Chalmers University of Technology, Sweden Katholieke Universiteit Leuven, Belgium INRIA Rennes - Bretagne Atlantique, France Allegheny College, USA

VIII

Conference Organization

Nikolai Kosmatov Victor Kuliamin Karl Meinke Antoni Oliv´e Holger Schlingloﬀ T.H. Tse Margus Veanes Burkhart Wolﬀ Fatiha Zaidi

CEA LIST, France Russian Academy of Sciences, Russia Royal Institute of Technology Stockholm, Sweden Universitat Polit`ecnica de Catalunya, Spain Fraunhofer FIRST and Humboldt University, Germany University of Hong Kong, China Microsoft Research, USA University Paris-Sud, France University Paris-Sud, France

External Referees Andrea Baruzzo Puneet Bhateja Matthieu Carlier Gidon Ernst Marcelo Frias Rene Just Nadjib Lazaar Stefan Loewe Florian Lonsing Fei Niu Malte Rosenthal Sagar Sen Muddassar Sindhu Jan Smans

University of Udine, Italy Chennai Mathematical Institute, Chennai, India IRISA, France Augsburg University, Germany Universidad de Buenos Aires, Argentina Ulm University, Germany INRIA, France University of Passau, Germany Johannes Kepler University, Austria Royal Institute of Technology Stockholm, Sweden University of Passau, Germany INRIA, France Royal Institute of Technology Stockholm, Sweden Katholieke Universiteit Leuven, Belgium

Local Organizers Christian Estler Claudia G¨ unthart Yu Pei

ETH Z¨ urich, Switzerland ETH Z¨ urich, Switzerland ETH Z¨ urich, Switzerland

Table of Contents

Checking Models, Proving Programs, and Testing Systems . . . . . . . . . . . . Marie-Claude Gaudel

1

Tests from Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Patrice Godefroid

14

Incorporating Coverage Criteria in Bounded Exhaustive Black Box Test Generation of Structural Inputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nazareno M. Aguirre, Valeria S. Bengolea, Marcelo F. Frias, and Juan P. Galeotti

15

Checking the Behavioral Conformance of Web Services with Symbolic Testing and an SMT Solver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Lina Bentakouk, Pascal Poizat, and Fatiha Za¨ıdi

33

Association of Under-Approximation Techniques for Generating Tests From Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pierre-Christophe Bu´e, Jacques Julliand, and Pierre-Alain Masson

51

Security Mutants for Property-Based Testing . . . . . . . . . . . . . . . . . . . . . . . . Matthias B¨ uchler, Johan Oudinet, and Alexander Pretschner The SANTE Tool: Value Analysis, Program Slicing and Test Generation for C Program Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Omar Chebaro, Nikolai Kosmatov, Alain Giorgetti, and Jacques Julliand Abstraction Based Automated Test Generation from Formal Tabular Requirements Speciﬁcations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Renzo Degiovanni, Pablo Ponzio, Nazareno Aguirre, and Marcelo Frias

69

78

84

Correct Code Containing Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Claire Dross, Jean-Christophe Filliˆ atre, and Yannick Moy

102

A Random Testing Approach Using Pushdown Automata . . . . . . . . . . . . . Pierre-Cyrille H´eam and Catherine Masson

119

Incremental Learning-Based Testing for Reactive Systems . . . . . . . . . . . . . Karl Meinke and Muddassar A. Sindhu

134

X

Table of Contents

Encoding OCL Data Types for SAT-Based Veriﬁcation of UML/OCL Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mathias Soeken, Robert Wille, and Rolf Drechsler

152

State Coverage Metrics for Speciﬁcation-Based Testing with B¨ uchi Automata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Li Tan

171

Lightweight Testing of Communication Networks with e-Motions . . . . . . . Javier Troya, Jos´e M. Bautista, Fernando L´ opez-Romero, and Antonio Vallecillo

187

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

205

Checking Models, Proving Programs, and Testing Systems Marie-Claude Gaudel Univ Paris-Sud, Laboratoire LRI, Orsay, F-91405, and CNRS, Orsay, F-91405 [email protected] http://www.lri.fr/~ mcg

Abstract. We are all faced up to a ﬂowering of concepts and methods in the area of software veriﬁcation and validation, due to signiﬁcant advances in the domain. This paper considers the main terms and expressions currently in use on the subjects of model, speciﬁcation, program, system, proof, checking, testing. Some analysis of the use and combination of these terms is sketched, pointing out some confusions and discrepancies. This leads to a plea for clariﬁcation of the taxonomy and terminology. The aim is a better identiﬁcation of the general concepts and activities in the area, and the development of some uniform basic terminology helping communication and cooperation among the scientiﬁc and industrial actors. Keywords: software veriﬁcation, software testing.

1

Introduction

We are all faced up to a ﬂowering of concepts and methods in the area of software veriﬁcation and validation, due to signiﬁcant advances in the domain. This paper proposes a tour, from a terminological point of view, of some of the methods for specifying, building, verifying and certifying high-quality software. It has no pretension of presenting a survey of the state-of-the art. It pleads the case of a consensual clariﬁcation of the vocabulary in order to improve mutual understanding, to save time, and to make comparisons and cross-fertilisation easier. It takes its inspiration from a well-known collective work led in the area of fault-tolerant computing under the auspices of IEEE and IFIP by a joint committee “Fundamental Concepts and Terminology”. The result of this eﬀort [2], [36] was widely published in 2004 and adopted by the research community in the area of dependable computing and fault tolerance. In the area of software veriﬁcation and validation, numerous methods, techniques, tools, are now available in order to ensure and verify software quality. it has been suggested for a long time (see for instance [23]), and it is now quite well acceptedthat activities such as model-checking, proof-supported reﬁnement, program proving, system testing, etc, are complementary. This being said, it is M. Gogolla and B. Wolﬀ (Eds.): TAP 2011, LNCS 6706, pp. 1–13, 2011. c Springer-Verlag Berlin Heidelberg 2011

2

M.-C. Gaudel

not yet completely clear how to organise this complementarity and how to assess its beneﬁts. Moreover, several divergent interpretations of this complementarity can be found in the literature: for instance it is diﬀerent to perform concurrently two of the activities mentioned above, drawing global conclusions at the end, and to transpose one method developed for one of these activities to another one in order to improve it. In an attempt of clariﬁcation, this paper reserves the use of the word “complementarity” to the ﬁrst type of approach, and will refer to “cross-fertilisation” for the second case. Several success stories report on new, and less new, approaches such as: program veriﬁcation [9] [52], run-time veriﬁcation [41], semi-proving programs [14], software model-checking [7] [24], model-based testing [19], speciﬁcation-based testing [30], coverage in model-checking [32], bounded model-checking [8], symbolic model-checking [43], symbolic execution [16] [45], property testing [26], theorem proving for program checking [18], black-box testing [5], black-box checking [46], etc. It would be quite satisfactory to establish some classiﬁcation of these methods taking into account for instance: their input (speciﬁcation/model, program text, executable system); their result and the sort of guarantee they yield (full certainty, certainty w.r.t. some hypotheses, probability); the classes of software they can deal with (sequential, concurrent, ...); their scalability and the complexities of the underlying algorithms. This paper continues as follows. Section 2 recalls and discusses the basic deﬁnitions of the terms used in the title: models, programs, systems, modelchecking, program proving, system testing. Section 3 gives some examples of cross-fertilisation: the use of model-checkers for testing, the use of testing strategies for model-checking, the use of theorem provers for testing. Section 4 raises the issue of the distinction between static and dynamic approaches, which becomes a bit cloudy when considering for example symbolic evaluation, concolic testing or run-time veriﬁcation. The last section comes back to the need of a consensual clariﬁcation of the terminologies, and brieﬂy evokes some research perspectives.

2

Some Preliminaries on Entities and Activities in Software Development

There is a ﬂourishing literature on software engineering and the development process. Here, we focus on approaches related to formal treatments of highlevel descriptions, programs, and systems, these last ones being, we must never forget it, the ultimate aims of the development process. This section aims at the clariﬁcation of the concepts of model, program, system, and of some related activities, namely, model-checking, program proving, system testing in order to use them in the next sections. In these next sections, we will see that these notions overlap in some cases, there are variants of them, and the literature is sometimes confusing and not enough precise with the terms used for the qualiﬁcation of new approaches.

Checking Models, Proving Programs, and Testing Systems

3

Let us start with this dramatically oversimpliﬁed schema: M odel/specif ication → P rogram → System Models or speciﬁcations omit details that appear in the program. They are used for system description, design and analysis. The program and the system must conform to the model. A program is a very detailed solution to a much more abstract problem that the system is required to deal with. A system is an executable entity that results from the compilation, installation and activation of the program. These three points are developed in the rest of the section and activities that are respectively performable on models, programs, systems are brieﬂy characterised. Before going further, let us recall the opposition between static approaches, which work on the basis of the texts of speciﬁcations or programs, and dynamic ones, which require the execution of the system. Examples of the ﬁrst approaches are static analysis and proofs; examples of the second are testing methods. The schema above is just a starting point for introducing the concepts of model, program, system. As it is presented, it emphasizes the development process. It is well-known that there are backtracking phenomena in this process, for instance when verifying or validating programs and systems against a model or a speciﬁcation. 2.1

Models and Model-Checking

Model is an heavily overloaded term: for a physicist a model may be a diﬀerential equation; for a biologist it is often an homogeneous population of mice or frogs... In computer science, there is as well quite a variety of models depending on the kind of requirements to be described and checked. Most the time, when speaking of models, computer scientists mean ﬁnite behavioural models based on states and transitions between states. States are labelled by atomic propositions. Executions of the modelled system are supposed to correspond to sequences of adjacent transitions in the model. The model-checking activity consists in checking that the model satisﬁes a temporal property via an (often) exhaustive exploration of the model, searching for some counter-example. The property to be checked is sometimes called “the speciﬁcation”. Interesting models often reach enormous sizes, for instance when models of concurrent systems are considered or when the modelled problem involves many variables on large domains. However, the size limits have been pushed further and further thanks to a lot of powerful techniques: BDD representation, partial order and symmetry reductions, abstraction, on-the-ﬂy treatments of components to avoid the construction of global models, etc. A problematic issue is the gap from the model to the program and the system. What is checked is that the model satisﬁes a property. It is not a guarantee that the program or the system do so but in special circumstances such as: the program is derived via some certiﬁed translation from the model, or conversely, the model is extracted from the program. This last case is often referred to as software model checking [24], or program model-checking [3].

4

M.-C. Gaudel

Remark 1. Other sorts of high-level descriptions, that could be also called models, are based on logical formulas: set of axioms, pre- and postconditions, predicate transformers, etc. They are traditionally called formal speciﬁcations. The activity of verifying that a required property is a consequence of the speciﬁcation is naturally supported by theorem provers. This could be called speciﬁcation-checking. Remark 2. There is a tendency in the literature on model-checking to use the term veriﬁcation, or automatic veriﬁcation, as a synonym of model-checking. However, according to numerous standards and references, veriﬁcation has a much broader meaning. For instance, quoting the SWEBOK (Software Engineering Body of Knowledge, see www.swebok.org), we have: “Veriﬁcation is an attempt to ensure that the product is built correctly, in the sense that the output products of an activity meet the speciﬁcations imposed on them in previous activities. Validation is an attempt to ensure that the right product is built, that is, the product fulﬁlls its speciﬁc intended purpose.” Actually, in term of category of activities, model-checking could be as well mentioned as a validation activity, since checking requirements is usually referred to as validation [2], and program proving and system testing clearly come under veriﬁcation activities. 2.2

Programs and Program Proving

A program is a piece of text written in a well-deﬁned language. In some case it is annotated by assertions: pre- and postconditions, invariants, that are formulas in another well-suited language, for instance JML for Java [38] or Spec# for C# [4]. It is possible to perform formal reasoning on programs either by using the rules of the operational semantics of the programming language, or by using a formal system that considers annotated programs as formulas like for instance in [9] [18] [38]. Actually, reasoning on programs is a very old idea [33], which has been known for quite a while under quite a variety of terms, the main ones being program proof and static analysis of programs. One can see the ﬁrst one as an extreme version of the second, since its goal is to guarantee full correctness of the program with respect to logical assertions, while static analysis techniques only guarantee the absence of certain types of faults (data-ﬂow analysis, alias analysis, buﬀer overﬂow, etc). But there is a signiﬁcant diﬀerence between the tools that are used: powerful theorem provers versus specialised algorithms. However, the current progresses in theorem proving, constraint solving, and invariant generation [28] [47] and the emergence of logics, such as separation logics [49] that address properties that were traditionally in the scope of static analysis, have a tendency to blur this diﬀerence. 2.3

Systems and Testing Systems

A system is a dynamic entity, embedded in the physical world. It is observable via some limited interface/procedure. It is not always controllable. It is in essence quite diﬀerent from a piece of text (formula, program) or a diagram. A program text, or a speciﬁcation, or a model, are not the system but some description of the system.

Checking Models, Proving Programs, and Testing Systems

5

The map is not the territory [35]. The only way to interact with a system, unless it comes with some sophisticated instrumentation, is to trigger its execution by giving some inputs and observing the outputs or the absence of output. When testing, the actual system is executed for a ﬁnite set of selected inputs. These executions are observed, and a decision is made on their conformance w. r. t. the expected behaviour. This expected behaviour is known via some speciﬁcation, some model, or the program. There is a classical distinction between black-box testing and white-box testing, the ﬁrst one being independent from the program, the second one exploiting the structure of the program for selecting the test inputs. Black-box testing may be completely in the black, for instance selecting the test inputs at random in the input domain, or may use a model or a speciﬁcation to perform this selection as in the subsection below. Actually this terminology is not adequate, ﬁrst because a white box is as opaque as a black one, second, and more seriously, because a system is always an opaque box, even when the program of origin is available. In this case all what can be said is that the internal system behaviour is likely to be close to the symbolic executions of the program. But nowadays, there exist sophisticated optimising compilers that are likely to falsify this assumption. Model-based testing. Using speciﬁcations or models for selecting test cases and stating the corresponding verdicts is now recognized as a major application of formal methods; we refer to [6] [10] [15] [39] among many other pioneering papers and surveys. However, embedding testing activities within a formal framework is far from being obvious. One tests a system: as said above, a system is not a formula, even if it can be (partially) described as such. Thus, testing is related to, but very diﬀerent from proof of correctness based on the program text using, for example, an assertion technique. Similarly, testing is diﬀerent from model checking, where veriﬁcations are based on a known model of the system: when testing, the model corresponding to the actual system under test is unknown. If it was known, testing would not be necessary... Moreover, it is often diﬃcult to observe the state of the system under test (observability) [15] [22] [39], and to force the execution of certain behaviours (controllability). These points have been successfully circumvented in several testing methods that are based on formal speciﬁcations or models and on conformance relations, which precisely state what it means for a system under test to satisfy a speciﬁcation or to be conform to a model [10] [21] [29] [51]. The gap between systems and models is generally taken into account by explicit assumptions on the systems under test [15] [6] [39] that are called “testability hypotheses” in [6] or “test hypotheses” in [10]. Such hypotheses are generally related to the fact that, given that model-based testing is based on a conformance relation between models, the actual system under test must be observable as some (unknown) model of the same nature as the ones considered by this relation.

6

M.-C. Gaudel

Similarly, when selecting a ﬁnite subset of test cases, there is an assumption that from the success of the associated ﬁnite test set one can extrapolate the success of exhaustive testing and the conformance of the system under test. Such testability and test assumptions are fundamental in the proof that the success of the test set selected from a model or a speciﬁcaton establishes the conformance relation. They can be seen as providing either proof obligations or hints on complementary tests, which are required to ensure conformance.

3

Using Each other Methods: Cross-Fertilisation

This section gives a few examples of approaches where methods and techniques developed for one of the activities presented above are used for a diﬀerent purpose: model-checking is used for test generation; some test methods are used for approximate model-checking; some test generators are based on theorem-provers. This section just aims at giving a sample of such approaches in the framework sketched in Section 2. It does not pretend to be comprehensive and could easily be extended. 3.1

Using Model-Checkers For Test Generation

When using model-checkers for test generation, the basic idea is to exploit the fact that model-checkers can yield counter-examples [30] [42]. Given a model M of the system under test and φ a required property, model-checking M for ¬φ yields a counter-example, i.e. a trace of M that satisﬁes φ. This trace can be used as a basis for a test sequence to be submitted to the system under test It is a popular approach: most model-checkers have been experienced for test generation and even customised for that [1], and many success stories have been reported. However, even if it brings much, it raises some new issues and does not solve ... some old ones: – φ must be a formula of some temporal logic: it is often diﬃcult to express realistic properties; – for some formulas, for instance those that are universally quantiﬁed, one test sequence is not enough: one is faced up to good old issues like exhaustivity, test selection, and ﬁnally assessment of test selection strategies; – most the times, as M is ﬁnite, it gives an over-approximation of the system under tests. It means that some traces of the model may not be executable by this system, raising the well-known issue of feasibility of the test sequences. 3.2

Using Test Methods for Model-Checking

The big challenge of model-checking is the enormous sizes of the models. Even when the best possible abstractions and restrictions methods have been applied, it may be the case that the remaining size is still signiﬁcantly too large to perform exhaustive model explorations. As seen in Section 2.3, testing is by essence a non exhaustive activity. Giving up the idea of exhaustivity for model-checking leads

Checking Models, Proving Programs, and Testing Systems

7

to the idea of using test selection methods for limiting the exploration of models. However, it is of ﬁrst importance to assess in a qualitative or quantitative way the approximation or the incompleteness induced by the selection method. One of these methods is randomisation of the search algorithm used for model exploration. Random exploration of models is a classical approach in simulation and testing. A ﬁrst transposition into model-checking has been described and implemented in [27] as a Monte-Carlo algorithm for LTL model-checking. The underlying random exploration is based on a classical uniform drawing among the transitions starting from a given state. The drawback of such random explorations is that the resulting distribution of the exploration paths is dependent on the topology of the model, and some paths may have a very low probability to be traversed. An improvement has been recently proposed in [44], which is more expensive in memory, but provides a uniform random generation of lassos, which are the kind of paths of interest for LTL model-checking. It maximises the minimal probability to reach a counter-example, and makes it possible to state a lower bound of this probability after N drawings, giving an assessment of the quality of the approximation. Another approach of model-checking that presents some similarity with testing is bounded model-checking [8]. It limits the length of the paths explored by the model-checker. In bounded model checking, some upper bounds on the execution paths to search for some error is stated for some class of formulas. In practice, one progressively increases the bound, looking for counter-examples in longer and longer execution paths. For every ﬁnite transition system M and LTL formula φ, there exists a number k such that the absence of errors up to k proves that M |= ∀φ. k is called the completeness treshold of M with respect to φ. Thus, the method is complete when this threshold is reached, but incomplete if the bound cannot be high enough for instance because there is not enough resources available to reach k. 3.3

Using Theorem Provers for Test Generation

Speciﬁcation-based testing often requires sophisticated reasoning on the speciﬁcation and the test purpose. An example of a speciﬁcation and test case generation environment based on theorem proving is HOL-TestGen [11] [12]. HOL-TestGen is an extension of Isabelle/HOL. It makes it possible both to generate test cases and to make explicit the associated test hypotheses [6] [22]. Starting from a test speciﬁcation, which is a property to be tested and a program under test, the system decomposes it into some normal form called a test theorem. The test theorem indicates a list of test cases and their associated hypotheses that implies the test speciﬁcation. The meaning of the test theorem is “if the program under test passes one instance of all test cases and if it satisﬁes the test hypotheses, it is correct with respect to the test speciﬁcation”. Thank to the extensibility of Isabelle/HOL and its large collection of theories, this environment allows to deal with several sorts of programs and various

8

M.-C. Gaudel

logics. Using theorem proving techniques for simplifying test speciﬁcations can improve dramatically the eﬃciency of test generation, and reduce the number of generated tests. Moreover, this environment provides more than a possibility to use theorem proving for test generation: the test hypotheses yielded by the decomposition can be seen as proof obligations associated to the generated tests. This approach actually supports complementarity of testing and proving.

4

What Is Static, What Is Dynamic?

It is a well-known distinction. Static program analysis methods extract useful information from the text of a program, without performing any execution. For instance data ﬂow analysis records the dependencies between deﬁnitions and uses of variables. Due to some undecidability results, these methods generally produce a cautious over-approximation, i. e. they may signal potential problems that do not correspond to actual behaviours. On the contrary, dynamic methods perform program executions on some test inputs and draw some conclusions from the observed behaviours, i.e. they are system testing activities. These two kinds of methods are often interdependent, static analysis being used for selecting test cases that are likely to produce interesting behaviours (see for instance [48]). However, in some cases the distinction is not so clear. Recently, several methods, that combines program executions and on-line veriﬁcations have blurred the border. 4.1

Symbolic Executions

The sort of static program analysis that is probably the closest to dynamic methods is symbolic execution [34]. It consists in symbolically interpreting the program text, starting with symbolic values to the inputs, and representing the successive values of program variables as symbolic expressions. The state of a symbolically executed program is composed of the symbolic expressions associated with the program variables, a path predicate and a program location. A symbolic path is a path in the control ﬂow graph of the program where the vertices are decorated by the successive symbolic expressions associated with the program variables. The path predicate is the conjunction of the accumulated conditions, or their negations, encountered when symbolically traversing the path. The program location indicates the next place to be considered in the text of the program. The set of symbolic execution of a program can be characterised by a so-called symbolic evaluation tree, where the nodes are the states as deﬁned above and the links record transitions between states. Symbolic executions can be used for some static veriﬁcation. However, there is often an explosion of the number of symbolic paths, and in the presence of loops the execution tree may be inﬁnite. It has been rather used as a static preliminary to testing, exploiting the fact that the symbolic execution tree represents all the potential actual executions of the programs in a way that naturally induces for each path the characterisation of those inputs that provoke its execution. But a serious issue is that symbolic execution yields more paths than the actual ones...

Checking Models, Proving Programs, and Testing Systems

9

Unfeasible paths. A symbolic path is feasible if its predicate is satisﬁable, i.e. there exist some input values that ensure its execution by the system under test. Unfeasibility of a symbolic path results from the presence of contradictory conditions among the ones accumulated when traversing it. The existence of infeasible symbolic paths is an habitual problem for all static analysis techniques and structural testing methods. There is no general algorithm allowing to identify them since the satisﬁability problem of the kind of predicate to be considered is known to be undecidable [53]. Depending on the kind of formula and expression allowed in the program, diﬀerent constraint solvers may be used to check feasibility and to eliminate some classes of clearly infeasible symbolic paths. Dealing with unfeasible paths or traces remains a challenge for static analysis methods of programs (or models or speciﬁcations). There are currently spectacular advances in constraint solving, which open new perspectives for the static detection of unfeasibilities. Moreover, some attempts at mixing static and dynamic techniques have been developed to cope with this problem. 4.2

Concolic Testing

Concolic testing combines actual execution of the system under test with symbolic execution of the program. The system is instrumented in order to record the symbolic path followed by some actual execution and gathering the corresponding path predicate. This yields some deﬁnitely feasible symbolic path that is used as a starting point to build other feasible paths, exploiting the simple idea that the preﬁx of a symbolic feasible path is feasible and may be extended. Some examples of tools based on this principle are described in [54] [50] [25]. The discovery of new feasible paths is made by backtracking in the path predicate, i.e. negating the last encountered condition, or removing it and adding one, using the program text. The generation of the new test inputs is obtained by constraint solving and makes use of the fact that unfeasibility, if any, is due to the last modiﬁed condition. It must be noted that in presence of loops the method may not terminate, since it explores in some way the symbolic execution tree, and the problem of the explosion of the number of paths is still there. This approach is an example of strong integration of dynamic and static methods. 4.3

Runtime Verification

Depending on the context, runtime veriﬁcation is a revival of passive testing [40] or of self-checking components, either in hardware or in software [2]. This revival is due to the successful use of some variants of LTL formula as a basis for the generation of controllers or monitors. Runtime veriﬁcation deals with execution traces of the considered system. During normal operations of the system, these traces are either recorded and checked a posteriori, or observed and checked online. The aim of this activity is to check whether these executions satisfy a given property, sometimes called the speciﬁcation. This is realised by instrumentation of the system, The checks are

10

M.-C. Gaudel

performed by a component called a monitor. Monitors can work online on one trace, in interaction with the system, or oﬄine on a ﬁnite set of recorded traces. The requirements and associated techniques for these two kinds of monitor are rather diﬀerent. The main challenge of runtime veriﬁcation is the synthesis of monitors form the property to be checked. As explained in [41], the current revival of these methods has its roots in model-checking: it turns out that there exist variants and fragments of linear temporal logic from which it is possible to generate eﬃcient monitors automatically. Thus, runtime veriﬁcation is a pure dynamic activity, which takes advantage of the corpus of knowledge developed for model-checking. Besides, the principles presented above are not only useful for veriﬁcation and failure detection, but also as design methods for robust, and easy to maintain, systems.

5

Conclusion

More and more, exigencies of trust are raising for software based systems. At the same time, quite a variety of sophisticated and eﬃcient methods are appearing for the validation and the veriﬁcation of these systems. It would be very fruitful to speak a common language and to clarify the concepts and the terminology for this mine of powerful techniques and methods. This is a condition for comparisons, cross-fertilisation, and improvements. It would also save time when presenting new ideas, writing research papers and transferring results to industry. This paper sketched a very rough classiﬁcation of the main activities by subjects (model, program, system) and then gave a few examples of mixing approaches where the classiﬁcation is less clear. It could be extended very easily given the number of such approaches, and a whole class of other ones, which were not mentioned here, and rely upon controlled approximations [37], probabilities [17], rare events[20]. They are the keys for scalability in size, reliability, and dependability. A lot of work, and many mutual exchanges are required but it is clear that it is worth the eﬀort. Acknowledgments. This paper has greatly beneﬁted from comments and discussions during the summer school of the Resist European Network of Excellence held in Porquerolles in September 2007. Special thanks are due to my late friend Jean-Claude Laprie, who left us much too early.

References 1. Anand, S., Pasareanu, C.S., Visser, W.: JPF-SE: A symbolic execution extension to Java PathFinder. In: Grumberg, O., Huth, M. (eds.) TACAS 2007. LNCS, vol. 4424, pp. 134–138. Springer, Heidelberg (2007) 2. Avizienis, A., Laprie, J.C., Randell, B., Landwehr, C.: Basic concepts and taxonomy of dependable and secure computing. IEEE Trans. on Dependable and Secure Computing 1, 11–33 (2004)

Checking Models, Proving Programs, and Testing Systems

11

3. Ball, T., Podelski, A., Rajamani, S.K.: Boolean and cartesian abstraction for model checking C programs. In: Margaria, T., Yi, W. (eds.) TACAS 2001. LNCS, vol. 2031, pp. 144–152. Springer, Heidelberg (2001) 4. Barnett, M., DeLine, R., F¨ ahndrich, M., Jacobs, B., Leino, K.R.M., Schulte, W., Venter, H.: The Spec# Programming System: Challenges and Directions. In: Meyer, B., Woodcock, J. (eds.) VSTTE 2005. LNCS, vol. 4171, pp. 144–152. Springer, Heidelberg (2008) 5. Beizer, B.: Black-Box Testing: Techniques for Functional Testing of Software and Systems. John Wiley & Sons, Chichester (1995) 6. Bernot, G., Gaudel, M.C., Marre, B.: Software testing based on formal speciﬁcations: a theory and a tool. Software Engineering Journal 6(6), 387–405 (1991) 7. Beyer, D., Henzinger, T.A., Jhala, R., Majumdar, R.: The software model checker Blast: Applications to software engineering. International Journal on Software Tools for Technology Transfer (STTT) 9(5-6), 505–525 (2007) 8. Biere, A., Cimatti, A., Clarke, E.M., Strichman, O., Zhu, Y.: Bounded model checking. Advances in Computers 58, 118–149 (2003) 9. B¨ ohme, S., Moskal, M., Schulte, W., Wolﬀ, B.: HOL-boogie - an interactive proverbackend for the verifying C compiler. J. Autom. Reasoning 44(1-2), 111–144 (2010) 10. Brinksma, E., Tretmans, J.: Testing transition systems: An annotated bibliography. In: Cassez, F., Jard, C., Rozoy, B., Dermot, M. (eds.) MOVEP 2000. LNCS, vol. 2067, pp. 187–195. Springer, Heidelberg (2001) 11. Brucker, A.D., Br¨ ugger, L., Krieger, M.P., Wolﬀ, B.: HOL-TestGen 1.5.0 user guide. Tech. Rep. 670, ETH Zurich (2010), http://www.lri.fr/~ wolff/papers/other/HOL-TestGen_UserGuide.pdf 12. Brucker, A.D., Wolﬀ, B.: HOL-TestGen: An Interactive Test-case Generation Framework. In: Chechik, M., Wirsing, M. (eds.) FASE 2009. LNCS, vol. 5503, pp. 417–420. Springer, Heidelberg (2009) 13. Brucker, A.D., Wolﬀ, B.: Test-sequence generation with HOL-TestGen – with an application to ﬁrewall testing. In: Meyer, B., Gurevich, Y. (eds.) TAP 2007. LNCS, vol. 4454, pp. 149–168. Springer, Heidelberg (2007) 14. Chen, T.Y., Tse, T., Zhou, Z.Q.: Semi-proving: An integrated method for program proving, testing, and debugging. IEEE Transactions on Software Engineering 37, 109–125 (2011) 15. Chow, T.: Testing software design modeled by ﬁnite-state machines. IEEE Transactions on Software Engineering SE 4(3), 178–187 (1978) 16. Coen-Porisini, A., Denaro, G., Ghezzi, C., Pezz`e, M.: Using symbolic execution for verifying safety-critical systems. In: ESEC / SIGSOFT FSE, pp. 142–151 (2001) 17. Courcoubetis, C., Yannakakis, M.: The complexity of probabilistic veriﬁcation. J. ACM 42(4), 857–907 (1995) 18. Detlefs, D., Nelson, G., Saxe, J.B.: Simplify: a theorem prover for program checking. J. ACM 52(3), 365–473 (2005) 19. Finkbeiner, B., Gurevich, Y., Petrenko, A. (eds.): Proceedings of the Fourth Workshop on Model Based Testing, ENTCS, Budapest, Hungary. Elsevier, Amsterdam (2008) 20. Galves, A., Gaudel, M.C.: Rare events in stochastic dynamical systems and failures in ultra-reliable reactive programs. In: FTCS, pp. 324–333. IEEE, Los Alamitos (1998) 21. Gaudel, M.C., James, P.J.: Testing algebraic data types and processes: a unifying theory. Formal Aspects of Computing 10(5-6), 436–451 (1998)

12

M.-C. Gaudel

22. Gaudel, M.C.: Testing can be formal, too. In: Mosses, P.D., Nielsen, M. (eds.) CAAP 1995, FASE 1995, and TAPSOFT 1995. LNCS, vol. 915, pp. 82–96. Springer, Heidelberg (1995) 23. Geller, M.M.: Test data as an aid in proving program correctness. Commun. ACM 21(5), 368–375 (1978) 24. Godefroid, P.: Software model checking: The VeriSoft approach. Formal Methods in System Design 26(2), 77–101 (2005) 25. Godefroid, P., Levin, M.Y., Molnar, D.A.: Automated whitebox fuzz testing. In: Proceedings of the Network and Distributed System Security Symposium, NDSS 2008. pp. 151–166. The Internet Society, San Diego (2008) 26. Goldreich, O.: A brief introduction to property testing. In: Goldreich, O. (ed.) Property Testing. LNCS, vol. 6390, pp. 1–5. Springer, Heidelberg (2010) 27. Grosu, R., Smolka, S.A.: Monte Carlo Model Checking. In: Halbwachs, N., Zuck, L.D. (eds.) TACAS 2005. LNCS, vol. 3440, pp. 271–286. Springer, Heidelberg (2005) 28. Gupta, A., Rybalchenko, A.: Invgen: An eﬃcient invariant generator. In: Bouajjani, A., Maler, O. (eds.) CAV 2009. LNCS, vol. 5643, pp. 634–640. Springer, Heidelberg (2009) 29. Helke, S., Neustupny, T., Santen, T.: Automating test case generation from Z speciﬁcations with Isabelle. In: ZUM, pp. 52–71 (1997) 30. Hierons, R.M., Bogdano, K., Bowen, J.P., Cleaveland, R., Derrick, J., Dick, J., Gheorghe, M., Harman, M., Kapoor, K., Krause, P., L¨ uttgen, G., Simons, A.J.H., Vilkomir, S., Woodward, M.R., Zedan, H.: Using formal speciﬁcations to support testing. ACM Computing Surveys 41(2), 1–76 (2009) 31. Holzmann, G.: Spin Model Checker, the primer and reference manual, 1st edn. Addison-Wesley Professional, Reading (2003) 32. Hoskote, Y.V., Kam, T., Ho, P.H., Zhao, X.: Coverage estimation for symbolic model checking. In: DAC, pp. 300–305 (1999) 33. Jones, C.B.: The early search for tractable ways of reasoning about programs. IEEE Annals of the History of Computing 25(2), 26–49 (2003) 34. King, J.C.: Symbolic execution and program testing. Commun. ACM 19(7), 385–394 (1976) 35. Korzybski, A.: Science and Sanity: A Non-Aristotelian System and its Necessity for Rigour in Mathematics and Physics. Institute of General Semantics (1933) 36. Laprie, J.C.: Dependability: Basic Concepts and Terminology. Dependable Computing and Fault-Tolerant Systems. Springer, Heidelberg (1991); in English, French, German, Italian and Japanese 37. Lassaigne, R., Peyronnet, S.: Probabilistic veriﬁcation and approximation. Ann. Pure Appl. Logic 152(1-3), 122–131 (2008) 38. Leavens, G.T., Leino, K.R.M., M¨ uller, P.: Speciﬁcation and veriﬁcation challenges for sequential object-oriented programs. Formal Asp. Comput. 19(2), 159–189 (2007) 39. Lee, D., Yannakakis, M.: Principles and methods of testing ﬁnite state machines A survey. Proceedings of the IEEE 84, 1090–1126 (1996) 40. Lee, D., Netravali, A.N., Sabnani, K.K., Sugla, B., John, A.: Passive testing and applications to network management. In: ICNP, pp. 113–122. IEEE Computer Society, Los Alamitos (1997) 41. Leucker, M., Schallhart, C.: A brief account of runtime veriﬁcation. J. Log. Algebr. Program. 78(5), 293–303 (2009) 42. L¨ uttgen, G.: Formal veriﬁcation & its role in testing. Tech. Rep. YCS-2006-400, Department of Computer Science, University of York, England (2006)

Checking Models, Proving Programs, and Testing Systems

13

43. McMillan, K.: Symbolic Model Checking. Kluwer Academic Publishers, Dordrecht (1993) 44. Oudinet, J., Denise, A., Gaudel, M.-C., Lassaigne, R., Peyronnet, S.: Uniform monte-carlo model checking. In: Giannakopoulou, D., Orejas, F. (eds.) FASE 2011. LNCS, vol. 6603, pp. 127–140. Springer, Heidelberg (2011) 45. Pasareanu, C.S., Visser, W.: A survey of new trends in symbolic execution for software testing and analysis. STTT 11(4), 339–353 (2009) 46. Peled, D., Vardi, M.Y., Yannakakis, M.: Black box checking. Journal of Automata, Languages and Combinatorics 7(2), 225–246 (2002) 47. Podelski, A., Rybalchenko, A.: Transition invariants and transition predicate abstraction for program termination. In: Abdulla, P.A., Leino, K.R.M. (eds.) TACAS 2011. LNCS, vol. 6605, pp. 3–10. Springer, Heidelberg (2011) 48. Rapps, S., Weyuker, E.J.: Data ﬂow analysis techniques for test data selection. In: ICSE, pp. 272–278 (1982) 49. Reynolds, J.C.: Separation logic: A logic for shared mutable data structures. In: LICS, pp. 55–74. IEEE Computer Society, Los Alamitos (2002) 50. Sen, K., Marinov, D., Agha, G.: Cute: a concolic unit testing engine for C. In: ESEC/SIGSOFT FSE, pp. 263–272. ACM, New York (2005) 51. Tretmans, J.: A formal approach to conformance testing. In: Protocol Test Systems. IFIP Transactions, vol. C-19, pp. 257–276. North-Holland, Amsterdam (1993) 52. Vardi, M.Y., Wolper, P.: An automata-theoretic approach to automatic program veriﬁcation (preliminary report). In: LICS, pp. 332–344. IEEE Computer Society, Los Alamitos (1986) 53. White, L.J.: Basic mathematical deﬁnitions and results in testing. In: Chandrasekaran, B., Radicchi, S. (eds.) Computer Program Testing, pp. 13–24. NorthHolland, Amsterdam (1981) 54. Williams, N., Marre, B., Mouy, P., Roger, M.: Pathcrawler: Automatic generation of path tests by combining static and dynamic analysis. In: Dal Cin, M., Kaˆ aniche, M., Pataricza, A. (eds.) EDCC 2005. LNCS, vol. 3463, pp. 281–292. Springer, Heidelberg (2005)

Tests from Proofs Patrice Godefroid Microsoft Research [email protected]

Abstract. Test generation has recently become the largest application of SMT solvers as measured by computational usage. At Microsoft, the Z3 SMT solver has solved more than 200 million constraints over the last two years as a component of the whitebox fuzzer SAGE. Whitebox fuzzing extends dynamic test generation based on symbolic execution and constraint solving from unit testing to whole-application security testing. Since 2009, SAGE has been running non-stop on (average) 100+ machines automatically “fuzzing” hundreds of applications in a dedicated lab owned by the Microsoft Windows security test team. In the process, SAGE found many new security vulnerabilities (missed by blackbox fuzzing and static program analysis) and was credited to have found roughly one third of all the bugs discovered by file fuzzing during the development of Microsoft’s Windows 7, saving millions of dollars by avoiding expensive security patches to nearly a billion PCs. In the second part of the talk, I will present a new form of test generation, named higher-order test generation, where imprecision in symbolic execution is represented using uninterpreted functions in logic path constraints. I will explain why such functions need be universally quantified, hence requiring tests to be generated from validity proofs of first-order logic formulas, rather than from satisfiability proofs of quantifier-free first-order logic formulas as usual.

M. Gogolla and B. Wolﬀ (Eds.): TAP 2011, LNCS 6706, p. 14, 2011. c Springer-Verlag Berlin Heidelberg 2011

Incorporating Coverage Criteria in Bounded Exhaustive Black Box Test Generation of Structural Inputs Nazareno M. Aguirre1 , Valeria S. Bengolea1, Marcelo F. Frias2 , and Juan P. Galeotti3 1

2

Departamento de Computaci´ on, FCEFQyN, Universidad Nacional de R´ıo Cuarto and CONICET, R´ıo Cuarto, C´ ordoba, Argentina {naguirre,vbengolea}@dc.exa.unrc.edu.ar Departamento de Ingenier´ıa Inform´ atica, Instituto Tecnol´ ogico Buenos Aires and CONICET, Buenos Aires, Argentina [email protected] 3 Departamento de Computaci´ on, FCEyN, Universidad de Buenos Aires and CONICET, Buenos Aires, Argentina [email protected]

Abstract. The automated generation of test cases for heap allocated, complex, structures is particularly diﬃcult. Various state of the art tools tackle this problem by bounded exhaustive exploration of potential test cases, using constraint solving mechanisms based on techniques such as search, model checking, symbolic execution and combinations of these. In this article we present a technique for improving the bounded exhaustive constraint based test case generation of structurally complex inputs, for “ﬁltering” approaches. The technique works by guiding the search considering a given black box test criterion. Such a test criterion is incorporated in the constraint based mechanism so that the exploration of potential test cases can be pruned without missing coverable classes of inputs, corresponding to the test criterion. We present the technique, together with some case studies illustrating its performance for some black box testing criteria. The experimental results associated with these case studies are shown in the context of Korat, a state of the art tool for constraint based test case generation, but the approach is applicable in other contexts using a ﬁltering approach to test generation.

1

Introduction

Testing is a powerful and widely used technique for software quality assurance [6]. The technique essentially consists of executing a piece of code, whose quality needs to be assessed, under a number of particular inputs, or test cases. For these test cases to be adequate, they generally need to try the software under diﬀerent circumstances. A variety of test criteria have been devised, which basically deﬁne what are the diﬀerent situations that a set of test cases must exercise, or cover [17]. M. Gogolla and B. Wolﬀ (Eds.): TAP 2011, LNCS 6706, pp. 15–32, 2011. c Springer-Verlag Berlin Heidelberg 2011

16

N.M. Aguirre et al.

Generating test cases is generally a complex activity, in which the engineer in charge of the generation has to come up with inputs that satisfy, in many cases, complex constraints. The problem is particularly diﬃcult when the inputs to be generated involve complex, heap allocated, structures, such as balanced trees, graphs, etc. Some tools [2,16,7,10,4] tackle this problem rather successfully, by bounded exhaustive exploration of potential test cases. More precisely, these tools work by generating all the inputs, within certain bounds (maximum number of objects of each of the classes involved in the structure), that satisfy a given constraint using some kind of constraint solver. Among the possible constraint solving techniques, model checking and other search related mechanisms have been implemented into state of the art tools. In order to make the bounded exhaustive generation feasible, diﬀerent mechanisms are implemented so that, in some cases, redundant structures are avoided, and such that parts of the state space corresponding to invalid structures are not explored. For instance, Korat implements a symmetry breaking mechanism together with an approach for avoiding the generation of invalid structures based on a sophisticated pruning technique; TestEra uses Alloy and its underlying symmetry breaking and optimisation mechanisms to improve the generation; UDITA implements a novel lazy evaluation mechanism, which in combination with symbolic execution greatly improve the test generation process. In this work, we consider a complement to the so called filtering approach [4] to bounded exhaustive test generation, i.e., the process of exhaustively generating all possible structures (within the established bounds) and “ﬁltering” to keep the valid or well formed ones. This complement takes into account a black box test criterion as part of the “generate and ﬁlter” process. Basically, in the same way that symmetric structures are avoided, we propose to also avoid the exploration of portions of the search space for test input candidates when such portions are guaranteed to provide test inputs corresponding to classes already covered by other test inputs previously generated. The result is somehow in between bounded exhaustive and “optimal” equivalence class coverage, and the actual “exhaustiveness” of the technique depends on the interaction of the test criterion (e.g., the adequacy of the predicates used for equivalence class coverage) and the generation procedure. The motivation for this work lies in the fact that, by bounded exhaustive generation of test cases, in many cases, it becomes costly or infeasible to test a piece of code for all valid bounded inputs, even for small bounds, due to the large number of inputs obtained. Thus, a test criterion might be employed in order to “prune” the test generation, achieving a bounded exhaustive coverage of equivalence classes associated with the criterion. For instance, suppose that one counts with a black box test criterion for a given program to test. If one is interested in equivalence class coverage, it would be enough to generate a single test input per each (feasible) equivalence class. On the other hand, by bounded exhaustive generation one would be building all valid structures within the provided bounds. Instead, we propose to do a kind of exhaustive generation, but exploiting the possibility of pruning parts of the search space when one

Incorporating Coverage Criteria

17

is certain that all candidates in the pruned part correspond to classes already covered. More precisely, our proposed technique works based on the following observation. The test generation mechanisms that follow a bounded exhaustive, ﬁltering approach, generally contain a process for avoiding the generation of redundant structures. Independently of how such processes are implemented, they all correspond essentially to a pruning operation. For instance, the symmetry breaking formulas that TestEra incorporates in the Alloy model resulting from a program to be tested, instruct the underlying SAT solver to skip parts of the search space (in this context, assignments to propositional variables), thus constituting a pruning [7]. Similarly, UDITA prunes the search space to avoid isomorphic structures by incorporating isomorphism avoidance into the object pool abstraction and the operations for obtaining new objects from it in the construction of heap allocated structures [4]; Korat performs a similar pruning, imposing an ordering in the objects of the same type in the process of building the heap allocated structures [2]. Our approach proposes to take advantage of such pruning processes but in a diﬀerent way; instead of just eliminating isomorphic (redundant) structures, we propose to take extra advantage of the pruning, and use it for skipping portions of the search space that would produce test cases covering classes that have already been covered by previous tests. We present the approach by implementing it as a variant of the Korat algorithm/tool [10]. This variant is based on the use of a routine that we call eqClass(), that given a (valid) test input indicates which is the equivalence class it corresponds to, according to a test criterion. Basically, our variant of Korat, which we will refer to as Korat+1, works as follows: when a candidate is found to be a valid test case, we invoke the eqClass() routine for this candidate, and look at what ﬁelds are observed to determine its equivalence class. We then try to “skip” the structures that coincide, for the observed ﬁelds, with the current candidate. Clearly, for any other candidate with the same values in these observed ﬁelds, its equivalence class would be the same as that of the current candidate. We describe our approach in detail, via the mentioned variant of Korat. We provide examples and case studies together with their associated experimental results, using some black box test criteria. As it will be shown later on, our variant results in signiﬁcant improvements, compared to Korat’s search, for some of our case studies. As it is further explained later on, in some cases we were able to reduce the search space substantially, as well as to produce signiﬁcantly less test cases, compared to bounded exhaustive generation. As we mentioned before, the technique results to be between bounded exhaustive, and “optimal” equivalence class coverage. 1

Korat+ is only a variant of Korat that we present in this paper, for illustrating our technique. The name Korat+ is used only for reference purposes in the presentation of our approach. We fully acknowledge that the Korat tool and algorithm, as well as the name Korat, are the intellectual property of the authors of [2], and the technique we present, which could be applied in other contexts, should not be associated with the name Korat+.

18

2

N.M. Aguirre et al.

Preliminaries

In this section we describe the Korat algorithm, which we use to present our technique. We also introduce a motivating example to drive the presentation. 2.1

The Korat Algorithm

Korat is an algorithm, and a tool implementing it, that allows for the generation of test cases composed of complex, heap allocated, structures [2]. Suppose, for instance, that we need to test a procedure that takes as a parameter a sorted singly linked list. Let us consider the following deﬁnition of the structure of sorted singly linked lists (a variant of SinglyLinkedList, as provided in Korat’s distribution): class SortedSinglyLinkedList { Node header; int size; }

class Node { Integer elem; Node next; }

A list is composed of a reference to a header node, and an integer value indicating the length of the list. The linked list of nodes starts with a header node with no element (a traditional dummy node); the actual contents of the list starts from the second node. The list should be acyclic, sorted (disregarding the header node), and the number of nodes in it minus one (the header) should coincide with the size value of the list. Korat can be used to generate such lists automatically, so that the procedure under analysis can be tested. Korat requires two routines accompanying the class associated with the input (in our example, SortedSinglyLinkedList). One of them is a boolean parameterless routine, called repOk() [8], that checks whether the structure satisﬁes its representation invariant. In our case, repOk() should check that the header is not null, and that no element is stored in it, that the list is acyclic and sorted, and the number of nodes in it minus one coincides with the value of the size ﬁeld, as we explained before. The other routine that Korat requires is a finitisation procedure, that provides the bounds for the domains involved in the structure. This routine indicates the range for primitive type ﬁelds (e.g., that size in SortedSinglyLinkedList goes from 0 to 3), and the minimum/maximum number of objects of the classes involved in the structure (e.g., 1 list, 0 to 4 nodes, 1 to 3 integer objects). Korat generates all possible valid structures within the provided bounds. By valid we mean that they satisfy the repOk() routine. For our example, this means that Korat will generate all acyclic sorted singly linked lists with dummy header, where the size coincides with the number of nodes in the linked list minus one, of size at most 3, containing integers from 1 to 3. In order to do so, Korat builds a tuple, where each entry corresponds to a value of a ﬁeld of the involved objects. In our example, the tuple would have length 10, two values for the header and size of the lone list object, and the other 8 for the corresponding two ﬁelds of the four nodes that the list might at most contain. For instance, the following tuple

Incorporating Coverage Criteria

19

0, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL would represent the empty list (where the ﬁrst zero in the tuple is the reference to the ﬁrst node object). Each entry in this tuple has a domain, which is deﬁned by the ﬁnitisation procedure. Korat’s actual algorithm works on what are called candidate vectors, vectors that represent the candidate tuples, but where the actual entries are replaced by indices into the respective domains. For instance, the candidate vector 1, 0, 0, 0, 0, 0, 0, 0, 0, 0 would correspond to the previously shown candidate tuple (each tuple entry has the ﬁrst possible value in its domain, i.e., the value with index 0, except for the ﬁrst entry, the head of the list, which points to the ﬁrst node). Typically, most of the candidate vectors correspond to invalid structures, i.e., structures that do not satisfy the repOk(). Indeed, the space of candidates is in our example 3,200,000 (55 × 45 ), but there exist only 8 singly sorted linked lists (up to isomorphism, as it will be explained later on), within the provided bounds. Korat exhaustively explores the space of candidate vectors, using backtracking with a sophisticated pruning mechanism. More precisely, Korat works as follows: it starts with the initial candidate vector, with all indices in zero. It then executes repOk() on this candidate, monitoring the ﬁelds accessed in the execution, and storing these in a stack. Korat will then use this stack in order to backtrack over candidate vectors, as follows. If the current candidate satisﬁes repOk(), it is considered a valid test case (in this case, all reachable ﬁelds must be in the stack of accessed ﬁelds). If repOk() fails, then the candidate is discarded. In order to build the next candidate, Korat increments the last accessed ﬁeld to its next value. If one or more of the last accessed ﬁelds are already in their corresponding maximum values, then these are resetted to 0, and the ﬁeld accessed before them is incremented. If all ﬁelds are already at their maximum values, then the state space of candidate vectors has been exhaustively explored, and Korat terminates. Notice that when repOk() fails, not all reachable ﬁelds might have been accessed, since its failure might be determined before exploring all reachable ﬁelds (for instance, in our example, if the ﬁrst two nodes of the list are unordered, then repOk() fails without the need to explore the remaining part of the structure). Backtracking only on accessed ﬁelds is what enables Korat to prune large parts of the space of candidate vectors. It is sound since if the last accessed ﬁeld is not modiﬁed, the output for repOk() would not change due to its determinism, (i.e., the parts of the structure visited by repOk() would remain the same, and therefore repOk() would fail again). Besides the described search mechanism, with its incorporated search pruning, Korat also avoids generating isomorphic candidates [2]. Basically, two candidates are isomorphic if they only diﬀer in the object identities of their constituents (i.e., if one of the candidates can be obtained from the other by changing the object identities). Most applications do not depend on the actual identities of objects (which represent the memory addresses or heap references of objects), and thus if one generated a structure, it is desirable to avoid generating its isomorphic structures, whose treatment would be redundant. Korat avoids generating isomorphic candidates by deﬁning a lexicographic order between candidate vectors, and gen-

20

N.M. Aguirre et al.

erating only the smallest in the order, among all isomorphic candidates. Basically, when considering the range of a class-typed ﬁeld (i.e., its possible values) in the construction of candidates during the search, it is restricted to up to one “untouched” (i.e., not previously referenced in the structure) object of its corresponding domain. For example, suppose that in the construction of candidates one needs to consider diﬀerent values for a given position i in the candidate vector. Suppose further that the ith position corresponds to a class domain D, and no ﬁelds of that domain have been accessed before i in the last invokation of repOk(). Then the only possible value for the i-th position is 0. More generally, if k objects of domain D have been accessed before in the last invokation of repOk(), these must be indexed 0 to k − 1, and thus the ith position can go from 0 to k, but not beyond k. Korat’s pruning and isomorphism elimination mechanisms allow the tool to reduce the search space signiﬁcantly, in many cases. For our example, for instance, Korat explores only 319 out of the 3200000 possible cases, for linked lists with length 0 to 3, up to 4 nodes, and values ranging in integer objects from 1 to 3. For more details, we refer the reader to [2,10].

3

Incorporating Black Box Coverage to Bounded Exhaustive Search

In this section, we describe our proposal for improving a ﬁltering approach to bounded exhaustive generation, by incorporating pruning associated with test criteria. Essentially, the approach is based on the observation that in many cases, the number of valid test cases, bounded by a value k, can be too large even for small bounds, and therefore evaluating the software under all these cases might be impractical. Then, our intention is to skip the generation of some test cases; the idea is to avoid generating test cases whose corresponding equivalence classes, for the test criterion under consideration, have already been covered. The idea is not to do “optimal” equivalence class coverage (one per equivalence class), but to approximate somehow to bounded exhaustive generation. That is, we would like to do a kind of bounded exhaustive generation, but with some pruning based on what the test criterion provides as information. We present the approach by implementing it as a variant of the Korat algorithm, introduced in the previous section. In the same way that Korat requires an imperative predicate repOk(), we require a routine that we call eqClass(). This routine returns, given a valid candidate (i.e., a candidate satisfying repOk()), the equivalence class the candidate corresponds to, according to a test criterion. As for repOk(), this routine must be deterministic for exactly the same reason that repOk() must be deterministic. As opposed to Korat, which prunes (advances various candidates at once) only when repOk() fails, since if it does not fail all reachable ﬁelds must be in the stack of accessed ﬁelds, we prune the search space both when repOk() succeedes and when it fails: when repOk() fails, we advance various candidates at once using the fact that if none of the accessed ﬁelds is changed, then repOk() would fail again. When repOk()

Incorporating Coverage Criteria

21

succeedes, we execute eqClass() and monitor the accessed ﬁelds; we then advance various candidates at once to force a change in the last accessed ﬁeld, since if none of the accessed ﬁelds changes the equivalence class would be the same of the previous valid candidate, which is already covered. In order to better understand how this mechanism works, let us brieﬂy expand our example. Suppose that we need a procedure listAsSet that, given a list l and a set s, both implemented over linked lists, determines whether s is the result of converting l to a set, i.e., disregarding repetitions and the order of elements in the list. From an implementation point of view, and taking into account the representation invariant of sets over singly linked lists, s should be the result of removing repetitions and sorting the list l. It is not diﬃcult to ﬁnd contexts in which a procedure of the kind of listAsSet is relevant. An obvious application of such a function would be an oracle for checking whether a list-to-set routine works as expected. If we want to generate test cases for listAsSet, we need to provide two objects, namely an arbitrary (acyclic) singly linked list of integers (the list l), and a strictly sorted acyclic singly linked list (the “set” s); the repOk() routine for this pair of objects checks ﬁrst whether l is acyclic, and if so, it then checks whether s is acyclic and strictly sorted. Korat’s candidate vectors will be composed of values for the ﬁelds of all objects of the two lists. Moreover, suppose that our test criterion takes into consideration all the combinations of four predicates: – – – –

the the the the

ﬁrst list is empty ﬁrst list has repeated elements ﬁrst list is sorted second list is empty

The criterion is satisﬁed if at least one test case is produced for each of the satisﬁable combinations of the truth values for the above predicates. Now, suppose that, in the search for valid candidates, Korat constructs the following pair of lists (for the linked list and the “set”, respectively): N0 header

N0’ header’

N1’ 1

Let us refer to this pair of lists as (l,s). Clearly, (l,s) satisﬁes repOk(). Let us analyse how Korat would proceed. Since repOk() is satisﬁed, Korat will move to the next candidate, which corresponds to advancing the last accessed ﬁeld, i.e., N1’.next, assuming that repOk() checks the representation invariants of l and s in this order. Furthermore, because of the way repOk() works, Korat will produce all valid sets “greater than” (in the sense of the order in which Korat produces them) s, in combination with the empty l, before advancing l, i.e., producing a nonempty list. Now let us analyse how our approach would proceed. According to the test criterion described before, this pair of lists corresponds to the equivalence class

22

N.M. Aguirre et al.

T, F, T, F (i.e., the ﬁrst list is empty, with no repeated elements and sorted, whereas the second one is nonempty). In order to determine the equivalence class for this test case, the parts of the structure that are examined are header (N0), N0.next, header’ (N0’), N0’.next, in this order. Thus, if none of these ﬁelds are modiﬁed, the candidates produced would correspond to the same equivalence class as our current candidate (l,s), which is already covered by this valid candidate. So, the approach “prunes” the search by attempting to advance the last accessed ﬁeld, namely N0’.next, which is already at its maximum possible value (due to the rule of “at most one untouched object”). We move then to trying to advance header’, which again is at its maximum for the same reason as before, and thus we start considering greater values for N0.next. Notice how we avoided generating many (nonempty) sets, which in combination with the empty list would cover an already covered equivalence class. For example, if the ﬁnitisation procedure establishes that both lists have 0 to 4 nodes, and integers go from 1 to 3, then the described pruning, associated with our approach, constructs 679 candidates (320 of which are valid), skipping the construction of 14000 candidates (240 of which are valid, but cover already covered classes) that Korat would generate. 3.1

Soundness of the Approach

Let us argue about the soundness of the approach with respect to equivalence class coverage, i.e., that any valid test case in the pruned state space corresponds to a previously covered equivalence class. For comparison purposes, let us introduce a pseudo-code description of the standard Korat algorithm: function korat() { Vector curr = initVector; Stack fields = new Stack(); boolean ok; do { (ok, fields) = curr.repOk(); if (ok) { reportValid(curr); fields.push(curr.reachFields - fields); } field = fields.pop(); while (!fields.isEmpty() && curr[field] >= nonIsoMax(curr, fields, field)) { curr[field] = 0; field = fields.pop(); } if (!fields.isEmpty()) curr[field]++; } while (curr != lastVector && !fields.isEmpty()) }

In this pseudo-code description of the algorithm, we make an abuse of notation and make repOk(), which applies to candidate vectors, return both the

Incorporating Coverage Criteria

23

result of executing this function on the corresponding vector (a boolean, indicating whether the candidate is a valid one or not) and a stack with the ﬁelds accessed in the execution (fields). Notice how the backtracking is performed on the ﬁelds accessed by repOk(); also, when the current vector is valid, then all reachable ﬁelds are forced into the accessed ﬁelds, so that these are considered for backtracking and no candidates are missed. Finally, notice that an auxiliary function called nonIsoMax, which returns the maximum index possible for a given ﬁeld, is used in order to determine the range of values for each ﬁeld. This is crucial for the generation of nonisomorphic instances [2]. Our technique, which in this context we present as a variant of Korat referred to as Korat+, performs an extra pruning. It works by “popping out” more items from fields, the stack of accessed ﬁelds. In order to perform this pruning, the algorithm needs to compute the equivalence class for each valid candidate, monitoring the ﬁelds accessed in this computation (stored in eqFields). It then checks whether Korat’s standard “next candidate” computation already advanced some of the ﬁelds accessed by the eqClass() routine, and if not it forces such an advance. The pseudo-code for our variant is the following: function koratPlus() { Vector curr = initVector; Stack fields = new Stack(); boolean ok; do { (ok, fields) = curr.repOk(); if (ok) { reportValid(curr); fields.push(curr.Fields - fields); (eqClass, eqFields) = curr.eqClass(); reportEqClass(eqClass); } List modified = new List(); field = fields.pop(); while (!fields.isEmpty() && curr[field] >= nonIsoMax(curr, fields, field)) { curr[field] = 0; modified.add(field); field = fields.pop(); } if (!fields.isEmpty()) { curr[field]++; modified.add(field); } // extra pruning if (ok && (eqFields - modified == eqFields)) { for each field in modified { curr[field] = 0 } boolean found = false;

24

N.M. Aguirre et al. while (!fields.isEmpty() && !found) { field = fields.pop(); if (eqFields.contains(field)) { found = true; } else { curr[field] = 0; } } if (found) { while (!fields.isEmpty() && curr[field] >= nonIsoMax(curr, fields, field)) { curr[field] = 0; field = fields.pop(); } if (!fields.isEmpty()) { curr[field]++; } } } } while (!fields.isEmpty())

}

Guaranteeing the soundness of this pruning approach is relatively straightforward. First, notice that the backtracking order of the original Korat algorithm is preserved: Korat+ backtracks over fields, the ﬁelds accessed by repOk(). Our variant can only “pop” more items, but not modify the accessed ﬁelds (and thus the order of backtracking) in any other way. Let us see that this new pruning can only skip valid candidates of already covered classes. Suppose that this new pruning stage is activated. Then, the previous candidate, which we will refer to as vp , is a valid candidate, since ok is true; moreover, the standard Korat computation of the next candidate did not modify any of the ﬁelds accessed by eqClass(). This last pruning stage modiﬁes the last ﬁeld, according to fields, appearing in eqFields. Let v be a candidate vector pruned by this process. Assume further that v is a valid candidate. Since this candidate was pruned in this extra pruning, it corresponds to the pruned search space, which coincides in its values of the eqFields with vp . Then, v corresponds to the same test equivalence class as vp , due to the determinism of eqClass(). Therefore, the candidates pruned in the extra pruning stage correspond to the same equivalence class of vp , which has already been covered by this test case.

4

Case Studies

We now describe some of the case studies we selected for assessing the technique. At the end of this section we will brieﬂy analyse the experimental results associated with these case studies.

Incorporating Coverage Criteria

25

listAsSet. Our ﬁrst case study corresponds to the listAsSet routine, and the black box test criterion described before, which requires covering all combinations of the predicates “ﬁrst list is empty”, “ﬁrst list has repeated elements”, “ﬁrst list is sorted”, and “second list is empty”. The repOk() and eqClass() routines have been implemented as described in Section 3. This is a simple case study, with few equivalence classes, but it is still an interesting “toy” case study which serves the purpose of showing the beneﬁts of the technique. The experimental results are shown in the table below. The scope indicates the size ranges for the two lists (as separated scopes), the maximum number of nodes in each list (as separated scopes), and the number of diﬀerent integer values allowed, respectively. For Korat and Korat with coverage pruning (Korat+), the table shows the number of explored vectors, together with how many of these are valid test cases (i.e., satisfying repOk()). We also indicate the time taken by both algorithms, and the number of classes covered (CC) for the corresponding scope (the covered classes are the same for Korat and Korat+, due to the soundness of the technique). Scope Korat Korat+ Time Korat Time Korat+ CC 0-2,0-2,3,3,3 1,121(91) 185(26) 0.331s 0.249s 8 0-4,0-4,3,3,3 1,485(91) 211(26) 0.277s 0.241s 8 0-4,0-4,4,4,3 14,679(320) 679(80) 0,422s 0,269s 10 0-5,0-5,5,5,4 1,274,977(5,456) 6,798(682) 2,39s 0,395s 10 0-5,0-5,5,5,5 6,692,357(24,211) 16,369(1,562) 10.961s 0.586s 10 0-7,0-7,7,7,6 - 1,453,804(111,974) TIMEOUT 3,36s 10

Binomial Heap (Merge). Our second case study consists of generating test data for binomial heaps. A fundamental operation of binomial heaps is the merge of two heaps, which can be performed very eﬃciently for this structure. Assuming one is interested in testing such a routine, it is necessary to provide pairs of binomial heaps. The merging of two binomial heaps depends very much on how these are composed, and the degrees of their composing binomial trees. Considering equivalence class partitioning as the black box test criterion, the following predicates should provide a suitable coverage: – – – – – – –

the ﬁrst heap is empty, the second heap is empty, the ﬁrst heap has more elements than the second, both heaps have the same number of elements, the ﬁrst heap has a larger degree than the second, both heaps have the same degree, and the heaps contain a tree with the same degree.

We have used the implementation of binomial heaps, with its corresponding repOk(), exactly as provided in the Korat distribution, replicated for the two binomial heaps. The domains for each of these have been deﬁned disjoint, in the ﬁnitisation procedure. The experimental results regarding this case study are shown in the table below. The scope indicates the maximum number of elements both heaps might have. The keys in the nodes range from zero to this value

26

N.M. Aguirre et al.

(repeated elements are allowed). The table shows the number of explored vectors, together with how many of these are valid test cases. We also indicate the number of equivalence classes covered, and the times taken by the two algorithms. Scope Korat Korat+ Time Korat Time Korat+ 2 348(36) 147(15) 0,42s 0,394s 3 5,389(784) 1,315(56) 0,656s 0,454s 4 150,448(14,400) 46,786(435) 1,436s 0,86s 5 3,125,314(876,096) 647,410(1,872) 16,347s 2,492s 6 274,808,123(57,790,404) 55,745,855(43,134) 1360,323s 178,072s

CC 6 10 10 10 10

Directed Graphs. Our third case study corresponds to generating test cases for a routine manipulating a directed graph. The implementation of directed graphs is a standard object oriented implementation, consisting of a vector of vertices, each of which has a corresponding strictly sorted linked list, its adjacency list. Suppose that one is interested in generating case studies of varied arc “densities” and covering border cases; so, the combined graph characteristics considered for equivalence class partitioning could be the following: – emptiness, – density, and – completeness. The experimental results for this case study are shown in the table below. The scope indicates the exact number of nodes in the directed graph. As for the previous cases, the table shows the number of explored vectors, together with how many of these are valid test cases, the number of classes covered, etc. Notice that the number of valid cases grows too quickly, preventing us from reporting results for scopes higher than 3. Scope Korat Korat+ Time Korat Time Korat+ CC 2 1,624(382) 518(126) 0,343s 0,265s 4 3 372,861,255(47,672,840) 11,670,154(899,852) 1145,341s 37,854s 4

Weighted Directed Graphs. Our fourth case study extends the previous one, to generating test cases for weighted directed graphs. The graph implementation is an extension of the one described above, in which each entry in the adjacency list of a vertex has a corresponding weight. Some typical algorithms on weighted directed graphs are calculations of transitive closure or minimal path information, as for instance using Floyd’s algorithm. From the deﬁnition of minimal path some representative equivalence classes can be deﬁned, based on properties of the graph: – acyclicity, – presence of negative weights, and – connectedness of the graph. They all play signiﬁcant roles in the calculation of transitive closure or minimal path information. Thus, these are adequate predicates to consider for equivalence

Incorporating Coverage Criteria

27

class coverage. In order to also get cases of varied arc “densities” and cover border cases, we also take into account emptiness, density and completeness of the structure, as for the previous case study. The experimental results for this case study are shown in the table below. The scope indicates the exact number of nodes in the directed graph, and the range for weights. As for the previous cases, the table shows the number of explored vectors, together with how many of these are valid test cases, the number of classes covered, etc. Scope Korat Korat+ Time Korat Time Korat+ CC 2,-1-1 1,062(332) 984(256) 0.326s 0.324s 11 2,-2-2 2,272(1,542) 1,750(1,022) 0,632s 0,534s 11 3,-1-1 18,003,420(493,232) 17,815,155(304,982) 55.483s 52.759s 13 3,-2-2 33,122,848(15,612,660) 25,486,513(7,976,340) 278.562s 169.82s 13 3,-3-3 205,397,228(187,887,040) 103,127,315(85,617,142) 2812,665s 1333,942s 13

Search Tree (Delete). Our last case study is concerned with deletion in search trees. In this case, the test data to generate is composed of a combination of a search tree and a value to be deleted from it. The search tree implementation we considered is the one provided in the Korat distribution. The test case equivalence classes in this case correspond to the “position” of the value to be deleted in the tree; we have chosen the following cases: – – – – – –

the the the the the the

value value value value value value

is is is is is is

not in the tree, in the root, in a leaf, in a node with two (nonempty) subtrees, in a node with a left subtree only, and in a node with a right subtree only.

The experimental results for this case study are shown in the table below. The scope indicates the maximum number of nodes in the tree, the range for the size of the tree, and the number of keys allowed in the tree. Scope Korat Korat+ Time Korat Time Korat+ CC 3,0-3,3 534(45) 500(43) 0.272s 0.251s 8 3,0-3,4 1,152(148) 1,011(125) 0.255s 0.271s 8 3,0-3,6 4,290(822) 3,331(586) 0,423s 0,359s 8 3,0-3,8 12,144(2,760) 8,675(1,793) 0.661s 0.562s 8 5,0-5,8 477,888(29,416) 338,292(16,137) 1,607s 1,333s 9 6,0-6,9 4,597,299(167,814) 3,213,270(83,511) 7,529s 5,724s 9

4.1

Analysing the Assessment of our Case Studies

Let us brieﬂy discuss now the results of our experimental analyses on our case studies. First, notice that we have chosen to report, for each of the case studies, the number of explored candidates, accompanied by the corresponding number

28

N.M. Aguirre et al.

of valid candidates found. This is, in our opinion, the most reasonable measure to employ if one is interested in evaluating the level of pruning that our technique contributes to standard ﬁltering. In our cases, these numbers reﬂect directly in running times, because our eqClass() routines, the most inﬂuencial (with respect to running time) part of the extra pruning section of our variant of Korat, do not increase in a noticeable way the running times of Korat for the scopes considered in these case studies. However, we only report the running time for generation. One would expect that this would also reﬂect in the time necessary for actually testing for the produced inputs. All the experiments were run on a 3.06GHz Intel Core 2 Duo with 4GB of RAM, and the reported data correspond to experiments that terminated within our timeout of 5 hours. The performance of the technique, in this case implemented as a variant of Korat, depends greatly on the quality of repOk() and eqClass(), and how these relate to each other. For instance, in cases in which eqClass() needs to visit the whole structure in order to determine the equivalence class for the test case, there will be no extra pruning at all; this is due to the fact that the “next candidate” computation of Korat would have already advanced one of the ﬁelds observed by eqClass(), since it “observes everything”. So, the technique provides better results when the test criterion under consideration is such that examining a small part of the structure one can determine a test case’s equivalence class. This is exactly the case in our two ﬁrst case studies, in which the technique exhibits a better proﬁt. Another important factor in the performance of our technique implemented as Korat+ compared to Korat is in the size of the “valid candidates” space over the search space. More precisely, when repOK() fails very often, i.e., when the conditions for valid structures are stronger, then Korat exploits its associated pruning mechanism. It is when repOK() succeedes more often than it fails when Korat+ contributes more to the pruning, since while Korat would advance to the next candidate with no pruning, our extra pruning mechanism would try to prune candidates corresponding to the just covered equivalence class. Notice that when for Korat the number of valid test cases is large in comparison with the number of explored candidates (repOK() succeedes more often), our extra pruning tends to contribute more to the pruning. We have tried to foresee potential threats to the validity of our experimental results. We tried to be careful about the chosen case studies. Although our case studies correspond to relatively small pieces of code, they represent, in our opinion, rather natural testing situations in the context of the implementation of complex, heap allocated data structures (which is the main target for Korat). We have accompanied the presentation of each case study by a short justiﬁcation of its appropriateness. We have included in our evaluation some case studies that have been successfully tackled by Korat, employing the same implementation available with Korat’s distribution (for which repOK() routines are tailored to exploit Korat’s search process). One might argue that the equivalence classes used in these cases might prune too much, i.e., that these would show good pruning but would not be helpful

Incorporating Coverage Criteria

29

for ﬁnding bugs. We decided then to take the three case studies for which we achieved more pruning, and make an analysis of how good would the obtained test suites be for ﬁnding bugs. These case studies are list as set, binomial heaps and search trees. We took three programs, namely standard implementations of listToSet, merge and deleteFromTree, for these structures, and performed the following experiment. We used muJava [9] in order to generate all method mutants of these three programs, and employed the test cases produced by Korat, by Korat+ and optimal equivalence class coverage (i.e., “one per equivalence class”), to see how many mutants can be killed by each of these test suites. The mutants are those obtained by the application of 12 diﬀerent method-level mutation operators, e.g., arithmetic, logical and relational operator replacements, etc. (see [11] for a complete list of method level mutation operators). Not all of these mutation operators were applicable to our programs (6 were applicable to list as set, 5 were applicable to binomial heaps, and 4 were applicable to search trees). The results obtained are shown in the tables at the end of this section. Each table shows the total number of mutants and how many remained live after testing using the corresponding test suite. Notice that the results for Korat correspond to optimal mutant killing, since their corresponding test suites are bounded exhaustive (i.e., Korat kills as many mutants as possible within the corresponding bounds). In order to obtain a test suite for optimal equivalence class coverage (one per equivalence class), we take the ﬁrst test case of each equivalence class from the bounded exhaustive test suite produced by Korat. As these experiments show, we achieve better results compared to one per equivalence class, and as the bounds are increased we get closer to bounded exhaustive test suites. Our intuition of being somehow “in between” optimal equivalence class coverage and bounded exhaustive is supported by the results. List as Set (49 mutants) Scope Korat Korat+ One Per Class 0-2,0-2,3,3,3 3 9 15 0-4,0-4,3,3,3 3 9 15 0-4,0-4,4,4,3 3 9 11 0-5,0-5,5,5,4 3 9 10 0-5,0-5,5,5,5 3 9 10

Binomial Heaps (117 mutants) Scope Korat Korat+ One Per Class 2 38 39 44 3 8 8 17 4 7 7 17 5 7 7 17 6 7 7 17

Search Trees (24 mutants) Scope Korat Korat+ One Per Class 3,0-3,3 2 2 2 3,0-3,4 2 2 2 3,0-3,6 2 2 2 3,0-3,8 2 2 2 5,0-5,8 0 0 2 6,0-6,9 0 0 2

30

5

N.M. Aguirre et al.

Conclusions and Future Work

We have presented a technique for improving bounded exhaustive test case generation using a ﬁltering approach, by incorporating black box test criteria and employing these for pruning the search of valid test inputs. The approach targets structurally complex inputs, and essentially consists of incorporating into the usual pruning processes present in test generation techniques, an extra pruning that skips parts of the search space when one is certain that only candidates of classes of inputs already covered would be found. We implemented this technique as a variant of Korat, a tool/algorithm that automatically generates test cases by a “generate and ﬁlter” mechanism [2] . We argued about the technique’s correctness, and developed some case studies, whose associated experimental results enabled us to assess the beneﬁts of the technique. The technique is somehow in between bounded exhaustive and “optimal” equivalence class coverage, and the actual “exhaustiveness” of the technique depends on the interaction of the test criterion (e.g., the adequacy of the predicates used for equivalence class coverage) and the generation procedure. This is reﬂected by the fact that, in our implementation, the performance depends on the quality of the repOk() and eqClass(), and how these relate to each other. In particular, when eqClass() roughly respects the order in which repOk() visits the ﬁelds of the structure, and in cases in which a relatively small part of it suﬃces to determining its equivalence class, the technique is more beneﬁcial. We also found that standard Korat works well when the valid test cases are relatively few with respect to the number of general structures, i.e., when the restrictions for the structure to be valid are stronger. In these cases, repOk() fails often, and thus Korat’s pruning improves the search signiﬁcantly. On the contrary, when repOk() does not fail very often, Korat’s pruning is not exercised much. These are the cases in which our technique shows more proﬁt. For instance, structures such as directed acyclic graphs or linked lists show better results than structures such as red black or AVL trees. Automatic test case generation is an active area of research. For the particular case of bounded exhaustive test case generation of structurally complex, heap allocated inputs, various tools have been proposed. Among these we may cite Java PathFinder [15], Alloy [5], CUTE [12] and obviously Korat. A thorough comparison between these tools, reported in [13], shows that Korat (seen as a kind of specialised solver) is generally the most eﬃcient, justifying our selection of Korat for implementing the technique. Although our approach is implemented for Korat, the technique applies to other tools that perform bounded exhaustive generation by ﬁltering. Examples of such tools would be Alloy [5], UDITA [4] (which also supports a generative approach) and AsmL [1]. As future work, we plan to develop a more signiﬁcant evaluation of our technique, over larger source code than that used in the experiments presented in this paper (these experiments were limited to a number of case studies regarding heap allocated data structure implementations, and a few algorithms over these). We are also currently exploring the use of SAT based analysis for test case generation guided by test criteria, exploiting the scalability improvements

Incorporating Coverage Criteria

31

achieved in [3]. We plan to continue this line of work by exploring the parallelisation of the approach (e.g., by combining the pruning with mechanisms such as those in [14]), as well as by deﬁning generic (i.e., not user provided) mechanisms for considering inputs to be similar. This would enable us to implement a similar technique, without the need for a user provided test criterion.

Acknowledgements The authors would like to thank Darko Marinov and the anonymous referees for their valuable comments. This work was partially supported by the Argentinian Agency for Scientiﬁc and Technological Promotion (ANPCyT), through grant PICT 2006 No. 2484. The ﬁrst author’s participation was also supported through ANPCyT grant PICT PAE 2007 No. 2772.

References 1. Barnett, M., Grieskamp, W., Nachmanson, L., Schulte, W., Tillmann, N., Veanes, M.: Model-Based Testing with AsmL.NET. In: Proceedings of the 1st European Conference on Model-Driven Software Engineering (2003) 2. Boyapati, C., Khurshid, S., Marinov, D.: Korat: Automated Testing based on Java Predicates. In: Proceedings of International Symposium on Software Testing and Analysis ISSTA 2002. ACM Press, New York (2002) 3. Galeotti, J.P., Rosner, N., L´ opez Pombo, C., Frias, M.: Analysis of invariants for eﬃcient bounded veriﬁcation. In: Proceedings of the 19th International Symposium on Software Testing and Analysis ISSTA 2010. ACM Press, Trento (2010) 4. Gligoric, M., Gvero, T., Jagannath, V., Khurshid, S., Kuncak, V., Marinov, D.: Test generation through programming in UDITA. In: Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering ICSE 2010. ACM Press, Cape Town (2010) 5. Jackson, D.: Software Abstractions: Logic, Language, and Analysis. The MIT Press, Cambridge (2006) 6. Kaner, C., Bach, J., Pettichord, B.: Lessons Learned in Software Testing. Wiley, Chichester (2001) 7. Khurshid, S., Marinov, D.: TestEra: Speciﬁcation-Based Testing of Java Programs Using SAT. Automated Software Engineering 11(4) (2004) 8. Liskov, B., Guttag, J.: Program Development in Java: Abstraction, Speciﬁcation and Object-Oriented Design. Addison-Wesley, Reading (2000) 9. Ma, Y.-S., Oﬀutt, J., Kwon, Y.-R.: MuJava: An Automated Class Mutation System. Journal of Software Testing, Veriﬁcation and Reliability 15(2) (2005) 10. Milicevic, A., Misailovic, S., Marinov, D., Khurshid, S.: Korat: A Tool for Generating Structurally Complex Test Inputs. In: Proceedings of International Conference on Software Engineering ICSE 2007. IEEE Press, Los Alamitos (2007) 11. MuJava Home Page, http://www.cs.gmu.edu/offutt/mujava/ 12. Sen, K., Marinov, D., Agha, G.: CUTE: A Concolic Unit Testing Engine for C. In: Proceedings of the 5th Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering ESEC/FSE 2005. ACM Press, New York (2005)

32

N.M. Aguirre et al.

13. Siddiqui, J., Khurshid, S.: An Empirical Study of Structural Constraint Solving Techniques. In: Breitman, K., Cavalcanti, A. (eds.) ICFEM 2009. LNCS, vol. 5885, pp. 88–106. Springer, Heidelberg (2009) 14. Siddiqui, J., Khurshid, S.: PKorat: Parallel Generation of Structurally Complex Test Inputs. In: Proceedings of the 2nd International Conference on Software Testing Veriﬁcation and Validation ICST 2009. IEEE Computer Society, Los Alamitos (2009) 15. Visser, W., Pasareanu, C., Khurshid, S.: Test Input Generation with Java PathFinder. In: Proceedings of International Symposium on Software Testing and Analysis ISSTA 2004. ACM Press, New York (2004) 16. Xie, T., Marinov, D., Notkin, D.: Rostra: A Framework for Detecting Redundant Object-Oriented Unit Tests. In: Proceedings of the 19th IEEE International Conference on Automated Software Engineering ASE 2004. IEEE Computer Society, Linz (2004) 17. Zhu, H., Hall, P., May, J.: Software Unit Test Coverage and Adequacy. ACM Computing Surveys 29(4) (1997)

Checking the Behavioral Conformance of Web Services with Symbolic Testing and an SMT Solver Lina Bentakouk1, Pascal Poizat1,2 , and Fatiha Za¨ıdi1 1

LRI; Univ. Paris-Sud, CNRS, Orsay France ´ Univ. Evry Val d’Essonne, Evry, F-91000, France {lina.bentakouk,pascal.poizat,fatiha.zaidi}@lri.fr 2

Abstract. Workflow-based service composition languages foster the rapid design and development of distributed applications. The behavioral verification of service Compositions has widely been addressed at design time, using modelchecking. Testing is a complementary technique when it comes to check the behavioral conformance of a service implementation with respect to its specification or to a user or a service need. In this paper we address this issue with an automatic approach based on symbolic testing and an SMT solver. Keywords: services, orchestration, formal testing, test-case generation, WS-BPEL, transition systems, symbolic execution, SMT solver.

1 Introduction Context. Web services are gaining industry-wide acceptance and usage as the main implementation of the Service Oriented Architecture. As such, they support the construction of distributed applications out of reusable heterogeneous and loosely coupled software entities, namely services. WS-BPEL [1] (BPEL for short) has been normalised as the language for Web service orchestration, i.e., centralised service composition. BPEL is twofold. It may be used to describe orchestration implementations. Abstracting details and refereed to as Abstract BPEL (ABPEL), it may also be used to specify orchestrations or to publish information about the behaviour of a service, i.e., the ordering in which its operations should be called. Motivations. Since BPEL is a semi-formal workflow language, a significant research effort has been produced in the last years in order to propose formal models for orchestration verification [2]. Numerous work in this area have addressed model-checking, to check if properties are verified by an orchestration specification. Still, in presence of a black box implementation, one cannot retrieve a model from it. To help establishing the conformance wrt. a specification, testing should be used. However, to the contrary of model-checking, testing is incomplete. One rather focuses on generating the good test cases to search for errors. Testing enables one to ensure both that sub-services participating to an orchestration conform to their publicised behavioural interface, and that the orchestration itself conforms to the behavioural interface to be publicised after its deployment. Orchestration testing has mainly been addressed from a white-box perspective, assuming that the orchestration implementation source code is available. In practise, the source code is often not available as it constitutes an added-value for the M. Gogolla and B. Wolff (Eds.): TAP 2011, LNCS 6706, pp. 33–50, 2011. c Springer-Verlag Berlin Heidelberg 2011

34

L. Bentakouk, P. Poizat, and F. Za¨ıdi

service providers. To overcome this limit, we propose a black-box conformance testing approach to automate the conformance checking of composed services. We base our approach on a symbolic treatment of data and on an efficient SMT solver. Contributions. The contributions of our work are manifold. (i) Based on a behavioural specification, we produce a formal model, namely a Symbolic Transition System (STS) [3,4] by means of transformation rules written in a process algebraic style [5]. We handle as well interfaces as behavioural parts of composed services. (ii) A particular accent is put on the processing of structured data types in such a way that they will be handled by a more powerful constraints solving tool compared to our previous work. (iii) The testing process is based on the definition of requirements that allow a user to specify the service behaviour to test. Based on requirements or user needs, the tester provides test purposes. We suppose these are given as STS too, yet, it has been shown that such transition system models could be obtained automatically either from abstract properties given in a temporal logic or from human-friendly descriptions such as sequence diagrams. (iv) We support the test of a real service orchestrator, in which we use some knowledge about the database of the partner services. This assumption is needed in order to provide pertinent data input to the tested orchestrated service. (v) We clearly explain our stepwise approach, which starts with the definition of a formal model of the specification and of the test purposes. Afterwards, we produce by means of a symbolic execution tree (SET) the test cases that cover the test purpose. The SET allows to avoid the unfolding of the STS product that would yield state space explosion. Then, an online oracle tester executes the test cases against the real implementation. Outline. The remainder of the paper is structured as follows. The next section presents the related work. Then a running example is provided in Section 3. A formal approach is detailed in Section 4 in which we describe the use of test purposes. In Section 5 the automatic test case generation, based on the STS product and SET generation, is explained. In Section 6 we present the tools chain with an emphasis on the use of a solving tool. Finally Section 7 ends with conclusion and perspectives.

2 Related Work Web service (WS) testing has been addressed differently depending on the available service description level and on the testing generation approach [6]. A simple but limited technique is to inject hand-crafted test cases on a service implementation using tools such as soapUI or BPELUnit [7]. However, these tools do not allow to do intensive testing, since they require a manual instantiation of the test cases. When the WS source code is available, white-box testing criteria can be used to generate test cases [8]. Sometimes, only service signatures are available. Still, this enables to test WS by generating relevant input data for each operation [9,10]. In the later work, the control flow for Java control flow code is also considered. Going beyond WSDL WS, composite services, e.g., orchestrations, feature a behavioural interface. It is important to take this information into account in the testing process. In [11], they have addressed grey-box testing using a BPEL to IF (Intermediate Format) translation and extending the IF simulator to generate test cases given a test objective. A limitation is state space explosion in presence of complex data types. In [12]

Conformance Testing of Service Orchestration Using an SMT Solver

35

the authors propose an approach for compliance testing of the specification wrt. the implementation of a composed Web services, using Petri-nets. This approach allows to automatically generate test cases. However, they did not address the problem of generating input data for test. To solve this problem out, we propose to rely on symbolic models and their symbolic semantics. Symbolic testing approaches have been investigated for several years, e.g., in [13,14,15]. Our test purpose driven test case generation is closely related to [13,14]. However, except for [15] (still, from a theoretical and generic point of view), the above mentioned approaches have not addressed application to the component or service composition domain. This application requires a comprehensive language-to-language approach with, e.g., transformation rules supporting a workflow-based language with complex data types and communication mechanisms. We have proposed in [4] a WS symbolic testing approach based on specification coverage criteria. Here, this work is extended to formally propose a more precise and stepwise approach driven by symbolic test purpose which allows a user to test specific part of an orchestration. In other words, the test purposes described in this paper refer to behavioral functionalities that a tester wants to check. Further, in [4] we also used the UMLtoCSP [16] tool to solve constraints, which required a very fine tuning of the variables’ domains. Moreover, a more complex model transformation between this tool and ours was needed, which is no more the case with the Z3 SMT solver.

3 Running Example In this Section we present our e-Conference case study. This medium-size orchestrated service provides functionalities to researchers going to a conference such as information

Fig. 1. e-Conference Example – Data and Service Architecture (UML Extended)

36

L. Bentakouk, P. Poizat, and F. Za¨ıdi

Fig. 2. e-Conference Example – Orchestration Specification (Inspired and extended from BPMN )

about the conference, flight booking and fees refunding. Parts of the orchestration, e.g., the e-governance rules, correspond to reality. Other parts, e.g., the sub-service used to book plane tickets, represent a simplified yet realistic version of it. e-Conference is based on three sub-services: ConferenceService, FlightService, and e-govService. Its specification is as follows. A user starts the process by providing the conference name and edition, together with personal information (ordersSetup

Conformance Testing of Service Orchestration Using an SMT Solver

37

operation). Conference information is first retrieved using ConferenceService then e-govService is invoked to check if any travel alerts exist in the conference country. If there is one, the user is informed and the process stops. If not, an orders id is asked to e-govService, a plane ticket is bought using FlightService and all gathered information (orders id, conference and plane) is sent back to the user who may then go to the conference. Upon return, the user may choose to be refunded from either a fees or package basis (returnSetup operation). In both cases, the user will end by validating the mission (validate operation) and e-Conference will reply with refunding information. If fees basis has been chosen, the user will be able to send information on fees (using addFee operation several times). Figure 1 exhibits our extension of the UML notation corresponding to the e-Conference orchestration. Within this diagram we highlight the stereotypes for message types, correlations and properties to represent the orchestration architecture and the imported data (XML schema files structured in namespaces nsx ). Concerning orchestration specification shown in Figure 2, we take inspiration from BPMN, while adding our own annotations supporting relation with BPEL. Communication activities are represented with the concerned partnerlink (USER for the user, cS, egovS, and fS for the sub-services), operation, input/output variables, and, when it applies, information about message correlation.

4 Formal Models for Test Case Generation In this section we briefly present our formal model for service orchestration, the motivations and steps that lead us to reuse the same model to represent test purposes. 4.1 Service Orchestration Model Different models have been proposed to support behavioural service discovery, verification, testing, composition or adaptation [2,17,11,18]. They mainly differ in their formal grounding (Petri nets, transition systems, or process algebra), and the subset of service languages being supported. We base on [19] due to its wide coverage of BPEL language constructs. Moreover, its process algebraic style for transformation rules enables a concise yet precise and operational model, which is, through extension, amenable to symbolic execution. With reference to [19], we support data (in computation, conditions and messages), message faults (enabling a service to inform its partners about internal errors), message correlation (enabling BPEL engines to correlate messages in-between service instances), flows (parallel processing) and the until activity. More specifically, support for data yields grounding on (discrete time) Symbolic Transitions Systems and their symbolic execution, rather than on (discrete time) Labelled Transition Systems. Unlike the model described in [3] we have modified the WS-STS model to take into account the specific characteristics of Web Services. STS modelling. A Web Service Symbolic Transition System (WS-STS), is a tuple (D, V, S, s0 , T ) where D is a set of domains (data type specifications), V is a set of variables with domain in D, S is a non empty set of states, s0 ∈ S is the initial state, and T is a

38

L. Bentakouk, P. Poizat, and F. Za¨ıdi

(potentially nondeterministic) transition relation, T ⊆ S×TBool,V ×Ev×seq(Act)×S, with TBool,V denoting Boolean terms possibly with variables in V, Ev a set of events represents the messages communication. D and V are often omitted when clear from the context (e.g., V are variables used in transitions). A reception or the return of an invocation event noted pl.o?x corresponds to the reception of the input message x from the partner pl of the operation o. We omit BPEL port types for simplicity reasons (full event prefixes would be, e.g., pl.pt.o?x). Accordingly, we define a reply or invocation event pl.o!x corresponds to an emission of an output message. Ev? (resp. Ev ! ) is the set of reception events (resp. emission events). Ex is the set of internal fault events, that corresponds to faults possibly raised internally (not in messages) by the orchestration process. We also introduce √ specific events: τ denotes non-observable internal computations or conditions and denotes the termination of a conversation √ (end of a session). For reasons of simplicity we denote Ev = Ev? ∪ Ev! ∪ Ex ∪ {τ, }. seq(Act) is a set of actions denoting computation (data processing) that will be executed in a sequential way (of the form v := t where v ∈ V is a variable and t ∈ TD,V is a term). The transition system is called symbolic as the guards, events, and actions may contain [g]e/A

[g]e/A

variables. (s, g, e, A, s ) ∈ T is also written s −−−−−→T s or simply s −−−−−→ s when clear from the context. The WS-STS moves from the state s to the state s if the event e occurs and the guard g is satisfied and some actions can be performed. When there is no guard (i.e., it is true) it is omitted. The same yields for the actions. We impose that variables used in the WS-STS transitions are variables from BPEL and anonymous variables used for interacting with other services. Notice that each data type (structured or scalar) of BPEL corresponds to an element of D and each variable of BPEL corresponds to a variable in V. An orchestration is built around a partnership, i.e., a set of (partners) signatures corresponding to required operations and a set of signatures corresponding to provided operations. In the sequel, we suppose, without loss of generality, that an orchestration has only one of the later, named USER. STS have been introduced under different forms (and names) in the literature [3], to associate a behaviour with a specification of data types that is used to evaluate guards, actions and sent values. This role is played by D which is a superset of all partner’s domains. Transformation rules from BPEL to WS-STS are provided in [5]. Application. From the e-Conference specification (see Figure 2), we obtain the √ STS in Figure 3 (49 states, 57 transitions) where tau (resp. term) denote τ (resp. ). The zoom (grey states) corresponds to fees loop. One may notice states 39 (while condition test) and 40 (pick). In states 41/45 it is checked if incoming messages (validate or addFee) come from the same user than the previous ones in the conversation (ordersSetup and returnSetup). When it is not the case (correlation failure) an exception is raised (in state 26). Variables names, in our example are prefixed with namespaces (e.g., vns5:VOut is the variable storing values of type ns5:VOut) to help the reader. 4.2 Test Purpose Model A Test Purpose. (TP) is a set of functional properties allowed by the specification and that one is interested to test. Usually, the formal model for TP follows the one used for the system specification. Hence Labelled Transition Systems (LTS) is the most popular

Conformance Testing of Service Orchestration Using an SMT Solver

39

0

Gno41:vcsUSERinit=false \/ not (vans5:VIn/ordersId=vcsUSER/ordersId) Gno45:vcsUSERinit=false \/ not (vans5:AFIn/ordersId=vcsUSER/ordersId)

USER.ordersSetup ?vans5:OSIn / vns5:OSIn:=vans5:OSIn; vcsUSERinit:=false 1 tau / vns2:GIIn/name:=vns5:OSIn/confName; vns2:GIIn/edition:=vns5:OSIn/edition 2 tau / vans2:GIIn:=vns2:GIIn 3 cS.getInfo !vans2:GIIn 4 cS.getInfo ?vans2:GIOut / vns2:GIOut:=vans2:GIOut 5 tau / vns3:TAIn/country:=vns2:GIOut/conf/loc/country 6 tau / vans3:TAIn:=vns3:TAIn 7 egovS.travelAlert !vans3:TAIn 8 egovS.travelAlert ?vans3:TAIn / vns3:TAOut:=vans3:TAOut 9 [not vns3:TAOut/hasAlert== false] tau

[vns3:TAOut/hasAlert== false] tau

10

14

tau / vns5:OSOut/conf:=vns2:GIOut/conf; vns5:OSOut/riskLevel:=vns3:TAOut/riskLevel

tau / vans3:GOIIn:=vns3:GOIIn

11

15

tau / vans5:OSOut:=vns5:OSOut

egovS.getOrdersId !vans3:GOIIn

12

16 egovS.getOrdersId ?vans3:GOIOut / vns3:GOIOut:=vans3:GOIOut

USER.ordersSetup !vans5:OSOut

17 tau / vns4:BIn/arrival:=vns2:GIOut/conf/start; vns4:BIn/departure:=vns2:GIOut/conf/end; vns4:BIn/to/city:=vns2:GIOut/conf/loc/city; vns4:BIn/to/country:=vns2:GIOut/conf/loc/country; vns4:BIn/form/city:=vns5:OSIn/uinfo/address/city; vns4:BIn/form/country:=vns5:OSIn/uinfo/address/country

13

18 tau / vans4:BIn:=vns4:BIn 19 fS.book !vans4:BIn 20 fS.book ?vans4:BOut / vns4:BOut:=vans4:BOut 21 tau / vns5:OSOut/conf:=vns2:GIOut/conf; vns5:OSOut/ticket:=vns4:BOut/ticket; vns5:OSOut/ordersId:=vns3:GOIOut/ordersId 22 tau / vans5:OSOut:=vns5:OSOut 23 USER.ordersSetup !vans5:OSOut 24 USER.returnSetup ?vans5:RSIn / vns5:RSIn:=vans5:RSIn 25 [not Gno 25] tau 27 [vns5:RSIn/choice!= ’package’] tau term

38

[not vns5:RSIn/choice!= ’package’] tau 28

tau / vns5:VOut/value:=0; finish:=false

USER.validate ?vans5:VIn / vns5:VIn:=vans5:VIn

39

[Gno 25] tau

29

[not finish==false] tau

[finish==false] tau

[not Gno 29] tau 40

30

tau USER.validate ?vans5:VIn / vns5:VOut/value:=vns5:AFIn+vns5:VOut/value / vns5:VIn:=vans5:VIn

47

41 [not Gno41] tau

USER.validate !vans5:VOut

46

USER.addFee ?vans5:AFIn / vns5:AFIn:=vans5:AFIn

45 [Gno41] tau

tau / vans3:PVIn:=vns3:PVIn

26

32

egovS.packageValue !vans3:PVIn

43

33

tau / vans5:VOut:=vns5:VOut term

tau / vns3:PVIn/country := vns2:GIOut/conf/loc/country 31

[not Gno45] [Gno45] tau tau

42 tau / finish:=true

[Gno 29] tau

egovS.packageValue ?vans3:PVOut / vns3:PVOut:=vans3:PVOut

44

34 tau / vns5:VOut/value:=vns3:PVOut/value bpel:cv

35 tau / vans5:VOut:=vns5:VOut 36 USER.validate !vans5:VOut 37

term 48

Fig. 3. e-Conference Example – Orchestration model (STS)

model for TP. However in our case it will be formalised as an STS according to the specification model. Note that a TP can also be modelled using Linear Temporal Logic (LTL) to be more abstract. The average user may prefer more user friendly notation eg. MSC or UML sequence diagrams [20,21] that describe the interactions between system components. In both case we can get back to transition system model, LTL can be transformed in Buch¨ı automata [22] while, MSC and UML sequence diagrams can be transformed in LTS [23]. To represent formally requirements as a test purpose we were inspired by the work of [13]. However, the way to express a test purpose is simpler because we don’t need reject states to specify an undesired behaviour. Thus the WS-STS resulting product contains only the paths that run through an accept state. TP models are defined according to the orchestration (specification) models they refer to. Therefore, given an orchestration model B = (DB , VB , SB , s0B , TB ), a TP for B is an WS-STS T P = (DT P , VT P , ST P , s0T P , TT P ) with some constraints. T P may use a set of additional variables VI for expressiveness (see Application, below), disjoint from B variables. VT P = VI ∪ VB where VI ∩ VB = ∅, accordingly DT P ⊇ DB , with [g]e/v:=t

∀t s −−−−−−−→ s ∈ TT P v ∈ VI . Assignments in T P can only operate on VI . The events labelling T P transitions correspond to the B ones. More specifically, we impose for simplicity sake that variables used in message exchanges (events of the form pl.op . . .) correspond to the ones in B. This constraint can be lifted using substitutions. TP also introduce a specific event, *. Transitions labelled with * may neither have a guard, nor actions, and are used to abstract in T P one or several B transitions that are not relevant for the expression of the requirement. A TP defines a specific set of states,

40

L. Bentakouk, P. Poizat, and F. Za¨ıdi

P0

P0

P0

*

*

*

USER.addFee ? vans5:AFIn

P1

USER.returnSetup ? vans5:RSIn

USER.returnSetup ? vans5:RSIn

P1

P1

* # [vns5:RSIn/choice!= ’package’]tau

P2

[vns5:RSIn/choice!= ’package’]tau / n:=0

[n
* #

P2

*

USER.validate ! vans5:VOut

[vns5:AFIn/fee>=10 ]tau / n:=n+1

P3

P4

[vans5:VOut/value<1000]tau

P5

* #

[n=N]USER.addFee ? vans5:AFIn P6

Fig. 4. e-Conference Example - Test Purposes (STS)

Accept (Accept ⊆ ST P ), that denotes TP satisfaction. Those states have transitions labelled by an event #. Finally, we impose that T P is consistent with B, i.e., T P symbolic traces are included in B ones. This can be checked using symbolic execution (see Sect. 5.2), where we also have to check that the path condition corresponding for the T P trace implies the path condition of the B trace. Application. Let us assume one wants to focus on testing refunding on a fees basis. Possible TPs are given in Figure 4. In the first TP, one may note the use of the addFee transition, leading to an acceptance state (Labelled with #). This specifies addFee must be part of the generated test cases. Several transitions in the specification could be done before and after the addFee one, therefore, the two TP states are equipped with a * loop. The * loop is interpreted as being the execution of any other communication messages (including calls to sub-services), except those explicitly expressed. The first TP requires that there is at least one addFee in the test cases. One may want to take also into account the case where there are none. This can be specified as in the second TP. There, one relies on the returnSetup transitions that carry user requests relative to the return mode (package or fees basis). In order to specify it is the later which is required, a guarded transition is used (choice should be different from ’package’). Note that the guard is put on a τ transition after the reception in order to be consistent with (symbolic) execution semantics: the guard can only be evaluated once the variable has been received. The last TP, used in the sequel, is more realistic and demonstrates TPs expressiveness with four requirements: (i) return is on a fees basis, (ii) each fee is greater or equal to 10, (iii) the total amount for the mission is less than 1000, and (iv) there are at most N fees added. The later requires to use a TP additional variable, n, to count addFee iterations.

5 Automatic Test Case Generation 5.1 WS-STSs Product To generate test cases we have to take into account constraints specified both in the specification and in the TPs. This is achieved using WS-STS (symbolic) product. Given a specification model B = (DB , VB , SB , s0B , TB ), and a test purpose TP = (DT P , VT P , ST P , s0T P , TT P ), their product, P rod = B ⊗ TP, is the WS-STS (DP rod , VP rod , SP rod , s0P rod , TP rod ) where DP rod = DT P , VP rod = VT P , SP rod ⊆ SB × ST P , s0P rod = (s0B , s0T P ), and TP rod is built using four rules:

Conformance Testing of Service Orchestration Using an SMT Solver

• ∀e ∈ {τ, χ, },

41

√ • ∀e ∈ Ev ? ∪ Ev ! ∪ Ex ∪ { }, [gB ]e/AB

sB −−−−−−−→B sB , [gT P ]e/AT P

sT P −−−−−→T P sT P

sT P −−−−−−−−−→T P sT P

(sB ,sT P )−−−−−→P rod (sB ,sT P )

(sB ,sT P )−−−−−−−−−−−−−−→P rod (sB ,sT P )

• ∀e ∈ {τ, χ, },

√ • ∀e ∈ Ev ? ∪ Ev! ∪ Ex ∪ { },

[g]e/A

[g]e/A

[gB ∧gT P ]e/AT P ;AB

[gB ]e/AB

sB −−−−−−−→B sB , ∗ sT P −−→T P sT P , [gT P ]e/AT P

sB −−−−−→B sB

∃ sT P −−−−−−−−−→T P sT P

(sB ,sT P )− −−−−−→P rod (sB ,sT P )

(sB ,sT P )−−−−−−−→P rod (sB ,sT P )

[g]e/A

[g]τ /A

[gB ]e/AB

The two rules on the left side denote that T P (resp. B) evolves independently for non observable events. The first rule on the right side corresponds to synchronising between T P and B. Finally, the second rule on the right side defines the * semantics. It corresponds to observable events but for the ones that are captured by the first rule on the right side. To enforce the acceptance states semantics, the product is cleaned up by pruning states (and related transitions) that are not co-reachable from acceptance states, i.e., any s such that ∃s = (sB , sT P ) ∈ SP rod . s − → ∗s ∧ s ∈ Accept. Notice that a cleaning of the WS-STS product is performed to keep only the paths that pass through an accept state. Application. The product of the orchestration (Figure 3) with the third TP (Figure 4) is given in Figure 5. It has 68 states and 85 transitions (89 states and 119 transitions before pruning). Its set of symbolic traces is a subset of TP one (hence also of the orchestration one). One may note for example that receiving addFee is possible only if done less than N times (see guard [n
42

L. Bentakouk, P. Poizat, and F. Za¨ıdi 24

0

25

23

1 tau / vns5:OSOut/conf:=vns2:GIOut/conf; vns5:OSOut/ticket:=vns4:BOut/ticket; vns5:OSOut/ordersId:=vns3:GOIOut/ordersId

USER.returnSetup ?vans5:RSIn / vns5:RSIn:=vans5:RSIn

26

27

2

33 [vns5:RSIn/choice!= ’package’] tau / n:=0

fS.book ?vans4:BOut / vns4:BOut:=vans4:BOut

29

[not vns5:RSIn/choice!= ’package’] tau

tau / vans2:GIIn:=vns2:GIIn

21

[vns5:RSIn/choice!= ’package’] tau / n:=0

3

[vns5:RSIn/choice!= ’package’] tau

31

fS.book !vans4:BIn

34

[not vns5:RSIn/choice!= ’package’] tau

37

tau / vns5:VOut/value:=0; finish:=false

USER.validate ?vans5:VIn / vns5:VIn:=vans5:VIn

39

cS.getInfo !vans2:GIIn

20

[vns5:RSIn/choice!= ’package’] tau / n:=0

[vns5:RSIn/choice!= ’package’] tau

36

tau / vns2:GIIn/name:=vns5:OSIn/confName; vns2:GIIn/edition:=vns5:OSIn/edition

22

[vns5:RSIn/choice!= ’package’] [not Gno 25] tau tau / n:=0

[not Gno 25] tau

USER.ordersSetup ?vans5:OSIn / vns5:OSIn:=vans5:OSIn; vcsUSERinit:=false

tau / vans5:OSOut:=vns5:OSOut

USER.ordersSetup !vans5:OSOut

4

tau / vns5:VOut/value:=0; finish:=false

tau / vans4:BIn:=vns4:BIn

38

19

40

5 tau / vns4:BIn/arrival:=vns2:GIOut/conf/start; vns4:BIn/departure:=vns2:GIOut/conf/end; tau vns4:BIn/to/city:=vns2:GIOut/conf/loc/city; / vns3:TAIn/country:=vns2:GIOut/conf/loc/country vns4:BIn/to/country:=vns2:GIOut/conf/loc/country; vns4:BIn/form/city:=vns5:OSIn/uinfo/address/city; vns4:BIn/form/country:=vns5:OSIn/uinfo/address/country

[vns5:RSIn/choice!= ’package’] [finish==false] tau tau / n:=0

42

[not Gno 29] tau

17

6

[vns5:RSIn/choice!= ’package’] tau / n:=0

[finish==false] tau

43

egovS.getOrdersId ?vans3:GOIOut / vns3:GOIOut:=vans3:GOIOut

45

tau / vns3:PVIn/country := vns2:GIOut/conf/loc/country [vns5:AFIn/fee>=10] tau / n:=n+1

50

48

52

egovS.packageValue !vans3:PVIn

57

USER.validate ?vans5:VIn / vns5:VIn:=vans5:VIn

[vns5:AFIn/fee>=10] tau / n:=n+1

54

8

egovS.travelAlert ?vans3:TAIn / vns3:TAOut:=vans3:TAOut

tau / vans3:GOIIn:=vns3:GOIIn

51

tau / finish:=true

61 [finish==false] tau

64

9

[vns3:TAOut/hasAlert== false] tau

58

egovS.packageValue ?vans3:PVOut / vns3:PVOut:=vans3:PVOut

egovS.travelAlert !vans3:TAIn

13

[not Gno 41] tau

[vns5:AFIn/fee>=10] tau tau / vns5:VOut/value:= / n:=n+1 vns5:AFIn+vns5:VOut/value

[not Gno 45] tau

59

7

egovS.getOrdersId !vans3:GOIIn

47

[vns5:AFIn/fee>=10] [not Gno 45] tau tau / n:=n+1

tau / vans3:PVIn:=vns3:PVIn

tau / vans3:TAIn:=vns3:TAIn

15

[n
46

cS.getInfo ?vans2:GIOut / vns2:GIOut:=vans2:GIOut

11

tau / vans5:VOut:=vns5:VOut

67

65

tau / vns5:VOut/value:=vns3:PVOut/value

USER.validate !vans5:VOut

70

71

tau / vans5:VOut:=vns5:VOut

[not finish==false] [finish==false] tau tau

72

74

76

term

[vans5:VOut/value<1000] tau

[vans5:VOut/value<1000] tau

75

73

[vans5:VOut/value<1000] [vans5:VOut/value<1000] tau tau

USER.validate !vans5:VOut

77

79

USER.addFee ?vans5:AFIn / vns5:AFIn:=vans5:AFIn

82

80

term [vans5:VOut/value<1000] tau

[finish==false] tau

78 USER.validate ?vans5:VIn / vns5:VIn:=vans5:VIn

term

term

[not finish==false] tau

[Gno 41] tau 84

bpel:cv

81

tau / vns5:VOut/value:=vns5:AFIn+vns5:VOut/value

83 [not Gno 41] tau

[Gno 45] tau 85

[not Gno 45] tau

USER.validate !vans5:VOut

86 tau / finish:=true 87 tau / vans5:VOut:=vns5:VOut 88

Fig. 5. e-Conference Example - Product (STS)

(may be non deterministic), where Evsymb corresponds to the WS-STS product events (Ev) with symbolic variables in place of variables. Each feasible path of a SET (i.e., each path in this SET with a PC that can be solved) represents a possible test case. 5.3 SET Computation The SET is computed in a BFS fashion as follows. The root is (s0 , true, σ0 ) where s0 is the WS-STS product initial state and σ0 is the mapping of a fresh variable for each variable of the WS-STS (σ0 : ∀v ∈ V, v → newV ar) and π0 = true. Each transition [g]e/A

e

s −−−−−→ s then corresponds to an edge (s, π, σ) −−−→ (s , π , σ ), computed as described in [4]. 1. guard: π G = π ∧ gσ (π G = π if there is no guard) ⎧ ⎨ pl.o?v, σ[v → vs ] if e = pl.o?v if e = pl.o!v 2. event: e , σE = pl.o!σ(v), σ ⎩ e, σ otherwise with vs = new(Vsymb , σ). If e is a sub-service invocation return (e = pl.o?vout ∧pl = USER), we set π E = π(o)[σE (vin )/in, vs /out], where e = pl.o!vin is the label of the (unique) transition before the one we are dealing with, to take into account the operation specification (π(o)). Else, π E = πG . 3. actions (A = {xi /pathi := ti }i,i∈{1,...,n} ): A πiA = πi−1 ∧ (vsxi /pathi = ti [σ E (vj )/vj ]vj ∈vars(ti ) )

Conformance Testing of Service Orchestration Using an SMT Solver

43

with Δ = {x ∈ V | (x/pathi := ti ) ∈ A}, {vsx }x∈Δ = new#Δ (Vsymb , σ E ), σ = σE {[vsx /x]}x∈Δ , π0A = πE , and π = πnA . where vars denotes the variables in a term, newn (Vsymb , σ) denotes the creation of n new (fresh) symbolic variables wrt. σ, t[y/x] denotes the substitution of x by y in t, and σ[x → xs ] denotes σ where the mapping for x is overloaded by the one from x to xs . Δ is the set of variables that are modified by the assignments. For each of these, we have a new symbolic variable. Note that we suppose without losing of generality that in practise one assign with parallel instructions is executed sequentially. L

[g]pl.o!v/A

We denote may(η), η ∈ NSET , the set {pl.o!v | ∃η −−→ ∗η −−−−−−−−→ η”} with L a sequence of labels such that the corresponding word (keeping only the event in labels), √ contains only non observable events or communication events with partners ({τ, χ, } ∪ {pl.o ∗ v | ∗ ∈ {?, !} ∧ pl = U SER}). This set will be used later on for the test verdict emission. Pruning infeasible paths. Edges with inconsistent path conditions are cut off while computing the SET. For this, we check when computing a new node η if π(η) is satisfiable (there exists a valuation of variables in π such that π is true, if not, we cut the edge off). This is known to be an undecidable problem in general. Therefore, if the constraint solver does not yield a solution (or a contradiction) in a given amount of time, we cut the edge off and we issue a warning specifying that the test process is to be incomplete. Pruning redundant paths. A WS-STS product may contain loops that would cause SET unboundedness. To solve this issue out, we propose two compatible techniques. We can first take into account a path length criterion while computing the SET. Given a constant k, we stop the SET computation at some node whenever this node is at k edges from the SET root, this technique is inspired by the k bounded model checking [27]. The user can re-start the SET computation process with k + 1 for more test cases. A complementary approach is to use the inclusion criterion as proposed by [14]. Let us consider η = (s, π, σ), a reachable node in the SET. Solving the associate path condition π means that there exists at least a value for each symbolic variable satisfying the constraint with regards to the σ mapping (V → Vsymb ). Such constraint allows several interpretations (combinations of possible values). MV η represents the set of all the possible interpretations for the variables V of the node η. Consider now, an other node η = (s, π , σ ) which has the same state as η. We say that η is included in η V (η ⊆ η), when MV η ⊆ Mη . That means that the set of interpretations of the variables of V belonging to η is bigger than the one belonging to η , so π σ =⇒ πσ. If the two nodes are on the same branch, we can prune the sub-tree associated to the node η as it is included in the sub-tree of the node η. We can then stop the generation of the SET at the node η . Application. Among the 85 paths in the SET (k = 15) of the product STS (see Figure 5), there are 7 complete paths corresponding to the coverage of the last test purpose described in Section 4.2. A path example is:

44

L. Bentakouk, P. Poizat, and F. Za¨ıdi

USER.ordersSetup?vs36 tau(x2) cS.getInfo!vs40 cS.getInfo?vs41 tau(x2) egovS.travelAlert!vs44 egovS.travelAlert?vs45 tau(x2) egovS.getOrdersId!vs47 egovS.getOrdersId?vs48 tau(x2) fS.book!vs51 fS.book?vs52 tau(x2) USER.ordersSetup!vs55 USER.returnSetup?vs58 tau(x5) USER.addFee?vs69 tau(x4) USER.validate?vs74 tau(x3) USER.valivate!vs77 tau(x2) term.

6 Tools Support and Experimentations Figure 6 exhibits the different steps of our approach that are supported by tools. In step 1: from a specification represented as a BPEL file we use the BPEL2STS tool to generate an STS model. Followed by step 2: the produced model and a test purpose STS (in the same format) are provided as inputs files to our STSprod tool that computes the product. In step 3: the STS2SET tool takes as input the STS product and uses the Z3 solver [28] to compute the SET. Finally in step 4: using the SET paths we execute the test cases as described in the test cases realisation with an automated call to Z3 but the oracle algorithm is exercised manually. All intermediate formats follows the Aldebaran format (.aut) which supports a graphical representation with the CADP tool [29]. SMT Solver Use during the SET construction. Our approach relies on a symbolic execution semantic of our WS-STS. Compared to work that enumerate the possible values of variables [11] for the test cases, we rely on a solving tool to instantiate the path condition. Enumeration techniques can be faced to the state space explosion problem and false verdicts can be emitted as explained later on. In our case we can overcome the state space explosion problem by avoiding the unfolding of the WS-STS and with our on line testing algorithm we can obtain the possible values by a call to the solving tool and hence instantiate the values by interacting step by step with the service under

Fig. 6. Overview of the tools chain

Conformance Testing of Service Orchestration Using an SMT Solver

45

test. To solve the constraints we use an efficient SMT (Satisfiability Modulo Theories) solving tool, i.e. Z3. Z3 allows a direct use of arithmetic in Boolean formulae. Z3 is called by a satisfiability function. First when we compute a node η of the SET related to a path condition π that has been augmented by a condition because of a guard in the WS-STS (an if or a while), a call to the satisfiability function is performed. An SMT file is built to be given to Z3. For the inclusion criterion depicted above for the pruning of redundant paths, we also call this function to check the satisfiability of π σ =⇒ πσ and the complexity is on the number of edges from η to η . Symbolic test case extraction. Symbolic test cases correspond to the SET paths. √ However, it may be relevant to test only paths leading to orchestration termination ( ) even if it is not strongly required for Web services since different instances are run for each test case. Due to our k path length criterion in the SET computation, it follows that symbolic test cases have a length n ≤ k. Notice that we can increase the value of k if we did not find errors during the execution of tests. Online realisation with a constraint solving tool. Since Web services are reactive systems, test case realisation has to be performed step by step, by interacting with the Service Under Test (SUT). This is to avoid emitting erroneous verdicts. Take a path pl.o?x.pl.o!y, with σ = {x → vs0 , y → vs1 } and π = vs0 > 2∧vs1 > vs0 . Realisation all-at-once would yield a realised path p?vs0 , p!vs1 with, e.g., {vs0 → 3, vs1 → 4}. Suppose now we send message p with value 3 to the SUT and that it replies with value 5. We would emit a Fail verdict (5 = 4), while indeed 5 would be a correct reply (5 > 3). Concretely, the tester is implemented as a reactive process that mirrors the observable behaviour of the test case. We begin with the PC in the last state of the SET path corresponding to the test case. Whenever the tester has to send a message to the SUT (USER.o? in the test case), PC is solved to get correct data values to send. Whenever the tester has to receive a message from the SUT (USER.o! in the test case) a timeout is run. If it ends before we receive the message there is an Inconclusive verdict. If we receive the message from the SUT, data is extracted from it and the PC is updated with a correspondence between the reception variable(s) and the data, and PC is then checked to see if the data is correct or not. In both cases, we rely on a constraint solving tool, the Z3 solver. Call of the Z3 Solver for the Online Realisation. We present in the following how the solving function based on the call of the Z3 tool is used by the online algorithm to retrieve values in order to execute the tests. The solving function takes as inputs variables and their types, a predicate represented as a conjunction of clauses and a set of words that will be used as a small database to assign a string to a string variable. This function supports the use of the following datatypes: boolean (Bool) integer (Int) double (Real) and string (String). For sake of simplicity we use only the type Int in the sequel. First, the function builds for each variable its associated tree. Each tree is constructed according to its variable type. Hence, if it is a simple type variable then the associated tree will be reduced to a simple type leaf. Else if it is a complex type variable, then an isomorphic tree will be constructed. We need to manipulate tree types to be compliant with simple or complex types of XSD schema. Let us assume a variable x, a

46

L. Bentakouk, P. Poizat, and F. Za¨ıdi

variable y and a complex type named T1 . T1 consists of a part a and another part b, both are integer. We provide the following annotation for this complex type: T1 : {a : Int, b : Int}. The variables x and y have the following types: x :: Int and y :: T1 . The corresponding tree for x will be reduced to an integer leaf and the one associated to y is a tree with an integer leaf through a and an integer leaf through b. Notice that we use the term Labels to refer to a or b (the label a and the label b). We give below a graphical view of the trees. y

◦

b

•

a

x

Once the trees are constructed, a new variable is assigned to each leaf of the tree. These new variables will be used as pointers when the instantiation of variables of complex type will be performed. In fact, each path of the tree starting from the root to a leaf is represented with a unique variable as shown below: y v1 •

◦

b

•

a

x

• v2

Once the previous step is achieved, we process the predicate. As we said above a predicate is a conjunction of clauses. Theses clauses may represent an equivalence operations between variables or/and paths of trees as well as arithmetic and logic operations. A path in predicate clause is represented with an XP ath expression. If we consider the same previous variables x and y and the following predicate: x = 3 + 4 ∧ y/b = x, the clause y/b = x expresses the equivalence between the value of the variable x and the value of the location pointed by the path y/b. If the XP ath expression y/b does not correspond to a leaf, but to a sub-tree then a new variable is created to represent this sub-tree. We would like to point out that if the XP ath expression leads to a leaf, then the path is replaced by its associated variable. Let us consider z an input variable (as x and y) and a new complex type T2 where: T2 : {a : Int, b : T1 } and z :: T2 . The associated predicate to those variables is specified as: x = 3 + 4 ∧ y/b = x ∧ z/b = y, in this case a new variable will be created by the program to represent z/b. The graphical view is given below. z

◦ v6 v3 • ◦ v4 • • v5

a

b

• v2

b

v1 •

◦

a

y

b

•

a

x

The next step is to submit the predicate using the new variables instead of the XP ath expressions to the Z3 solver. The function generates an SMT input file for Z3. This input file represents a problem to be submitted to Z3 and has to satisfy the following BNF rules (note that we use the teletype font to indicate the part that will appear in the Z3 input file).

Conformance Testing of Service Orchestration Using an SMT Solver Problem Prefix Theory Var-Declaration Constraint-Formula Suffix Labels-Declaration String-Declaration Tree-Declaration

47

::= Prefix Theory Var-Declaration Constraint-Formula Suffix ::= ” ( benchmark data ” ::= Labels-Declaration String-Declaration Tree-Declaration ::= ” :extrafuns (” (variableName variableType)+ ”)” ::= ”:formula ( and ” variable-Structure constraintClauses”)” ::= ” )” ::= ” :datatypes ((label ” Labels∗ ” ) ” ::= ” (String (”word+ | ”noString)” ) ::= ”(tree (nil) (IntLeaf (val Int)) (RealLeaf (val Real))(StringLeaf (val String)) (BoolLeaf (val Bool)) (cons (firstChildLab label) (firstChild tree) (siblingChilds tree))))”

A problem is divided into three main sub-parts: the Theory, the Var-Declaration and the Constraint-Formula. The Theory is used to specify the string words allowed as a given database and also to specify the structure of a complex type variable. The second part, i.e. the Var-Declaration, allows to declare each variable and its type. The different types can be: Int, Real, Bool or a tree type. Finally the Constraint-Formula subpart allows to represent at first the structure of a complex variable type as specified by the theory (this corresponds to the ”variable-Structure” in the rules above). Secondly the predicate (or the constraint), presented as a conjunction of clauses, will be written into a particular form (this corresponds to the constraintClauses in the rules above). To explain this last point, if we consider the very simple example x = 3 + 4, the corresponding constraint with the concrete syntax of Z3 will be written as follows: (= x (+ 3 4)). The Labels∗ represent the set of tree labels of complex type variables, there is no labels for simple type variable. word+ represent the words given as input to the program. The Tree-Declaration defines a tree structure. We give below an example of a tree given as an input to the solving function and how we encode it with the concrete syntax of Z3. We give a linear representation of the tree, with the label connected to its ancestor, its first children with its descendants and all the siblings of this children and their descendants. The nil constructor represents an empty tree. Example. Let us consider the variable w with a tree as graphically represented below. The associated translation in the concrete syntax of Z3 is as follows. We give also the graphical representation of the encoding of the tree in Z3. Let us note that in the Z3 graphical representation we have the nil symbol associated to w for the empty tree for the sibling of the root w. Nevertheless, in the constraints below we simplify the structure by avoiding the nil for w as we are interested in the tree accessible through the label connected to w. (= w (cons a (IntLeaf v0 ) (cons b (cons d (IntLeaf v1 ) (cons e (IntLeaf v2 ) (nil)))(cons c (cons f (IntLeaf v3 ) (cons g (IntLeaf v4 ) (nil)))) (nil))))

48

L. Bentakouk, P. Poizat, and F. Za¨ıdi

w

◦

a v0 • d v1 •

b

w

⊥

a

b

v0

d

e

⊥ f

g

v1

v2

v3

v4

c

◦

◦ e f • v2 v3 •

⊥

c

g • v4

⊥

Once the SMT input file for Z3 is generated, we submit it to the solver. This one can return an instantiation of variables if the predicate is satisfiable (sat), an unsatisfiable answer (unsat) otherwise. An other answer could be given which is the unknown response (unknown) if the solver cannot give a response in a given amount of time. The solving function retrieves the values returned by the solver and assigns them to the variables. In the case of variables represented as trees, the leaves variables will receive the different values and thus complete the construction of trees. The instantiated variables will be used during the test case execution. The function can be easily extended to interact with different solving tool such as Alt-Ergo [30] for instance. We need to add some methods for the dumping of the problem to be solved in the syntax accepted by the tool and also to add a method to retrieve the responses of this tool. Observability and controllability. When the tester receives an unexpected message (USER.o! in the test case), it does not always corresponds to an error (and hence, to a Fail verdict). This is due first to non-controllability of the SUT outputs (one cannot √ prevent a SUT to send a message). Moreover, due to non observable events ({τ, χ, }) and internal communications with sub-services in orchestrations, the state we are in the test case execution, say η, does not always correspond to the state in which a correct implementation may have evolved. Therefore, whenever we receive an unexpected message, we check if it is in may(η). If it is the case, we produce an Inconclusive verdict, else a Fail verdict. Notice that the Inconclusive verdict related to may(η) is a consequence of the reception of an event related to the test of another branch of the test purpose. The conformance relation considered is a symbolic input-output trace inclusion as defined in [14]. Application. For the example e-Conference (see Section 3), we have defined five test purposes that allow to cover the different functional behaviour of the service. One is related to a demand of a user with a country having a high alert. Two others are related to an order with a country with no alert and with either a fees or package refund. The two last are related to orders with fees refund and a variation on the number of addfee. For each test purpose, we have produced several test cases, and only one can be executed with an uniformity hypothesis on the different values of the variables. In this work, we focus on generating test cases and solving constraints in order to first compute the SET then use the instantiate data returned by the solver to interact with the implementation. The interaction with the implementation can be done using the soapUI tool as depicted in [4].

Conformance Testing of Service Orchestration Using an SMT Solver

49

7 Conclusion and Perspectives The emergence of business models based on service composition calls for dedicated verification techniques. In this paper we have presented a formal testing framework addressing this issue. Formal models are retrieved from a real service composition language, WS-BPEL, thanks to a comprehensive set of transformation rules. Using symbolic models and their symbolic execution semantics, model state space explosion can be avoided. We do not only address test case generation but also tackle the way test cases are run against a service implementation. Using symbolic test purposes, the designer of an orchestration, or anyone interested in reusing it in a value-added composition, may have the testing process focus on functional properties of interest. Our framework is automated using prototypes. Our framework has been applied to realistic-size case studies. The main perspective of this work is to go beyond a pure black-box approach, defining an off-context testing approach where the test oracle would simulate orchestration partners. We also plan to define a language based on workflow (like BPMN) to express the test purposes. Acknowledgement. This work was supported by the project “WEB service MOdelling and Validation” (WEBMOV) of the French National Agency for Research.

References 1. OASIS: Web Services Business Process Execution Language (WSBPEL) Version 2.0. Technical report, OASIS (April 2007) 2. ter Beek, M.H., Bucchiarone, A., Gnesi, S.: Formal Methods for Service Composition. Annals of Mathematics, Computing & Teleinformatics 1(5), 1–10 (2007) 3. Poizat, P., Royer, J.C.: A Formal Architectural Description Language based on Symbolic Transition Systems and Modal Logic. JUCS 12(12), 1741–1782 (2006) 4. Bentakouk, L., Poizat, P., Za¨ıdi, F.: A Formal Framework for Service Orchestration Testing based on Symbolic Transition Systems. In: Proc. of TESTCOM, pp. 16–32 (2009) 5. Bentakouk, L., Poizat, P., Za¨ıdi, F.: A Formal Framework for Service Orchestration Testing based on Symbolic Transition Systems. Long version, in P. Poizat Webpage (2009) 6. Bozkurt, M., Harman, M., Hassoun, Y.: Testing Web Services: A Survey. Technical report, King’s College London (2010) 7. Mayer, P.: Design and Implementation of a Framework for Testing BPEL Compositions. PhD thesis, Leibnitz University, Germany (2006) 8. Bartolini, C., Bertolino, A., Marchetti, E., Parissis, I.: Data flow-based validation of web services compositions: Perspectives and examples. In: de Lemos, R., Di Giandomenico, F., Gacek, C., Muccini, H., Vieira, M. (eds.) Architecting Dependable Systems V. LNCS, vol. 5135, pp. 298–325. Springer, Heidelberg (2008) 9. Bartolini, C., Bertolino, A., Marchetti, E., Polini, A.: Towards automated WSDL-based testing of web services. In: Bouguettaya, A., Krueger, I., Margaria, T. (eds.) ICSOC 2008. LNCS, vol. 5364, pp. 524–529. Springer, Heidelberg (2008) 10. Hall´e, S., Hughes, G., Bultan, T., Alkhalaf, M.: Generating interface grammars from wsdl for automated verification of web services. In: ICSOC/ServiceWave, pp. 516–530 (2009) 11. Lallali, M., Za¨ıdi, F., Cavalli, A., Hwang, I.: Automatic Timed Test Case Generation for Web Services Composition. In: Proc. of ECOWS (2008)

50

L. Bentakouk, P. Poizat, and F. Za¨ıdi

12. Kaschner, K., Lohmann, N.: Automatic test case generation for interacting services. In: ICSOC Workshops, pp. 66–78 (2008) 13. Jeannet, B., J´eron, T., Rusu, V., Zinovieva, E.: Symbolic test selection based on approximate analysis. In: Halbwachs, N., Zuck, L.D. (eds.) TACAS 2005. LNCS, vol. 3440, pp. 349–364. Springer, Heidelberg (2005) 14. Gaston, C., Le Gall, P., Rapin, N., Touil, A.: Symbolic execution techniques for test purpose ¨ Duale, A.Y., Fecko, M.A. (eds.) TestCom 2006. LNCS, vol. 3964, definition. In: Uyar, M.U., pp. 1–18. Springer, Heidelberg (2006) 15. Frantzen, L., Huerta, M., Kiss, Z., Wallet, T.: On-The-Fly Model-Based Testing of Web Services with Jambition. In: Laneve, C., Su, J. (eds.) WS-FM 2009. LNCS, vol. 6194. Springer, Heidelberg (2010) 16. Cabot, J., Claris´o, R., Riera, D.: UMLtoCSP: a Tool for the Formal Verification of UML/OCL Models using Constraint Programming. In: Proc. of ASE, pp. 547–548. 17. Kov´acs, M., Varr´o, D., G¨onczy, L.: Formal analysis of BPEL workflows with compensation by model checking. Int. Journal of Computer Sciences & Engineering, 35–49 (2008) 18. Mateescu, R., Poizat, P., Sala¨un, G.: Adaptation of service protocols using process algebra and on-the-fly reduction techniques. In: Bouguettaya, A., Krueger, I., Margaria, T. (eds.) ICSOC 2008. LNCS, vol. 5364, pp. 84–99. Springer, Heidelberg (2008) 19. Mateescu, R., Rampacek, S.: Formal Modeling and Discrete-Time Analysis of BPEL Web Services. In: Advances in Enterprise Engineering I, vol. 10. LNBIP, pp. 179–193 (2008) 20. Uchitel, S., Kramer, J., Magee, J.: Synthesis of behavioral models from scenarios. IEEE Trans. Softw. Eng. 29(2), 99–115 (2003) 21. Ziadi, T., H´elou¨et, L., J´ez´equel, J.M.: Towards a uml profile for software product lines. In: Proc. of Software Product-Family Engineering, pp. 129–139 (2003) 22. Gastin, P., Oddoux, D.: Fast LTL to B¨uchi Automata Translation. In: Berry, G., Comon, H., Finkel, A. (eds.) CAV 2001. LNCS, vol. 2102, p. 53. Springer, Heidelberg (2001) 23. Pickin, S., Jard, C., J´eron, T., J´ez´equel, J.M., Traon, Y.L.: Test synthesis from uml models of distributed software. IEEE Trans. Software Eng. 33(4), 252–269 (2007) 24. King, J.C.: Symbolic Execution and Program Testing. Communications of the ACM 19(7), 385–394 (1976) 25. Khurshid, S., P˘as˘areanu, C.S., Visser, W.: Generalized symbolic execution for model checking and testing. In: Garavel, H., Hatcliff, J. (eds.) TACAS 2003. LNCS, vol. 2619, pp. 553– 568. Springer, Heidelberg (2003) 26. Frantzen, L., Tretmans, J., Willemse, T.A.C.: A Symbolic Framework for Model-Based Testing. In: Proc. of FATES/RV (2006) 27. Biere, A., Cimatti, A., Clarke, E.M., Strichman, O., Zhu, Y.: Bounded model checking. Advances in Computers 58, 118–149 (2003) 28. de Moura, L., Bjørner, N.S.: Z3: An efficient SMT solver. In: Ramakrishnan, C.R., Rehof, J. (eds.) TACAS 2008. LNCS, vol. 4963, pp. 337–340. Springer, Heidelberg (2008) 29. Garavel, H., Mateescu, R., Lang, F., Serwe, W.: CADP 2006: A toolbox for the construction and analysis of distributed processes. In: Damm, W., Hermanns, H. (eds.) CAV 2007. LNCS, vol. 4590, pp. 158–163. Springer, Heidelberg (2007) 30. Conchon, S., Contejean, E., Kanig, J.: Ergo: a theorem prover for polymorphic first-order logic modulo theories (2006), http://ergo.lri.fr/papers/ergo.ps

Association of Under-Approximation Techniques for Generating Tests from Models Pierre-Christophe Bu´e, Jacques Julliand, and Pierre-Alain Masson LIFC, Universit´e de Franche-Comt´e 16, route de Gray F-25030 Besan¸con Cedex France {bue, julliand, masson}@lifc.univ-fcomte.fr

Abstract. In this paper we present a Model-Based Testing approach with which we generate tests from an abstraction of a source behavioural model. We show a new algorithm that computes the abstraction as an under-approximation of the source model. Our ﬁrst contribution is to combine two previous approaches proposed by Ball and Pasareanu et al. to compute May, Must+ and Must- abstract transition relations. Proof techniques are used to compute these transition relations. The tests obtained by covering the abstract transitions have to be instantiated from the source model. So, following Pasareanu et al., our algorithm additionally computes a concrete transition relation: the tests obtained as sequences of concrete transitions need not be instantiated from the source model. Another contribution is to propose a choice of relevant parameters and heuristics to pilot the tests computation. We experiment our approach and compare it with a previous approach of ours to compute tests from an abstraction that over-approximates the source model. Keywords: Model-Based Testing, Abstraction, Over and underapproximations.

1

Motivations

The process of software testing can be automated by means of a Model-Based Testing (MBT) approach [1]. A formal behavioural model is designed from which a set of tests is computed that ensure a given coverage of the model. An adaptation layer ﬁlls the gap between the model and the implementation to produce executable tests. The conformance of the implementation to the model is assessed by comparing, modulo the adaptation layer, the outputs of the executable tests with the ones as predicted by the model. A frequent limit to the scalability of this approach is the size of the state space deﬁned by the model: it can be inﬁnite or very large, thus making its coverage impossible in practice. An abstraction of the behavioural model can be used to overcome this limitation. Abstraction techniques [2,3,4] allow for making ﬁnite or drastically reducing the state space representation of a formal speciﬁcation (or of a program), for example by gathering into one single abstract state several concrete states. Our M. Gogolla and B. Wolﬀ (Eds.): TAP 2011, LNCS 6706, pp. 51–68, 2011. c Springer-Verlag Berlin Heidelberg 2011

52

P.-C. Bu´e, J. Julliand, and P.-A. Masson

framework is that of predicate abstraction [2,3], where the states and the transition relation of the abstraction are deﬁned according to a set of predicates over the model variables. Once computed from the abstraction, the tests have to be instantiated on the concrete model. This may not be possible for some of the tests if the abstraction is an over-approximation of the source model, thus deﬁning execution paths that may not exist concretely. We have previously deﬁned and presented in [5] a method where this could happen. The abstraction was computed by a theorem-prover that tried to prove the potential feasibility of the transitions rather than their reachability. As a result, time could be spent uselessly to search for an instantiation that does not exist. We adopt another approach in this paper by considering only under-approximations. Since all the paths of an underapproximation exist in the concrete model, it is possible to instantiate every test from it on the concrete model. We present in this paper a new algorithm, based on previous works from Ball [6] and P˘as˘areanu et al. [7], to compute abstract transition relations based on predicate abstraction for test generation. Our algorithm combines the two approaches of Ball and P˘ as˘areanu et al., and applies to behavioural models instead of programs. We use SAT-solving techniques to compute the transition relations. We also compute a concrete transition relation and try to directly connect concrete states to each other, so as to obtain concrete tests that need not be instantiated from the model. This complements the tests obtained by covering the abstract transition relation, for which an instantiation is required. But what part of the behavioural model will be covered by tests computed from an under-approximation of it? We have implemented the algorithm, and used it to compute tests for six case studies. These experimental results are compared in terms of coverage of the abstraction with the ones obtained by our over-approximation method of [5]. We consider a set of optimisations of the concretisation computed by the algorithm, and evaluate experimentally their practical impact on our case studies. We also provide a set of parameters to the user as well as heuristics to improve the method. The paper is organised as follows. Section 2 presents the process for generating tests from an abstraction. The background required for reading the paper is given in Sec. 3. Two examples to illustrate our approach are described in Sec. 4. The algorithm for computing the abstraction as well as its properties are described in Sec. 5. Section 6 presents the experimental results. We present in Sec. 7 the works to which ours are related. We conclude and indicate future research directions in Sec. 8.

2

Test Generation Process from an Abstraction

In a previous work [5], we have presented a test generation method based on abstraction. The abstraction was computed as an over-approximation of a formal behavioural model M of the system, written by a validation engineer. The engineer also wrote a test purpose, by means of a language proposed in [8], to describe

Association of Under-Approximation Techniques

53

how he intended to test the system, according to his know-how. A set of abstraction predicates was automatically deduced from the test purpose, from which the abstraction was computed. The tests issued from the over-approximation had to be instantiated afterwards on M, which was not always possible since an overapproximation deﬁnes more executions than the concrete model. We improve in this paper the test generation method of [5] by computing an abstraction that is an under-approximation of M rather than an over-approximation. This guarantees that every test generated from it can be instantiated on M. Also, we now instantiate the tests on-the-ﬂy, and not a posteriori as was done in [5]. We sketch our process in Fig. 1. Notice that the abstraction predicates on input of the “Predicate Abstraction and Concretisation” box could be obtained from a test purpose, identically to what was proposed in [5]. We also could synchronise such a test purpose with M before the abstraction computation, to reduce the state space of the resulting abstraction. But we adopt in this paper a more generic presentation of our process. We combine predicate abstraction and SAT-solving for generating the under-approximation. The tests are generated from it by applying a selection criterion that ensures a coverage of the states, of the transitions or of the paths of the abstraction. In our implementation of the process, we have used a chinese postman algorithm to compute tests by covering all the transitions of the abstraction. The tests obtained are valid executions of M, intended to be executed, via the adaptation layer, on the implementation of the system. Structural Selection Criteria

Set of Abstraction Predicates

Behavioural Model M

Predicate Abstraction and Concretisation

Underapproximation A

Test Generation

Tests

Fig. 1. Process of Test Generation from Test Purpose by Abstraction

3

Background

Our behavioural models are described as B event systems, for which we provide the necessary background. We also present the concept of predicate abstraction and formalise the abstraction notions by means of Symbolic labelled Transition Systems (STS), that deﬁne the semantics of the abstractions of event systems. 3.1

Model Syntax

Introduced by J.-R. Abrial [9], a B event system deﬁnes a closed speciﬁcation of a system by a set of events. This syntax is used as an example to present the results in this paper. But these results are generic in the sense that this syntax is a concise form that has the same expressivity as some generic modelling

54

P.-C. Bu´e, J. Julliand, and P.-A. Masson

languages as the guarded actions of [10,11] or the labelled transition systems. In the sequel, we use the following notations: x, y, z are variables and X, Y , Z are sets of variables. I is an invariant and P is used to denote other predicates. The actions of modiﬁcation of the variables are called substitutions in B, following [12] where the semantics of an assignment is deﬁned as a substitution. In B, substitutions are generalized, i.e. the semantics of every kind of action is deﬁned by a substitution calculus that allows to compute the weakest precondition of a substitution a to satisfy the predicate P , denoted [a]P in B. We use a, a1 and a2 to denote B generalized substitutions, and E and F to denote B expressions. All the substitutions allowed in B event systems can be rewritten by means of the ﬁve B primitive forms of Def. 1. Notice that the multiple assignment can be generalized to n variables and that all the substitutions terminate. Definition 1 (Substitution). The following ﬁve substitutions are primitive: – – – – –

single and multiple assignments, denoted as x := E and x, y := E, F substitution with no eﬀect, denoted as skip guarded substitution, denoted as P ⇒ a bounded non-deterministic choice, denoted as a1 []a2 substitution with local variable z, denoted as @z.a.

The substitution with local variable is mainly used to express the unbounded non-deterministic choice denoted by @z.(P ⇒ a). Let us specify that among the usual structures of speciﬁcation languages, the conditional substitution IF P THEN a1 ELSE a2 END is denoted by (P ⇒ a1 )[](¬P ⇒ a2 ) with the primitive forms. Deﬁnition 2 deﬁnes correct B event systems. The events are deﬁned by an equation e = a where e is the name of an event and a is a generalized substitutions modifying the state variables. Definition 2 (Correct Event System). A correct B event system is a tuple X, I, Init, EvDef where: – X is a set of state variables, – I is an invariant predicate over X, – Init is a substitution called initialisation, such that the invariant holds in any initial state, i.e. [Init]I is valid, – EvDef is a set of event deﬁnitions in the shape of e = a such that every event preserves the invariant, i.e. I ⇒ [a]I is valid. 3.2

Predicate Abstraction

Predicate abstraction [2] is a special instance of the framework of abstract interpretation [3] that maps the potentially inﬁnite state space C of a transition system onto the ﬁnite state space Q of a symbolic transition system via a set of predicates P = {p1 , p2 , . . . , pn } over the model variables. A state of C is a valuation of the state variables of the model. The set of abstract states Q contains at most 2n states. Each state is a tuple q = (q1 , q2 , . . . , qn ) with qi being

Association of Under-Approximation Techniques

55

n equal either to pi or to ¬pi , and we also consider q as the predicate i=1 qi . We deﬁne an abstraction function αP : C → Q such that αP (c) is an abstract state q where c satisﬁes qi for all i ∈ 1..n. By a misuse of language, we say that c is in q. The predicate abstraction is based on ﬁve primitive functions denoted as follows. We denote by wp(a, q ) the weakest precondition [11] of an action a to reach a target state deﬁned by a predicate q . The weakest precondition wp(a, q ) is the largest set of states from which the execution of a necessary leads to a state that satisﬁes q . In B, wp(a, q ) can be computed by substitution calculus, denoted by [a]q . The weakest conjugate precondition [13], denoted by wcp(a, q ) is equal to ¬[a]¬q in B, i.e. ¬wp(a, ¬q ). The conjugate weakest precondition wcp(a, q ) is the largest set of states from which the execution of a can lead in a state that satisﬁes q . Since the events always terminate in B, wcp is identical to wp for the deterministic subset of B. In contrast, for a non deterministic substitution, wp(a, q ) ⇒ wcp(a, q ) because there may exist states for which a non deterministic choice leads to q or not, depending on the chosen branch. These states satisfy wcp(a, q ) but not wp(a, q ). We denote by sp(a, q) the strongest postcondition of an action a from a source state deﬁned by a predicate q. It is the smallest set of states reached by the execution of a from a state that satisﬁes q. We denote by grd(a) the condition under which the action a is triggerable. It is deﬁned as in B: grd(skip) = true, grd(x, y := E, F ) = true, grd(P ⇒ a) = grd(a1 ) ∨ grd(a2 ), grd(@z.a) = ∃z.grd(a). Last, we P ∧ grd(a), grd(a1 []a2 ) = denote by SAT (P ) the satisﬁability value of a predicate P . Let us now deﬁne the abstract transitions as may-transitions. Consider two abstract states q and q and an event e. There exists a may transition from q e to q by e, denoted by q → q , if and only if there exists a concrete transition e c → c where c and c are concrete states with αP (c) = q and αP (c ) = q . If we assume the event e to be deﬁned by an action a, there is a may transition e q → q iﬀ SAT (wcp(a, q ) ∧ q). As in [6], we deﬁne two other kinds of abstract transitions: must- and must+. The must+ transitions are may transitions that are triggerable from all the concrete states of the abstract source state. The must- transitions are may transitions that reach all the concrete states of the abstract target state. Let e e = a be an event deﬁnition. There exists a must+ transition q → q iﬀ q ⇒ wp(a, q ) ∧ grd(a) is valid, i.e. ¬SAT (¬(wp(a, q ) ∧ grd(a)) ∧ q). That is, for any state c concretising the abstract state q, there is a concrete state c in q e e such that c → c . There exists a must- transition q → q iﬀ q ⇒ sp(a, q) is valid, i.e. ¬SAT (¬sp(a, q) ∧ q ). That is for any concrete state c for which αP (c ) = q , e there is a concrete state c for which αP (c) = q and such that c → c . 3.3

Abstraction Formalisation

We deﬁne in Def. 3 a kind of STS well suited to represent abstractions. Deﬁnition 4 associates an abstraction deﬁned by an STS to an event system. Definition 3 (Symbolic Labelled Transition System (STS)). Let Ev be a ﬁnite set of event names. Let A be a ﬁnite set of symbolic states on P. Let C be

56

P.-C. Bu´e, J. Julliand, and P.-A. Masson

a ﬁnite or inﬁnite set of concrete state and αP an abstraction function from C to A. A tuple Q, Q0 , C0 , Δ, Δ+ , Δ− , Γ, Δc is an STS if it satisﬁes the following conditions: Q(⊆ A) is a ﬁnite set of states, Q0 (⊆ Q) is a set of abstract initial states, C0 (⊆ C) is a set of concrete initial states, Δ(⊆ Q × Ev × Q) is a may labelled transition relation, Δ+ (⊆ Q×Ev ×Q) is a must+ labelled transition relation such that Δ+ ⊆ Δ, Δ− (⊆ Q × Ev × Q) is a must- labelled transition relation such that Δ− ⊆ Δ, Γ (⊆ Q × C) is a concretisation relation that associates concrete states to any abstract state such that (q, c) ∈ Γ ⇒ αP (c) = q, e – Δc (⊆ C × Ev × C) is a concrete transition relation such that c → c ∈ Δc e iﬀ c ∈ C ∧ c ∈ C ∧ q → q ∈ Δ ∧ (q, c) ∈ Γ ∧ (q , c ) ∈ Γ . – – – – – – –

Definition 4 (STS associated to an ES). Let ES = X, Init, {e = a|e ∈ Ev} be an event system and P = {p1 , p2 , ..., pn } be a set of n predicates deﬁning a set of 2n abstract states A = {p1 , ¬p1 } × {p2 , ¬p2 } × ... × {pn , ¬pn }. A tuple Q, Q0 , C0 , Δ, Δ+ , Δ− , Γ, Δc is an STS associated to ES and P if it satisﬁes the following conditions: – – – – – – – –

4

{q|q ∈ A ∧ SAT (sp(Init, true) ∧ q)} , Q0 = C0 = {c|∃q0 .(q0 ∈ Q0 ∧ c ∈ C ∧ c = SAT (wcp(Init, q0 )))}, e Δ= {q → q |q ∈ A ∧ q ∈ A ∧ e ∈ Ev ∧ SAT (wcp(a, q ) ∧ q)}, e e Q= {q|∃(q , e).(q → q ∈ Δ ∨ q → q ∈ Δ)}, e {q → q |q ∈ A ∧ q ∈ A ∧ e ∈ Ev ∧ ¬SAT (¬(wp(a, q ) ∧ grd(a)) ∧ q)}, Δ+ = e Δ− = {q → q |q ∈ A ∧ q ∈ A ∧ e ∈ Ev ∧ ¬SAT (¬sp(a, q) ∧ q )}, e Δc (⊆ C × Ev × C) is such that c → c ∈ Δc ⇒ SAT (wcp(a, c ) ∧ c), e e Γ = {(q, c)|q ∈ Q ∧ c ∈ C ∧ αP (c) = q ∧ ∃(e, c ).(c → c ∈ Δc ∨ c → c ∈ Δc )}.

Examples

For the sake of readability, let us now introduce two examples that will be used to illustrate our propositions. Each one is intended to illustrate a diﬀerent point while remaining small and easy to read. The Electrical System example of Sec. 4.1 is a ﬁnite state control and command system that illustrates the various kinds of transition relations Δ, Δ− and Δ+ , as represented in Fig. 3. The example of Sec. 4.2 is a very simple model, but with an inﬁnite state space. Its aim is to illustrate that combining the two under-approximation techniques gives better transition and path coverage than each one separately (see Sec. 5.3, paragraph “Comparison with the other methods”).

Association of Under-Approximation Techniques X I Init T ic Com

F ail

Rep

(a) Physical tion

57

= {H, Sw, Bat} = H ∈ {tic, tac} ∧ Sw ∈ 1..3 ∧ (Bat ∈ 1..3 → {ok, ko}) ∧ Bat(Sw) = ok = H, Sw, Bat := tac, 1, {1 → ok, 2 → ok, 3 → ok} = H = tac ⇒ H := tic = ∃(i, j).(i ∈ 1..3 ∧ j ∈ 1..3 ∧ i = j ∧ Bat(i) = ok ∧ Bat(j) = ok) ∧ H = tic ⇒ @ns.(ns ∈ 1..3 ∧ Bat(ns) = ok ∧ ns = Sw ⇒ H, Sw := tac, ns) = ∃(i, j).(i ∈ 1..3 ∧ j ∈ 1..3 ∧ i = j ∧ Bat(i) = ok ∧ Bat(j) = ok) ⇒ @nb.(nb ∈ 1..3 ∧ Bat(nb) = ok) ⇒ (nb = Sw ⇒ @ns.(ns ∈ 1..3 ∧ ns = Sw ∧ Bat(ns) = ok ⇒ Sw, Bat(nb) := ns, ko)) [](nb = Sw ⇒ Bat(nb) := ko)) = @nb.(nb ∈ 1..3 ∧ Bat(nb) = ko) ⇒ Bat(nb) := ok)

Representa-

(b) Speciﬁcation

Fig. 2. The Electrical System and its Formal Behavioural Speciﬁcation

4.1

Electrical System Example

Figure 2(a) shows a device D powered via a switch to one of three batteries B1 , B2 , B3 . A clock H periodically sends a signal that causes a commutation of the closed switch. The system has to meet the following requirements: one switch and only one is closed at a time and a clock signal changes the switch that is closed. The batteries may break down. If it happens to the one that is powering D, an exceptional commutation is triggered. We assume that there is always at least one battery working. When there is only one battery working, the clock signals are ignored. Figure 2(b) models the system by means of three variables. H models the clock and takes two values: tic to ask for a commutation and tac when it has occurred. Sw models the switches by an integer that indicates which one is closed. Bat models the batteries breakdowns by a total function that associates ok or ko (for a broken battery) to each battery. The state changes are described by means of four events: Tic sends a commutation signal, Com changes the closed switch, Fail breaks down a battery and Rep repairs a battery. Figure 3 shows the STS without C0 , Γ and Δc that abstracts the model of Fig. 2(b) from the set of abstraction predicates P = {p1 , p2 } where p1 = H = tic and p2 = ∃(i, j).(i ∈ 1..3 ∧ j ∈ 1..3 ∧ i = j ∧ Bat(i) = ok ∧ Bat(j) = ok). All the transitions belong to Δ. Those followed by ”-” and/or ”+” belong to Δ− and/or Δ+ . In Fig. 7(a) and Fig. 7(b), we see a fragment of C0 , Γ and Δc . 4.2

Simple Illustrative Model

The speciﬁcation presented in Fig. 4 models a small conditional computation over a variable x. Its semantics is an inﬁnite state transition system for unbounded integers. Our abstraction method computes the ﬁnite state symbolic transition system of Fig. 5(a). The variable pc is not abstracted whereas x is abstracted according to the two predicates x < 3 and x ≥ 3. Figure 5(a) shows the STS that abstracts the model of Fig. 4 from the set of abstract states A = {x < 3, x ≥ 3} × {pc = 0, pc = 1, pc = 2, pc = 3}. The numbered points represent concrete states by deﬁning the x value. The may

58

P.-C. Bu´e, J. Julliand, and P.-A. Masson Rep F ail

Rep F ail

T ic, −, + q0 {¬p1 , p2 }

Rep, +

q2 {p1 , p2 }

Com, −, +

F ail, −

q1 {¬p1 , ¬p2 }

Rep, +

T ic

F ail, −

q3 {p1 , ¬p2 }

Fig. 3. STS of the Abstract Electrical System

X I Init e1 e2 e3 e4 e5

= = = = = = = =

{pc, x} pc ∈ 0..3 ∧ x ∈ Z @z.(x ∈ Z ⇒ pc, x := 0, z) pc = 0 ∧ x < 3 ⇒ pc, x := 1, x + 1 pc = 0 ∧ x ≥ 3 ⇒ pc, x := 1, x − 1 pc = 1 ∧ x < 3 ⇒ pc, x := 2, x + 1 pc = 1 ∧ x ≥ 3 ⇒ pc, x := 2, x − 1 pc = 2 ⇒ pc := 3

Fig. 4. A Simple Illustrative Model

existential transition relation Δ is represented by the dashed arrows. It deﬁnes an over-approximation. The transitions of Δ− and Δ+ are labelled with ”+” and/or ”-”. A Δc relation is represented by the full arrows. All the concrete states in q0 and q1 belongs to C0 .

5

Abstraction Computation

We ﬁrst present in Sec. 5.1 the basic algorithm to compute an existential overapproximation deﬁned by Δ. Some implementation-dependent heuristics have been abstracted. This algorithm also computes Δ+ and Δ− , and we improve it by computing on-the-ﬂy a concrete transition relation Δc that is an underapproximation obtained by a partial concretisation of Δ. We improve in Sec. 5.2 the under-approximation computation of Δc by taking the union with an under approximation deﬁned in [6] from Δ+ and Δ− . Last, we discuss in Sec. 5.3 the properties of this combined approach. 5.1

Under-Approximation by Existential Concretisation

The algorithm of Fig. 6 relies on satisﬁability evaluations of predicates by means of a SAT -solver. We consider a SAT -procedure as returning either the false value

Association of Under-Approximation Techniques q0 , {pc = 0, x < 3}

q2 , {pc = 1, x < 3} e1 , −

0 1 2

e3 , −

0 1

e1

2

e2 q1 , {pc = 0, x ≥ 3} 3

q4 , {pc = 2, x < 3}

e3

e4 q3 , {pc = 1, x ≥ 3} e2 , −

3

e4 , −

0

q6 , {pc = 3, x < 3} e5 , −, +

0

1

1

2

2

q5 , {pc = 2, x ≥ 3}

q7 , {pc = 3, x ≥ 3}

3

59

e5 , −, +

3

4

4

4

4

5

5

5

5

(a) Abstraction of the Small Model of Fig. 4

t1 t2 t3 t4 t5

e

1 = b q0 .0 → q2 .1 e1 = b q0 .2 → q3 .3 e2 = b q1 .3 → q2 .2 e2 = b q1 .4 → q3 .3 e2 = b q1 .5 → q3 .4

e

e

3 5 → q4 .2 → q6 .2, e4 e5 → q4 .2 → q6 .2, e3 e5 → q5 .3 → q7 .3, e4 e5 → q4 .2 → q6 .2, e4 e5 → q5 .3 → q7 .3.

(b) Concrete Tests

Fig. 5. Abstraction and Tests for the Small Model of Fig 4

if the predicate is not satisﬁable, or a concrete state c otherwise, that is also interpreted as the true value. In practice, SAT -solvers may also return an unknown value when they manage to prove neither the satisﬁability nor the unsatisﬁability of a predicate. We liken this unknown value to false in our algorithm. The algorithm computes an abstraction (an STS) in two steps. Lines 1-5 compute the initial abstract and concrete states Q0 and C0 while lines 7-33 compute the set Q of reachable abstract states and the transitions of Δ, Δ+ , Δ− and Δc . QR is the set of source states whose successors have to be computed, either because they are initial, or target states of reachable transitions of Δ. To compute the successors of a state q (lines 9-32), the algorithm enumerates all the possible target states q (line 10) and all the events (line 11) that could e lead to q . The transition q → q is added to Δ when wcp(a, q ) ∧ q is satisﬁable (lines 12-14). Lines 15-19 search for a concretisation c of the target state q , that would be the target of an existing concrete state c of the source state q. If such e a c is found (line 20), the transition c → c is added to Δc (line 21), otherwise e a transition nc → c is added (line 23-24) where nc is the new concretisation of q resulting of the satisﬁability condition SAT (wcp(a, q ) ∧ q) that guarantees its existence in lines 12-13. Line 26 stores the concretisation of the target state in Γ . If q was not already known as reachable (q ∈ / Q), it is added to QR (line 27). Then, if there exists a may transition, lines 28-29 complete the must+ and must- transitions when they exist. Finally, this algorithm computes a ﬁrst under-approximation deﬁned by C0 and Δc by concretising Δ. After that, a second step, described in the next section, completes this under-approximation by concretising Δ+ and Δ− . 5.2

Under-Approximation by Universal Concretisation

T. Ball proposes in [6] a method to compute the Δ, Δ+ and Δ− transition relations as deﬁned in Sec. 3. An under-approximation L is deﬁned by the set of states that are reachable from the initial ones by a sequence of must- transitions,

60

P.-C. Bu´e, J. Julliand, and P.-A. Masson

Let C be the set of concrete states of all the abstract states of the set Q Inputs X, Init, EvDef : an Event System where EvDef = {e = a|e ∈ Ev} Results Q, Q0 , C0 , Δ, Δ+ , Δ− , Γ, Δc : a Symbolic Transition System Variables QR : Set of abstract states remaining to be handled CR : Set of concrete states remaining to be handled q, q : source and target abstract states of the current transition c, c : source and target concrete states of respectively q and q e : event name of the current transition nc : other concrete state of the source state q Begin /* Computation of the initial abstract and concrete states and of a concrete instance */ (1) Q0 := ∅ ; Γ := ∅ ; C0 := ∅ ; (2) ForAll q ∈ A Do (3) c := SAT (sp(Init, true) ∧ q) ; (4) If c Then Q0 := Q0 ∪ {q} ; C0 := C0 ∪ {c} ; Γ := Γ ∪ {(q, c)} EndIf (5) EndForAll ; (6) /* Computation of the reachable states Q, the transitions in Δ, Δ+ , Δ− and the concrete transitions in Δc */ (7) Δ := ∅ ; Q := ∅ ; QR := Q0 ; Δc := ∅ ; Δ+ := ∅ ; Δ− := ∅ ; (8) While QR = ∅ Do (9) Choose q in QR ; QR := QR − {q} ; Q := Q ∪ {q} ; (10) ForAll q ∈ A Do (11) ForAll e ∈ Ev Do /* e = a in EvDef */ (12) nc := SAT (wcp(a, q ) ∧ q) ; (13) If nc Then /* nc is false if wcp(a, q ) ∧ q is not satisfiable, true otherwise */ e (14) Δ := Δ ∪ {q → q } ; (15) CR := Γ ({q}) ; c := f alse ; /* Γ (Z) is the relational image of the set Z */ (16) While ¬c ∧ CR = ∅ Do (17) Choose c in CR ; CR := CR − {c} ; (18) c := SAT (sp(a, c) ∧ q ) (19) EndWhile ; (20) If c Then e (21) Δc := Δc ∪ {c → c } (22) Else /* There is no concrete target state for the existing concrete source states */ (23) c := SAT (sp(a, nc) ∧ q ) ; /* we compute one from nc */ e (24) Γ := Γ ∪ {(q, nc)}; Δc := Δc ∪ {nc → c } (25) EndIf ; (26) Γ := Γ ∪ {(q , c )} ; (27) If q ∈ / Q Then QR := QR ∪ {q } EndIf ; e (28) If ¬SAT (¬(wp(a, q ) ∧ grd(a)) ∧ q) Then Δ+ := Δ+ ∪ {q → q } EndIf ; e (29) If ¬SAT (¬sp(a, q) ∧ q ) Then Δ− := Δ− ∪ {q → q } EndIf ; (30) EndIf (31) EndForAll (32) EndForAll (33) EndWhile End

Fig. 6. Computation of an Existential Concretisation

followed by at most one may transition and a sequence of must+ transitions. Ball deﬁnes a reachability function for an STS to formalise L. Let Q be a set of states and Δ a transition relation. The reachability function on Δ and Q ⊆ Q is deﬁned by R[Δ](Q ) = μZ.(Q ∪ Δ(Z, Ev)) where μ is the least ﬁx-point and Δ(Z, Ev) is the relational image of the sets Z and Ev by Δ. We adapt the deﬁnition of L as: L = {q | ∃ (q, e, q ) · (q ∈ R[Δ− ] (Γ −1 (R[Δc ](C0 )))) ∧ e → q ∈ Δ) ∧ (q = q ∨ q − + q ∈ R[Δ ]({q })}. For example, this process applied to the speciﬁcations of Fig. 2(b) and 4, gives the transition relations Δ− and Δ+ represented respectively by the transitions labelled with ”-” and/or ”+” in Fig. 3 and in Fig. 5(a). A contribution of our paper is to combine a concretisation of the underapproximation L with the one computed by our algorithm of Fig. 6. After the

Association of Under-Approximation Techniques

61

existential concretisation performed by our algorithm of Fig. 6, we concretise the longest executions of L that start in a concrete state that is reachable from the initial ones. The concretisation of the sequences of must+ transitions is performed by a forward depth search, detecting the cycles from one concrete state beginning such a sequence. The concretisation of the sequences of musttransitions is performed by a backward depth search detecting the cycles from one of the ﬁnal concrete states of these sequences. Finally, the relation Δc is the union of the relation computed by the algorithm of Fig. 6 with the one computed by this concretisation of L. For the example of Fig. 5(a), the ﬁnal Δc relation is represented by the full e2 q3 .4 concretises the ﬁrst transition of the following arrows. The transition q1 .5 → e2 ,− e4 ,− e5 ,+ sequence in L: q1 → q3 → q5 → q7 . Notice that this transition does not exist in the concretisation of Δ, but only in that of L. Finally, the whole underapproximation made of all the concrete transitions allows for generating the ﬁve test cases shown in Fig 5(b). They cover all the abstract states and all the concrete transitions of the STS of Fig 5(a). But they do not cover all the may e1 executions of the abstract model. For example, the abstract execution q0 → e4 e5 q3 → q5 → q7 is not covered, but it is not executable. This shows the interest of generating the tests from the under-approximation. This avoids generating non executable tests that give birth to a costly process of searching for an execution that does not exist. 5.3

Discussion about the Properties of the Algorithm

In this section, we explain why the algorithm of Fig. 6 terminates and is sound. We then explain that the enumeration order of the states and events in lines 1011 impacts the accuracy of the computed under-approximation. Last we compare our combined method with the two methods of [6,7]. Termination. Since A is a ﬁnite set, the external loop terminates in the worst case with Q = A. The loops that compute the initial states (lines 1-5) and the two loops of lines 10-11 terminate because the sets A and Ev are ﬁnite. The internal one (lines 16-19) also terminates because the number of concrete states built for all the abstract states is ﬁnite. This number is bounded by (|A| × |Ev|) × 2 + 1 in the worst case where, for any state, any event is ﬁreable leading to any other state and is reachable by any event from every other state. Soundness. The transition relation Δc is an under-approximation of the concrete transition system associated to an event system. On the one hand, the algorithm of Fig. 6 computes a subset of the concrete states of the model associated to an event system. On the other hand, all the states verify the correction conditions. e e For a transition c → c concretising a transition q → q , the source state c veriﬁes the condition SAT (wcp(a, q ) ∧ q), and the target state c veriﬁes the condition SAT (sp(a, c) ∧ q ) (see lines 18 and 23). Search Order and Accuracy of the Abstractions. The number of reachable concrete executions and their size depend on the enumeration order of the events

62

P.-C. Bu´e, J. Julliand, and P.-A. Masson q0 q0 q0 , c1 tac, 1, ok, ok, ok Rep

q0 , c5 tac, 1, ok, ok, ok

q1 T ic

F ail

q0 , c2 tac, 1, ok, ko, ok

q1 , c4 tic, 1, ok, ok, ok

q2 q0 , c3 tac, 3, ok, ok, ok

q2 F ail

q2 , c3 tac, 1, ok, ko, ko

Rep

F ail

q0 , c4 tac, 3, ok, ko, ok

(a) First Concretisation of q0

F ail

q2 , c1 tac, 3, ko, ko, ok q1

T ic

q1 , c2 tic, 3, ok, ko, ok

(b) Second Concretisation of q0

Fig. 7. Concretisations of q0 for the Abstract Electrical System

and concrete states, as Figs. 7(a) and 7(b) illustrate. Two test cases with a total of ﬁve test steps could be generated from the approximation of Fig. 7(a), while no test could be generated from the one of Fig. 7(b). Indeed, c5 , the concrete initial and reachable state of Fig. 7(b), is not connected to any concrete transition. There are three reasons for that. Concretising the reﬂexive transition Rep before F ail generates the state c4 , because Rep is not ﬁreable in the state c5 . The Rep transition does not connect c4 to c5 because c4 has been built by the solver with an upper limit choice for the connected battery, i.e. battery 3, while the connected battery in c5 is battery 1 (Rep doesn’t change the connected battery). Consequently, the state c3 is generated. Then F ail is applied, and as the possible concrete target states are enumerated in reverse order w.r.t their creation, c3 is connected to c4 rather than to c5 . As a result, no transition is reachable from the state c5 . This illustrates that according to the building order of the abstraction, the under-approximation contains more or less reachable concrete transitions. Notice that our implementation optimises the size of the under-approximation by considering the heuristics and parameters presented in Sec. 6.1. Comparison with the other Methods. The combined method that we propose generates under-approximations that are more accurate than with any of the two methods presented in [6,7]. These two methods are not comparable as illustrated by Fig. 5(a) and shown in [7]. The concretisation of the existential abstraction, as in [7], allows for generating the four test cases t1 to t4 of Fig. 5(b). Our method additionally generates the test case t5 thanks to the concretisation of L. In contrast, the method of [6] would only generate the test cases t1 and t5 . Our method ﬁnally generates the union of the test cases generated by each of the two methods.

6

Implementation and Experimentations

We present in this section some optimisations to the algorithm of Fig 6. Our experimentations show their impact on test generation. 6.1

Implementation and Optimisation

We have used the SMT-solver Z3 [14] to implement the SAT procedure and the B substitution calculus to implement wp, wcp, grd and sp. Our implementation

Association of Under-Approximation Techniques

63

of the algorithm of Fig 6 optimises it in two ways. Firstly, we improve its practical complexity by not examining all the triplets (q, e, q ). Secondly, we better connect the concrete transitions to produce longer executions. But the universal concretisation described in Sec. 5.2 has not been implemented yet. Improvement of the Practical Complexity. The worst case theoretical complexity of our algorithm is |A|2 × |Ev| when the concretisation is not taken into account. Taking into account the concretisation, i.e. the loop in lines 16-19 of Fig. 6, the complexity is |A|3 × |Ev|2 because the maximal size of Γ ({q}) is 2 × |A| × |Ev| when there is one transition for each event towards each abstract state because any abstract transition is concretised by one concrete transition. The number of triplets (q, e, q ) to explore can be reduced by computing on the one hand for each abstract state the events that can be ﬁred on it, independently of the target states, and on the other hand, for each abstract state, the events able to reach it, independently of the source states. Let F ireable be a partial function that maps each abstract state of A to the set of events ﬁreable on it. Let Reaching be a partial function that maps each abstract state of A to the set of events able to reach it. The computation of F ireable and Reaching is in |A| × |Ev| and allows for the following optimisations. – In the third loop, on line 11, instead of enumerating all the events in Ev, it is possible to enumerate only the events in F ireable(q) ∩ Reaching(q ): they are both ﬁreable on q and able to reach q . – In the second loop (line 10), it is suﬃcient to enumerate the set of reachable states (domain of Reaching) instead of the set of all the abstract states in A. Improvement of the Connectivity Between the Concrete Transitions. To get a “good” connectivity of the concrete transitions, the algorithm tries to concretise e an abstract transition (q → q ) by considering all the existing pairs of concrete states of q and q . If it fails, the algorithm asks the solver to ﬁnd for each concrete state of q a new concrete state of q such that the concrete transition exists. If it fails again, a new pair of concrete states is built by the solver. Indeed, many concrete transitions between two diﬀerent concrete states are transformed into reﬂexive may transitions on an abstract state. In practice, to go from an abstract state to another, it is often necessary to apply these reﬂexive transitions to connect inside an abstract state the concrete target state of the incoming abstract transition to the concrete source state of the outgoing abstract transition. The concretisation could be improved at no supplementary computing cost by two implementation choices and three user parameters. 1. Concretise in priority reﬂexive transitions so as to increase the number of concrete states of q. 2. Enumerate in priority the concrete states reachable in q (line 17) and unreached in q (line 18). This requires to mark as reachable or not the concrete states during their computation and to arrange Γ in order on-the-ﬂy. 3. Enumerate the events in an order that causes the reﬂexive transition to chain. For example in the Electrical System, F ail has to be called before Rep since a battery can be repaired only if it is broken.

64

P.-C. Bu´e, J. Julliand, and P.-A. Masson

4. Indicate the number of repetitions of some of the events, that need to be consecutively repeated to eﬀectively make the system progress. 5. Provide a partial concretisation of the numerical variables of some targeted concrete states to guide the global concretisation. The last three parameters rely on the hypothesis that the tester has a suﬃciently good knowledge of the system and its model to be able to provide them. Knowing the “common sense” sequencing order of the events and their number of repetitions seems natural. But a partial concretisation is surely a more diﬃcult thing to know about. 6.2

Experimental Results

In the tables, the symbols “”, “Pot.”, “Trans.”, “Inst.” and “Abs.” respectively stand for number of, Potential, Transitions, Instantiated and Abstract. Table 1 compares two processes of test generation from abstraction. We compare test sequences that come from the abstraction algorithm deﬁned in Sec. 5 (Process 1) with previous results obtained with a process (Process 2) in which we generate tests as executions of Δ and then instantiate them a posteriori. The comparison is based on the coverage of the abstract transitions and the abstract states. For Process 1, we give also the coverage of concrete transitions and concrete states of Δc . In Process 2, we have used the theorem prover of Atelier B [15] instead of the SMT-solver Z3. Since the tools used to generate the abstractions are not the same in Process 1 and Process 2, directly comparing their execution times does not make sense. Except for Demoney, these times are small with both processes. We notice however that the computation of the Demoney abstraction took 1381 s. with Process 1 vs. and 12917 s. with Process 2. We have observed that 9522 s. was spent at instantiating the tests a posteriori with Process 2. This indicates that instantiating the tests on-the-ﬂy is more eﬃcient. Process 1 gives better coverage ratios for the abstract states (up to 40%) and transitions (up to 45%), excepted for the small model of Fig. 4, where Process 2 also covers the test t5 that should be found by the second step of Process 1 described in Sec. 5.2. Table 1. Eﬃciency of the Test Generation Methods Process 1 : Concretisation Process Process 2 : Process with instantiation Pot. Abstract Abstract Concrete Concrete Abstract Abstract Tests Inst. Tests Tests states Tests States Trans. States Trans. States Trans. Steps / Abs. Tests Steps Coverage Coverage Coverage Coverage Coverage Coverage Small 8/8 8/10 11/15 8/11 8/8 10/10 ∞ 7 13 4/4 (100%) 12 Model (20) (100%) (80%) (73%) (73%) (100%) (100%) SysAlim 4/4 11/11 8/8 15/15 4/4 11/11 36 1 28 4/4 (100%) 21 (100) (100%) (100%) (100%) (100%) (100%) (100%) QuiDonc 5/5 22/22 13/13 40/40 3/5 14/22 13 2 90 2/5 (40%) 16 (170) (100%) (100%) (100%) (100%) (60%) (64%) Robot 4/4 21/31 44/59 50/72 3/4 17/31 384 29 224 2/4 (50%) 25 (100) (100%) (68%) (74%) (69%) (75%) (55%) Partition 4/7 5/23 9/41 5/28 3/7 5/23 ∞ 5 9 2/14 (14%) 6 (80) (57%) (22%) (22%) (18%) (43%) (22%) DeMoney 10/14 237/368 31/161 499/1669 9/14 197/368 ∞ 34 591 17/18 (95%) 170 (330) (71%) (64%) (19%) (30%) (64%) (54%) Model (loc)

Association of Under-Approximation Techniques

65

Table 2. Impact of the Use of Parameters for Test Generation Parameter

Events Enumeration Order Events Multiple Call

Parameter Value

Model

Loading order Inverse loading order Valid credit order DeMoney Inverse valid credit order Robot

Robot DeMoney

Part Arrival: 3 Default parameter Put Data: 4 Default parameter

Partial Concrete A partial concretisation DeMoney Valuation Default parameter

Tests

Coverage Coverage Tests Abstract Abstract Concrete Concrete Steps States Trans. States Trans.

2 1 34 20

16 2 591 422

50% 25% 71% 50%

26% 6% 64% 45%

21% 7% 19% 15%

26% 6% 30% 25%

29 7 34 1

224 35 591 22

100% 75% 71% 7%

68% 35% 64% 6%

74% 34% 19% 2%

69% 42% 30% 6%

34 1

591 22

71% 7%

64% 6%

19% 2%

30% 6%

The experimental results of Table 2 are about the impact of the parameters of Sec. 6.1 (items 3, 4 and 5), in terms of number of tests, number of test steps and coverage of the states and the transitions of both the abstraction and its concretisation. We have observed each parameter variation with the two other parameters ﬁxed at the best value that we found on the examples. The variations are as follows. There are two events sequencing: intuitive or non-intuitive. Each event is repeated either once (default parameter) or the number of times indicated in Table 2. Only Demoney has numerical variables: they are either not at all (default parameter) or partially concretised. The results show that modifying the sequencing of the events for the Robot and Demoney examples changes the coverage ratios (up to 4 times more) and the number of test and test steps (up to 8 times more) very signiﬁcantly. Repeating some of the events considerably improves the quality of the results, in terms of abstraction coverage (up to 10 times more) and of number of tests and test steps (up to 26 times more). The partial concretisation of the targeted states also substantially improves the coverage of the abstraction (up to 10 times more) and the number of tests and test steps (up to 26 times more). We have also noticed with Demoney that playing with the three parameters at once could be necessary to better improve the results.

7

Related Works

Some algorithms that compute over-approximations based on predicate abstraction can be found e.g. in [2,16]. Predicate abstraction is also used by Ball in [6,17] and P˘as˘areanu et al. in [7] to compute program abstractions that are underapproximations for generating tests. Ball computes an under-approximation from the reachable sequences of must- transitions followed by at most one may transition and a sequence of must+ transitions. P˘as˘areanu et al. also compute an under-approximation, but by concretisation of the may transitions. We combine these two methods in this paper by concretising may transitions and the sequences of must transitions. We do not reﬁne the set of abstraction predicates as in [7,18] to search for an exact abstraction. But we present a parametrised algorithm and many optimisations by heuristics to improve the concretisation. We have deﬁned in [5] a method to extract a set of predicates P from a test purpose and a behavioural model for generating tests in an MBT approach.

66

P.-C. Bu´e, J. Julliand, and P.-A. Masson

In [6], P is made of all the atomic predicates that appear in the control structure of a C program. In [18], the set of predicates is iteratively reﬁned in order to compute a bisimulation of the initial model when it exists. SYNERGY [19] and DASH [20] also combine under-approximation and over-approximation computations to check safety properties on programs. As we aim at proposing an eﬃcient MBT method, our algorithm always terminates because it does not reﬁne the over-approximation. Moreover, it generates on-the-ﬂy the under-approximations by using the witnesses of satisﬁability proofs to build the over-approximation. Other works are about generating tests from abstraction. The tools Agatha [21], DART [22], CUTE [23], EXE [24] and PEX [25] also compute abstractions from models or from programs, but by means of symbolic execution [4]. This data abstraction approach computes an execution graph. Its set of abstract states is possibly inﬁnite whereas it is ﬁnite with the predicate abstraction method. The methods of [26] implemented in STG [27] use abstractions deﬁned by the user and modelled by IOSTS (Input Output Symbolic Transition System). They use test purposes synchronised with abstractions, both deﬁned as IOSTS. Then, the synchronised product allows for generating tests after an optimisation step, which consists of pruning the unreachable states by abstract interpretation. Our approach is very similar in that we also use test purposes and abstractions, as well as synchronisation. But there are three diﬀerences. First, our abstractions are computed from a set of predicates deﬁned from the test purposes, whereas STG uses user-deﬁned abstractions. Second, an optimisation is performed by the abstraction computation by using the invariant properties (that do not exist in an IOSTS) speciﬁed in the B models used in our experimentations. It allows, for the weakest precondition computation, to minimise the symbolic state space and the feasible transitions. Third, we use SMT solvers, that combine constraint solving and theories for proof, instead of pure constraint solving to instantiate the symbolic tests. Similarly to the concolic execution in [23], we combine concrete execution with predicate abstraction. But concolic tools use symbolic execution instead of predicate abstraction. Furthermore, we use a dual combination by performing predicate abstraction and concretising incrementally the abstract transitions, whereas concolic execution performs a concrete execution and at the same time collects the symbolic path constraints. Moreover, hybrid concolic execution [28] combines random generation of input values.

8

Conclusion and Further Works

We have presented a method of model-based test generation and an algorithm that combines two under-approximation computations (deﬁned in [6,7]) by predicate abstraction. Our contributions are this combination and the design of three adequate parameters to capture the know-how of the tester and the deﬁnition of pertinent heuristics for improving the accuracy of the under-approximations. Our experimental results indicate that generating the tests from this underapproximation with an on-the-ﬂy instantiation is more eﬃcient and gives better coverage ratios than an over-approximation based process with afterwards instantiation. Our experimental results also measure the impact of each parameter

Association of Under-Approximation Techniques

67

on the test coverage ratios. This work shows that using under-approximations is decisive for the scalability of the method. This paper also shows that the eﬃciency of the method depends on capturing of the tester’s expertise. We propose three parameters to the tester: the order in which the operations are considered, the number of their consecutive repetition and a subset of the concrete states to target. The practical usability of these parameters by a validation engineer has to be assessed by experimentations on larger case studies. Also, we intend as future works to deﬁne better parameters that would depend on the application domains. We also have to improve the implementation of the concretisation of the underapproximation L deﬁned in Sec. 5.2. Additionally, for the SMT-solvers to be able to deal with our behavioural models, we have by now to restrict our usage of some of the B set operators. To overcome this limitation, it will be necessary either to deﬁne new theories for the SMT-solvers, or to automatically rewrite all B predicates as ﬁrst-order logic formulas.

References 1. Utting, M., Legeard, B.: Practical Model-Based Testing. Morgan Kaufmann, San Francisco (2006) 2. Graf, S., Sa¨ıdi, H.: Construction of abstract state graphs with PVS. In: Grumberg, O. (ed.) CAV 1997. LNCS, vol. 1254, pp. 72–83. Springer, Heidelberg (1997) 3. Cousot, P., Cousot, R.: Abstract interpretation frameworks. J. Log. Comput. 2(4), 511–547 (1992) 4. P˘ as˘ areanu, C.S., Visser, W.: A survey of new trends in symbolic execution for software testing and analysis. STTT 11(4), 339–353 (2009) 5. Bouquet, F., Bu´e, P.C., Julliand, J., Masson, P.A.: Test generation based on abstraction and test purposes to complement structural tests. In: A-MOST 2010, 6th Int. Workshop on Advances in Model Based Testing, Paris, France (April 2010) 6. Ball, T.: A theory of predicate-complete test coverage and generation. In: de Boer, F.S., Bonsangue, M.M., Graf, S., de Roever, W.-P. (eds.) FMCO 2004. LNCS, vol. 3657, pp. 1–22. Springer, Heidelberg (2005) 7. P˘ as˘ areanu, C.S., Pel´ anek, R., Visser, W.: Predicate abstraction with underapproximation reﬁnement. LMCS 3(1) (2007) 8. Julliand, J., Masson, P.A., Tissot, R.: Generating security tests in addition to functional tests. In: AST 2008, pp. 41–44. ACM Press, NY (2008) 9. Abrial, J.R.: Modeling in Event-b: System and Software Design. Cambridge Univ. Press, Cambridge (2010) 10. Dijkstra, E.: Guarded commands, nondeterminacy, and formal derivation of programs. C. ACM 18 (1975) 11. Dijkstra, E.: A Discipline of Programming. Prentice-Hall, Englewood Cliﬀs (1976) 12. Hoare, C.: An axiomatic basis for computer programming. CACM 12, 576–580 (1969) 13. Bert, D., Cave, F.: Construction of ﬁnite labelled transistion systems from b abstract systems. In: Grieskamp, W., Santen, T., Stoddart, B. (eds.) IFM 2000. LNCS, vol. 1945, pp. 235–254. Springer, Heidelberg (2000) 14. de Moura, L., Bjørner, N.: An eﬃcient smt solver. In: Ramakrishnan, C.R., Rehof, J. (eds.) TACAS 2008. LNCS, vol. 4963, pp. 337–340. Springer, Heidelberg (2008)

68

P.-C. Bu´e, J. Julliand, and P.-A. Masson

15. Atelier B: Case tool for developing software proven without default, http://www.atelierb.eu 16. Ball, T., Majumdar, R., Millstein, T.D., Rajamani, S.K.: Automatic predicate abstraction of c programs. In: PLDI, pp. 203–213 (2001) 17. Ball, T., Kupferman, O., Yorsh, G.: Abstraction for falsiﬁcation. In: Etessami, K., Rajamani, S.K. (eds.) CAV 2005. LNCS, vol. 3576, pp. 67–81. Springer, Heidelberg (2005) 18. Namjoshi, K.S., Kurshan, R.P.: Syntactic program transformations for automatic abstraction. In: Emerson, E.A., Sistla, A.P. (eds.) CAV 2000. LNCS, vol. 1855, pp. 435–449. Springer, Heidelberg (2000) 19. Gulavani, B.S., Henzinger, T.A., Kannan, Y., Nori, A.V., Rajamani, S.K.: Synergy: a new algorithm for property checking. In: SIGSOFT FSE, pp. 117–127 (2006) 20. Beckman, N.E., Nori, A.V., Rajamani, S.K., Simmons, R.J., Tetali, S., Thakur, A.V.: Proofs from tests. IEEE Trans. Software Eng. 36(4), 495–508 (2010) 21. Rapin, N., Gaston, C., Lapitre, A., Gallois, J.P.: Behavioral unfolding of formal speciﬁcations based on communicating extended automata. In: ATVA 2003 (2003) 22. Godefroid, P., Klarlund, N., Sen, K.: DART: directed automated random testing. In: PLDI, pp. 213–223 (2005) 23. Sen, K., Marinov, D., Agha, G.: CUTE: a concolic unit testing engine for C. In: ESEC/SIGSOFT FSE, pp. 263–272 (2005) 24. Cadar, C., Ganesh, V., Pawlowski, P.M., Dill, D.L., Engler, D.R.: EXE: automatically generating inputs of death. In: ACM Conference on Computer and Communications Security, pp. 322–335 (2006) 25. PEX: Automated exploratory testing for .NET, http://research.microsoft.com/en-us/projects/pex 26. Calam´e, J., Ioustinova, N., van de Pol, J.: Automatic model-based generation of parameterized test cases using data abstraction. ENTCS, vol. 191, pp. 25–48 (2007) 27. Jeannet, B., J´eron, T., Rusu, V., Zinovieva, E.: Symbolic test selection based on approximate analysis. In: Halbwachs, N., Zuck, L.D. (eds.) TACAS 2005. LNCS, vol. 3440, pp. 349–364. Springer, Heidelberg (2005) 28. Majumdar, R., Sen, K.: Hybrid concolic testing. In: ICSE 2007, pp. 416–426 (2007)

Security Mutants for Property-Based Testing Matthias B¨ uchler, Johan Oudinet, and Alexander Pretschner Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany {buechler,oudinet,pretschner}@kit.edu

Abstract. The last decade has witnessed impressive progress in terms of dedicated approaches to formally analyzing security properties of models. However, related approaches to generating tests generally rely on purely syntactic test selection criteria. In this paper, we consider models of protocols and describe an approach to generate tests from security properties. Security-speciﬁc mutation operators are deﬁned and used to introduce potential security-speciﬁc leaks into the model. Then, if the leak is conﬁrmed by a model analyzer, a test case for the security property is generated. We present examples for security-relevant mutants at the model level and show how they correspond to security-ﬂawed implementations, thus providing evidence that model-level mutants are indeed useful for doing security testing. Keywords: Property-based testing, Security protocols, Mutation, Test generation.

1

Introduction

One of the main challenges in testing systems is the selection of test cases. Ideally, test cases unveil potential and actual failures, are cheap to generate and run, make it easy to track down underlying faults, and make it possible to assess their quality. Accordingly, there is a plethora of test selection criteria. Among them random criteria as well as criteria that reﬂect coverage of the implementation’s or model’s structure are particularly appealing because they lend themselves to the automated generation of test cases. Both kinds of criteria usually do not take into account properties of the system (see Section 3 for a discussion of exceptions). This situation is not entirely satisfying when it comes to characteristics of a system that pertain to a system’s performance or security. Many security properties of protocols can today be formally veriﬁed at the model level. Once a model has been veriﬁed w.r.t. conﬁdentiality, integrity, or availability properties, the natural next step is to derive tests from this correct model for some implementation. In terms of structural coverage criteria, one approach to doing so is to pick a set of properties Φ that reﬂects the coverage criterion (e.g., it is possible to reach state σ, for all states σ), and use a model checker to generate a test case per property (that drives the model into state σ). Unfortunately, this simple approach does not work for non-structural universal properties like the aforementioned security properties. Let ψ be such a property. M. Gogolla and B. Wolﬀ (Eds.): TAP 2011, LNCS 6706, pp. 69–77, 2011. c Springer-Verlag Berlin Heidelberg 2011

70

M. B¨ uhler, J. Oudinet, and A. Pretschner

It is then reasonable to assume M |= ψ—the model should satisfy the property. Using a model checker to verify M |= ¬ψ then yields all traces of the model as counter examples (test cases); no discrimination between more or less “interesting” traces takes place. In this paper, we tackle the problem of automatically generating test cases that are “interesting” when it comes to testing a speciﬁc security property according to potential implementation vulnerabilities. Indeed, we provide evidence that our generated test cases are correlated to targeted ﬂaws at the source code level, which is not addressed by related work that focus on security properties only and leave out the implementation level. Hence, rather than simply negating the property, we follow a diﬀerent approach (that others have followed as well, see Section 3). Intuitively, we modify the model in a way that the security property is violated in a speciﬁc way, related to a common mistake at the implementation level. More speciﬁcally, we describe the following process. We start from a model, M, of the system in which a speciﬁed property, ϕ, holds: M |= ϕ. Then, we apply mutations to the model in order to get a new model M in which the property doesn’t hold anymore: M |= ϕ. In this way, we obtain one or several traces of M that are counter-examples for ϕ, also called attack traces. An attack trace describes a sequence of messages that an intruder should be able to play entirely in order to prove the presence of a real ﬂaw at the implementation level. If an attack trace can be reproduced on the implementation, then we have found a real security ﬂaw. Moreover, this ﬂaw is related to a speciﬁc security property and a speciﬁc vulnerability at the implementation level, according to the process used to generate these test cases. In sum, the problem that we tackle is the derivation of test cases for speciﬁc universal security properties. Our solution is based on a set of mutation operators at the model level that are related to possible errors at the source code level. We consider our main contribution in providing premature evidence for the correlation between High Level Protocol Speciﬁcation Language (HLPSL) mutants and implementation-level vulnerabilities. We complement the procedure for testing non-structural universal properties [5,23], in a sense that if a ﬂaw is found, we can boil it down to a speciﬁc vulnerability related to the mutant involved. In the remainder of this paper, we will focus on mutation operators for models of protocols described in HLPSL [19]. In Section 2, we present two mutation operators and explain what kind of real implementation-level security problems they reﬂect. In Section 3, we review related work and conclude in Section 4.

2

HLPSL Mutation Operators

The Avispa tool [3] is used for automated validation of Internet security protocols and applications. It provides a dedicated language for protocol speciﬁcations called High Level Protocol Speciﬁcation Language (HLPSL) which is used to specify protocols and their security properties. A model checker checks if the protocol model satisﬁes the speciﬁed properties. If security properties are violated the model checker outputs an attack trace that violates the property. To

Security Mutants for Property-Based Testing

71

ﬁnd attack traces for implementations on the basis of a correct model, we use mutation-based testing. The selection criterion for such mutants is the positive correlation with common errors at the implementation level. Before describing the mutant operators, we give a short description of the HLPSL input language and the output of the Avispa tool. HLPSL Input Language. A HLPSL example speciﬁcation of the NSPK-ﬁx protocol1 can be found in Appendix A. A security protocol speciﬁed in HLPSL is based on the A-B notation and is role based. It usually consists of several sections: (1) basic role speciﬁcations for protocol participants (line 1-37), (2) a session section where multiple participant roles are instantiated (line 40-46), (3) an environment section where sessions are combined and the intruder knowledge is deﬁned (line 50-63), and (4) a goal section for security properties (line 67-71). In general a basic role deﬁnes transitions (line 10-17,30-36) that usually describe the receipt of a message and the sending of a reply. To do so it speciﬁes preconditions and actions that have to be executed when the preconditions are true. Avispa Tool Output. The Avispa tool deﬁnes an output format OF which is described as follows: The result of the tool is either conclusive or inconclusive where in the ﬁrst case either saf e or unsaf e is reported. If the tool ﬁnds an attack (the tool reports unsaf e) parts of the output consists of the name of the violated security property and an attack trace which shows the exchange of messages between participants in order to violate the property. For example, in the NSPK protocol the attack trace looks like as follows: ATTACK TRACE: i -> a: start a -> i: {Na(1).a}_ki i -> b: {Na(1).a}_kb b -> i: {Nb(2).Na(1)}_ka i -> a: {Nb(2).Na(1)}_ka a -> i: {Nb(2)}_ki

It is a sequence of messages, such that X sends m to Y is represented by: X -> Y: m The agent i is a special agent — the intruder — that behaves as deﬁned by the attacker model. The Avispa tool only deﬁnes the Dolev Yao intruder model, speciﬁed as a channel parameter (e.g. line 1 or 21 in Listing 1.1). 2.1

Agent Identifier Mutant

A HLPSL speciﬁcation speciﬁes a protocol and security properties like secrecy or authentication. Security ﬂaws are often based on man-in-the-middle attacks where a message from a session can be used in another session and therefore 1

This example comes from http://avispa-project.org/library/NSPK-fix.html

72

M. B¨ uhler, J. Oudinet, and A. Pretschner

violate speciﬁc security properties of the protocol. By modifying agent IDs in the HLPSL speciﬁcation, one may produce test cases that are related to manin-the-middle attacks in particular and the violation of the secrecy property in general. The HLPSL language supports send and receive statements for messages. To generate such test cases, we consider variables in receive statements like the following: RCV ({A.B }K ). It speciﬁes that received messages are encrypted with key K and consist of two concatenated values A and B. If a message is received, the primed variable B is bound to the corresponding value in the message. The non-primed variable A already has been bound to a value and operates as a selector value. E.g. using the above receive statement at the left side of a transition, the receive statement is only triggered if the incoming message matches with the value of variable A. Therefore primed and non-primed variables can allow or prevent man-in-the-middle attacks at the HLPSL level. Violation of Authentication and Secrecy. We apply the previous idea to the ﬁrst message of the Needham-Schroeder Public-Key (NSPK) protocol. Correctly, Bob only accepts {N a .A}Kb from Alice in a session if A corresponds to the intended sender of that session — due to the unprimed selector variable A in RCV ({N a .A}Kb ) and the sharing of the agent identiﬁer A when a session between Alice and Bob is deﬁned. To invalidate this check, the mutant primes variable A so that A in RCV ({N a .A }Kb ) now adapts its value to the value in the received message. This means, an intruder can successfully use message {N a.A}Kb, originated from a session Alice↔Bob, in a session intruder↔Bob. Checking the modiﬁed model the Avispa tool returns the following attack trace. An expression X → Y : m means that X sends m to Y . The partial trace shows that in the third step, the intruder i can forward the message to Bob. Because Bob doesn’t check the agent ID, it accepts the message in the session intruder↔Bob and sends back an answer encrypted with key ki , instead of ka . ATTACK TRACE: i -> a: start a -> i: {Na(1).a}_kb i -> b: {Na(1).a}_kb b -> i: {Na(1).Nb(2).b}_ki

2.2

Nonce Mutant

In the NSPK protocol both Alice and Bob are creating nonces which are sent to the other partner. Nonces are generally used to guarantee the freshness of messages and that an agent is present in a session. Both are security properties and the latter can be speciﬁed with the keyword authentication on together with (w)request and witness in a HLPSL speciﬁcation. Therefore modifying nonces may aﬀect the authentication property. The following mutant modiﬁes the HLPSL model in such a way that the generated attack trace exactly addresses the part of the source code that deals with the authentication on security property.

Security Mutants for Property-Based Testing

73

Violation of Authentication. Alice uses the nonce N a in the ﬁrst message to verify that Bob participates in the current session. She expects that the ﬁrst reply from Bob contains N a too. The mutant in this section replaces RCV (N a.N b .B) by RCV (N a .N b .B) that means that Alice does not check the received nonce N a anymore. The Avispa tool indeed conﬁrms that the mutant aﬀects the authentication property of the protocol: ATTACK TRACE: i -> a: start a -> i: {Na(1).a}_kb i -> a: {x238.x239.b}_ka a -> i: {x239}_kb

This attack trace for the modiﬁed protocol shows that an intruder was able to ﬁnish the protocol with Alice although Alice thinks she is talking to Bob. 2.3

Mutant-Implementation Error-Correspondence

To show that the described mutants reﬂect common mistakes at source code level we consider the C implementation given in [11]. The security of the NSPK protocol is based on diﬀerent checks. E.g. when Bob receives the ﬁrst message from Alice, he has to check if variable A in message {N a.A}Kb is set correctly. Similarly when Bob replies to Alice, she needs to check if N a in message {N a.N b.B}Ka is correct. It’s crucial that these checks are performed at the implementation level as well. In our implementation Bob performs the above check if the sender ID is correct with an if statement given as follows: if(strncmp(alice_msg.id,ALICE_ID,sizeof(ALICE_ID))) {...} else {...}

Similarly, Alice executes the following if statement to check N a: if(strncmp(alice_msg.nonceA,nonceA,sizeof(nonceA))) {...} else {...}

Therefore applying the above mutants at the HLPSL level corresponds to the case where the software developer has either (1) forgotten to implement the if statements, (2) has misplaced the if statements and therefore has made them ineﬀective, or (3) has messed up the conditions in the if statements.

3

Related Work

Our work is closely related to mutation testing [7,12]. The goal of mutation testing is to assess the eﬀectiveness of test suites (or test selection criteria) which is done by introducing small syntactic changes into a program and then see if the test suite detects these changes. Two hypotheses underlie the generalizability of results obtained with mutation testing: the competent programmer hypothesis (programmers develop programs that are close to the correct version) and the coupling eﬀect (small syntactic faults correlate with major faults). These assumptions have been subject to quite some validation research that is summarized in [12, Section II.A]. The idea of manually injecting real world faults into systems

74

M. B¨ uhler, J. Oudinet, and A. Pretschner

to the end of assessing the quality of quality assurance is common practice [20] for instance in the telecommunication industries; and so is fault-based testing where systems are checked for the occurrence of speciﬁc errors, e.g., stuck-at-1 errors in circuit designs. Security-related mutation testing has also been successfully performed at the level of access control policies [13,15,16,17]; we diﬀer in that we consider protocol models rather than access control policies as a basis. Test assessment criteria can also be understood as test selection criteria [24]. In our context, this means that mutation testing can also be used to generate rather than assess tests, an idea that was, among others, successfully applied for speciﬁcation-based testing from AutoFocus or SMV models in the security context [23,1] and in the context of HLPSL [5]. These three papers are, in terms of related work, closest to our approach. Our work diﬀers from them in that we provide evidence for the correlation between model-level mutants and implementationlevel faults, instead of just a discussion about why implementations can go wrong. Model checkers have been used for test case generation since at least 1998 [2], in a variety of contexts that has been surveyed elsewhere [9]. Most of this work concentrates on generating tests that satisfy structural criteria such as state coverage, transition coverage, MC/DC coverage, etc., on the model. In this context, coverage criteria have also been successfully applied to temporal logic properties [21,8,22]. Our work diﬀers in that we rely on a domain-speciﬁc fault model. Formal models and model checking for security properties have been used by others [6,10,18,4,14]. They rely either on dedicated security rules that are used for test case generation, or are concerned with functional testing of security-critical subsystems. In contrast, our work is based on a dedicated fault model. In practice, security testing is today usually performed with penetration testing tools (e.g., http://sectools.org/). These tools are diﬀerent in that they do not rely on protocol models to perform the tests, and do not use model checking technology for the generation of tests.

4

Conclusion

We describe mutants at the HLPSL level that are closely related to implementationlevel security faults. Actually, we showed on a C implementation which lines of code are addressed by the described mutants. One drawback of generating mutants at a higher-level language like HLPSL is that the number of generated mutants is rather small (see for example the experimental results in [5]). We are currently working on producing mutants at a lower-level intermediate language (i.e., Intermediate Format (IF)) in order to introduce more subtle changes that cannot be described at a higher level. To be consistent with practices in security testing, we must facilitate the use of our test cases in a penetration testing tool. We are currently working on translating our test cases as exploits in Metasploit (http://metasploit.com). The main diﬃculties for this translation are: intercepting exchanged messages, ﬁltering those that are relevant to the attack trace, building and sending messages from the intruder that are accepted by the honest agents.

Security Mutants for Property-Based Testing

75

Acknowledgments This work was partially supported by the FP7-ICT-2009-5 Project no. 257876, “Secure Provision and Consumption in the Internet of Services” (http://www.spacios.eu).

References 1. Ammann, P., Ding, W., Xu, D.: Using a model checker to test safety properties. In: International Conference on Engineering of Complex Computer Systems, pp. 212–221. IEEE, Los Alamitos (2001) 2. Ammann, P.E., Black, P.E., Majurski, W.: Using model checking to generate tests from speciﬁcations. In: ICFEM, pp. 46–54. IEEE, Los Alamitos (1998) 3. Armando, A., Basin, D., Boichut, Y., Chevalier, Y., Compagna, L., Cuellar, J., Drielsma, P.H., He´ am, P.C., Kouchnarenko, O., Mantovani, J., M¨ odersheim, S., von Oheimb, D., Rusinowitch, M., Santiago, J., Turuani, M., Vigan` o, L., Vigneron, L.: The AVISPA tool for the automated validation of internet security protocols and applications. In: Etessami, K., Rajamani, S.K. (eds.) CAV 2005. LNCS, vol. 3576, pp. 281–285. Springer, Heidelberg (2005) 4. Brucker, A.D., Br¨ ugger, L., Wolﬀ, B.: Model-based ﬁrewall conformance testing. In: TestCom/FATES, pp. 103–118 (2008) 5. Dadeau, F., H´eam, P.-C., Kheddam, R.: Mutation-based test generation from security protocols in HLPSL. In: ICST, p. 9. IEEE, Los Alamitos (2011) 6. Darmaillacq, V., Fernandez, J.-C., Groz, R., Mounier, L., Richier, J.-L.: Test gen¨ Duale, A.Y., Fecko, M.A. (eds.) eration for network security rules. In: Uyar, M.U., TestCom 2006. LNCS, vol. 3964, pp. 341–356. Springer, Heidelberg (2006) 7. DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Program Mutation: A New Approach to Program Testing. In: Infotech State of the Art Report, Software Testing, pp. 107–126 (1979) 8. Fraser, G., Wotawa, F.: Complementary criteria for testing temporal logic properties. In: TAP, pp. 58–73 (2009) 9. Fraser, G., Wotawa, F., Ammann, P.: Testing with model checkers: a survey. Softw. Test., Verif. Reliab. 19(3), 215–261 (2009) 10. Garc´ıa-Alfaro, J., Cuppens, F., Cuppens-Boulahia, N.: Towards ﬁltering and alerting rule rewriting on single-component policies. In: G´ orski, J. (ed.) SAFECOMP 2006. LNCS, vol. 4166, pp. 182–194. Springer, Heidelberg (2006) 11. Goubault-Larrecq, J., Parrennes, F.: Cryptographic protocol analysis on real C code. In: Cousot, R. (ed.) VMCAI 2005. LNCS, vol. 3385, pp. 363–379. Springer, Heidelberg (2005) 12. Jia, Y., Harman, M.: An Analysis and Survey of the Development of Mutation Testing (2011); To appear in IEEE TSE 13. LeTraon, Y., Mouelhi, T., Baudry, B.: Testing security policies: Going beyond functional testing. In: ISSRE, pp. 93–102 (2007) 14. Mallouli, W., Morales, G., Cavalli, A.: Testing security policies for web applications. In: Proc. 1st Workshop on Security Testing (2008) 15. Martin, F., Xie, T.: A fault model and mutation testing of access control policies. In: Proc. 16th Intl. Conf. on the World Wide Web, pp. 667–676 (2007) 16. Mouelhi, T., LeTraon, Y., Baudry, B.: Mutation analysis for security tests qualiﬁcation. In: Proc. 3rd Workshop on Mutation Analysis, pp. 233–242 (2007)

76

M. B¨ uhler, J. Oudinet, and A. Pretschner

17. Pretschner, A., Mouelhi, T., LeTraon, Y.: Model-based tests for access control policies. In: ICST, pp. 338–347 (2008) 18. Senn, D., Basin, D.A., Caronni, G.: Firewall conformance testing. In: Khendek, F., Dssouli, R. (eds.) TestCom 2005. LNCS, vol. 3502, pp. 226–241. Springer, Heidelberg (2005) 19. The AVISPA Team. AVISPA User Manual, 1.1 edn. (2006), http://www.avispa-project.org/package/user-manual.pdf 20. Voas, J., McGraw, G.: Software Fault Injection: Innoculating Programs Against Errors. John Wiley & Sons, Chichester (1997) 21. Weiglhofer, M., Fraser, G., Wotawa, F.: Using coverage to automate and improve test purpose based testing. In: INFSOF, vol. 51(11), pp. 1601–1617 (2009) 22. Whalen, M.W., Rajan, A., Per Erik Heimdahl, M., Miller, S.P.: Coverage metrics for requirements-based testing. In: ISSTA, pp. 25–36 (2006) 23. Wimmel, G., J¨ urjens, J.: Speciﬁcation-based test generation for security-critical systems using mutations. In: George, C.W., Miao, H. (eds.) ICFEM 2002. LNCS, vol. 2495, pp. 471–482. Springer, Heidelberg (2002) 24. Zhu, H., Hall, P.A.V., May, J.H.R.: Software unit test coverage and adequacy. ACM Comput. Surv. 29, 366–427 (1997)

A

1 2 3 4

HLPSL Model for the Corrected Version of Needham-Schroeder Public-Key Authentication Protocol r o l e a l i c e (A, B : agent , Ka , Kb : p u b l i c k e y , SND, RCV: c h a n n e l ( dy ) ) p l a y e d b y A d e f= l o c a l S t a t e : nat , Na , Nb : t e x t

5 6

i n i t S t a t e := 0

7 8

transition

9 10

0.

S t a t e = 0 /\ S t a t e ’ : = 2 /\ /\ /\

2.

S t a t e = 2 /\ RCV( {Na . Nb ’ . B} Ka ) =|> S t a t e ’ : = 4 /\ SND( {Nb’ } Kb ) /\ r e q u e s t (A, B , a l i c e b o b n b , Nb ’ )

11 12 13

RCV( s t a r t ) =|> Na ’ := new ( ) /\ SND( {Na ’ . A} Kb ) s e c r e t (Na ’ , na , { A, B}) w i t n e s s (A, B , b o b a l i c e n a , Na ’ )

14 15 16 17 18

end r o l e

19 20 21 22 23 24

r o l e bob (A, B : agent , Ka , Kb : p u b l i c k e y , SND, RCV: c h a n n e l ( dy ) ) p l a y e d b y B d e f= l o c a l S t a t e : nat , Na , Nb : t e x t

25 26

i n i t S t a t e := 1

27 28 29

transition

Security Mutants for Property-Based Testing 30

RCV( { Na ’ . A} Kb ) =|> Nb ’ := new ( ) /\ SND( {Na ’ . Nb ’ . B} Ka ) s e c r e t (Nb’ , nb , { A, B} ) w i t n e s s (B, A, a l i c e b o b n b , Nb ’ )

1.

S t a t e = 1 /\ S t a t e ’ : = 3 /\ /\ /\

3.

S t a t e = 3 /\ RCV( {Nb} Kb ) =|> S t a t e ’ : = 5 /\ r e q u e s t (B, A, b o b a l i c e n a , Na)

31 32 33 34 35 36 37

end r o l e

38 39 40 41

r o l e s e s s i o n (A, B : agent , Ka , Kb: p u b l i c k e y ) d e f= l o c a l SA , RA, SB , RB: c h a n n e l ( dy )

42 43 44 45 46

composition a l i c e (A, B, Ka , Kb, SA ,RA) /\ bob (A, B, Ka , Kb, SB ,RB) end r o l e

47 48 49 50 51 52 53 54 55

r o l e environment ( ) d e f= const a , b : agent , ka , kb , k i : public key , na , nb , alice bob nb , bob alice na : protocol id

56 57

i n t r u d e r k n o w l e d g e = {a , b , ka , kb , k i , i n v ( k i ) }

58 59 60 61 62 63

composition s e s s i o n ( a , b , ka , kb ) /\ s e s s i o n ( a , i , ka , k i ) /\ s e s s i o n ( i , b , k i , kb ) end r o l e

64 65 66 67 68 69 70 71

goal s e c r e c y o f na , nb authentication on alice bob nb authentication on bob alice na end g o a l

72 73 74

environment ( ) Listing 1.1. HLPSL model of NSPK-ﬁx

77

The SANTE Tool: Value Analysis, Program Slicing and Test Generation for C Program Debugging Omar Chebaro1,2, Nikolai Kosmatov1 , Alain Giorgetti2,3, and Jacques Julliand2 1

CEA, LIST, Software Safety Laboratory, PC 94, 91191 Gif-sur-Yvette France [email protected] 2 LIFC, University of Franche-Comté, 25030 Besançon Cedex France [email protected] 3 INRIA Nancy - Grand Est, CASSIS project, 54600 Villers-lès-Nancy France

Abstract. This short paper presents a prototype tool called SANTE (Static ANalysis and TEsting) implementing an original method combining value analysis, program slicing and structural test generation for verification of C programs. First, value analysis is called to generate alarms when it can not guarantee the absence of errors. Then the program is reduced by program slicing. Alarm-guided test generation is then used to analyze the simplified program(s) in order to confirm or reject alarms. Keywords: static analysis, program slicing, all-paths test generation, run-time errors, alarm-guided test generation.

1 Introduction Software validation remains a crucial part in software development process. Software testing accounts for about 50% of the total cost of software development. Automated software validation is aimed at reducing this cost. The increasing demand on software validation has motivated much research and two major techniques have improved in recent years, static and dynamic analysis. They arose from different communities and evolved along parallel but separate tracks. Traditionally, they were viewed as separate domains. However, static and dynamic analysis have complementary strengths and weaknesses and combining them is of significant interest for program debugging. This paper presents our tool called SANTE (Static ANalysis and TEsting) combining value analysis, program slicing and structural testing for the verification of C programs. In [1], we described an earlier version of the SANTE method combining value analysis and structural testing for C program debugging. The method used value analysis to report alarms of possible run-time errors (some of which may be false alarms), and test generation to confirm or to reject them. The method produced for each alarm a diagnostic that can be safe for a false alarm, bug for an effective bug confirmed by some input state, or unknown if it does not know whether this alarm is an effective error or not. Experimental results showed that the combined method is better than each technique used independently. It is more precise than a static analyzer and more efficient in terms of time and number of detected bugs than a concolic structural testing tool used alone, or even guided by the exhaustive list of alarms for all potentially threatening statements. M. Gogolla and B. Wolff (Eds.): TAP 2011, LNCS 6706, pp. 78–83, 2011. c Springer-Verlag Berlin Heidelberg 2011

The SANTE Tool: Value Analysis, Program Slicing and Test Generation

79

Context

Program p

Value Analysis

p, Alarms

Slicing or

pall

or

p1

...

pn

Dynamic Analysis

Diagnostic Fig. 1. SANTE Debugging Process

In the new version of the SANTE tool presented in this paper, we add program slicing to our combination in order to simplify and reduce the source code before test generation. Program slicing [2] is a technique for decomposing programs based on data and control-flow information with respect to a given slicing criterion (e.g. one or several program statements). We present two different usages of program slicing. First program slicing is performed one time with respect to the set of all alarms. Second program slicing is performed n times, once with respect to each alarm (n is the number of alarms). Our implementation uses F RAMA -C, a framework for static analysis of C programs, and PATH C RAWLER, a structural test generation tool. F RAMA -C [3] is being developed in collaboration between CEA LIST and the ProVal project of INRIA Saclay. Its software architecture is plug-in-oriented and allows fine-grained collaboration of analysis techniques. Static analyzers are implemented as plug-ins and can collaborate with one another to examine a C program. F RAMA -C is distributed as open source with various plug-ins (i.e. value analysis, dependency analysis, program slicing, weakest precondition, ...). Developed at CEA LIST, PATH C RAWLER [4,5,6] is a test generation tool for C functions respecting the all-paths criterion, which requires to cover all feasible program paths, or the k-path criterion, which restricts the generation to the paths with at most k consecutive iterations of each loop. The paper is organized as follows. Section 2 describes our tool and its implementation. Section 3 provides some perspectives and concludes.

2 The SANTE Tool on a Running Example This section demonstrates how, given a C program p and its execution context, the SANTE tool applies value analysis, program slicing and dynamic analysis for its debug-

80

O. Chebaro et al.

0 i n t e u r o c h e c k ( char ∗ s t r ) { 1 u n s i g n e d char sum ; 2 char c [ 9 ] [ 3 ] = { "ZQ" , "YP" , "XO" , 3 "WN" , "VM" , "UL" , "TK" , " SJ " , " RI " } ; 4 u n s i g n e d char checksum [ 1 2 ] ; 5 int i = 0 , len = 0;

17 18 19

checksum [ i ] = s t r [ i ] −48} sum = 0 ; f o r ( i = 1 ; i < l e n ; i ++)

20 21 22 23 24 25 26 27 28

sum+= checksum [ i ] ; w h i l e ( sum > 9 ) sum = ( ( sum / 1 0 ) + ( sum % 1 0 ) ) ; f o r ( i = 0 ; i < 9 ; i ++) i f ( checksum [ 0 ] = = c [ i ] [ 0 ] ) break ; i f ( sum ! = i ) r e t u r n 5 ; / / wrong checksum r e t u r n 0 ; } / /OK

0 i n t e u r o c h e c k ( char ∗ s t r ) { 1 u n s i g n e d char sum ; 2 char c [ 9 ] [ 3 ] = { "ZQ" , "YP" , "XO" , 3 "WN" , "VM" , "UL" , "TK" , " SJ " , " RI " } ; 4 u n s i g n e d char checksum [ 1 2 ] ; 5 int i = 0 , len = 0; 60 / /@ a s s e r t ( \ v a l i d ( s t r + 0 ) ) ; 6 i f ( s t r [ 0 ] > = 9 7 && s t r [ 0 ] < = 1 2 2 ) 7 s t r [0] −=32; 8 i f ( s t r [ 0 ] < ’ I ’ | | s t r [ 0 ] > ’Z ’ ) 9 return 2; 10 i f ( s t r l e n ( s t r ) != 12) 11 return 3; 12 len = st r l e n ( s t r ) ; 130 / /@ a s s e r t ( \ v a l i d ( s t r + i ) ) ; 13 checksum [ i ] = s t r [ i ] ; 14 f o r ( i = 1 ; i < l e n ; i ++){ 150 / /@ a s s e r t ( \ v a l i d ( s t r + i ) ) ; 15 i f ( s t r [ i ] <48 | | s t r [ i ] > 5 7 ) 16 return 4; 170 / /@ a s s e r t ( \ v a l i d ( checksum+ i ) ) ; 17 checksum [ i ] = s t r [ i ] −48} 18 sum = 0 ; 19 f o r ( i = 1 ; i < l e n ; i ++) / /@ a s s e r t ( \ v a l i d ( checksum+ i ) ) ; 200 20 sum+= checksum [ i ] ; 21 w h i l e ( sum > 9 ) 22 sum = ( ( sum / 1 0 ) + ( sum % 1 0 ) ) ; 23 f o r ( i = 0 ; i < 9 ; i ++) 24 i f ( checksum [ 0 ] = = c [ i ] [ 0 ] ) 25 break ; 26 i f ( sum ! = i ) 27 return 5; 28 r e t u r n 0 ; }

a) Function eurocheck

b) Function eurocheck with alarms

6 7 8 9 10 11 12

i f ( s t r [ 0 ] > = 9 7 && s t r [ 0 ] < = 1 2 2 ) s t r [0] −=32; / / c a p i t a l i z e i f ( s t r [ 0 ] < ’ I ’ | | s t r [ 0 ] > ’Z ’ ) r e t u r n 2 ; / / i n v a l i d char i f ( s t r l e n ( s t r ) != 12) r e t u r n 3 ; / / wrong l e n g t h len = s t r l en ( s t r ) ;

13 14

checksum [ i ] = s t r [ i ] ; f o r ( i = 1 ; i < l e n ; i ++){

15 16

i f ( s t r [ i ] <48 | | s t r [ i ] > 5 7 ) return 4 ; / / not a d i g i t

Fig. 2. Running example before and after value analysis

ging (see Fig. 1). This process is fully-automatic. The execution context, or precondition, defines value ranges for acceptable inputs of p and relationships between them. We illustrate each step of the method on the example of Fig. 2a. Given a string str representing the serial number of a euro banknote, this function determines whether the serial number is valid or not. Such a number normally contains one letter followed by several digits. We define the precondition for the function eurocheck as: str is NULL or a zero-terminated string. 2.1 Step 1: Value Analysis SANTE starts by applying value analysis (see Fig. 1) to eliminate as many potential threats as possible. When the risk of a run-time error cannot be excluded by the (overapproximated) sets of possible values of variables for some statement, value analysis reports a threat for this statement, that is also called an alarm. In other words, value

The SANTE Tool: Value Analysis, Program Slicing and Test Generation

81

0 v o i d e u r o c h e c k ( char ∗ s t r ) { 5 int i , len ; 60 6 7 8 9 10 11 12 14 150 15 16

0 v o i d e u r o c h e c k ( char ∗ s t r ) { 5 int i , len ; i f ( 0 >= l e n g t h ( s t r ) ) 61 error (); 62 63 / /@ a s s e r t \ v a l i d ( s t r + 0 ) ; else i f ( s t r [ 0 ] > = 9 7 && s t r [ 0 ] < = 1 2 2 ) 6 i f ( s t r [ 0 ] > = 9 7 && s t r [ 0 ] < = 1 2 2 ) s t r [0] −=32; 7 s t r [0] −=32; i f ( s t r [ 0 ] < ’ I ’ | | s t r [ 0 ] > ’Z ’ ) 8 i f ( s t r [ 0 ] < ’ I ’ | | s t r [ 0 ] > ’Z ’ ) return ; 9 return ; i f ( s t r l e n ( s t r ) != 12) 10 i f ( s t r l e n ( s t r ) != 12) return ; 11 return ; len = s t r l en ( s t r ) ; 12 len = s t r l e n ( s t r ) ; f o r ( i = 1 ; i < l e n ; i ++){ 14 f o r ( i = 1 ; i < l e n ; i ++){ i f ( i >= l e n g t h ( s t r ) ) 151 error (); 152 153 / /@ a s s e r t \ v a l i d ( s t r + i ) ; else i f ( s t r [ i ] <48 | | s t r [ i ] > 5 7 ) 15 i f ( s t r [ i ] <48 | | s t r [ i ] > 5 7 ) return ;}} 16 return ;}} a) The slice without error branches

b) The slice with error branches

Fig. 3. The slice with respect to line 15, before and after adding error branches

analysis proves the absence of errors for some potential threats and computes a set of alarms reporting the remaining threats. Our implementation uses the value analysis plug-in of F RAMA -C. For the program of Fig. 2a, value analysis returns five alarms for (the statements at) lines 6, 13, 15, 17 and 20. At line 6, we are reading the first character str[0]. This alarm is a bug since str can be empty. At line 13, value analysis reports that str[i] may be an out-of-bound access. This alarm is a false alarm because if the length of str is not equal to 12, the program will return wrong length at line 11 and the execution will never reach line 13. At line 15, the alarm reported is also a false alarm. Here value analysis does not unroll all iterations, it is configured to unroll the first two iterations and then it approximates. Same for the alarms at line 17 and line 20. Technically, the FRAMA -C value analyzer marks each alarm by an annotation printed just before it using the assert keyword (see Fig. 2b). For instance, at line 15, the overapproximated set of values calculated for i contains values greater than the length of str and the annotation / /@ a s s e r t ( \ v a l i d ( s t r + i ) ) ;

is added just before line 15 (see line 150 in Fig. 2b) to report that the array access str[i] may be out-of-bound. The reader will find more information on the ACSL annotation language used by F RAMA -C in [3]. 2.2 Step 2: Program Slicing The second step automatically simplifies the program by program slicing. In this tool demostration, we show three different ways to simplify the program p.

82

O. Chebaro et al.

1. The program p is directly analyzed by dynamic analysis without any simplification by program slicing. The earlier version of the SANTE method presented in [1] was limited to this unique option. Its main drawback is that dynamic analysis on a large non-simplified program may take much time or not terminate, leaving a lot of alarms unknown. 2. Program slicing is applied once and the slicing criterion is the set of all alarms of p (formally speaking, the set of threatening statements containing these alarms). We obtain one simplified program pall containing the same threats as the original program p. Then dynamic analysis is applied to pall (see Fig. 1). Dynamic analysis is executed only once and runs faster than for p since it is applied to its simplified version pall . For the running example, pall contains only 18 lines. 3. Let n be the number of alarms in p. Program slicing is performed n times, once with respect to each alarm ai , producing simplified programs pi (1 ≤ i ≤ n). Then dynamic analysis is called n times to analyze the n resulting programs pi (see Fig. 1). The advantage of this option is producing for each alarm ai the minimal slice pi preserving the threatening statement of ai . For the running example, we obtain five slices whose sizes vary from 3 to 16 lines. Fig. 3a shows an example of a slice for the threat in line 15. 2.3 Step 3: Dynamic Analysis Program slicing is followed by the last step, dynamic analysis, applied to all simplified programs. Dynamic analysis tries to activate each potential threat, i.e. to cover execution paths in which the associated alarms are triggered. This step produces for each alarm a diagnostic: safe, bug or unknown. In our implementation, we use the PATH C RAWLER tool [5] whose method is similar to the concolic testing [7], also called dynamic symbolic execution. Given the C source code of the function under test, the generator explores program paths in a depth-first search using symbolic and concrete execution. Technically, in order to force test generation to activate potential errors on each feasible program path in p, we add special error branches into the source code of p in the following way. For each alarm, its threatening statement, say threatStatement ;

is automatically replaced by the following branching statement: if ( errorCondition ) error (); else threatStatement ;

where the condition determines if the error reported by the alarm occurs. For the running example, the result is shown in Fig. 3b. Test generation is then executed for the C program with error branches denoted p . We call this technique alarm-guided test generation. If the errror condition is verified in p , a run-time error can occur in p, so the function error() reports the error and stops the execution of the current test case. If there is no risk of run-time error, the execution continues normally and p behaves

The SANTE Tool: Value Analysis, Program Slicing and Test Generation

83

exactly as p. The transformation of p into p adds new branches for error and error-free states so that PATH C RAWLER algorithm will automatically try to cover error states. For an alarm a, PATH C RAWLER may confirm it as a bug when it finds an input state and an error path leading to the bug. PATH C RAWLER may also prove that the alarm is safe when all-paths test generation on p terminates without activating the corresponding threat. When all-paths test generation on p does not terminate, or when incomplete test coverage criterion was used (e.g. k-path), no alarm is classified safe. Finally, all alarms that are not classified as bug or safe remain unknown. For the running example, without slicing (cf Sec. 2.2.1), test generation on the original program with error branches takes around 25 seconds. When the program is sliced with respect to all alarms (cf Sec. 2.2.2), test generation finishes in around 7 seconds. For each of the five programs sliced with respect to one alarm (cf Sec. 2.2.3), test generation takes between 1 and 6 seconds. The complete time needed for the five slices is around 13 seconds. The value analysis and slicing steps are much faster than test generation (much less than 1 sec. for this example). In all cases, test generation concludes that among the five alarms, there is one bug and four false alarms.

3 Conclusion In this demonstration paper, we presented the SANTE tool combining value analysis, program slicing and structural testing for C program debugging. The method was illustrated on a running example. Future work includes proving the soundness of the method, studying other ways to combine different analyses and transformations, and experiments on more examples. Acknowledgments. The authors thank the members of the PathCrawler and Frama-C teams for providing the tools and support. Special thanks to Loïc Correnson and Bruno Marre for their helpful advice and fruitful suggestions.

References 1. Chebaro, O., Kosmatov, N., Giorgetti, A., Julliand, J.: Combining static analysis and test generation for C program debugging. In: Fraser, G., Gargantini, A. (eds.) TAP 2010. LNCS, vol. 6143, pp. 94–100. Springer, Heidelberg (2010) 2. Weiser, M.: Program slicing. In: ICSE 1981, pp. 439–449 (1981) 3. Frama-C: Framework for static analysis of C programs (2007-2011), http://www.frama-c.com/ 4. Williams, N., Marre, B., Mouy, P., Roger, M.: PathCrawler: Automatic generation of path tests by combining static and dynamic analysis. In: Dal Cin, M., Kaâniche, M., Pataricza, A. (eds.) EDCC 2005. LNCS, vol. 3463, pp. 281–292. Springer, Heidelberg (2005) 5. Botella, B., Delahaye, M., Hong-Tuan-Ha, S., Kosmatov, N., Mouy, P., Roger, M., Williams, N.: Automating structural testing of C programs: Experience with PathCrawler. In: AST, pp. 70–78 (2009) 6. Kosmatov, N.: Online version of the PathCrawler test generation tool (2010-2011), http://pathcrawler-online.com/ 7. Sen, K., Marinov, D., Agha, G.: CUTE: a concolic unit testing engine for C. In: ESEC/FSE 2005, pp. 263–272 (2005)

Abstraction Based Automated Test Generation from Formal Tabular Requirements Specifications Renzo Degiovanni1 , Pablo Ponzio1 , Nazareno Aguirre1 , and Marcelo Frias2 1

2

Departamento de Computaci´ on, FCEFQyN, Universidad Nacional de R´ıo Cuarto and CONICET, R´ıo Cuarto, C´ ordoba, Argentina {rdegiovanni,pponzio,naguirre}@dc.exa.unrc.edu.ar Departamento de Ingenier´ıa Inform´ atica, Instituto Tecnol´ ogico Buenos Aires and CONICET, Buenos Aires, Argentina [email protected]

Abstract. We propose an automated approach for generating tests from formal tabular requirements speciﬁcations, such as SCR speciﬁcations. The technique is based on counterexample guided abstraction reﬁnement and the use of SMT solving. Moreover, in order to eﬀectively perform automated test generation, we take advantage of particular characteristics of tabular requirements descriptions to aid the abstraction and abstraction reﬁnement processes. The exploited characteristics are, most notably, the organisation of the requirements speciﬁcation in modes, which is used to build an initial abstraction, and the execution model of tabular speciﬁcations, which is directed by changes observed in environment variables and is exploited for modularising the transition relation associated with tables, simplifying the calculation of abstractions. These characteristics enable us to eﬀectively perform automated test generation achieving good levels of coverage for diﬀerent criteria relevant to this context. We compare our approach with a standard abstraction analysis, showing the beneﬁts that exploiting the mentioned characteristics of tables provide. We also compare the approach with model checking based generation, using several model checking tools. Our experiments show that the presented approach is able to generate test cases from models whose complexity, with respect to the sizes of variables and data domains, cannot be coped with well by the model checkers we used.

1

Introduction

It is generally accepted that the quality of requirements speciﬁcations has a great impact in the whole development process, since crucial activities such as validation against user expectations, system veriﬁcation against requirements, and the coherence of the requirements (and therefore also the system to be developed), depend on these speciﬁcations. Requirements speciﬁcations are mostly expressed in natural language, in textual form. Various approaches deal with this informal M. Gogolla and B. Wolﬀ (Eds.): TAP 2011, LNCS 6706, pp. 84–101, 2011. Springer-Verlag Berlin Heidelberg 2011

Abstraction Based Automated Test Generation

85

textual representation, and how its quality can be improved and assessed. However, formal requirements speciﬁcations may provide useful features, diﬃcult to achieve if informal notations are used. More precisely, formally speciﬁed requirements are better suited for analysis and its automation, which can be exploited for (semi-)automatically ﬁnding problems in the speciﬁcation itself (inconsistencies, imprecisions, etc.), and contrasting the speciﬁcation against the system (i.e., veriﬁcation), or user expectations (i.e., validation). Tabular notations, originally used to document requirements by D. Parnas and others [14], have proved to be a useful means for concisely and formally describing expressions characterising complex requirements. Indeed, tables have been successfully incorporated into various formalisms for requirements speciﬁcation, most notably those reported in [17,12]. Tables are used for formally describing various relations involved in requirements, such as the expected relationship between the observed and controlled environment variables once the system is put in place, assumptions about the environment due to conditions external to the system, or the relations that the system must maintain with the environment. Tables impose a structure on speciﬁcations, which provides useful features: it helps in organising large and complex formulas into well distinguished smaller formulas that are easier to follow, and impose simple constraints for guaranteeing characteristics such as certain forms of completeness (no missing cases) and consistency (no contradicting requirements). In this paper, we are concerned with the automated analysis of formal tabular notations. More precisely, we propose the use of automated analysis techniques for generating tests based on requirements speciﬁcations. The approach we use for test generation is a variant of a lazy abstraction mechanism for automated analysis, and relies on scalability mechanisms that take advantage of the tabular structure of the speciﬁcations, i.e., exploiting characteristics inherent to the tabular speciﬁcations. This variant yields, in this context, performance enhancements, compared to standard lazy abstraction. It is automated, based on counterexample guided abstraction reﬁnement and the use of SMT solving. Tests from tabular speciﬁcations correspond to executions of the requirements speciﬁcation, i.e., sequences of events and states respecting the constraints prescribed by tables. These tests have an obvious role in the validation, i.e. contrasting with user expectations, and veriﬁcation, i.e., contrasting against system behaviour. Moreover, they can also help in identifying problems in the speciﬁcation (e.g., missing cases, contradictory statements, etc.). Especially for complex requirements, coming up with a test suite that “exercises” the speciﬁcation under a good variety of cases (i.e., complying with particular coverage criteria) can be extremely diﬃcult. Especially for validation and veriﬁcation, activities in which the engineer needs to compare the behaviours expected by users and the actual system behaviour against the formal speciﬁcation of requirements, it is useful that the tests (i.e., the execution traces over the speciﬁcation) maintain the level of detail of the original speciﬁcation. Usually, due to analysis reasons, the engineer needs to manually simplify the speciﬁcations, so that analysis tools can deal with these;

86

R. Degiovanni et al.

if test cases are generated from these simpliﬁed models, then they have to be “concretised” before contrasting them with the expected or actual system behaviours, in order to disregard spurious tests resulting from the abstraction. This obviously complicates the validation and veriﬁcation tasks. Our approach attempts to deal with speciﬁcations at their original level of abstraction, generating an abstraction level suitable for test generation, but producing (non spurious) test cases at the level of abstraction of the original speciﬁcation. For some case studies we are able to generate test cases which several model checking tools are unable to cope with, in particular due to the sizes of variables and data domains of the models. Our approach is based on the following few observations regarding formal tabular requirements speciﬁcations: – In tabular requirements speciﬁcations, a special set of variables, known as mode classes, are employed in order to organise the states of the system into modes; mode changes are described via corresponding (mode) transition tables. – The execution model of tabular speciﬁcations is directed by changes observed in environment variables, whose alterations are observed once at a time. – The deﬁnition of variables involved in a tabular speciﬁcation lead to a dependency relation which is inherently acyclic. Each symbol describes a diﬀerent speciﬁcation element, thus ruling out aliasing in speciﬁcations. – Tabular requirements speciﬁcations often involve numeric variables, whose ranges are often larger than what automated tools (e.g., model checkers) are able to handle. Contributions of this paper. The contribution of this paper is an approach for automatically generating tests based on formal tabular requirements speciﬁcations. The approach is automated, based on abstraction and counterexample guided abstraction reﬁnement. The underlying technology supporting the approach is SMT solving, but the actual beneﬁts come from the identiﬁcation of the above elements, inherent to this kind of speciﬁcations, and mechanisms to exploit them in order to contribute to the abstraction. The technique exploits the above described characteristics of tables in the following way: – Modes and mode transitions are employed as part of the initial abstraction of the speciﬁcation. This provides important analysis beneﬁts, as we will demonstrate later on, especially because both properties to be analysed and the structure of tables in the speciﬁcation typically strongly depend on modes. We also observe that states within a mode tend to coincide in the level of “preciseness” for analysis, which we exploit to deﬁne our variant of lazy abstraction. – The execution model of tabular speciﬁcations and the inherent acyclic dependence of syntactic elements in tabular requirements speciﬁcations are exploited for modularising the transition relation associated with tables according to monitored variables changes, and their dependencies. This allows us to speed up the calculation of abstractions, as well as reuse calculations.

Abstraction Based Automated Test Generation

87

– We deﬁne a technique for dealing with discrete numeric datatypes in abstracting tabular speciﬁcations, so that the degree of detail that is necessary to incorporate as part of the abstraction reﬁnement in relation to numerical variables is “localised” to parts of the abstract state space. As we show later on, this leads to better scalability when dealing with numerical variables in tabular speciﬁcations. The above ways of exploiting tabular speciﬁcations signiﬁcantly contributes to the abstraction process. We show the beneﬁts of our approach by assessing it, in comparison with standard abstraction, for various case studies. We also use these case studies to compare the technique with several model checking tools, used for test case generation from requirements speciﬁcations. The technique is able to generate test cases from models whose complexity, with respect to the sizes of variables and data domains, cannot be handled well by the model checkers we used. It is important to notice that our approach does not constitute a testing criterion. Our technique is a test generation mechanism, which, provided a test criterion, allows for the automated generation of test cases according to the test criterion. We refer the reader to [9] for a thorough description of testing strategies applied to tabular speciﬁcations. Related work. Tabular notations, in particular SCR, have associated tool support providing diﬀerent kinds of analysis, e.g., syntax checking, consistency analysis via theorem proving and model checking, and the veriﬁcation of properties of requirements [2,6,13]. With respect to testing, the simulator in the SCR toolset [13] allows the developer to load speciﬁc scenarios, which are in principle provided by the engineer, and check whether certain associated assertions are violated or not in the particular executions described by the scenarios. Gargantini and Heitmeyer [11] used a model checker for automatically obtaining tests (table executions) transiting through particular modes. Recently, Fraser and Gargantini [10] made a thorough comparison between various model checkers (symbolic, bounded, explicit state, etc.) in order to automatically generate test cases from tables, and analysed the achieved coverage and scalability issues. Our evaluation of the technique is based on case studies analysed by Fraser and Gargantini, and our comparison with model checkers uses Fraser and Gargantini’s employment of model checkers for test case generation from tables. Bultan and Heitmeyer [6] recently employed the ALV inﬁnite state model checker to analyse tables; we do not report results using ALV for test case generation because the tool is discontinued, and in our preliminary experiments with it other model checkers exhibited better performance for this task. To our knowledge, automated abstraction techniques have been applied to tabular requirements speciﬁcations infrequently, particularly for testing purposes. In [4] Bharadwaj and Heitmeyer applied abstraction to SCR speciﬁcations, for scalability purposes related to model checking. As opposed to our work, Bharadwaj and Heitmeyer’s approach is based on the removal of irrelevant variables (slicing), and the transformation of “internal” variables into input ones (i.e.,

88

R. Degiovanni et al.

monitored), with the aim of removing monitored variables; given a property to be veriﬁed, their abstraction is fixed, it does not admit reﬁnement. Many successful approaches to veriﬁcation and test generation have been proposed. Some of these are based on SMT solving, abstraction, combinations and variants. Most works target code analysis rather than requirements speciﬁcations. In particular, lazy abstraction with abstraction reﬁnement based on interpolation was used for automatically generating tests leading to the reachable locations of a program, with successful applications in device drivers and security critical programs [3]. Other related and successful approaches are reported in [16,7]. Our approach is based on that presented in [3], which employs lazy abstraction [15] for test generation, but targets requirements speciﬁcations. Requirements speciﬁcations are not “control intensive”, as the programming domains in which abstraction is successfully applied [8], thus constituting a novel interesting domain. The behavioural model corresponding to SCR speciﬁcations has a signiﬁcant degree of nondeterminism, leading to high levels of “interleavings”, which makes it diﬃcult to apply techniques that work well in control intensive environments. Other automated tools such as Pex [20] and JPF [21] successfully implement automated white box test case generation for .NET and Java programs, respectively. These target code, and are based on symbolic execution instead of predicate abstraction with automated reﬁnement.

2

Preliminaries

Tabular Requirements Specifications. Tables provide an unambiguous yet clear and concise way of writing formulas. This has been found to be particularly useful for describing complex requirements of software systems, most notably in the work of D. Parnas and collaborators [14], and in the SCR method [17,12]. Our presentation is based on SCR because it is a very good representative of formal requirements languages using tables, which has extensive tool support and many real speciﬁcations to assess analysis techniques. In the SCR methodology, requirements are speciﬁed following Parnas’ four-variable model [19]. In this context, tables are used, among other things, for describing REQ, the requirements as a relationship that the system should induce between monitored and controlled variables, environment variables that the system is able to observe and control, respectively. In order to describe this relationship, SCR uses events, conditions, mode classes and terms. Events occur when changes in the variables observed by the system take place, and conditions are logical expressions referring to these variables. Modes represent classes of states of the system (the values of particular variables called mode classes), and typically capture historical information of the system [4]; terms are functions on the variables of the speciﬁcation. For describing events, SCR provides a simple notation. The notation @T(c) WHEN d describes the event in which expression c becomes true, when d is true in the current state, i.e., it represents the expression c’ ∧ c ∧ d, where the primed expression refers to the next state. If d is true, the ‘WHEN’ section is not written.

Abstraction Based Automated Test Generation

89

Let us consider an example, taken from [4]. Suppose that one needs to specify a safety injection system, whose task is to control the level of water pressure of a nuclear plant’s cooling system. The system monitors the water pressure, and a couple of switches for blocking and resetting (represented by monitored variables mWaterPres, mBlock and mReset, respectively), and it controls a single boolean variable, indicating whether the safety injection system is on or oﬀ (controlled variable cSafetyInjection). Usually the behaviour of the system depends on a number of previous events or conditions (i.e., the history); these are characterised by the so called modes of the system (possible values of mode classes). In this case, a single mode class, mcPressure, whose corresponding modes are TooLow, Permitted and High, indicates whether the water pressure is considered to be too low (below a constant Low), in a permitted level or high (over a constant Permit), respectively. Basically, the system must start safety injection when the pressure becomes too low. The system can be “disengaged” via the switches, indicating that its actions are overridden. This is captured by a term tOverridden. Tables are used to deﬁne dependent symbols, i.e., mode classes, terms and controlled variables. Basically, tables are of two kinds: event tables, to deﬁne symbols whose changes are driven by the occurrences of events, and condition tables, which deﬁne symbols in terms of conditions over other symbols. Mode changes depend on events, and are described via special event tables, the mode transition tables. For the safety injection system, table in Figure 1(a) indicates how system modes change when certain events occur. For instance, when the system is in mode TooLow and the water pressure becomes higher than or equal to Low, the mode changes to Permitted (see ﬁrst row of the mode transition table). Term tOverridden is deﬁned via the event table in Figure 1(c). This table indicates exactly when the system actions are overridden (e.g., if the block switch is pressed while in any mode other than High, with the reset switch oﬀ). Finally, the controlled variable cSafetyInjection is deﬁned via a condition table, shown in Figure 1(b). Tables have associated well formedness conditions. For instance, the disjunction of all conditions in a row of a condition table must be True, to ensure completeness (no missing cases), and the conjunction of any pair of diﬀerent cells in the same row must be False (disjointness), to ensure no contradictions. A ﬁnite run is a sequence σ = s0 , s1 , . . . , sk of states such that s0 satisﬁes some speciﬁed initial condition, and every state si+1 is obtained from its predecessor si by modifying a single monitored variable mV , and propagating the modiﬁcations of all variables depending on mV , according to what is prescribed by the corresponding tables. Variables not depending on mV maintain their corresponding values in si . The change in a monitored variable that triggers the move from a state to another can be interpreted as an input event, while the resulting changes in controlled variables can be interpreted as the output. Thus, a run is a sequence of input events and corresponding outputs. For example, consider the state s = High, 150, on, off, True, off, corresponding to the values of mcPressure, mWaterPres, mBlock, mReset, tOverridden and cSafetyInjection, respectively; from state s, the input event mWaterPres’ = mWaterPres-1 takes the system to the state Permitted, 149, on, off, False, off, assuming that Permit = 150.

90

R. Degiovanni et al.

(a) Mode class mcPressure

(b) Controlled var. cSafetyInjection

(c) Term tOverridden Fig. 1. Tabular speciﬁcation of the Safety Injection System

Abstraction. A labelled transition system (LTS) S = (S, Σ, →) consists of a set S of states, a ﬁnite set Σ of labels, and a labelled transition relation →⊆ S × Σ × S. A region structure R = (R, ⊥, , , pre, post, [.]) for an LTS S is a structure consisting of a set R of regions, where each region r represents a set [r]1 of states of S; ⊥ represents the empty set of states, r r and r r are the union and intersection operators of [r] and [r ], respectively; pre(r, l) and post(r, l) are the weakest precondition and strongest postcondition operators with respect to labels (e.g., pre(r, l) returns the largest region such that, from all its states and traversing arcs labelled by l one arrives at states in r), respectively. We denote by r ⊆ r the fact that [r] ⊆ [r ], and by r ≡ r that [r] = [r ]. An abstraction structure A = (R, preA , postA , ) for an LTS S consists of a region structure R for S, abstract pre and post operators preA and postA , and a precision preorder , such that pre(r, l) ⊆ preA (r, l), post(r, l) ⊆ postA (r, l), and preA and postA are monotonic with respect to ( ∩ ⊆). A region r can be thought of as representing an abstract state, in some abstract state space, whose concretisation is [r], with the abstract operators preA and postA enabling abstract state propagations. The precision preorder indicates how close the abstract pre and post operators are to the concrete ones, for given regions. A particular way of abstracting a state space is via predicate abstraction. In this context, regions are characterised by sets of state properties called support predicates, and the concretisation is simply the set of states satisfying the corresponding predicates. More precisely, let S = (S, Σ, →) be an LTS, and P a set of predicates over S (i.e., for every p ∈ P , [p] ⊆ S). An abstraction structure AP (S) = (RP (S), preAP , postAP , P ) can be deﬁned, as follows: – RP (S) = (R, ⊥, , , pre, post, [.]), where the regions in R are pairs (ϕ, Γ ), with Γ ⊆ P being a ﬁnite set of (local) support predicates, and ϕ is a boolean formula over the predicates of Γ . The remaining elements of RP (S) are deﬁned as follows: ⊥= (f alse, ∅); (ϕ, Γ ) (ϕ , Γ ) = (ϕ ∨ ϕ , Γ ∪ Γ ); (ϕ, Γ ) pre (ϕ , Γ ) = (ϕ ∧ ϕ , Γ ∪ Γ ); pre((ϕ, Γ ), l) = (ϕpre is the l , Γls ) where ϕl 1

[.] : R → 2S is the function that maps each region to the set of states it represents.

Abstraction Based Automated Test Generation

91

weakest precondition of ϕ with respect to l and Γls is the least superset of Γ which contains all predicates in ϕpre l . Operator post is deﬁned in a similar way. The concretisation [(ϕ, Γ )] of (ϕ, Γ ) is deﬁned as [ϕ] (the set of states satisfying ϕ). – The abstract operator postAP is deﬁned as follows: let (ϕ, Γ ) be a region with ϕ = ϕ1 ∨ · · · ∨ ϕk in DNF (with support predicates as atomic formulas), and l be a label. postAP ((ϕ, Γ ), l) is the disjunction ψ1 ∨ ψ2 ∨ · · · ∨ ψk , where each ψi is a conjunction of all literals γ appearing positively or negatively in Γ , and such that ϕi ⇒ pre(γ, l). The operator preAP is deﬁned in a similar way. – is deﬁned in the following way: (ϕ, Γ ) (ϕ , Γ ) iﬀ Γ ⊇ Γ . This characterisation corresponds to lazy predicate abstraction, as introduced in [15]. The process is called lazy because the abstraction predicates are local to the regions, enabling the reﬁnement of the portions of the abstract computation tree that require it. The lazy predicate abstraction algorithm is a symbolic forward search algorithm with the capability of reﬁning the abstract regions as needed. Given an abstraction structure A, an initial region r0 and an error region ε, the procedure tries to verify that ε is not reachable in the abstract model by constructing an abstract reachability tree. Each node can be in one of two states: unmarked (not yet treated) or marked (already treated). The algorithm starts by creating an unmarked initial node with region r0 . It then iterates, by taking an unmarked node n, and doing one of the following steps, depending on the characteristics of n: (i) if the region of n intersects ε, then the abstract error region is reachable; (ii) if n’s region has already been covered, i.e., it is included in the join of marked nodes, then n is marked too; (iii) if n’s region does not intersect ε and has not been covered, then n is marked, the abstract successors of n’s region, with respect to postA and all labels, are calculated, and new unmarked nodes are created to hold these regions. The algorithm terminates, in principle, when the error region is reached, or no unmarked nodes remain. When the error region is reached, i.e., an abstract trace leading to an error is found, it remains to check whether the counterexample is spurious or not. This is done by checking the feasibility of the abstract trace. If it is feasible, a real counterexample has been found; if not, then a new suitable predicate can be added to one of the regions to reﬁne the abstraction, removing the spurious counterexample. The abstract computation subtree with the reﬁned region as a root has to be recalculated, and the process can continue. If the formalism to express the predicates is chosen appropriately, both the feasibility of the abstract trace and the calculation of a suitable predicate to reﬁne the abstraction can be done automatically. A way of computing suitable predicates to add to the regions in the above abstraction setting is via the use of interpolation [18]. Let us suppose that σ = ϕ1 , ϕ2 , . . . , ϕk is a sequence of formulas such that their conjunction is inconsistent (e.g., these formulas might correspond to an abstract counterexample trace). An interpolant for σ is a sequence of formulas ψ0 , ψ1 , . . . , ψk such that: (a) ψ0 = True and ψk = False, (b) ∀1 ≤ i ≤ k · ψi−1 ∧ ϕi ⇒ ψi , and (c) ∀1 ≤ i ≤ k · vars(ψi ) ⊆

92

R. Degiovanni et al.

vars(ϕ1 , . . . , ϕi ) ∩ vars(ϕi+1 , . . . , ϕk ). Given an abstract spurious trace σ, the interpolants provide suitable predicates to add to each of the regions in the trace so that σ is ruled out as an abstract behaviour.

3

Abstracting Requirements Specifications

We now deﬁne an abstraction for tabular requirements speciﬁcations. A main hypothesis in our construction is that modes structure the tabular speciﬁcations in a way that is relevant to many properties of interest on tables, particularly those having to do with test case generation. Thus, exploiting this structure for abstraction may provide signiﬁcant beneﬁts. Moreover, we base the construction on lazy abstraction motivated by the fact that, when the system is in diﬀerent modes it generally has diﬀerent capabilities, i.e., it responds to diﬀerent input events and in diﬀerent ways; so, quite possibly diﬀerent kinds and levels of precision might be necessary when the system is in diﬀerent modes. Since we want to make the process automated, we choose linear integer arithmetic (LIA), one of the formalisms behind the SMT solver MathSAT [5] which we use as a decision process for calculating and reﬁning abstractions, as our abstract domain. This language, which due to space restrictions we do not describe here, is expressive enough for our needs. For the interpolants, we use diﬀerence logic, a sublanguage of LIA for which MathSAT is able to calculate interpolants automatically. This is essential for making the abstraction reﬁnement process completely automated. Let us start describing the construction. First, an SCR requirements speciﬁcation Spec corresponds to an LTS SSpec = (S, E, →T ), where S is the set of all possible values for variables in the speciﬁcation, E is the set of events on monitored variables (i.e., input events), and →T contains a tuple (si , e, sj ) iﬀ sj is obtained from si as a consequence of input event e and the propagation of changes to all variables according to what is prescribed by the tables in Spec [4]. The set R of regions for our abstract domain is MSpec × LIA × ℘(LIA), with MSpec being the set of modes in Spec, and LIA and ℘(LIA) the domains of formulas and sets of formulas in LIA, respectively. The concretisation of a region is deﬁned as follows: [(m, ϕ, Γ )] = {s ∈ S|s |= ϕ and m is the mode of s}. The formula ϕ must be a conjunction of literals based on predicates from Γ as atomic formulas. As it can be noticed in the deﬁnition of R, the current mode of the system is an essential part of a region, meaning that the abstraction is precise with respect to “mode location”. If the speciﬁcation has more than one mode class, MSpec would correspond to the cartesian product of mode classes. The remaining elements of our region structure RSpec are deﬁned in a standard way, from SSpec . As we explained previously, in a lazy setting the support predicates are local to regions. In our case, and for the reasons explained at the beginning of this section, the support predicates will be made local to a mode. That is, all regions that share the mode will also share the support predicates. Given a mode m, we will denote by SP(m) the current set of support predicates for mode m. In order to complete the deﬁnition of our abstract domain ASpec , we have to deﬁne the abstract operators preASpec and postASpec , and the precision preorder

Abstraction Based Automated Test Generation

93

Spec . Since the approach is based on a forward construction, we concentrate on the deﬁnition of postASpec (preASpec is deﬁned similarly). Our abstract operator postASpec is deﬁned as follows: for every region (m, ϕ, Γ ) and input event l, postASpec ((m, ϕ, Γ ), l) is the set of all tuples (m , ψ, Γ ), such that: – m is reachable from m via l, – Γ = SP(m ), and – ψ is the conjunction γ1 ∧· · ·∧γk , where each γi is a literal appearing positively or negatively in Γ , and ϕ ⇒ pre(γi , l). Finally, the precision preorder Spec is deﬁned in the following way: for every pair of regions (m, ϕ, Γ ) and (m , ϕ , Γ ), (m, ϕ, Γ ) Spec (m , ϕ , Γ ) iﬀ Γ ⊇ Γ . The proofs of pre((m, ϕ, Γ ), l) ⊆ preASpec ((m, ϕ, Γ ), l), post((m, ϕ, Γ ), l) ⊆ postASpec ((m, ϕ, Γ ), l), and the monotonicity of preASpec and postASpec with respect to (Spec ∩ ⊆), are relatively straightforward. Let us provide a very simple example illustrating the above deﬁnition of postASpec . Consider the speciﬁcation of the safety injection system given previously. Suppose that, for each of the three modes, the following are the current support predicates: – TooLow: tOverridden, mWaterPres < Low – Permitted: tOverridden, mWaterPres < Low, mBlock = on – High: tOverridden, mWaterPres >= Low, mBlock = on, mReset = on Consider the abstract region r = Permitted, True, False, False. Its successor with respect to postASpec and the event mWaterPres’ = mWaterPres-1 is the disjunction of r itself, and TooLow, True, True. Similarly, the successor of r with respect to the event mWaterPres’ = mWaterPres+1 is the disjunction of r, High, False, True, False, False and High, False, True, False, True. This algorithm works essentially in the same way as the lazy abstraction algorithm explained previously, but using our abstract strongest postcondition operator. More precisely, given a property of interest P whose reachability needs to be analysed, the process starts with the initial abstract state, with only P as a support predicate; it then starts calculating the abstract reachability tree using postASpec , trying to reach an abstract state satisfying P and reﬁning the regions via interpolants resulting from spurious counterexamples. Whenever a new predicate is added to the support predicates of a region r, the same predicate is incorporated to all regions of nodes whose modes coincide with that of r; similarly, when a new region is incorporated, this region “inherits” the support predicates of its corresponding mode, as the deﬁnition of postASpec indicates. When no unmarked nodes remain and P has not been reached, then P is unreachable. If a concrete trace reaching a state satisfying P is constructed, a concrete run witnessing the reachability of P is obtained. The process can however “diverge” when the number of support predicates for some region becomes too large to be dealt with. As opposed to the original lazy abstraction approach, our abstraction is precise with respect to mode location, and that support predicates are local to modes instead of regions. Experimenting with tabular speciﬁcations, we found out that

94

R. Degiovanni et al.

as support predicates are discovered, these tend to be shared (i.e., would be “rediscovered” by the original algorithm) by abstract states with the same mode. Adding these predicates to all regions with the same mode enabled us to improve the construction of the abstraction. This fact is related to the “shape” of the LTS corresponding to a tabular speciﬁcation, in which every state admits changes in any of the monitored variables (i.e., the LTS is not “control intensive”, as opposed to some applications of abstraction for automated analysis [8]). Another important diﬀerence of our approach has to do with the mechanism for checking whether a state is covered or not. While the standard lazy abstraction algorithm keeps the value of the already computed symbolic abstract space (the join of all the already computed regions) and a decision procedure is employed to check if the current node is included there, our approach looks at all the other nodes within the same mode, to see if there is another one weaker than the current. This last check can be done without the use of a decision procedure, since all the nodes with the same mode share the same predicates. Modularising the Transition Relation. Given a (concrete) state of the system and an event, the transition relation leads to a process for computing the next state: when a monitored variable changes, the tables are looked up to update other variables whose values depend on the change of the monitored variable. The modularity of the transition relation deﬁned by the tabular structure, together with the variable dependencies and the absence of aliasing, can be straightforwardly used to “localise” changes, and save time in the calculation of the next state. We can do something similar for the case of computing the abstract successors of an abstract state. However, taking into account the structure of our abstraction states, we modularise the transition relation in a diﬀerent way. Basically, we take the global state transition relation T , as deﬁned by the tables, and for each monitored variable mV, we produce transition (sub)relations mk m1 mi TmV , . . . , TmV , where m1 , . . . , mk are the modes of the speciﬁcation. Each TmV corresponds to the transition subrelation associated with the behaviour of the tables when the monitored variable that changes is mV, and the current mode is mi . This modularisation is straightforward to obtain from the tables, and is similar to a kind of “cone of inﬂuence” approach. The acyclicity of the dependency between symbols of the speciﬁcation and the absence of aliasing enables us to perform this modularisation of the transition relation easily. As we show in the following section, this modularisation provides an important beneﬁt with respect to the calculation of abstract successors, since besides the simpliﬁcations in the calculations of concrete weakest preconditions (notice that these are necessary for computing abstract successors), it enables us to identify predicates whose current boolean value will not be altered, since none of its associated syntactic elements depends on the modiﬁed variable that changed. The next section provides an evaluation of the beneﬁts of this modularisation. Treating Numerical Domains. A characteristic that our abstraction process is sensitive to is the use of large numerical domains in the models. These numerical domains are rather common in SCR requirements speciﬁcations, so we need to propose a mechanism to deal with these. Basically, the problem has to do

Abstraction Based Automated Test Generation

95

with our lazy abstraction having support predicates local to modes. This means that whenever a location s needs to incorporate a support predicate (because of abstraction reﬁnement), that predicate is added to all locations with the same mode as s. Suppose that the behaviours of the speciﬁcation require the system to be within a mode m along long “chains” of successive numerical values for some variable before producing a change to a mode m . What would typically happen is that the abstraction reﬁnement process will need to introduce support predicates for distinguishing all these successive values of the variable (in our case, these support predicates will be introduced by the interpolation process), in order to remove spurious counterexamples taking the system from m to m . All these support predicates will be associated with a single mode (m), making our abstraction process impractical. In order to deal with this problem, we use a heuristics that consists of introducing intervals over these numerical variables. Basically, we learn from short executions of the abstraction process which numerical variables potentially have the issue we just described. We then take these variables, partition their domains in a number of intervals, and make support predicates to be local to mode, and corresponding domain interval. The actual degree used to partition intervals is calculated from the size of the numerical domain being partitioned, and the maximum number of support predicates for a location; notice that locations corresponded to modes, whereas now they will correspond to a mode, and intervals for the numerical variables that require partitioning. Consider, for instance, the safety injection system described previously. Suppose also that variable mWaterPres’s range is [0..5000], and it changes in steps of 1..10 (decrementing or incrementing a value in this range). The reﬁnement based on interpolation will then introduce support predicates mWaterPress ≤ 0, mWaterPress ≤ 10, mWaterPress ≤ 20, and so on. So, if we do not want to introduce more than 20 support predicates (per location) associated with mWaterPres, then we would introduce intervals 0 ≤ mWaterPres ≤ 199, 200 ≤ mWaterPres ≤ 399, 400 ≤ mWaterPres ≤ 599, and so on, to deﬁne the new, ﬁner locations. That is, these intervals are now part of the locations, meaning that abstraction reﬁnement will be local to a mode and interval of mWaterPres. Generating Test Cases from Tables using Abstraction. Test case generation via predicate abstraction is performed in a similar way as the generation using model checking, as reported, e.g., in [3]. First, one needs to build all test predicates corresponding to the coverage criterion of interest. Each of these test predicates characterises a particular equivalence class of test cases, in the corresponding test criterion; these are used as “trap properties”, one at a time, for the abstraction algorithm to produce concrete traces reaching the predicates. For each test predicate P , we run the algorithm and obtain three possible outputs: – the abstract state space is covered without reaching P , meaning that the corresponding test case equivalence class is infeasible, or – a concrete run reaching P is produced (i.e., a test case), or

96

R. Degiovanni et al.

– the process does not converge, and is stopped due to timeout, exhausted memory or any other resource related problem (e.g., excessive introduction of support predicates). In this case the process is inconclusive, and the corresponding test predicate is marked as an error, as these are called in [10]. We perform the test case generation using two stages. First, we use a so called cartesian abstraction, in which diﬀerent but related abstract states are combined into a single representation; this is equivalent to having three possible values for support predicates: true, false or *, the latter meaning “don’t care”. In this way the number of states that need to be handled for the test case generation, but the interpolation based reﬁnement might fail (since we are not dealing with actual abstract runs, but sets of abstract runs, when using these cartesian abstract states). Whenever the interpolation process produces, for a given mode, a support predicate that has already been introduced previously, we stop the test generation process for the current test predicate, and move to a second stage, in which the abstraction is “precise” (i.e., non cartesian), for this test predicate (the remaining test predicates will be treated ﬁrst with cartesian abstraction, and then precise abstraction if necessary). This latter process, the precise abstraction, has no issues regarding abstraction reﬁnement, but its scalability diminishes. Reusing Calls to the Decision Procedure. Given a particular test criterion, the corresponding test predicates are in general related in some way to each other. For instance, in table coverage predicates correspond to cells of tables; two different predicates originating in the same table might, for instance, share a row, meaning that they coincide in the mode, or share a column, meaning that the resulting value is the same, or if in diﬀerent columns, in the same table, the satisfaction of one of these implies the unsatisﬁability of the other, and vice versa. So, the predicates discovered while covering a particular test predicate P might also be relevant for the covering of other test predicates of the same test criterion. For this reason, we “cache” the calls to the decision procedure while computing abstract successors in the covering of a test predicate, so that these calculations can be reused in the covering of other test predicates within the same test criterion. This resulted to be fruitful, as we show in the next section.

4

Experimental Results

The contents of our experimental results section are two fold. First, we provide an evaluation of our approach in comparison with standard predicate abstraction, which enables us to assess the beneﬁts of our variant that exploits characteristics of tables. Second, we compare our approach with automated test case generation based on model checking. The experiments are based on some of the case studies presented in [10], where a thorough comparison between diﬀerent model checkers used for test generation is carried out. Therein, a number of test criteria, such as table coverage and modified condition decision coverage, relevant to tabular speciﬁcations are used. We generate test cases for the same criteria, and refer the reader to [10] for a description of these. The case studies in [10], some of

Abstraction Based Automated Test Generation

97

which we also use in this paper, are models available in the literature, regarding a cruise control system (ccs), a safety injection system (sis), an aircraft’s autopilot (autopilot), and a car overtaking protocol for coordinating smart vehicles (car3prop). Our models diﬀer, for part of the experimentation, from the ones used in [10]; for assessing the beneﬁts that exploiting characteristics of tables provides, compared with lazy abstraction, we use essentially the same models as in [10], which have been manually simpliﬁed (by using smaller constants and numeric ranges, mostly). However, in order to compare our approach with model checking test generation we use versions of the case studies which are larger than those in [10], i.e., where the sizes of constants and numeric ranges are larger (basically, at the level of abstraction of the textual descriptions of the corresponding systems). For instance, autopilot reduced deals with integer variables in the range [0..500], while the original autopilot, which we use for comparison with model checkers, has these same variables over the range [0..10000]. As we mentioned previously, we are interested in this because, especially for validation and veriﬁcation, it is important to generate test cases at the same level of abstraction expected by the user, and/or used in the implementation. All the experiments were run on an 2.6GHz Intel Core 2 Duo with 3GB of RAM (2.5GB maximum memory set for the analysis tools), running GNU/Linux 2.6. The following table compares, for three case studies (sis, car3prop and autopilot), standard predicate abstraction (PA, i.e., support predicates are globally shared by regions, modes are not part of the initial abstraction, no modularisation of the transition relation with respect to monitored variables/modes), with PA plus support predicates local to modes and these considered in the initial abstraction (PA + m.), PA plus modularised transition relation with respect to monitored variables/modes (PA + mt.), and ﬁnally our approach (PA + m. + mt.). The data corresponds to the total number of test predicates for a test criterion, the number of runs of each technique, and the corresponding numbers of covered (i.e., those for which a test case was generated), infeasible (those that the corresponding technique identiﬁed as unrealisable), and uncovered test predicates (i.e., errors, those in which the corresponding technique is inconclusive). For the covered test predicates, we also indicate between parentheses the number of traces, since a single trace can cover several test predicates. When any individual run of any of the processes executed for over an hour, it was stopped and the corresponding test predicate marked as uncovered. Notice that we marked the models autopilot and sis as “reduced”, meaning that they have been manually simpliﬁed with respect to the original description, by using small constants and numerical ranges. These models are the same as those used in [10] (car3prop is not reduced because it has no numerical constants or ranges). CS

car3prop (498 TPs.) runs c/i/u time PA 110 402(14)/90/6 33620 PA + m. 114 401(17)/96/1 8858 PA + mt. 118 402(22)/74/22 69197 PA + m. + mt. 119 402(29)/96/0 1795

sis reduced (91 TPs.) runs c/i/u time 19 80(4)/11/0 8496 21 80(10)/11/0 5319 20 80(9)/11/0 13032 23 80(12)/11/0 4724

autopilot reduced (409 TPs.) runs c/i/u time 21 395(7)/10/4 23497 81 347(19)/10/52 113401 51 389(31)/10/10 22201 54 399(44)/10/0 3951

98

R. Degiovanni et al.

In our approach, support predicates are local to modes instead of regions. We argued that, due to the structure of tables, regions with the same mode tend to share the support predicates. In order to validate this hypothesis, we have randomly chosen a few test predicates, and gathered the number of savings in predicate discovery associated with the “support predicates local to regions” approach. Basically, we measure for standard lazy abstraction and our approach, the number of nodes visited and introduced support predicates, and for standard lazy abstraction also the support predicate with largest number of “rediscoveries” for diﬀerent regions with the same mode. These results are summarised in the following table. CS car3prop car3prop sis reduced sis reduced autopilot reduced autopilot reduced

Lazy Lazy + m. + mt. nodes predicates most repeated nodes predicates 41 48 916 113 1037 965

94 75 158 25 126 126

30 33 50 8 21 21

76 75 936 113 1192 1120

11 8 65 17 32 32

Models involving numeric variables over large ranges, such as sis, are those in which “support predicates local to regions” provides more beneﬁts. We now compare our technique, referred to in the tables as “Lazy abs. +”, with model checkers used for test generation. Our experiments are based on those presented in [10], so we compare our technique with Spin, NuSMV, Cadence SMV and SAL. The four case studies used for the assessment are the ones mentioned above, using larger more realistic constants and numeric ranges, compared to the models used in [10]. We have run these tools with a variety of settings, and we report the best result obtained for each tool, in the table at the end of this section. When a tool is not mentioned for one of the case studies, that is because it performed signiﬁcantly worse than other model checkers. As opposed to the previous experiments, and because we have increased the sizes of the models, we set the timeout for covering single test predicates to 3 hours, and the total analysis time for a test criterion to 2 days. We report the number of individual runs for each technique, the covered, uncovered (nor covered, nor identiﬁed as infeasible, referred to as errors) and infeasible test predicates, the total time for the generation (in seconds), and the largest trace produced. For our technique, we also mention the number of automated reﬁnements that were necessary. As our experimental results show, our technique is able to deal with the generation in cases in which model checkers fail to do so. Let us explain our assessment of the experiments reported in the table at the end of this section. Notice that, whenever Spin is able to generate a test case, it does so very fast, but being an explicit state model checker, it generally runs out of memory quickly for models with large numerical domains. In speciﬁcations such that car3prop and sis, with either small ranges for integer variables or relatively few interleavings (due to having a small number of monitored variables), model checkers perform very well, being able to generate test cases much faster than our technique. On the other hand, for speciﬁcations such as ccs and autopilot, with large ranges for numerical variables and several monitored variables (leading to a

Abstraction Based Automated Test Generation

99

higher degree of interleaving), model checkers perform poorly and our technique shows better proﬁt. Runs

Spin NuSMV Cad. SMV SAL/SMC Lazy abs. +

169 409 378 296 62

Spin 582 Cad. SMV 582 Lazy abs. + 120

5

NuSMV Cad. SMV SAL/BMC Lazy abs. +

142 160 133 125

Spin NuSMV Cad. SMV SAL/SMC Lazy abs. +

19 27 31 27 24

Tests Time Max. Trace Cov. Errors Infeas. autopilot (409 test predicates) 288(48) 121 0 2351 168 0 409 0 timeout 36(5) 373 0 timeout 13 116(3) 293 0 timeout 6 390(43) 10 9 164003 127 ccs (582 test predicates) 0 582 0 timeout 0 582 0 timeout 494(32) 0 88 312 8 car3prop (498 test predicates) 402(46) 0 96 261.45 13 402(64) 0 96 77.58 10 402(37) 15 81 56.32 11 402(29) 0 96 1795 9 sis (91 test predicates) 80(8) 0 11 145.72 23515 80(16) 0 11 995.23 406 80(20) 0 11 420.34 402 80(16) 0 11 37.94 403 80(13) 0 11 32742 402

Rfnmts.

7099 167 1713 2955

Conclusion and Future Work

We have identiﬁed a number of characteristics inherent to formal tabular requirements speciﬁcations that can be exploited for improving automated analysis for test generation from these speciﬁcations. Indeed, we argued that certain features of these speciﬁcations can be exploited for deﬁning an abstraction process able to eﬀectively generate test cases, i.e., runs of the speciﬁcation, corresponding to a variety of test criteria relevant to tables. Basically, the identiﬁed characteristics have to do with typical elements used by the engineer in the construction of the formal speciﬁcation, and the inherent behavioural model of tables. We have compared our developed approach with test case generation from tables using some model checkers, as well as with a standard abstraction approach not exploiting the identiﬁed characteristics of tables. Our experimental results show that the identiﬁed characteristics play a signiﬁcant role in the scalability of abstraction employed in test case generation from tables, and that the resulting technique is able to deal better with some tabular speciﬁcations which, due to their complexity with respect to the size of numerical ranges and constants present in the model, and not handled well by some model checkers. The motivation for dealing with “larger” speciﬁcations is straightforward: it contributes to scalability in this analysis, and facilitates the validation and veriﬁcation activities, for which it is important that the generated tests maintain the level of abstraction/detail expected by users, and present in the implementation. Although abstraction has been successfully applied for test case generation via model checking, it has generally been applied in “control intensive” domains [8]. In contrast, requirements speciﬁcations are not control intensive, thus constituting a challenging domain to apply abstraction for test case generation. As our experiments show, directly

100

R. Degiovanni et al.

applying lazy abstraction for test case generation in the context of requirements speciﬁcations does not perform well, so our variant, exploiting characteristics of tables, shows its proﬁt. As future work, we plan to explore the use of terms for the abstraction, complementing our current use of mode classes; this is motivated by the fact that, according to [6], terms typically capture historical information, as modes do. We also plan to apply the presented approach for the veriﬁcation of speciﬁcations, i.e., to guaranteeing the properties associated with the well formedness of tables, as well as the veriﬁcation of state and transition invariants over tabular speciﬁcations. We are also extending the ideas presented in this paper to the framework for describing behaviours over tabular speciﬁcations presented in [1].

Acknowledgements The authors would like to thank Angelo Gargantini, who kindly provided the model checking speciﬁcations of the SCR models we used in our experiments, as well as a prototype tool to automate test generation from tables using various model checkers. This greatly simpliﬁed our experimental evaluation. We would also like to thank the anonymous referees for their helpful comments. This work was partially supported by the Argentinian Agency for Scientiﬁc and Technological Promotion (ANPCyT), through grant PICT 2006 No. 2484. The third author’s participation was also supported through ANPCyT grant PICT PAE 2007 No. 2772.

References 1. Aguirre, N.M., Frias, M.F., Moscato, M.M., Maibaum, T.S.E., Wassyng, A.: Describing and Analyzing Behaviours over Tabular Speciﬁcations Using (Dyn)Alloy. In: Chechik, M., Wirsing, M. (eds.) FASE 2009. LNCS, vol. 5503, pp. 155–170. Springer, Heidelberg (2009) 2. Atlee, J., Gannon, J.: State-Based Model Checking of Event-Driven System Requirements. IEEE Trans. Software Eng. 19(1) (1993) 3. Beyer, D., Chlipala, A., Henzinger, T., Jhala, R., Majumdar, R.: Generating Tests from Counterexamples. In: Proc. of ICSE 2004. IEEE, Los Alamitos (2004) 4. Bharadwaj, R., Heitmeyer, C.: Model Checking Complete Requirements Speciﬁcations Using Abstraction. Automated Software Engineering 6(1) (1999) 5. Bruttomesso, R., Cimatti, A., Franz´en, A., Griggio, A., Sebastiani, R.: The mathSAT 4 SMT solver. In: Gupta, A., Malik, S. (eds.) CAV 2008. LNCS, vol. 5123, pp. 299–303. Springer, Heidelberg (2008) 6. Bultan, T., Heitmeyer, C.: Applying Inﬁnite State Model Checking and Other Analysis Techniques to Tabular Requirements Speciﬁcations of Safety-Critical Systems. Design Automation for Embedded Systems 12(1-2) (2008) 7. Chaki, S., Clarke, E., Groce, A., Jha, S., Veith, H.: Modular Veriﬁcation of Software Components in C. Trans. on Software Engineering 30(6) (2004) 8. Clarke, E., Gupta, A., Jain, H., Veith, H.: Model Checking: Back and Forth between Hardware and Software. In: Meyer, B., Woodcock, J. (eds.) VSTTE 2005. LNCS, vol. 4171, pp. 251–255. Springer, Heidelberg (2008)

Abstraction Based Automated Test Generation

101

9. Feng, X., Parnas, D., Tse, T., O’Callahan, T.: A Comparison of Tabular Expression-Based Testing Strategies. IEEE Transactions on Software Engineering (to appear), http://www.cs.hku.hk/research/techreps/document/TR-2009-19.pdf 10. Fraser, G., Gargantini, A.: An Evaluation of Model Checkers for Speciﬁcation Based Test Case Generation. In: Proc. of ICST 2009. LNCS. Springer, Heidelberg (2009) 11. Gargantini, A., Heitmeyer, C.: Using Model Checking to Generate Tests from Requirements Speciﬁcations. In: Proc. of ESEC/FSE 1999. LNCS. Springer, Heidelberg (1999) 12. Heitmeyer, C., Jeﬀords, R., Labaw, B.: Automated consistency checking of requirements speciﬁcations. Trans. on Soft. Eng. and Methodology 5(3) (1996) 13. Heitmeyer, C., Archer, M., Bharadwaj, R., Jeﬀords, R.: Tools for constructing requirements speciﬁcations: the SCR Toolset at the age of nine. Computer Systems: Science & Engineering 20(1) (2005) 14. Heninger, K., Kallander, J., Parnas, D., Shore, J.: Software Requirements for the A-7E Aircraft, NLR Memorandum Report 3876, US Naval Research Lab (1978) 15. Henzinger, T., Jhala, R., Majumdar, R., Sutre, G.: Lazy abstraction. In: Proc. of POPL 2002. ACM, New York (2002) 16. Henzinger, T., Jhala, R., Majumdar, R., McMillan, K.: Abstractions from proofs. In: Proc. of POPL 2004. LNCS. Springer, Heidelberg (2004) 17. Leveson, N., Heimdahl, M., Hildreth, H., Reese, J.: Requirements Speciﬁcations for Process-Control Systems. Trans. on Software Engineering 20(9) (1994) 18. McMillan, K.L.: Interpolation and SAT-Based Model Checking. In: Hunt Jr., W.A., Somenzi, F. (eds.) CAV 2003. LNCS, vol. 2725, pp. 1–13. Springer, Heidelberg (2003) 19. Parnas, D., Madey, J.: Functional Documentation for Computer Systems. Science of Computer Programming 25(1) (1995) 20. Tillmann, N., de Halleux, J.: Pex–White Box Test Generation for .NET. In: Beckert, B., H¨ ahnle, R. (eds.) TAP 2008. LNCS, vol. 4966, pp. 134–153. Springer, Heidelberg (2008) 21. Visser, W., Pˇ asˇ areanu, C., Khurshid, S.: Test input generation with Java PathFinder. In: Proc. of ISSTA 2004. ACM, NY (2004)

Correct Code Containing Containers Claire Dross1 , Jean-Christophe Filliˆ atre2 , and Yannick Moy1 1

2

AdaCore, 46 rue d’Amsterdam, F-75009 Paris, France {dross,moy}@adacore.com CNRS, INRIA Saclay ˆIle-de-France, Universit´e Paris-Sud 11 [email protected]

Abstract. For critical software development, containers such as lists, vectors, sets or maps are an attractive alternative to ad-hoc data structures based on pointers. As standards like DO-178C put formal veriﬁcation and testing on an equal footing, it is important to give users the ability to apply both to the veriﬁcation of code using containers. In this paper, we present a deﬁnition of containers whose aim is to facilitate their use in certiﬁed software, using modern proof technology and novel speciﬁcation languages. Correct usage of containers and user-provided correctness properties can be checked either by execution during testing or by formal proof with an automatic prover. We present a formal semantics for containers and an axiomatization of this semantics targeted at automatic provers. We have proved in Coq that the formal semantics is consistent and that the axiomatization thereof is correct. Keywords: containers, iterators, veriﬁcation by contracts, annotations, axiomatization, API usage veriﬁcation, SMT, automatic provers.

1

Introduction

Containers are generic data structures oﬀering a high-level view of collections of objects, while guaranteeing fast access to their content to retrieve or modify it. The most common containers are lists, vectors, sets and maps, which are usually deﬁned in the standard library of languages, like in C++ STL, Ada Standard Libraries or Java JCL, and sometimes even as language elements, like sets in SETL [17] or maps in Perl. In critical software where veriﬁcation objectives severely restrict the use of pointers, containers oﬀer an attractive alternative to pointer-intensive data structures. Containers oﬀer both a better defense against errors than low-level code manipulating pointers, and a rich high-level API to express properties over data. This is particularly evident when the implementation of containers themselves obeys the coding standards of critical software, with no dynamic allocation and few pointers, as is the case for the bounded containers deﬁned in the proposed Ada 2012 standard [3]. Standards for critical software development deﬁne comprehensive veriﬁcation objectives to guarantee the high levels of dependability we expect of life-critical and mission-critical software. All requirements must be shown to be satisﬁed by the software, which is a costly activity. In particular, veriﬁcation of low-level M. Gogolla and B. Wolﬀ (Eds.): TAP 2011, LNCS 6706, pp. 102–118, 2011. c Springer-Verlag Berlin Heidelberg 2011

Correct Code Containing Containers

103

requirements is usually demonstrated by developing unit tests, from which high levels of conﬁdence are only obtained at a high cost. This is the driving force for the adoption of formal veriﬁcation on an equal footing with testing to satisfy veriﬁcation objectives. The upcoming DO-178C avionics standard states: Formal methods [..] might be the primary source of evidence for the satisfaction of many of the objectives concerned with development and verification. Although there are some areas where formal veriﬁcation can be applied independently [18], most areas where testing is the main source of evidence today would beneﬁt from an integration of formal veriﬁcation with existing testing practice. At the simplest, this combination should be provably sound and it should guarantee a coverage of atomic veriﬁcations through formal veriﬁcation and testing. This is the goal of project Hi-Lite [14], a project aiming at combined unit testing and unit proof of C and Ada programs. In the context of project Hi-Lite, this paper deals with the deﬁnition of suitable containers in Ada, based on Ada standard containers, whose properties can be both tested dynamically and proved automatically. Properties over containers oﬀer a high level description of the code, suitable for expressing partial correctness in the form of code contracts. Therefore, we are not only interested in correct usage of container APIs, but also in partial correctness of functional properties of interest to users. Before they can be used in the context of avionics (or similar) safety-critical software, the new library of formal containers will need to be certiﬁed and the veriﬁcation tools we present will have to undergo qualiﬁcation [2]. We present in this paper: (a) a formal proof of correctness of an implementation of the new library in Coq [1], a well-known formal programming language; and (b) a formal proof of the new library properties used in our veriﬁcation tools, expressed in the Why language [12] for formal veriﬁcation. Thus, this work can be seen as a contribution to the argument-based approach to certiﬁcation [16]. In Section 2, we detail the changes that we introduce in formal containers w.r.t. containers as deﬁned in language standards. In the following sections, we describe in detail formal containers for doubly linked lists, and we sketch formal containers for vectors, sets and maps: formal semantics in Section 3, formal speciﬁcation in the Why language in Section 4, formal proof of correctness in Coq in Section 5. We ﬁnally give a survey of related works in Section 6. A longer version of this article with more details and proofs is available on the web [4]. A web page gives an introduction to the code for containers’ implementation and proofs [5], and instructions to anonymously access the git repository where all the source code is stored.

2

Formal Containers

We will use the following Ada code as a running example throughout the section. Procedure Map_F 1 modiﬁes a list in place, by replacing each element initially contained in the list by its image through function F. 1

Note that Map F is client code, not part of the API.

104

C. Dross, J.-C. Filliˆ atre, and Y. Moy

procedure Map_F ( L : in out List ) i s Current : Cursor := First ( L ); begin while Current /= No_Element loop Replace_Elem en t (L , Current , F ( Element ( Current ))); Next ( Current ); end loop ; end Map_F ;

2.1 Contracts in Ada 2012 The forthcoming version of the Ada standard, called Ada 2012 [3], oﬀers a variety of new features to express properties of programs. New checks are deﬁned as aspects of program entities, for which the standard deﬁnes precisely the various points at which the check is performed during execution. The most prominent of these new checks are the Pre and Post aspects which deﬁne respectively the precondition and postcondition of a subprogram. These are deﬁned as Boolean expressions over program variables and functions. For example, a simple contract on function Map_F could specify that its parameter list should not be empty, and that the call does not modify its length: procedure Map_F ( L : in out List ) with Pre = > Length ( L ) /= 0 , Post = > Length ( L ) = Length (L ’ Old );

Notice that, in the precondition, L refers to the list in the pre-state while, in the postcondition, L refers to the list in the post-state, hence the need to refer to the special attribute L’Old in the postcondition, which designates the value of L before the call. 2 The execution model for these aspects is simply to insert assertions at appropriate locations, which raise exceptions when violated. For each variable V whose pre-state value may be read in the postcondition (V’Old), the compiler inserts a shallow copy of the variable’s value at the beginning of the subprogram body. It is this copy which is read to give the value of V’Old. Expressing properties in contracts is greatly facilitated by the use of ifexpressions, case-expressions, quantiﬁed-expressions and expression-functions, all deﬁned in Ada 2012. The main objective of these logic features is veriﬁcation by testing, based on their executable semantics. In particular, quantiﬁedexpressions are always expressed over ﬁnite ranges or (obviously ﬁnite) containers, with a loop through the range or container as execution model: for all J in 0 .. 10 => P (J) is true if-and-only-if the subprogram P returns True for every argument, starting from 0 up to 10. 2.2 Ada Standard Containers Like in many other languages, Ada standard containers deﬁne two mutually dependent data structures: containers proper which hold the data, and iterators (or cursors) which provide handles to individual pieces of data. In function 2

More precisely, attribute ’Old can be applied to all names (as deﬁned in Ada standard), so that we could use Length (L)’Old instead of Length (L’Old).

Correct Code Containing Containers

105

Map_F, the container has type List and the iterator has type Cursor, which are both deﬁned by the standard Ada lists. A cursor is implicitly associated with a container (implemented as a pointer ﬁeld in the cursor structure), in which it designates an element. An important feature of Ada containers is that cursors remain valid as long as the container exists and the element referenced is not deleted, like many iterators in other languages (for example those in C++ STL and Java JCL). This allows modifying a container while iterating over its content with cursors, without risk of invalidating these cursors. 2.3

API Modiﬁcation: Independent Cursors

Problem. A useful postcondition for Map_F is to state how elements are modiﬁed by the call. All cursors are preserved through replacement of an element in a list. Thus, for every cursor Cu that designates an element E in L before the call, Cu designates F (E) in L after the call. It seems like we could express it with a quantiﬁed-expression: procedure Map_F ( L : in out List ) with Post = > ( f o r a l l Cu in L = > Element ( Cu ) = ???);

The expression denoted by ??? should designate the value obtained by calling F (Element (Cu)) in the pre-state, which could be intuitively written as F (Element (Cu))’Old. Unfortunately, this expression is not valid because it refers to the value in the pre-state of Cu, which is not deﬁned in the pre-state since it is quantiﬁed. As a side note, notice that, even for a cursor Cu deﬁned outside of the Map_F function, Cu’Old would be the same as Cu in our example, because the semantics of attribute ’Old is to perform a shallow copy, so it does not copy the implicit container in a cursor. Approach. To solve the previous two problems, we break the implicit link between cursors and containers, so that the same cursor can be used both in the pre-state and in the post-state. Then, the previous postcondition can be expressed easily: procedure Map_F ( L : in out List ) with Post = > ( f o r a l l Cu in L = > Element (L , Cu ) = F ( Element (L ’ Old , Cu )));

Notice that we passed an additional argument to function Element, to indicate the container from which the element at cursor Cu should be retrieved. This is true for every function in the API of containers which previously accessed implicitly the container through a cursor, such as Next in the example of Section 2. This is the only modiﬁcation to the API of containers that we introduce. The alternative of using existing containers both greatly complicates the execution model and the formal veriﬁcation. Using existing containers would require a diﬀerent semantics for the ’Old attribute, which would reach to the complete pre-state including the stack and

106

C. Dross, J.-C. Filliˆ atre, and Y. Moy

heap, similar to the work by Kosiuczenko on Java programs [15] that builds a complete history of updates alongside execution. Our solution has the beneﬁts of sticking to the standard semantics for ’Old in Ada 2012, leading to a simple and eﬃcient execution model. The semantics of standard cursors also leads to more complex veriﬁcation conditions to check the correct use of containers’ API: each access through cursor Cu to container Co is valid only if 1) Co is alive, which can be hard to know if Co is implicit, and 2) Cu is associated to container Co, which amounts to deciding whether Co is the same as the implicit container in Cu. With the semantics of formal containers, both veriﬁcation conditions above disappear. 2.4

API Addition: Parts of Containers

Problem. In order to prove the postcondition of Map_F stated above, we need to annotate the loop in Map_F with a loop invariant, which states the accumulated eﬀect of N iterations through the loop. We would like to state that list elements already scanned have been modiﬁed by F (as in the postcondition) and that list elements not yet scanned are unchanged. This pattern of loop invariant, consisting of two parts for the elements scanned and the elements not yet scanned, is typical of loops that iterate over a container. Approach. We introduce two new functions, called Left and Right, which return the containers holding respectively the elements preceding (exclusively) or following (inclusively) a given cursor in the container. With these functions, the eﬀect of the loop on the elements already scanned resembles the postcondition: pragma Assert ( f o r a l l Cu in Left (L , Current ) = > Element (L , Cu ) = F ( Element (L ’ Old , Cu )));

The eﬀect of the loop on the elements not yet scanned is not simply the equality of right containers. Indeed, equality of lists L1 and L2 only implies that, while iterating separately through each of the lists, the same elements are encountered in the same order. Here, we also need to be able to iterate on both lists with the same cursor, so that the ﬁrst cursors of L1 and L2 should be equal and then each call to Next should return the same cursor on both lists which is not implied by equality. This is expressed with a new function, called Strict_Equal: pragma Assert ( Strict_Equal ( Right (L , Current ) , Right (L ’ Old , Current )));

3

Formal Semantics

In this section, we present a formal semantics for lists. We show brieﬂy how the formal semantics of other containers compares to the one of lists in the last subsection.

Correct Code Containing Containers

3.1

107

Syntax of Lists

A program is a sequence of variable declarations for lists (in Lvar) and cursors (in Cvar) followed by a sequence of instructions (in Instr). Procedures Insert, Delete and Replace Element modify their ﬁrst argument, which must be therefore a variable, and have no return value. The remaining instructions are assignments. Notice that list assignment makes an explicit copy of its argument, which prevents aliasing between lists. LExpr is the set of list expressions. Empty is the empty list constant. Functions Left and Right return parts of containers as deﬁned in Section 2.4. CExpr is the set of cursor expressions. No Element is the constant invalid cursor. Functions First, Last, Next and Previous are used for iterating over lists. EExpr is the set of element expressions. Function Element accesses the element designated by a cursor in a list. BExpr is the set of Boolean expressions. Has Element checks for the validity of a cursor in a given container, = is the structural equality and Strict Equal is the more constraining equality described in Section 2.4. Finally, IExpr is the set of integer expressions. Function Length returns the length of a list. Instr := Insert (LVar, CExpr, EExpr)

CExpr := Cvar

|

Delete(LVar, CVar)

|

No Element

|

Replace Element

|

First(LExpr)

|

Last (LExpr)

(LVar, CExpr, EExpr) |

Cvar := CExpr

|

Next (LExpr, CExpr)

|

Lvar := Copy(LExpr)

|

Previous (LExpr, CExpr)

|

...

LExpr := Lvar

EExpr := Element (LExpr, CExpr) |

...

|

Empty

|

Left(LExpr, CExpr)

|

LExpr = LExpr

|

Right (LExpr, CExpr)

|

CExpr = CExpr

|

Strict Equal (LExpr, LExpr)

|

...

IExpr := Length (LExpr) |

...

BExpr := Has Element(LExpr, CExpr)

For the sake of simplicity, we only list instructions and expressions that are speciﬁc to containers. Thus, we have not included loops or branching statements in the set of instructions, or arithmetic operations in the set of integer expressions. 3.2

Operational Semantics of Lists

Since lists are generic in the kind of element they contain, the type E of elements is left unspeciﬁed. The type D of cursors can be any countably inﬁnite type. We add a speciﬁc element ⊥ to this set, D∪⊥ is written D⊥ . There is an environment for lists ΓL , and an environment for cursors ΓC :

108

C. Dross, J.-C. Filliˆ atre, and Y. Moy

ΓL : Lvar → L = {Len : N, Fc : [1..Len] → D, Fe : Im(Fc) → E} ΓC : Cvar → D⊥ Intuitively, lists can be seen as an array Fc of cursors with a mapping Fe from cursors to elements. Fc is injective, so Fc restricted to Im(Fc) is bijective, and Fc −1 : Im(Fc) → [1..Len] is its inverse. We extend Fc −1 to Fc −1 + : Im(Fc) ∪ −1 {⊥} → [1..Len + 1] with Fc + (⊥) = Len + 1. Given an instruction I in Instr, a list l in LExpr, a cursor c in CExpr, an expression e in EExpr, a Boolean expression b in BExpr and an integer expression i in IExpr, judgments take the following form: ΓL , ΓC I ⇒ ΓL , ΓC ΓL , ΓC e ⇒ E

ΓL , ΓC c ⇒ D⊥

ΓL , ΓC l ⇒ L ΓL , ΓC b ⇒ B

ΓL , ΓC i ⇒ Z

If expr is an expression, ΓL , ΓC expr ⇒ val means that expr evaluates in environments ΓL and ΓC to a value represented by val in the semantics. If instr is an instruction, ΓL , ΓC instr ⇒ ΓL , ΓC means that instr change the environments ΓL and ΓC into ΓL and ΓC . Below are the description of the semantics of integer, element and boolean expressions. The result of function Length on a list evaluating to {Len, Fc, Fe} is Len. Similarly, function Element returns the value of Fe on d, where cursor argument c evaluates to d. Notice that Element (l, c) is deﬁned only when d ∈ Im(Fc), which is expressed in the informal semantics as c designates an element in l. Indeed, the associated Ada function will raise a run-time error otherwise. Has Element (l, c) checks if c eﬀectively designates an element in l. Equality over lists (=) is the structural equality. It only implies that the elements in its two list arguments appear in the same order, i.e., the equality of Fe ◦ Fc : [1..Len] → E. Strict Equal is stronger than =, as expected, since it also implies the equality of Fc and Fe. Equality of cursors is simply equality of their evaluations. ΓL , ΓC l ⇒ {Len, Fc, Fe} ΓL , ΓC Length (l) ⇒ Len ΓL , ΓC l ⇒ {Len, Fc, Fe}

ΓL , ΓC c ⇒ d

d ∈ Im(Fc)

ΓL , ΓC Element(l, c) ⇒ Fe(d) ΓL , ΓC l ⇒ {Len, Fc, Fe}

ΓL , ΓC c ⇒ d

ΓL , ΓC Has Element(l, c) ⇒ d ∈ Im(Fc) ΓL , ΓC l1 ⇒ {Len 1 , Fc 1 , Fe 1 }

ΓL , ΓC l2 ⇒ {Len 2 , Fc 2 , Fe 2 }

ΓL , ΓC l1 = l2 ⇒ Len 1 = Len 2 & Fe 1 ◦ Fc 1 = Fe 2 ◦ Fc 2 ΓL , ΓC l1 ⇒ {Len 1 , Fc 1 , Fe 1 }

ΓL , ΓC l2 ⇒ {Len 2 , Fc 2 , Fe 2 }

ΓL , ΓC Strict Equal (l1 , l2 ) ⇒ Len 1 = Len 2 & Fc 1 = Fc 2 & Fe 1 = Fe 2 ΓL , ΓC c1 ⇒ d1

ΓL , ΓC c2 ⇒ d2

ΓL , ΓC c1 = c2 ⇒ d1 = d2

Below are the description of the semantics of cursor expressions. The special invalid cursor No Element evaluates to ⊥. This is possible because ⊥ cannot

Correct Code Containing Containers

109

appear in Im(Fc), as Fc : [1..Len] → D. Therefore ⊥ is not a valid cursor, i.e., it designates no element, in any list. Function Next is deﬁned for both valid cursors and No Element . It returns No Element when applied to a cursor which has no valid successor (i.e., for No Element and the last cursor). Previous is similar. Function First is deﬁned on every list. It returns No Element when called on an empty list. Last is similar. ΓL , ΓC No Element ⇒ ⊥ ΓL , ΓC l ⇒ {Len, Fc, Fe}

ΓL , ΓC c ⇒ d1

d1 ∈ Im(Fc) ∪ {⊥}

ΓL , ΓC Next (l, c) ⇒ d2 where d2 = {Len = 0 & d1 ∈ Im(Fc)\{Fc(Len)} → Fc(Fc −1 (d1 ) + 1), else → ⊥} ΓL , ΓC l ⇒ {Len, Fc, Fe}

ΓL , ΓC c ⇒ d1

d1 ∈ Im(Fc) ∪ {⊥}

ΓL , ΓC Previous(l, c) ⇒ d2 where d2 = {Len = 0 & d1 ∈ Im(Fc)\{Fc(1)} → Fc(Fc −1 (d1 ) − 1), else → ⊥} ΓL , ΓC l ⇒ {Len, Fc, Fe} ΓL , ΓC First(l) ⇒ d

where d = {Len = 0 → ⊥, Len > 0 → Fc(1)} ΓL , ΓC l ⇒ {Len, Fc, Fe}

ΓL , ΓC Last (l) ⇒ d

where d = {Len = 0 → ⊥, Len > 0 → Fc(Len)}

Below are the description of the semantics of list expressions. The empty list, returned by Empty, is the only list whose length is null (F∅ is the only function that is deﬁned on the empty set ∅). Left is deﬁned for both valid cursors and No Element . Its evaluation yields a list whose valid cursors are the valid cursors of the list argument which precede cursor argument c (when c is No Element , that means all cursors). Right is similar. ΓL , ΓC Empty ⇒ {0, F∅ , F∅ } ΓL , ΓC c ⇒ d

ΓL , ΓC l ⇒ {Len, Fc, Fe} d ∈ Im(Fc) ∪ {⊥} n = Fc −1 + (d) − 1

ΓL , ΓC Left(l, c) ⇒ {n, Fc , Fe } ΓL , ΓC c ⇒ d

where Fc = Fc|[1..n]

Fe = Fe|Im(Fc )

ΓL , ΓC l ⇒ {Len, Fc, Fe} d ∈ Im(Fc) ∪ {⊥} n = Fc −1 + (d) − 1

ΓL , ΓC Right (l, c) ⇒ {Len − n, Fc , Fe } where Fc = λi : [1..Len − n].Fc(n + i) Fe = Fe|Im(Fc )

The rules below describe the semantics of instructions. Rules concerning reads or assignment of variables are omitted (they are the usual ones). Insert modiﬁes the environment so that its list variable argument designates, after the call, a list where a cursor and an element have been inserted at the proper place. The cursor argument, which can be either a valid cursor or No Element , encodes the place the new element is inserted. The newly created cursor is not speciﬁed. It should be diﬀerent from No Element and from every valid cursor in the argument list. Delete modiﬁes the environment so that its cursor variable argument (which must reference a valid cursor before the call) is deleted from the list referenced by its list variable argument. The cursor variable references the special invalid cursor No Element after the call. Replace Element modiﬁes the environment so

110

C. Dross, J.-C. Filliˆ atre, and Y. Moy

that, after the call, its cursor argument (which must be valid) designates its element argument in the list referenced by its list variable argument. d1 ∈ Im(Fc) ∪ {⊥}

ΓL (l) = {Len, Fc, Fe} ΓL , ΓC c ⇒ d1 n = Fc −1 (d ) d ∈ / Im(Fc) ∪ {⊥} 1 2 +

ΓL , ΓC e ⇒ Elt

ΓL , ΓC Insert (l, c, e) ⇒ ΓL [l → {Len + 1, Fc , Fe }], ΓC where Fc = λi : [1..Len + 1]. {i ∈ [1..n − 1] → Fc(i), i = n → d2 , i ∈ [n + 1..Len + 1] → Fc(i − 1)} Fe = λd : Im(Fc ).{d ∈ Im(Fc) → Fe(d), d = d2 → Elt} ΓL (l) = {Len, Fc, Fe}

ΓC (c) = d

d ∈ Im(Fc)

n = Fc −1 (d)

ΓL , ΓC Delete(l, c) ⇒ ΓL [l → {Len − 1, Fc , Fe }], ΓC [c → ⊥] where Fc = λi : [1..Len − 1].{i ∈ [1..n − 1] → Fc(i), i ∈ [n..Len − 1] → Fc(i + 1)} Fe = Fe|Im(Fc )

ΓL (l) = {Len, Fc, Fe}

ΓL , ΓC c ⇒ d1

d1 ∈ Im(Fc)

ΓL , ΓC e ⇒ Elt

ΓL , ΓC Replace Element(l, c, e) ⇒ ΓL [l → {Len, Fc, Fe }], ΓC where Fe = λd : Im(Fc).{d = d1 → Elt, d = d1 → Fe(d)}

3.3

Vectors, Sets and Maps

Sets. Sets do not allow duplication of elements, the order of iteration in a set is not user-deﬁned and the link between cursors and elements is preserved in most cases. In the semantics, sets can be modeled as the same tuples as lists where Fe is injective: {Len : N, Fc : [1..Len] → D, Fe : Im(Fc) → E}. For non ordered sets, the order of iteration is not speciﬁed. If the set is ordered, the order of iteration is constrained by the order over elements. As a consequence, function Fe ◦ Fc has to preserve order. The longer version of this article presents the modiﬁed inference rule for Insert for each container. Maps. Maps behave just like sets of pairs key/element except that they only constrain keys: (k1 , e1 ) < (k2 , e2 ) ↔ k1 < k2 and (k1 , e1 ) = (k2 , e2 ) ↔ k1 = k2 . Vectors. Vectors do not expect cursors to keep designating the same element in every case. Instead, as for arrays, elements can be accessed through their position (an index). As a consequence, we model vectors as tuples {Len : N, Fc : [1..Len] → D, Fe : [1..Len] → E} where Fc is injective. When an element is inserted or deleted from a vector, nothing can be said for the cursors that follow the place of insertion/deletion.

4

Axiomatization

In this section, we present an axiomatization of lists in the language Why, targeted at automatic provers. We show that this axiomatization is correct w.r.t. the formal semantics we gave in Section 3. We will formally prove it is correct w.r.t. formal semantics in Coq in Section 5.

Correct Code Containing Containers

4.1

111

Presentation of Why

The Why platform [12] is a set of tools for deductive program veriﬁcation. The ﬁrst feature of Why is to provide a common frontend to a wide set of automated and interactive theorem provers. Why implements a total, polymorphic, ﬁrstorder logic, in which the user can declare types, symbols, axioms and goals. These goals are then translated to the native input languages of the various supported provers. In our case, we are using the backends for the Coq proof assistant [1] and the three SMT solvers Alt-Ergo [8], Z3 [9] and Simplify [10]. For example, here is some Why syntax that declares a modulo operation over integers, together with some possible axiomatization: l o g i c mod_ : int , int -> int axiom mod__ : f o r a l l a , b : int . 0 < b -> e x i s t s q : int . a = b * q + mod_ (a , b ) and 0 <= mod_ (a , b ) < b goal test : mod_ (7 , 2) = 1

The second feature of Why we use here is to provide a veriﬁcation condition generator for an idealized, alias-free, Hoare-logic-like programming language. The user declares and implements programs, which are annotated with preand postconditions, and local assertions such as loop invariants. Veriﬁcation conditions are then generated and transmitted to the theorem provers. Here is for instance the declaration of a program function parameter mod : a : int -> b : int -> {0 < b } int { result = mod_ (a , b )}

which takes two integers a and b as arguments, has precondition 0 < b, returns a result of type int and has postcondition result = mod_(a, b). When used inside programs, this function will trigger veriﬁcation conditions (namely, that its second argument is positive). This is one way to express that modulo is a partial operation. 4.2

Axiomatization of Lists

Note that this axiomatic is not meant to be the exact translation of our semantics, but rather is written to facilitate the veriﬁcation of programs with automatic provers. Types. The type of elements is irrelevant, and so it is deﬁned as an abstract type element_t. Cursors and lists are described respectively with abstract types cursor and list, further axiomatized in the following. Properties. To encode the semantics deﬁned in Section 3, we introduce three logic functions: l o g i c length_ : list -> int l o g i c position_ : list , cursor -> int l o g i c element_ : list , cursor -> element_t

112

C. Dross, J.-C. Filliˆ atre, and Y. Moy

Logic functions length_ and element_ deﬁne accessors to the ﬁelds Len and Fe of a list. The encoding is more complex for the ﬁeld Fc, due to the fact that Fc is used in three diﬀerent ways in the speciﬁcation: 1) directly, 2) through its inverse Fc −1 , and 3) through its domain Im(Fc). Function position_ is the extension of Fc −1 to cursors not in Im(Fc), whose image is set to 0. It gives access to both Im(Fc) (c ∈ Im(Fc) ⇔ position_(l, c) > 0) and Fc −1 . We rewrite almost all rules of the semantics to remove occurrences of Fc. For example, in the rule for Next, d2 = Fc(Fc −1 (d1 ) + 1) can be rewritten as Fc −1 (d2 ) = Fc −1 (d1 ) + 1. The only rule that cannot be translated that way is =. For this one rule, we can use an existential quantiﬁcation, −1 ∀ d1 : Im(Fc 1 ).∃ d2 : Im(Fc 2 ).Fc −1 1 (d1 ) = Fc 2 (d2 ) & Fe 1 (d1 ) = Fe 2 (d2 ). Why functions length_, element_ and position_ are related to the semantics as follows: ∀ l, i, length_(l) = i ⇔ ∃ Len, Fc, Fe, ΓL , ΓC l ⇒ {Len, Fc, Fe} & Len = i ∀ l, c, i, position_(l, c) = i ⇔ ∃ Len, Fc, Fe, d, ΓL , ΓC l ⇒ {Len, Fc, Fe} & ΓL , ΓC c ⇒ d & i ≥ 0 & (i = 0 → d ∈ / Im(Fc)) & (i > 0 → d ∈ Im(Fc) & Fc −1 (d) = i) ∀ l, c, e, position_(l, c) > 0 → element_(l, c) = e ⇔ ∃ Len, Fc, Fe, d, Elt, ΓL , ΓC l ⇒ {Len, Fc, Fe} & ΓL , ΓC c ⇒ d & ΓL , ΓC e ⇒ Elt & (d ∈ Im(Fc) → Fe(d) = Elt)

Axioms. We encode the semantic properties of functions length_ and position_ into axioms, while ensuring that the axiomatic is not unnecessarily restrictive (all semantic lists should be also axiomatic lists). We have four axioms: 1. 2. 3. 4.

∀ ∀ ∀ ∀

l, l, l, l,

length_(l) >= 0 c, length_(l) >= position_(l, c) >= 0 position_(l, no_element) = 0 c1, c2, position_(l,c1) = position_(l,c2) > 0 → c1 = c2

It is rather straightforward to check that these axioms are implied by the semantics. These proofs can be found in the longer version of this article. Semantic Rules. Each Ada function represented in the semantics is translated Pre S into a Why program. Given the semantic rule deﬁning this function, we Post S deﬁne a precondition Pre A and a postcondition Post A on the Why program, so that Pre A ⇒ Pre S and Post S ⇒ Post A . In fact, since preconditions are quite

Correct Code Containing Containers

113

simple, we usually have Pre A = Pre S . We illustrate the general pattern that we applied with function Next, which we translate into program next in Why: parameter next : l : list -> c : cursor -> { c = no_element or position_ (l , c ) > 0 } cursor { result = next_ (l , c ) }

The precondition of this function states that either the argument cursor c is valid in the argument list l (because position_(l, c) > 0 ⇔ c ∈ Im(Fc)) or the argument cursor is equal to No Element . This precondition is exactly the condition for the application of the semantic rule for Next. An axiom next__ deﬁnes the behavior of logic function next_ over the allowed cases only, leaving the value of other applications of next_ unspeciﬁed, as seen in Section 4.1: axiom next__ : f o r a l l l : list . f o r a l l c : cursor . f o r a l l nxt : cursor . ( length_ ( l ) > position_ (l , c ) > 0 -> position_ (l , nxt ) = position_ (l , c ) + 1) and ( length_ ( l ) > 0 and position_ (l , c ) = length_ ( l ) or c = no_element -> nxt = no_element )

Axiom next__ is deﬁned as two implications: in the ﬁrst case, the next cursor of the cursor argument is valid and we deﬁne its position; in the second case, the result is No Element. Intuitively, this is the same as the semantic rule for Next. Using the same equivalences as above, we can rewrite axiom next__ into a logic formula that is exactly the semantics of Next, rewritten to use only Fc −1 . This proof is presented in the longer version of this article. 4.3

Eﬀectiveness

It is worth noting that the main diﬃculty we faced, when developing the axiomatization presented above, was to match the somewhat fuzzy expectations of SMT automatic provers: some work best with large predicates and few axioms, some work best with many smaller axioms, etc. We wrote a number of tests (30) to convince ourselves that the axiomatization of lists presented is eﬀective. With a combination of provers we managed to prove all the generated veriﬁcation conditions. Our running example is proved rather quickly (less than 1s per VC). To facilitate proofs, which impacts both provability and speed, we deﬁned 15 lemmas. These are not a burden in the maintainability of the axiomatic and cannot introduce inconsistencies since they are also automatically proved. Most of the tests correspond to unit-tests for speciﬁc properties of list expressions (especially for complex ones, such as Left ) and instructions. For example, there is a test to check that after replacing the value of an element in the list, the list indeed contains this value at this position after the call:

114

C. Dross, J.-C. Filliˆ atre, and Y. Moy

l e t test_repla ce _ el e m en t ( co : list ref ) ( cu : cursor ) ( e : element_t ) = { has_element_ ( co , cu ) } replace_elem en t co cu e { element_ ( co , cu ) = e }

A few tests, such as our running example, combine expressions and instructions to validate more complex behaviors of the API. All tests were proved automatically. 4.4

Vectors, Sets and Maps

Here is a table referencing the work done for each container. It contains the size of its Why ﬁle (its number of lines), the number of lemmas given and the number of tests passed. The code and tests are available on the web. Container Lines Lemmas Tests Container Lines Lemmas Tests List 352 15 30 Ordered Sets 506 30 38 Vectors 298 0 22 Hashed Maps 394 27 22 Hashed Sets 429 24 35 Ordered Maps 476 35 25

5

Validation of the Axiomatization

We have presented a formal semantics for containers in Section 3 and its axiomatization in Why in Section 4. We have shown a pen-and-paper proof that the axiomatization presented is correct w.r.t. formal semantics. Given the size of the axiomatization, such a manual proof may easily contain errors. In this section, we describe an implementation in Coq of the formal semantics of containers, and a formal proof of correctness of the Why axiomatization of lists w.r.t. the Coq implementation, (which also implies the consistency of the axiomatization). 5.1

Coq Implementation and Proof for Lists

Types. Our proofs are generic in the element type. For the representation of cursors, we use positive natural numbers to which we add 0 to model the special cursor ⊥. For lists, we model the tuple with a functional list of pairs cursorelement (if a is an element of this list, fst a refers to the associated cursor and snd a to the element). The ﬁeld Len is the length of the functional list, the ﬁeld Fc is the function that, for an integer i ∈ [1..Len], returns the cursor of the pair at the ith position in the list and Fe returns, for each cursor in Im(Fc), its ﬁrst association in the list. Therefore, a list of this kind always deﬁnes one and only one tuple {Len, Fc, Fe}. D e f i n i t i o n cursor : Set := nat . D e f i n i t i o n Rlist : Set := List . list ( cursor * element_t ).

To keep only tuples where Fc is injective and has value in D, we constrain the functional lists with a predicate. This predicate states that every cursor that

Correct Code Containing Containers

115

appears in a list is diﬀerent from ⊥ (positive) and does not appear again in the same list. Fixpoint well_formed ( l : Rlist ) : Prop := match l with nil = > True | a :: ls = > fst a > 0 /\ has_element ls ( fst a ) = false /\ well_formed ( ls ) end. Record list := { this : > Rlist ; wf : well_formed this }.

The property of well formedness has to be preserved through every modiﬁcation of the list. With the Coq lists restricted that way, there is one and only one list per tuple {Len : N, Fc : [1..Len] → D, Fe : Im(Fc) → E}. The two representations are equivalent. Axioms. Thanks to the -coq option of Why, we translate automatically our Why axioms into Coq. They can then be proved valid formally. Semantic Rules. We have written an implementation for each construct of our language. Since Coq is a pure functional language, the instructions that modify their list argument, such as Insert , are modeled by a function that returns a new list. The implementations are as close as possible to the semantic of their corresponding construct. Since they are Coq functions, these implementations have to be total, so we complete them for the cases that are not described in the semantics. For example, the semantics of Next is only deﬁned when the cursor that designates the element to be replaced is valid in the list or equal to No Element 3 . In the Coq implementation below, it returns No Element if we are not in that case (we could have chosen any other return value). Fixpoint next ( l : Rlist ) ( cu : cursor ) := match l with nil = > no_element | a :: ls = > i f beq_nat ( fst a ) cu then first ls e l s e next ls cu end.

The result of the semantic rule for Next is completely deﬁned. It is easy to be convinced that, when the cursor given as a parameter is indeed valid in the list or equal to No Element , the Coq function next returns the appropriate cursor. We use this representation to prove formally that the contracts in our axiomatic are indeed implied by the semantics. For nearly every rule of the semantics of lists, the result of the modiﬁcation is completely deﬁned in terms of the value of the arguments. The only rule that is not completely determined is Insert, since the value of the new cursor is not 3

Using it elsewhere reﬂects a mistake and is reported as an error when executed.

116

C. Dross, J.-C. Filliˆ atre, and Y. Moy

given. For the implementation, we deﬁne a function new that returns a valid cursor. To keep our proofs as general as possible, we took care to use only the properties of new that were deﬁned in the semantic (i.e., that the result of new ∈ / Im(Fc) ∪ ⊥) by enforcing it thanks to Coq’s module system. 5.2

Vectors, Sets and Maps

All containers have the same Coq representation. Therefore, some parts of the proofs are shared. Sets and maps (ordered or not) share the same lemmas and heavily rely on those of lists. Vectors also rely on the lemmas of lists but less heavily (they are quite diﬀerent). To make the proofs more reliable, general lemmas, which will not be aﬀected by a slight change in the API, are collected in a separate ﬁle named “Raw”. Here are the size of the ﬁles (the number of lemmas in each ﬁle) and the architecture. Raw Lists (154) Raw Vectors (108) Raw Sets (139) Lists Vectors Hashed Ordered Hashed Ordered (21) (29) Sets (64) Sets (69) Maps (64) Maps (70) Like for the Insert rule for lists, every unspeciﬁed part of the semantic of every container is kept as general as possible thanks to a sealed Coq module that only allows proofs to use the speciﬁed parts. The whole proof is 16,000+ lines of Coq. All of it plus commented excerpts can be found on the web.

6

Related Work

Formal proof over containers is an active area of research. There are two important, complementary areas in this domain: certifying user code that uses containers while assuming that their implementation complies with their speciﬁcation (what we are doing) and certifying that an implementation of containers indeed complies with its speciﬁcations. On the one hand, Bouillaguet et al. [7] focus on verifying that a container’s implementation indeed complies with its speciﬁcations. They use resolution based ﬁrst-order theorem provers to verify that the invariants of data structures such as sets and maps are preserved when considering operations on their encodings as arrays and trees. Zee et al. [19] even presented the ﬁrst veriﬁcation of full functional correctness for some linked data structure implementations. Unlike Bouillaget et al., they use interactive theorem provers as well to discharge their veriﬁcation conditions. Since they aim at certifying an implementation once, it does not seem to be too heavy a burden. On the other hand, Gregor and Schupp [13] focus, like we do, on the certiﬁcation of user programs. They present an unsound static analysis of C++ programs using STL containers. They generate partially executable assertions in C++ to express the constraints over containers’ usage, in particular a nonexecutable foreach quantiﬁer to iterate over all objects of a given type in the

Correct Code Containing Containers

117

current memory state. Blanc et al. [6] also work on certifying user code using the C++ STL containers. Just as we did in this work, they axiomatize the containers and then construct some preconditions (resp. postconditions) that are more (resp. less) constraining than those of the semantics. Their work is still substantially diﬀerent from what we did since they only check that the containers are properly used and they have no annotation language to allow the user to express other properties. Dillig et al. [11] present a static analysis for reasoning precisely over the content of containers. While they assume that we can analyze the code that ﬁlls the containers to provide constraints over the containers’ content, we rely instead on user annotations. This gives us the possibility to verify user properties expressed in the same annotation language.

7

Conclusion

We have presented a library of formal containers, a slightly modiﬁed version of the standard Ada containers. The aim was to make them usable in programs annotated with properties of interest to the user, that can be both tested and formally proved automatically. Although we have limited experience with using this library, our experiments so far indicate that most user-deﬁned annotations can now be expressed with few quantiﬁers, leading to automatic proofs of rich properties with SMT provers. We are now looking forward to working with our industrial partners in project Hi-Lite to develop large use-cases with formal containers. We have given a formal semantics for these containers, and we have proved that this semantics is consistent by implementing it in Coq. We have developed an axiomatization of these containers in the language Why, targeted at automatic proofs with SMT provers, and we have proved in Coq that this axiomatization is correct w.r.t. the formal semantics of containers. On the one hand, this formalization is an essential step towards an argument-based certiﬁcation of the library of formal containers, for their use in safety-critical software development. On the other hand, the proof of correctness of the axiomatization used in automatic provers is a very strong assurance against inconsistencies in proofs, which are a sour point of formal methods in industry. Formal containers have been implemented in Ada, and could be included in any Ada compiler’s library. They have been included in the standard library of the not yet released GNAT 6.5 Ada compiler. Theoretically, the Why axiomatization could be reused to model containers in other languages, with a few modiﬁcations to comply with the particularities of their respective APIs. The implementation we provide for Ada containers should be correct w.r.t. the formal semantics in Coq, but we have not proved it formally. This is an interesting (but diﬃcult) problem for the future. Acknowledgement. We would like to thank Ed Schonberg, David Lesens and the anonymous referees for their useful reviews of this paper.

118

C. Dross, J.-C. Filliˆ atre, and Y. Moy

References 1. The Coq Proof Assistant, http://coq.inria.fr 2. DO-178B: Software considerations in airborne systems and equipment certiﬁcation (1982) 3. http://www.ada-auth.org/standards/12rm/html/RM-TTL.html 4. http://www.open-do.org/wp-content/uploads/2011/01/main.long.pdf 5. http://www.open-do.org/projects/hi-lite/formal-containers/ 6. Blanc, N., Groce, A., Kroening, D.: Verifying C++ with STL containers via predicate abstraction. In: Proceedings of the 22nd IEEE/ACM International Conference on Automated Software Engineering, ASE 2007, pp. 521–524. ACM, USA (2007) 7. Bouillaguet, C., Kuncak, V., Wies, T., Zee, K., Rinard, M.: Using First-Order Theorem Provers in the Jahob Data Structure Veriﬁcation System. In: Cook, B., Podelski, A. (eds.) VMCAI 2007. LNCS, vol. 4349, pp. 74–88. Springer, Heidelberg (2007) 8. Conchon, S., Contejean, E.: The Alt-Ergo automatic theorem prover (2008), http://alt-ergo.lri.fr/ 9. de Moura, L., Bjørner, N.: Z3, An Eﬃcient SMT Solver, http://research.microsoft.com/projects/z3/. 10. Detlefs, D., Nelson, G., Saxe, J.B.: Simplify: a theorem prover for program checking. J. ACM 52(3), 365–473 (2005) 11. Dillig, I., Dillig, T., Aiken, A.: Precise reasoning for programs using containers. In: Proceedings of the 11th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL (January 2011) 12. Filliˆ atre, J.-C., March´e, C.: The Why/Krakatoa/Caduceus Platform for Deductive Program Veriﬁcation. In: Damm, W., Hermanns, H. (eds.) CAV 2007. LNCS, vol. 4590, pp. 173–177. Springer, Heidelberg (2007) 13. Gregor, D., Schupp, S.: STLlint: lifting static checking from languages to libraries. Softw. Pract. Exper. 36, 225–254 (2006) 14. http://www.open-do.org/projects/hi-lite/ 15. Kosiuczenko, P.: An abstract machine for the old value retrieval. In: Bolduc, C., Desharnais, J., Ktari, B. (eds.) MPC 2010. LNCS, vol. 6120, pp. 229–247. Springer, Heidelberg (2010) 16. Rushby, J.: Formalism in safety cases. In Chris Dale and Tom Anderson, editors, Making Systems Safer. In: Dale, C., Anderson, T. (eds.) Making Systems Safer: Proceedings of the Eighteenth Safety-Critical Systems Symposium, pp. 3–17. Springer, Heidelberg (2010) 17. Schwarz, J.T.: Programming with sets: an introduction to SETL. Lavoisier (October 1986) 18. Souyris, J., Wiels, V., Delmas, D., Delseny, H.: Formal Veriﬁcation of Avionics Software Products. In: Cavalcanti, A., Dams, D.R. (eds.) FM 2009. LNCS, vol. 5850, pp. 532–546. Springer, Heidelberg (2009) 19. Zee, K., Kuncak, V., Rinard, M.: Full functional veriﬁcation of linked data structures. In: Proceedings of the 2008 ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2008, pp. 349–361. ACM, New York (2008)

A Random Testing Approach Using Pushdown Automata Pierre-Cyrille Héam and Catherine Masson LIFC INRIA-CASSIS 16 route de Gray, 25030 Besançon Cedex, France [email protected]

Abstract. Developing eﬃcient and automatic testing techniques is one of the major challenges facing the software validation community. Recent work by Denise and al. (in MBT’08 proceedings) shows how to draw traces uniformly at random in large systems modeled by ﬁnite automata for testing purposes. Since ﬁnite automata are strong abstractions of systems, many generated test cases following this approach may be un-concretizable, i.e., do not correspond to any concrete execution of the system under test. In this paper, we propose to tackle this problem by extending the approach to pushdown systems that can encode either a stack data structure or the call stack. The method is based on context-free grammar algorithms, and relies on combinatorial techniques to guarantee the uniformity of generated traces.

1 1.1

Introduction General Overview

Producing secure, safe and bug-free programs is one of most challenging problems of modern computer science. In this context, two complementary approaches address this problem: veriﬁcation and testing. On one hand, veriﬁcation techniques mathematically prove that a code or a model of an application is safe. However, complexity bound makes veriﬁcation diﬃcult to apply on large-sized systems. On the other hand, testing techniques do not provide any proof but are relevant, in practice, in order to produce high quality software. Last years, many works have been done in order to upgrade hand-made (or experience-based) testing techniques to scientiﬁc based frameworks. Since every conﬁguration of a software cannot be practically explored, one of the key problem for a validation engineer is to choose a relevant test suite while controlling the number of tests. The crucial question raised is then: “what does relevant mean?”. A frequent answer, in the literature and in practice, is to consider a test suite as relevant if it fulﬁls some well-known coverage criteria; for instance, a code coverage criterion, that is satisﬁed if all the lines of the codes are executed at least once when running the tests. It is important to point out that coverage criteria can be applied on the code (white box or structural testing) or on a model of the implementation (black box or functional testing [1]). Since M. Gogolla and B. Wolﬀ (Eds.): TAP 2011, LNCS 6706, pp. 119–133, 2011. c Springer-Verlag Berlin Heidelberg 2011

120

P.-C. Héam and C. Masson

1 a 0

c 2

b d

4

f

3

e Fig. 1. Example of graph

there are many ways to fulﬁl coverage criteria [2], other criteria can be taken into account, for example based either on computing minimal/maximal length test suites, or on selecting boundary or random values for the test data. In [3] authors proposed a method for drawing paths in ﬁnite graphs uniformly and showed how to use these techniques in a control ﬂow graph based testing approach of C programs. The goal is to deﬁne probabilistic coverage criteria for testing. The method consists in proposing a combinatorial based algorithm to solve the following probabilistic problem. Random Generation of a Path in a Finite Graph Input: A ﬁnite labelled graph G, a vertex v0 , a positive integer n Output: Randomly generate a path in G of length n, starting from v0 . The generation has to be uniform relatively to all paths of length n in G. This technique developed in [3] allows an eﬃcient way to answer the above problem, even for very large graphs. Nevertheless, a ﬁnite graph often represents a strong abstraction of the system under test, and many abstract tests generated by the approach may be impossible to play on the implementation. Firstly, this paper improves the random approach proposed in [3] by extending it to pushdown graphs, that are ﬁner abstractions, particularly to encode the call stack of a program (see Sec.1.2). Secondly we will show how to eﬃciently solve the problem Random Generation of a Path in a Deterministic Normalized Pushdown Automaton (see Sec. 3) described below. Finally, the approach is experimented in Sec. 4. The precise deﬁnition of a successful path in a pushdown automaton is deﬁned in Sec. 2; informally it’s a path in the underlying automaton coherent with the stack operations. Random Generation of a Path in a Deterministic Normalized Pushdown Automaton Input: A deterministic normalized pushdown automaton A, the initial state v0 , a positive integer n Output: Randomly generate a successful path in A of length n, starting from v0 . The generation has to be uniform relatively to all successful paths of length n in A.

A Random Testing Approach Using Pushdown Automata int power(float x, int n){ int res; if (n==0) { return 1; } else { res = power(x,n/2); if (n%2==0) {

}

}

121

return res*res ; } else { return res*res*x ; }

Fig. 2. C program computing xn

1.2

Motivation

The context of this paper is to randomly generate test cases from a graph based model. In this section it is explained why classical random walks are unfair for testing purposes and why pushdown graphs are better compared to classical graphs for the concretization step. Layout of the paper is described in Sec. 1.3. Finally, Sec. 1.4 presents a short view on related work. Random Paths Generation vs Random Walks. A ﬁrst intuitive way to generate random paths is to perform a Markov-like random generation: starting from v0 , at each step the next vertice is randomly and uniformly picked in the neighborhood of the current vertices. This technique leads to an unknown distribution on paths of length n and is very sensitive to the topology of the graph. Consider for instance the labelled graph depicted in Fig. 1. Using a Markovian approach to generate a path of length 2 starting from 0: path ef occurs with probability 1/2, and paths ab and ac both with probability 1/4. The generation is not uniform. For a graph with a complex topology, severe disparities may be observed on the occurring probabilities of paths of same length. The Markovian approach is therefore unfair to ensure a nice probabilistic coverage of the graph. The technique developed in [3] to handle the Random Generation of a Path in a Graph problem requires two steps: In a ﬁrst step the number of paths of length i ≤ n between each pair of vertices is computed recursively and using combinatorial techniques. In a second step, the random path is recursively generated using the computed probabilities. Finite Automata vs. Pushdown Automata. Consider the recursive C program described in Fig. 2 and computing xn : the control ﬂow graph of this program is depicted in Fig. 3; on the left the control ﬂow graph is represented ignoring the recursive calls to the function power. On the right, calls to the function power are represented by dashed arrows labelled either by call(power) (for a call to the function power) or by return(power) for the end of a call to the function power). Since in the left graph recursive calls to power are ignored, it is impossible to compute arbitrarily long paths. It is however possible on the right graph. For instance, the run of power(3,2) corresponds to path 0 → 1 → 5 0 → 1 → 5 0 → 1 → 2 → 4 6 → 9 → 10 6 → 7 → 8

122

P.-C. Héam and C. Masson

The ﬁrst 5 0 corresponds to the call to power(3,1). The second 5 0 corresponds to the call to power(3,0). The transition 4 6 closes the call to power(3,0) and the transition 10 6 closes the call to power(3,1). call(power) int res;

0

1

int res;

0

2

5

4

2

6

4

9

return res*res; return res*res*x; 8

return(power)

n%2 ==0

n%2 !=0

7

5

return 1;

return 1; res=power(x,n/2);

n%2 ==0

n!=0

n==0

n!=0

n==0

1

6

n%2 !=0

7 return(power)

9

return(power)

return res*res; return res*res*x;

10

8

10

Fig. 3. Example of a control ﬂow graph Automatic Tool

Program

Direct Translation

Control Flow Graph

Section 3.1

NPA

Section 3.2

Context-free Grammar

Execution Trees

Direct translation Concretization Abstract Tests

Fig. 4. Random Generation Process

Now a stands for int res;, b for n==0, c for n!=0, d for call(power), e for return 1;, f for return(power), g for n%2==0, h for n%2!=0, i for the instruction return res*res; and j for return res*res*x;. Words accepted by the automaton depicted on the right of Fig. 3 (see also Fig. 5), are those in [(acd)∗ abef (gif + hjf )(gif + hjf )∗ (gi + hj)] + (acd)∗ abe (if considered paths are from 0 to a ﬁnal state). Moreover, accepted words corresponding to correct occurrences1 of call(power)’s and return(power)’s are those of the form 1

It means that the word is coherent with the calls to function power, like a well braced expression.

A Random Testing Approach Using Pushdown Automata

123

[(acd)k abef (gif + hjf )(gif + hjf )k−1 (gi + hj)], where k ≥ 1. For instance, path 0 → 1 → 5 0 → 1 → 2 → 4, corresponding to acdabe is a successful path in the automaton but cannot be concretized since the call to function power (letter d) will never be returned. Therefore, the probability that a path of length n in the automaton corresponds to a correct path w.r.t. calls/returns of function power converges to 0 while n grows to +∞. It follows that the approach consisting in generating paths uniformly at random in the ﬁnite automaton will most of time lead to un-concretizable tests. Moreover, it rules out the possibility to use a rejection approach to generate long test cases. 1.3

Layout of the Approach

In order to generate abstract tests from a program, our approach consists in (1) generating its control ﬂow graph, (2) translating its graph ﬂow into a normalized pushdown automata, (3) transforming this NPA into an equivalent nonambiguous context-free grammar, (4) uniformly generating execution trees of the grammar and (5) translating back these trees into paths in the program. This approach is depicted in Fig. 4. Note that automatic tools exist for Step (1) for a variety of used programming languages. Step (2) only consists in transforming function calls into stack operations, and can easily be automatized. Note that the concretization step requires the use of speciﬁc tools, for instance constraint solvers. 1.4

Related Work

Graph-based Testing. Testing systems from ﬁnite state machines representing models of the system [4,1] (model based testing) consists in describing the system by a labeled transition system on which diﬀerent algorithms may be used to extract the test cases. This is, for instance, the principle of SpecExplorer [5] or TGV [6]. Testing from control-ﬂow graphs (structural testing) is one of the major testing approaches developed in hundreds of articles. Interested reader is referred to [7] or to [8] for more information. Grammar-based Testing. The approach presented in [9] addresses parser testing and [10] focuses on testing re-factoring engines (program transformation software). A systematic generation for grammar based inputs is proposed in [11]. However because of the explosion of tests, symbolic approaches are preferred as in [12,13,14]. Recently, in [15], a generic tool for test generation from grammars is proposed. This tool does not provide any random feature but is based on rule coverage approaches/algorithms as deﬁned in [9,16,17,18]. Random Testing. Random based approaches for testing were initially proposed in [19,20]. Random testing can be employed for generating test data, such as in DART [21] or to generate complete test sequences, as in the Jartege tool [22]. The recent work [3] provides an approach combining random-testing and modelchecking and is discussed deeper in this paper. Approaches [23,24,25,26] propose random and grammar-based techniques for testing: in this context, grammar

124

P.-C. Héam and C. Masson

are used to specify input data-structures. Note that such algorithms were used in [14] for testing in a white-box fuzzing context. In [27] an approach combining model-based testing and randomness is presented.

2

Formal Background

If X is a ﬁnite set, X ∗ denotes respectively the set of ﬁnites words over X. The empty word (on every alphabet) is denoted ε. We denote by X + the set X ∗ \{ε}. 2.1

Pushdown Automata

A deterministic finite automaton is a tuple (Q, Σ, δ, qinit, F ) where Q is a ﬁnite set of states, Σ is a ﬁnite alphabet, qinit ∈ Q is the initial state, F is the set of ﬁnal states and δ is a partial function from Q × Σ into Q. A successful path in a ﬁnite automaton is a (possibly empty) ﬁnite sequence of elements of Q × Σ × Q of the form (p1 , a1 , q1 ) . . . (pn , an , qn ) such that p1 = qinit , qn ∈ F and for each i, qi = pi+1 and δ(pi , a) = qi . The integer n is the length of the path and a1 . . . an is its label. A pushdown automaton2 is a tuple A = (Q, Σ, Γ, δ, qinit, F ) where Q is a ﬁnite set of states, Σ and Γ are disjoint ﬁnite alphabets satisfying ε ∈ / Σ and ⊥ ∈ Γ , qinit ∈ Q is the initial state, F is the set of ﬁnal states and δ is a partial function from Q × (Σ ∪ {ε}) × Γ into Q × Γ ∗ such that for every q ∈ Q, for every X ∈ Γ , a ∈ Σ ∪ {ε}, (1) if δ(q, a, X) = (p, w) then w ∈ (Γ \ {⊥})∗ , and (2) if δ(q, a, ⊥) = (p, w), then the ﬁrst letter of w is ⊥. Letter ⊥ is called the empty stack letter. A configuration of a pushdown automaton is an element of Q × {⊥}(Γ \ {⊥})∗ that is a pair whose ﬁrst element is in Q and the second is a word starting by ⊥ and whose others letters are not ⊥. Informally, the second part encodes the current value of the stack. The initial configuration is (qinit , ⊥). Two conﬁgurations (q, ⊥u) and (p, ⊥v) are a-consecutive, with a ∈ Σ ∪{ε}, either if u = ε and δ(q, a, ⊥) = ⊥v, or if u is of the form u0 X with X ∈ Γ \ {⊥} and there exists w ∈ Γ ∗ such that δ(q, a, X) = (p, w) and v = u0 w. A successful execution of length n in a pushdown automaton is a sequence C1 a1 C2 a2 . . . Cn an Cn+1 where the Ci ’s are conﬁgurations, the ai ’s are in Σ ∪{ε} and such that C1 is the initial conﬁguration, for each i, Ci and Ci+1 are ai consecutive, and Cn+1 is of the form (p, ⊥) with p ∈ F . A normalized pushdown automaton (NPA) is a pushdown automaton such that for every state q, every a ∈ Σ ∪ {ε}, every X ∈ Γ , if δ(q, a, X) = (p, w) then one of the following case arises: (i) a = ε and w = ε, or (ii) a = ε and w is of the form w = XY with Y ∈ Γ , or (iii) a = ε and w = X. 2

In this paper we use a restricted deﬁnition of pushdown automaton.

A Random Testing Approach Using Pushdown Automata

125

pop(S)

10

j

push(S) 9

h

6

5 c

pop(S)

g

1

pop(S) b

8

i

7

4

e

a 2

0

Fig. 5. Example of NPA

Intuitively, case (i) corresponds to pop on the stack, case (ii) to push and case (iii) to an action that does not modify the stack. The underlying finite automaton of an NPA A = (Q, Σ, Γ, δ, qinit, F ) is the ﬁnite automaton (Q, Σ ∪ {pop(X), push(X) | X ∈ Γ }, μ, qinit, F ) where μ is deﬁned by (i) δ(q, ε, X) = (p, ε) iﬀ μ(q, pop(X)) = p, (ii) δ(q, ε, X) = (p, XY ) iﬀ μ(q, push(Y)) = p, (iii) δ(q, a, X) = (p, X) iﬀ μ(q, a) = p. The above deﬁnitions can be illustrated with the NPA depicted in Fig. 5, corresponding to the graph on the right of Fig. 3. For this automaton Q = {0, . . . , 10}, Σ = {a, b, c, d, e, g, h, i, j}, Γ = {⊥, S}, qinit = 0, F = {4, 8, 10} and δ is deﬁned by the arrows: if there is an edge of the form (q, a, p) with a ∈ Σ, then one has δ(q, a, ⊥) = (p, ⊥) and δ(a, ⊥) = (p, S); if there is an edge of the form (q, pop(S), p) then δ(q, ε, S) = (p, ε); if there is an edge of the form (q, push(S), p) then δ(q, ε, S) = (p, SS) and δ(q, ε, ⊥) = (p, ⊥S). The sequence (0, ⊥)a(1, ⊥)c(5, ⊥)ε(0, ⊥S)a(1, ⊥S)c(5, ⊥)ε(0, ⊥SS)a(1, ⊥SS)b(2, ⊥SS) e(4, ⊥SS)ε(6, ⊥S)h(9, ⊥S)j(10, ⊥S)ε(6, ⊥)g(7, ⊥)i(8, ⊥) is a successful execution in the NPA. This path corresponds to the execution of power(3,2), already pointed out in Sec. 1.2. Note that each pair (q, w) encodes that the system is in state q and that the call stack is w. For instance, (2, ⊥SS) means that system is in state 2 and that there are two no-closed calls to the function power. Each successful execution of an NPA is associated to a successful path in the underlying automaton using the natural projection: the successful execution C1 a1 . . . an Cn is associated to the path (q1 , a1 , q2 ) . . . (qn−1 , an , qn ), where the Ci ’s are of the form (qi , wi ). For instance, the successful path associated to the above successful execution is (0, a, 1)(1, c, 5)(5, ε, 0) . . . (6, g, 7)(7, i, 8). This projection is denoted proj and is injective. Its image forms a subset of successful paths in the underlying automaton, which is in bijection with the set of successful executions and whose elements are called successful traces of the underlying automaton.

126

P.-C. Héam and C. Masson

2.2

Context-Free Grammars

A Context-free grammar is a tuple G = (Σ, Γ, S0 , R) where Σ and Γ are disjoint ﬁnite alphabets, S0 ∈ Γ is the initial symbol and R is a ﬁnite subset of Γ × (Σ ∪ Γ )∗ . An element of R is called a rule of the grammar. A word w ∈ (Σ ∪ Γ )∗ is a successor of v ∈ (Σ ∪ Γ )∗ for the grammar G if there exist v0 ∈ Σ ∗ , v1 , v2 ∈ (Σ ∪ Γ )∗ , S ∈ Γ such that v = v0 Sv1 and w = v0 v2 v1 and (S, v2 ) ∈ R. A complete derivation of the grammar G is a ﬁnite sequence x0 , . . . , xn of words of (Σ ∪ Γ )∗ such that x0 = S0 , xn ∈ Σ ∗ and for every i, xi+1 is a successor of xi . A derivation tree of G is a ﬁnite tree whose internal nodes are labeled by letters of Γ , whose leaves are labelled by elements of Σ ∪ {ε}, whose root is labelled by S0 and satisfying: if a node is labeled by X ∈ Γ and if its children are labelled by α1 , . . . , αk (in this order), then either α1 = ε and n = 1, or all the αi ’s are in Γ ∪ Σ and (X, α1 . . . αk ) ∈ R. Consider for instance the grammar G = ({a, b}, {S, T }, S, R), with R = {(S, T b), (S, aSb), (T, ε)}). The sequence S, aSb, aT bb, abb is a complete derivation of the grammar. The associated derivation tree is a S

S b

b T

ε

Note that there is a natural bijection between the set of complete derivations of a grammar and the set of derivation trees of this grammar.

3

Uniform Random Generation of Successful Paths

Given a NPA, remind that the goal is to uniformly generate successful traces of a given length in its underlying automaton. In Sec. 3.1 a well-known connection between NPA and context-free grammars is recalled. Uniform random generation of successful traces is then explained in Sec. 3.2. 3.1

NPA to Context Free Grammars

Transforming a NPA into a context-free grammar can be done using classical algorithms on pushdown automata; see, for instance, [28]. The following result is a direct combination of well-known results on pushdown automata. Theorem 1. Let A be a pushdown automaton. One can compute in polynomial time a grammar G satisfying the following assertions: – The size of G is at most quadratic in the size of A and there is no rule of the form (X, Y ) in G, where X and Y are stack symbols. – There exists a bijection ϕ from the set of complete derivations of G and the set of successful executions of A. – Given a complete derivation of G, its image by ϕ can be computed in polynomial time.

A Random Testing Approach Using Pushdown Automata

127

Note that the precise complexity of algorithms depends on the chosen datastructures, but all of them can be implemented in a very eﬃcient way. Consider for instance the underlying automaton depicted in Fig. 5. Successful traces of A are generated by the grammar (ΣG , ΓG , X0 , RG ) where ΓG = {X0 , X1 , T }, ΣG ={(0, a, 1), (1, c, 5), (5, push(S), 0), (1, b, 2), (2, ε, 4), (4, pop(S), 6), (6, g, 7), (7, i, 8), (8, pop(S), 6), (6, h, 9), (9, j, 10), (10, pop(S), 6)}, and RG ={(X0 , (0, a, 1)(1, c, 5)(5, push(S), 0)X1 T (6, h, 9)(9, j, 10)), (X0 , (0, a, 1)(1, c, 5)(5, push(S), 0)X1 T (6, g, 7), (7, i, 8)), (X0 , (0, a, 1)(1, b, 2)(2, ε, 4)), (X0 , (0, a, 1)(1, c, 5)(5, push(S), 0)(0, a, 1)(1, b, 2)(2, ε, 4)(4, pop(S), 6)), (X1 , (0, a, 1)(1, c, 5)(5, push(S), 0)X1 T ), (X1 , (0, a, 1)(1, c, 5)(5, push(S), 0)(0, a, 1)(1, b, 2)(2, ε, 4)(4, pop(S), 6)), (T, (6, g, 7), (7, i, 8)(8, pop(S), 6)), (T, (6, h, 9)(9, j, 10)), (10, pop(S), 6))}. 3.2

Random Generation of Successful Traces

The random generation of successful traces of a NPA is performed using Theorem 1. First, the related grammar G is computed. Next, random derivation trees are randomly generated: successful traces are computed using ϕ. Since ϕ is bijective, if the random generation of derivation trees is uniform, so will be the random generation of successful traces (using the fact that proj is bijective.) The general scheme of the generation process is sketched in Fig. 6.

NPA

Thm. 1

Algo. Rand. Gen. Grammar

Derivation Trees bijection

proj (bijection) Succ. Traces

ϕ Succ. Executions

Comp. deri.

Fig. 6. Random Generation of Successful Traces

The random generation of derivation trees is performed using classical combinatorial techniques [29] that are sketched below. Let G = (Σ, Γ, X, R) be a context-free grammar satisfying the conditions of Theorem 1. For each symbol S ∈ Γ , we introduce the sequence of positive integers s(1), . . . , s(k), . . ., where s(i) is the number of size i derivation tree of (Σ, Γ, S, R). These s(i)’s are

128

P.-C. Héam and C. Masson

computed recursively as follows. For each strictly positive integer k and each rule r = (S, w1 S1 . . . wn Sn wn+1 ) ∈ R, with wj ∈ Σ ∗ and Si ∈ Γ , let ⎧ βr = 1 + n+1 ⎪ i=1 |wi | ⎪ ⎪ j=n ⎨α (k) = r j=1 sj (ij ) if n = 0 i1 +i2 +...+in =k−βr ⎪αr (k) = 0 if n = 0 and k = βr ⎪ ⎪ ⎩ αr (βr ) = 1 if n = 0. It is known [29, Theorem I.1] that s(k) = r∈R∩(S×(Σ∪Γ )∗ ) αr (k). Since, by hypotheses, there is no rule of the form (S, T ) in R, with S, T ∈ Γ , all ij ’s involved in the deﬁnition of βr are strictly less than k. This way, the s(i)’s can be recursively computed. Consider for instance the grammar ({a, b}, {X}, X, {r1, r2 , r3 }) with r1 = (X, XX) r2 = (X, a) and r3 = (X, b). One hasβr1 = 1 + 0 = 1, βr2 = 1 + 1 = 2, βr3 = 1 + 1 = 2. Therefore x(k) = i+j=k−1 x(i)x(j) if k = 2 and x(2) = 1 + 1 + i+j=2−1 x(i)x(j) = 2 otherwise. It follows that x(1) = 0, x(2) = 2, x(3) = x(1)x(1) = 0, x(4) = x(1)x(2) + x(2)x(1) = 0, x(5) = x(2)x(2) = 4, etc. The two derivation trees of size 2 are X /\ Z1 Z2

X | a

and

X |. b

The four derivation trees of size 4 are the trees of the form

where Z1 and Z2 are both ones of the size 2 derivation trees.

In order to generate derivation trees of size n, all s(i) s, for S ∈ Γ and i ≤ n have to be computed with the above methods. This can be performed in time O(n2 ). Next the random generation is done recursively using the algorithm of Fig. 7. It is known [29] that this algorithm provides a uniform generation of derivation trees of size n, i.e. each derivation tree occurs with the same probability. Note that an exception is returned at Step 2 if there are no elements of the given size: on the example presented above, there are no elements of size 4, then it is impossible to generate one. Running the algorithm on this example with n = 2, one consider at Step 1 the set {r1 , r2 , r3 } since all these rules have X as left element. Since αr1 (2) = 0, αr2 (2) = 1, αr3 (2) = 1, at Step 3 the probability that i = 1 is null, the probability that i = 2 is 1/2 and the probability that i = 3 is 1/2. If i = 2 is picked, then the generated tree has X as root symbol and a as unique child. Running the algorithm on this example with n = 3 will stop at Step 2 since there is no tree of size 3. Running the algorithm on this example with n = 5, the set {r1 , r2 , r3 } is considered at Step 1. Since αr1 (5) = 4, αr2 (5) = 0, αr3 (5) = 0, i = 1 is picked with probability 1. Therefore, the tree has X as root symbol, and its two children are both labelled by X. Therefore, at Step 7, the considered set is {1, 2}. At Step 8, one has n − βr1 = 5 − 1 = 4. The probability that i1 = 1 and i2 = 3 is null since x(1) = 0. Similarly the probability that i1 = 3 and i2 = 1 is null too. Now the probability that i1 = 2 and i2 = 2 is 1. Now the algorithm is recursively called on each child with n = 2: each of the 4 trees is picked with probability 1/4. From a NDPA with n states, the generation of k paths of length n can be performed in time O(n4 + n2 k log(n)).

A Random Testing Approach Using Pushdown Automata

129

Random Generation Input: G = (Σ, Γ, X, R) a context-free grammar, n a strictly positive integer. Output: a derivation tree t of G size n. Algorithm: 1. Let{r1 , r2 , . . . , r } be set of the elements of R whose ﬁrst element is X. 2. If j= j=1 αrj (n) = 0, then return “Exception”. 3. Pick i ∈ {1, . . . , } with probability P rob(i = j) = 4. 5. 6. 7. 8.

αri (n) . j= j=1 αrj (n)

Let ri = (X, Z1 . . . Zk ), with Zj ∈ Σ ∪ Γ . Root symbol of t is X. Children of t are Z1 , . . . , Zk in this order. Let {i1 , . . . , im } = {j | Zj ∈ Γ }. Pick (x1 , . . . , xm ) ∈ Nm such that x1 + . . . + xm = n − βri with probability j=m j=1 zij (j ) P rob(x1 = 1 , . . . , xm = m ) = . αri (n)

9. For each ij , the ij -th sub-tree of T is obtained running the Random Generation algorithm on (Σ, Γ, Zij , R) and j . 10. Return t. Fig. 7. The trandom generation algorithm

4

Experimentation

The experimentation has been performed on the running example proposed in Section 4.1. The transformation into NPA was done by hand, as the translation into a context-free grammar, but there is no theoretical problem to automate these steps. Random generation of execution trees was done using GenRgenS [30]. 4.1

Running Example

We illustrate our approach on the program of Fig. 8, using two functions S and M. int S(int x, int y, int n){ int z; if (y == 1){ z = M(x,n); return z; } else { z = x + S(x,y-1,n); z = M(z,n); return z; } }

int M(int x, int n){ if (x < 1){ return x; } else { return M(x-n,n); } }

Fig. 8. Mutually Recursive Functions

130

P.-C. Héam and C. Masson

FunctionM(x,n) computes x%n, and S(x,y,n) computes (x ∗ y)%n. Our objective is to test the function S using the random approach developed in this paper. The NPA associated to this pair of functions is depicted in Fig. 9. The top part describes the function S and the bottom one describes the function M. The recursive calls to M in S are encoded by the stack symbols M1 (for the case y==1) and M2 (for the case y!=1). The stack symbol M3 encodes the call to M in S. push(S1 ) 1

int z

3

y!=1

8

z=x+S()

2

9

10

z=M()

11 return z

pop(S1 ) y==1

4

5

z=M()

6

return z

7

pop(S1 )

12

pop(M1 ) push(M1 ) push(M2 ) I

x
II

x>=n

return x

pop(M2 )

III

pop(M2 )

pop(M3 )

pop(M1 )

push(M3 ) IV

V

return M;

VI

pop(M3 )

Fig. 9. NPA of the running example

One can compute a related grammar satisfying the properties of Theorem 1. It has 24 stack symbols and 31 rules. 4.2

Experimental Results

From the grammar, the random generation of the execution trees is performed using GenRgenS [30]. In order to compare our approach with [3], we also implement the related random generation approach. Results are reported in Fig. 10. The ﬁrst column is the length of generated traces/paths in the underlying automaton of Fig 9. For each size, we generate 10 traces-paths. The second column reports the generated successful traces: we only give the sequence of states and the number in brackets is the number of times this path was randomly generated. The last column is similar for successful paths. One can ﬁrst observe that there is a unique successful trace of length 8: 12-4-I-II-III-5-6-7, which is obviously generated at each time. Note this trace corresponds to the execution of S(3,1,2). Conversely, there are two successful paths of length 8: the ﬁrst one corresponding to the successful trace and 1-2-4I-II-III-10-11-12, which is not coherent with stack calls (it does not correspond

A Random Testing Approach Using Pushdown Automata

131

size of the paths/traces Our approach from 1-7 no trace of these lengths 8 9 10

11

Approach of [3] no path of these lengths 1-2-4-I-II-III-10-11-12 (6) 1-2-4-I-II-III-5-6-7 (10) 1-2-4-I-II-III-5-6-7 (4) no trace of this length no path of this length 1-2-4-I-IV-I-II-III-5-6-7 (3) 1-2-4-I-IV-I-II-III-10-11-12 (2) 1-2-4-I-IV-I-II-III-5-6-7 (10) 1-2-4-I-II-III-V-VI-10-11-12 (2) 1-2-4-I-II-III-V-VI-5-6-7 (2) A-2-3-1-2-4-I-II-III-10-11-12 (6) no trace of this length 1-2-31-2-4-I-II-III-5-6-7 (4)

Fig. 10. Qualitative experimental results

to any successful trace). Therefore, this path cannot be concretized. The same situation occurs for generation of size 10: there is only one successful traces but four successful paths. For size 11, there are two successful paths but none can be concretized. These experimental results show that for paths of length 10, 70% of traces generated by [3] would be un-concretizable, and 100% for paths of length 11. The results were obtained in an unobservable time.

5

Conclusion

In this paper an extension of the approach developed in [3] is proposed. It is shown how to uniformly generate traces in a control-ﬂow graph respecting call stack constraints. This generation is not as eﬃcient as [3] but is still polynomial and can be applied on large graphs. The two main challenges while testing software is to concretize the test cases (concretization) and to know whether the tests success or fail (the oracle problem). Our approach is fruitful for the ﬁrst problem, providing more coherent tests. However, the concretization problem is still diﬃcult and the combination with the use of constraint solvers have to be investigated. In the future, we plan to apply the proposed technique on larger examples. It supposes to use advanced techniques for the random generation, as Botlzmann samples, ﬂoating point arithmetics [31] or dedicated works [24,32]. Moreover, we would also like to use the same technique in a model-based testing context: the pushdown automaton is not a control ﬂow graph but an abstraction of the system under test. For instance, models of go-back functions based programs can be used, as in [33]. The stack would be either a data-structure involved in the system or the abstraction of a data-structure. Similarly, generated traces would be more frequently concretizable. Another perspective is to adapt the framework to randomly generate tests under a distribution given by statistical information on the systems (statistical testing). Acknowledgement.Authors would like to thank Frédéric Dadeau and Fabien Peureux for helpful comments. This work was partially granted by the french ANR project FREC.

132

P.-C. Héam and C. Masson

References 1. Beizer, B.: Black-Box Testing: Techniques for Functional Testing of Software and Systems. John Wiley & Sons, New York (1995) 2. Oﬀutt, A., Xiong, Y., Liu, S.: Criteria for Generating Speciﬁcation-Based Tests. In: 5th International Conference on Engineering of Complex Computer Systems (ICECCS 1999), p. 119. IEEE Computer Society, Las Vegas (1999) 3. Groce, A., Joshi, R.: Random testing and model checking: building a common framework for nondeterministic exploration. In: WODA 2008, pp. 22–28. ACM, New York (2008) 4. Lee, D., Yannakakis, M.: Principles and methods of testing ﬁnite state machines a survey. Proceedings of the IEEE, 1090–1123 (1996) 5. Campbell, C., Grieskamp, W., Nachmanson, L., Schulte, W., Tillmann, N., Veanes, M.: Testing concurrent object-oriented systems with spec explorer. In: Fitzgerald, J.S., Hayes, I.J., Tarlecki, A. (eds.) FM 2005. LNCS, vol. 3582, pp. 542–547. Springer, Heidelberg (2005) 6. Jard, C., Jéron, T.: TGV: theory, principles and algorithms, a tool for the automatic synthesis of conformance test cases for non-deterministic reactive systems. Software Tools for Technology Transfer (STTT) 6 (2004) 7. Gotlieb, A., Botella, B., Rueher, M.: Automatic test data generation using constraint solving techniques. In: ISSTA, pp. 53–62 (1998) 8. Ammann, P., Oﬀutt, J.: Introduction to Software Testing. Cambridge University Press, Cambridge (2008) 9. Purdom, P.: A sentence generator for testing parsers. BIT 12, 366–375 (1972) 10. Daniel, B., Dig, D., Garcia, K., Marinov, D.: Automated testing of refactoring engines. In: ESEC/FSE 2007: Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering. ACM Press, New York (2007) 11. Coppit, D., Lian, J.: Yagg: an easy-to-use generator for structured test inputs. In: Redmiles, D.F., Ellman, T., Zisman, A. (eds.) ASE, pp. 356–359. ACM, New York (2005) 12. Lämmel, R., Schulte, W.: Controllable combinatorial coverage in grammar-based testing. In: Uyar, M., Duale, A., Fecko, M. (eds.) TestCom 2006. LNCS, vol. 3964, pp. 19–38. Springer, Heidelberg (2006) 13. Majumdar, R., Xu, R.G.: Directed test generation using symbolic grammars. In: Stirewalt, R.E.K., Egyed, A. (eds.) ASE, pp. 134–143. ACM, New York (2007) 14. Godefroid, P., Kiezun, A., Levin, M.: Grammar-based whitebox fuzzing. In: Gupta, R., Amarasinghe, S.P. (eds.) PLDI, pp. 206–215. ACM, New York (2008) 15. Xu, Z., Zheng, L., Chen, H.: A toolkit for generating sentences from context-free grammars. In: Software Engineering and Formal Methods, pp. 118–122. IEEE, Los Alamitos (2010) 16. Lämmel, R.: Grammar testing. In: Hußmann, H. (ed.) FASE 2001. LNCS, vol. 2029, pp. 201–216. Springer, Heidelberg (2001) 17. Zheng, L., Wu, D.: A sentence generation algorithm for testing grammars. In: Ahamed, S., Bertino, E., Chang, C., Getov, V., Liu, L., Ming, H., Subramanyan, R. (eds.) COMPSAC, vol. (1), pp. 130–135. IEEE Computer Society, Los Alamitos (2009) 18. Alves, T., Visser, J.: A case study in grammar engineering. In: Gašević, D., Lämmel, R., Van Wyk, E. (eds.) SLE 2008. LNCS, vol. 5452, pp. 285–304. Springer, Heidelberg (2009)

A Random Testing Approach Using Pushdown Automata

133

19. Duran, J., Ntafos, S.: A report on random testing. In: ICSE 1981: Proceedings of the 5th International Conference on Software Engineering, pp. 179–183. IEEE Press, Piscataway (1981) 20. Hamlet, R.: Random testing. In: Encyclopedia of Software Engineering, pp. 970– 978. Wiley, Chichester (1994) 21. Godefroid, P., Klarlund, N., Sen, K.: DART: directed automated random testing. In: PLDI 2005: Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 213–223. ACM, New York (2005) 22. Oriat, C.: Jartege: A tool for random generation of unit tests for java classes. In: Reussner, R., Mayer, J., Staﬀord, J.A., Overhage, S., Becker, S., Schroeder, P.J. (eds.) QoSA 2005 and SOQUA 2005. LNCS, vol. 3712, pp. 242–256. Springer, Heidelberg (2005) 23. McKenzie, B.: Generating string at random from a context-free grammar. Technical Report TR-COSC 10/97, Univ ersity of Canterbury (1997) 24. Hickey, T.J., Cohen, J.: Uniform random generation of strings in a context-free language. SIAM J. Comput. 12, 645–655 (1983) 25. Maurer, P.: The design and implementation of a grammar-based data generator. Softw. Pract. Exper. 22, 223–244 (1992) 26. Heam, P.C., Nicaud, C.: Seed: an easy to use random generator of recursive data structures for testing. In: ICST 2011. IEEE, Los Alamitos (to appear, 2011) 27. Dadeau, F., Levrey, J., Héam, P.C.: On the use of uniform random generation of automata for testing. Electr. Notes Theor. Comput. Sci. 253, 37–51 (2009) 28. Sipser, M.: 2. In: Introduction to the Theory of Computation. PWS (1996) 29. Flajolet, P., Sedgewick, R.: Analytic Combinatorics. Cambridge University Press, Cambridge (2008) 30. Ponty, Y., Termier, M., Denise, A.: Genrgens: Software for generating random genomic sequences and structures. Bioinformatics 22, 1534–1535 (2006) 31. Denise, A., Zimmermann, P.: Uniform random generation of decomposable structures using ﬂoating-point arithmetic. Theor. Comput. Sci. 218, 233–248 (1999) 32. Goldwurm, M.: Random generation of words in an algebraic language in linear binary space. Inf. Process. Lett. 54, 229–233 (1995) 33. Belli, F., Beyazit, M., Takagi, T., Furukawa, Z.: Testing of "go-back" functions based on pushdown automata. In: ICST 2011. IEEE, Los Alamitos (to appear, 2011)

Incremental Learning-Based Testing for Reactive Systems Karl Meinke and Muddassar A. Sindhu School of Computer Science and Communication, Royal Institute of Technology, 100-44 Stockholm, Sweden [email protected], [email protected]

Abstract. We show how the paradigm of learning-based testing (LBT) can be applied to automate speciﬁcation-based black-box testing of reactive systems. Since reactive systems can be modeled as Kripke structures, we introduce an eﬃcient incremental learning algorithm IKL for such structures. We show how an implementation of this algorithm combined with an eﬃcient model checker such as NuSMV yields an effective learning-based testing architecture for automated test case generation (ATCG), execution and evaluation, starting from temporal logic requirements.

1

Introduction

A heuristic approach to automated test case generation (ATCG) from formal requirements speciﬁcations known as learning-based testing (LBT) was introduced in Meinke [9] and Meinke and Niu [11]. Learning-based testing is an iterative approach to automate speciﬁcation-based black-box testing. It encompasses both test case generation, execution and evaluation (the oracle step). The aim of this approach is to automatically generate a large number of high-quality test cases by combining a model checking algorithm with an optimised model inference algorithm. For procedural programs, [11] has shown that LBT can signiﬁcantly outperform random testing in the speed with which it ﬁnds errors in a system under test (SUT). In this paper we consider how the LBT approach can be applied to a quite different class of SUTs, namely reactive systems. Conventionally, reactive systems are modeled as Kripke structures and their requirements are usually speciﬁed using a temporal logic (see e.g. [6]). To learn and test such models eﬃciently, we therefore introduce a new learning algorithm IKL (Incremental Kripke Learning) for Kripke structures. We show that combining the IKL algorithm for model inference together with an eﬃcient temporal logic model checker such as NuSMV yields an eﬀective LBT architecture for reactive systems. We evaluate the eﬀectiveness of this testing architecture by means of case studies. In the remainder of Section 1 we discuss the general paradigm of LBT, and speciﬁc requirements on learning. In Section 2 we review some essential mathematical preliminaries. In Section 3, we consider the technique of bit-sliced learning M. Gogolla and B. Wolﬀ (Eds.): TAP 2011, LNCS 6706, pp. 134–151, 2011. c Springer-Verlag Berlin Heidelberg 2011

Incremental Learning-Based Testing for Reactive Systems

135

of Kripke structures. In Section 4, we present a new incremental learning algorithm IKL for Kripke structures that uses distinguishing sequences, bit-slicing, and lazy partition reﬁnement. In Section 5 we present a complete LBT architecture for reactive systems testing. We evaluate this architecture by means of case studies in Section 6. Finally, in Section 7 we draw some conclusions. 1.1

Learning-Based Testing

Several previous works, (for example Peled et al. [16], Groce et al. [8] and Raﬀelt et al. [17]) have considered a combination of learning and model checking to achieve testing and/or formal veriﬁcation of reactive systems. Within the model checking community the veriﬁcation approach known as counterexample guided abstraction reﬁnement (CEGAR) also combines learning and model checking, (see e.g. Clarke et al. [5]). The LBT approach can be distinguished from these other approaches by: (i) an emphasis on testing rather than veriﬁcation, and (ii) use of incremental learning algorithms speciﬁcally chosen to make testing more eﬀective and scalable (c.f. Section 1.2). The basic LBT paradigm requires three components: (1) a (black-box) system under test (SUT) S, (2) a formal requirements speciﬁcation Req for S, and (3) a learned model M of S. Now (1) and (2) are common to all speciﬁcation-based testing, and it is really (3) that is distinctive. Learning-based approaches are heuristic iterative methods to automatically generate a sequence of test cases. The heuristic approach is based on learning a black-box system using tests as queries. An LBT algorithm iterates the following four steps: (Step 1) Suppose that n test case inputs i1 , . . . , in have been executed on S yielding the system outputs o1 , . . . , on . The n input/output pairs (i1 , o1 ), . . . , (in , on ) are synthesized into a learned model Mn of S using an incremental learning algorithm (see Section 1.2). This step involves generalization from the given data, (which represents an incomplete description of S) to all possible data. It gives the possibility to predict previously unseen errors in S during Step 2. (Step 2) The system requirements Req are satisﬁability checked against the learned model Mn derived in Step 1 (aka. model checking). This process searches for a counterexample in+1 to the requirements. (Step 3) The counterexample in+1 is executed as the next test case on S, and if S terminates then the output on+1 is obtained. If S fails this test case (i.e. the pair (in+1 , on+1 ) does not satisfy Req) then in+1 was a true negative and we proceed to Step 4. Otherwise S passes the test case in+1 so the model Mn was inaccurate, and in+1 was a false negative. In this latter case, the eﬀort of executing S on in+1 is not wasted. We return to Step 1 and apply the learning algorithm once again to n + 1 pairs (i1 , o1 ), . . . , (in+1 , on+1 ) to infer a reﬁned model Mn+1 of S. (Step 4) We terminate with a true negative test case (in+1 , on+1 ) for S.

136

K. Meinke and M.A. Sindhu

Thus an LBT algorithm iterates Steps 1 . . . 3 until an SUT error is found (Step 4) or execution is terminated. Possible criteria for termination include a bound on the maximum testing time, or a bound on the maximum number of test cases to be executed. This iterative approach to TCG yields a sequence of increasingly accurate models M0 , M1 , M2 , . . ., of S. (We can take M0 to be a minimal or even empty model.) So, with increasing values of n, it becomes more and more likely that satisﬁability checking in Step 2 will produce a true negative if one exists. If Step 2 does not produce any counterexamples at all then to proceed with the iteration, we must construct the next test case in+1 by some other method, e.g. randomly. 1.2

Eﬃcient Learning Algorithms

As has already been suggested in Step 1 of Section 1.1, for LBT to be eﬀective at ﬁnding errors, it is important to use the right kind of learning algorithm. A good learning algorithm should maximise the opportunity of the satisﬁability algorithm in Step 2 to ﬁnd a true counterexample in+1 to the requirements Req as soon as possible. An automata learning algorithm L is said to be incremental if it can produce a sequence of hypothesis automata A0 , A1 , . . . which are approximations to an unknown automata A, based on a sequence of information (queries and results) about A. The sequence A0 , A1 , . . . must ﬁnitely converge to A, at least up to behavioural equivalence. In addition, the computation of each new approximation Ai+1 by L should reuse as much information as possible about the previous approximation Ai (e.g. equivalences between states). Incremental learning algorithms are necessary for eﬃcient learning-based testing of reactive systems for two reasons. (1) Real reactive systems may be too big to be completely learned and tested within a feasible timescale. This is due to the typical complexity properties of learning and satisﬁability algorithms. (2) Testing of speciﬁc requirements such as use cases may not require learning and analysis of the entire reactive system, but only of a fragment that implements the requirement Req. For testing eﬃciency, we also need to consider the type of queries used during learning. The overhead of SUT execution to answer a membership query during learning can be large compared with the execution time of the learning algorithm itself (see e.g. [3]). So membership queries should be seen as “expensive”. Therefore, as many queries (i.e. test cases) as possible should be derived from model checking the hypothesis automaton, since these are all based on checking the requirements Req. Conversely as few queries as possible should be derived for reasons of internal book-keeping by the learning algorithm (e.g. for achieving congruence closure prior to automaton construction). Book-keeping queries make no reference to the requirements Req, and therefore can only uncover an SUT error by accident. Ideally, every query would represent a relevant and interesting requirements-based test case. In fact, if the percentage of internally

Incremental Learning-Based Testing for Reactive Systems

137

generated book-keeping queries is too high then model checking becomes almost redundant. In this case we might think that LBT becomes equivalent to random testing. However [18] shows that this is not the case. Even without model checking, LBT achieves better functional coverage than random testing. In practise, most of the well-known classical regular inference algorithms such as L* (Angluin [2]) or ID (Angluin [1]) are designed for complete rather incremental learning. Among the much smaller number of known incremental learning algorithms, we can mention the RPNII algorithm (Dupont [7]) and the IID algorithm (Parekh et al. [15]) which learn Moore automata, and the CGE algorithm (Meinke [10]) which learns Mealy automata. To our knowledge, no incremental algorithm for learning Kripke structures has yet been published in the literature. Thus the IKL algorithm, and its application to testing represent novel contributions of our paper.

2

Mathematical Preliminaries and Notation

Let Σ be any set of symbols then Σ ∗ denotes the set of all ﬁnite strings over Σ including the empty string ε. The length of a string α ∈ Σ ∗ is denoted by |α| and |ε| = 0. For strings α, β ∈ Σ ∗ , α . β denotes their concatenation. For α, β, γ ∈ Σ ∗ , if α = βγ then β is termed a preﬁx of α and γ is termed a suﬃx of α. We let Pref (α) denote the preﬁx closure of α, i.e. the set of all preﬁxes of α. We can also apply preﬁx closure pointwise to any set of strings. The set diﬀerence operation between two sets U, V , denoted by U − V , is the set of all elements of U which are not members of V . The symmetric diﬀerence operation on pairs of sets is deﬁned by U ⊕ V = (U − V ) ∪ (V − U ). A deterministic ﬁnite automaton (DFA) is a ﬁve-tuple A =< Σ, Q, F, q0 , δ > where: Σ is the input alphabet, Q is the state set, F ⊆ Q is the accepting state set and q0 ∈ Q is the starting state. The state transition function of A is a mapping δ : Q×Σ → Q with the usual meaning, and can be inductively extended to a mapping δ∗ : Q × Σ ∗ → Q where δ ∗ (q, ε) = q and δ ∗ (q, σ1 , . . . σn+1 ) = δ(δ ∗ (q, σ1 , . . . σn ), σn+1 ). Since input strings can be used to name states, given a distinguished dead state d0 (from which no accepting state can be reached) we deﬁne string concatenation modulo the dead state d0 , f : Σ ∗ ∪ {d0 } × Σ → Σ ∗ ∪ {d0 }, by f (d0 , σ) = d0 and f (α, σ) = α . σ for α ∈ Σ ∗ . This function is used for automaton learning in Section 4. The language L(A) accepted by A is the set of all strings α ∈ Σ ∗ such that δ ∗ (q0 , α) ∈ F . A language L ⊆ Σ ∗ is accepted by a DFA if and only if, L is regular, i.e. L can be deﬁned by a regular grammar. A generalisation of DFA to multi-bit outputs on states is given by deterministic Kripke structures. 2.1. Deﬁnition. Let Σ = { σ1 , . . . , σn } be a ﬁnite input alphabet. By a k-bit deterministic Kripke structure A we mean a ﬁve-tuple 0 A = ( QA , Σ, δA : QA × Σ → QA , qA , λA : QA → Bk )

138

K. Meinke and M.A. Sindhu

0 where QA is a state set, δA is the state transition function, qA is the initial state ∗ and λA is the output function. As before we let δA : QA × Σ ∗ → QA denote the ∗ ∗ iterated state transition function, where δA (q, ε) = q and δA (q, σ1 , . . . , σi+1 ) = ∗ ∗ ∗ k δA (δA (q, σ1 , . . . , σi ), σi+1 ). Also we let λA : Σ → B denote the iterated ∗ 0 output function λ∗A (σ1 , . . . , σi ) = λA (δA (qA , σ1 , . . . , σi )).

If A is a Kripke structure then the minimal subalgebra M in(A) of A is the unique subalgebra of A which has no proper subalgebra. (We implicitly assume that all input symbols σ ∈ Σ are constants of A so that M in(A) has a non-trivial state set.) Note that a 1-bit deterministic Kripke structure A is isomorphic to the 0 DFA A = ( QA , Σ, δA : QA × Σ → QA , qA , FA ), where FA ⊆ QA and λA (q) = true if, and only if q ∈ FA .

3

Bit-Sliced Learning of Kripke Structures

We will establish a precise basis for learning k-bit Kripke structures using regular inference algorithms for DFA. The approach we take is to bit-slice the output of a k-bit Kripke structure A into k individual 1-bit Kripke structures A1 , . . . , Ak , which are learned in parallel as DFA by some regular inference algorithm. The k inferred DFA B1 , . . . , Bk are then recombined using a subdirect product construction to obtain a Kripke structure that is behaviourally equivalent to A. This approach has three advantages: (1) We can make use of any regular inference algorithm to learn the individual 1-bit Kripke structures Ai . Thus we have access to the wide range of known regular inference algorithms. (2) We can reduce the total number of book-keeping queries by lazy book-keeping. This technique maximises re-use of book-keeping queries among the 1-bit structures Ai . In Section 4, we illustrate this technique in more detail. (3) We can learn just those bits which are necessary to test a speciﬁc temporal logic requirement. This abstraction technique improves the scalability of testing. It usually suﬃces to learn automata up to behavioural equivalence. 3.1. Deﬁnition. Let A and B be k-bit Kripke structures over a ﬁnite input alphabet Σ. We say that A and B are behaviourally equivalent, and write A ≡ B if, and only if, for every ﬁnite input sequence σ1 , . . . , σi ∈ Σ ∗ we have λ∗A ( σ1 , . . . , σi ) = λ∗B ( σ1 , . . . , σi ). Clearly, by the isomorphism identiﬁed in Section 2 between 1-bit Kripke structures and DFA, for such structures we have A ≡ B if, and only if, L(A ) = L(B ). Furthermore, if M in(A) is the minimal subalgebra of A then M in(A) ≡ A. Let us make precise the concept of bit-slicing a Kripke structure. 3.2. Deﬁnition. Let A be a k-bit Kripke structure over a ﬁnite input alphabet Σ, 0 A = ( QA , Σ, δA : QA × Σ → QA , qA , λA : QA → Bk ).

Incremental Learning-Based Testing for Reactive Systems

139

For each 1 ≤ i ≤ k deﬁne the i-th projection Ai of A to be the 1-bit Kripke structure where 0 , λAi : QA → B ), Ai = ( QA , Σ, δA : QA × Σ → QA , qA

and λAi (q) = λA (q)i , i.e. λAi (q) is the i-th bit of λA (q). A family of k individual 1-bit Kripke structures can be combined into a single k-bit Kripke structure using the following subdirect product construction. (See e.g. [13] for a general deﬁnition of subdirect products and their universal properties.) 3.3. Deﬁnition. Let A1 , . . . , Ak be a family of 1-bit Kripke structures, Ai = ( Qi , Σ, δi : Qi × Σ → Qi , qi0 , λi : Q → B ) for i = 1, . . . , k. Deﬁne the product Kripke structure k

Ai = ( Q, Σ, δ : Q × Σ → Q, q 0 , λ : Q → Bk ),

i=1

where Q =

k i=1

Qi = Q1 × . . . × Qk and q 0 = ( q10 , . . . , qk0 ). Also

δ(q1 , . . . , qk , σ) = ( δ1 (q1 , σ), . . . , δk (qk , σ) ), λ(q1 , . . . , qk ) = ( λ1 (q1 ), . . . , λk (qk ) ). Associated with the direct product ki=1 Ai we have i-th projection mapping proj i : Q1 × . . . × Qk → Qi , Let Min(

k i=1

proj i (q1 , . . . , qk ) = qi , 1 ≤ i ≤ k

Ai ) be the minimal subalgebra of

k i=1

Ai .

k The reason for taking the minimal subalgebra of the direct product i=1 Ai is to avoid the state space explosion due to a large number of unreachable states k in the direct product itself. The state space size of i=1 Ai grows exponentially with k. On the other hand since most of these states are unreachable from the initial state, then from the point of view of behavioural analysis these states are irrelevant. Note that this minimal subalgebra can be computed in linear time from its components Ai (w.r.t. state space size). As is well known from universal algebra, the i-th projection mapping proj i is a homomorphism. 3.4. Proposition. Let A1 , . . . , Ak beany minimal 1-bit Kripke structures. k (i) For each 1 ≤ i ≤ k, proj i : Min( i=1 Ai ) → Ai is an epimorphism, and k hence Min( i=1 Ai ) is a subdirect product of the Ai . k k (ii) Min( i=1 Ai ) ≡ i=1 Ai . Proof. (i) Immediate since the Ai are minimal. (ii) Follows from M in(A) ≡ A.

140

K. Meinke and M.A. Sindhu

The following theorem justiﬁes bit-sliced learning of k-bit Kripke structures using conventional regular inference methods for DFA. 3.5. Theorem. Let A be a k-bit Kripke structure over a finite input alphabet Σ. Let A1 , . . . , Ak be the k individual 1-bit projections of A. For any 1-bit Kripke structures B1 , . . . , Bk , if, A1 ≡ B1 & . . . & Ak ≡ Bk then A ≡ Min(

k

Bi ).

i=1

Proof. Use Proposition 3.4.

4

Incremental Learning for Kripke Structures

In this section we present a new algorithm for incremental learning of Kripke structures. We will brieﬂy discuss its correctness and termination properties, although a full discussion of these is outside the scope of this paper and is presented elsewhere in [12]. Our algorithm applies bit-slicing as presented in Section 3, and uses distinguishing sequences and lazy partition reﬁnement for regular inference of the 1-bit Kripke structures. The architecture of the IKL algorithm consists of a main learning algorithm and two sub-procedures for lazy partition reﬁnement and automata synthesis. Distinguishing sequences were introduced in Angluin [1] as a method for learning DFA. Algorithm 1 is the main algorithm for bit-sliced incremental learning. It learns a sequence M1 , . . . , Ml of n-bit Kripke structures that successively approximate a single n-bit Kripke structure A, which is given as the teacher. In LBT, the teacher is always the SUT. The basic idea of Algorithm 1 is to construct in parallel a family Ei11 , . . . , Einn of n diﬀerent equivalence relations on the same set Tk of state names. For each equivalence relation Eijj , a set Vj of distinguishing strings is generated iteratively to split pairs of equivalence classes in Eijj until a congruence is achieved. Then a quotient DFA M j can be constructed from the partition of Tk by Eijj . The j and thus the IKL algorithm is congruences are constructed so that Eij ⊆ Ei+1 incremental, and fully reuses information about previous approximations, which is eﬃcient. Each n-bit Kripke structure Mt is constructed using synthesis algorithm 3, as a subdirect product of n individual quotient DFA M 1 , . . . , M n (viewed as 1-bit Kripke structures). When the IKL algorithm is applied to the problem of LBT, the input strings si ∈ Σ ∗ to IKL are generated as counterexamples to correctness (i.e. test cases) by executing a model checker on the approximation Mt−1 with respect to some requirements speciﬁcation φ expressed in temporal logic. If no counterexamples to φ can be found in Mt−1 then si is randomly chosen, taking care to avoid all previously used input strings.

Incremental Learning-Based Testing for Reactive Systems

141

Algorithm 1. IKL: Incremental Kripke Structure Learning Algorithm Input: A ﬁle S = s1 , . . . , sl of input strings si ∈ Σ ∗ and a Kripke structure A with n-bit output as teacher to answer queries λ∗A (si ) = ? Output: A sequence of Kripke structures Mt with n-bit output for t = 0, . . . , l. 1. begin 2. //Perform Initialization 3. for c = 1 to n do { ic = 0, vic = ε, Vc = {vic } } 4. k = 0, t = 0, 5. P0 = {ε}, P0 = P0 ∪ {d0 }, T0 = P0 ∪ Σ 6. //Build equivalence classes for the dead state d0 7. for c = 1 to n do { E0c (d0 ) = ∅ } 8. //Build equivalence classes for input strings of length zero and one 9. ∀α ∈ T0 { 10. (b1 , . . . , bn ) = λ∗A (α) 11. for c = 1 to n do 12. if bc then Eicc (α) = {vic } else Eicc (α) = ∅ 13. } 14. //Reﬁne the initial equivalence relations E01 , . . . , E0n 15. //into congruences using Algorithm 2 16. 17. //Synthesize an initial Kripke structure M0 approximating A 18. //using Algorithm 3. 19. 20. //Process the ﬁle of examples. 21. while S = empty do { 22. read( S, α ) 23. k = k+1, t = t+1 24. Pk = Pk−1 ∪ P ref (α) //preﬁx closure 25. Pk = Pk ∪ {d0 } 26. Tk = Tk−1 ∪ P ref(α) ∪ {α . b | α ∈ Pk − Pk−1 , b ∈ Σ} //for preﬁx closure 27. Tk = Tk ∪ {d0 } 28. ∀α ∈ Tk − Tk−1 { 29. for c = 1 to n do E0c (α) = ∅ //initialise new equivalence class E0c (α) 30. for j = 0 to ic do { 31. // Consider adding distinguishing string vj ∈ Vc 32. // to each new equivalence class Ejc (α) 33. (b1 , . . . , bn ) = λ∗A (α . vj ) 34. if bc then Ejc (α) = Ejc (α) ∪ { vj } 35. } 36. } 37. //Reﬁne the current equivalence relations Ei11 , . . . , Einn 38. // into congruences using Algorithm 2 39. 40. if α is consistent with Mt−1 41. then Mt = Mt−1 42. else synthesize Kripke structure Mt using Algorithm 3. 43. } 44. end.

142

K. Meinke and M.A. Sindhu

Algorithm 2. Lazy Partition Reﬁnement 1. while (∃ 1 ≤ c ≤ n, ∃α, β ∈ Pk and ∃σ ∈ Σ such that Eicc (α) = Eicc (β) but Eicc (f (α, σ)) = Eicc (f (β, σ)) do { 2. //Equivalence relation Eicc is not a congruence w.r.t. δc 3. //so add a new distinguishing sequence. 4. Choose γ ∈ Eicc (f (α, σ)) ⊕ Eicc (f (β, σ)) 5. v=σ.γ 6. ∀α ∈ Tk { 7. (b1 , . . . , bn ) = λ∗A (α . v) 8. for c = 1 to n do { 9. if Eicc (α) = Eicc (β) and Eicc (f (α, σ)) = Eicc (f (β, σ)) then { 10. // Lazy reﬁnement of equivalence relation Eicc 11. ic = ic + 1, vic = v, Vc = Vc ∪ {vic } 12. if bc then Eicc (α) = Eicc −1 (α) ∪ {vic } else Eicc (α) = Eicc −1 (α) 13. } 14. } 15. }

Algorithm 3. Kripke Structure Synthesis 1. for c = 1 to n do { 2. // Synthesize the quotient DFA (1-bit Kripke structure) M c 3. The states of M c are the sets Eicc (α), where α ∈ Tk 4. Let q0c = Eicc (ε) 5. The accepting states are the sets Eicc (α) where α ∈ Tk and ε ∈ Eicc (α) 6. The transition function δc of M c is deﬁned as follows: 7. ∀α ∈ Pk { 8. if Eicc (α) = ∅ then ∀b ∈ Σ { let δc (Eicc (α), b) = Eicc (α) } 9. else ∀b ∈ Σ { δc (Eicc (α), b) = Eicc (α . b) } 10. } 11. ∀β ∈ Tk − Pk { 12. if ∀α ∈ Pk { Eicc (β) = Eicc (α) } and Eicc (β) = ∅ then 13. ∀b ∈ Σ { δc (Eicc (β), b) = ∅ } 14. } 15. // ComputeMt in linear time as a subdirect product of the M c c 16. Mt = Min( n c=1 M )

Incremental Learning-Based Testing for Reactive Systems

143

Algorithm 2 implements lazy partition reﬁnement, to extend Ei11 , . . . , Einn from being equivalence relations on states to being a family of congruences with respect to the state transition functions δ1 , . . . , δn of M 1 , . . . , M n . Thus line 1 searches for congruence failure in any one of the equivalence relations Ei11 , . . . , Einn . In lines 6-14 we apply lazy partition reﬁnement. This technique implies reusing the new distinguishing string v wherever possible to reﬁne each equivalence relation Eijj that is not yet a congruence. On the other hand, any equivalence relation Eijj that is already a congruence is not reﬁned, even though the result bj of the new query α . v might add some new information to M j . This helps minimise the total number of partition reﬁnement queries (cf. Section 1.2). Algorithm 3 implements model synthesis. First, each of the n quotient DFA M 1 , . . . , M n are constructed. These, reinterpreted as 1-bit Kripke structures, are then combined in linear time as a subdirect product to yield a new n-bit approximation Mt to A (c.f. Section 3). 4.1

Correctness and Termination of the IKL Algorithm

The sequence M1 , . . . , Ml of hypothesis Kripke structures which are incrementally generated by IKL can be proven to ﬁnitely converge to A up to behavioural equivalence, for suﬃciently large l. The key to this observation lies in the fact that we can identify a ﬁnite set of input strings such that the behavior of A is completely determined by its behaviour on this ﬁnite set. Recall that for a DFA A =< Σ, Q, F, q0 , δ > a state q ∈ Q is said to be live if for some string α ∈ Σ ∗ , δ ∗ (q, α) ∈ F . A ﬁnite set C ⊆ Σ ∗ of input strings is said to be live complete for A if for every reachable live state q ∈ Q there exists a string α ∈ C such that δ ∗ (q0 , α) = q. More generally, given a ﬁnite collection A1 , . . . , Ak of DFA, then C ⊆ Σ ∗ is live complete for A1 , . . . , Ak if, and only if, for each 1 ≤ i ≤ k, C is a live complete set for Ai . Clearly, for every ﬁnite collection of DFA there exists at least one live complete set of strings. 4.1.1. Theorem. Let A be a k-bit Kripke structure over a finite input alphabet Σ. Let A1 , . . . , Ak be the k individual 1-bit projections of A. Let C = { s1 , . . . , sl } ⊆ Σ ∗ be a live complete set for A1 , . . . , Ak . The IKL algorithm terminates on C and for the final hypothesis structure Ml we have Ml ≡ A. Proof. See [12].

5

A Learning-Based Testing Architecture Using IKL

Figure 1 depicts an architecture for learning-based testing of reactive systems by combining the IKL algorithm of Section 4 with a model checker for Kripke stuctures and an oracle. In this case we have chosen to use the NuSMV model checker (see e.g. Cimatti et al. [4]), which supports the satisﬁability analysis of Kripke

144

K. Meinke and M.A. Sindhu

!

"

#

$$ % !

#$ %%

!

"

!

Fig. 1. A Learning-Based Testing Architecture using the IKL algorithm

structures with respect to both linear temporal logic (LTL) and computation tree logic (CTL) [6]. To understand this architecture, it is useful to recall the abstract description of learning-based testing as an iterative process, given in Section 1.1. Following the account of Section 1.1, we can assume that at any stage in the testing process we have an inferred Kripke structure Mn produced by the IKL algorithm from previous testing and learning. Test cases will have been produced as counterexamples to correctness by the model checker, and learning queries will have been produced by the IKL algorithm during partition reﬁnement. (Partition reﬁnement queries are an example of what we termed book-keeping queries in Section 1.2.) In Figure 1, the output Mn of the IKL algorithm is passed to an equivalence checker. Since this architectural component is not normally part of an LBT framework, we should explain its presence carefully. We are particularly interested in benchmarking the performance of LBT systems, both to compare their performance with other testing methodologies, and to make improvements to existing LBT systems. (See Section 6.) In realistic testing situations, we do not anticipate that an entire SUT can be learned in a feasible time (c.f. the discussion in Section 1.2). However, for benchmarking with the help of smaller case studies (for which complete learning is feasible) it is useful to be able to infer the earliest time at which we can say that testing is complete. Obviously testing must be complete at time ttotal when we have learned the entire SUT (c.f. Section 6). Therefore the equivalence checker allows us to compute the time ttotal simply to conduct benchmarking studies. (Afterwards the equivalence checker is removed.) The equivalence checker compares the current Kripke structure Mn with the SUT. A positive result from this equivalence test stops all further learning and testing after one ﬁnal model check. The algorithm we use has been adapted from

Incremental Learning-Based Testing for Reactive Systems

145

the quasi-linear time algorithm for DFA equivalence checking described in [14] and has been extended to deal with k-bit Kripke structures. In Figure 1, the inferred model Mn is passed to a model checker, together with a user requirement represented as a temporal logic formula φ. This formula is constant during a particular testing experiment. The model checker attempts to identify at least one counterexample to φ in Mn as an input sequence i. If φ is a safety formula then this input sequence will usually be ﬁnite i = i1 , . . . , ik . If φ is a liveness formula then this input sequence i may be ﬁnite or inﬁnite. Recall that inﬁnite counterexamples to liveness formulas can be represented as inﬁnite sequences of the form x y ω . In the case that i = x y ω then i is truncated to a ﬁnite initial segment that would normally include the handle x and at least one execution of the inﬁnite loop y ω , such as i = x y or i = x y y. Observing the failure of an inﬁnite test case is of course impossible. The LBT architecture implements a compromise solution that runs the truncated sequence only, in ﬁnite time, and issues a warning rather than a fail verdict. Note that if the next input sequence i cannot be constructed either by partition reﬁnement or by model checking then in order to proceed with iterative testing and learning, another way to generate i must be found. (See the discussion in Section 1.1.) One simple solution, shown in Figure 1, is to use a random input sequence generator for i, taking care to discard any previously used sequences. Thus from one of three possible sources (partition reﬁnement, model checking or randomly) a new input sequence i = i1 , . . . , ik is constructed. Figure 1 shows that if i is obtained by model checking then the current model Mn is applied to i to compute a predicted output p = p1 , . . . , pk for the SUT that can be used for the oracle step. However, this is not possible if i is random or a partition reﬁnement since then we do not know whether i is a counterexample to φ. Nevertheless, in all three cases, the input sequence i is passed to the SUT and executed to yield an actual or observed output sequence o = o1 , . . . , ok . The ﬁnal stage of this iterative testing architecture is the oracle step. Figure 1 shows that if a predicted output p exists (i.e. the input sequence i came from model checking) then actual output o and the predicted output p are both passed to an oracle component. This component implements the Boolean test o = p. If this equality test returns true and the test case i = i1 , . . . , ik was originally a ﬁnite test case then we can conclude that the test case i is deﬁnitely failed, since the behaviour p is by construction a counterexample to the correctness of φ. If the equality test returns true and the test case i is ﬁnitely truncated from an inﬁnite test case (a counterexample to a liveness requirement) then the verdict is weakened to a warning. This is because the most we can conclude is that we have not yet seen any diﬀerence between the observed behaviour o and the incorrect behaviour p. The system tester is thus encouraged to consider a potential SUT error. On the other hand if o = p, or if no output prediction p exists then it is quite diﬃcult to issue an immediate verdict. It may or may not be the case that the observed output o is a counterexample to the correctness of φ. In some cases the syntactic structure of φ is simple enough to semantically evaluate the formula

146

K. Meinke and M.A. Sindhu

φ on the ﬂy with its input and output variables bound to i and o respectively. However, sometimes this is not possible since the semantic evaluation of φ also refers to global properties of the automaton. Ultimately, this is not a problem for our approach, since Mn+1 is automatically updated with the output behaviour o. Model checking Mn+1 later on will conﬁrm o as an error if this is the case. 5.1

Correctness and Termination of the LBT Architecture

It is important to establish that the LBT architecture always terminates, at least in principle. Furthermore, the SUT coverage obtained by this testing procedure is complete, in the sense that if the SUT contains any counterexamples to correctness then a counterexample will be found by the testing architecture. When the SUT is too large to be completely learned in a feasible amount of time, this completeness property of the testing architecture still guarantees that there is no bias in testing so that one could somehow never discover an SUT error. Failure to ﬁnd an error in this case is purely a consequence of insuﬃcient testing time. The termination and correctness properties of the LBT architecture depend on the following correctness properties of its components: (i) the IKL algorithm terminates and correctly learns the SUT given a ﬁnite set C of input strings which is live complete (c.f. Theorem 4.1.1); (ii) the model checking algorithm used by NuSMV is a terminating decision procedure for the validity of LTL formulas; (iii) each input string i ∈ Σ ∗ is generated with non-zero probability by the random input string generator. 5.1.1. Theorem. Let A be a k-bit Kripke structure over an input alphabet Σ. (i) The LBT architecture (with equivalence checker) terminates with probability 1.0, and for the final hypothesis structure Ml we have Ml ≡ A. (ii) If there exists a (finite or infinite) input string i over Σ which witnesses that an LTL requirement φ is not valid for A, then model checking will eventually find such a string i and the LBT architecture will generate a test fail or test warning message after executing i as a test case on A. Proof. (i) Clearly by Theorem 4.1.1, the IKL algorithm will learn the SUT A up to behavioural equivalence, given as input a live complete set C for the individual 1-bit projections A1 , . . . , Ak of A. Now, we cannot be sure that the strings generated by model checking counterexamples and partition reﬁnement queries alone constitute a live complete set C for A1 , . . . , Ak . However, these sets of queries are complemented by random queries. Since a live complete set is ﬁnite, and every input string is randomly generated with non-zero probability, then with probability 1.0 the IKL algorithm will eventually obtain a live complete set and converge. At this point, equivalence checking the ﬁnal hypothesis structure Ml with the SUT will succeed and the LBT architecture will terminate.

Incremental Learning-Based Testing for Reactive Systems

147

(ii) Suppose there is at least one (ﬁnite or inﬁnite) counterexample string i over Σ to the validity of an LTL requirement φ for A. In the worst case, by part (i), the LBT architecture will learn the entire structure of A. Since the model checker implements a terminating decision procedure for validity of LTL formulas, it will return a counterexample i from the ﬁnal hypothesis structure Ml , since by part (i), Ml ≡ A and A has a counterexample. For such i, comparing the corresponding predicted output p from Ml and the observed output o from A we must have p = o since Ml ≡ A. Hence the testing architecture will issue a fail or warning message.

6

Case Studies and Performance Benchmarking

In order to evaluate the eﬀectiveness of the LBT architecture described in Section 5, we conducted a number of testing experiments on two SUT case studies, namely an 8 state cruise controller and a 38 state 3-ﬂoor elevator model1 . For each SUT case study we chose a collection of safety and liveness requirements that could be expressed in linear temporal logic (LTL). For each requirement we then injected an error into the SUT that violated this requirement and ran a testing experiment to discover the injected error. The injected errors all consisted of transition mutations obtained by redirecting a transition to a wrong state. This type of error seems quite common in our experience. There are a variety of ways to measure the performance of a testing system such as this. One simple measure that we chose to consider was to record the ﬁrst time tfirst at which an error was discovered in an SUT, and to compare this with the total time ttotal required to completely learn the SUT. (So tfirst ≤ ttotal .) This measure is relevant if we wish to estimate the beneﬁt of using incremental learning instead of complete learning. Because some random queries are almost always present in each testing experiment, the performance of the LBT architecture has a degree of variation. Therefore, for the same correctness formula and injected error, we ran each experiment ten times to try to average out these variations in performance. This choice appeared adequate to obtain a representative average. Subsections 6.1 and 6.2 below set out the results obtained for each case study. 6.1

The Cruise Controller Model

The cruise controller model we chose as an SUT is an 8 state 5-bit Kripke structure with an input alphabet of 5 symbols. Figure 2 shows its structure2 . The four requirements shown in Table 1 consist of: (1,2) two requirements on speed maintenance against obstacles in cruise mode, and (3,4) disengaging cruise mode by means of the brake and gas pedals. To gain insight into the LBT 1 2

Our testing platform was based on a PC with a 1.83 GHz Intel Core 2 duo processor and 4GB of RAM running Windows Vista. The following binary data type encoding is used. Modes: 00 = manual, 01 = cruise, 10 = disengaged. Speeds: 00 = 0, 01 = 1, 10 = 2.

148

K. Meinke and M.A. Sindhu

Fig. 2. The cruise controller SUT

Table 1. Cruise Controller Requirements as LTL formulas Req Req Req Req

1 2 3 4

G( G( G( G(

mode mode mode mode

= = = =

cruise cruise cruise cruise

& & & &

speed = 1 & in = dec -> X( speed = 1 )) speed = 1 & in = acc -> X( speed = 1 ) ) in = brake -> X( mode = disengaged ) ) in = gas -> X( mode = disengaged ) )

Table 2. LBT performance for Cruise Controller Requirements Requirement tfirst (sec) ttotal (sec) M CQfirst M CQtotal Req 1 3.5 21.5 3.2 24.3 Req 2 2.3 5.7 5.5 18.2 Req 3 2.3 16.0 1.7 33.7 Req 4 2.9 6.1 4.7 20.9

P Qfirst 7383 8430 6127 7530

P Qtotal RQfirst RQtotal 30204 8.2 29.3 27384 10.4 23.1 34207 6.8 38.8 24566 10.4 20.9

Incremental Learning-Based Testing for Reactive Systems

149

architecture performance, Table 2 shows average ﬁgures at times tfirst and ttotal for the numbers: (i) M CQfirst and M CQtotal of model checker generated test cases, (ii) P Qfirst and P Qtotal of partition reﬁnement queries, (iii) RQfirst and RQtotal of random queries. In Table 2, columns 2 and 3 show that the times required to ﬁrst discover an error in the SUT are between 14% and 47% of the total time needed to completely learn the SUT. The large query numbers in columns 6 and 7 show that partition reﬁnement queries dominate the total number of queries. Columns 8 and 9 show that the number of random queries used is very low, of and of the same order of magnitude as the number of model checking queries (columns 4 and 5). Thus partition reﬁnement queries and model checker generated test cases come quite close to achieving a live complete set, although they do not completely suﬃce for this (c.f. Section 4.1). 6.2

The Elevator Model

The elevator model we chose as an SUT is a 38 state 8-bit Kripke structure with an input alphabet of 4 symbols. Figure 3 shows its condensed structure as a hierarchical statechart. The six requirements shown in Table 3 consist of requirements that: (1) the elevator does not stop between ﬂoors, (2) doors are closed when in motion, (3) doors open upon reaching a ﬂoor, and (4, 5, 6) closed doors can be opened by pressing the same ﬂoor button when stationary at a ﬂoor.

Fig. 3. The 3-ﬂoor elevator SUT (condensed Statechart ) Table 3. Elevator Requirements as LTL formulas Req Req Req Req Req Req

1 2 3 4 5 6

G( G( G( G( G( G(

Stop -> ( @1 | @2 | @3 ) ) !Stop -> cl ) Stop & X( !Stop ) -> X( !cl ) ) Stop & @1 & cl & in=c1 & X( @1 ) -> X( !cl ) ) Stop & @2 & cl & in=c2 & X( @2 ) -> X( !cl ) ) Stop & @3 & cl & in=c3 & X( @3 ) -> X( !cl ) )

150

K. Meinke and M.A. Sindhu Table 4. LBT performance for Elevator Requirements

Requirement tfirst (sec) ttotal (sec) M CQfirst M CQtotal Req 1 0.34 1301.3 1.9 81.7 Req 2 0.49 1146 3.9 99.6 Req 3 0.94 525 1.6 21.7 Req 4 0.052 1458 1.0 90.3 Req 5 77.48 2275 1.2 78.3 Req 6 90.6 1301 2.0 60.9

P Qfirst 1574 2350 6475 15 79769 129384

P Qtotal RQfirst RQtotal 729570 1.9 89.5 238311 2.9 98.6 172861 5.7 70.4 450233 0.0 91 368721 20.5 100.3 422462 26.1 85.4

Table 4 shows the results of testing the requirements of Table 3. These results conﬁrm several trends seen in Table 2. However, they also show a signiﬁcant increase in the eﬃciency of using incremental learning, since the times required to ﬁrst discover an error in the SUT are now between 0.003% and 7% of the total time needed to completely learn the SUT. These results are consistent with observations of [12] that the convergence time of IKL grows quadratically with state space size. Therefore incremental learning gives a more scalable testing method than complete learning.

7

Conclusions

We have presented a novel incremental learning algorithm for Kripke structures, and shown how this can be applied to learning-based testing of reactive systems. Using two case studies of reactive systems, we have conﬁrmed our initial hypothesis of Section 1.2, that incremental learning is a more scalable and efﬁcient method of testing than complete learning. These results are consistent with similar results for LBT applied to procedural systems in [11]. Further research could be carried out to improve the performance of the architecture presented here. For example the performance of the oracle described in Section 5 could be improved to yield a verdict even for random and partition queries, at least for certain kinds of LTL formulas. Further research into scalable learning algorithms would be valuable for dealing with large hypothesis automata. The question of learning-based coverage has been initially explored in [18] but further research here is also needed. We gratefully acknowledge ﬁnancial support for this research from the Swedish Research Council (VR), the Higher Education Commission (HEC) of Pakistan, and the European Union under project HATS FP7-231620.

References 1. Angluin, D.: A note on the number of queries needed to identify regular languages. Information and Control 51(1), 76–87 (1981) 2. Angluin, D.: Learning regular sets from queries and counterexamples. Information and Computation 75(1), 87–106 (1987)

Incremental Learning-Based Testing for Reactive Systems

151

3. Bohlin, T., Jonsson, B.: Regular inference for communication protocol entities. Technical Report 2008-024, Dept. of Information Technology, Uppsala University (2008) 4. Cimatti, A., Clarke, E., Giunchiglia, F., Roveri, M.: NUSMV: A new symbolic model veriﬁer. In: Halbwachs, N., Peled, D.A. (eds.) CAV 1999. LNCS, vol. 1633, pp. 495–499. Springer, Heidelberg (1999) 5. Clarke, E., Gupta, A., Kukula, J., Strichman, O.: SAT based abstractionreﬁnement using ILP and machine learning techniques. In: Brinksma, E., Larsen, K.G. (eds.) CAV 2002. LNCS, vol. 2404, p. 265. Springer, Heidelberg (2002) 6. Clarke, E.M., Grumberg, O., Peled, D.A.: Model Checking. MIT Press, Cambridge (1999) 7. Dupont, P.: Incremental regular inference. In: Miclet, L., de la Higuera, C. (eds.) ICGI 1996. LNCS (LNAI), vol. 1147. Springer, Heidelberg (1996) 8. Groce, A., Peled, D., Yannakakis, M.: Adaptive model checking. Logic Journal of the IGPL 14(5), 729–744 (2006) 9. Meinke, K.: Automated black-box testing of functional correctness using function approximation. In: ISSTA 2004: Proceedings of the 2004 ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 143–153. ACM, New York (2004) 10. Meinke, K.: CGE: A sequential learning algorithm for Mealy automata. In: Sempere, J.M., García, P. (eds.) ICGI 2010. LNCS, vol. 6339, pp. 148–162. Springer, Heidelberg (2010) 11. Meinke, K., Niu, F.: A learning-based approach to unit testing of numerical software. In: Petrenko, A., Simão, A., Maldonado, J.C. (eds.) ICTSS 2010. LNCS, vol. 6435, pp. 221–235. Springer, Heidelberg (2010) 12. Meinke, K., Sindhu, M.: Correctness and performance of an incremental learning algorithm for Kripke structures. Technical report, School of Computer Science and Communication, Royal Institute of Technology, Stockholm (2010) 13. Meinke, K., Tucker, J.V.: Universal algebra. In: Handbook of Logic in Computer Science, 1st edn., pp. 189–411. Oxford University Press, Oxford (1993) 14. Norton, D.A.: Algorithms for testing equivalence of ﬁnite state automata, with a grading tool for jﬂap. Technical report, Rochester Institute of Technology, Department of Computer Science (2009) 15. Parekh, R.G., Nichitiu, C., Honavar, V.G.: A polynomial time incremental algorithm for regular grammar inference. In: Honavar, V.G., Slutzki, G. (eds.) ICGI 1998. LNCS (LNAI), vol. 1433, p. 37. Springer, Heidelberg (1998) 16. Peled, D., Vardi, M.Y., Yannakakis, M.: Black-box checking. In: Formal Methods for Protocol Engineering and Distributed Systems FORTE/PSTV, pp. 225–240. Kluwer, Dordrecht (1999) 17. Raﬀelt, H., Steﬀen, B., Margaria, T.: Dynamic testing via automata learning. In: Yorav, K. (ed.) HVC 2007. LNCS, vol. 4899, pp. 136–152. Springer, Heidelberg (2008) 18. Walkinshaw, N., Bogdanov, K., Derrick, J., Paris, J.: Increasing functional coverage by inductive testing: a case study. In: Petrenko, A., Simão, A., Maldonado, J.C. (eds.) ICTSS 2010. LNCS, vol. 6435, pp. 126–141. Springer, Heidelberg (2010)

Encoding OCL Data Types for SAT-Based Verification of UML/OCL Models Mathias Soeken, Robert Wille, and Rolf Drechsler Institute of Computer Science, University of Bremen Computer Architecture Group, D-28359 Bremen, Germany {msoeken,rwille,drechsle}@informatik.uni-bremen.de

Abstract. Checking the correctness of UML/OCL models is a crucial task in the design of complex software and hardware systems. As a consequence, several approaches have been presented which address this problem. Methods based on satisfiability (SAT) solvers have been shown to be very promising in this domain. Here, the actual veriﬁcation task is encoded as an equivalent bit-vector instance to be solved by an appropriate solving engine. However, while a bit-vector encoding for basic UML/OCL constructs has already been introduced, no encoding for nontrivial OCL data types and operations is available so far. In this paper, we close this gap and present a bit-vector encoding for more complex OCL data types, i.e. sets, bags, and their ordered counterparts. As a result, SAT-based UML/OCL veriﬁcation becomes applicable for models containing these collections types. A case study illustrates the application of this encoding.

1

Introduction

The Unified Modeling Language (UML) [1] is a de-facto standard in the domain of software development. Besides that, in recent years UML is also being employed for the speciﬁcation of hardware systems [2] as it is a promising abstraction level enabling the modeling of a complex system while hiding concrete implementation details. Within UML, the Object Constraint Language (OCL) enables the enrichment of object-oriented models by textual constraints which add vital information. Using OCL, it is possible to restrict valid system states by invariants or to control the applicability of operation calls by pre- and post-conditions. However, adding too restrictive OCL constraints leads to an inconsistent model, i.e. a model from which no valid system state can be constructed. Consistency and other veriﬁcation tasks refer to the static aspects of a UML model. Further, wrong pre- and post-conditions can cause that an operation ω is not reachable, i.e. no system state can be constructed due to calls of other operations such that the pre-conditions of ω are satisﬁed. This property is called reachability and refers along with other veriﬁcation tasks to the dynamic aspects of a UML model. Accordingly, several approaches to (semi-)automatically solve these veriﬁcation tasks have been proposed in the last years. For this purpose, diﬀerent solving methodologies and engines are applied ranging from (1) interactive theorem M. Gogolla and B. Wolﬀ (Eds.): TAP 2011, LNCS 6706, pp. 152–170, 2011. c Springer-Verlag Berlin Heidelberg 2011

Encoding OCL Data Types for SAT-Based Veriﬁcation

153

provers [3,4] which require manual interactions over (2) enumeration techniques as e.g. provided in the UML Specification Environment (USE) [5] to (3) automatic approaches based on Constraint Programming (CSP) [6,7] or speciﬁcation languages such as Alloy [8]. However, these approaches suﬀer either from the need for manual interaction, their enumerative behavior resulting in low scalability, or complicated transformations and solving steps. In order to tackle these drawbacks, an alternative automatic UML/OCL veriﬁcation method based on Boolean Satisfiability (SAT) and SAT Modulo Theories (SMT) has recently been suggested in [9,10]. Here, the actual veriﬁcation task is directly encoded as an equivalent bit-vector instance which, afterwards, is solved by an appropriate solving engine. The impressive improvements of SAT and SMT solvers achieved in the past years are exploited enabling the treatment of static veriﬁcation problems for signiﬁcantly larger UML/OCL instances. Furthermore, also dynamic issues (e.g. reachability of function calls) are addressed by this method. However, while the work in [9,10] provides initial implementations and ﬁrst experimental results, the description on how to encode the respective UML/OCL components into a proper bit-vector formulation was limited to basic data types and operations, respectively. In particular, non-trivial OCL constructs such as collection types and operations using them have not been introduced so far. But, this is essential in order to provide an eﬃcient solution for UML/OCL veriﬁcation tasks with full support of the modeling language. In this paper, we cover this missing link. More precisely, we show how OCL constraints can be encoded as a bit-vector instance in order to apply them to the previously introduced SAT/SMT-based veriﬁcation of UML/OCL models. Besides basic data types, we also consider more complex constructs, i.e. sets, bags, and their ordered counterparts in detail. A case study illustrates the applicability of the proposed encoding by means of a practical example. The remainder of this paper is structured as follows. Preliminaries on OCL, bit-vector logic, and the satisﬁability problem are given in the next section. Afterwards, the background on UML/OCL veriﬁcation is brieﬂy reviewed in Sect. 3 leading to the main motivation for the contribution of this paper. Section 4 eventually introduces the respective encodings of the OCL data types into an equivalent bit-vector formulation. Afterwards, the applicability of this encoding is illustrated in Sect. 5 before the paper is concluded in Sect. 6.

2

Preliminaries

To keep the paper self contained, preliminaries on OCL, bit-vector logic, and the satisﬁability problem are brieﬂy reviewed in the following. 2.1

Object Constraint Language

In UML models, the set of valid system states can be restricted by UML constraints, i.e. associations between classes. Multiplicities annotated at the

154

M. Soeken, R. Wille, and R. Drechsler inv unsold: cars->forAll(c|c.sold = False)

context sell(car: Car) pre: car.dealer = self post: car.sold = True post: cars->excludes(car)

Dealer sell(car: Car)

0..1 dealer

Stock

* cars

Car sold: Boolean

Fig. 1. Simple UML diagram enriched with OCL expressions

association ends deﬁne how the connected classes are related to each other. Further constraints, that e.g. restrict the attributes of the connected classes, cannot be expressed using UML. For this purpose, the Object Constraint Language (OCL) [11] has been employed. OCL enables to extend UML diagrams with textual constraints which further restrict the set of valid system states. OCL constrains are primarily used to complete class diagrams, interaction diagrams, and state charts, but can also be applied to activity diagrams, component diagrams, and use cases. In this work, we focus on the application of OCL in UML class diagrams. However, the proposed techniques can be transferred to applications in other diagram types as well. OCL is commonly applied to constrain diﬀerent components (e.g. classes, attributes) in a model. Thus, they are modeled as an expression which evaluates to a Boolean value, i.e. True or False. In a UML class diagram, OCL expressions appear as both invariants as well as pre- and post-conditions (associated with operations). Invariants restrict the relations between classes and values of attributes, whereas pre- and post-conditions specify in which conditions an operation can be called and how the system state is speciﬁed afterwards. Example 1. Figure 1 shows a UML class diagram for a simple car dealing model. Through the association Stock, a dealer can contain several cars, and a car can be assigned to a dealer or not. A car can be sold by a dealer using the sell method. The state whether a car is sold or not is stored in the attribute sold of class Car. One invariant unsold is attached to the class Dealer ensuring that all cars in stock must not be sold. Furthermore, the operation sell is annotated by pre- and post-conditions. In order to sell a car, the car must be assigned to the dealer. After a car is sold, the respective attribute must be updated and the car has to be removed from the stock. 2.2

Bit-Vector Logic

This paper aims to encode OCL expressions into equivalent bit-vector expressions. The deﬁnition of bit-vectors, their notation, and the applicable operations are brieﬂy reviewed in the following. Given the set of truth values IB := {0, 1}, the set IBn is referred to as bitvectors of size or dimension n. Let b ∈ IBn with b = (bn−1 . . . b1 b0 ) be a bitvector. Then the ith component of b is b[i] := bi . This index operation b[i]

Encoding OCL Data Types for SAT-Based Veriﬁcation

155

is a shorthand notation for b[i : 1] and the extraction operation is deﬁned as b[i : l] := (bi+l−1 . . . bi ). Thus, b[i : l] is a bit-vector of dimension l starting from the bit bi . The counterpart to extraction is the concatenation. Given two bit-vectors b, c ∈ IBn with b = (bn−1 . . . b0 ) and c = (cn−1 . . . c0 ), the concatenation ◦ is deﬁned as b ◦ c := (bn−1 . . . b0 cn−1 . . . c0 ). The bit-vectors are big-endian to emphasize the correspondence to natural numbers, since bit-vectors represent the binary expansion of positive numbers, e.g. 11002 = 1210 . More formally, to obtain a natural number from a bit-vector, the function nat : IBn → [0, 2n − 1] is deﬁned as nat : b →

n−1

b[i] · 2i .

(1)

i=0

The inverse function of nat is bv := nat−1 which returns the binary expansion of a natural number. Further, there are logical and arithmetic operations which can be applied to bit-vectors. Bit-wise logical operations are amongst others =, ∧, ∨, ⊕ referring to equivalence, conjunction, disjunction, and exclusive disjunction (EXOR), respectively. Analogously, arithmetic operations are amongst others ·, +, − referring to multiplication, addition, and subtraction, respectively. All arithmetic operations map into the same domain, that is e.g. a multiplication gets two n-bit bit-vectors as input and returns a n-bit bit-vector resulting value. Since obviously the result of a multiplication requires up to 2n-bit, the operations follow an overﬂow arithmetic in the general case. However, in special cases a saturation arithmetic [12] ˆ −. ˆ In this case, the maximal or minimal possible can be applied, denoted by ˆ·, +, number which can be represented is taken in case of an overﬂow or underﬂow, ˆ respectively. For example, 11002 + 11002 = 10002 , but 11002+1100 2 = 11112 . 2.3

Satisfiability Problem

The Boolean satisfiability problem (SAT) is deﬁned as the task to determine a (satisfying) assignment to all inputs of a function f so that f evaluates to 1 or to prove that no such assignment exists. More formally: Given a function f : IBn → IB, the function f is satisfiable, if and only if there exists an assignment α ∈ IBn such that f (α) = 1. In this case, α is called a satisfying assignment. Otherwise, f is unsatisfiable. Usually, the Boolean satisﬁability check is conducted on a function in conjunctive normal form. Although the SAT problem is N P-complete [13], much research was dedicated to the investigation of SAT solvers in the recent decades (see e.g. [14,15,16]). Thus, many hard instances of practical problems can be transformed into SAT problems and, afterwards, solved quite eﬃciently [17]. To further enhance the eﬃciency of satisﬁability solvers, researchers combined SAT techniques with solving strategies for higher levels of abstraction, e.g. arithmetic or bit-vector logic. This resulted in the development of solving methods for SAT Modulo Theories (SMT) [18,19]. Here, instead of a Boolean conjunctive

156

M. Soeken, R. Wille, and R. Drechsler

normal form, instances may include more complex expressions, e.g. composed of bit-vector variables and operations as introduced above. In [20], it has been demonstrated that problems having a more complex structure tend to be solved more eﬃciently when retaining the level of abstraction in the solving process. The common form of a satisﬁability problem is a conjunction, in which several constraints to be satisﬁed are deﬁned. SMT solvers exploit this structure by applying Boolean SAT solvers to handle the respective conjunction and specialized theory solvers to handle the single constraints provided in a higher abstraction. Besides that, also bit-blasting techniques (see e.g. [21]) are common to solve SMT instances. The theory of quantiﬁer-free bit-vectors (QF_BV) is an established higher abstraction which corresponds to the bit-vectors described in the previous section. The resulting problems can be formulated in the SMT-LIB ﬁle format that can be processed by oﬀ-the-shelf SMT solvers. Example 2. Given three bit-vectors a, b, c ∈ IB4 , consider the function f : IB12 → IB with f (a, b, c) = (a = b + bv(2)) ∧ (b = c · a) .

(2)

This bit-vector formula can be rewritten as an SMT instance using the bit-vectors theory as: (benchmark f :logic QF_BV :extrafuns ((a BitVec[4]) (b BitVec[4]) (c BitVec[4]))

)

:assumption (= a (bvadd b bv2[4])) :assumption (= b (bvmul c a))

When solving this instance with an SMT solver, e.g. the satisfying assignment α = (a = 00112, b = 00012, c = 10112)

(3)

is returned.

3

Problem Formulation

The design of complex systems is a non-trivial task. To ease the design process, UML provides several description models that enable to explicitly specify the system to be realized, while, at the same time, hiding concrete implementation details. Properties of the design can additionally be speciﬁed using OCL. However, even on this higher abstraction, errors frequently arise leading e.g. to (1) an over-constraint model from which no valid system state can be derived or (2) to operations which can never be executed due to too restrictive pre- or postconditions. Thus, veriﬁcation approaches are applied to check the correctness

Encoding OCL Data Types for SAT-Based Veriﬁcation Class Diagram OCL Constraints Veriﬁcation Task Problem Bounds .. .

157

Witness sat

Satisﬁability Problem unsat

verification task disproven

Fig. 2. General ﬂow for solving static veriﬁcation tasks

of a model. Typical veriﬁcation tasks include checks for consistency (in a static view) or reachability (in a dynamic view). In order to solve these veriﬁcation problems, methods based on SAT/SMT have been shown to be quite promising [9,10]. Here, the general ﬂow as depicted in Fig. 2 is applied. The given UML model together with the OCL constraints, the veriﬁcation task, and further information (e.g. the number of objects to instantiate or the number of operation calls to consider) is taken and encoded as a satisﬁability instance. Depending on the addressed veriﬁcation task, the resulting instance becomes satisﬁable if, e.g. in case of a consistency check, a valid system state exists. In contrast, if e.g. reachability is considered, the respective instance becomes satisﬁable if a sequence diagram exists conﬁrming that an operation can be executed. These solutions are called witnesses since they are witnessing the satisﬁability of the considered veriﬁcation task. For this purpose, the number of objects to consider is restricted. While this is an essential requirement for the veriﬁcation of UML/OCL models, this restriction is also justiﬁed by the fact that, eventually, the implemented system will be composed of a ﬁnite set of objects anyway. Further, the small scope hypothesis [22] supports the consideration of ﬁnite domains by stating that a large percentage of bugs can already be found by considering small state spaces. In order to solve the created satisﬁability instance, common SAT or SMT solvers are applied [16,21]. If the instance is satisﬁable, the corresponding witness can be derived from the satisfying assignment to the variables. Otherwise, it has been proven that no such witness exists within the selected problem bounds. The concrete encoding of the UML model and the OCL constraints into a proper bit-vector formulation is thereby crucial. So far, only encodings for basic data types and basic operations have been introduced. Non-trivial OCL constructs such as collection types and respective operations require more sophisticated encodings which are not available so far. Since this kind of OCL data types frequently occurs in UML/OCL models, we cover this missing link in this paper. That is, we address the following question: How can we encode OCL data types and their respective operations in bit-vector logic so that they can be applied in SAT/SMT-based UML/OCL verification?

158

4

M. Soeken, R. Wille, and R. Drechsler

Encoding of OCL Data Types

This section brieﬂy reviews the encoding of basic OCL data types into a bitvector formulation. Based on that, the encodings of OCL sets are introduced and extended for further collection types. 4.1

Basic Data Types

Already for trivial OCL data types such as Boolean variables, a special bit-vector encoding is needed. This is because, although a Boolean variable can only be set to True and False, a third value, namely ⊥ (undeﬁned), has to be considered when checking for UML/OCL consistency. Accordingly, to encode such a variable in bit-vector logic at least two bits are required. More formally: Encoding 1 (Boolean). An OCL Boolean variable b is encoded by a bit-vector b ∈ IB2 . The truth values True and False are represented by the bit-vector values 002 and 012 , respectively, whereas ⊥ is encoded as 102 . The remaining possible value 112 has to be prohibited in the satisﬁability instance by adding the bit-vector constraint (b = 112 ). The encoding of integer values is based on the same principle. However, there is another issue to consider: The domain of integer values in OCL is inﬁnite. But in order to solve veriﬁcation tasks using bit-vector logic, ﬁxed sizes have to be assumed. While, at a ﬁrst glance, this might look like an illegal simpliﬁcation, it becomes reasonable considering that, at least for the concrete implementation of the considered UML/OCL model, ﬁnite bounds are applied nevertheless. Accordingly, integers are encoded as follows: Encoding 2 (Integer). An OCL integer variable n is encoded by a bit-vector n ∈ IBl , where l is the precision assumed for integers when encoding the considered problem. Thus, n is suitable to encode l-bit integer values (except one). That is, if n is set to a value v ∈ [0, 2l − 2], then n = (nl−1 . . . n0 ) such that n is the binary expansion of n, i.e. n = bv(n). The value ⊥ is encoded by the remaining bit-vector value bv(2l − 1) = 1 . . . 12 . Note that this encoding does allow the representation of all l-bit integer except 2l − 1. If the value 2l − 1 is essential in order to check an UML/OCL model, the value of l has to be increased by 1. Then, 2l − 1 can be represented. If necessary, all other l + 1-bit values can be prohibited in the satisﬁability instance, e.g. by adding the bit-vector constraint (n = 1 . . . 12 ) ∨ (n < 10 . . . 02 ). Furthermore, note that negative values cannot be represented by this encoding. However, negative values can be enabled by substituting the binary expansion with the two-complement expansion. Another basic data type are strings. In a straight-forward view, each string can be seen as ﬁxed length sequence of characters with a terminating character such as char arrays in the C programming language. Given an l-bit character encoding and strings of maximal length n (including the terminating character), a

Encoding OCL Data Types for SAT-Based Veriﬁcation

159

bit-vector of size l ·n is required to encode the string. For ASCII strings of size 80 already 8 · 80 = 640 bits are required for each string variable. However, these bits are only required if the exact content of the string matters in the OCL constraints. This is the case, when OCL expressions such as length or startsWith are applied. If no such expressions are used, it only matters to distinguish diﬀerent string values. Then, the content of the strings can be abstracted and a more eﬃcient encoding can be applied. Encoding 3 (String). Given a system state consisting of n string variables. Then, each string variable s is encoded by a bit-vector s ∈ IBld(n+1) with ld := log2 . With this encoding, each variable can be set to a unique value (including ⊥) to distinguish them. Therewith, indeed the exact content of the string cannot be determined, but operations like comparison of two string variables can be encoded. The remaining basic data type is a real number. It has been observed that SAT instances including this kind of variables often are hard to solve [23] and, thus, should be avoided. Fortunately, the considered problems to be veriﬁed usually do not utilize real numbers since they are diﬃcult to realize both in hardware and in software. However, if real values are needed nevertheless, abstractions such as ﬁxed point or ﬂoating point numbers can be used. This can be encoded using bit-vectors of appropriate size. Using these encodings of basic data types many “standard” operations of OCL constraints like logical or arithmetical expressions can already be encoded into bit-vector logic. Therefore, existing bit-vector constraints (as the one brieﬂy sketched in Sect. 2.3) can be applied, but need to be extended to support the additional value ⊥. Example 3. Figure 3(a) shows a UML model with OCL invariants over basic data types, i.e. a string, a Boolean value, and a 32-bit integer. Considering a system state with three objects of class Car, an excerpt of the corresponding bit-vector encoding is depicted in Fig. 3(b). In Fig. 3(b), c denotes the car object being considered. Thus, these bit-vector expressions are repeated three times for each object. Car name: String expensive: Boolean price: Integer

inv i1: name.isDefined() inv i2: price > 30000 implies expensive

(a) UML diagram consisting of basic OCL data types αcname ∈ IB2 αcexpensive ∈ IB2 αcprice ∈ IB32

c i1 : α = 112 name i2 : αcprice > bv(30000) ⇒ αcexpensive = 012

(b) Encoding of the OCL invariants Fig. 3. OCL encoding of basic data types and operations

160

4.2

M. Soeken, R. Wille, and R. Drechsler

Sets

While the basic principle of encoding OCL data types into bit-vector logic has been illustrated in the previous section, the encoding of more complex OCL data types is introduced in the following by means of the Set container. Three scenarios of the car dealing example depicted in Fig. 1 are considered to illustrate the idea, namely 1. to address all objects of class Car connected to an object of class Dealer via the Stock association, i.e. by cars, 2. to state that all cars in the stock must not be sold (see the forAll operation in the invariant unsold), and 3. to state that a car is not included in the stock after it has been sold (see the excludes operation in the post-condition of the sell operation). As already discussed above, the bit-vectors that encode OCL constraints must be of a ﬁxed size in order to be suitable for SAT/SMT-based veriﬁcation. However, the cardinalities of the sets within a UML model and within the respective system states, respectively, may be of dynamic size. Thus, the total number of objects in the system state is incorporated in the bit-vector encoding. More precisely: Encoding 4 (Set). Let A be a UML class and < be a total order on the objects of class A, i.e. a0 < · · · < a|A|−1 where a0 , . . . , a|A|−1 are objects derived from class A and |A| denotes the total number of these objects. Then, each OCL variable v:Set(A) is encoded by a bit-vector v ∈ IB|A| with v = v|A|−1 . . . v0 , such that vi = 1 if and only if v.includes(ai). Example 4. Let D = {d0 , . . . , dm−1 } be the set of all objects of class Dealer and let C = {c0 , . . . , cn−1 } be the set of all objects of class Car, respectively. Using this encoding, all three scenarios mentioned above can be encoded into a bit-vector instance as follows: 1. The set cars of objects derived from class Car that are associated to class Dealer is represented by one bit-vector λdcars ∈ IBn for each object d ∈ D. In accordance with Enc. 4, d is linked to an object ci , if the corresponding bit in λdcars is set to 1. 2. The invariant unsold in Fig. 1 constrains that the sold attribute for each car associated to a dealer should be False. Although the size of the set cars is dynamic, the size of the corresponding bit-vector λdcars is ﬁxed. The invariant is modeled as a bit-vector expression as follows: n i=0

i λdcars [i] ⇒ (αcsold = 002 )

(4)

Thus, each bit in the bit-vector representing all possible elements in the set is considered. Only for those elements in the set, the invariant condition is forced. This is done by implication.

Encoding OCL Data Types for SAT-Based Veriﬁcation

161

Table 1. OCL collection types Type Set

Description Example Each element can occur at Set (b1 , b5 , b3 ) = Set (b1 , b3 , b5 ) most once OrderedSet As set, but ordered OrderedSet (b1 , b5 , b3 ) = OrderedSet (b1 , b3 , b5 ) Bag Elements may be presence Bag (b1 , b3 , b3 ) = Bag (b3 , b1 , b3 ) more than once Sequence As bag, but ordered Sequence (b1 , b3 , b3 ) = Sequence (b3 , b1 , b3 )

3. To state that the car being sold is not in the stock anymore (as constrained in the post-condition in Fig. 1), the corresponding bit in the bit-vector must be set to 0. Let ci be the parameter of the operation, then ¬λdcars [i] encodes the post-condition. 4.3

Further Collection Types

Based on the encoding of the Set data type, encodings for the remaining OCL collection types, namely OrderedSet, Bag, and Sequence, are introduced in this section. The diﬀerences of these data types are as follows: In a Set, each element can occur at most once, whereas in a Bag each element may occur more than once. For both, sets and bags, counterparts exists in which the elements follow an order, i.e. OrderedSet and Sequence, respectively. Table 1 brieﬂy summarizes the semantics of all these data types. Note that an ordered set and a sequence are ordered, but not sorted. That is, successive elements are not greater or less than the element before (see column Example in Table 1). Before outlining the encoding for ordered collections, the transformation of bags into bit-vectors is described ﬁrst. Encoding of Bags. What makes a bag diﬀerent from a set is the property that elements can occur more than once. The idea of encoding a bag is similar to the one of encoding a set. The diﬀerence is that the bits in the encoding of a set represent whether an element is contained or not. For bags, each bit is replaced by a cardinality number. More formally: Encoding 5 (Bag). Let A be a UML class and < be a total order on the objects of class A, i.e. a0 < · · · < a|A|−1 where a0 , . . . , a|A|−1 are objects derived from class A. Furthermore, it is assumed that each object occurs at most 2m times in m·|A| a bag. Then, each OCL variable v:Bag(A) is encoded by a bit-vector v ∈ IB with v = v|A|−1 . . . v0 , such that nat(vi ) = v.count(ai). The number of occurrences of objects in a bag (i.e. the respective cardinality) is thereby crucial. For sets, the total number of objects can be used as an upper bound. This is not possible for bags, since here an arbitrary number of equal objects may be contained. Thus, a reasonable upper bound of possible objects has to be deﬁned. Similar to the encoding of integer values, this is a simpliﬁcation which, however, becomes reasonable considering that at least for the concrete implementation ﬁnite bounds are applied nevertheless.

162

M. Soeken, R. Wille, and R. Drechsler

Encoding of Ordered Sets. To encode an ordered set in bit-vector logic, the position of the elements needs to be incorporated. This can be done as follows: Encoding 6 (Ordered Set). Let A be a UML class with a total order < and a set of derived objects {a0 , . . . , a|A|−1}. Then, an ordered set v:OrderedSet(A) is encoded by a bit-vector v ∈ IB|A|·l with l = ld(|A| + 1) . For each element (|A| times), l bits are devoted to encode |A| + 1 diﬀerent values, i.e. the values 0, . . . , |A| − 1 specify positions of the elements and 2l − 1 = 11 . . . 12 expresses that an element is not in the ordered set. Furthermore, the following three constraints have to be added to the satisﬁability instance in order to keep the semantics of the ordered set consistent: 1. There can be at most one element at each position, i.e. |A|−1 |A|−1 |A|−1

i=0

j=0

k=0 k=i

(v[il : l] = bv(j)) ⇒ (v[kl : l] = bv(j)) .

(5)

2. If an element is encoded to be at the j th position (with j > 0), then there must be some element at position j − 1, i.e. |A|−1 |A|−1

i=0

j=1

|A|−1

(v[i : il] = bv(j)) ⇒

v[kl : l] = bv(j − 1) .

(6)

k=0 k=i

3. Since l bits can possibly encode more than |A| + 1 values, illegal assignments must be prohibited, i.e. |A|−1

v[il : l] < bv(|A|) ∨ v[il : l] = bv(2l − 1) .

(7)

i=0

Encoding of Sequences. Sequences are the most expensive data type to encode. Using the same argumentation used within the encoding of bags, the number of elements appearing in a sequence is not limited by the system state. Thus, again a reasonable upper bound has to be determined before encoding the satisﬁability instance. Encoding 7 (Sequence). Let A be a UML class with a total order < and a set of derived objects {a0 , . . . , a|A|−1}. Then, a sequence v:Sequence(A) is encoded m by a bit-vector v ∈ IB(2 ·|A|·l) with l = ld(2m · |A| + 1) . Otherwise, the same semantics as for ordered sets apply, however, for sequences 2m · |A| possible positions have to be encoded and not just |A|, since each element can occur up to 2m times (cf. Enc. 5).

Encoding OCL Data Types for SAT-Based Veriﬁcation OCL Collection v1=Set{a2 ,a4 } v1 ∈ IB|A| IB5 v2=Bag{a0 ,a1 ,a1 ,a4 } m=2

v2 ∈ IBm·|A| IB10 v3=OrderedSet{a4 ,a0 ,a3 } l=3 v3 ∈ IB|A|·l IB15 v4=Sequence{a4 ,a0 ,a4 ,a2 ,a1 ,a4 } m

v4 ∈ IB(2

·|A|·l) l=5

IB100

163

Bit-Vector a0 a1 a2 a3 a4

0 0 1 0 1 a0 a1 a2 a3 a4

1 2 0 0 1 0 1 1 0 0 0 0 0 0 1 a0 a1 a2 a3 a4

1 7 7 2 0 0 0 1 1 1 1 1 1 1 0 1 0 0 0 0 a0 a1 a2 a3 a4 1 313131 4 313131 3 31313131313131 0 2 5 31

...

0 0 0 1 1 1 1 1 1 1

...

Fig. 4. Overview of encodings for OCL collection data types

Example 5. Figure 4 illustrates all encodings applied to a base collection A = {a0 , a1 , a2 , a3 , a4 }. For the bag and sequence, the cardinality of elements is set to 4, i.e. m = 2. The value of l is determined according to the maximal number of elements in the respective collection. Thus, for an ordered set this is l = ld(|A| + 1) = ld 6 = 3, and for a sequence it is l = ld(2m · |A| + 1) = ld 20 = 5, respectively. In case of the bag, a0 is contained once and a1 is contained twice. Thus, the respective ﬁelds in the bit-vector are 012 for a0 and 102 for a1 , respectively. 4.4

Operations on Collection Types

Having the encodings of the collection data types available, they can be used to encode the respective operations on them. Example 4 already illustrated the encoding of the excludes operation. In a similar way, this can be done for the remaining operations as well. In fact, many of the OCL operations can be mapped to a corresponding bitvector counterpart. To illustrate this, consider the encoding of a set. The elements in both, the set as well as the corresponding bit-vector encoding, are supposed to follow a total order. That is, each element in the set corresponds to a ﬁxed bit in the bit-vector. Because of this, the set-operations union and intersection can be mapped to the bit-wise disjunction and bit-wise conjunction, respectively. Analogously, this can be done for the remaining set-operations. This is summarized in detail in Table 2 which lists all set-operations together with the respective encoding for a class A with objects {a0 , . . . , an−1 } and sets v1:Set(A) as well as v2:Set(A)1. Note that the operations asBag, asOrderedSet and 1

For simplicity we omitted exceptional cases in the encodings such as the treatment of undeﬁned collections. However, they can easily be supported by adding case differentiation to the bit-vector expressions.

164

M. Soeken, R. Wille, and R. Drechsler Table 2. Encoding of set operations into bit-vector operations

Operation v1 = v2 v1 <> v2

Encoding v1 = v2 v1 = v2

v3 = v1->asBag()

v3 ∈ IBm·n s.t. v3 [j] =

v3 = v1->asOrderedSet()

v3 ∈ IBn·l s.t. v3 [il : l] =

v1 [i] if j = im, 0 otherwise.

i−1 bv if v 1 [i] = 1, j=0 v 1 [j]

bv(2l − 1) with l = ld(n + 1) v3 = v1->asSequence() see v1->asBag()->asSequence() v1->count(ai ) v1 [i] v1->excludes(ai ) ¬v1 [i] v1->excludesAll(v2) ¬v1 ∧ v2 = v2 v1->excluding(ai ) v1 ∧ ¬ bv(2i ) v1->includes(ai ) v1 [i] v1->includesAll(v2) v1 ∧ v2 = v2 v1->including(ai ) v1 ∨ bv(2i ) v1->intersection(v2) v1 ∧ v2 v1->isEmpty() v1 = bv(0) v1->notEmpty() v1 = bv(0) n−1 v1->size() i=0 v1 [i] v1->symmetricDifference(v2) v1 ⊕ v2 v1->union(v2) v1 ∨ v2

otherwise.

asSequence require thereby auxiliary variables since the operation results in a diﬀerent bit-vector domain. Example 6. Consider the operation v1->including(ai) which results in a set containing all elements of v1 and the element ai . This can be rewritten as v1->union(Set{ai}). A set containing only the element ai can be expressed as a bit-vector with only one bit set at position i, which corresponds to the natural number 2i . Using the bit-wise disjunction to express the union of two sets, the operation results in v1 ∨ bv(2i ). Accordingly, bit-vector expressions to model operations on bags are outlined in Table 3. Example 7. Consider e.g. the including transformation applied to a bag. Instead of activating the ith bit, ﬁrst all bits are erased at position i, i.e. im+m−1 v1 ∧ ¬ bv 2i , (8) k=il

before to the result of that expression ˆ · 2im bv (nat(v1 [im : m])+1)

(9)

is added by disjunction. That is, to the current amount of ai , i.e. nat(v1 [im : m]), ﬁrst 1 is added before shifting by im bits to the left so that they replace the current cardinality of ai . Further, consider the expressions for intersection and union. Both bags are element-wise concatenated, whereby for the intersection the respective minimal amount of elements and for the union the sum of both amounts is used, respectively.

Encoding OCL Data Types for SAT-Based Veriﬁcation

165

Table 3. Mappings of bag operations into bit-vector operations Operation v1 = v2 v1 <> v2 v3 = v1->asOrderedSet() v3 = v1->asSequence()

v3 = v1->asSet() v1->count(ai ) v1->excludes(ai ) v1->excludesAll(v2) v1->excluding(ai ) v1->includes(ai ) v1->includesAll(v2) v1->including(ai ) v1->intersection(v2) v1->isEmpty() v1->notEmpty() v1->size() v1->union(v2)

Mapping v1 = v2 v1 = v2 see v1->asSet()->asOrderedSet() m m v3 ∈ IB2 nl s.t. ∀n−1 ∀2j=0−1 : i=0 bv(j) + i−1 k=0 v1 [km : m] if j < nat(v1 [im : m]), v3 [i2m l + jl : l] = l bv(2 − 1) otherwise. 1 if v1 [im : m] = bv(0), v3 ∈ IBn s.t. v3 [i] = 0 otherwise. nat(v1 [im : m]) v1 [im : m] = bv(0) n−1 : m] = bv(0)) i=0 (v2 [im : m] = bv(0))

⇒ (v1 [im im+m−1 i ˆ · 2im v1 ∧ ¬ bv 2 ∨ bv nat(v1 [im : m]−1) k=im v1 [im : m] = bv(0) n−1 : m] = bv(0)) i=0 (v2 [im : m] = bv(0))

⇒ (v1 [im im+m−1 i ˆ · 2im v1 ∧ ¬ bv 2 ∨ bv (nat(v 1 [im : m])+1) k=im n−1 i=0 min {nat(v1 [im : m]), nat(v2 [im : m])} v1 = bv(0) v1 = bv(0) n−1 i=0 nat(v1 [im : m]) ˆ n−1 i=0 v1 [im : m]+v2 [im : m]

( is concatenation)

The mappings for operations on ordered sets are given in Table 4 considering ordered sets with at most n elements and l as described in Enc. 6. The function maxpos is used in some operations and returns the largest index in the ordered set. The function is deﬁned as n−1 maxpos(v) := max nat(v[kl : l]) | v[kl : l] = bv(2l − 1) . (10) k=0

Note that in OCL, the ﬁrst element in an ordered set has the index 1, while in the encoding the ﬁrst index is 0 due to advantages in the implementation. The bit-vector expressions for the OCL operations on ordered sets is described by the means of two examples. Example 8. Consider the operation v1->at(k) in Table 4. According to the encoding deﬁned in Enc. 6, the bit-vector is subdivided into several ﬁelds, where each ﬁeld corresponds to one item of all available items. The ﬁeld contains an index describing either the position of that item in the ordered set or the value bv(2l − 1) if the item is not contained in the ordered set. Thus, the ﬁeld containing the required position k has to be found: For each position in the encoding, the content is compared to the index with v1 [il : l] = bv(k − 1). This either evaluates to 0 or, in one case, to 1 assuming that v1 contains at most k items. Multiplying the result with bv(2l − 1), i.e. a bit-vector containing l ones, results in either a bit-vector containing only zeros or ones. This bit-vector is used as a bit-mask for the considered position, i.e. bv(i), and all these bitvectors are added. Since only one bit-vector does not contain of all zeros, which is the bit-vector containing the item, the result is a bit-vector encoding the item at position k.

166

M. Soeken, R. Wille, and R. Drechsler Table 4. Mappings of ordered set operations into bit-vector operations

Operation v1 = v2 v1 <> v2

Mapping v1 = v2 v1 = v2

v1->append(ai )

v1 [jl : l] =

v1->count(ai ) v1->excludes(ai ) v1->excludesAll(v2)

v1 [il : l] = bv(2l − 1) v1 [il : l] = bv(2l − 1)

n−1 v2 [il : l] = bv(2l − 1) ⇒ v1 [il : l] = bv(2l − 1) i=0

bv (maxpos(v1 ) + 1) if j = i ∧ v1 [jl : l] = bv(2l − 1), v1 [jl : l] otherwise. v3 = v1->asBag() see v1->asSet()->asBag() v3 = v1->asSequence() see v1->asSet()->asBag()->asSequence() 1 if v1 [il : l] = bv(2l − 1), v3 = v1->asSet() v3 ∈ IBn s.t. v3 [i] = 0 otherwise.

n−1 v1->at(k) (v1 [il : l] = bv(k − 1)) · bv(2l − 1) ∧ bv(i) i=0

v1->excluding(ai ) v1->first() v1->includes(ai ) v1->includesAll(v2) v1->including(ai ) v1->indexOf(ai ) v1->insertAt(k, ai ) v1->isEmpty() v1->last() v1->notEmpty() v1->prepend() v1->size()

l v1 ∨ bv(2 − 1) · 2il

n−1 (v1 [jl : l] = bv(0)) · bv(2l − 1) ∧ bv(j) j=0

v1 [il : l] = bv(2l − 1)

n−1 v2 [il : l] = bv(2l − 1) ⇒ v1 [il : l] = bv(2l − 1) i=0 see v1->append(ai ) nat(v1 [il : l]) ⎧+1 if j = i ∧ v1 [jl : l] = bv(2l − 1), ⎨ bv(k − 1) v1 [jl : l] = v1 [jl : l] if v1 [jl : l] < bv(k) ∨ v1 [jl : l] = bv(2l −1), ⎩ v1 [jl : l] + bv(1) otherwise. ln v1 = bv(2 − 1)

n−1 (v1 [jl : l] = bv(maxpos(v1 )) · bv(2l − 1) ∧ bv(j) j=0 v1 = bv(2ln⎧− 1) if j = i ∧ v1 [jl : l] = bv(2l − 1), ⎨ bv(0) v1 [jl : l] = bv(2l − 1) if v1 [jl : l] = bv(2l − 1), ⎩ v1 [jl : l] + bv(1) otherwise. bv(maxpos(v1 )) + 1

On the other hand, the operation v1->indexOf(ai) is encoded straight forward. Since the ﬁeld corresponding to ai can be determined directly by v1 [il : l], the result is its natural representation incremented by 1. We omitted the detailed table of bit-vector expressions for operations on sequences due to page limitations. However, they can be derived by combining the bit-vector expressions for the respective operations on bags and ordered sets.

5

Case Study

In this section, we illustrate the application of the proposed encoding by means of a case study. Therefore, the UML/OCL model depicted in Fig. 5 and representing a car dealing scenario is considered. A car dealer (Dealer ) oﬀers cars (Car ) according to a preferred color and preferred type. The associations CarsOfColor and CarsOfType are used to model which cars belong to the dealer (distinguished with respect to the color and the type, respectively). A car dealer has at least one car by color and by type, and each car can only be assigned to one dealer. In the following, selected OCL invariants for this model along with the resulting bit-vector encoding are outlined. The respective veriﬁcation task is to generate a valid system state composed of three dealers, i.e. three objects D =

Encoding OCL Data Types for SAT-Based Veriﬁcation

167

CarsOfColor colorDealer 0..1

1..* colorCars

Dealer prefColor: Color prefType: Type

Car color: Color type: Type

typeDealer 0..1

1..* typeCars

enum

Color red yellow green

enum

Type coupet convertible suv

CarsOfType inv inv inv inv inv

defined: prefColor.isDefined() and prefType.isDefined() carsHaveSameColor: colorCars->forAll(c|c.color = prefColor) carsHaveSameType: typeCars->forAll(c|c.type = prefType) disjointSets: typeCars->intersection(colorCars)->size() = 0 competition: Dealer.allInstances()->forAll(s|s <> self implies ((s.prefColor <> prefColor) and (s.prefType <> prefType))) inv balance: Dealer.allInstances()->forAll(s|s.colorCars->union(s.typeCars)->size() = colorCars->union(typeCars)->size()) inv defined: color.isDefined() and type.isDefined() inv oneCategory: colorDealer.isUndefined() or typeDealer.isUndefined() inv mustBeAssigned: colorDealer.isDefined() or typeDealer.isDefined()

Fig. 5. Car dealing example

{d0 , d1 , d2 } derived from class Dealer, and 15 cars, i.e. objects C = {c0 , . . . , c14 } derived from class Car. To encode the attributes, we introduce the bit-vector variables αdprefColor ∈ IB2 for each d ∈ D. Other bit-vectors are created accordingly for the other attributes of the class diagram. In the same manner as in Example 4, bit-vectors are created for the associations, i.e. λdcolorCars ∈ IB15 for each d ∈ D. The defined invariants for both classes can be encoded according to Fig. 3. Next, the invariants carsHaveSameColor and carsHaveSameType ensure that cars who are in the stock of a dealer must meet the preferred color or type respective to the association. The respective bit-vector encoding for the invariant carsHaveSameColor reads as follows: |C|−1

∀d ∈ D :

i = αdprefColor λdcolorCars [i] ⇒ αccolor

(11)

i=0

The bit-vector encoding for the invariant carsHaveSameType is formulated analogously. The invariant disjointSets assures that cars are either connected by their color or by their type, i.e. the intersection of colorCars and typeCars must be empty for each dealer. Using the encodings suggested in Table 2, the following bit-vector expression results: |C|−1

∀d ∈ D :

λdtypeCars ∧ λdcolorCars [i] = 0 i=0

(12)

168

M. Soeken, R. Wille, and R. Drechsler

In this expression, the number of bits of the intersection (bit-wise conjunction) are counted and forced to be 0. To ensure a variety of car dealers, the competition invariant is added to ensure that there are no dealers with the same preferred color or type. This is encoded as:

(d = d ) ⇒ αdprefColor = αdprefColor ∧ αdprefType = αdprefType ∀d ∈ D : d ∈D

(13) The invariant balance ensures that all dealers have the same number of cars, regardless of whether by color or by type. Thus, the size of the unions of both sets are compared: ⎛ ⎞ |C|−1

|C|−1 ⎝ ∀d ∈ D : λdcolorCars ∨ λdtypeCars [i]⎠ λdcolorCars ∨ λdtypeCars [i] = d ∈D

i=0

i=0

(14) The invariant disjointSets assures that one car cannot be used both by color and by type for one dealer. However, using the invariants introduced so far, a car can still be assigned by color to one dealer and by type to another one. To prevent this, the invariant oneCategory is added to the Car class, stating that one of the association ends has to be undeﬁned: ∀c ∈ C : λccolorDealer = 112 ∨ λctypeDealer = 112

(15)

Note that in this case the λ variables are not interpreted as bit-vectors but as natural numbers directly pointing to the dealer object. This is done, since a car can only be assigned to at most one dealer. In this sense, the λ vectors can be interpreted analogously to the α attribute vectors. Finally, we want to assign each car to a dealer by car or by color. Analogously to the previous invariant, in this case it has to be assured that at least one association end is deﬁned resulting in the mustBeAssigned invariant, encoded as: ∀c ∈ C : λccolorDealer = 112 ∨ λctypeDealer = 112

(16)

Solving the resulting satisﬁability instance with an SMT solver such as Boo8 = lector [21], a satisfying assignment is returned, amongst others assigning αccolor c11 d0 002 , αtype = 102 , and λcolorCars = 00010 01000 000002. From the assignments, a system state can be constructed, e.g. Car8 is assigned the color red, Car11 is assigned the type suv, and Dealer0 is connected via CarsOfColor to Car8 and Car11. Together with other assignments, an object diagram representing the system state can be obtained. This is partially depicted in Fig. 6. Here, one dealer together with its connected cars and all attribute assignments is shown.

Encoding OCL Data Types for SAT-Based Veriﬁcation

Car8

Dealer0

color: red type: coupet

CarsOfColor

CarsOfType

prefColor: red prefType: coupet

169

Car11: CarsOfColor

CarsOfType

color: red type: suv

CarsOfType

Car4

Car5

Car13

color: red type: coupet

color: red type: coupet

color: red type: coupet

Fig. 6. Derived system state

Solving this particular problem with 15 cars the solver Boolector [21] requires less than 0.1 seconds to determine the solution on an Intel 2.26 GHz Core 2 Duo processor with 3 GB main memory. Scaling the example to determine a solution with 150 cars, the solver takes 6.9 seconds. Further experiments with run-times can be found in [9,10].

6

Conclusion

In this work, encodings for both OCL basic and collection data types have been presented. The encoding of these data types and their operations into bit-vector expressions enables their application in satisﬁability based veriﬁcation techniques proposed in the past. OCL and satisﬁability instances follow diﬀerent design paradigms. One example is the number of variables and their size, which is dynamic in OCL, whereas it must be deﬁned initially with a static bit-width in a satisﬁability instance. This leads to non trivial encodings for both, the data types and their operations. The applicability of the encodings has been demonstrated in a case study by means of a practical example.

References 1. Rumbaugh, J., Jacobson, I., Booch, G.: The Uniﬁed Modeling Language reference manual. Addison-Wesley Longman, Essex (1999) 2. Vanderperren, Y., Müller, W., Dehaene, W.: UML for electronic systems design: a comprehensive overview. Design Automation for Embedded Systems 12(4), 261–292 (2008) 3. Kyas, M., Fecher, H., de Boer, F.S., Jacob, J., Hooman, J., van der Zwaag, M., Arons, T., Kugler, H.: Formalizing UML Models and OCL Constraints in PVS. Electronic Notes in Theoretical Computer Science 115, 39–47 (2005) 4. Beckert, B., Hähnle, R., Schmitt, P.: Veriﬁcation of Object-Oriented Software: The KeY Approach. Springer, Secaucus (2007) 5. Gogolla, M., Kuhlmann, M., Hamann, L.: Consistency, Independence and Consequences in UML and OCL Models. In: Tests and Proof, pp. 90–104. Springer, Heidelberg (2009)

170

M. Soeken, R. Wille, and R. Drechsler

6. Cabot, J., Clarisó, R., Riera, D.: Veriﬁcation of UML/OCL Class Diagrams using Constraint Programming. In: IEEE Int. Conf. on Software Testing Veriﬁcation and Validation Workshop, pp. 73–80 (April 2008) 7. Cabot, J., Clarisó, R., Riera, D.: Verifying UML/OCL Operation Contracts. In: Leuschel, M., Wehrheim, H. (eds.) IFM 2009. LNCS, vol. 5423, pp. 40–55. Springer, Heidelberg (2009) 8. Anastasakis, K., Bordbar, B., Georg, G., Ray, I.: UML2Alloy: A Challenging Model Transformation. In: Int. Conf. on Model Driven Engineering Languages and Systems, pp. 436–450. Springer, Heidelberg (2007) 9. Soeken, M., Wille, R., Kuhlmann, M., Gogolla, M., Drechsler, R.: Verifying UML/OCL models using Boolean satisﬁability. In: Design, Automation and Test in Europe, pp. 1341–1344. IEEE Computer Society, Los Alamitos (2010) 10. Soeken, M., Wille, R., Drechsler, R.: Verifying Dynamic Aspects of UML Models. In: Design, Automation and Test in Europe. IEEE Computer Society, Los Alamitos (2011) 11. Warmer, J., Kleppe, A.: The Object Constraint Language: Precise modeling with UML. Addison-Wesley Longman, Boston (1999) 12. Constantinides, G.A., Cheung, P.Y.K., Luk, W.: Synthesis of Saturation Arithmetic Architectures. ACM Trans. Design Autom. Electr. Syst. 8(3), 334–354 (2003) 13. Cook, S.A.: The complexity of theorem-proving procedures. In: ACM Symp. on Theory of Computing, pp. 151–158. ACM, New York (1971) 14. Moskewicz, M.W., Madigan, C.F., Zhao, Y., Zhang, L., Malik, S.: Chaﬀ: Engineering an Eﬃcient SAT Solver. In: Design Automation Conference, pp. 530–535. ACM, New York (2001) 15. Goldberg, E.I., Novikov, Y.: BerkMin: A Fast and Robust Sat-Solver. In: Design, Automation and Test in Europe, pp. 142–149. IEEE Computer Society, Los Alamitos (2002) 16. Eén, N., Sörensson, N.: An Extensible SAT-solver. Theory and Applications of Satisﬁability Testing, 502–518 (May 2003) 17. Biere, A., Heule, M.J.H., van Maaren, H., Walsh, T. (eds.): Handbook of Satisﬁability, February 2009. IOS Press, Amsterdam, NL (February 2009) 18. Armando, A., Castellini, C., Giunchiglia, E.: SAT-Based Procedures for Temporal Reasoning. In: Biundo, S., Fox, M. (eds.) ECP 1999. LNCS, vol. 1809, pp. 97–108. Springer, Heidelberg (2000) 19. Ganzinger, H., Hagen, G., Nieuwenhuis, R., Oliveras, A., Tinelli, C.: DPLL(T): Fast Decision Procedures. In: Alur, R., Peled, D.A. (eds.) CAV 2004. LNCS, vol. 3114, pp. 175–188. Springer, Heidelberg (2004) 20. Wille, R., Große, D., Soeken, M., Drechsler, R.: Using Higher Levels of Abstraction for Solving Optimization Problems by Boolean Satisﬁability. In: IEEE Symp. on VLSI, pp. 411–416. IEEE Computer Society, Los Alamitos (2008) 21. Brummayer, R., Biere, A.: Boolector: An Eﬃcient SMT Solver for Bit-Vectors and Arrays. In: Tools and Algorithms for Construction and Analysis of Systems, pp. 174–177. Springer, Heidelberg (2009) 22. Jackson, D., Damon, C.: Elements of Style: Analyzing a Software Design Feature with a Counterexample Detector. IEEE Trans. on Software Engineering 22(7), 484–495 (1996) 23. Davenport, J.H., Heintz, J.: Real Quantiﬁer Elimination is Doubly Exponential. Journal of Symbolic Computation 5(1-2), 29–35 (1988)

State Coverage Metrics for Specification-Based Testing with B¨ uchi Automata Li Tan School of Electrical Engineering and Computer Science Washington State University, Richland, WA 99354 [email protected]

Abstract. B¨ uchi automata have been widely used for specifying linear temporal properties of reactive systems and they are also instrumental for designing eﬃcient model-checking algorithms. In this paper we extend speciﬁcation-based testing to B¨ uchi automata. A key question in speciﬁcation-based testing is how to measure the quality (relevancy) of test cases with respect to system speciﬁcation. We propose two state coverage metrics for measuring how well a test suite covers a B¨ uchi-automaton-based requirement. We also develop test generation algorithms that use counter-example generation capability of an oﬀ-theshelf model checker to generate test cases for the coverage criteria inferred by these metrics. In our experiment we demonstrate the feasibility and performance of the coverage criteria and test generation algorithms for these criteria. In [13] we proposed testing coverage metrics and criteria for properties in Linear Temporal Logic (LTL) and referred to the new approach as property-coverage testing. This research shares the same motivation as in [13] and it extends property-coverage testing to speciﬁcations in B¨ uchi automata. Since automaton minimization techniques can be used to reduce the structural diversity of semantically equivalent B¨ uchi automata, we argue that a coverage metric based on B¨ uchi automata is less susceptible to syntactic changes of a property than a LTLbased coverage metric, and hence the proposed coverage metrics measure the relevancy of a test suite to the semantics of a linear temporal property. We also discuss an algorithm for reﬁning a B¨ uchi-automaton-based requirement based on its strong state coverage metric. Our experiment demonstrates the feasibility and performance of our coverage criteria and test generation algorithms.

1

Introduction

Testing and formal veriﬁcation are considered as two important and yet complemental methods in verifying and validating reactive systems. Reactive systems refer to those dynamic systems that continuously operate and interact with their environment. Examples of reactive systems include engine control modules (ECMs) in automobiles, and autopilot modules in airplanes. Ensuring the correct functioning of reactive systems is of uttermost interest to automobile, aerospace, and many other industries where reactive systems are widely deployed in safe- and/or M. Gogolla and B. Wolﬀ (Eds.): TAP 2011, LNCS 6706, pp. 171–186, 2011. c Springer-Verlag Berlin Heidelberg 2011

172

L. Tan

mission-critical applications. Whereas testing is to check the behavior of a system under a controlled environment, formal veriﬁcation is to algorithmically analyze a system. A critic of testing may be best summarized by Dijkstra’s notable statement “testing shows the presence, not the absence of bugs”. Nevertheless, despite this shortcoming, testing can work where formal veriﬁcation stops short. Compared with formal veriﬁcation, testing usually has a better scalability, and it can be applied to implementation directly, for instance, in a setting such as hardwarein-loop testing. In the foreseeable future testing will continue to play a predominant role on validating and verifying reactive systems. A direction of our research is to study how to harness the synergy of testing and formal veriﬁcation. For instance, formal veriﬁcation techniques such as model checking (cf. [3]) have enjoyed a great deal of successes in past two decades. As the result, rigorous formalisms such as temporal logics and B¨ uchi automata are increasingly popular for specifying requirements for high-dependable reactive systems. A research question is how testing can beneﬁt from the proliferation of these high-quality formal speciﬁcations. One of important formal veriﬁcation techniques is linear temporal model checking (cf. [15]), in which a system design is checked against a linear temporal property. B¨ uchi automata have been used for specifying linear temporal properties. Since other formalisms such as Linear Temporal Logic (LTL) can be translated into B¨ uchi automata, B¨ uchi automata are also used as a uniﬁed theoretical platform for reasoning about linear temporal model checking algorithms. For instance, eﬃcient linear temporal model checking algorithms such as those in [5,6] have been proposed with the use of B¨ uchi automata. The purpose of this research is to develop a framework for speciﬁcation-based testing with B¨ uchi automata. We will address the following issues: 1. We need to deﬁne the relevancy of a test case to a speciﬁcation in a B¨ uchi automata. For this purpose, we propose two variants of state coverage metrics that measure how a B¨ uchi automaton is covered during a test. A weak variant indicates that a particular state may be reached during a test, whereas a strong variant requires that the state must be reached during a test; 2. We need to develop a practical way to produce test cases for the testing criteria inferred by the proposed metrics. For this purpose, we propose the algorithms that can use the counterexample mechanism of an oﬀ-the-shelf model checker to generate test cases for state coverage criteria. In addition, we consider property reﬁnement based on the proposed metrics. Lack of state coverage may be caused by incorrect/incomplete implementation, and/or imprecise/loose speciﬁcation. Whereas testing helps identify the ﬁrst problem, a careful examination and reﬁnement of the speciﬁcation will address the latter issue. We will discuss how to enhance our test generation algorithms to systematically reﬁne a property expressed as a B¨ uchi automaton. The rest of the paper is organized as follows: Section 2 prepares the notations that will be used in the rest of the paper; Section 3 proposes two variants of state coverage metrics for B¨ uchi automata; Section 4 gives the algorithms for generating tests for state coverage using a model checker; Section 5 discusses

State Coverage Metrics for Speciﬁcation-Based Testing

173

the property reﬁnement based on state coverage metrics; Section 6 discusses the experimental results; and ﬁnally Section 7 concludes the paper. Related Works. Test generation using model checkers attracted much research eﬀorts in recent years as a way to harness the synergy of testing and model checking. The ﬁrst and foremost question in model-checking-based test generation is how to formulate test generation as a model checking problem, more speciﬁcally, how to encode a testing criterion as a temporal property accepted by a model checker. It has been shown that traditional structural coverage criteria such as Modiﬁed Condition/Decision Coverage (MCDC) can be encoded in Computational Tree Logic (CTL) (cf. [4]), which can be used by a model checker such as NuSMV for generating tests. In [2] Calvagna et al encoded several combinatorial testing criteria in Linear Temporal Logic (LTL), which was then used by the model checker SAL to generate tests. In [8] Hong et al expressed dataﬂow criteria in CTL. These previous works emphasize on applying model-checking-based test generation to existing testing criteria. In contrast, in [13] we considered speciﬁcation-based with temporal logic requirements. We proposed a coverage metric that measures how well a linear temporal logic requirement was covered during a test. Our coverage metric in [13] is inspired by the notion of (non-) vacuity in [10]. Intuitively the vacuity-based coverage criterion requires that a test suite tests the relevancy of each subformula of a LTL property to a system. For a comparison of these techniques, interested readers may refer to [4]. This work can be seen as an extension of our previous work in [13]. Whereas our vacuity-based coverage metric helps deﬁne and develop test cases relevant to temporal speciﬁcations in LTL, a feature but also a critic of the coverage metric is that it heavily depends on syntactical structures of LTL formulae. For example, the LTL formula f0 : G(brake ⇒ F stop) ∧ (brake ⇒ F stop) is semantically equivalent to f1 : G(brake ⇒ F stop). Yet for vacuity-based coverage metric, the coverage of a test case for f0 always subsumes its coverage for f1 . Deﬁning coverage metrics based on B¨ uchi automata will help alleviate this syntactical dependency. Moreover, there are several existing algorithms for minimizing B¨ uchi automata, which can be used as a preprocess to further reduce syntactical variance of otherwise semantically equivalent B¨ uchi automata. In addition, in this paper we will also discuss property reﬁnement based on the proposed coverage metrics.

2 2.1

Preliminaries Kripke Structures, Traces, and Tests

We model systems as Kripke structures. A Kripke structure is a ﬁnite transition system in which each state is labeled with a set of atomic propositions. Semantically atomic propositions represent primitive properties held at a state. Deﬁnition 1 formally deﬁnes Kripke structures. Definition 1 (Kripke Structures). Given a set of atomic proposition A, a Kripke structure is a tuple V, v0 , →, V, where V is the set of states, v0 ∈ V is

174

L. Tan

the start state, →⊆ V × V is the transition relation, and V : V → 2A labels each state with a set of atomic propositions. We write v → v in lieu of v, v ∈→. We let a, b, · · · range over A. We denote A¬ for the set of negated atomic propositions. Together, P = A ∪ A¬ deﬁnes the set of literals. We let l1 , l2 , · · · and L1 , L2 , · · · range over P and 2P , respectively. We use the following notations for sequences: let β = v0 v1 · · · be a sequence, we denote β[i] = vi for i-th element of β, β[i, j] for the subsequence vi · · · vj , and β (i) = vi · · · for the i-th suﬃx of β. A trace τ of the Kripke structure V, v0 , →, V is deﬁned as a maximal sequence of states starting with v0 and respecting the transition relation →, i.e., τ [0] = v0 and τ [i − 1] → τ [i] for every i < |τ |. We also extend the labeling function V to traces: V(τ ) = V(τ [0])V(τ [1]) · · ·. Definition 2 (Lasso-Shaped Sequences). A sequence τ is lasso-shaped if it has the form α(β)ω , where α and β are finite sequences. |β| is the repetition factor of τ . The length of τ is a tuple |α|, |β|. Definition 3 (Test and Test Suite). A test is a word on 2A , where A is a set of atomic propositions. A test suite ts is a finite set of test cases. A Kripke structure K = V, v0 , →, V passes a test case t if K has a trace τ such that V(τ ) = t. K passes a test suite ts if and only if it passes every test in ts. 2.2

Generalized B¨ uchi Automata

Definition 4. A generalized B¨ uchi automaton is a tuple S, S0 , Δ, F , in which S is a set of states, S0 ⊆ S is the set of start states, Δ ⊆ S × S is a set of transitions, and the acceptance condition F ⊆ 2S is a set of sets of states. uchi automaton is an We write s → s in lieu of s, s ∈ Δ. A generalized B¨ ω-automaton. That is, it can accept the inﬁnite version of regular languages. A run of a generalized B¨ uchi automaton B = S, S0 , Δ, F is an inﬁnite sequence ρ = s0 s1 · · · such that s0 ∈ S0 and si → si+1 for every i ≥ 0. We denote inf(ρ) for a set of states appearing for inﬁnite times on ρ. A successful run of B is a run of B such that for every F ∈ F , inf(ρ) ∩ F = ∅. The plain version of generalized B¨ uchi automata in Deﬁnition 4 deﬁnes successful runs as sequences of states. In order to accept inﬁnite words, we need to extend a generalized B¨ uchi automaton with an alphabet. The alphabet is a set of sets of atomic propositions. It shall be noted that there are more than one way to extend a generalized B¨ uchi automaton with an alphabet. One may label states with a set of sets of atomic propositions, as in [5], or label transitions with a set of sets of atomic propositions, as in [6]. Just like Moore and Mealy versions of ﬁnite systems, these two representations are equivalent. In this paper we will use state labeling approach in [5] with one modiﬁcation: instead of labeling a state with a set of sets of atomic propositions in [5], we label the state with a set of literals. A set of literals is a succinct representation of a set of sets of atomic propositions: let L be a set of literals labeling state s, then semantically s is labeled with a set of sets of atomic propositions Λ(L), where

State Coverage Metrics for Speciﬁcation-Based Testing

175

Λ(L) = {A ⊆ A | (A ⊇ (L ∩ A)) ∧ (A ∩ (L ∩ A¬ ) = ∅)}, that is, every set of atomic propositions in Λ(L) must contain all the atomic propositions in L but none of its negated atomic propositions. In the rest of the paper, generalized B¨ uchi automata (GBA) refer to labeled generalized B¨ uchi automata in Deﬁnition 5. Definition 5. A labeled generalized B¨ uchi automaton is a tuple P, S, S0 , Δ, L, F , in which S, S0 , Δ, F is a generalized B¨ uchi automaton, P is a set of literals, and the label function L : S → 2P maps each state to a set of literals. A GBA B = A ∪ A¬ , S, S0 , Δ, L, F accepts inﬁnite words over the alphabet 2A . Let α be a word on 2A , B has a run ρ induced by α, written as α ρ, if and only if for every i < |α|, α[i] ∈ Λ(L(ρ[i])). B accepts α, written as α |= B if and only if B has a successful run ρ such that α ρ. Generalized B¨ uchi automata are of special interests to the model checking community. Because a GBA is an ω-automaton, it can be used to describe temporal properties of a ﬁnite-state reactive system, whose executions are inﬁnite words of an ω-language. Formally, a GBA accepts a Kripke structure K = V, v0 , →, V if for every trace τ of K, V(τ ) |= B. Eﬃcient B¨ uchi automaton-based algorithms have been developed for linear temporal model checking. The process of linear temporal model checking generally involves translating the negation of a linear temporal logic property φ to a GBA B¬φ , and then checking the emptiness of the product of the GBA and the Kripke structure K. If the product automaton is not empty, then a model checker usually outputs an accepting trace of the product automaton, which serves as a counterexample to K |= φ.

3

State Coverage for Generalized B¨ uchi Automata

Definition 6 (Covered States). Given a generalized B¨ uchi automaton B = P, S, S0 , Δ, L, F , a test t weakly covers a state s if B has a successful run ρ such that t ρ and s is on ρ. A test t strongly covers a state s if B accepts t and for B’s every successful run ρ such that t ρ, s is on ρ. Since a GBA B may have nondeterministic transitions, B may have more than one successful run induced by a test. A weakly covered state s shall appear on some successful run induced by t, whereas a strongly covered state s has to appear on every successful run induced by t. By imposing the additional requirement that B accepts t, Deﬁnition 6 also requires that at least one of such successful runs exists for a state s strongly covered by t. Definition 7 (Weak State Coverage Metrics and Adequacy Criterion). Given a generalized B¨ uchi automaton B = P, S, S0 , Δ, L, F , let S ⊆ S be a set of states, the weak state coverage metric for a test suite ts on S is defined as |S | , where S = {s | s ∈ S ∧ ∃t ∈ ts.(t weakly covers s)}. ts weakly covers S if |S| and only if S = S.

176

L. Tan

Definition 8 (Strong State Coverage Metrics and Adequacy Criterion). Given a generalized B¨ uchi automaton B = P, S, S0 , Δ, L, F , let S ⊆ S be a set of states, the strong state coverage metric for a test suite ts on S is | defined as |S |S| , where S = {s | s ∈ S ∧ ∃t ∈ ts.(t strongly covers s)}. ts strongly covers S if and only if S = S. Theorem 1 shows that the strong state coverage criterion subsumes the weak path coverage criterion. Theorem 1. Let S be a set of states of a GBA B = P, S, S0 , Δ, L, F , if a test suite ts strongly covers S, then ts also weakly covers S. Proof. Since ts strongly covers S, by Deﬁnition 8, for every s ∈ S, there exists a t such that (i) B accepts t; and (ii) for every B’s successful run ρ such that t ρ, s is on ρ. By (i) and (ii), B has at least one successful run ρ such that t ρ and s on ρ. Therefore, by Deﬁnition 8, ts also weakly covers S. 2

4

Model-Checking-Based Test Generation

Model checking via generalized B¨ uchi automata is a well-studied subject and eﬃcient algorithms have been developed over years (cf. [5]). We consider the approach that uses a B¨ uchi automaton-based model checker to generate test cases for model-based testing. Model-based testing is an important component in the workﬂow of model-based design. In model-based design, engineers work on mathematical models of systems. These mathematical models are the abstractions of ﬁnish products. Model-based design helps improve the quality of ﬁnish products by supporting veriﬁcation and validation at early design stage. Model-based testing extends this beneﬁt by supporting eﬃcient test generations from design models and then applying generated test suites to ﬁnish products. The inputs to our test generation algorithms are (a model of ) a transition system and a linear temporal property in GBA. A system model is a behaviorial abstraction of the system. In this paper we consider the path abstraction in Deﬁnition 9. We denote Ks ≺ Km , if Km is a path abstraction of Ks . In modelbased design, a system model exists as part of design artifact so we can use it directly as an input to our test generation algorithms. Definition 9 (Path Abstraction). Let Km = Sm , s0m , →m , Vm and Ks = Ss , s0s , →s , Vs be two Kripke structures, Km is a path abstraction of Ks , written as Ks ≺ Km if and only if for every trace τs of Ks , there is a trace τm of Km such that Vs (τs ) = Vm (τm ). Theorem 2. Given two Kripke structures Km and Ks such that Ks ≺ Km , if Km passes a test suite ts, then Ks also passes ts. Proof. Since Km passes the test suite ts, for every t ∈ ts, there is a trace τm of Km such that Vm (τm ) = t, where Vm is Km ’s labeling function. By Deﬁnition

State Coverage Metrics for Speciﬁcation-Based Testing

177

9, there exists a trace τs of Ks such that Vs (τs ) = Vm (τm ), therefore, Ks passes t and hence ts. 2 By Theorem 2, a test suite generated for a design model shall also be passed by its implementation, if the design model is a path abstraction of the implementation. In model-based testing a test suite is generated from a design model and then applied to the actual implementation. We will generate a test suite by utilizing the counterexample capability of a linear temporal model checker. As we discussed before, the ﬁrst and foremost question in model-checking-based test generation is to formulate test generation as a model-checking problem. In our case, we need to translate state coverage criteria to temporal properties accepted by a linear temporal model checker. These temporal properties are also referred to as “trapping properties”. Since we express temporal properties as GBAs, we will also formulate these “trapping properties” as GBAs. In our approach a model checker takes ( the negation of ) a trapping property in GBA and a system model, and produces a counterexample that is essentially a trace satisfying a given state coverage. The trapping property for generating a test case weakly covering a state sm is given as a state marking generalized B¨ uchi automaton (SM-GBA) in Deﬁnition 10. Definition 10 (State Marking Generalized B¨ uchi Automata (SM-GBA)). Let B = P, S, S0 , Δ, L, F be a GBA, a state marking generalized B¨ uchi automaton for state sm ∈ S is a GBA B(sm ) = P, S × {0, 1}, S0 × {0}, Δ , L , F , where, – Δ =

s,s ∈Δ {(s, 0), (s

, 0), (s, 1), (s , 1)} ∪ sm ,s ∈Δ {(sm , 0), (s , 1)} ;

– For every s ∈ S, L ((s, 0)) = L ((s, 1)) = L(s); – F = F ∈F {F × {1}}.

A SM-GBA indexes each state of the original GBA with a number from {0, 1}. The start states are always indexed with 0. The ﬁnal states are always indexed with 1. The indexing number will be changed from 0 to 1 on the outgoing transitions from the marked state sm . By the construction of the SM-GBA, the only way that a run can reach an acceptance state from a start state is through state sm . Therefore, every successful run of the SM-GBA must have sm on it. Algorithm 1 generates a test suite that weakly covers all the states of a given GBA. A B¨ uchi-automaton-based linear temporal model checker like SPIN [7] veriﬁes the system model against a linear temporal property in two stages: ﬁrst it builds a GBA for the negation of a given property, and then it checks the emptiness of the product of the GBA and the system model. If the system model satisﬁes the linear temporal property, then the product of the GBA and the system model shall be empty, that is, the product does not accept any word. The emptiness checking is at the core of many B¨ uchi-automaton-based model checkers. Algorithm 1 uses the core emptiness checking algorithm of an existing model checker, which is represented by the function M C isEmpty. By our deﬁnition function M C isEmpty returns an empty word if the product of the GBA and the system model is empty, otherwise it returns a successful run of the product of the GBA and the system model Km . The test case obtained

178

L. Tan

Algorithm 1. TestGen WSC(B = P, S, S0 , Δ, L, F , Km = S, s0 , →, V) Require: B is GBA, Km is a system model, and Km satisﬁes B; Ensure: Return the test suite ts that weakly covers all the states of B and Km passes ts. Return ∅ if such a test suite is not found; 1: for every s ∈ S do 2: Construct a SM-GBA B(s) from B that marks the state s; 3: τ = M C isEmpty(B(s), Km ); 4: if |τ | = 0 then 5: ts = ts ∪ {V(τ )} 6: else 7: return ∅; 8: end if 9: end for 10: return ts;

from this successful run is added to the resulting test suite. Theorem 4 shows the correctness of Algorithm 1. Theorem 3. If the test suite ts returned by Algorithm 1 is not empty, then (i) Km passes ts and (ii) ts weakly covers all the states of B. Proof. (i) For each t ∈ ts, there is a related state s, and M C isEmpty(B(s), Km ) returns a successful run τ of the product of B(s) and Km such that V(τ ) = t. Since any successful run of the product of B(s) and Km shall also be a trace of Km , τ is also a trace of Km . Therefore, Km shall pass t. Furthermore, Km passes every test case in ts. (ii) As shown in (i), for each t ∈ ts, there is a related state s and a successful run τ of the product of B(s) and Km such that V(τ ) = t. We will show that t weakly covers s. By Deﬁnition 7, we need to show that there is a successful τ of B such that τ goes through the state s. We obtain τ by taking the projection of τ on the states of B as follows: since τ is a run of the product of B(s) and Km , each state on τ has the form of s , i, v, where s is a state of B, i is an index number from {0, 1}, and v is a state of Km . We project state s , i, v to state s on B, and let τ be the resulting sequence. Clearly τ is also a successful run of B because, by Deﬁnition 10, each transition in B(s) is mapped to a transition in B and each acceptance state in B(s) is mapped to an acceptance state in B. In addition, τ has to go through s, 0 because, by Deﬁnition 10, acceptance states of a SM-GBA are indexed by 1, whereas start states are indexed by 0. The only way the index number is changed from 0 to 1 is to go through s, 0. Therefore, τ has to go through s, and we have proved (ii). 2 By Deﬁnition 8, a test case t strongly covers a state s of a GBA only if every successful run of the automaton accepting t has to visit state s. To generate such a test case, we will deﬁne an automaton whose successful runs are precisely those of the original automaton visiting state s. Such an automaton can be deﬁned as the negation of the state excluding generalized B¨ uchi automaton (SE-GBA) in Deﬁnition 11. SE-GBA for state s removes s from the original automaton.

State Coverage Metrics for Speciﬁcation-Based Testing

179

Algorithm 2. TestGen SSC(B, Km = S, s0 , →, V) Require: B is a GBA, Km is a system model, and Km satisﬁes B; Ensure: Return the test suite ts that strongly covers all the states of B and is passed by Km . Return ∅ if such a test suite is not found; 1: for every s ∈ S do 2: Construct a SE-GBA Bs¯ for B’s state s 3: τ = M C isEmpty(¬Bs¯, Km ); 4: if |τ | = 0 then 5: ts = ts ∪ {V(τ )} 6: else 7: return ∅; 8: end if 9: end for 10: return ts;

Transitions, start states, and acceptance states are updated accordingly to reﬂect the removal of s. By the construction of the SE-GBA, its successful runs are exactly the subset of the original GBA’s successful runs that do not visit s. Therefore, the test cases accepted by the SE-GBA do not strongly cover state s of the original GBA. Algorithm 2 uses the negation of the SE-GBA as an input to the core emptiness checking routine of a model checker to produce a test case with strong state coverage. Theorem 4 shows the correctness of Algorithm 2. Definition 11 (State Excluding Generalized B¨ uchi Automata (SM-GBA)). Let B = P, S, S0 , Δ, L, F be a GBA, a state excluding generalized B¨ uchi automaton for s ∈ S is a GBA Bs¯ = P, S − {s}, S0 − {s}, Δ − {s , s ∈ Δ | s = s ∨ s = s}, L, F ∈F {F − {s}}. Theorem 4. If the test suite ts returned by Algorithm 2 is not empty, then (i) Km passes ts; and (ii) ts strongly covers all the states of B. Proof. (i) For each t ∈ ts, there is a related state s, and M C isEmpty(¬(Bs¯), Km ) returns a successful run τ of the product of ¬Bs¯ and Km such that V(τ ) = t. Since any successful run of the product of ¬Bs¯ and Km shall also be a trace of Km , τ is also a trace of Km . Therefore, Km shall pass t. That is, Km passes every test case in ts. (ii) As shown in (i), for each t ∈ ts, there is a related state s and a successful run τ of the product of ¬Bs¯ and Km such that V(τ ) = t. We will show that t strongly covers s. First, since τ is a trace of Km and Km satisﬁes B by the precondition of Algorithm 2, B accepts t = V(τ ). Next, we will prove by contradiction that every successful run of B induced by the test case t shall visit s at least once: suppose not, and let ρ be a successful run of B induced by t and ρ does not visit s. It follows that ρ shall also be a successful run of Bs¯, because, with the exception of missing state s, Bs¯ is the same as B. Therefore, Bs¯ shall accept t. By Algorithm 2, there is a τ such that

180

L. Tan

t = V(τ ) and τ is a successful run of the product of ¬Bs¯ and Km . It follows that t = V(τ ) is accepted by ¬Bs¯ and hence it cannot be accepted by Bs¯. We reach a contradiction. Therefore, every successful run of B that accepts t shall visit s at least once. 2 A generalized B¨ uchi automaton B can be translated to a B¨ uchi automaton by indexing acceptance states. The resulting B¨ uchi automaton has the size O(|F | · |B|), where |F | is the number of acceptance state sets in B, and |B| is the size of B. The emptiness checking for a B¨ uchi automaton can be done in linear time (cf. [15]). Therefore, generating a test case weakly covering a state s can be done in O(|K| · |F | · |B|), where |K| is the size of the model, and generating a test suite weakly covering all the states in B can be done in O(|K| · |F | · |B|2 ). Algorithm 2 starts with the construction of a SE-GBA for a state, which can be done in linear time, and Algorithm 2 then negates SE-GBA. Michel [12] provided a lower bound of 2O(nlogn) for negating a B¨ uchi automaton of size n. Therefore, Algorithm 2 takes at least O(|K| · 2O(|F |·|B|log(|F |·|B|)) to generate a test case strongly covering a state and at least O(|K| · 2O(|F |·|B|log(|F |·|B|)) to generate a test suite strongly covering all the states of a GBA. The reason why generating test cases for strong state coverage is much more expensive than for weak state coverage can be traced back to Deﬁnition 6: to strongly cover a state s we need to examine all the successful runs and make sure that they visit s, whereas to weakly cover s we only need to ﬁnd a single successful run visiting s.

5

Testing-Based Property Refinement

In speciﬁcation-based testing, the correctness of a system is deﬁned as the system’s conformance to its speciﬁcations. Inadequate coverage on speciﬁcation may suggest a problem in a system, but it may also indicate a problem in speciﬁcation. For example, the speciﬁcation may be imprecise and/or too general. Besides producing test suites, model-checking-based test generation algorithms also provide valuable information on how a property is related to a system model. Here we consider property reﬁnement using feedbacks from test generation algorithms. To facilitate our discussion, we need to formalize the notion of “reﬁnement”. Language inclusion is a natural candidate for deﬁning a reﬁnement preorder. Formally we deﬁne B B if and only if L(B) ⊆ L(B ), that is, B is a reﬁnement of B if the language accepted by B is a subset of the language accepted by B . By Deﬁnition 3, we can infer that a test accepted by B will also be accepted by B . We use state coverage metrics to guide the property reﬁnement process. The purpose of the reﬁnement is to ﬁne-tune a property so it can more closely describe the behaviors of a system, measured by increased state coverage on the property. Lemma 1. Given a GBA B = P, S, S0 , Δ, L, F with a state s ∈ S, let Bs¯ = P, S − {s}, S0 − {s}, Δ , L, F be the state excluding GBA for s, then Bs¯ B.

State Coverage Metrics for Speciﬁcation-Based Testing

181

Proof. By Deﬁnition 11 the SE-GBA Bs¯ misses state s. It follows that the successful runs of Bs¯ are those that do not visit s in the original GBA. Therefore, L(Bs¯) ⊆ L(B) and hence Bs¯ B. 2 Theorem 5. Given a GBA B = P, S, S0 , Δ, L, F with a state s ∈ S and a Kripke structure K = V, v0 , →, V, let Bs¯ = P, S − {s}, S0 − {s}, Δ , L , F be the state excluding GBA for s, if K passes a test case t strongly covering s and K |= B, then, K |= Bs¯. Proof. We will prove by contradiction. Suppose that K |= Bs¯. Since K passes t, K has a trace τ such that V(τ ) = t. Since K |= Bs¯, t |= Bs¯. Let ρ be a successful runs of Bs¯ induced by t, that is, t ρ. ρ is also a successful run of B because by Lemma 1 Bs¯ is a reﬁnement of B. Since ρ does not visit s, t does not strongly covers s, which contradicts to the condition of the theorem. Therefore, K |= Bs¯. 2 Definition 12 (Vacuous States). Given a generalized B¨ uchi automaton B = P, S, S0 , Δ, L, F and a Kripke structure K, a state s of B is vacuous with respect to K if and only if K |= B implies K |= Bs¯, where Bs¯ = P, S − {s}, S0 − {s}, Δ , L, F is the SE-GBA for s. Deﬁnition 12 deﬁnes vacuous states. Since Bs¯ is a reﬁnement of B, K |= B implies K |= Bs¯. Therefore, Deﬁnition 12 indicates that a vacuous state s of a GBA B for a Kripke structure K does not aﬀect whether K satisﬁes B. That is, if we remove the vacuous state s from B, the outcome of whether the system K satisﬁes GBA B will stay same. This observation prompts us to introduce the notion of state-coverage-induced reﬁnement: for a given system and a property in a GBA, if a state of the GBA is vacuous to the system, the state can be removed from the GBA, and the system still satisfies this refinement of the original GBA. Corollary 1. Given a generalized B¨ uchi automaton B and a Kripke structure K = V, v0 , →, V, s is not vacuous with respect to K if and only if K |= B and there exists a test t such that t strongly covers s and K passes t. Proof. Note that K |= B implies K |= Bs¯ since L(Bs¯) ⊆ L(B). By Deﬁnition 12, s is not vacuous with respect to K if and only if K |= B and K |= Bs¯. All we need to show that: if K |= B, then K |= Bs¯ if and only if there exists a test t strongly covering s and K passes t. (⇒) Since K |= B and K |= Bs¯, there must be a trace τ of K such that (1) B has a successful run ρ such that V(τ ) ρ, and (2) Bs¯ does not have a successful run ρ such that V(τ ) ρ . Since Bs¯ is obtained by removing state s from B, it follows that B’s every successful run ρ such that V(τ ) ρ shall go through s, otherwise, ρ is also a successful run of Bs¯, which contradicts to the condition of our selection of τ . Let t = V(τ ), by Deﬁnition 6, t strongly covers s and K passes t. (⇐) Since K passes t, K has a trace τ such that V(τ ) = t. Since t strongly covers s, we have (i) t |= B, and hence B has a successful run ρ such that

182

L. Tan

Algorithm 3. State Reﬁnement(B, Km = S, s0 , →, V)

. Require: B is a GBA, Km is a system model, and Km satisﬁes B; Ensure: Return a GBA as a reﬁnement of B, and a test suite ts that strongly covers all the states of the new GBA; 1: for every s ∈ S do 2: Construct the SE-GBA Bs¯ for B’s state s; 3: τ = M C isEmpty(¬Bs¯, Km )); 4: if |τ | = 0 then 5: ts = ts ∪ {V(τ )} 6: else 7: B = Bs¯; 8: end if 9: end for 10: return B, ts;

t ρ; and (ii) for every successful run ρ of B such that t ρ , ρ goes through s. We will prove by contradiction that K |= Bs¯. Suppose that K |= Bs¯. It follows that every trace of K shall be accepted by Bs¯, and hence Bs¯ has a successful run ρ such that V(τ ) ρ . Note that Bs¯ is obtained by removing s from B, ρ is also a successful run of B but ρ does not visit s. It follows that t cannot strongly cover s because B has a successful run induced by t that does not visit s. We reach a contradiction. Therefore, K |= Bs¯. 2 Corollary 1 shows the relation between the strong state coverage and nonvacuousness of a state in a GBA. It shall be noted that testing alone cannot prove the non-vacuousness of a state of a GBA. This is because the non-vacuousness of a state s of B for a system K requires that s aﬀects the outcome of whether K satisﬁes B, that is, either K |= B and K |= Bs¯, or K |= B and K |= Bs¯. Since Bs¯ is obtained by removing s from B, K |= B implies K |= Bs¯. The only option left is that K |= B and K |= Bs¯, but testing alone cannot show that K satisﬁes B. Nevertheless, lack of the strong coverage for a state s indicates that s is a vacuous state for K. If that happens, we can remove s from B without aﬀecting the outcome of whether K satisﬁes B. Algorithm 3 reﬁnes a GBA while generating a test suite strongly covering states of the new GBA. Algorithm 3 is a modiﬁcation of Algorithm 2. The diﬀerence is at line 7. Instead of returning with a failed attempt in Algorithm 2 when strong state coverage cannot be obtained, Algorithm 2 reﬁnes the input GBA by removing vacuous states. The output will be a GBA reﬁned by removing vacuous states with respect to the model, and a test suite for the reﬁned GBA.

6

Experiment

To assess the feasibility and performance of our proposed coverage metrics and test generation algorithms, we test them on three examples using SPIN. The

State Coverage Metrics for Speciﬁcation-Based Testing

183

ﬁrst example is the General Inter-ORB Protocol, a key component of Common Object Request Broker Architecture (CORBA) speciﬁcation deﬁned by Object Management Group (OMG). GIOP deﬁnes the inter-operability between Object Request Brokers (ORBs). The Promela model and the properties are provided by Kamel and Leue [9]. The second application is to generate tests for the NeedhamSchroeder Public Key Protocol. The model and the properties are provided by Maggi and Sisto [11]. The last one is the Go-Back-N sliding window protocol as described by Tanenbaum [1]. The model and the properties come from the SPIN website 1 . In all three cases, the properties are initially provided in LTL. We use GOAL [14] to generate a B¨ uchi automaton from a LTL property, and we also use it to synthesize state marking and state excluding automata required for generating tests for weak and strong state coverage criteria. It shall be noted that SPIN takes an automaton in its negation form through its never claim form. For example, the GBA for a LTL property f in its never claim form is ¬B(¬f ), where B(¬f ) is the GBA for the LTL property ¬f . SPIN produces an error trace if it ﬁnds a violation of the property in never claim form. Using SPIN, we replace line 3 with spin(¬B(f )(s), Km ), where B(f ) is a B¨ uchi automaton for the LTL property f and B(f )(s) is its corresponding SM-GBA for state s. To generate test cases for strong state coverage, we replace line 3 in Algorithm 2 with spin(¬(¬B(f )s¯), Km ). Note that the property is given as its negation in never claim form never{¬B(f )s¯}. As we discussed in Section 4, negating a B¨ uchi automaton is an expensive computation for test generation not just in its own right but also because the complemented automaton suﬀers an exponential blowup [12]. Since SPIN produces an inﬁnite error trace as a lasso-shaped sequence, we measure its length in a form deﬁned in Deﬁnition 2. As a comparison, we also measure the coverage of generated test cases using a traditional structural coverage metric, which in our experiment, is branch coverage. We use SPIN version 6.0.1. All the experiments are run on a Dell server with one 2.33 GHz quadcore Xeon 5410 and 8 GB RAM. Table 1 shows the experiment results. For each model, column 1 of Table 1 speciﬁes the properties that we generate test cases for. We refer to these properties by their names originally used by their respective authors. For each property, column 2 speciﬁes which state coverage metric, weak or strong, is used. Column 3 speciﬁes which state a test case is to cover. Column 4 provides the length of each lasso-shaped test case, as deﬁned in Deﬁnition 2 and column 5 is the time used to generate a test case. To compare the performance of the proposed metrics and a traditional coverage metric, we measure the branch coverage of each individual test case, which is given as the ratio of covered branches v.s. total branches. We also measure the accumulative coverage of each test suite, which consists of all the test cases generated for a property and a particular (weak or strong) state coverage metric. With only exception of property v7 in GIOP model, the branch coverage achieved by a test suite for strong state coverage criterion is better than that for weak state coverage criterion. That is because strong state coverage criterion subsumes weak state coverage criterion. Generating a test case strongly covering 1

http://spinroot.com/spin/man/exercises.html

184

L. Tan

Table 1. Experiment results. A test case t = α · β ω , where α is the preﬁx and β is a circular sequence. s0 is the default and only start state for all the properties. Acceptance states are marked with ∗. Test case length Branch coverage time (sec.) |α| |β| Individual Overall General Inter-ORB Protocol s1 577 1 0.01 46/70 Weak s2 ∗ 577 1 0.01 46/70 51/70 s3 ∗ 449 1 0.57 47/70 v6b s1 577 1 0.01 46/70 Strong s2 ∗ 577 1 1.00 46/70 51/70 s3 ∗ 449 1 1.00 47/70 s2 577 1 0.01 46/70 Weak s2 ∗ 577 1 0.01 46/70 54/70 s3 ∗ 521 1 2.04 53/70 v7 s1 577 1 0.01 46/70 Strong s2 ∗ 577 1 0.01 46/70 46/70 s3 ∗ (Exec. Time > 300 min.) Needham-Schroeder Security Protocol s1 22 1 0.01 15/59 Weak s2 ∗ 22 1 0.01 15/59 15/59 s3 ∗ 22 1 0.01 15/59 m x1init fixed s1 24 1 0.01 17/59 Strong s2 ∗ 24 1 0.01 17/59 28/59 s3 ∗ 23 1 0.01 15/59 Sliding Window Protocol s1 (Not covered) Weak s2 550 111 0.01 14/23 14/23 s3 ∗ 185 111 0.01 7/23 ltl3 s4 ∗ (Not covered) s1 (Not covered) s2 550 111 0.01 14/23 Strong 14/23 s3 ∗ 185 111 0.01 7/23 s4 ∗ (Not covered) Property

Coverage State

state s3 for property v7 does not terminate in a reasonable time frame. Test suites generated for v6b and v7 achieve a reasonable branch coverage (72.8% for v6b and 77.1% for v7), especially when taking into account that these two properties capture only a very limit requirement for the GIOP protocol. The model for Needham-Schroeder security protocol involves three parties: an initiator, a responder, and an intruder. The property m 1init fixed is a liveness property requiring that the initiator sends a message only after the responder is up and running. It does not describes the security aspect of the protocol, that is, safety properties involving the intruder. Almost all of branches not covered by generated test suites are within the logic of the intruder. Lack of coverage in this case is related to the deﬁciency in the speciﬁcation. In the model for the sliding

State Coverage Metrics for Speciﬁcation-Based Testing

185

window protocol, the branch points not covered by test suites for state coverage are within the logic of timeout mechanism. The timeout mechanism is put into place to handle packet loss. A close look at the model reveals that it does not contain a lossy channel. So in this case lack of coverage indicates the deﬁciency in the model. In these experiments, our proposed state coverage metrics do help reveal deﬁciency in speciﬁcations and/or models.

7

Conclusions

We considered speciﬁcation-based testing for linear temporal properties expressed in generalized B¨ uchi automata (GBAs). We proposed two variants of state coverage metrics for measuring how well test cases cover states of a GBA. The immediate application of these two metrics is to select test cases based on their relevancy to a GBA-based speciﬁcation. For this application we provide modelchecking-based test generation algorithms for proposed coverage criteria. This research extends our previous work on vacuity-based coverage metric for LTL formula. By using GBA as the underlying representation for linear temporal properties, we can use existing automaton minimization techniques to reduce syntactical variances of these temporal properties, and hence our state coverage metrics deﬁned on GBAs are less susceptible to syntactical changes of properties. We argued that our property-based state coverage metrics also helped detect the deﬁciency in speciﬁcation and one may use them to guide the reﬁnement of requirement speciﬁcations. For this application, we deﬁned the notion of vacuous states for a GBA and a system. Vacuous states are those states of the GBA that do not aﬀect whether the system satisﬁes the GBA. Removing these vacuous states from the GBA yields a reﬁned GBA that describes the behaviors of the system more closely. We provided a model-checking-based property reﬁnement algorithm based on the notion of vacuous states and strong state coverage metric. Our experiment results demonstrated the feasibility and performance of our coverage metrics and test generation algorithms. For the further research on the subject, we will study other GBA-based coverage metrics, and we will also consider a general framework for unifying temporal logic-based and GBA-based coverage metrics. Acknowledgement. The author would like to thank Bolong Zeng for his assistance in the experimental study.

References 1. Tanenbaum, A.S.: Computer Networks, 5th edn. Prentice Hall, Englewood Cliﬀs (2010) 2. Calvagna, A., Gargantini, A.: A Logic-Based Approach to Combinatorial Testing with Constraints. In: Beckert, B., H¨ ahnle, R. (eds.) TAP 2008. LNCS, vol. 4966, pp. 66–83. Springer, Heidelberg (2008) 3. Clarke, E., Grumberg, O., Peled, D.: Model Checking. MIT Press, Cambridge (1999)

186

L. Tan

4. Fraser, G., Gargantini, A.: An Evaluation of Speciﬁcation Based Test Generation Techniques Using Model Checkers. In: TAIC-PART 2009, pp. 72–81. IEEE Press, Los Alamitos (2009) 5. Gerth, R., Peled, D., Vardi, M., Wolper, P.: Simple on-the-ﬂy automatic veriﬁcation of linear temporal logic. In: PSTV 1995, pp. 3–18. Chapman and Hall, Boca Raton (1995) 6. Giannakopoulou, D., Lerda, F.: From States to Transitions: Improving Translation of LTL Formulae to B¨ uchi Automata. In: Peled, D.A., Vardi, M.Y. (eds.) FORTE 2002. LNCS, vol. 2529, pp. 308–326. Springer, Heidelberg (2002) 7. Gerard, J.: The Model Checker SPIN. IEEE Transactions on Software Engineering 23(5), 279–295 (1997) 8. Hong, H.S., Lee, I., Sokolsky, O., Ural, H.: A Temporal Logic Based Theory of Test Coverage and Generation. In: Katoen, J.-P., Stevens, P. (eds.) TACAS 2002. LNCS, vol. 2280, pp. 327–341. Springer, Heidelberg (2002) 9. Kamel, M., Leue, S.: Formalization and validation of the General Inter-ORB Protocol (GIOP) using PROMELA and SPIN. International Journal on Software Tools for Technology Transfer (STTT) 2(4), 394–409 (2000) 10. Kupferman, O., Vardi, M.Y.: Vacuity detection in temporal model checking. International Journal on Software Tools for Technology Transfer (STTT) 4(2), 224–233 (2003) 11. Maggi, P., Sisto, R.: Using SPIN to Verify Security Properties of Cryptographic Protocols. In: Boˇsnaˇcki, D., Leue, S. (eds.) SPIN 2002. LNCS, vol. 2318, pp. 187– 204. Springer, Heidelberg (2002) 12. Michel, M.: Complementation is more diﬃcult with automata on inﬁnite words. CNET, Paris (1988) 13. Tan, L., Sokolsky, O., Lee, I.: Speciﬁcation-based Testing with Linear Temporal Logic. In: IRI 2004, pp. 493–498. IEEE Society, Los Alamitos (2004) 14. Tsay, Y.-K., Chen, Y.-F., Tsai, M.-H., Wu, K.-N., Chan, W.-C.: GOAL: A Graphical Tool for Manipulating B¨ uchi Automata and Temporal Formulae. In: Grumberg, O., Huth, M. (eds.) TACAS 2007. LNCS, vol. 4424, pp. 466–471. Springer, Heidelberg (2007) 15. Vardi, M.Y.: Automata-theoretic model checking revisited. In: Cook, B., Podelski, A. (eds.) VMCAI 2007. LNCS, vol. 4349, pp. 137–150. Springer, Heidelberg (2007)

Lightweight Testing of Communication Networks with e-Motions Javier Troya, Jos´e M. Bautista, Fernando L´opez-Romero, and Antonio Vallecillo GISUM/Atenea Research Group. Universidad de M´ alaga, Spain {javiertc,jmbautista,fernando,av}@lcc.uma.es

Abstract. This paper illustrates the use of high-level domain speciﬁc models to specify and test some performance properties of complex systems, in particular Communication Networks, using a light-weight approach. By following a Model-Driven Engineering (MDE) approach, we show the beneﬁts of constructing very abstract models of the systems under test, which can then be easily prototyped and analysed to explore their properties. For this purpose we use e-Motions, a language and its supporting toolkit that allows end-user modelling of real-time systems and their analysis in a graphical manner.

1

Introduction

Lightweight modelling is the use of small, abstract models of the system under study, and of push-button veriﬁcation techniques [1]. The key ideas behind this approach, as proposed by Pamela Zave, are the construction of a very abstract model of the system and the use of analysis tools to explore its properties. Being the model very abstract in comparison to a real implementation, and focusing only on the relevant concepts, it becomes small, tractable, and can be constructed quickly. Being the analysis tools simple and pushbutton-based, they yield results with little eﬀort. Thus it becomes easy for the system designer to prototype the system, test its properties and re-adjust the designs in a costeﬀective manner. Moreover, this enables an incremental and iterative approach to system design and testing, where the system is progressively speciﬁed and its properties analysed for correctness and against a set of quality requirements. The problems found during the testing process can be carefully analysed and either the system design or the quality requirements reﬁned accordingly. In this paper we show how Domain Speciﬁc Modelling Languages (DSMLs) can help realizing this approach. In the ﬁrst place, they allow end-users to create models of their systems at the right level of abstraction and with the appropriate precision. Secondly, the produced models can be connected to powerful simulation and analysis tools using model transformations, to provide the push-button capabilities required for accomplishing the analysis. We illustrate our approach using the e-Motions language and supporting toolkit [2], which enables the precise deﬁnition of real-time models in a graphical and intuitive way, as well as its simulation and analysis [3]. M. Gogolla and B. Wolﬀ (Eds.): TAP 2011, LNCS 6706, pp. 187–204, 2011. c Springer-Verlag Berlin Heidelberg 2011

188

J. Troya et al.

Fig. 1. Communication Network Metamodel

As a running example we use a re-conﬁgurable Communication Network system, composed of computers that transmit messages through nodes that process and forward them to other nodes until messages reach their ﬁnal destinations. Additional supporting nodes can be activated in case of network congestion to alleviate the temporary traﬃc bottlenecks. Assuming that the cost of acquiring and maintaining these extra nodes is not negligible, there are some tradeoﬀs between the quality of service provided by the network and its overall cost. We show how this kind of analysis and tests can be conducted with our proposal in an easy and cost-eﬀective manner. The structure of this paper is as follows. Section 2 introduces the running example and provides the motivation of our work. Then, Section 3 describes the structural model of the system, and them how to model its behaviour so that it can be later simulated and analysed, as discussed in Section 4. Finally, Section 5 describes some related works and Section 6 presents the conclusions.

2

A Running Example

Let us start by describing the system that we want to model and whose performance and behaviour we want to analyse. It consists of a communication network composed of diﬀerent kinds of Components that can contain Packets. Each component has a speciﬁc location, given by two coordinates. The metamodel of the network is shown in Fig. 1. Users produce packets, while Servers consume them (i.e., they act as sources and sinks of the network, respectively). Components can exchange Packets only if they are connected. Such connection between components is modelled by the neighbours reference, that reﬂects the components that are reachable from a given component. The network itself is modelled by a set of packet switching Nodes, which are the network elements in charge of receiving, processing and forwarding packets to other components. Nodes have one attribute (pkPr) in our

Lightweight Testing of Communication Networks with e-Motions

189

model to keep track of the packets they have processed so far. The buﬀer with the set of received packets that a node has to process is modelled by means of the composition relation between Component and Packet. One characteristic of our network is that the time each node spends in processing a packet depends on the number of packets in its buﬀer. The more packets in the buﬀer, the more the node takes to process each one. This simulates a behaviour where nodes need to perform some operations on the ﬂow of packets, such as sorting or merging them according to a given algorithm, for instance. Packets have two attributes, is being processed and timeStamp. The former indicates whether the packet is currently being processed by a node, while the latter stores the moment in time at which the packet enters the network. For routing packets, nodes decide to forward packets to the neighbour node which is less loaded, i.e., the one with the smallest buﬀer size. In order to alleviate network congestion, an additional kind of nodes (called SupportNodes) exists in the network. They can be activated and de-activated depending on the load of the neighbouring nodes. Each SupportNode activates itself if the number of packets in the buﬀer of any of the nodes connected to it via the support relationship goes above the value deﬁned in its threshold attribute. Similarly, it deactivates itself when the load of all connected nodes is above the threshold. Attribute activations keeps track of the moments in time at which the support node changes its state. Let us assume that the cost of acquiring, maintaining and running these extra support nodes cannot be ignored, as it happens for instance if support nodes are hired from external network providers, and their running costs depend on the time they are active or on the number of packets they process. In this setting the system owner is faced with several decisions in order to maximize the quality of service provided by the network while minimizing its overall cost. Firstly, how many supporting nodes need to be hired/purchased to guarantee a minimum level of throughput? Secondly, which is the optimal value for the threshold of each support node that provides a required level of throughput with the minimum time of support node activation (hence minimizing the running cost of the node)? In order to be able to respond to these questions, we need to identify which are the system parameters that are relevant to our analysis. In our case, we will focus on the following ones: – Throughput and delay of the overall network. They indicate how fast nodes process packets. Throughput tells us how many packets are processed by the network per unit of time. Delay indicates how many time units the packets spend within the network. The higher the throughput, the lower the delay, and so the higher the performance of the network. – Packets processed per node. This measure provides an indication of the work load supported by each node. This is however a complex indicator due to the way in which packets are processed in this network, and how they arrive to nodes. The fact that processing time depends on the length of the buﬀer of pending packets may cause diﬀerent behaviour depending on whether packets are coming in bursts or at a regular pace.

190

J. Troya et al.

Fig. 2. Initial model of the network

– Packets processed per SupportNode. This measure is important because it provides an indication on the real need of these nodes. – Activation times of SupportNodes. The time and frequency of activation of this kind of nodes also provides useful information about their actual usage in the current network conﬁguration.

3 3.1

Modelling the Communication Network Using e-Motions Modelling the Structure

The ﬁrst step is to model the initial conﬁguration of the system. This is nothing but a model that conforms to the Network metamodel. A possible conﬁguration example of a network is shown in Fig. 2 (please ignore the area within the dotted lines for now). This conﬁguration deﬁnes three Users feeding packets into the network and one Server consuming them. Each user accesses the network using diﬀerent nodes. The network is composed of 8 (normal) nodes and 2 support nodes (n9 and n10), which are initially deactivated. The activation of the support

Lightweight Testing of Communication Networks with e-Motions

191

nodes depends on the buﬀer size of nodes n3 and n4 for support node n9, and of nodes n5 and n6 for support node n10. This is speciﬁed by the corresponding support relations between the support nodes and the nodes they try to help. 3.2

Modelling Behaviour

Apart from the structure of our system, which is captured by the model shown in Fig. 2, we need to be able to describe its behavioural dynamics in a way that allow us to reason about them. One way to do this is by describing the evolution of the modelled artifacts along some time model. In MDE, this can be done using model transformations supporting in-place updates [4]. The behaviour of the system is then speciﬁed in terms of the permitted actions, which are in turn modelled by the model transformation rules. In-place transformations are composed of a set of rules, each of which represents a possible action of the system. These rules are of the form l : [NAC]∗ × LHS → RHS, where l is the rule’s label (its name), and LHS (left-hand side), RHS (right-hand side) and NAC (negative application conditions) are model patterns that represent certain (sub-)states of the system. The LHS and NAC patterns express the preconditions for the rule to be applied, whereas the RHS represents its postcondition, i.e., the eﬀect of the corresponding action. Thus, a rule can be applied, i.e., triggered, if an occurrence (or match) of the LHS is found in the model and none of its NAC patterns occurs. Generally, if several matches are found, one of them is non-deterministically selected and applied, producing a new model where the match is substituted by the appropriate instantiation of its RHS pattern (the rule’s realization). The model transformation proceeds by applying the rules in a non-deterministic order, until none is applicable — although this behaviour can be usually modiﬁed by some execution control mechanism, e.g., strategies [5]. 3.3

e-Motions

In [2] we presented e-Motions, a tool for the formal and precise deﬁnition of realtime DSMLs in a graphical and intuitive way developed for Eclipse. It extends inplace model transformation with a model of time and mechanisms to state action properties, designed for the speciﬁcation of Domain Speciﬁc Visual Languages (DSVL) of real-time systems. Time-related attributes can be added to in-place rules to represent features like duration, periodicity, etc. Two types of rules were deﬁned to specify time-dependent behaviour, namely, atomic and ongoing rules. Atomic rules represent atomic actions with a speciﬁc duration, which is speciﬁed by an interval of time with any OCL [6] expression. In fact, e-Motions has full support for OCL thanks to mOdCL [7], which implements and give semantics to OCL in Maude [5]. The mentioned rules can be periodic, i.e., they admit a parameter that speciﬁes the amount of time after which the action is periodically triggered (if the rule’s precondition holds, of course). In our latest version of eMotions, probability distributions can be used for specifying these times. In this way, we can, for example, let the arrival of packets to a system depend on a poisson distribution. Ongoing rules represent actions that progress continuously

192

J. Troya et al.

Fig. 3. NewPacket Rule

with time while the rule preconditions (LHS and not NACs) hold. Both atomic and ongoing rules can be scheduled, or be given an execution interval. In order to be able to model both state-based and action-based properties, we have also proposed extending model patterns with action executions to specify action occurrences. These action executions specify the type of the action (i.e., the name of the atomic rule), its status (e.g., if the action is unﬁnished or realized) and its identiﬁer. They may also specify its starting, ending and execution time and the set of participants involved in it. This provides a very useful mechanism when we want to check whether an object is participating in an action or not, or if an action has already been executed. A special kind of object, named Clock, represents the current global time elapse. Designers are allowed to use it in their timed rules (using its attribute time) to know the amount of time that the system has been working. e-Motions oﬀers automated bridges to the Maude [5] executable language and its formal toolkit. Maude is used as a formal notation to provide the precise semantics of the corresponding e-Motions speciﬁcations (as described in [8]), while at the same time the model transformations between e-Motions and Maude (implemented in ATL [9]) allow the Maude tools to become available in the eMotions environment. In this way, both simulation and the use of some formal analysis tools are possible for e-Motions speciﬁcations [10]. 3.4

Specifying the Behaviour of the Network

The behaviour of the network will be speciﬁed by a set of rules, each one describing one possible action. The NewPacket rule, shown in Fig. 3, simulates the generation of packets by users. This process follows a uniform distribution in the interval [1,7], i.e., a user generates a packet every duration time units. Here, duration determines

Lightweight Testing of Communication Networks with e-Motions

193

Fig. 4. Forwarding Rule

the duration of the rule and is calculated using a random number generator (eMotions.random(6) returns a value between 0 and 6). Packet attributes are initialized at creation as shown in the right hand side of the rule. The Forwarding rule (Fig. 4) models the forwarding of packets among components and nodes. This rule is ﬁred when sending packets from users to nodes and from nodes to nodes. To apply this rule, the packet must not be being processed. Furthermore, there are two OCL expressions that have to be satisﬁed in order to launch the rule. They state that the target node is the component’s neighbour which is processing a lower number of packets, and that it cannot be a deactivated support node. In the RHS pattern of the rule the packet has moved to the node and it has started being processed. The duration of this rule can be either 0 or 1 time units (the fact that a packet can take 0 units simulates the situation in which several packets are forwarded together to optimize an open connection). In Fig. 5(a) we can see the PacketProcessing rule. It models the processing of a packet by a node by modifying its is being processed attribute. The pckPr attribute of the node is increased in one unit as it has processed a new packet. The time this rule spends is directly proportional to the number of packets being processed by the node. PacketArrival rule (Fig. 5(b)) models the arrival and consumption of a packet from a node to the server. The time this rule consumes is either 0 or 1 time units. Finally, activation and deactivation of support nodes is speciﬁed by two rules. ActivationSupport rule (Fig. 6(a)) deals with the activation of a support node when it is deactivated and one of the nodes it supports is processing more packets than indicated by the node threshold. DeactivationSupport rule (Fig. 6(b)) carries out the opposite action. In both rules, the time unit when the activation/deactivation occurs is added to the node’s activations attribute. These rules are instantaneous rules, i.e., atomic rules with duration 0.

194

J. Troya et al.

(a) PacketProcessing Rule

(b) PacketArrival Rule

Fig. 5. PacketProcessing and PacketArrival Rules

(a) Activate Rule

(b) Deactivate Rule

Fig. 6. Activate and Deactivate Rules

3.5

Adding Observers for System Monitoring

Apart from the intrinsic properties of the system, there are also other features that we may need to express and capture in our models. For example, in this network we are interested in monitoring the throughput and delay of the packets processed by the network as well as in the number of packets processed by each node, especially the support nodes. The activation/deactivation frequency of the support nodes is also relevant. Although some of these properties could be analysed using the model element attributes (e.g., number of processed packets), other features should be expressed using additional elements. The traditional solution has normally consisted in extending the system metamodel with additional attributes, i.e., extending the structure of the system to accommodate new state variables. In [3] we introduced observers for tackling this problem using a modular and reusable approach.

Lightweight Testing of Communication Networks with e-Motions

195

Fig. 7. Observers Metamodel

An observer is an object whose purpose is to monitor the state of the system: the state of the objects, of the actions, or both. Observers, as any other objects, have a state and a well-deﬁned behaviour. The attributes of the observers capture their state and are used to store the variables that we want to monitor. We have deﬁned an Observers metamodel, which is shown in Fig. 7. We have three diﬀerent observers: – ThroughputOb, in charge of monitoring the throughput of the system (the number of packets processed by the network per time unit). – DelayOb, that tracks with its delay attribute the average time spent by packets to be processed by the network. – CounterOb, responsible for counting packets. The packetsServer attribute counts the number of packets that arrive at the server. It is used to calculate the throughput and delay of the system. Attribute packetsNetwork stores the number of packets that users introduce in the network. Finally, totalPackets determines an upper limit for the simulation process, specifying the total number of packets that the network will process. The idea for analysing the system with observers is to combine the original metamodel (Fig. 1) with the Observers’ metamodel (Fig. 7) to be able to use the observers in our DSVL. Since e-Motions allows users to merge several metamodels in the deﬁnition of the behaviour of a DSVL, we can deﬁne the Observers metamodel in a non-intrusive way, i.e., we do not need to modify the system metamodel to add attributes to their objects. Furthermore, this approach also enables the reuse of observers across diﬀerent DSVLs. The behaviour of the observers is speciﬁed using rules, too. To specify how observers monitor the non-functional properties of the system we have included them in the rules (inside the area delimited by dotted lines—these dotted lines do not form part of the rules, they have been added to the diagrams of this paper for understandability reasons). Thus, starting with the initial model of the system (Fig. 2), we see how we include an observer of each type in the network and we give their attributes some initial values. We see that the network will process up to 500 packets. Continuing with the NewPacket rule (Fig. 3), we use here the CounterOb observer to stop users generate packets when the speciﬁed upper limit is reached. PacketArrival rule (Fig. 5(b))

196

J. Troya et al.

models the arrival of a packet to the server, updating the three observers’ state appropriately: the CounterOb observer updates the number of packets arrived to the server; the DelayOb observer updates its attributes to properly compute the delay, and ﬁnally the ThroughputOb computes the current throughput.

4

Simulating and Analysing the Network

Once the speciﬁcations are written, this section describes how they can be simulated and analysed with the e-Motions tool. In e-Motions, the semantics of the real-time speciﬁcations is deﬁned by means of transformations to other domain with well-deﬁned semantics, namely RealTime Maude [11]. The e-Motions environment not only provides an editor for writing the visual speciﬁcations, but also implements their automatic transformation (using ATL) into the corresponding formal speciﬁcations in Maude. One of the beneﬁts of this approach is that it allows to make use of the Maude facilities and tools available for executing and analysing the system speciﬁcations once they are expressed in Maude. In [10,8] we showed some examples of analyses that can be performed on the Maude speciﬁcations. Furthermore, Maude rewriting logic speciﬁcations are executable, and therefore they can be used as a prototype of the system and to run simulations. In Maude, the result of a simulation is the ﬁnal conﬁguration of objects reached after completing the rewriting steps, which is nothing but a model. This resulting model can then be transformed back into its corresponding EMF notation, allowing the end-user to manipulate it from the Eclipse platform. The semantic mapping as well as the transformation process back and forth between the e-Motions and Real-Time Maude speciﬁcations is described in detail in [8], although it is completely transparent to the e-Motions user. In this way the user perceives himself as working only within the e-Motions visual environment, without the need to understand any other formalism or being completely unaware of the Maude rewriting engine performing the simulation. Regarding the use of the resulting models by other tools, e-Motions implements a trivial model-to-text transformation that enables the creation of a .csv ﬁle from an Ecore model. Such a csv ﬁle contains the information of every object in the model, together with the values of all its attributes. Objects are named by their identiﬁers, and attributes are expressed as a list of name-value pairs. Such ﬁle can be directly imported by diﬀerent applications for performing different kinds of analysis. For example, it can be fed to an spreadsheet application that the domain expert can use to analyse the data, display charts, etc. In this way, the domain expert will be able to easily display charts with the result of a simulation (which is in fact a model) to graphically represent the values of the parameters monitored by the observers throughout the whole simulation. 4.1

Tests

In order to understand how the network works and in order to analyse the parameters mentioned in Section 2, we have simulated the network with diﬀerent

Lightweight Testing of Communication Networks with e-Motions

197

Fig. 8. Simulation time, delay and throughput

threshold values for the support nodes. They have ranged from −1 (the support node is always active) to 100 (they will never be active because the buﬀers of the nodes in our example keep always below that value). For every threshold value we have run ﬁve diﬀerent simulations since users introduce packets in the network in a random manner. The ﬁgures showed in the charts correspond to the average results for the obtained values. In all the simulations, we have limited the number of packets that enter the network and reach the server to 500. Fig. 8 shows the values of throughput, delay and the time units taken by the simulations. Most variations occur when the threshold is between −1 and 7, before they become stable. This is why the chart is divided in two horizontal parts, in which the left part zooms out the [1, 10] interval. The vertical axis has also been split into two sections, in order to distinguish the area where the throughput values reside. The examination of the chart reveals that, as expected, the best performance (highest throughput, lowest delay and lowest simulation time) is achieved when the support nodes are always active (threshold = −1). However, this is also the most expensive situation. The behaviour of the support nodes turns out to be more interesting when the thresholds are between 3 and 6. In that range, the three parameters experiment the biggest variation, making the network slower as the thresholds increase. We can also see in the right part of the chart a variation in the simulation time and throughput that is a bit more pronounced than the rest. It is between thresholds 70 and 80. We give an explanation to this fact when we discuss the chart shown in Fig. 9. Fig. 9 shows the chart with the number of packets processed by the two support nodes for each threshold value. We see that the second support node (node n10 in Fig. 2) processes packets when its threshold is within the range [−1, 7). In fact, in the range [−1, 5] it processes more packets than the ﬁrst support node (node n9 in Fig. 2). However, this latter node keeps on processing packets until the threshold is 80. These results were initially unexpected, and

198

J. Troya et al.

Fig. 9. Packets processed by the support nodes 1 (n9) and 2 (n10)

Fig. 10. Activation/Deactivation of support node 1 (n9)

they are a result of the topology of the network and the way in which the packets are processed. By deﬁning a diﬀerent topology to the network and/or by changing the algorithm used to process packets to speed it up (i.e., changing the duration of rule PacketProcessing), the results would be diﬀerent. Regarding the the behaviour of the system in the range [70, 80], the slope is more pronounced there because this is the threshold value from which the ﬁrst support node (n9) stops processing packets, i.e., it is not needed at all. Focusing on the ﬁrst support node (n9), it ﬁnally stops processing packets when the threshold value is 80. By looking at its behaviour in the range [−1, 7], we cannot expect when this node will stop processing packets. In fact, looking at the behaviour of the second support node (at the beginning it processes more packets than the ﬁrst support node but then it stops processing packets from the threshold value 7), we may expect that the ﬁrst support node will stop processing packets earlier than it actually does. To help us see how the activation/deactivation of a support node evolves, we have also displayed charts for it. Thus, Fig. 10 shows four charts with the activation/deactivation of the ﬁrst support node when the threshold values are 40, 50, 60 and 70. We see how the activation of the node is carried out later when the threshold increases. This is because, as thresholds increase, the nodes being supported do not need the help of the support nodes very soon. Support nodes are also deactivated earlier when the threshold is smaller. We have to clarify here that the fact that a support node is deactivated does not mean that it stops processing packets. In fact, it only means that new packets stop coming in, but the node still has to process its buﬀer of pending packets.

Lightweight Testing of Communication Networks with e-Motions

199

Fig. 11. Packets processed by nodes n4 and n6

Not all the graphs of activations/deactivations are as uniform as the four shown here. The complete set of charts and values obtained for the simulations can be consulted in [12]. Finally, let us show a graph that we also consider of interest (Fig. 11). It displays the number of packets processed by two nodes, n4 and n6 in Fig. 2). This graph is related to the one shown in Fig. 9, since the processing of packets by the support nodes makes nodes n4 and n6 process less packets. In general, we can see that the more packets the ﬁrst support node (n9) processes, the less packets n4 processes, and the same thing happens with the second support node and n6. For every threshold value, n6 processes more packets than n4 because one of the users sends packets directly to n6.

5

Related Work

There are many diﬀerent proposals for monitoring and improving the performance and reliability of communication networks, from diﬀerent perspectives. In the ﬁrst place we have those approaches that focus on the actual systems and not on their prototypical models, such as [13,14,15,16]. These works measure the performance of existing network connections (ATMs, multimedia networks, etc.) using dedicated tools. Of course their accuracy and level of precision is very good, but they cannot be used in a predictive way. In other words, these methods and tools are excellent for a posteriori testing the network and for checking that it behaves as expected, but they cannot be used for planning purposes in the very early phases of the network design to, e.g., evaluate design alternatives or diﬀerent routing protocols. Other kind of approaches focus on design models of the system, before it is actually built. In these (model-driven) approaches, a prototype model of the system is constructed prior to the actual development and deployment, and then analysed for performance or reliability [17,18]. Model-driven proposals can be diﬀerentiated depending on three main characteristics: the level of detail used in the models (from very abstract to very detailed); the kind of analysis that they allow (analytical methods, such as Queueing Networks; formal analysis based on the exhaustive exploration of the execution tree, such as model checking; or analysis methods based on simulation techniques); and the level of ﬂexibility provided by the supporting tools.

200

J. Troya et al.

Some proposals, such as [18,19,20], are based on UML for modelling systems and networks, and normally make use of UML proﬁles like MARTE [21] for annotating the models with the speciﬁcation of QoS and other quality properties. These approaches normally provide considerable level of detail and tend to be very precise. Moreover, their models can be transformed into other formalisms such as Stochastic Petri Nets (SPN) [22], Queue Network Models (QNM) [23] and Stochastic Process Algebras (SPA) [24] for performance or reliability analysis. As weak points, their speciﬁcations normally remain at a low level of abstraction, and require skilled levels of expertise from the user. Furthermore, their corresponding analytical models can handle certain types of behaviours, but they are limited when the behaviour of the system does not follow speciﬁc patterns. For example, modelling networks with complex forwarding algorithms or packet arrival times that do not follow negative exponential distributions are hard to model and to analyse with these approaches. This is something at which we are very strong with e-Motions, being able to simulate models whose behaviours follow diﬀerent distributions [3]. Flexibility is another essential characteristic of any modelling approach in order to be useful during the ﬁrst phases of the design, but it is a common limitation of many existing approaches. For example, [25] presents a powerful proposal for evaluating the performance of packet switching communication networks using stochastic processes. However, they have a ﬁxed routing strategy and the way of specifying the network is also ﬁxed. With our approach, many diﬀerent types of networks can be modelled (by simply changing the metamodel or the rules) and diﬀerent properties can be easily observed. Visual languages based on graph transformations seem to provide the level of ﬂexibility required for specifying the structure and behaviour of this kind of systems. For example, Reiko Heckel speciﬁes in [26] two protocols for reconﬁguring P2P networks, and analyse their reliability. Protocols are modelled using stochastic graph transformations, which, as in our approach, take advantage of negative application conditions (NACs) and path expressions. For the speciﬁcation and analysis of the system he uses model checking, chaining several tools (namely GROOVE and PRISM ). This approach allows very interesting and useful kinds of analysis, but it also has some limitations. Firstly, even the very high-level models of the system cause a state explosion that can become unmanageable very soon. Secondly, in this approach users have to change the modelling environment when moving from the system design to the analysis, with the need to be familiar with more than one environment. The ﬁrst problem is intrinsic to the complexity of the networks to analyse and this is why simulation is sometimes more eﬀective to reveal design problems, specially in the early phases of the system design (at least, until the structure and dynamics of the system become stable). Then model checking or any other tool-supported mechanism that allows exploring the execution tree may be used. In e-Motions we can use not only simulation but also Maude’s search facilities, without having to escape the e-Motions environment.

Lightweight Testing of Communication Networks with e-Motions

201

There are also other interesting approaches for modelling and analysing networks at a very high level of abstraction. For example, de Lara et al. present in [27] a DSVL for the deﬁnition of traﬃc networks, using ATOM3 . Once the network is designed, they map the system models into both untimed and timed Petri nets and show how Petri net analysis and synthesis techniques can be effectively used to analyse the models. In our approach, we map our model into Real Time Maude, and the simulations performed are executed in Maude using its rewrite engine — but in a transparent way to the user. There are many other proposals based on graph transformations that allow modelling timed behaviours [28,29,30,31,32]. However, none of them allows the use of OCL for specifying expressions in attribute calculations and in rule durations. In addition to the ﬂexibility it oﬀers, the expressiveness provided by OCL becomes very useful for specifying complex behaviours at the right level of abstraction and using an appropriate notation. The ﬁnal group of works that is related to our proposal allows conducting simulations of the network models. We already mentioned the fact that the analyses based on the exploration of the execution tree may be too heavy-weight for the early phases of system design. And analytical methods, such as QNM or SPA may not be expressive enough to capture some particular characteristics of the systems under study. This is when simulation techniques for analysing performance requirements can be very useful. This is especially important in wireless self-organizing networks (WSONs), which need to be able to respond to dynamic changing environments, operating conditions and practices of use, in a robust way. In fact, the success of WSON-related applications seems to be strongly related to the validation of properties of the network protocols used in these systems. In [33] Alina et al. present a survey with a comprehensive review of the literature on protocol engineering techniques and they discuss the challenges imposed by WSONs to the protocol engineering community. Many of the approaches presented in that paper can be extended to other kinds of networks. They present formal and non-formal approaches. With respect to the latter ones, in many works, e.g., [34,35,36], simulation is used to check protocol designs. They also use the concept of monitors, as we use observers, to deﬁne entities that check the performance metrics during the simulation executions and generate the traces ﬁles accordingly. However, most of these proposals do not use model-driven techniques: they implement the algorithms in general purpose programming languages such as Java. This results in expensive development costs and eﬀorts, in lack of ﬂexibility and in error-prone simulators due to the complexity of the systems to simulate. Other works (e.g., [37]) have developed prototyping environments for certain kinds of networks, in order to avoid this problem. This is also what we have done, but showing how the use of domain speciﬁc languages and a model-driven environment such as the one provided by e-Motions can be even more ﬂexible, eliminating the restriction of having to deal with particular kinds of networks (the ones for which the simulation framework was developed).

202

J. Troya et al.

The basic e-Motions language has been presented in other papers, e.g., [2,3,8]. This paper demonstrates how some of its features and mechanisms can be combined and used to accomplish lightweight modelling and testing of systems, and in particular of communication networks.

6

Conclusions and Future Work

In this paper we have presented a lightweight approach for the design and analysis of non-trivial systems, and showed how these can be realized using e-Motions. A communication network example has been used as a proof-of-concept of our proposal, although similar kinds of systems and analysis can be applied in other environments such as P2P networks or the Cloud [20]. There are several lines of work that we would like to explore next. For example, we would like to assign probabilities to the rules. In this way, apart from needing a match of the LHS pattern of the rule, we would also need some probability parameter to be satisﬁed in order to ﬁre the rule. The use of a similar approach to the one used in Probabilistic Rewrite Theories (pMaude) [38] could be interesting, although other possibilities can also be considered. We also plan to connect e-Motions to other interesting Maude tools, such as the LTL model checker [5]. The connection is now possible but requires human intervention. Our plans are to fully integrate some of these tools so that they become accessible to the user in a transparent way. Finally, we are deﬁning connections to other formalisms in addition to Maude, in order to make use of the tools available in these semantic domains for performance and reliability analysis. In particular we are considering connections with Stochastic Petri Nets (SPN), Queue Networks Models (QNM) and Stochastic Process Algebras (SPA), and their associated tools. Although the expressiveness of these notations is diﬀerent from the expressiveness of e-Motions, having access from e-Motions to their analytic tools can be of great help. Making the connections work in both ways can also be interesting. In this way these notations can have a direct access to the simulation facilities provided by e-Motions. Acknowledgements. We would like to thank the anonymous reviewers for their useful suggestions. This work has been supported by Spanish Research Projects P07-TIC-03184 and TIN2008-03107.

References 1. Zave, P.: Lightweight modeling of network protocols, http://www2.research.att.com/~ pamela/model.html 2. Rivera, J.E., Dur´ an, F., Vallecillo, A.: A graphical approach for modeling timedependent behavior of DSLs. In: Proc. of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2009), pp. 51–55. IEEE Computer Society, Los Alamitos (2009)

Lightweight Testing of Communication Networks with e-Motions

203

3. Troya, J., Rivera, J.E., Vallecillo, A.: Simulating domain speciﬁc visual models by observation. In: Proc. of the Symposium on Theory of Modeling and Simulation (DEVS 2010), Orlando, FL, US (April 2010) 4. Czarnecki, K., Helsen, S.: Classiﬁcation of model transformation approaches. In: OOPSLA 2003 Workshop on Generative Techniques in the Context of MDA (2003) 5. Clavel, M., Dur´ an, F., Eker, S., Lincoln, P., Mart´ı-Oliet, N., Meseguer, J., Talcott, C.: All About Maude - A High-Performance Logical Framework. LNCS, vol. 4350. Springer, Heidelberg (2007) 6. Object Management Group: Object Constraint Language (OCL) Speciﬁcation. Version 2.2, OMG Document formal/2010-02-01 (February 2010) 7. Rold´ an, M., Dur´ an, F.: Representing UML models in mOdCL (2008), http://maude.lcc.uma.es/mOdCL 8. Rivera, J.E., Dur´ an, F., Vallecillo, A.: On the behavioral semantics of real-time ¨ domain speciﬁc visual languages. In: Olveczky, P.C. (ed.) WRLA 2010. LNCS, vol. 6381, pp. 174–190. Springer, Heidelberg (2010) 9. Jouault, F., Allilaire, F., B´ezivin, J., Kurtev, I.: ATL: A model transformation tool. Science of Computer Programming 72(1-2), 31–39 (2008) 10. Rivera, J.E., Vallecillo, A., Dur´ an, F.: Formal speciﬁcation and analysis of domain speciﬁc languages using Maude. Simulation: Transactions of the Society for Modeling and Simulation International 85(11/12), 778–792 (2009) ¨ 11. Olveczky, P., Meseguer, J.: Semantics and pragmatics of Real-Time Maude. HigherOrder and Symbolic Computation 20(1-2), 161–196 (2007) 12. Atenea: Packet Switching Simulation Results (2011), http://atenea.lcc.uma.es/index.php/Page/Resources/ E-motions/PacketSwitchingExample/Results 13. Jain, M., Dovrolis, C.: End-to-end available bandwidth: measurement methodology, dynamics, and relation with tcp throughput. IEEE/ACM Transactions Networking 11(4), 537–549 (2003) 14. Carter, R.L., Crovella, M.E.: Measuring bottleneck link speed in packet-switched networks. Perform. Eval. 27-28, 297–318 (1996) 15. Lindh, T.: Performance management in switched ATM networks. In: Trigila, S., Mullery, A., Campolargo, M., Vanderstraeten, H., Mampaey, M. (eds.) IS&N 1998. LNCS, vol. 1430, pp. 439–450. Springer, Heidelberg (1998) 16. Paciﬁci, G., Stadler, R.: Integrating resource control and performance management in multimedia networks. In: Proc. of the IEEE International Conference on Communications, Seattle, WA, vol. 3, pp. 1541–1545 (1995) 17. Balsamo, S., Marco, A.D., Inverardi, P., Simeoni, M.: Model-based performance prediction in software development: A survey. IEEE Trans. on Software Engineering 30(5), 295–310 (2004) 18. Cortellessa, V., Di Marco, A., Inverardi, P.: Integrating performance and reliability analysis in a non-functional MDA framework. In: Dwyer, M.B., Lopes, A. (eds.) FASE 2007. LNCS, vol. 4422, pp. 57–71. Springer, Heidelberg (2007) 19. Tawhid, R., Petriu, D.C.: Integrating performance analysis in the model driven development of software product lines. In: Busch, C., Ober, I., Bruel, J.-M., Uhl, A., V¨ olter, M. (eds.) MODELS 2008. LNCS, vol. 5301, pp. 490–504. Springer, Heidelberg (2008) 20. Li, J., Chinneck, J., Woodside, M., Litoiu, M., Iszlai, G.: Performance model driven QoS guarantees and optimization in clouds. In: Proceedings of the 2009 ICSE Workshop on Software Engineering Challenges of Cloud Computing, CLOUD 2009, pp. 15–22. IEEE Computer Society, Vancouver (2009) 21. OMG: UML Proﬁle for Modeling and Analysis of Real-time and Embedded Systems (MARTE). Object Management Group (June 2008), OMG doc. ptc/08-06-08

204

J. Troya et al.

22. Marsan, A.: Stochastic petri nets: An elementary introduction. In: Rozenberg, G. (ed.) APN 1989. LNCS, vol. 424, pp. 1–29. Springer, London (1990) 23. Denning, P.J., Buzen, J.P.: The operational analysis of queueing network models. ACM Comput. Surv. 10, 225–261 (1978) 24. Clark, A., Gilmore, S., Hillston, J., Tribastone, M.: Stochastic process algebras. In: Bernardo, M., Hillston, J. (eds.) SFM 2007. LNCS, vol. 4486, pp. 132–179. Springer, Heidelberg (2007) 25. Yaron, O., Sidi, M.: Performance and stability of communication networks via robust exponential bounds. IEEE/ACM Transactions on Networking 1, 372–385 (1993) 26. Heckel, R.: Stochastic analysis of graph transformation systems: A case study in P2P networks. In: Van Hung, D., Wirsing, M. (eds.) Theoretical Aspects of Computing ICTAC 2005. LNCS, vol. 3722, pp. 53–69. Springer, Heidelberg (2005) 27. de Lara, J., Vangheluwe, H., Mosterman, P.J.: Modelling and analysis of traﬃc networks based on graph transformation. In: Proceedings of the FORMS/FORMATS 2004 Symposium on Formal Methods for Automation and Safety in Railway and Automotive Systems, Braunschweig, Germany, pp. 120–127 (2004) 28. Burmester, S., Giese, H., Hirsch, M., Schilling, D., Tichy, M.: The Fujaba real-time tool suite: model-driven development of safety-critical, real-time systems. In: ICSE 2005, pp. 670–671. ACM, NY (2006) 29. Gyapay, S., Heckel, R., Varr´ o, D.: Graph transformation with time: Causality and logical clocks. In: Corradini, A., Ehrig, H., Kreowski, H.-J., Rozenberg, G. (eds.) ICGT 2002. LNCS, vol. 2505, pp. 120–134. Springer, Heidelberg (2002) 30. Syriani, E., Vangheluwe, H.: Programmed graph rewriting with time for simulationbased design. In: Vallecillo, A., Gray, J., Pierantonio, A. (eds.) ICMT 2008. LNCS, vol. 5063, pp. 91–106. Springer, Heidelberg (2008) ¨ 31. Boronat, A., Olveczky, P.C.: Formal real-time model transformations in MOMENT2. In: Rosenblum, D.S., Taentzer, G. (eds.) FASE 2010. LNCS, vol. 6013, pp. 29–43. Springer, Heidelberg (2010) 32. de Lara, J., Vangheluwe, H.: Automating the transformation-based analysis of visual languages. Formal Aspects of Computing 22(3-4), 297–326 (2010) 33. Viana, A.C., Maag, S., Zaidi, F.: One step forward: Linking wireless self-organizing network validation techniques with formal testing approaches. ACM Comput. Surv. 43, 7:1–7:36 (2011) 34. Girod, L., Elson, J., Cerpa, A., Stathopoulos, T., Ramanathan, N., Estrin, D.: Em*: a software environment for developing and deploying wireless sensor networks. In: Proceedings of the USENIX General Track (2004) 35. Girod, L., Stathopoulos, T., Ramanathan, N., Elson, J., Osterweil, E., Schoellhammer, T., Estrin, D.: A system for simulation, emulation, and deployment of heterogeneous sensor networks. In: Proceedings of the Second ACM Conference on Embedded Networked Sensor Systems, pp. 201–213. ACM Press, New York (2004) 36. Keshav, S.: Real: A network simulator. Technical report, Berkeley, CA, USA (1988) 37. Ben Abdesslem, F., Iannone, L., Dias de Amorim, M., Obraczka, K., Solis, I., Fdida, S.: A prototyping environment for wireless multihop networks. In: Fdida, S., Sugiura, K. (eds.) AINTEC 2007. LNCS, vol. 4866, pp. 33–47. Springer, Heidelberg (2007) 38. Agha, G., Meseguer, J., Sen, K.: PMaude: Rewrite-based speciﬁcation language for probabilistic object systems. Electronic Notes in Theoretical Computer Science 153(2), 213–239 (2006)

Author Index

Aguirre, Nazareno M.

15, 84

Oudinet, Johan

Degiovanni, Renzo 84 Drechsler, Rolf 152 Dross, Claire 102 102

Sindhu, Muddassar A. Soeken, Mathias 152 Tan, Li 171 Troya, Javier

H´eam, Pierre-Cyrille

Wille, Robert

Julliand, Jacques

51, 78

69

Poizat, Pascal 33 Ponzio, Pablo 84 Pretschner, Alexander

Galeotti, Juan P. 15 Gaudel, Marie-Claude 1 Giorgetti, Alain 78 Godefroid, Patrice 14 119

187

Masson, Catherine 119 Masson, Pierre-Alain 51 Meinke, Karl 134 Moy, Yannick 102

78

Filliˆ atre, Jean-Christophe Frias, Marcelo F. 15, 84

78

L´ opez-Romero, Fernando

Bautista, Jos´e M. 187 Bengolea, Valeria S. 15 Bentakouk, Lina 33 B¨ uchler, Matthias 69 Bu´e, Pierre-Christophe 51 Chebaro, Omar

Kosmatov, Nikolai

187

Vallecillo, Antonio

Za¨ıdi, Fatiha

152 33

187

69 134