This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
> Q5 ; ... Q5 : = Queue . c r e a t e <... >( M a x C a p a c i t y . QUEUE ) ; CAD . b i n d D i s p a t c h i n g Q u e u e <... >(Q5 , 5 ) ; ... thread threadgroup5 () { P a i r . t y p e<EventHandlerEnum , CAD . Event > p a i r ; EventHandlerEnum h a n d l e r ; CAD . E v e n t e v e n t ;
}
l o c l o c 0 : l i v e { h a n d l e r , e v e n t } when Queue . s i z e <... >(Q5) > 0 do i n v i s i b l e { p a i r : = Queue . g e t F r o n t <... >(Q5 ) ; Queue . dequeue <... >(Q5 ) ; h a n d l e r : = P a i r . f i r s t <... >( p a i r ) ; e v e n t : = P a i r . s e c o n d <... >( p a i r ) ; } goto l o c 1 ; l o c l o c 1 : l i v e {} i n v i s i b l e i n v o k e v i r t u a l f ( h a n d l e r , e v e n t ) goto l o c 0 ;
Fig. 8. Bogor dispatch queue and thread model for ModalSP (excerpts).
the Bogor model, queues are modeled using Queue and Pair extensions. Figure 8 illustrates the 5 hertz rate queue of pending event dispatches and the thread, threadgroup5, that cyclically dequeues dispatch pairs and invokes the component event handler encoded in each pair (note that pair type declarations are elided (i.e., <...>) for improved readability). Each correlator is represented as a deterministic finite-state automaton whose transition function is encoded as a static transition table. For each correlator, there is a single state variable that holds the current correlator state. Since the structure of correlators is fixed for a given system, the transition tables are not held in the state vector. 5.4
Summary of Data Portion of State-Vector
To summarize the modeling strategy discussed above, we present the state vector components related to data state of Cadena systems. The observable state of a Cadena assembly is comprised of all non-fixed system data. As we have noted above, correlator transition tables, subscriber lists, and component connection information are all fixed and are not considered part of the observable state. Definition 1. Cadena Data States are tuples (c, r, a, t, p) where: c = (c1 , . . . , ck ) stores the data states of component instances, each of which is comprised of a, possibly empty, set of mode attributes as defined by ci ’s component type. r = (qr1 , . . . , qrn ) are rate-specific queues of pairs, (c, e), recording the dispatch of event e to port c. a = (a1 , . . . , al ) stores the current states of each of the event correlation recognition automata. t records an abstraction of time used to trigger timeouts. p records the priority of the current thread being executed.
Model-Checking Middleware-Based Event-Driven Real-Time
171
The initial state is defined to have instance modes set to their initial values, correlation automata set to their start state, rate specific queues to be empty, t = 0, and the priority variable is set to the highest priority. The values of local variables in component handlers and methods and in the implementation of push methods and rate-specific threads cannot be observed outside their method activation by other threads or by property observables and are also not considered part of the observable state. Local variable are held in the state vector, but only during the corresponding method activations. In addition to the data state described above, Bogor maintains control information in the form of a program counter and a method call-stack frame for each thread. 5.5
Strategies for Modeling Scheduling and Time
The behavior of Cadena systems is driven by the triggering of middleware timeouts as described in Section 4 and is controlled by the scheduling policies of the thread-pool in the real-time event channel. Finding an effective strategy for modeling these timeouts and thread-scheduling is a central issue in the construction of Cadena models. When analyzing concurrent systems, most model-checkers do not attempt to exploit knowledge of specific timing or scheduling strategies but instead explore all possible interleavings of concurrent actions. If we followed this approach, we would allow timeout events to occur non-deterministically between every system transition and we would allow actions from different threads to be interleaved non-deterministically without consideration of priorities or other scheduling constraints. While such a strategy is sound in that it covers all possible system behaviors, the number of states generated makes it impractical for all but the smallest systems. In the subsections below, we describe several strategies that we use to reduce infeasible interleavings. Each strategy incorporates constraints based on observations about priority scheduling and timeout policies implemented by the real-time middleware. Priority-based scheduling: Having the model-checker non-deterministically explore interleavings without considering thread priorities obviously introduces schedules that are infeasible in the actual system, e.g., a schedule that continues to execute transitions from a lower priority thread even though a higher-priority thread is enabled. Inter-rate-group timeout constraints: Having the model-checker nondeterministically generate timeout events introduces schedules that are infeasible in the actual system, e.g., a 5 Hz timeout event should not occur more frequently than a 20 Hz timeout event. We present strategies that reduce infeasible interleavings by taking into account the appropriate relative frequency of timeout events, i.e., by taking into account constraints that exist between timeouts of different rate groups. Intra-rate-group timeout constraints: Having the model-checker nondeterministically generate timeout events introduces infeasible schedules
172
X. Deng et al.
where a timeout for a rate group r occurs before all events in the current frame for r are dispatched or before the previous timeout from group r is even dispatched. We constrain the generation of time-out events to ensure that timeouts from the same rate group are not triggered “too quickly”. This strategy constrains the occurrence of timeouts by considering the relative lengths of the real-time frames and constrains scheduling by considering priority information. Lazy-time with priority scheduling: In addition to the techniques used in the strategy above, this strategy also considers timing estimates for system transition which allows additional infeasible schedules to be removed from consideration. 5.6
Representing Priority-Based Scheduling Information
Bold Stroke systems are priority scheduled based on the results of rate monotonic analysis of a set of harmonic rate groups. The CAD call connectEvent(), illustrated in Figure 7, assigns a rate, and hence a priority, to each component handler for a given event. The default non-deterministic scheduling policy in Bogor is implemented by a module that calculates the set of enabled transitions in a given state and passes that set to the state exploration module, which explores each possible outgoing transition. When reporting our experiments, we refer to models that use this strategy as priority-unaware. For Cadena models, a Bogor plugin is used that intercepts the set of enabled transitions in a given state, selects the transitions with the highest priority and passes these transitions on to the state exploration module. As expected, this yields dramatic reductions in the state space, as shown in Section 6, and also improves the precision of the state space since only infeasible schedules are eliminated (i.e., ones on which a lower-priority transition executes when a higher-priority transition is enabled). We refer to models that use this strategy as priority aware. Variations of this plugin are used in the following models to allow for interleaving of timeouts with the highest-priority enabled transition. 5.7
Representing Intra-rate-group Timing Constraints
The treatment of time, t, determines, in part, the fidelity of the model with respect to the real system’s behaviors. If detailed timing information is available one can keep track of time as component actions are executed and use that time value to trigger periodic events. However, even when timing information is not available, one can still reduce the occurrence of timeout events based on both intra- and inter-rate-group constraints. Intra-rate-group constraints that we consider involve the notion of frame overrun. A frame overrun occurs when a timeout event er for rate group r occurs before all events e triggered directly or indirectly by the previous timeout for r are processed by the rate group’s thread tr . In normal situations, a timeout er occurs and is dispatched, other events arrive in the event channel’s dispatch queues (including those associated with r), and thread tr becomes idle after all
Model-Checking Middleware-Based Event-Driven Real-Time
173
events associated with r have been dispatched. The time that tr remains idle waiting for the next r timeout is called slack time. If a system has a frame overrun error, a thread tr has no slack time – it is unable to finish all of its work before the next timeout er arrives. Note that exploring the state-space of systems where arbitrary frame overruns are modeled results in a huge number of additional system behaviors that would very likely be infeasible if actual timing data were considered (timing data would allow us to conclude that in most cases frame overruns do not occur). While frame overruns are a real source of bugs in Bold Stroke systems, engineers have other tools and methods for detecting these types of errors. Accordingly, we will reduce the state space that we explore using two strategies. The first strategy which we call no overruns assumes that no frame overruns occur at all. This is implemented by having the model-checker scheduler only emit a timeout event for rate group r if there are no enabled transitions associated with rate group r – which models the situation where tr has become idle because its associated dispatch queue is empty. The second strategy which we call limited overruns is implemented by having the model-checker scheduler only emit a timeout event er if there is no other timeout event remaining in the r dispatch queue (but other non-timeout events may still be waiting in the queue for dispatch). Intuitively, this model includes overruns that only spill over into the very next frame but does not include overruns where processing is ‘late’ by more than one additional frame. 5.8
Representing Inter-rate-group Timing Constraints
The strategies related to frame overrun in the previous section constrain timeout events by considering when they should occur relative to other timeouts from the same rate group. We now describe a strategy which we call the relativetime (RT) strategy that constrains the issuing of timeout events by considering when a timeout for r should occur relative to a timeout for a different rate group r . Specifically, we take advantage of the fact that in rate-monotonic scheduling theory (which is used in Bold Stroke systems), the frame associated with a rate can be evenly divided into some whole number of r -frames for each rate r that is higher than r. In the example system of Figure 3, the frame of the slowest rate (1 Hz) can be divided into 5 5 Hz frames, and each 5 Hz frame can be divided into 4 20 Hz frames. The longest frame/period (the frame associated with the lowest rate) is called the hyper-period. The relative-time model enforces the following constraints related to issuing of timeouts: – a single timeout is issued for the slowest rate group in the hyper-period, – timeouts for rate groups, ri and rj where ri > rj , are issued such that ri /rj timeouts of rate ri are issued in a rj frame. These constraints determine the total number and relative ordering of instances of timeouts that may occur in the hyper-period.
174
X. Deng et al.
CAD . Component Timer ; ... Timer : = CAD . c r e a t e C o m p o n e n t ( ” Timer ” ) ; CAD . d e c l a r e E v e n t S o u r c e P o r t <EventType >(Timer , ” timeOut5 ” , EventType . TimeOut ) ; ... thread timerThread () { l o c l o c 0 : l i v e {} when t r u e do { t i m e : = ( t i m e + 1 ) % 2 0 ; } goto l o c 0 ; } ... thread timeOutSenderThread ( ) { ... l o c l o c 1 : / / 5 Hz t i m e o u t c a s e when t i m e % ( 2 0 / 5 ) = = 0 do i n v i s i b l e { } goto l o c I n v o k e 5 ; when t i m e % ( 2 0 / 5 ) ! = 0 do i n v i s i b l e { } goto l o c 2 ; ... loc locInvoke5 : l i v e { localTime} i n v i s i b l e i n v o k e p u s h O f P r o x y ( Timer , ” timeOut5 ” , CAD . c r e a t e E v e n t <EventType > ( EventType . TimeOut ) ) goto l o c 2 ; ... l o c l o c 2 : / / 1 Hz t i m e o u t c a s e ... }
Fig. 9. Timer and TimeOutSender thread models for ModalSP (excerpts).
Figure 9 shows the Bogor code for two threads that are used to model this strategy. Thread timerThread increments an abstraction of time where each ’tick’ (i.e., each increment of the time variable) represents the passing of time corresponding to the shortest frame in the system (e.g., in the ModalSP, each tick represents a 20 Hz frame). The time variable wraps around every 20 ticks which corresponds to the fact that there are 20 Hz frames in the 1 Hz hyperperiod. Thread timeOutSenderThread models the behavior of the rate-specific timer threads in the middleware discussed in Section 4. This thread monitors time and when it observes a change in the time value, it passes through a case statement to see which timeout events should be dispatched at that point. Since a time tick represents the period of the shortest frame, a new timeout event for the fastest rate is issued on each pass through the case statement. In our example system, the 5 Hz timeout happens every fourth tick. To represent the occurrence of a timeout, the thread enqueues the timeout event through the standard push call. From the explanation above, it is clear that the RT model only establishes the occurrence of timeouts relative to each other – it does not relate timeout occurrences to the time required by component event handlers and method execution. Thus, it is now important to understand when timeout actions may occur with respect to actions that occur inside of component handlers, i.e., when can these actions be interrupted by timeouts. To see that the model safely approximates all interleavings of timeouts and component actions (given the constraint on no frame overruns) consider Figure 10. This figure illustrates four points during a system execution which contains 5 Hz and 10 Hz rate processing. The 10 Hz and 5 Hz timeouts are queued together (e.g., at the point 1) since they both have frames that begin at the
Model-Checking Middleware-Based Event-Driven Real-Time
10Hz and 5Hz
175
10Hz timeout
10Hz
10Hz execution step 5Hz timeout 5Hz execution step
10Hz priority delay interleaving
5Hz time 1
2
3
4
Fig. 10. Relative-time Environment.
same point. However, the 10 Hz timeout event is dispatched first due to its higher priority. Once the all the actions associated with 10 Hz component processing complete (e.g., at point 2), the model-checker scheduler begins consideration of lower priority actions and the 5 Hz timeout is dispatched leading to 5 Hz component processing. Our no overruns assumption entails that processing the 10 Hz component actions does not require more time than the period of the 10 Hz frame — thus, the next 10 Hz timeout cannot occur before point 2. Since we are not modeling the actual time required for carrying out component actions, it impossible to determine the relationship between the time required for 5 Hz component action processing (e.g., the duration from point 2 to 4) and the time until the next 10 Hz timeout (e.g., the duration from point 2 to 3). To safely cover all possibilities, we must allow for any relationship between these durations. To model all such relationships, we adapt Bogor’s standard scheduling module to consider all interleavings of enabled timeouts with the enabled transitions. On the right in Figure 10 the interleavings of the 10 Hz timeout and the enabled transitions performed during 5 Hz component processing are illustrated. The dark grey double circle represents the dispatching of the leftmost 10 Hz timeout event. This is followed by three dark grey circles representing transitions in 10 Hz component processing: the first branch point represents the choice between the next 10 Hz timeout (on the left) or dispatching the already queued 5 Hz timeout event (on the right). If 5 Hz processing is selected then the choice between the 10 Hz timeout and 5 Hz processing repeats for the next enabled 5 Hz transition illustrated as a light grey circle. 5.9
Lazily-Timed Components
In the relative-time model, timeouts are arranged in a proper order and ratio with respect to each other, but there are no constraints that guarantee that, e.g., the interval between time outs is appropriate for the correspond period. This means that the model may have interleavings in which a timeout, e.g., for ri , occurs prematurely with respect to an action sequence whose duration is less than period(ri ). For example, if the 5 Hz component processing (i.e., from point 2 to 4) in Figure 10 is guaranteed to be less than the time to the next
176
X. Deng et al. 10Hz
10Hz priority delay
5Hz time 1
2
3
Fig. 11. Lazily-timed Environment.
10 Hz timeout (i.e., point 4 comes before point 3) then the interleavings of 10 Hz timeouts with 5 Hz processing in the RT model will be infeasible. The lazilytimed (LT) component model addresses this by leveraging worst-case estimates of the running time of components; these will be available for Cadena systems to support rate monotonic analysis. This model can be configured for whatever granularity of timing information is available. Here we consider worst-case timing estimates for event handlers. Conceptually, the estimates are used to determine whether a handler can run without interruption before the next timeout occurs and, if not, the model non-deterministically interleaves action sequences from the handler with timeouts and higher-priority actions that follow from timeouts. This model modifies the data associated with time to record the intra-hyperperiod (IHP) time normalized by the least common factor of all handler durations and timeout periods, the guards in timeOutSenderThread from Figure 9 are adjusted accordingly, and each component handler is modified to include an increment of time. Figure 11 illustrates how these increments are performed. It shows the execution of 5 Hz component processing subsequent to completion of 10 Hz processing in a frame. There are two cases: (1) the worst-case time estimate of the 5 Hz processing (i.e., which runs up to point 2) is less than or equal to the next timeout (i.e., timeout occurs at point 3) or (2) it is not (i.e., timeout occurs at point 1 and interrupts the 5 Hz actions). In case (1), the IHP time is incremented by the worst-case timing estimate of the currently running 5 Hz event handler and the state space exploration algorithm proceeds; note that there is no branching in the state space for this case. In case (2), the IHP time is incremented to the next timeout (i.e., point 1), a non-deterministically chosen prefix of the currently running 5 Hz handler is executed, and then the 10 Hz timeout is performed. By choosing a prefix of the handler actions, we are modeling all possible distributions of timing across the actions of the handler. The remaining portion of the handler is left for the state-space exploration algorithm after the 10 Hz timeout, and subsequent 10 hertz processing is performed. The difference between point 2 and point 1 (i.e., the worst-case execution time of the handler remaining time of the 10 Hz frame) is assigned to that remaining portion as its duration. This model can be seen as a refinement of the RT model. It eliminates interleavings when the timing estimates guarantee that a group of highest-priority enabled transitions are guaranteed to complete before the next timeout. In the example in Figure 10, if the right-most three light-grey circles correspond to a
Model-Checking Middleware-Based Event-Driven Real-Time
177
5 Hz component handler body whose worst-case execution bound is less than the time to the next 10 Hz timeout, then there would be no branching in that portion of the state space (i.e., the lower two left outgoing arcs to 10 Hz timeouts are eliminated).
6
Experimental Results
Table 1 shows the results of evaluating our strategies using four example systems provided by Boeing engineers. As an example of how to read a system description, the ModalSP scenario that we have used as an example has three threads (for rate groups 1 Hz, 5 Hz, and 20 Hz), 8 components, an event correlation (e/c), and 125 events being generated per one second hyper-period (hp). For each scenario, we give data for five models that incorporate the modeling strategies presented in the previous section. – (R) is the reference model. There is no scheduling policy for the thread groups in the scenario (it is priority unaware and has no intra-rate-group timing constraints). Since a completely interleaved execution is infeasible to check, the relative time constraints are used though. – (RT-1) uses two policies: priority aware scheduling and the relative time environment where we implement the no frame overruns strategy for the highest-priority thread only. – (RT-2) is like (RT-1), but also assumes there are no frame overruns for all threads. – (LT) is like (RT-2) but uses the lazy time environment model. For each example, we collect the number of transitions trans, states, time, and memory consumption mem at the end of the search. The numbers of transitions and states are both listed because some of steps in the model are marked as invisible (atomic) for which Bogor will not save the states. The experiments were run on a Pentium 4 2.53 GHz with 1.5Gb RAM using the Java 2 Platform. Bogor’s collapse compression[17] and heap symmetry [18] and process symmetry [2] reductions are used in all of the experiments. Each of the experiments represents a complete exploration of the state-space of the system. From the table, the state space generally decreases from model (R), (RT-1), (RT-2), to (LT). This shows that by incorporating more knowledge (e.g., the scheduling policy) of the model that is being checked, less states need to be explored. For example, Medium, the largest scenario that we have, cannot be model checked using Bogor or our previous dSpin implementation [14] without employing the reduction strategies used in (RT-2) and (LT). For Basic the states are the same for model (RT-1) and (RT-2) because it only has a single thread (thus, there is no interleaving). Model (R) has a larger number of states because the lack of constraints allows the timeout to occur even when events associated with the current frame are still being dispatched. Model (LT) has two more states than (RT-2) due to the overhead introduced by the timing transitions. Bogor runs out of memory checking ModalSP (R) (at 3 million states) and Medium (RT-1) (at 13 million states). It is interesting that the states for ModalSP
178
X. Deng et al. Table 1. Experiment Data. Example System Basic Scenario Threads: 20Hz Components: 3 Events: 2 per .05sec hp Multi-Rate Scenario Threads: 20Hz, 40Hz Components: 6 Events: 6 per .05sec hp ModalSP Scenario Threads: 1Hz, 5Hz, 20Hz Components: 8 (e/c) Events: 125 per 1sec hp Medium Scenario Threads: 1Hz, 20Hz Components: 50 Events: 820 per 1sec hp
(R) trans 111 states 20 time .16 sec mem .51Mb trans 1.36M states .12M time 5 min mem 16Mb trans o.m. states 3M+ time o.m. mem o.m. trans o.m. states — time o.m. mem o.m.
(RT-1) 42 12 .11 sec .5Mb 7.5K 1.5K 1.9 sec .77Mb .92M 20.9K 20 sec 4.1Mb o.m. 13M+ o.m. o.m.
(RT-2) 42 12 .09 sec .5Mb .98K .1K .38 sec .61Mb 38.2K 9.1K 8.59 sec 1.61Mb 3.79M .74M 29 min 71.8 Mb
(LT) 44 14 .11 sec .51Mb .15K 33 .19 sec .61Mb 6.27K 1.56K 2.11 sec 1.45Mb .36M 74.5K 3 min 21.5Mb
(R) require more memory than the states for Medium (RT-1). This is an effect of the collapse compression that is used. Specifically, there are three threads in ModalSP (R), but only two threads in Medium (RT-1). In addition, ModalSP (R), which has fewer scheduling constraints, allows more interleaving than Medium (RT-1). Thus, the collapse compression can save more in Medium (RT-1) than ModalSP (R), because there are more similar state bit patterns in Medium (RT-1) than in ModalSP (R).
7
Related Work
Garlan and Khersonsky [11] describe an approach for checking publish-subscribe systems (which they refer to as implicit invocation systems) using SMV. We build on their key insights of factoring system models into two parts: (1) a reusable model of run-time event-delivery infrastructure, and (2) application dependent, user-specified component models. However, we support much more directly the forms of component structure and connections (i.e., CCM structures and object references) and event-delivery mechanisms (i.e., RT CORBA middleware) found in real systems. This advance is achieved by leveraging Bogor’s direct modeling of OO concepts (as compared to the SMV input language which provides very little support for modeling programming language features) and Bogor’s extension mechanism (which allows complex middleware behavior to be captured internally to the model-checker). In addition, [11] does not provide any performance data, nor does that work consider any state-space reductions based on priorities, scheduling, or timing constraints that seem critical for scaling to realistic applications.
Model-Checking Middleware-Based Event-Driven Real-Time
179
There has been a large body of work on timing and schedulability analysis for component-based systems (see also [12] in this volume). As these techniques have matured, they have been integrated into environments that support the development of real-time systems. For example, MetaH [25] and Geodesic [6] are frameworks that support the reuse of components written in Ada and Java, respectively, in real-time systems. These frameworks include a range of timing analyses and automatically generate infrastructure code that coordinates the execution of component code in a way that achieves the system’s timing requirements. Cadena is complementary to this work in that it targets logical properties of a system using both light-weight and heavy-weight analysis techniques. Ptolemy [20] is a framework that allows a wide variety of formal descriptions of components and their behavior to be integrated into a single system. User’s provide sufficient detail in these descriptions to allow implementations to be automatically generated. Ptolemy provides a run-time infra-structure to mediate between components that have different execution models. In contrast, Cadena models intentionally leave out detail in order to provide more abstract system descriptions that are amenable to analysis for large systems. While Cadena provides some code generation capabilities, we do attempt to generate component method implementations. Model checking [4] has become extremely popular as a technology for analyzing behavioral models of software artifacts. Researchers have extracted such models from source code (e.g., [15,5]), UML design artifacts (e.g., [21,19]) and architectural descriptions (e.g., [1]). The difficulty with all applications of model checking is scaling it up to apply to realistically large and complex systems. Recent years have seen an enormous amount of research on the systematic abstraction of models to enable more tractable reasoning. We take a different approach in Cadena by exploiting the natural abstractions that arise when developing high-level design models of systems.
8
Conclusions
We believe the idea of a flexible model-checking framework that allows domainspecific extensions (such as the ones that we have used for encoding models of CORBA communication layers) can be a very effective approach for modelchecking modern distributed system designs and implementations. We are currently working with Boeing engineers to incorporate other forms of domain information (e.g., common specification idioms) and other forms of light-weight checking (e.g., interface protocol checking, refinement checking) and static analysis into Cadena.
Acknowledgements Thanks to various members of the Boeing Bold Stroke team including David Sharp, Carol Sanders, Wendy Roll, and Dennis Noll for their comments on the work described in this paper.
180
X. Deng et al.
References 1. R. Allen and D. Garlan. A formal basis for architectural connection. ACM Transactions on Software Engineering and Methodology, July 1997. 2. D. Bosnacki, D. Dams, and L. Holenderski. Symmetric spin. In International Journal on Software Tools for Technology Transfer. Springer-Verlag, 2002. 3. G. Brat, K. Havelund, S. Park, and W. Visser. Java PathFinder – a second generation of a Java model-checker. In Proceedings of the Workshop on Advances in Verification, July 2000. 4. E. Clarke, O. Grumberg, and D. Peled. Model Checking. MIT Press, 2000. 5. J. C. Corbett, M. B. Dwyer, J. Hatcliff, S. Laubach, C. S. P˘ as˘ areanu, Robby, and H. Zheng. Bandera : Extracting finite-state models from Java source code. In Proceedings of the 22nd International Conference on Software Engineering, June 2000. 6. D. de Niz and R. Rajkumar. Geodesic - a reusable component framework for embedded real-time systems. Technical report, Carnegie Mellon University, 2002. 7. C. Demartini, R. Iosif, and R. Sisto. dspin : A dynamic extension of SPIN. In Theoretical and Applied Aspects of SPIN Model Checking (LNCS 1680), Sept. 1999. 8. W. Deng, M. Dwyer, J. Hatcliff, G. Jung, Robby, and G. Singh. Model-checking middleware-based event-driven real-time embedded software (extended version). Forthcoming – April 2003. 9. B. Doerr and D. Sharp. Freeing product line architectures from execution dependencies. In Proceedings of the Software Technology Conference, May 1999. 10. Eclipse Consortium. Eclipse website. http://www.eclipse.org, 2001. 11. D. Garlan and S. Khersonsky. Model checking implicit-invocation systems. In Proceedings of the 10th International Workshop on Software Specification and Design, Nov. 2000. 12. G. Goessler and J. Sifakis. Composition for component-based modeling. In Proceedings of the First International Symposium on Formal Methods for Components and Objects, volume (this volume) of Lecture Notes in Computer Science. Springer, 2002. 13. T. H. Harrison, D. L. Levine, and D. C. Schmidt. The design and performance of a real-time corba event service. In Proceedings of the 1997 ACM SIGPLAN conference on Object-oriented programming systems, languages and applications, pages 184–200. ACM Press, 1997. 14. J. Hatcliff, W. Deng, M. Dwyer, G. Jung, and V. Prasad. Cadena: An integrated development, analysis, and verification environment for component-based systems. In Proceedings of the 25th International Conference on Software Engineering (to appear), 2003. 15. K. Havelund and T. Pressburger. Model checking Java programs using Java PathFinder. International Journal on Software Tools for Technology Transfer, 1999. 16. G. J. Holzmann. The model checker SPIN. IEEE Transactions on Software Engineering, 23(5):279–294, May 1997. 17. G. J. Holzmann. State compression in SPIN: Recursive indexing and compression training runs. In Proceedings of Third International SPIN Workshop, Apr. 1997. 18. R. Iosif. Symmetry reduction criteria for software model checking. In Proceedings of Ninth International SPIN Workshop, volume 2318 of Lecture Notes in Computer Science, pages 22–41. Springer-Verlag, Apr. 2002.
Model-Checking Middleware-Based Event-Driven Real-Time
181
19. D. Latella, I. Majzik, and M. Massink. Automatic verification of a behavioural subset of UML statechart diagrams using the SPIN model-checker. Formal Aspects of Computing, 11(6):637–664, 1999. 20. E. A. Lee. Overview of the ptolemy project. Technical Report UCB/ERL M01/11, University of California, Berkeley, Mar. 2001. 21. J. Lilius and I. P. Paltor. vUML: A tool for verifying UML models. In Proceedings of the 14th IEEE International Conference on Automated Software Engineering, 1999. 22. Robby, M. B. Dwyer, and J. Hatcliff. Bogor: An extensible and highly-modular model checking framework. In Proceedings of the 2003 ACM Symposium on Foundations of Software Engineering (FSE 2003), 2003. 23. Robby, M. B. Dwyer, and J. Hatcliff. Bogor Website. http://www.cis.ksu.edu/bandera/bogor, 2003. 24. H. Sipma. Event correlation: A formal approach. Technical Report Draft, Stanford University, July 2002. 25. S. Vestal. Metah user’s manual. http://www.htc.honeywell.com/metah, 1998.
Equivalent Semantic Models for a Distributed Dataspace Architecture Jozef Hooman1,2 and Jaco van de Pol2 1
University of Nijmegen, Nijmegen, The Netherlands [email protected] http://www.cs.kun.nl/∼hooman/ 2 CWI, Amsterdam, The Netherlands [email protected] http://www.cwi.nl/∼vdpol/
Abstract. The general aim of our work is to support formal reasoning about components on top of the distributed dataspace architecture Splice. To investigate the basic properties of Splice and to support compositional verification, we have defined a denotational semantics for a basic Splice-like language. To increase the confidence in this semantics, also an operational semantics has been defined which is shown to be equivalent to the denotational one using the theorem prover PVS. A verification framework based on the denotational semantics is applied to an example of top-down development and transparent replication.
1
Introduction
The general aim of our work is to support the development of complex applications on top of industrial software architectures by means of formal methods. As a particular example, we consider in this paper components on top of the software architecture Splice [Boa93,BdJ97] which has been devised at Thales Nederland (previously called Hollandse Signaalapparaten). It is used to build large and complex embedded systems such as command and control systems, process control systems, and air traffic management systems. The main goal of Splice is to provide a coordination mechanism between loosely-coupled heterogeneous components. In typical Splice-applications it is essential to deal with large data streams from sensors, such as radars. Hence Splice should support real-time and high-bandwidth distribution of data. Another aim is to support fault-tolerance, e.g., it should be possible to replicate components transparently, i.e. without affecting the overall system behaviour. Splice is data-oriented, with distributed local databases based on keys. It uses the publish-subscribe paradigm to get loosely-coupled data producers and consumers. An important design decision is to have minimal overhead for data management, allowing a fast and cheap implementation that indeed allows huge data streams. For instance, Splice has no standard built-in mechanisms to ensure global consistency or global synchronization. If needed, this can be constructed for particular data types on top of the Splice primitives. Note that this is quite F.S. de Boer et al. (Eds.): FMCO 2002, LNCS 2852, pp. 182–201, 2003. c Springer-Verlag Berlin Heidelberg 2003
Equivalent Semantic Models for a Distributed Dataspace Architecture
183
different from Linda [Gel85] and JavaSpaces [FHA99], which have a central data storage. The latter also has a transaction mechanism to handle distributed access. Our aim is to reason about components of the distributed dataspace architecture Splice in a compositional way. This means that we want to deduce properties of the parallel composition of Splice-components using only the specifications of the externally visible behaviour of these components. Compositionality supports verification of development steps during the development process. Many examples in the literature show that it is convenient to specify components using explicit assumptions about the environment. Concerning Splice, in [HH02] we propose a framework with an explicit assumption about the quality of data streams published by environment and a similar commitment of the component about its produced data. When putting components in parallel, assumptions can be discharged if they are guaranteed by other components. Reasoning with assumption/commitment [MC81] or rely/guarantee [Jon83] pairs, however, easily leads to unsound reasoning. There is a danger of circular reasoning, two components which mutually discharge each others assumptions, leading to incorrect conclusions. Hence it is important to prove the soundness of the verification techniques. Correctness of compositional verification rules is usually based on a denotational semantics which assigns a meaning to compound constructs based on the meaning of its constituents. Earlier work on the verification of Splicesystems [HvdP02] was based on a complex semantics with environment actions and its correctness was not obvious. In this paper we define a denotational semantics for a simple Splice-like language which is more convenient as a basis for compositional reasoning using assumptions about the environment. It is, however, far from trivial that this semantics captures the intuitive understanding of the Splice architecture. Hence, we also provide a more intuitive operational semantics and prove formally that it is equivalent to the denotational one. Moreover, we show by a small example of transparent replication how the denotational framework can be used to support verification. Both versions of the semantics have been formulated in the language of the interactive theorem prover PVS [OSRSC01]. The equivalence result has been checked completely using PVS. Also the example application (transparent replication) has been verified in detail using PVS. Related to our semantic study is earlier work on the semantics of Splice-like languages such as a transition system semantics for a basic language of write and read statements (without query) [BKBdJ98] and a comparison of semantic choices using an operational semantics [BKBdJ98,BKZ99]. In previous work on a denotational semantics for Splice [BHdJ00] the semantics of local storages is not very convenient for compositional verification; it is based on process identifiers and a partial order of read and write events with complex global conditions. New in this paper is an operational semantics that deals with local time stamps and their use for updating local databases. We define an equivalent denotational semantics which includes assumptions about the environment of a
184
J. Hooman and J. van de Pol
application process
application process
...... read
read
write
write local database
local database
......
agent send
agent receive
send
receive
Fig. 1. Splice applications.
component. Moreover, we show that this denotational semantics forms the basis of a formal framework for specifying and verifying Splice applications. This paper is structured as follows. Section 2 contains a brief informal explanation of Splice. In Sect. 3 we present a formal syntax of the Splice primitives considered in this paper. The operational and denotational semantics of this language are defined in Sect. 4 and Sect. 5, respectively. The main outline of the equivalence proof can be found in Sect. 6. A framework for specification and verification is described in Sect. 7 and applied to an example with top-down design and transparent replication in Sect. 8. Finally, Sect. 9 contains a number of concluding remarks.
2
Informal Splice Introduction
The Splice architecture provides a coordination mechanism for concurrent components. Producers and consumers of data are decoupled. They need not know each other, and communicate indirectly via the Splice primitives; basically readand write- operations on a distributed dataspace. This type of anonymous communication between components is strongly related to coordination languages such as Linda [Gel85] and JavaSpaces [FHA99]. These languages, however, have a single shared dataspace, whereas in Splice each component has its own dataspace, see Fig. 1. Communication between components takes place by means of local agents. A data producer writes data records to the other dataspaces via its agent. A data consumer uses its agent to subscribe to the required types of data; only data which matches this subscription is stored. Data items may be delayed and re-ordered and sometimes may even get lost. It is possible to associate certain quality-of-service policies with data delivery and data storage. For instance, for a particular data type, delivery maybe guaranteed (each item is delivered at least once) or best effort (zero or more times). Data storage can be volatile, transient, or persistent.
Equivalent Semantic Models for a Distributed Dataspace Architecture
185
Each data item within Splice has a unique sort, specifying the fields the sort consists of and defining the key fields [BdJ97]. In each local dataspace, at most one data item is present for each key. Basically, a newly received data item overwrites the current item with the same key (if any). To avoid that old data items overwrite newer information (recall that data may be delayed and re-ordered), data records include a time stamp field. A time stamp of a data item is obtained from the local clock of the data producer when the item is published. At the local storage of the consumer, data items are only overwritten if their time stamp is smaller than that of a newly arrived item (with the same key). This overwriting technique reduces memory requirements and allows a decoupling of frequencies between producers and consumers. It also reduces the number of updates to be performed on the dataspace, as not all received records get stored. The timestamps improve the quality of the data stored, as no record can be overwritten by older data. To program components on top of Splice, a Splice API can be called within conventional programming languages such as C and Java. Splice provides, for instance, constructs for retrieving (reading) data from the local dataspace, and for publishing (writing) data. Read actions contain a query on the dataspace, selecting data items that satisfy certain criteria. Data has a life-cycle attribute, such as “read”, “unread”, “new”, and “update”, which may be used for additional filtering.
3
Syntax of a Simple Splice-Like Language
In this section we define the formal syntax of a very simple Splice-like language. We have embedded the basic Splice primitives in a minimal programming language to be able to high-light the essential features and to prove equivalences between various semantic definitions in a formal way. In Sect. 7.1 we show that it is easy to extend the language with, e.g. if-then-else and loop constructs. Lifecycle attributes are not included in the current version, because we detected some problems with the informal definition (as mentioned in Sect. 9). We consider only one sort. Let Data be some data domain, with a set KeyData of key data and a function key: Data → KeyData. Assume a given type LocalT ime to represent values of local clocks (in our PVS representation we choose the natural numbers). The type DataItems of time-stamped data items, consists of records with two fields: dat of type Data and ts of type LocalT ime. A record of type DataItems can be written as (#dat := ..., ts := ...#), following the PVS notation. Hence, for di ∈ DataItems, we have dat(di) ∈ Data and ts(di) ∈ LocalT ime. The function key is extended to DataItems by key(di) = key(dat(di)). Let SV ars be a set of variables ranging over sets of elements from DataItems. A data expression e yields an element from Data (dependent on the current value of the program variables). Henceforth we typically use the following variables ranging over the types mentioned above:
186
– – – – –
J. Hooman and J. van de Pol
d, d0 , d1 , . . . over Data lt, lt0 , lt1 , . . . over LocalT ime dati, dati0 , dati1 , . . . over DataItems diset, diset0 , diset1 , . . . over sets of DataItems x, x0 , x1 , . . . , y, y0 , y1 , . . . over SV ars
A query q ⊆ P(diset) specifies sets of (time-stamped) data items (it may also depend on program variables). A few examples: – q1 = {diset | for all dati ∈ diset: ts(dati) > 100} = 0} – q2 = {diset | for all dati ∈ diset: dat(dati) = ø and diset = x} – q2 = {diset | diset For simplicity, we do not give the syntax of data expressions and queries here. The syntax of our programming language is given in Table 1. Table 1. Syntax Programming Language. Sequential program S ::= Write(e) | Read(x, q) | S1 ; S2 Process
P ::= S | P1 P2
Informally, the statements of this language have the following meaning: – Write(e) publishes the data item with value e (in the current state) and the current time stamp (from the local clock). We model best effort delivery; a data item arrives 0 or more times at each process, where it might be used to update the local storage if there is no current value with the same key which has a larger or equal time stamp. – Read(x, q) assigns to x some set of data items from local storage that satisfies query q (if there are several sets satisfying q, the choice is non-deterministic). For instance, for query q1 above, we assign to x a set of data items from local storage such that each data item has time stamp greater than 100. If there are no such items in local storage, x becomes the empty set. Note that query q3 above does not allow the empty set; then the read statement blocks until the local storage contains a set of items satisfying the query. Hence a read statement may be blocking, depending on the query. – S1 ; S2 : sequential composition of sequential programs S1 and S2 . – P1 P2 : parallel composition of processes. A process is a sequential program or a parallel composition of processes. As a very simple example, consider a few producers and consumers of flight data. Let Data be a record with two fields: flightnr (a string, e.g. KL309) and pos a position in some form, here a natural number for simplicity. The flight number is the key, that is, key(dati) = flightnr(dati). Consider a producer of flight data
Equivalent Semantic Models for a Distributed Dataspace Architecture
187
P1 = Write((#flightnr := KL567, pos := 1#)) ; Write((#flightnr := LU 321, pos := 6#)) ; Write((#flightnr := KL567, pos := 2#)) ; Write((#flightnr := KL567, pos := 3#)) and two consumers: C1 = Read(x1 , true) ; Read(y1 , q1 ) ; Read(z1 , q1 ) C2 = Read(x2 , q1 ) ; Read(y2 , q2 ) whose queries are specified as follows: = ø and for all dati ∈ diset : flightnr(dati) = KL567} q1 = {diset | diset = ø and for all dati ∈ diset : flightnr(dati) = KL567 and q2 = {diset | diset for all dati1 ∈ x2 : ts(dati) > ts(dati1 )} Consider the process P1 C1 C2 and assume there are no other producers of data. Note that the producer does not specify the local time stamp explicitly; this is added implicitly. Recall that the items produced by P1 may arrive at a different order at the consumers, and they may arrive several times. Variable x1 may be empty (if no data item has been delivered yet – note that this read is not blocking) or a set with one or two elements, at most one for each flight number. For instance, it may contain position number 3 for KL567. Variable y1 will be a singleton, since the local storage contains at most one item with flight number KL567 and the second read is blocking (the query requires a non-empty set). If there is a position for KL567 in x1 , then the position in y1 will be greater or equal (lower values are produced earlier, hence have a smaller local clock value, and thus they cannot overwrite greater values). Similarly for z1 , where the position is greater or equal than the one in y1 . It is possible that z1 = y1 . For consumer C2 the second read action requires a newer time stamp, hence we always have y2 = x2 and the position in y2 is at least 2.
4
Operational Semantics
We define an operational semantics for a process S1 ... Sn of the syntax of Sect. 3. where the Si are sequential programs. We first define an operational status of a sequential process (Def. 1) and a configuration (Def. 3) which represents the state of affairs during operational execution of a process. Next computation steps are defined (Def. 5), leading to the operational semantics (Def. 6). Let DataBases be the type consisting of sets of data items with at most one item for each key, i.e. DataBases = {diset ⊆ DataItems | for all dati1 , dati2 ∈ diset: key(dati1 ) = key(dati2 ) → dati1 = dati2 } Definition 1 (Operational Status). An operational status of a sequential process, denoted os, os0 , os1 , .., is a record with three fields, st, clock and db: – st : SV ars → ℘(DataItems), represents the local state, assigning to each variable a set of data items; – clock ∈ LocalT ime, the value of the local clock;
188
J. Hooman and J. van de Pol
– db ∈ DataBases, the local database, a set of data items (at most one for each key) representing local storage. Definition 2 (Variant). The variant of local state st with respect to variable x ∈ SV ars and value diset ⊆ DataItems, denoted diset if y = x st[x := diset], is defined as (st[x := diset])(y) = st(y) if y =x Similarly, the variant of a record r with fields f1 . . . fm is defined by v if fi = f r[f := v](fi ) = =f fi (r) if fi Produced data items are sent to an underlying network. Here this is represented by N , a set of data items, i.e. N ⊆ DataItems. Note that we do not use a multiset, although a particular item might be produced several times. The use of a set is justified by the fact that data items may always be delivered more than once and hence are never deleted from N . Definition 3 (Configuration). The state of affairs of a process S1 ... Sn during execution is represented by a configuration: (S1 , os1 ), ..., (Sn , osn ), N It denotes for each sequential process Si , the status osi and the remaining part Si that still has to be executed, and the current contents N of the network. For convenience, we introduce the empty statement E indicating that the process has terminated. An execution of S1 ... Sn is represented by sequence of configurations C0 −→ C1 −→ C2 −→ ... where C0 = (S1 ; E, os1 ), ..., (Sn ; E, osn ), ø with, for all i, db(osi ) = ø. The idea is that each step in the sequence C0 −→ C1 −→ C2 −→ ... represents the execution of an atomic action by some process i. We define the update of a database. Definition 4 (Update Database). The update of database db using a new database db1 , denoted UpdateDb(db, db1 ) is defined by dati ∈ UpdateDb(db, db1 ) iff – either dati ∈ db and for all dati1 ∈ db1 with key(dati1 ) = key(dati) we have ts(dati1 ) ≤ ts(dati), – or dati ∈ db1 and for all dati0 ∈ db with key(dati0 ) = key(dati) we have ts(dati0 ) < ts(dati). Computation steps are defined formally as follows. Definition 5 (Computation Step). We have a step of process i, i.e., (S1 , os1 ), .., (Si , osi ), .., (Sn , osn ), N −→ (S1 , os1 ), .., (Si , osi ), .., (Sn , osn ), N iff one of the following clauses holds: = E and Si = Si , N = N , st(osi ) = st(osi ), clock(osi ) = (Update) Si clock(osi ), and there exists a database db1 ⊆ N that is used to update
Equivalent Semantic Models for a Distributed Dataspace Architecture
189
db(osi ) such that an element of db1 is added if its key is not yet present and, otherwise, it replaces an element of db(osi ) with the same key if its local time-stamp is strictly greater. Formally, using Def. 4, there exists a database db1 ⊆ N such that db(osi ) = UpdateDb(db(osi ), db1 ). For simplicity, we did not change the local clock (it is not needed), but alternatively we might require clock(osi ) ≥ clock(osi ). Note that the network has not been changed, since data items might be used several times for an update (modeling the fact that an item might be delivered by the network several times). (Write) Si = Write(e) ; Si , st(osi ) = st(osi ), db(osi ) = db(osi ), clock(osi ) > clock(osi ), and N = N ∪ {(v, clock(osi ))} where v is the value of e in the current state. Note that, given a syntax of expressions, it would be easy to define the values of expressions and queries in the current status (see, e.g. [dRdBH+ 01]). (Read) Si = Read(x, q); Si , N = N , db(osi ) = db(osi ), clock(osi ) = clock(osi ), and there exists a set of data items diset ⊆ db(osi ) satisfying query q and such that st(osi ) = st(osi )[x := diset], i.e. assigning diset to x. For two configurations C1 and C2 , define C1 −→k C2 , for k ∈ IN, by C1 −→0 C1 and C1 −→k+1 C2 iff there exists a configuration C such that C1 −→k C and C −→ C2 . Define C1 −→∗ C2 iff there exists a k ∈ IN such that C1 −→k C2 . Typically, the operational semantics yields some abstraction of the execution sequence, depending on what is observable. Here we postulate that only the set of produced data items in the last configuration of an execution sequence is (externally) observable. Definition 6 (Operational Semantics). The operational semantics of a program S1 ... Sn , given an initial operational status os0 , is defined by O(S1 ... Sn )(os0 ) = {N ⊆ DataItems | (S1 ; E, os0 ), .., (Sn ; E, os0 ), ø −→∗ (E, os1 ), .., (E, osn ), N , with db(os0 ) = ø } Thus the operational semantics of a program yields a set of sets of produced data items, where each set of produced data items represents a possible execution of the program. Example 1. Let 0 be the query specifying that the value 0 should be read, similarly for 1. Observe that, for any os0 , O((Read(x, 0) ; Write(1)) (Read(x, 1) ; Write(0)))(os0 ) = ø.
5
Denotational Semantics
We define the denotational semantics of a program, given an initial status, i.e. the state of affairs when execution starts. To support our aim to reason with assumptions about the items produced by the environment, such assumptions are
190
J. Hooman and J. van de Pol
included in the status. The semantics yields a set of statuses, each representing a possible execution of the program. To achieve compositionality and to describe a process in isolation, it is quite common that information has to be added to the status to express relations with the environment explicitly. Here we add, e.g. the set of written data items and the set of items that are assumed to be produced by the environment. Moreover, we need a way to represent causality between the written items. As shown below, this is needed to assign a correct meaning to the program (Read(x, 0); Write(1)) (Read(x, 1); Write(0)). Here this is achieved by the use of conceptual logical clocks that are added to the status of each process. They are updated similarly to Lamport’s logical clocks [Lam78], which ensures that there exists a global, total order on the produced items. Let T ime be the domain of logical clock values, here we use the natural numbers. We add a logical clock value to the data items produced. Type ExtDataItems, with typical variables edi, edi0 , edi1 , . . ., consists of records with two fields: – di of type DataItems, and – tm of type T ime. Field tm represents the logical moment of publication. It can be used to construct a global partial order on the produced data items that reflects causality. Field selector di is extended to remove the logical clock values from a set of extended data items. For a set ediset ⊆ ExtDataItems define di(ediset) = {di(edi) | edi ∈ ediset}. A status, typically denoted by s, s0 , s1 , .., representing the current state of affairs of a program, is a record with six fields. In addition to the three fields of the operational status: – st : SV ars → ℘(DataItems), the local state (values of variables); – clock ∈ LocalT ime, the value of the local clock; – db ∈ DataBases, the local database (a set of data items, unique per key); we have three new fields: – time ∈ T ime, the logical clock, to represent causality; – ownw ⊆ ExtDataItems, the set of extended data items written by the program itself in the past; – envw ⊆ ExtDataItems, the set of extended data items written by the environment of the program; this is an assumption about all items produced (including present and future). Below we define a meaning function M for programs by induction on their structure. The possible behaviour of a program prog, i.e. a set of statuses, is defined by M(prog)(s0 ), where s0 is the initial status at the start of program execution. Note that this includes an assumption about all data items that have been or will be produced by the environment. The semantics will be such that if s ∈ M(prog)(s0 ) then
Equivalent Semantic Models for a Distributed Dataspace Architecture
191
– ownw(s) equals the union of ownw(s0 ) and the items written by prog. – envw(s) = envw(s0 ); the field envw is only used by prog to update its local storage. Although all items are available initially, constraints on logical clocks prevent the use of items “too early”. Next we define M(prog) by induction on the structure of prog. Write. In the semantics of the write statement the published item, extended with the current value of the local clock, is added to the ownw field. The local clock is increased to ensure that subsequent written items get a later time stamp. In a similar way, a logical clock value has been added. M(Write(e))(s0 ) = {s | clock(s) > clock(s0 ), time(s) > time(s0 ), ownw(s) = ownw(s0 ) ∪ {(#di := dati, tm := time(s)#)}, where dati = (#dat := v, ts := clock(s0 )#), with v the value of e in s0 , and s equals s0 for the other fields (st, db and envw) } Read. To define the meaning of a read statement, we first introduce an auxiliary Update function which may update the local database with data items written (by the process itself or by its environment), using UpdateDb of Def. 4. Update(s0 ) = {s | there exists an ediset ⊆ ownw(s0 ) ∪ envw(s0 ) and a database db1 ⊆ di(ediset), such that db(s) = UpdateDb(db(s0 ), db1 ), time(s) >= time(s0 ), for all edi ∈ ediset, we have tm(edi) < time(s), and s equals s0 for the other fields (st, clock, ownw and envw) } Then the read statement Read(x, q) first updates the local storage and next assigns to x a set of data items that satisfies the query q. M(Read(x, q))(s0 ) = {s | exists s1 ∈ Update(s0 ) and diset ⊆ db(s1 ) such that diset satisfies query q and st(s) = st(s1 )[x := diset], and s equals s1 for the other fields (clock, db, time, ownw and envw) } Note that we only represent terminating executions; blocking has not been modeled explicitly. Sequential Composition. Since we only model terminating executions, the meaning of the sequential composition S1 ; S2 is defined by applying the meaning of S2 to any status that results from executing S1 . In Sect. 7.1 we show how this can be extended to deal with non-terminating programs. M(S1 ; S2 )(s0 ) = {s | exists s1 with s1 ∈ M(S1 )(s0 )∧ s ∈ M(S2 )(s1 )} Parallel Composition. To define parallel composition, let init(s0 ) be the condition db(s0 ) = ø ∧ ownw(s0 ) = ø. Moreover, we use s + ediset to add a set
192
J. Hooman and J. van de Pol
ediset ⊆ ExtDataItems to the environment writes of s, i.e. envw(s + ediset) = envw(s) ∪ ediset and all other fields of s remain the same. In the semantics of P1 P2 , starting in initial status s0 , the main observation is that envw(s0 ) contains only the data items produced outside P1 P2 . Hence the semantic function for P1 is applied to s0 where we add the items written by P2 to the environment writes. Similarly for P2 . Then parallel composition is defined as follows: M(P1 P2 )(s0 ) = {s | init(s0 ) and there exist s1 and s2 with s1 ∈ M(P1 )(s0 + ownw(s2 )), s2 ∈ M(P2 )(s0 + ownw(s1 )), ownw(s) = ownw(s1 ) ∪ ownw(s2 ), envw(s) = envw(s0 )} Example 2. Consider again the program of Example 1: (Read(x, 0) ; Write(1)) (Read(x, 1) ; Write(0)) Without using logical clocks, the semantics of parallel composition would allow for this program a status where envw = ø and ownw contains 0 and 1 (each component produces the item required by the other one). This, however, does not correspond to the operational semantics which yields the empty set. In the current version of the semantics, we can indeed show that, for any s0 , M((Read(x, 0) ; Write(1)) (Read(x, 1) ; Write(0)))(s0 ) = ø.
6
Equivalence of Denotational and Operational Semantics
In this section we first define what it means that the operational and the denotational semantics of Sect. 4 and 5, resp., are equivalent. Next we give an outline of how we proved this equivalence formally. Note that equivalence is far from trivial, since there a number of prominent differences: – The operational semantics allows updates of the local database at any point in time, whereas in the denotational semantics we minimized the number of updates to keep verification simple, allowing it only once, immediately before reading items. – The parallel composition of the denotational semantics is defined by a few recursive equations and it is not clear whether this indeed corresponds to operational execution. – The denotational semantics has additional fields in a status and the read statement contains an additional check on logical clock values. – The underlying network is modeled in different ways. Equivalence is based on what is externally observable, i.e. two semantics functions are equivalent if they assign the same observable behaviour to any program. For the denotational semantics, we choose the same notion of observable behaviour as has been used in the operational semantics, namely the set of published data items. For a set D of denotational statuses, define the observations
Equivalent Semantic Models for a Distributed Dataspace Architecture
193
by Obs(D) = {di(ownw(s)) | s ∈ D}. Define for an n-tuple (s1 , . . . , sn ) of statuses Obs(s1 , . . . , sn ) = di(∪i,1≤i≤n ownw(si )), and for a set T of such tuples Obs(T ) = {Obs(s1 , . . . , sn ) | (s1 , . . . , sn ) ∈ T }. To relate the operational and the denotational semantics, we use a function Ext to extend an operational status to a status of the denotational semantics; Ext(os) copies the fields st, clock, and db of os and it sets the fields time to 0 and ownw and envw to ø. This leads to the main theorem. Theorem 1. O(S1 ... Sn )(os) = Obs(M(S1 (S2 (... Sn )...))(Ext(os))) We give an outline of the proof that has been checked completely using the interactive theorem prover PVS. Below we only give the main steps, for instance, ignoring details about initial conditions. The proof uses three intermediate versions of the semantics and corresponding lemma’s: – M which is the same as M except that also the write-statement is preceded by an update action. Lemma 1. Obs(M(S1 (S2 (... Sn )...))(Ext(os))) = Obs(M (S1 (S2 (... Sn )...))(Ext(os))) Note that M and M are defined for the parallel composition of two processes. But we derive a similar formulation for n processes: Lemma 2. Obs(M (S1 (S2 (... Sn )...))(Ext(os))) = Obs({(s1 , . . . , sn ) | init(Ext(os)), envw(Ext(os)) = ø and for all i, 1 ≤ i ≤ n, si ∈ M (Si )(Ext(os)[envw := ∪j=i ownw(sj )]) }) – OD which extends the operational semantics O to the status of the denotational semantics (adding time, ownw and envw). Moreover, the network N is removed. This is achieved by defining the atomic steps of a single sequential process as (S, s) −→ediset (S , s ), where ediset represents the set of items written in the step (a singleton if S starts with a write statement, the empty set otherwise). This also includes update steps, similar to the update inside a read statement of the denotational semantics, so including a condition of the values of logical clocks. Based on this step relation for one process, we have a step of n parallel processes, denoted (S1 , s1 ), ..., (Sn , sn ) −→ (S1 , s1 ), ..., (Sn , sn ), if there = i, Sj = Sj and exists an i with (Si , si ) −→ediset (Si , si ), and for all j sj = sj + ediset. This leads to the following semantics for n processes: OD(S1 ... Sn )(s0 ) = {(s1 , . . . sn ) | init(s0 )∧ (S1 ; E, s0 ), ..., (Sn ; E, s0 ) −→∗ (E, s1 ), ..., (E, sn ) } Lemma 3. O(S1 ... Sn )(os) = Obs(OD(S1 ... Sn )(Ext(os)) – OS which extends the operational semantics of a sequential process to the status of the denotational semantics, using the relation (S, s) −→ediset (S , s ) mentioned above. Lemma 4. OS(S) = M (S), for sequential programs S.
194
J. Hooman and J. van de Pol
Now, by the lemmas 1, 2, 3, and 4, theorem 1 reduces to the following lemma. Lemma 5. Obs(OD(S1 ... Sn )(Ext(os)) = Obs({(s1 , . . . , sn ) | init(Ext(os)), envw(Ext(os)) = ø and for all i, 1 ≤ i ≤ n, si ∈ OS(Si )(Ext(os)[envw := ∪j=i ownw(sj )]) }) Proof. We give the main ideas of the proof, showing that the sets are contained in each other. ⊆ Suppose (s1 , . . . sn ) ∈ OD(S1 ... Sn )(Ext(os)), that is, (S1 ; E, Ext(os)), ..., (Sn ; E, Ext(os)) −→∗ (E, s1 ), ..., (E, sn ). We have proved by induction on the number of steps in the execution sequence that this implies, for all i, 1 ≤ i ≤ n, (Si ; E, Ext(os)[envw := ∪j=i ownw(sj )]) −→ediset1 . . . −→edisetk (E, si ), that is, si ∈ OS(Si )(Ext(os)[envw := ∪j=i ownw(sj )]). ⊇ Assume, for all i, 1 ≤ i ≤ n that si ∈ OS(Si )(Ext(os)[envw := ∪j=i ownw(sj )]). Thus we have an operation execution (Si ; E, Ext(os)[envw := ∪j=i ownw(sj )]) −→ediset1 . . . −→edisetk (E, si ) for each of the sequential processes. We have to show, (S1 ; E, Ext(os)), ..., (Sn ; E, Ext(os)) −→∗ (E, s1 ), ..., (E, sn ), i.e., we have to show that these sequential executions can be merged into a global execution sequence for the parallel program. Basically, this is done by induction on the total number of steps in all sequential executions. The global execution sequence is constructed by selecting for each step the process with the lowest logical time after its next possible step. This construction is far from trivial, since the local, sequential executions start with all available environment writes, whereas in the global execution a process can only use what has been produced up to the current moment. However, the constraints on the logical clocks (that have been included in this extended operational semantics), ensure that only items produced before its current logical time are used. Formally, this is captured by the property that if (S, s+ediset) −→ediset1 (S , s ) (representing a step of a local process) and for all edi ∈ ediset, tm(edi) ≥ time(s ) then (S, s) −→ediset1 (S , s [envw := envw(s)]) (i.e. it can be used in the global sequence). Since we execute the process with the earliest logical time, we can also show that all items produced later cannot have a smaller logical time stamp.
7
Verification Framework
In this section we provide a framework that can be used to specify and verify processes, as shown in Sect. 8. First, in Sect. 7.1, the programming language is extended with a number of useful constructs. Section 7.2 contains the main specification and verification constructs.
Equivalent Semantic Models for a Distributed Dataspace Architecture
195
7.1 Language Extensions To deal with some more interesting examples, the simple programming language of Sect. 3 is extended with an assignment, if-then-else construct, and infinite loops. Accordingly, the denotational semantics of Sect. 5 is extended. Since infinite loops introduce non-terminating computations, we add one field to the status: – term ∈ {true, f alse}: indicates termination of the process; if it is false all subsequent statements are ignored. Henceforth, we assume that s0 is such that term(s0 ) = true, i.e. after s0 we can still execute subsequent statements. The definition of sequential composition has to be adapted, since it is possible that the first process does not terminate and thus prohibits execution of the second process. M(S1 ; S2 )(s0 ) = {s | s ∈ M(S1 )(s0 ) ∧ ¬term(s)}∪ {s | there exists an s1 with s1 ∈ M(S1 )(s0 )∧ term(s1 ) ∧ s ∈ M(S2 )(s1 )} The meaning of the skip statement can be defined easily. M(Skip)(s0 ) = {s0 } Similarly we can easily define x := e where x ∈ Svars and e an expression yielding a set of data items. We also add variables that range over data items and define assignments for such variables. Next we define an if-then-else statement, where b is a boolean expression. M(If b Then S1 Else S2 Fi)(s0 ) = {s | if b is true in st(s0 ) then s ∈ M(S1 )(s0 ) else s ∈ M(S2 )(s0 ) } Finally, we define an infinite loop by means of an infinite sequence of statuses s0 , s1 , s2 , . . ., where si is the result of executing the loop body i times, provided all these executions terminate. Otherwise the term-field of si is false. The written items are collected by taking the union of the produced items in each execution of the body. Note that only the own and environment writes are relevant. M(Do S Od)(s0 ) = {s | there exists a sequence s0 , s1 , s2 , . . . such that for all i, if term(si ) then si+1 ∈ M(S)(si ) else term(si+1 ) = f alse, and ownw(s) = ∪{i|term(si )} ownw(si+1 ) and envw(s) = envw(s0 ) } 7.2
Specification and Verification
To obtain a convenient specification and verification framework, we define a mixed formalism in which one can freely mix programs and specifications, based on earlier work [Hoo94]. Specifications are part of the program syntax; let p, p0 , p1 , . . . , q, q0 , q1 , . . . be assertions, that is, predicates over statuses. A specification is a “program” of the form Spec(p, q) with the following meaning.
196
J. Hooman and J. van de Pol
M(Spec(p, q))(s0 ) = {s | (p(s0 ) implies q(s)) and envw(s) = envw(s0 ) } Next we define a refinement relation ⇒ between programs (which now may include specifications). Definition 7 (Refinement). For any two programs P1 , P2 , we define P1 ⇒ P2 iff for all s0 , we have M(P1 )(s0 ) ⊆ M(P2 )(s0 ). Note that it is easy to prove that the refinement relation is reflexive and transitive. We have the usual consequence rule. Lemma 6 (Consequence). Spec(p, q).
If p → p0 and q0 → q then Spec(p0 , q0 ) ⇒
Based on the denotational semantics for Splice, we checked in PVS the soundness of a number of proof rules for programming constructs. For instance, for sequential composition we have a composition rule and a monotonicity rule which allows refinements in a sequential context. Lemma 7 (Sequential Composition). (Spec(p, r); Spec(r, q)) ⇒ Spec(p, q). Lemma 8 (Monotonicity of Sequential Composition). If P3 ⇒ P1 and P4 ⇒ P2 then (P3 ; P4 ) ⇒ (P1 ; P2 ). The reasoning about parallel composition in PVS mainly uses the semantics directly. We only have a monotonicity rule, which forms the basis of stepwise refinement of components. Lemma 9 (Monotonicity of Parallel Composition). If P3 ⇒ P1 and P4 ⇒ P2 then (P3 P4 ) ⇒ (P1 P2 ).
8
Verification Example
To illustrate our reasoning framework, we first show top-down development of a typical application consisting of a producer, a transformer, and a consumer in Sect. 8.1. Next, in Sect. 8.2, we consider transparent replication. The general question is whether we can replace a single process by two, or more, copies of the same process without having to change the environment. So, given the monotonicity rules mentioned above, we look for sufficient conditions for having P P ⇒ P . We investigate this for the concrete case study developed in Sect. 8.1. 8.1
Top-Down Development
Here we consider a concrete and simple case study, which is prototypical for the application area, with the following processes: – Producer: provides monotonically increasing data (here simply natural numbers) with name SensData.
Equivalent Semantic Models for a Distributed Dataspace Architecture
197
– Transformer: assuming it gets increasing SensData items, it provides monotonically increasing data with name Intern. – Consumer: assuming the environment provides increasing data with name Intern, it produces monotonically increasing Display items. To formalize this, we define the following types and functions: – DataN ame = {SensData, Intern, Display}, with typical variable dn. – DataV al = IN. – Data is a type of records with two fields: name of type DataN ame and val of type DataV al. Let dvar be a variable of type Data. – KeyData = DataN ame and key(dvar) = name(dvar). To formulate the specifications, first a few preliminary definitions are needed, where wset is a set of extended data items: – N ameOwnw(dn)(s) = ∀edi ∈ ownw(s) : name(edi) = dn – SensData(wset) = {edi | edi ∈ wset ∧ name(edi) = SensData} Similarly, we have Intern(wset) and Display(wset). – Increasing(wset) = ∀edi1 ∈ wset, edi2 ∈ wset : (val(edi1) < val(edi2) ↔ ts(edi1) < ts(edi2)) Let pre be an assertion expressing that db = ø, ownw = ø, and all variables ranging over sets of data items are initialized to the empty set. It is used as a precondition for the processes. The top-level specification of the overall system is defined as follows. postT opLevel(s) = envw(s) = ø → Increasing(Display(ownw(s))) T opLevel = Spec(pre, postT opLevel) We implement this top-level specification by the parallel composition of the three processes mentioned above: Producer Transformer Consumer We first specify the processes, without considering their implementation, and show that these specifications lead to the top-level specification. The producer writes an increasing sequence of sensor data. postP rod(s) = N ameOwnw(SensData)(s)∧ Increasing(ownw(s)) P rod = Spec(pre, postP rod) The consumer should produce an increasing sequence of Display data, assuming the environment provides increasing internal values. postCons = N ameOwnw(Display)(s)∧ Increasing(Intern(envw(s))) → Increasing(Display(ownw(s))) Cons = Spec(pre, postCons) To connect producer and consumer, we specify the transformer as follows.
198
J. Hooman and J. van de Pol
postT rans = N ameOwnw(Intern)(s)∧ Increasing(SensData(envw(s))) → Increasing(Intern(ownw(s))) T rans = Spec(pre, postT rans) We have proved in PVS that this leads to a correct refinement of the top-level specification. Theorem 2. (P rod (T rans Cons)) ⇒ T opLevel Next we consider the implementation of the three components. Let d be a variable ranging over Data, whereas dset and dold are variables ranging over sets of data items. P roducer = d := (#name := SensData, val := 0#) ; Do Write(d) ; d := (#name := SensData, val := val(d) + 1#) Od Note that the producer writes all natural numbers, which is not required by the specification. Also note that subscribers to data with name SensData will usually read only a (increasing) subsequence of these items. We have proved in PVS that this is indeed a correct refinement. Lemma 10. P roducer ⇒ P rod Similarly we provide a program for the transformer and show that it is a correct implementation. Let q(dn, old) be the query that specifies a set of data items with name dn and local time stamp different from those in variable old (which is initially empty). The set is allowed to be empty, to avoid blocking computations; otherwise it will be a singleton with a new, unread, item. The item that has been read is transformed using a function T r(dn, dset) which, for simplicity, just changes the name of the record of the item in dset into dn. Transformer = Do Read(dset, q(SensData, old)) ; If dset =ø Then Write(T r(Intern, dset)) ; old := dset Else Skip Fi Od Lemma 11. Transformer ⇒ T rans Similarly for the consumer. For simplicity, we used the same name transformation, which makes it possible to derive the correctness of transformer and consumer from a single proof that was parameterized by the name of the data sort. Consumer = Do Read(dset, q(Intern, old)) ; If dset =ø Then Write(T r(Display, dset)) ; old := dset Else Skip Fi Od Lemma 12. Consumer ⇒ Cons Finally observe that top-level theorem 2 and the lemmas 10, 11, and 12 for the components lead by the monotonicity property (lemma 9) to (P roducer (Transformer Consumer)) ⇒ T opLevel
Equivalent Semantic Models for a Distributed Dataspace Architecture
8.2
199
Transparent Replication
While investigating transparent replication, we observed that the current version of the transformer specification is not suitable. The next lemma expresses that we cannot replicate the transformer in a transparent way. Lemma 13. ¬((T rans T rans) ⇒ T rans) The lemma has been proved by a concrete counter example in PVS. The basic idea is that the two transformers each have increasing output, but - because they may write this output at different moment - the local time stamps in these sequences might be different and hence merging these output streams need not lead to an increasing sequence. The main problem is that the items produced get their local time stamp from the local clock when the item is written. This need not be related to the temporal validity of the data. Hence we propose an alternative specification for the transformer where the local time stamp is just copied from the incoming data item (as we will see later, this also requires a modified write statement). M aintainT s(s) = ∀edi ∈ ownw(s) : ∃edi1 ∈ envw(s) : ts(edi) = ts(edi1 )∧ name(edi1 ) = SensData∧ val(edi) = val(edi1 ) postT ransN ew = N ameOwnw(Intern) ∧ M aintainT s T ransN ew = Spec(pre, postT ransN ew) This indeed refines the transformer specification: Lemma 14. T ransN ew ⇒ T rans Moreover, this specification can indeed be replicated. Lemma 15 (Transformer Replication). (T ransN ew T ransN ew) ⇒ T ransN ew Hence we have (by theorem 2, lemma 14 and monotonicity): (P rod (T ransN ew Cons)) ⇒ T opLevel and by lemma 15 and monotonicity: (P rod ((T ransN ew T ransN ew) Cons)) ⇒ T opLevel It remains to implement T ransN ew. Important part of this specification is that it just copies the local time stamp (the ts-field). This, however, cannot be implemented with the current write statement, since it always use the current value of the local clock as its time stamp (the ts field is set to clock(s0 )). Therefore we introduce a new write statement Write(e, texp) which has an additional parameter, a time expression texp, which specifies the ts value, i.e. in the semantics of this extended write statement the ts-field in the data item gets the value of texp. Query Qtrans specifies a set of data items that is either empty or a singleton containing an item with name SensData. Expression MkIntern(dset) changes the name of the item in dset to Intern and copies it value. Now the write statement has a time expression T s(dset) which just yields the ts-field of the data item in dset.
200
J. Hooman and J. van de Pol
NewTransformer = Do Read(dset, Qtrans) ; If dset =ø Then Write(MkIntern(dset), T s(dset)) Else Skip Fi Od Lemma 16. NewTransformer ⇒ T ransN ew So, finally, we have (P roducer(NewTransformerConsumer)) ⇒ T opLevel and (P roducer ((NewTransformer NewTransformer) Consumer)) ⇒ T opLevel.
9
Concluding Remarks
We have defined a new denotational semantics for the main primitives of the industrial software architecture Splice. This architecture is data-oriented and based on local storages of data items. Communication between components is anonymous, based on the publish-subscribe paradigm. New is especially the modeling of time stamps, based on local clocks, and the update mechanism of local storages based on these time stamps. Moreover, the denotational semantics supports convenient compositional verification, based on assumptions about the data items produced by the environment of a component. Causality between data items is represented by logical time stamps. To simplify verification, we minimized the number of updates of the local storages and tried a short, but non-trivial, formulation of parallel composition. To increase the confidence in this denotational semantics, we also formulated a rather straightforward operational semantics. Using the interactive theorem prover PVS, we formally showed that the operational semantics is equivalent to the denotational one. The proof revealed a number of errors in earlier versions of the semantics, e.g. concerning the precise interpretation of logical time stamps representing causality. As a side-effect, the proof resulted in a number of useful properties. Classical ones, such as associativity of sequential and parallel composition, but also more Splice-oriented properties such as idempotence of updates. We also observed that the current definition of the notion of a lifecycle in Splice breaks idempotence of updates. Since it indicates a conceptual error in the definition, we have removed the lifecycle from the current formalization. A topic of current work concerns the equivalence classes induced by the semantics, i.e. which programs obtain the same semantics? The question is whether the denotational semantics is fully abstract with respect to the operational one, that is, does it only distinguish those programs that are observably different in some context? We have shown how the semantics can be used as a basis for a formal framework for specification and compositional verification. The study of transparent replication clarified the use of local time stamps and led to an improvement of the write primitive. In future work we intend to investigate the use of clock synchronization.
Equivalent Semantic Models for a Distributed Dataspace Architecture
201
References BdJ97.
M. Boasson and E. de Jong. Software architecture for large embedded systems. IEEE Workshop on Middleware for Distributed Real-Time Systems and Services, 1997. BHdJ00. R. Bloo, J. Hooman, and E. de Jong. Semantical aspects of an architecture for distributed embedded systems. In Proc. of the 2000 ACM Symposium on Applied Computing (SAC 2000), volume 1, pages 149– 155. ACM press, 2000. BKBdJ98. M.M. Bonsangue, J.N. Kok, M. Boasson, and E. de Jong. A software architecture for distributed control systems and its transition system semantics. In Proc. of the 1998 ACM Symposium on Applied Computing (SAC ’98), pages 159 – 168. ACM press, 1998. BKZ99. M. M. Bonsangue, J.N. Kok, and G. Zavattaro. Comparing coordination models based on shared distributed replicated data. In Proc. of the 1999 ACM Symposium on Applied Computing (SAC’99). ACM Press, 1999. Boa93. M. Boasson. Control systems software. IEEE Transactions on Automatic Control, 38(7):1094–1106, July 1993. dRdBH+ 01. W.P. de Roever, F. de Boer, U. Hannemann, J. Hooman, Y. Lakhnech, M. Poel, and J. Zwiers. Concurrency Verification, Introduction to Compositional and Noncompositional Methods. Cambridge Tracts in Theoretical Computer Science. Cambridge University Press, 2001. FHA99. E. Freeman, S. Hupfer, and K. Arnold. JavaSpaces: Principles, Patterns, and Practice. Addison-Wesley, Reading, MA, USA, 1999. Gel85. D. Gelernter. Genarative communication in Linda. Transactions on Programming Languages and Systems, 7(1):80–112, 1985. HH02. U. Hannemann and J. Hooman. Formal reasoning about real-time components on a data-oriented architecture. In Proc. of 6th World Multiconference on Systemics, Cybernetics and Informatics (SCI02), volume XI, pages 313–318, 2002. Hoo94. J. Hooman. Correctness of real time systems by construction. In Formal Techniques in Real-Time and Fault-Tolerant Systems, pages 19–40. LNCS 863, Springer-Verlag, 1994. HvdP02. J. Hooman and J. van de Pol. Formal verification of replication on a distributed data space architecture. In Proc. of the 2002 ACM Symposium on Applied Computing (SAC 2002), pages 351–358, 2002. Jon83. C.B. Jones. Tentative steps towards a development method for interfering programs. ACM Transactions on Programming Languages and Systems, 5(4):596–619, 1983. Lam78. L. Lamport. Time, clocks, and the ordering of events in a distributed system. Communications of the ACM, 21(7):558–565, 1978. MC81. J. Misra and K.M. Chandy. Proofs of networks of processes. IEEE Transactions on Software Engineering, 7(7):417–426, 1981. OSRSC01. S. Owre, N. Shankar, J.M. Rushby, and D.W.J. Stringer-Calvert. PVS System Guide. SRI International, Computer Science Laboratory, Menlo Park, CA, version 2.4 edition, December 2001. http://pvs.csl.sri.com.
Java Program Verification Challenges Bart Jacobs, Joseph Kiniry, and Martijn Warnier Dep. Comp. Sci., Univ. Nijmegen P.O. Box 9010, 6500 GL Nijmegen, The Netherlands {bart,kiniry,warnier}@cs.kun.nl http://www.cs.kun.nl/∼{bart,kiniry,warnier}
Abstract. This paper aims to raise the level of verification challenges by presenting a collection of sequential Java programs with correctness annotations formulated in JML. The emphasis lies more on the underlying semantical issues than on verification.
1
Introduction
In the study of (sequential) program verification one usually encounters the same examples over and over again (e.g., stacks, lists, alternating bit protocol, sorting functions, etc.), often going back to classic texts like [11,14,15]. These examples typically use an abstract programming language with only a few constructs, and the logic for expressing the program properties (or specifications) is some variation on first order logic. While these abstract formalisms were useful at the time for explaining the main ideas in this field, they are not very helpful today when it comes to actual program verification for modern programming and specification languages. The aim of this paper is to give an updated collection of program verification examples. They are formulated in Java [13], and use the specification language JML [19,21]. The examples presented below are based on our experience with the LOOP tool [3] over the past five years. Although the examples have actually been verified, the particular verification technology based on the LOOP tool does not play an important rˆ ole: we shall focus on the underlying semantical issues. The examples may in principle also be verified via another approach, like those of Jack [6], Jive [23], Krakatoa [10], and KeY [2]. We shall indicate when static checking with the ESC/Java [12] tool brings up interesting semantical issues in our examples1 . When ESC/Java checks an annotated program, it returns one of three results. The result “passed” indicates that ESC/Java believes the implementation of a method fulfills its specification; in that case we’ll say ESC/Java accepts the input. A result of “warning” indicates that an error potentially exists in the program but ESC/Java was unable to prove its existence definitively (e.g., a counter-example or instruction trace). Finally, the message “error” indicates a specific bug in the program, typically a potential run-time violation like a NullPointerException or an ArrayIndexOutOfBoundsException. 1
We use the final binary release (v. 1.2.4, 27 September 2001) of ESC/Java.
F.S. de Boer et al. (Eds.): FMCO 2002, LNCS 2852, pp. 202–219, 2003. c Springer-Verlag Berlin Heidelberg 2003
Java Program Verification Challenges
203
Of course the examples below do not cover all possible topics. For instance, floating point computations and multi-threading2 are not covered. In general, our work has focused on Java for smart cards and several examples stem from that area. Other examples are taken from work of colleagues [4], from earlier publications, or from test sets that we have been using internally. They cover a reasonable part of the semantics of Java and JML. Most of the examples can be translated easily in other programming languages. The examples do not involve semantical controversies, as discussed for instance in [5]. These examples are not meant to be canonical or proscriptive; they simply give a flavor of the level of completeness (and thus, complexity) necessary to cover modern programming language semantics. The verification examples are organised in the following categories. – – – – – –
Control flow and side-effects Overflow and bitwise operations Static initialization Inheritance Non-termination Specification issues
The organization of the paper roughly follows this categorization. Each example is accompanied by a short explanation of why the example is interesting and the challenges involved. But first we briefly describe the specification language JML.
2
JML
The Java Modeling Language, JML [19], is a behavioral interface specification language designed to specify Java modules. It can be used for classes as a whole, via class invariants and constraints, and for the individual methods of a class, via method specifications consisting of pre-, post- and frame-conditions (assignable clauses). In particular, it is also possible within a method specification to describe if a particular exception may occur and which post-condition results in that case. JML annotations are to be understood as predicates, which should hold for the associated Java code. These annotations are included in the Java source files as special comments indicated by //@, or enclosed between /*@ and */. They are recognised by special tools like the JML run-time checker [9], the LOOP compiler, and the Krakatoa verification condition generator. Class invariants and constraints are described as follows. /∗@ /∗@ 2 @ invariant <predicate> @ constraint
For a meta-theory on multithreaded Java programs see [1] in this volume.
204
B. Jacobs, J. Kiniry, and M. Warnier
exceptional) of methods, provided it holds before. Thus, invariants are implicitly added to postconditions of methods and constructors, and to preconditions of normal (non-constructor) methods. A constraint is a relation between two states, expressing what should hold about the pre-state and post-state of all methods. In this paper we use no constraints, and only one invariant (in Subsection 3.6). However, in practice they contain important information. Next we give an example JML method specification of some method m(). 2
4
6
/∗@ behavior @ requires <precondition>; @ assignable
Such method specifications may be understood as an extension of correctness triples {P }m{Q} used in Hoare logic, because they allow both normal and exceptional termination. Moreover, the postconditions in JML are relations, because the pre-state, indicated by \old(--), may occur. We shall see many method specifications below. JML is intended to be usable by Java programmers. Its syntax is therefore very much like Java. However, it has a few additional keywords, such as ==> (for implication), \old (for evaluation in the pre-state), \result (for the return value of a method, if any), and \forall and \exists (for quantification). This paper will not pay much attention to the semantics of JML (interested readers should see [21] for such). Hopefully, most of the JML assertions are self-explanatory. However, there are three points that we would like to mention. – In principle, expressions within assertions (such as an array access a[i]) may throw exceptions. The JML approach, see [21], is to turn such exceptions into arbitrary values. Of course, one should try to avoid such exceptions by including appropriate requirements. For instance, for the expression a[i] one should add a != null && i >= 0 && i < a.length, in case this is not already clear from the context. This is what we shall always do. – JML uses the subtype semantics for inheritance, see [22]. This means that overriding methods in subclasses should still satisfy the specifications of the overridden ancestors in superclasses. This is a non-trivial restriction, but one which is essential in reasoning about methods in an object-oriented setting. However, it does not hold for all our examples (see for instance Subsection 3.8). In that case we simply write no specification at all for the relevant methods. – JML method specifications form proof obligations. But also, once proved, they can be used in correctness proofs of other methods. In that case one first has to establish the precondition and invariant of the method that is called, and subsequently one can use the postcondition in the remainder of
Java Program Verification Challenges
205
the verification (which will rely heavily on the called method’s assignable clause). An alternative approach is to reason not with the specification, but with the implementation of the method that is called3 . Basically, this means that the body of the called method gets substituted at the appropriate place. However, this may lead to duplication of verification work, and makes proofs more vulnerable to implementation changes. But if no specification is available (see the previous point), one may be forced to reason with the implementation. In the examples below we shall see illustrations of method calls which are used both by specification and by implementation. For readers unfamiliar with JML, this paper may hopefully serve as an introduction via examples. More advanced use of JML in specifying API-components may be found for instance in [24] 4 .
3
Verification Challenges
This section describes our Java+JML examples in several subsections. Our explanations focus on the (semantic) issues involved, and not so much on the actual code snippets. They should be relatively self-explanatory. 3.1
Aliasing and Field Access
Our first example, seen in Figure 1, might seem trivial to some readers. The return expression of the method Alias.m() references the value of the field i of the object c value via an aliased reference to itself in the field a. We present this example because it represents (in our view) the bare minimum necessary to model a language like Java. ESC/Java has no problem verifying this program. Either the implementation or specification of the constructor of class C can be used to verify method Alias.m(). 3.2
Side-Effects in Expressions
One of the most common abstractions in program verification is to omit sideeffects from expressions in the programming language. This is a serious restriction. Figure 2 contains a nice and simple example from [4] where such side-effects play a crucial rˆ ole, in combination with the logical operators. Recall that in Java there are two disjunctions (| and ||) and two conjunctions (& and &&). The double versions (|| and &&) are the so-called conditional operators: their second argument is only evaluated if the first one is false (for ||) or true (for &&). 3 4
This only works if one actually knows the run-time type of the object on which the method is called. See also on the web at www.cs.kun.nl/∼ erikpoll/publications/jc211 specs.html for a specification of the Java API for smart cards.
206
2
B. Jacobs, J. Kiniry, and M. Warnier
class C { C a; int i ;
4
6
8
10
}
/∗@ normal behavior @ requires true; @ assignable a , i ; @ ensures a == null && i == 1; @∗/ C() { a = null; i = 1; }
12
class Alias { /∗@ normal behavior @ requires true; 16 @ assignable \nothing; @ ensures \ result == 4; 18 @∗/ int m() { 20 C c = new C(); c.a = c; 22 c. i = 2; return c.i + c.a. i ; 24 } } 14
Fig. 1. Aliasing via Field References.
In case the field b in Figure 2 is true, method m() yields f() ∨ ¬f() = false and ¬f() ∧ f() = true, going against standard logical rules. The verification of the specification for method m() may use either the implementation or the specification of f(). 3.3
Breaking out of a Loop
While and for loops are typically used for going through an enumeration, for instance to find or modify an entry meeting a specific condition. Upon hitting this entry, the loop may be aborted via a break statement. This presents a challenge for the underlying control flow semantics. Figure 3 presents a simple example of a for loop that goes through an array of integers in order to change the sign of the first negative entry. The two lines of java code are annotated with the loop invariant, with JML-keyword maintaining stating what holds while going through the loop, and the loop variant, with JML-keyword decreasing. The loop variant is a mapping to the natural numbers which decreases with every loop cycle. It is used in verifications to show that the repetition terminates.
Java Program Verification Challenges
207
boolean b = true; boolean result1, result2; 2
/∗@ normal behavior @ requires true; @ assignable b; 6 @ ensures b == !\old(b) && \result == b; @∗/ 8 boolean f() { b = !b; return b; } 4
/∗@ normal behavior @ requires true; 12 @ assignable b, result1 , result2 ; @ ensures (\old(b) ==> !result1 && result2) && 14 @ (!\ old(b) ==> result1 && result2); @∗/ 16 void m() { result1 = f () || ! f (); result2 = ! f() && f (); } 10
Fig. 2. Side-effects and Conditional Logical Operators.
The result of ESC/Java on this program is not very interesting because of its limited handling of loops: they are executed (symbolically) only once by default. In general, this may indicate a basic problem with the invariant, but the coverage is far from complete5 . 3.4
Catching Exceptions
Typical of Java is its systematic use of exceptions, via its statements for throwing and catching. They require a suitable control flow semantics. Special care is needed for the ‘finally’ part of a try-catch-finally construction. Figure 4 contains a simple example (adapted from [17]) that combines many aspects. The subtle point is that the assignment m+=10 in the finally block will still be executed, despite the earlier return statements, but has no effect on the value that is returned. The reason is that this value is bound earlier. 3.5
Bitwise Operations
Our next example in Figure 5 is not of the sort one finds in textbooks on program verification. But it is a good example of the ugly code that verification tools have to deal with in practice, specifically in Java Card applets6 . It involves a “command” byte cmd which is split in two parts: the first three, and last five bits. Depending on these parts, a mode field is given an appropriate value. This 5
6
As an aside: ESC/Java has difficulty with this example due to limitations in its parser as quantified expressions cannot be used in ternary operations. If we rewrite the specification of negatefirst as a conjuction of disjoint implications, ESC/Java accepts the program. ESC/Java does not handle a bitwise operator like signed right shift (>>) correctly.
208
B. Jacobs, J. Kiniry, and M. Warnier
int [] ia ; 2
/∗@ normal behavior @ requires ia != null ; @ assignable ia [∗]; 6 @ ensures \ forall int i; 0 <= i && i < ia.length ==> @ (\old(ia [ i]) < 0 && 8 @ (// i is the first position with negative value @ \ forall int j; 0 <= j && j < i ==> \old(ia[j]) >= 0)) 10 @ ? ( ia [ i] == −\old(ia[i ])) @ : ( ia [ i] == \old(ia[ i ])); 12 @∗/ void negatefirst () { 14 /∗@ maintaining i >= 0 && i <= ia.length && @ (\ forall int j; 0 <= j && j < i ==> 16 @ (ia [ j] >= 0 && ia[j] == \old(ia[j ]))); @ decreasing ia .length − i ; 18 @∗/ for(int i = 0; i < ia .length ; i++) { 20 if ( ia [ i ] < 0) { ia [ i] = −ia[ i ]; break; } } } 4
Fig. 3. Breaking out of a Repetition. int m; 2
/∗@ normal behavior @ requires true; @ assignable m; 6 @ ensures \ result == ((d == 0) ? \old(m) : \old(m) / d) @ && m == \old(m) + 10; 8 @∗/ int returnfinally (int d) { 10 try { return m / d; } catch(Exception e) { return m / (d+1); } 12 finally { m += 10; } } 4
Fig. 4. Return within try-catch-finally.
happens in a nested switch. The specification is helpful because it tells in decimal notation what is going on. 3.6
Class Invariants and Callbacks
Class invariants are extremely useful in specification, because they often make explicit what programmers have in the back of their mind while writing their code. A typical example is: “integer i is always non-zero” (so that one can safely divide by i).
Java Program Verification Challenges
2
209
static final byte ACTION ONE = 1, ACTION TWO = 2, ACTION THREE = 3, ACTION FOUR = 4; private /∗@ spec public @∗/ byte mode;
4
/∗@ behavior @ requires true; @ assignable mode; 8 @ ensures (cmd == 0 && mode == ACTION ONE) || @ (cmd == 16 && mode == ACTION TWO) || 10 @ (cmd == 4 && mode == ACTION THREE) || @ (cmd == 20 && mode == ACTION FOUR); 12 @ signals (Exception) @ ((cmd & 0x07) != 0 || (cmd != 0 && cmd != 16)) 14 @ && @ ((cmd & 0x07) != 4 || (cmd != 4 && cmd != 20)); 16 @∗/ void selectmode(byte cmd) throws Exception { 18 byte cmd1 = (byte)(cmd & 0x07), cmd2 = (byte)(cmd >> 3); switch (cmd1) { 20 case 0x00: switch (cmd2) { case 0x00: mode = ACTION ONE; break; 22 case 0x02: mode = ACTION TWO; break; default: throw new Exception(); } 24 break; case 0x04: switch (cmd2) { 26 case 0x00: mode = ACTION THREE; break; case 0x02: mode = ACTION FOUR; break; 28 default: throw new Exception(); } break; 30 default: throw new Exception(); } // ... more code 32 } 6
Fig. 5. Typical Mode Selection Based on Command Byte.
The standard semantics for class invariants is: when an invariant holds in the pre-state of a (non-constructor) method, it must also hold in the post-state. Note that this post-state can result from either normal or exceptional termination. An invariant may thus be temporarily broken within a method body, as long as it is re-established at the end. A simple example is method decrementk in Figure 6. Things become more complicated when inside such a method body the class invariant is broken and another method is called. The current object this is then left in an inconsistent state. This is especially problematic if control returns at some later stage to the current object. This re-entrance or callback phenomenon is discussed for instance in [25, Sections 5.4 and 5.5]. The commonly adopted solution to this problem is to require that the invariant of this is established before a method call. Hence the proof obligation in a method call a.m() involves the invariants of both the caller (this) and the callee (a).
210
2
B. Jacobs, J. Kiniry, and M. Warnier
class A { private /∗@ spec public @∗/ int k, m; B b;
4
/∗@ invariant k + m == 0; @∗/ 6
/∗@ normal behavior @ requires true; @ assignable k , m; @ ensures k == \old(k) − 1 && m == \old(m) + 1; @∗/ void decrementk () { k−−; m++; }
8
10
12
14
16
18
20
}
/∗@ normal behavior @ requires b != null ; @ assignable k , m; @ ensures true; @∗/ void incrementk () { k++; b.go(this); m−−; }
class B { /∗@ normal behavior 24 @ requires arg != null ; @ assignable arg.k , arg.m; 26 @ ensures arg.k == \old(arg.k) − 1 && @ arg.m == \old(arg.m) + 1; 28 @∗/ void go(A arg) { arg.decrementk(); } 30 } 22
Fig. 6. Callback with Broken Invariant.
This semantics is incorporated in the translation performed by the LOOP tool. Therefore we can not prove the specification for the method incrementk in Figure 6. However, a proof using the implementations of method go and decrementk is possible, if we make the additional assumptions that the run-time type of the field b is actually B, and that the method incrementk is executed on an object of class A. These restrictions are needed because if, for instance, field b has a subclass of B as run-time type, a different implementation will have to be used if the method go is overridden in the subclass. ESC/Java warns about the potential for invariant violation during the callback. Another issue related to class invariant is whether or not they should be maintained by private methods. JML does require this, but allows a special category of so-called ‘helper’ methods which need not maintain invariants. We don’t discuss this matter further.
Java Program Verification Challenges
2
211
class C { static boolean result1, result2 , result3 , result4 ;
4
6
8
10
12
14
16
}
/∗@ normal behavior @ requires !\ is initialized (C) && @ !\ is initialized (C1) && @ !\ is initialized (C2); @ assignable \ static fields of (C), @ \ static fields of (C1), @ \ static fields of (C2); @ ensures result1 && !result2 && result3 && result4; @∗/ static void m(){ result1 = C1.b1; result2 = C2.b2; result3 = C1.d1; result4 = C2.d2; }
18
class C1 { static boolean b1 = C2.d2; static boolean d1 = true; 22 } 20
24
class C2 { static boolean d2 = true; static boolean b2 = C1.d1; 28 } 26
Fig. 7. Static Initialization.
3.7
Static Initialization
Figure 7 shows an example of static initialization in Java (due to Jan Bergstra). In Java a class is initialized at its first active use (see [13]). This means that class initialization in Java is lazy, so that the result of initialization depends on the order in which classes are initialized. The rather sick example in Figure 7 shows what happens when two classes, which are not yet initialized, have static fields referring to each other. In the specification we use a new keyword \static_fields_of in the assignable clause. It is syntactic sugar for all static fields of the class. The first assignment in the body of method m() triggers the initialization of class C1, which in turn triggers the initialization of class C2. The result of the whole initialization is, for instance, that static field C2.b2 gets value false assigned to it. This can be seen when one realizes that the boolean static fields from class C1 initially get the default value false. Subsequently, class C2 becomes initialized and its fields also get the default value false. Now the assignments
212
B. Jacobs, J. Kiniry, and M. Warnier
in class C2 are carried out: d2 is set to true and b2 is set to false. Note that d1 is still false at this stage. Finally the assignments to fields in class C1 take place, both resulting in value true. One can see that the order of initializations is important. When the first two assignments in the method body of m() are switched, class C2 will be initialized before class C1 resulting in all fields getting value true. ESC/Java cannot handle this example as it cannot reason about static initialization. It provides no warnings for potential run-time errors in static initializers or in initializers for static fields. 3.8
Overriding and Dynamic Method Invocation
The example in Figure 8 is usually attributed to Kim Bruce. It addresses an issue which is often thought of as confusing in programming with languages which support inheritance. The overriding of the method equal makes it hard to tell which implementation is called: the one in the subclass or in the superclass. When a method is overridden, the run-time type of an object decides which implementation is called. This phenomena is also called late-binding. In the example three different objects are created, and the question is which equal method will be used. Notice that the equal methods in Figure 8 have no specifications. According to the behavioural subtyping semantics used in JML, the equal method in the subclass ColorPoint should also satisfy the specification of the equal method in superclass Point. This makes it impossible to prove a precise specification of the equal method in class ColorPoint. Therefore we proved the specification of method m() by using the implementations of the equal methods. 3.9
Inheritance
The program in Figure 9 is from [16] and was originally suggested by Joachim van den Berg. On first inspection it looks like the method test() will loop forever. The method test() calls method m() from class C, which calls method m() from class Inheritance, since ‘this’ has runtime-type Inheritance. Due to the subtype semantics used in JML for inheritance, we cannot write specifications for both of the m() methods with which we can reason. Therefore we can only prove the specification of method test() by using the method implementations. 3.10
Non-termination
The example in Figure 10 (due to Cees-Bart Breunesse) shows a program that does not terminate. The specification asserts that the program does not terminate normally or with an exception. The JML keyword diverges followed by the predicate true indicates that the program fails to terminate. The reader can easily see that this
Java Program Verification Challenges class Point { 2
int equal(Point x) { return 1; }
4
}
6
class ColorPoint extends Point {
8
}
int equal(Point x) { return 2; }
10
int r1,r2,r3,r4,r5,r6,r7,r8,r9; 12
/∗@ normal behavior @ requires true; @ assignable r1 , r2 , r3 , r4 , r5 , r6 , r7 , r8 , r9; @ ensures r1 == 1 && r2 == 1 && r3 == 2 && @ r4 == 2 && r5 == 2 && r6 == 2 && @ r7 == 1 && r8 == 2 && r9 == 2; @∗/ void m() { Point p1 = new Point(); Point p2 = new ColorPoint(); ColorPoint cp = new ColorPoint(); r1 = p1.equal(p1);r2 = p1.equal(p2);r3 = p2.equal(p1); r4 = p2.equal(p2);r5 = cp.equal(p1);r6 = cp.equal(p2); r7 = p1.equal(cp);r8 = p2.equal(cp);r9 = cp.equal(cp); }
14
16
18
20
22
24
26
Fig. 8. Overriding and dynamic method invocation.
2
class C { void m() throws Exception { m(); } }
4
6
class Inheritance extends C { void m() throws Exception { throw new Exception(); }
8
10
12
14
}
/∗@ exceptional behavior @ requires true; @ assignable \nothing; @ signals (Exception) true; @∗/ void test () throws Exception { super.m(); }
Fig. 9. Overriding and Dynamic Types.
213
214
B. Jacobs, J. Kiniry, and M. Warnier
class Diverges{ 2
4
6
8
10
12
14
}
/∗@behavior @ requires true; @ assignable \nothing; @ ensures false ; @ signals (Exception e) false ; @ diverges true; @∗/ public void m(){ for (byte b = Byte.MIN VALUE;b <= Byte.MAX VALUE;b++) ; }
Fig. 10. A Program that does not Terminate.
program does not terminate. Since Byte.MAX VALUE + 1 = Byte.MIN VALUE the guard in the for loop will never fail. Note that in order to verify this program both overflowing and non-termination have to be modeled appropriately. ESC/Java does not handle non-termination. 3.11
Specification
The final example exemplifies two commonplace complications in reasoning about “real world” Java. Representations of integral types (e.g., int, short, byte, etc.) in Java are finite. A review of annotated programs that use integral types indicates that specifications are often written with infinite numeric models in mind [7]. Programmers seem to think about the issues of overflow (and underflow, in the case of floating point numbers) in program code, but not in specifications. Additionally, it is often the case that specifications use functional method invocations. Methods which have no side-effects in the program are called called “pure” methods in JML and “queries” in the Eiffel and UML communities7 . The example in Figure 11 highlights both complications, as it uses method invocations in a specification and integral values that can potentially overflow. The method isqrt(), which computes an integer square root of its input, is inspired by a specification (see Figure 12) of a similar function included with early JML releases [20]. Note that the iabs() method in Figure 11 is used in the specification of isqrt() to stipulate that both the negative and the positive square root are an acceptable return value, as all we care about is its magnitude. Actually, our implementation of isqrt() only computes the positive integer square root. 7
There is still debate in the community about the meaning of “pure”. Many Java methods, for example, which claim to have no side-effects, and thus should be pure, actually do modify the state due to caching, lazy evaluation, etc.
Java Program Verification Challenges
215
/∗@ normal behavior @ requires true; @ assignable \nothing; 4 @ ensures \ result == ((x >= 0 || @ x == Integer.MIN VALUE) ? x : −x); 6 @∗/ /∗@ pure @∗/ int iabs(int x) { 8 if (x < 0) return −x; else return x; } 2
/∗@ normal behavior @ requires x >= 0 && x <= 2147390966; 12 @ assignable \nothing; @ ensures iabs(\ result ) < 46340 && 14 @ \ result ∗ \ result <= x && @ x < (iabs(\ result ) + 1) ∗ ( iabs(\ result ) + 1); 16 @∗/ int isqrt (int x) { 18 int count = 0, sum = 1; /∗@ maintaining 0 <= count && 20 @ count < 46340 && @ count ∗ count <= x && 22 @ sum == (count + 1) ∗ (count + 1); @ decreasing x − count; 24 @∗/ while (sum <= x) { count++; sum += 2 ∗ count + 1; } 26 return count; } 10
Fig. 11. Dependent Specifications and Integral Types.
2
4
6
/∗@ normal behavior @ requires x >= 0; @ ensures Math.abs(\result) <= x && @ \ result ∗ \ result <= x && @ x < (Math.abs(\result) + 1) ∗ @ (Math.abs(\result) + 1); @∗/ Fig. 12. JML Specification of Integer Square Root from [20].
The original specification included in early JML releases is provided in Figure 12. This specification uses the Math.abs() method instead of our iabs() method. We use our own equivalent implementation of square root out of convenience, not necessity. The definition of JML states that expressions in the requires and ensures clauses are to be interpreted in the semantics of Java. Consequently, a valid im-
216
B. Jacobs, J. Kiniry, and M. Warnier
plementation of the specification in Figure 12 is permitted to return Integer.MIN VALUE when x is 0 (as already noted in [20]). This surprising situation exists because Java’s integral are bounded and because the definition of unary negation in the Java Language Specification is somewhat unexpected. Because integral types are bounded, expressions in the postcondition, specifically the multiplication operations, can overflow. Additionally, the two implementations of integer absolute value both return a value of Integer.MIN VALUE when passed Integer.MIN VALUE. While this is the documented behavior of java.lang.Math.abs(int), it is often overlooked by programmers because they presume that a function as mathematically uncomplicated as absolute value will produce unsurprising, mathematically correct results for all input. The absolute value of Integer.MIN VALUE is equal to itself because Math.abs() is implemented with Java’s unary negation operator “-”. This operator, defined in Section 15.15.4 of the Java Language Specification, silently overflows when applied to maximal negative integer or long [13] 8 . The precondition of isqrt() in Figure 11 is explained in Figure 13. We wish to ensure that no operation, either in the implementation of isqrt() or in its specification, overflows. The critical situation that causes an overflow is when we attempt to take an integer square root of a very large number. In particular, if we attempt to evaluate the postcondition of isqrt() for values of x larger than 2,147,390,966 an overflow takes place. The small but critical interval between the precondition’s bound 2,147,390,966 and Integer.MAX VALUE = 2,147,483,647 is indicated by the dark interval on the right of Figure 13: to check the postcondition, the prospective root (the arrow labeled 1) must be determined, one is added to its value (arrow 2), and the result is squared (arrow 3). This final result will thus overflow. Indeed, (46, 340 + 1)2 > 2, 147, 483, 647. 2.
3.
Integer.MAX_VALUE
0 1.
Fig. 13. The Positive Integers.
The erroneous nature of a specification involving potential overflows should become clear when one verifies the method using an appropriate bit-level representation of integral types [18]. Unfortunately, such errors are not at all apparent, even when performing extensive unit testing, because the boundary conditions for arithmetic expressions, like the third term of the postcondition of isqrt() in Figure 11, are rarely automatically derivable, and full state-space coverage is simply too computationally expensive. 8
Interestingly, the documentation for java.lang.Math.abs() did not reflect this fact until the 1.1 release of Java.
Java Program Verification Challenges
217
Specifications involving integral types, and thus potential overflows, are frequently seen in application domains that involve numeric computation both complex (e.g., scientific computation, computer graphics, embedded devices, etc.) and relatively simple (e.g., currency and banking). The former category are obviously challenging due to the complexity of the related data-structures, algorithms, and their specifications, and the latter are problematic because it is there that implementation violations have egregious (financial) consequences. This specification raises the question: What is the appropriate model for arithmetic specifications9 ? Another challenge highlighted by this example might be called “formalism completeness”. Many semantic formalisms of modern programming languages like Java do not attempt to specify the semantics of complicated features like method invocation. Even fewer attempt to incorporate such semantics in method specifications, as used here. For example, ESC/Java is unable to deal with this example because of such interdependencies. This is a significant limitation as many, if not most, method specifications rely upon pure or JML helper methods. This is also a weakness of the LOOP tool, as dealing with (pure) method calls in specifications is tedious in verifications10 . In general, the semantics of method invocation in specifications is still an unclear issue at this time.
4
Conclusions
The starting point of this paper is the observation that the classical examples in (sequential) program verification are no longer very relevant in today’s context. They need to be updated in two dimensions: language complexity and size. This paper focuses on complexity, by presenting a new series of challenges, written in Java, with correctness assertions expressed in JML. The examples incorporate some of the ugly details that one encounters in real-world programs, and that any reasonable semantics should be able to handle. The fact that these example programs are small does not mean that we think size is unimportant. On the contrary, once a reasonably broad semantic spectrum is covered, the next challenge is to scale up one’s program verification techniques to larger programs. With our tools we are currently verifying programs with hundreds of lines of code.
Acknowledgments We want to thank Patrice Chalin for insightful comments on an earlier version of this paper. 9 10
We do not have answers for these questions, though investigations are underway [8]. The verification of the method isqrt() from Figure 11 uses the implementation of the absolute value method iabs().
218
B. Jacobs, J. Kiniry, and M. Warnier
References ´ 1. Erika Abrah´ am Mumm, Frank S. de Boer, Willem-Paul de Roever, and Martin Steffen. A Tool-supported Proof System for Multithreaded Java. In FMCO 2002 proceedings, Lect. Notes Comp. Sci. Springer, Berlin, 2003. 2. W. Ahrendt, Th. Baar, B. Beckert, R. Bubel, M. Giese, R. H¨ ahnle, W. Menzel, W. Mostowski, A. Roth, S. Schlager, and P.H. Schmitt. The KeY tool. Technical Report 2003-5, Chalmers and G¨ oteborg University, 2003. http://i12www.ira.uka.de/˜key. 3. J. van den Berg and B. Jacobs. The LOOP compiler for Java and JML. In T. Margaria and W. Yi, editors, Tools and Algorithms for the Construction and Analysis of Systems, number 2031 in Lect. Notes Comp. Sci., pages 299–312. Springer, Berlin, 2001. 4. J. Bergstra and M. Loots. Empirical semantics for object-oriented programs. Artificial Intelligence Preprint Series nr. 007, Dep. Philosophy, Utrecht Univ. http://preprints.phil.uu.nl/aips/, 1999. 5. E. B¨ orger and W. Schulte. Initialization problems in Java. Software—Concepts and Tools, 20(4), 1999. 6. Lilian Burdy, Jean-Louis Lanet, and Antoine Requet. JACK (Java Applet Correctness Kit), 2002. www.gemplus.com/smart/r_d/trends/jack.html. 7. P. Chalin. Back to basics: Language support and semantics of basic infinite integer types in JML and Larch. Technical Report 2002.003.1, Computer Science Department, Concordia University, 2002. www.cs.concordia.ca/˜faculty/chalin/. 8. P. Chalin. Improving JML: For a safer and more effective language. Technical Report 2003-001.1, Computer Science Department, Concordia University, March 2003. 9. Y. Cheon and G.T. Leavens. A runtime assertion checker for the Java Modeling Language (JML). In H.R. Arabnia and Y. Mun, editors, International Conference on Software Engineering Research and Practice (SERP ’02), pages 322–328. CSREA Press, Las Vegas, 2002. 10. E. Contejean, J. Duprat, J.-C. Filliˆ atre, C. March´e, C. Paulin-Mohring, and X. Urbain. The Krakatoa tool for JML/Java program certification. Available via the Krakatoa home page at www.lri.fr/˜marche/krakatoa/, October 2002. 11. E.W. Dijkstra. A Discipline of Programming. Prentice Hall, 1976. 12. C. Flanagan, K.R.M. Leino, M. Lillibridge, G. Nelson, J.B. Saxe, and R. Stata. Extended static checking for Java. In Proceedings of the 2002 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), volume 37(5) of SIGPLAN Notices, pages 234–245. ACM, 2002. 13. J. Gosling, B. Joy, G. Steele, and G. Bracha. The Java Language Specification Second Edition. The Java Series. Addison-Wesley, 2000. http://java.sun.com/docs/books/jls/second_edition/html/ j.title.doc.html. 14. D. Gries. The Science of Programming. Springer, 1981. 15. C.A.R. Hoare. An axiomatic basis for computer programming. Commun. ACM, 12:576–580, 583, 1969. 16. M. Huisman and B. Jacobs. Inheritance in higher order logic: Modeling and reasoning. In M. Aagaard and J. Harrison, editors, Theorem Proving in Higher Order Logics, number 1869 in Lect. Notes Comp. Sci., pages 301–319. Springer, Berlin, 2000.
Java Program Verification Challenges
219
17. B. Jacobs. A formalisation of Java’s exception mechanism. In D. Sands, editor, Programming Languages and Systems (ESOP), number 2028 in Lect. Notes Comp. Sci., pages 284–301. Springer, Berlin, 2001. 18. B. Jacobs. Java’s integral types in PVS. Manuscript, http://www.cs.kun.nl/˜bart/PAPERS/, 2003. 19. G.T. Leavens, A.L. Baker, and C. Ruby. JML: A notation for detailed design. In H. Kilov and B. Rumpe, editors, Behavioral Specifications of Business and Systems, pages 175–188. Kluwer, 1999. 20. G.T. Leavens, A.L. Baker, and C. Ruby. Preliminary design of JML: A behavioral interface specification language for Java. Techn. Rep. 98-06, Dep. of Comp. Sci., Iowa State Univ. http://www.cs.iastate.edu/˜leavens/JML.html, 1999. 21. G.T. Leavens, E. Poll, C. Clifton, Y. Cheon, and C. Ruby. JML reference manual (draft). www.jmlspecs.org, 2002. 22. B. Liskov and J.M. Wing. A behavioral notion of subtyping. ACM Trans. on Prog. Lang. & Syst., 7(16(6)):1811–1841, 1994. 23. J. Meyer and A. Poetzsch-Heffter. An architecture for interactive program provers. In S. Graf and M. Schwartzbach, editors, Tools and Algorithms for the Construction and Analysis of Systems, number 1785 in Lect. Notes Comp. Sci., pages 63–77. Springer, Berlin, 2000. 24. E. Poll, J. van den Berg, and B. Jacobs. Specification of the JavaCard API in JML. In J. Domingo-Ferrer, D. Chan, and A. Watson, editors, Smart Card Research and Advanced Application, pages 135–154. Kluwer Acad. Publ., 2000. 25. C. Szyperski. Component Software. Addison-Wesley, 1998.
ToolBus: The Next Generation Hayco de Jong1 and Paul Klint1,2 1
2
Centrum voor Wiskunde en Informatica Kruislaan 413, P.O.Box 94079 1090 GB Amsterdam, The Netherlands Informatics Institute, University of Amsterdam, The Netherlands {Hayco.de.Jong,Paul.Klint}@cwi.nl
Abstract. The ToolBus is a software coordination architecture for the integration of components written in different languages running on different computers. It has been used since 1994 in a variety of projects, most notably in the complete renovation of the Asf+Sdf Meta-Environment. In this paper we summarize the experience that has been gained in these projects and sketch preliminary ideas how the ToolBus can be further improved. Topics to be discussed include the structuring of message exchanges, crash recovery in distributed applications, and call-by-value versus call-by-reference.
1
Generic Language Technology
Our primary interest is generic language technology that aims at the rapid construction of tools for a wide variety of programming and application languages. Its central notion is a language definition of some programming or application language. The common methodology is that a language is identified in a given domain, that relevant aspects of that language are formally defined and that desired tools are generated on the basis of this language definition. This generative approach is illustrated in Fig. 1. Using a definition for some language L as starting point, a generator can produce a range of tools for editing, manipulating, checking or executing L programs. Language aspects have to be defined, analyzed, and used to generate appropriate tooling such as compilers, interpreters, type checkers, syntax-directed editors, debuggers, partial evaluators, test case generators, documentation generators, and more. Language definitions are used, on a daily basis, in application areas as disparate as Cobol renovation, Java refactoring, smart card verification and in application generation for domains including finance, industrial automation and software engineering. In the case of Cobol renovation, the language in question is Cobol and those aspects that are relevant for renovation have to be formalized. In the case of application generation, the language in question is probably new and has to be designed from scratch. 1.1
One Realization: The Asf+Sdf Meta-environment
The Asf+Sdf Meta-Environment [1,2] is an incarnation of the approach just described and covers both the interactive development of language definitions and the generation of tools based on these language definitions. F.S. de Boer et al. (Eds.): FMCO 2002, LNCS 2852, pp. 220–241, 2003. c Springer-Verlag Berlin Heidelberg 2003
ToolBus: The Next Generation
221
Fig. 1. From language definition to generated programming environment.
In this paper we are primarily interested in the software engineering aspects of building such a system. Starting point is the Asf+Sdf Meta-Environment as we had completed it in the beginning of the 1990’s. This was a monolithic 200 KLOC Lisp program that was hard to maintain. It had all the traits of a legacy system and was the primary motivation to enter the area of system and software renovation. 1.2 Towards a Component Based Architecture We give a brief time line of the efforts to transform the old, monolithic, implementation of the Meta-Environment into a well-structured, component-based, implementation. In 1992, first, unsuccessful, experiments were carried out to decompose the system into separate parts [3]. The idea was to separate the user-interface and the text editor from the rest of the system. The user-interface was completely re-implemented as a separate component and as text editor we re-used Emacs. In hindsight, we were unaware of the fact that we made the transition from a completely sequential system to a system with several concurrent components. Unavoidably, we encountered hard to explain deadlocks and race conditions. In 1993, a next step was to write a formal specification of the desired system behavior [4] using PSF, a specification language based on process algebra and algebraic specifications [5]. Simulation of this specification unveiled other, not yet observed, deadlocks. Although this was clearly an improvement over the existing situation, this specification approach also had its limitations and drawbacks: – The specification lacked generality. It would, for instance, have been a major change to add the description of a new component. – The effort to write the PSF specification was significant and there was no way to derive an actual implementation from it. In 1994, the first version of the ToolBus was completed [6,7]. The key idea was to organize a system along the lines of a software bus and to make this bus programmable by way of a scripting language (Tscript) that was based on ACP (Algebra of Communicating Processes, [8]). Another idea was to use a uniform data format (called
222
H. de Jong and P. Klint
ToolBus terms) to exchange data between ToolBus and tools. At the implementation level, Tscripts were executed by an interpreter and communication between tools and ToolBus took place using TCP/IP sockets. In this way, multi-language, distributed, applications could be built with significantly less effort than using plain C and sockets. Based on various experiments [9,10,11,12], in 1995 a new version of the ToolBus was designed and implemented: the Discrete Time ToolBus [13,14,15]. Its main innovations were primitives for expressing timing considerations (delay, timeout) and for operating on a limited set of built-in data-types (booleans, integers, reals, lists). The Discrete Time ToolBus has been used for the restructuring of the Asf+Sdf MetaEnvironment [16]. A first version was released in 2001 [2]. In the meantime, the exchange format has also evolved from the ToolBus terms mentioned above to ATerms [17]: a term format that supports maximal subterm sharing and a very concise, sharing preserving, binary exchange format. ATerms decrease memory usage thanks to sharing and they permit a very fast equality test since structural equality can be replaced by pointer equality thanks to the maximal subterm sharing. Another line of development is the ToolBus Integrated Debugging Environment (Tide) described in [18]. Today, beginning 2003, it turns out that the original software engineering goals that triggered the development of the ToolBus have been achieved and that the MetaEnvironment can now be even further stretched than anticipated [19]. Therefore, it is time for some reflection. What have we learned from this major renovation project and what are the implications for the ToolBus design and implementation? 1.3
Plan of this Paper
In Sect. 2 we discuss component coordination, representation and computation and introduce the ToolBus: our component coordination architecture. Following, in Sect. 3, we demonstrate some of the ToolBus-features by means of an example. In Sect. 4 we show how we used the ToolBus in the Asf+Sdf Meta Environment to migrate from a monolithic to a distributed architecture. Then, in Sect. 5 we elaborate on the various issues that we would like to tackle in a next generation of the ToolBus. We conclude the paper with an overview of the current status of our current implementation of this next generation ToolBus (Sect. 6) and some concluding remarks (Sect. 7).
2 The TOOLBUS Architecture In [20] it was advocated that the overall architecture of a software system can be improved by separating coordination from computation. In addition to this, we also distinguish representation and use the following definitions: – Coordination: the way in which program and system parts interact (using procedure calls, remote method invocation, middleware, and others). – Representation: language and machine neutral data exchanged between components. – Computation: program code that carries out a specialized task.
ToolBus: The Next Generation
223
Fig. 2. Separating coordination from computation.
Fig. 3. The ToolBus architecture.
The assumption is now that a rigorous separation of coordination, representation and computation leads to flexible and reusable systems. This subdivision is sketched in Fig. 2. Our ToolBus approach follows this paradigm and is illustrated in Fig. 3. The goal of the ToolBus is to integrate tools written in different languages running on different machines. This is achieved by means of a programmable software bus. The ToolBus coordinates the cooperation of a number of tools. This cooperation is described by a Tscript that runs inside the ToolBus. The result is a set of concurrent processes inside the ToolBus that can communicate with each other and with the tools. Tools can be written in any language and can run on different machines. They exchange data by way of ATerms. A typical cooperation scenario is illustrated in Fig. 4. A user-interface (UI) and a database (DB) are combined in an application. Pushing a button in the user-interface leads to a database action and the result is displayed in the user-interface. In a traditional approach, the database action is directly connected to the user-interface button by means of a call-back function. This implies that the user-interface needs some knowledge about
224
H. de Jong and P. Klint
Fig. 4. A typical cooperation scenario.
the database tool and vice versa. In the ToolBus approach the two components are completely decoupled: pushing the button only leads to an event that is handled by some process in the ToolBus. This process routes the event to the database tool (likely via some intermediary process) and gets the answer back via the inverse route. This implies that the configuration knowledge is now completely localized in the Tscript and that UI and DB do not even know about each others existence. The primitives that can be used in Tscripts are listed in Table 1.
3 An Example: The Address Book Service To make the scenario from Fig. 4 more concrete, we describe the construction of an address book holding (name, address) pairs. Typical uses include creating a new address, finding an address based on the name, etc. First we consider some aspects of the User Interface. An instance of the UI connects to the ToolBus and during the subsequent session, the user can: create a new entry in the address book database; delete an existing entry from the database; search for an entry in the database; update an existing entry in the database. Each of these use cases can be described as a ToolBus process which, together with a process that explains how these use cases interact, form the ToolBus script describing our Address Book Service. 3.1 TOOLBUS Processes for the Address Book Service The ADDRESSBOOK process tells the ToolBus that an instance of our address-book tool is to be executed, followed by a loop which invokes one of the processes CREATE,
ToolBus: The Next Generation
225
Table 1. Overview of ToolBus primitives. Primitive delta + . * create snd-msg rec-msg snd-note rec-note no-note subscribe unsubscribe snd-eval rec-value snd-do rec-event snd-ack-event if ... then ... fi if ... then ... else ... fi || let ... in ... endlet := delay abs-delay timeout abs-timeout rec-connect rec-disconnect execute snd-terminate shutdown attach-monitor detach-monitor
Description inaction (“deadlock”) choice between two alternatives (P1 or P2 ) sequential composition (P1 followed by P2 ) iteration (zero or more times P1 followed by P2 ) process creation send a message (binary, synchronous) receive a message (binary, synchronous) send a note (broadcast, asynchronous) receive a note (asynchronous) no notes available for process subscribe to notes unsubscribe from notes send evaluation request to tool receive a value from a tool send request to tool (no return value) receive event from tool acknowledge a previous event from a tool guarded command conditional expressions communication-free merge (parallel composition) local variables assignment relative time delay absolute time delay relative timeout absolute timeout receive a connection request from a tool receive a disconnection request from a tool execute a tool terminate the execution of a tool terminate ToolBus attach a monitoring tool to a process detach a monitoring tool from a process
DELETE, SEARCH or UPDATE in each iteration. This construction, using the + operator ensures that at this level, the sub-processes can be regarded atomically. This means that for example no DELETE will happen during an UPDATE. process ADDRESSBOOK is let AB : address-book in execute(address-book, AB?) . ( CREATE(AB) + DELETE(AB) + SEARCH(AB) + UPDATE(AB) ) * delta endlet
226
H. de Jong and P. Klint
The operating system level details of starting the tool are defined in a separate section (one for each tool if multiple tools are involved): tool address-book is { command = "java-adapter -class AddressBookService" }
In this case, the ToolBus is told that our tool is written in Java, and that the main class to be started is called AddressBookService. The CREATE process can be described as a ToolBus process as follows: process CREATE(AB : address-book) is let AID : int in rec-msg(create-address) . snd-eval(AB, create-entry) . rec-value(AB, new-entry(AID?)) . snd-msg(address-created(AID)) endlet
The request to create a new address book entry is received and delegated to the tool, so it can update its state. In this case, our tool yields a unique id for reference to the new entry, which is returned as the result of the creation message. Note that communication between processes involves the matching of the arguments of snd-msg and rec-msg. The same holds for the communication between a process and a tool using snd-eval and rec-value. In all these cases, a result variable of the form V? gets a value assigned as the result of a successful match. The DELETE process differs only from the CREATE process in that it does not need a return value: ... rec-msg(delete-address(AID?) . snd-do(AB, delete-entry(AID)) . snd-msg(address-deleted(AID)) ...
The SEARCH process in our example implements but a single query: finding an address book entry by name. It shows how different results from a tool-evaluation request can be processed in much the same way that different messages are handled. Upon receiving a find-by-name message from another process, this request is delegated to the tool. Depending on whether or not the entry exists in the database, the tool replies with a found or a not-found message, respectively. This result is then propagated to the process that sent the initial find-by-name message.
ToolBus: The Next Generation
227
process SEARCH(AB : address-book) is let Aid : int, Name : str in rec-msg(find-by-name(Name?)) . snd-eval(AB, find-by-name(Name)) . ( rec-value(AB, found(Aid?)) . snd-msg(found(Aid)) + rec-value(AB, not-found) . snd-msg(not-found) ) endlet
The UPDATE process is more interesting. It shows that each update of an address entry is guarded. A process wanting to update an entry first has to announce this fact by sending an update-entry message, before it can do one or more updates to the entry. It then finishes the update by sending an update-entry-done message. Because matching snd-msg and rec-msg messages are connected synchronously, it is not possible for one update transaction to interfere with another. After a sender and receiver of the update-entry message are connected, all other processes that want to send a update-entry message have to wait until the receiving process is ready to receive an update-entry message again. This message pair thus acts as a very primitive locking scheme. More elaborate schemes are very well possible, but are not discussed in this paper. Summarizing, the UPDATE process shows that outside the implementation of the address book service, we can enforce the order in which certain parts of the service are invoked, as well as mutual exclusion of some of its sections. process UPDATE(AB : address-book) is let AID : int, Name : str, Address : str in rec-msg(update-entry(AID?)) . ( rec-msg(set-name(Name?)) . snd-do(AB, set-name(AID, Name)) + rec-msg(set-address(Address?)) . snd-do(AB, set-address(AID, Address)) ) * rec-msg(update-entry-done(AID)) endlet
3.2 TOOLBUS Process for the User Interface Because users can connect at any time to the ToolBus to start a session with the Address Book Service, the ToolBus itself does not execute instances of the UI (as it did with the
228
H. de Jong and P. Klint
address book tool). Instead UITool instances can connect, make zero or more requests to the service, and disconnect at their convenience. A ui tool is declared to exist, but no operating system level details are provided. The following definition of the UI process shows how UI requests for the creation of a new entry and a name-change can be realized: tool ui is { /* the ToolBus does not execute ui-instances */ } process UI is let UITool : ui, AID : int, Name : str in rec-connect(UITool?) . ( rec-event(UITool, create-address) . snd-msg(create-address) . rec-msg(address-created) . snd-ack-event(UITool, create-address) + rec-event(UITool, update-name(AID?, Name?)) . snd-msg(update-entry(AID)) . snd-msg(set-name(Name)) . snd-msg(update-entry-done(AID)) . snd-ack-event(UITool, update-name(AID, Name)) + ... /* more UI requests */ ) * rec-disconnect(UITool) endlet
4 Application to the ASF+SDF Meta-environment As explained in Sect. 1.2, the ToolBus has been used to restructure the Asf+Sdf MetaEnvironment. It consists of a cooperation of 27 tools ranging from a user-interface, graph browser, various editors, compiler and interpreter, to a parser generator and a repository for parse trees. A simplified view is shown in Fig. 5. Our insight can be further increased by considering some statistics. Table 2 shows the relative sizes of the various implementation languages used in the Meta-Environment. In the column language the various languages are listed. In column KLOC the size (in Kilo Lines Of Code) is given for each language. The result is 107 KLOC for the whole system of which 4.6% are Tscripts. If we consider the fact that Asf+Sdf specifications are compiled to C code, another view is possible as well: 12 KLOC of Asf+Sdf generates 170 KLOC of C code. Taking this generated code into account, the total size of the whole system amounts to 277 KLOC of which 1.8% are Tscripts. This is compatible with the expectation that Tscripts are relatively small and form high-level “glue” to connect much larger components.
ToolBus: The Next Generation
229
Fig. 5. Architecture of the Asf+Sdf Meta-Environment. Table 2. Facts concerning implementation languages.
†
Language KLOC† Generated KLOC Total KLOC ASF+SDF 12 170 (C) C 80†† Java, Tcl/Tk 5 Makefiles, etc 5 Tscript 5 Total LOC: 107 170 277 Tscript 4.6% 1.8% Kilo Lines of Code excluding third party code such as emacs, dot, and the like. †† This includes 10 KLOC (C code) for the ToolBus implementation itself.
Part of the generated C code is currently done by ApiGen [21]. This is an API generator which takes an Sdf grammar as input and generates a C library which gives type-safe access to the underlying ATerm representation of the parse trees over this grammar. Another conclusion from these facts is that low-level information for building the software (makefiles and configuration scripts) are of the same size as the high level Tscripts. This points into the direction that the level of these build scripts should be raised. This conclusion will, however, not be further explored in this paper. Another view is given in Table 3 where the frequency of occurrence of Tscript primitives is shown. Clearly, sequential composition (.) is the dominant operator and sending/receiving (snd-msg, rec-msg) messages is the dominant communication mechanism, followed by communication with tools (snd-do, snd-eval). It may be surprising that parallel composition (||) is used so infrequently. However, one should be aware that at the top level all ToolBus processes run concurrently and that || is only used for explicit concurrency inside a process. The level of concurrency is therefore approximately 100 (104 process definitions and 3 explicit || operators). Empirical evidence shows that: – The ToolBus-based version of the Asf+Sdf Meta-Environment is more flexible as illustrated by the fact that clones of the Meta-Environment start to appear for other languages than Asf+Sdf. Examples are Action Semantics [22] and Elan [23].
230
H. de Jong and P. Klint Table 3. Facts concerning Tscript primitives. Primitive Number of occurrences process definitions 104 tool definitions 27 . (sequential composition) 4343 + (choice) 341 * (iteration) 243 || (parallel composition) 3 snd-msg 555 rec-msg 541 snd-note 100 rec-note 24 snd-do/snd-eval 220 rec-event 56 create 58
– Various components of the Asf+Sdf Meta-Environment are being reused in other projects [10,24].
5
Issues in a Next-Generation TOOLBUS
The ToolBus has been used in various applications of which the Meta-Environment is by far the largest. Some of the questions posed by our users and ourselves are: – I find it difficult to see which messages are requests and which are replies; can you provide support for this? See Sect. 5.1. – If a tool crashes, what is the best way to describe the recovery in the Tscript? See Sect. 5.2. – I have huge data values that are exchanged between tools and the ToolBus becomes a data bottleneck; can you improve this? See Sect. 5.3. – The ToolBus and tools are running as separate tasks of the operating systems. Would it not be more efficient to run ToolBus and tools in a single task? See Sect. 6. 5.1
Undisciplined Message Patterns
The classical pattern of a remote procedure call is shown in Fig. 6: a caller performs a call to a callee. During the call the caller suspends execution and the callee executes until it has computed a reply. At that point in time, the caller continues its execution. Compare this simple situation with general message communication as shown in Fig. 7: the caller continues execution after sending a message msg1 to Callee1 and may even send a message msg2 to Callee2. At a certain point in time Callee2 may send message msg3 back to Caller. In this case, the three parties involved continue their execution while messages are being exchanged and there is no obvious pairing of calls and replies.
ToolBus: The Next Generation
231
Fig. 6. Communication pattern for remote procedure call.
Fig. 7. Communication pattern for general messages.
In the ToolBus case, a snd-msg and a rec-msg can interact with each other if their arguments match. A typical sequence is: Process A: snd-msg(calculate(E)) . ... other actions ... rec-msg(value(E, V?))
Process B: rec-msg(calculate(E?)) . ... actions that compute value V ... snd-msg(value(E, V))
What we see here is that a form of call/reply regime is encoded in the messages: process B returns the value V that it has computed as snd-msg(value(E, V)). The E is completely redundant but serves as identification for process A to which message this is an answer. The call/reply regime is thus implicitly encoded in messages. This makes error handling harder (which reply did not come?) and makes the Tscripts harder to understand. This is particularly so, since unstructured combinations of snd-msg/rec-msg and sequential composition, choice, iteration and parallel composition are allowed.
232
H. de Jong and P. Klint
The only solution for the above problems is to limit the occurrences of snd-msg or rec-msg in such a way that a form of very general call/reply regime is enforced. Our approach is to syntactically enforce that snd-msg/rec-msg or rec-msg/snd-msg may only occur in (possibly nested) pairs and that in between arbitrary operations are allowed. In fact, the matching snd-msg or rec-msg may be an arbitrary expression provided that all its alternatives begin with a matching snd-msg or rec-msg. We replace thus snd-msg(req(E)) . arbitrary process expression . rec-msg(ans(A?)) by the syntactic construct snd-msg(req(E)) { arbitrary process expression } rec-msg(ans(A?)) and also allow more general cases like: snd-msg (req(E)) { arbitrary process expression} ( rec-msg(ans(A?)) + rec-msg(error(M?) ) It is an interesting property of Process Algebra that every process expression can be normalized to a so-called action prefix form: a list of choices where each choice starts with an atomic action. An action prefix form has the following structure: a1 .P1 + a2 .P2 +...+ an .Pn . Using this property we can formulate the most general constraint that we impose on occurrences of snd-msg and rec-msg. Consider P1 { Q } P2 and let P1 ’ and P2 ’ be the action prefix forms of, respectively, P1 and P2 . Our requirement is now that each choice in P1 ’ starts we a snd-msg and each choice in P2 ’ with a rec-msg, or vice versa. Note that this constraint can be checked statically. 5.2
Exception Handling
Exception handling is notorious for its complexity and impact on the structure of program code. The mainstream exception handling approach as used in, for instance, Java associates one or more exception handlers with a specific method call. If the call completes successfully, the handlers are ignored. If the call raises an exception, it is checked whether this exception can be handled locally by one of the given handlers. If not, the exception is propagated to the caller of the current code. This model does, however, not work well in a setting where multiple processes are active and the occurrence of an exception may require recovery in several processes. Local Exception Handling. We start with the simpler case of local error handling and introduce the disrupt operator (>>) proposed in LOTOS [25]. A process algebra variant of this operator is described in [26]. It has the form P >> E, where P describes the normal processing and E the exceptional processing. It adds the exception E as alternative to each atomic action in P. If the action prefix form of P is a1 .P1 + a2 .P2 +...+ an .Pn , then P >> E ≡ (a1 + E).(P1 >> E) +.. + (an + E).(Pn >> E).
ToolBus: The Next Generation
233
Global Exception Handling. Global exception handling in distributed systems is a very well-studied subject from the perspective of crash recovery and transaction management in distributed databases. An overview of rollback-recovery protocols in message-passing systems is, for instance, given in [27]. In the context of system reliability, the notion of a recovery block has been introduced by Randell [28]. Its purpose was to provide several alternative algorithms for doing the same computation. Upon completion of one algorithm, an acceptance test is made. If the test succeeds, the program proceeds normally, but if it fails a rollback is made to the system state before the algorithm was started and one of the alternative algorithms is tried. In [29] this idea is applied to backtracking in string processing languages. It turns out that the preservation of the system state can be done efficiently by only saving updates to the state after the last recovery point. Recovery blocks also form the basis for Coordinated Atomic Actions described in [30]. Recovery blocks are intended for the error recovery in a single process. They can be generalized to conversations between more than one process: several processes can enter a conversation at different times but they can only leave it simultaneously, when all participating processes satisfy their acceptance test. In case one participant fails to pass its test, each participant is rolled back to the state when it entered the conversation. We are currently studying this model since it can be fit easily in the ToolBus framework and seems to solve our problem of global exception handling. It is helpful that a backtrack operator similar to the one described in [29] has also been described for Process Algebra [31]. What remains to be studied is how the recovery of tools has to be organized. Most likely, we will add a limited undo request to the tool interface to recover from the last few operations carried out by a tool. 5.3
Call-by-Value Versus Call-by-Reference
Background. The concepts of call-by-reference and call-by-value are well-known in programming languages. They describe how an actual parameter value is transmitted from a procedure call to the body of the called procedure. In the case of call-by-reference, a pointer to the parameter is transmitted to the body. Call-by-reference is efficient (only a pointer has to be transmitted) and the parameter value can be changed during execution of the procedure body (via the pointer). In the case of call-by-value, a physical copy of the parameter is transmitted to the procedure body. Call-by value is less efficient for large values and does not allow the called procedure to make changes to the parameter value in the calling procedure. These considerations also apply to value transmissions in a distributed setting, with the added complication that values can be accessed or modified by more than one party. Call by reference (Fig. 8) is efficient for infrequent access or update. It is the prevalent mechanism in, for instance, CORBA [32]. However, uncontrolled modifications by different parties can lead to disaster. Call-by-value (Fig. 9) is inefficient for large values and any sharing between calls is lost. To us, this is of particular interest, because we need to preserve sharing in huge parse trees. In the case of Java RMI [33], value transmission is achieved via serialization and works only for communication with other Java components. Using IIOP [34] communication with non-Java components is possible.
234
H. de Jong and P. Klint
Fig. 8. Call-by-reference in a distributed application.
Fig. 9. Call-by-value in a (Java-based) distributed application.
Current ToolBus approach. Currently, the ToolBus provides a transport mechanism based on call-by-value as shown in Fig. 10(a). It is transparent since the transmitted values are ATerms (see Sect. 1.2) that can be exchanged with components written in any language. Since pure values are exchanged, there is no need for distributed garbage collection. Note that the call-by-reference model can easily be mimicked in the ToolBus. For instance, one tool can maintain a shared database and can communicate with other tools using record keys and field names so that only the values of record fields have to be exchanged (as opposed to complete records or even the complete database). In this way the access control to the shared database can be spelled out in detail and concurrency conflicts can be avoided. This solves one of the major disadvantages of the pure call-byreference model in a distributed environment. The downside is, however, that the ToolBus becomes a data bottleneck when huge values really have to be transmitted between tools. Currently, two workarounds are used. A first workaround is to get temporary relief by sending compressed values rather than the values themselves. A second workaround is to store the large value in the filesystem and to send a file name rather than the file itself. It does scale, but it also creates an additional inter-tool dependency and assumes that both tools have access to the same shared file system. We will now first discuss how related frameworks handle call-by-reference and then we come back to implications for the ToolBus design. In particular, we will discuss channel-based transmission as already shown in Fig. 10(b). 5.4
Related Frameworks: Java RMI, RMI-IIOP and Java IDL
Given our needs and desires for a next generation ToolBus it is interesting to see what other solutions are applied in similar projects. In this section, we briefly look at three related mechanisms:
ToolBus: The Next Generation
235
Fig. 10. Value-based (a) versus channel-based (b) transmission in the ToolBus.
– Java Remote Method Invocation (RMI) which connects distributed objects written in Java; – Java RMI over Internet Inter-ORB Protocol (IIOP) which is like RMI, but uses IIOP as the underlying protocol; – Java IDL which connects Java implementations of CORBA interfaces. Java RMI. Java Remote Method Invocation is similar to the ToolBus architecture in the sense that it connects different tools, possibly running on different machines. It differs from the ToolBus setting because it is strictly Java based: only components written in Java can communicate via RMI. For components to work together in RMI, first a remote interface is established. This is a Java interface that has a “real” implementation in the tool (or server) and a “stub” implementation on the client sides (Fig. 11). The interface is written by the programmer as opposed to the generated interfaces in a ToolBus setting where they are derived from the communication patterns found in the ToolBus script. The stubs in the RMI setting are then generated from this Java interface using rmic: the RMI compiler. Stubs act as a client-side proxy, delegating the method call via the RMI system to the server object. In RMI, any object that implements a remote interface is called a remote object. In RMI, arguments to or return values from remote methods can be primitive data (e.g. int), remote objects, or serializable objects. In Java, an object is said to be serializable if it implements the java.util.Serializable interface. Both primitive data and serializable objects are passed by value using Java’s object serialization. Remote objects are essentially passed by reference. This means that changes to them are actually performed on the server, and updates become available to all clients. Only the behavior that was defined in the remote interface is available to the clients. RMI programmers should be aware of the fact that any parameters, return values and exceptions that are not remote objects are passed by value. This makes it hard to understand when looking at a system of RMI objects exactly which method calls will result in a local (i.e. client side) state change, and which will have global (server side) effect.
236
H. de Jong and P. Klint
Fig. 11. Client-server model in RMI framework.
Consider, again, our address book example. If the AddressBookService is implemented as a remote object in RMI, then client-side invocations of the setAddress method will cause a global update. If, on the other hand, the AddressBookEntries are made serializable and instances of this class are returned as the result of a query to the AddressBookService, then updates on these instances will have a local state change only. Finally, before two RMI components can connect, the server side needs to register itself with an rmiregistry, after which the client needs to explicitly obtain a reference to the (remote) server object. Java RMI over IIOP. By making RMI programs conform to some restrictions, they can be made available over the Internet Inter-ORB Protocol (IIOP). This means that functionality offered by the RMI program can be made available to CORBA clients written in any (CORBA supported) language. The restrictions are mostly namespace oriented: programmers need to take special care not to use certain names that might collide with CORBA generated names, but some reservations should also be made regarding sharing preservation of object references. References to objects that are equal according to the == operator in one component, need not necessarily be equal in a remote component. Instead the equals method should be used to discern equality. RMI over IIOP is best used when combining several Java tools for which the programmer would like to use RMI, and some tools written in another CORBA-supported language need to use (some of) the services provided by the Java tools. The component’s interface is established by writing a Java interface, just as in plain RMI. Java IDL. Apart from Java RMI, which is optimized for connecting components that are all written in Java, there is also a connection from Java to CORBA using the Java Interface Definition Language (IDL). This alternative to Java RMI is for Java programmers who want to program in the Java programming language, based on interfaces defined in the CORBA Interface Definition Language. Using this bridge, it becomes possible to let Java components communicate with CORBA objects written in any language that has Interface Definition Language (IDL) mappings.
ToolBus: The Next Generation ToolBus Architecture Component coordination Interface Tscript GC yes parameters / by-value return values language any with TB adapter component yes coordination
RMI Client Server Java Interface yes local: by-value remote: by-ref only Java no
RMI-IIOP Client Client Java Interface no local: by-value remote: by-ref CORBA objects if interface in Java no
237
Java IDL Client Server IDL no depends on signature any with IDL binding no
Fig. 12. Related architectures: a feature overview.
Instead of writing a Java interface as is done in RMI, in Java IDL the definition is written in IDL: a special purpose interface language used as the base for CORBA implementations. This IDL definition is then used to generate the necessary stubs (client side proxies to delegate method invocations to the server) and skeletons, holder and helper classes (server side classes that hide low-level CORBA details). Feature summary. Fig. 12 shows some of the similarities and differences in ToolBus, RMI, RMI-IIOP and Java IDL. – RMI, RMI-IIOP and Java IDL make an explicit distinction between client and server sides of a set of cooperating components. In the ToolBus setting all components are considered equal (and none are more equal than others). – In RMI and RMI-IIOP, the programmer writes a Java interface which describes the component’s incoming and outgoing method signature, from which stubs and skeletons are generated. In Java IDL a CORBA interface is written. In the ToolBus setting, these signatures are generated from the ToolBus script which describes much more of the component’s behavior in terms of method call interaction, rather than just method signatures. – The ToolBus takes care of garbage collection of the ATerms that are used to represent data as it is sent from one component to another. RMI allows programmers access to Java’s Distributed Garbage Collection API. In RMI-IIOP and Java IDL however, this is not possible, because the underlying CORBA architecture is used, which does not support (distributed) GC, but places this burden entirely on the developer. – In the ToolBus all data is sent by-value. RMI and RMI-IIOP use both pass-by-value and pass-by-reference, depending on whether the relevant data is serializable (it is a primitive type, or it implements Serializable) or is a remote object. In Java IDL the components abide by IDL prescribed interfaces. Determination of whether a parameter is to be passed by-value or by-reference is made by examination of the parameter’s formal type (i.e. in the IDL signature of the method it is being passed to). If it is a CORBA value type, it is passed by-value. If it is an ordinary CORBA interface type (the “normal” case for all CORBA objects), it is passed by-reference.
238
H. de Jong and P. Klint
– The ToolBus allows components in any language for which a ToolBus adapter exists. Programming languages such as C and Java are supported, but adapters also exist for a wide range of languages and applications, including e.g., Perl, Prolog, MySQL, Tcl and Asf+Sdf. In RMI, only Java components can be connected; in RMIIIOP the service is implemented in Java, its functionality (client-side) is available to CORBA clients. The Java IDL framework is fully CORBA compliant. – Only the ToolBus has coordination support for component interaction. In the three other cases any undesired sequence of incoming and outgoing method calls will have to be prohibited by adding code to the component’s internals. Whereas RMI, RMI-IIOP and Java IDL just perform the wiring that connects the components, the ToolBus also provides workflow support. In relation to this workflow support, it would be interesting to compare the ToolBus to related workflow description languages such as the Business Process Modeling language [35] and the Web Services Description Language [36]. Implications for the ToolBus Approach. To overcome the problems of value-based transmission, we envisage the introduction of channels as sketched in Fig. 10(b). This model is inspired by the second workaround mentioned at the end of Sect. 5.3 and is completely transparent for the user. The idea is to stick to the strict call-by-value transmission model, but to implement the actual value transmission by data communication between sending tool and receiving tool thus offloading the ToolBus itself. Via the ToolBus, only an identification of the data value is transmitted between sender and receiver. The downside of this model is that it introduces the need for distributed garbage collection, since a value may be distributed to more than one receiving tool and the sender does not known when all receivers have retrieved their copy. Adding expiration times to values or reference counting at the ToolBus level may solve this problem.
6
Current Status
The current ToolBus was first specified in Asf+Sdf and has then been implemented manually in C. Its primary target was the renovation of the Asf+Sdf Meta-Environment. The next generation ToolBus is being implemented in Java and aims at supporting larger applications such as, for instance, a multi-user game site like www. gamesquare.nl with thousands of users. High performance and recovery of crashed game clients are then of paramount importance. The Java implementation is organized in such a way that the actual implementation of tools is as much hidden as possible. This is achieved by introducing the interface ToolInterface that describes the required ToolBus/tool interaction. This interface can be implemented by a variety of classes: ClassicToolBusTool: This implements the ToolBus/tool communication as used in current applications. The tool is executed as a separate, operating system level, process and the ToolBus/tool communication is achieved using sockets. JavaTool: This implements a new model that addresses one of the issues mentioned in Sect. 5: when ToolBus and tool run on the same computer and the tool is written
ToolBus: The Next Generation
239
in Java, then the tool can be loaded dynamically in the executing ToolBus, e.g. using Java Threads. In this way, the overhead of interprocess communication can be eliminated. JavaRMITool: This is a special case where a Java tool runs on another computer. SOAPTool: This implements communication with a tool that has a SOAP interface. A prototype implementation is under development that allows experimentation with the features mentioned in this paper.
7
Concluding Remarks
In this paper we have reflected on our experiences over the past years with the use of the ToolBus as a means to refactor a previously monolithic system: the Asf+Sdf Meta Environment. This real test case of the ToolBus has taught us some of its shortcomings: its data bottleneck in case very large data items are sent using pass-by-value, maintenance issues related to undisciplined message passing and questions such as how to deal with exceptions caused by e.g. crashing tools. Some of the ideas we showed in this paper could be implemented by changing or extending the Tscript (e.g. to implement a call-reply regime as discussed in Sect. 5.1), others will also require extending the ToolBus and the tool-adapters (e.g. to detect crashed tools in combination with exception handling as discussed in Sect. 5.2). We have also studied some related ideas and frameworks and we are now in a position where we have a new prototype of the ToolBus in Java, with a very open structure which allows for all sorts of experiments and case studies based on the experience we have with the existing ToolBus and the ideas presented in this paper.
Acknowledgments We thank Pieter Olivier for his contribution and input to the many interesting and fruitful discussions we have had about ToolBus related issues, and his efforts to get www.gamesquare.nl ToolBus enabled.
References 1. Klint, P.: A meta-environment for generating programming environments. ACM Transactions on Software Engineering and Methodology 2 (1993) 176–201 2. van den Brand, M.G.J., van Deursen, A., Heering, J., de Jong, H.A., de Jonge, M., Kuipers, T., Klint, P., Moonen, L., Olivier, P.A., Scheerder, J., Vinju, J.J., Visser, E., Visser, J.: The ASF+SDF Meta-Environment: a Component-Based Language Development Environment. In Wilhelm, R., ed.: Compiler Construction (CC ’01). Volume 2027 of Lecture Notes in Computer Science., Springer-Verlag (2001) 365–370 3. Bakker, H.C.N., Koorn, J.W.C.: Building an editor from existing components: an exercise in software re-use. Technical Report P9312, Programming Research Group, University of Amsterdam (1993) 4. van Vlijmen, S.F.M., Vriend, P.N., van Waveren, A.: Control and data transfer in the distributed editor of the ASF+SDF meta-environment. Technical Report P9415, University of Amsterdam, Programming Research Group (1994)
240
H. de Jong and P. Klint
5. Mauw, S., Veltink, G.J.: A process specification formalism. Fundamenta Informaticae XIII (1990) 85–139 6. Bergstra, J.A., Klint, P.: The ToolBus: a component interconnection architecture. Technical Report P9408, University of Amsterdam, Programming Research Group (1994) 7. Bergstra, J.A., Klint, P.: The ToolBus coordination architecture. In Ciancarini, P., Hankin, C., eds.: Coordination Languages and Models. Volume 1061 of Lecture Notes in Computer Science. (1996) 75–88 8. Bergstra, J.A., Klop, J.W.: Process algebra: specification and verification in bisimulation semantics. In Hazewinkel, M., Lenstra, J.K., Meertens, L.G.L.T., eds.: Mathematics & Computer Science II. Volume 4 of CWI Monograph., North-Holland (1986) 9. Olivier, P.A.: Embedded system simulation: testdriving the ToolBus. Technical Report P9601, University of Amsterdam, Programming Research Group (1996) 10. Dams, D., Groote, J.F.: Specification and Implementation of Components of a muCRL toolbox. Logic Group Preprint Series 152, Utrecht University, Dept. of Philosoph (1995) 11. Lisser, B., van Wamel, J.J.: Specification of components in a proposition solver. Technical Report SEN-R9720, Centrum voor Wiskunde en Informatica (CWI) (1997) 12. Diertens, B.: Simulation and animation of process algebra specifications. Technical Report P9713, Programming Research Group, University of Amsterdam (1997) 13. Bergstra, J.A., Klint, P.: The discrete time ToolBus. Technical Report P9502, University of Amsterdam, Programming Research Group (1995) 14. Bergstra, J.A., Klint, P.: The discrete time ToolBus. In Wirsing, M., Nivat, M., eds.: Algebraic Methodology and Software Technology. Volume 1101 of Lecture Notes in Computer Science., Springer-Verlag (1996) 286–305 15. Bergstra, J.A., Klint, P.: The discrete time ToolBus—a software coordination architecture. Science of Computer Programming 31 (1998) 205–229 16. van den Brand, M.G.J., Heering, J., Klint, P.: Renovation of the ASF+SDF meta-environment current state of affairs. In Sellink, M.P.A., ed.: Proceedings of the 2nd International Workshop on the Theory and Practice of Algebraic Specifications. electronic Workshops in Computing, Springer-Verlag (1997) 17. van den Brand, M.G.J., de Jong, H.A., Klint, P., Olivier, P.A.: Efficient Annotated Terms. Software, Practice & Experience 30 (2000) 259–291 18. Olivier, P.A.: A Framework for Debugging Heterogeneous Applications. PhD thesis, University of Amsterdam (2000) 19. Brand, M.G.J.v.d., Moreau, P.E., Vinju, J.J.: Environments for Term Rewriting Engines for Free. In: Rewriting Techniques and Applications. Lecture Notes in Computer Science, Springer Verlag (2003) To appear. 20. Gelernter, D., Carriero, N.: Coordination languages and their significance. Communications of the ACM 35 (1992) 96 21. de Jong, H.A., Olivier, P.A.: Generation of abstract programming interfaces from syntax definitions. Technical Report SEN-R0212, St. Centrum voor Wiskunde en Informatica (CWI) (2002) Submitted to Journal of Logic and Algebraic Programming. 22. Mosses, P.D.: System demonstration: Action semantics tools. In van den Brand, M.G.J., L¨ammel, R., eds.: Proceedings of the Second Workshop on Language Descriptions, Tools and Applications (LDTA 2002). Volume 65.3 of Electronic Notes in Theoretical Computer Science. (2002) 23. Brand, M.G.J.v.d., Ringeissen, C.: ASF+SDF parsing tools applied to ELAN. In Futatsugi, K., ed.: Third International Workshop on Rewriting Logic and its Applications (WRLA’2000). Volume 36 of Electronic Notes in Theoretical Computer Science., Elsevier Science Publishers (2001)
ToolBus: The Next Generation
241
24. Blom, S.C.C., Fokkink, W.J., Groote, J.F., van Langevelde, I., Lisser, B., van de Pol, J.C.: µCRL: A toolset for analysing algebraic specifications. In Berry, G., Comon, H., Finkel, A., eds.: Proc. of the CAV 2001. Volume 2102 of LNCS., Springer (2001) 250–254 25. Brinksma, E.: On the Design of Extended LOTOS–A Specification Language for Open Distributed Systems. PhD thesis, University Twente (1988) 26. Diertens, B.: New features in PSF I – Interrupts, Disrupts, and Priorities. Technical Report P9417, Programming Research Group, University of Amsterdam (1994) 27. Elnozahy, E.N.M., Alvisi, L., Wang, Y., Johnson, D.B.: A survey of rollback-recovery protocols in message-passing systems. ACM Computing Surveys (CSUR) 34 (2002) 375–408 28. Randell, B.: System structures for software fault tolerance. IEEE Transactions on Software Engineering SE–1 (1975) 21–232 29. Klint, P.: A Study in String Processing Languages. Volume 205 of Lecture Notes in Computer Science. Springer-Verlag (1985) 30. Zorzo, A., Romanovsky, A., Xu, J., Randell, B., Stroud, R., Welch, I.: Using coordinated atomic actions to design dependable distributed object systems. In: OOPSLA’97 Workshop on Dependable Distributed Object Systems. (1997) 31. Bergstra, J.A., Ponse, A., van Wamel, J.: Process algebra with backtracking. In: REX School/Symposium. (1993) 46–91 32. Object Management Group (OMG): The Common Object Request Broker: Architecture and Specification. (1999) http://www.omg.org/technology/documents/formal/corba_2.htm. 33. Sun MicroSystems Inc.: Java Remote Method Specification. (2003) http://java.sun.com/j2se/1.4/docs/guide/rmi. 34. Object Management Group: CORBA IIOP Specification. (2003) www.omg.org/technology/documents/formal/corba_iiop.htm. 35. Business Process Management Initiative: Business Process Modeling Language. (2002) http://www.bpmi.org/bpmi-downloads/BPML1.0.zip. 36. W3C: World Wide Web Consortium: Web Services Description Language. (2001) http://www.w3.org/TR/wsdl.
High-Level Specifications: Lessons from Industry Brannon Batson1 and Leslie Lamport2 1 2
Intel Corporation Microsoft Research
Abstract. We explain the rationale behind the design of the TLA+ specification language, and we describe our experience using it and the TLC model checker in industrial applications–including the verification of multiprocessor memory designs at Intel. Based on this experience, we challenge some conventional wisdom about high-level specifications.
1
Introduction
The first author is a computer architect with a master’s degree in electrical engineering. His work focuses on designing, implementing, and validating multiprocessor cache-coherence protocols. He has worked on TLA+ formal specifications for the cache-coherence protocols of two Digital/Compaq Alpha multiprocessors, and he is currently using TLA+ to model protocols on future Intel products. The second author is a computer science researcher who began verifying concurrent algorithms over 25 years ago [12]. About ten years ago, he devised TLA, a logic for reasoning about concurrent algorithms [15]. He later designed TLA+, a complete high-level specification language based on TLA [17]. The two authors view formal verification and TLA+ from two different, complementary vantage points. In this paper, we try to synthesize our two views to explore the rationale behind TLA+, describe our experience using it in industry, and derive some lessons from this experience. When discussing our individual experiences, we refer to the first and second authors as BB and LL, respectively. We begin by describing TLA+ and TLC, the TLA+ model checker. We then describe how TLA+ has been used at Digital/Compaq and at Intel. We next explore how our experience contradicts some conventional wisdom about specification, and we end with some simple conclusions.
2 2.1
TLA+ Desiderata
TLA+ is a high-level language for describing systems – especially asynchronous concurrent and distributed systems. It was designed to be simple, to be very expressive, and to permit a direct formalization of traditional assertional reasoning – the style of reasoning begun by Floyd [5] and Hoare [9] and extended to concurrent programs by Ashcroft [2], Owicki and Gries [21], Pnueli [24], and F.S. de Boer et al. (Eds.): FMCO 2002, LNCS 2852, pp. 242–261, 2003. c Springer-Verlag Berlin Heidelberg 2003
High-Level Specifications: Lessons from Industry
243
others [3,11,12,22]. Making it easy, or even possible, to build tools was not a design criterion for the language. The desire to formalize assertional reasoning, especially for liveness properties, led LL to base TLA+ on TLA (the Temporal Logic of Actions) [15], a simple variant of linear-time temporal logic [24]. To be practical, a temporal logic must be based on an expressive language for writing elementary, non-temporal expressions. The desire for simplicity and expressiveness led to the use of ordinary first-order logic and set theory for this underlying language of expressions. 2.2
From Math to TLA+
First-order logic and set theory provide a formalization of ordinary mathematics. TLA adds to them modalities for expressing temporal properties. Temporal modalities are useful for describing liveness (eventuality) properties. However, temporal logic, like any modal logic, is more complicated than ordinary math. TLA was therefore designed to put most of the complexity, both in describing a system and in reasoning about it, into the realm of ordinary math rather than into temporal logic. LL originally assumed that first-order logic and set theory extended with the temporal operators of TLA would provide the semantic basis for a language, but that a practical language would require conventional programming-language constructs such as assignment statements. However, not wanting to introduce unnecessary constructs, he decided to begin writing formal specifications using only mathematics, and to add other constructs as needed. To his surprise, he discovered that he did not need those conventional constructs. Instead, he added to TLA+ only the following extensions to ordinary mathematics: Unambiguous Syntax. A formal language must be unambiguous, meaning that it must be possible for a program to parse it. This led to eliminating from TLA+ two common practices of mathematicians: using juxtaposition as an operator and the overloading of operators. Mathematicians write the product of x and y as xy; in TLA+ it is written x ∗ y. (One could require a space between x and y to distinguish this product from the single variable xy, but that would make parsing difficult.) Mathematicians frequently overload operators—for example, f −1 could mean either the inverse of f or f raised to the power −1. There is no overloading of operators in TLA+. (The use of types can make some instances of overloading unambiguous; but for reasons explained below, TLA+ is untyped.) New Constructs. TLA+ borrows a few useful constructs from computer science—for example, allowing if/then expressions like if x = 0 then 1/x else 0 Also, mathematicians have no notation for explicitly describing a function— for example, the function whose domain is the set of reals and that maps every number to its negative. In TLA+, this function is written [x ∈ Real → −x ]
244
B. Batson and L. Lamport
(A computer scientist might write this as a λ expression, but TLA+ avoids the keyword lambda because of its potentially confusing connotations.) Definitions. Mathematicians have no standard convention for defining operators. They typically write something like “let ◦ be defined by letting a ◦ b equal . . . , for any a and b.” In TLA+, one writes: ∆
a ◦ b = ... Support for Large Specifications. Mathematicians typically introduce new variables informally as needed. This casual introduction of variables could lead to errors in large specifications, so TLA+ requires that variables be declared before they are used. Moreover, mathematicians will write “let x be an element of S ” even though two pages earlier they had defined x to have some other meaning. Formalizing this requires some method of restricting the scope of a declaration. TLA+ does this through the use of modules, which provide a mechanism for structuring large specifications. The mathematical operation of substituting expressions for variables is expressed in TLA+ by instantiating a module with expressions substituted for its declared variables. Support for Large Formulas. For a mathematician, a 20-line formula is large. In a specification, a 200-line formula is not unusual. To aid in writing long formulas, TLA+ allows bulleted lists of conjuncts and disjuncts, using indentation to eliminate parentheses [14]. For example, ∧ reqQ[p][i ].type = “MB” ∧ ∨ DirOpInProgress(p, reqQ[p][i ].adr ) ∨ reqQ[p][j ].adr = reqQ[p][i ].adr means (reqQ[p][i ].type = “MB”) ∧ ( (DirOpInProgress(p, reqQ[p][i ].adr )) ∨ (reqQ[p][j ].adr = reqQ[p][i ].adr )) TLA+ also has a let/in construct for making definitions local to an expression. This permits structuring an expression for easier reading as well as combining multiple instances of the same subexpression. TLA. TLA+ extends ordinary math by adding the modal operators of TLA. The most important of these is prime ( ), where priming an expression makes it refer to the value of the expression in the next state. For example, x = x + 1 is an action, a predicate on a pair of states (called the current and next state), that is true iff the value of x in the next state is one greater than its value in the current state. Although formally a modal operator, expressions with primes obey the rules of ordinary mathematics, where x is treated like a new variable unrelated to the variable x . TLA also has a few simple temporal operators, used mostly for expressing liveness properties. As we will see, these operators appear in only a small part of a specification. We have listed the ways in which TLA+ differs from math as used by ordinary mathematicians. Where practical, TLA+ maintains the richness and economy of
High-Level Specifications: Lessons from Industry
245
inline AppendNum(n) { i = 0; do :: i < MaxSeqLen && seq[i] != 0 && seq[i] != n -> i++ :: else -> break od; if :: i >= MaxSeqLen || seq[i] != 0 :: else -> seq[i] = n fi } Fig. 1. Part of a toy specification written in Promela. ∆
AppendNum(n) = ∧ ∀ i ∈ 1 . . Len(seq) : n = seq[i] ∧ seq = Append (seq, n) ∧ num = num Fig. 2. The TLA+ version of the piece of specification in Figure 1.
ordinary mathematical notation. For example, while a textbook on first-order logic typically defines only a simple quantified formula such as ∃ x : exp, mathematicians typically write formulas like: ∃ x , y ∈ S , z , w ∈ T : exp TLA+ allows this kind of richer syntax. On the other hand, mathematicians do not use extraneous keywords or punctuation. TLA+ maintains this simplicity of syntax; for example, successive statements are not separated by punctuation. This syntactic economy makes TLA+ specifications easy for people to read, but surprisingly hard for a program to parse. 2.3
A Brief Taste of TLA+
To provide a sense of how a specification written in TLA+ compares to one written in a typical specification language used in industry, Figures 1 and 2 give small parts of two versions of a toy specification. The first version is written in Promela, the input language of the Spin model checker [10]. (It was written by Gerard Holzmann, the designer of the language.) The second is written in TLA+. The figures contain corresponding parts of the two specifications, although those parts are not completely equivalent. (One significant difference is mentioned in Section 3 below.) Figure 3 shows a small part of a TLA+ specification of a real cache-coherence protocol for a multiprocessor computer. Observe that it looks very much like the piece of toy specification of Figure 2; it is just a little more complicated. In both examples, the TLA+ specification uses only simple mathematics.
246
B. Batson and L. Lamport
∧ req.type = “MB” ∧ ∀ i ∈ 1 . . (idx − 1) : ∧ reqQ[p][i].type = “MB” ∧ DirOpInProgress(p, reqQ[p][i].adr ) ∧ ∀ j ∈ 1 . . (i − 1) : reqQ[p][j ].adr = reqQ[p][i].adr ∧ ¬ ∃ m ∈ msgsInTransit : ∧ m.type ∈ {“Comsig”, “GetShared”, “GetExclusive”, “ChangeToExclusive”} ∧ m.cmdr = p Fig. 3. A small piece of a real TLA+ specification.
Crucial to the simplicity of TLA+ is that it is based on the ordinary mathematics used by ordinary mathematicians. Computer scientists have devised many forms of weird mathematics. They have introduced bizarre concepts such as “nondeterministic functions”, leading to strange formalisms in which the formula A = A is not necessarily true. Ordinary mathematics was formalized about a century ago in terms of first-order logic and (untyped) set theory. This is the formal mathematics on which TLA+ is based. The use of ordinary mathematics in TLA+ led BB to remark: if I want to find out what an expression in a TLA+ specification means, I can just look it up in a math book. Computer scientists and engineers, accustomed to computer languages, are likely to be surprised by the expressiveness of TLA+. BB describes the power of TLA+ in this way: A single line of TLA+ can do powerful manipulations of complex data structures. This allows me to focus on the algorithm without getting lost in bookkeeping tasks. TLA+ tops even perl in this regard. Unlike perl, however, TLA+ is unambiguous. The simplicity and expressiveness of TLA+ is not the result of any cleverness in the design of the language; it comes from two thousand years of mathematical development, as formalized by great mathematicians like Hilbert. TLA+ is described in a recent book by LL, which is available on the Web [17]. 2.4
The Structure of a TLA+ Specification
TLA+ does not enforce any particular way of structuring a specification, allowing declarations and definitions to appear in any order as long as every identifier is declared or defined before it is used. Moreover, by partitioning it into separate modules, one can structure a specification to allow reading higher-level definitions before reading the lower-level definitions on which they depend. Logically, most TLA+ specifications consist of the following sections: Declarations. The declarations of the constant parameters and the variables that describe the system. There are typically a dozen or so declared identifiers.
High-Level Specifications: Lessons from Industry
247
Definitions of Operations on Data Structures. These define mathematical operators used to describe operations specific to the particular system. For example, a specification of a system that can perform a masked store to a register might define MaskedStoreResult(curr , val , msk ) to be the new value of a register, whose current value is curr , after storing to it a value val through a mask msk . A specification usually has only a few such operator definitions, each just a few lines long. The operator MaskedStoreResult is more likely to be used only in the definition that describes the masked store action, in which case it would probably be defined locally in a let/in expression. The Initial Predicate. The definition of a formula that describes the possible initial values of the variables. It is a conjunction of formulas x = . . . or x ∈ . . . for each variable x , where the “. . .” is usually a simple constant expression. The Next-State Action. The definition of a formula containing primed and unprimed variables that describes the system’s possible next state as a function of its current state. It is generally defined as a disjunction of subactions, each describing one type of system step. For example, in the specification of a mutual-exclusion algorithm, the next-state action might have a disjunct ∃ p ∈ Proc : EnterCS (p) , where EnterCS (p) describes a step in which a process p enters its critical section. The definition of the next-state action comprises the bulk of the specification. Liveness. The definition of a temporal formula specifying the liveness properties of the system, usually in terms of fairness conditions on subactions of the next-state action. It typically consists of about a dozen lines. The Specification. This is the one-line definition ∆
Spec = Init ∧ 2[Next]v ∧ Liveness that defines Spec to be the actual specification, where Init is the initial predicate, Next is the next-state action, Liveness is the liveness formula, and v is the tuple of all variables. For a high-level specification that describes a system’s correctness properties, Spec is the inner specification in which internal variables are visible. The true specification would be obtained by hiding those internal variables. If h is the tuple of all internal variables, then Spec with those variables hidden is represented by the TLA formula ∃ h : Spec. For technical reasons, TLA+ requires that this formula be defined in a separate module from the one defining Spec. However, hiding the internal variables is done for philosophical correctness only. The TLC model checker cannot handle the TLA hiding operator ∃ , and in practice, one uses the inner specification Spec. The only temporal operators that appear in the entire specification are the ones in the liveness formula and the single 2 in the final specification. The rest of the specification—usually about 99% of it—consists entirely of ordinary math,
248
B. Batson and L. Lamport
with no temporal operators. Moreover, engineers generally do not use the liveness property. The complexity of model checking liveness properties is inherently much greater than that of checking safety properties, which means that liveness can be checked only for extremely small models of real systems. Engineers therefore usually do not even write the liveness property; instead their specification is ∆ Spec = Init ∧ 2[Next]v The only temporal operator in this specification is the single 2. 2.5
The Size of Specifications
In principle, one could write specifications of any size, from tiny toy examples to million-line monsters. But tiny toys are of no practical interest, and the module structure of TLA+ is probably inadequate for handling the complexity of specifications longer than ten or twenty thousand lines. We have identified the following three classes of applications for which TLA+ is useful, each with a surprisingly narrow range of specification sizes: Abstract Algorithms. These are the types of concurrent algorithms that are published in journals—for example, the Disk Paxos algorithm [6]. Their specifications seem to require a few hundred lines of TLA+. Interesting algorithms simple enough to have much shorter specifications seem to be rare, while an algorithm with a much longer specification is probably not abstract enough for journal publication. Correctness Properties. These are descriptions of the properties that protocols or systems should satisfy. One example is a description of the memory model that a cache-coherence protocol is supposed to implement [20]. Their specifications also seem to require a few hundred lines of TLA+. The requirements of a real system are seldom simple enough to have a shorter specification, while a statement of correctness requiring a much longer specification would probably be too complicated to be useful. High-Level Protocol or System Designs. These describe the high-level designs of actual systems or protocols. We know of no published example; such designs are usually proprietary. We have found that these specifications are about two thousand lines of TLA+. Any such specification is an abstraction of the actual lower-level implementation. Engineers want to describe their design in as much detail as they can. However, if the specification takes much more than two thousand lines, then the design is too complicated to understand in its entirety, and a higher-level abstraction is needed.
3
TLC: The TLA+ Model Checker
TLA+ was not designed with tools in mind; LL believed that a practical model checker for it was impossible and advised against trying to write one. Fortunately, Yuan Yu ignored him and wrote TLC, an explicit-state model checker for TLA+ programmed in Java [26].
High-Level Specifications: Lessons from Industry
249
TLA+ is an extremely expressive language—for example, it can easily be used to specify a program that accepts an arbitrary Turing machine as input and tells whether or not it will halt. No model checker can handle all TLA+ specifications. TLC handles a subset of TLA+ that seems to include most specifications of algorithms and correctness properties, as well as all the specifications of protocol and system designs that engineers actually write. Those few specifications arising in practice that TLC does not handle can be easily modified, usually by changing only a few lines, so they can be checked by TLC. Explicit-state model checking is possible only for bounded-state specifications. Most high-level specifications are not bounded-state because the state contains data structures such as unbounded queues. We want engineers to use the TLA+ specification as the official high-level specification of their system, and a major goal of TLC is that the specification should not have to be changed to allow it to be checked. So TLC accepts as input a TLA+ specification and a configuration file that defines a finite model. The configuration file instantiates the constant parameters of the specification—for example, instructing TLC to replace the parameter Proc that represents the set of processors with a set containing three elements. The configuration file can also specify a constraint on the state space, instructing TLC to explore only states satisfying the constraint. As an example, we return to the toy specification, part of which is specified in Figures 1 and 2. In that specification, the variable seq represents a queue that can grow arbitrarily large. To model check it with TLC, we write a simple module that imports the original specification, declares a constant MaxSeqLen, and defines a constraint asserting that the length of seq is at most MaxSeqLen. We then instruct TLC to check the specification using that constraint, substituting a specific value for MaxSeqLen. We do this for increasing values of MaxSeqLen until we are confident enough that the specification is correct, or until the space of reachable states becomes so large that it takes TLC too long to explore it. In contrast, note in Figure 1 that, to model check the specification with Spin, the parameter MaxSeqLen had to be made part of the actual Promela specification. Operations on many common data types are not built into TLA+, but are instead defined in standard modules. For example, the natural numbers are defined in the Naturals module to be an arbitrary set satisfying Peano’s axioms, and arithmetic operation are defined in the usual way in terms of the next-number function. A specification that uses operators like + on natural numbers imports the Naturals module. It would be rather difficult for TLC to compute 2 + 2 from the definition of + in that module. Instead, such arithmetic operations are programmed in Java using TLC’s general module overriding mechanism. When a specification imports a module named M , TLC looks for a Java class file named M.class. If it finds one, it replaces operators defined in M with their Java implementations in M.class. There are Java implementations for common operators on numbers, sequences (lists), finite sets, and bags (multisets) that are defined in the standard modules. Ordinary users can also write their own Java class files to provide more efficient implementations of the operators that they define. However, we know of no case where this was necessary, and we know of only one user (a researcher) who has done it.
250
B. Batson and L. Lamport
TLC has a primitive command-line interface. Debugging is done by adding print statements to the specification. Although this violates the principle of not having to modify the specification to check it, the print statements are usually removed as soon as initial debugging is completed and simple “coding” errors corrected. Design errors are generally found by examining error traces. We hope eventually to add a graphical user interface. TLC is coded in Java. It is a multithreaded program and there is a version that can use multiple machines. For checking safety properties, it obtains close to an n-fold speedup with n processors when run on Alphas using a high quality Java runtime. However, we have found that the poor implementation of multithreading in many Java runtimes can significantly reduce the speedup. The largest case we know of was one with 900 million reachable states that took about two weeks on a four-processor machine. The expressiveness of TLA+ makes it essentially impossible to compile TLA+ specifications into efficient code. Therefore, TLC must interpret specifications. We guess that this makes TLC about ten times slower than explicit-state model checkers that require specifications to be written in a low-level, compilable language. Because TLC maintains its data structures on disk, it has essentially no space limitations for checking safety properties. The goal of TLC is to help engineers find bugs in their designs. Experience tells an engineer what kind of bugs a particular finite model is and is not likely to find. For example, if a cache-coherence protocol handles different addresses independently, then it may suffice to check models with only a single memory address. A specification is an abstraction, and checking it can find only errors that are present in that abstraction. Lower levels of detail introduce many other sources of error not reflected in the specification. In most industrial applications, engineers need cost-effective methods of finding bugs; they are not seeking perfection.
4
TLA+ Use in Industry
4.1 Digital/Compaq/HP TLA+ and TLC were conceived at the Digital (later Compaq) Systems Research Center. The first serious industrial use of TLA+ was for specifying and writing part of a hand proof of the cache-coherence protocol of a multiprocessor codenamed Wildfire, based on the Alpha EV6 processor [7]. The Wildfire experience inspired Yuan Yu to write TLC. It also persuaded the verification team to write a TLA+ specification of the cache-coherence protocol of the next generation Alpha, the EV7. The initial specification viewed an entire processor chip as a single component; the level of abstraction was later lowered to model protocol interactions between on-chip components as well. TLC checked important properties of the protocol and helped find several bugs. TLC and the EV7 protocol specification were also used as the basis of a project to improve test coverage for hardware simulation [25]. The processor design team for the next Alpha processor, the EV8, began using TLA+ to write the official specification of its cache-coherence protocol. However, development of that processor was cancelled.
High-Level Specifications: Lessons from Industry
251
TLA+ and TLC were also applied by Compaq engineers to the cache- coherence protocols of two Itanium-based processors. Researchers used TLC to debug a bus protocol proposal and to help develop database recovery and cache management protocols. TLC is now used routinely by some researchers to check the concurrent algorithms that they develop. The use of TLA+ and TLC at Digital and Compaq, some of which continued at HP, is described in [18]. 4.2
Intel
We now describe the use of TLA+ and TLC by BB and his colleagues at Intel. The actual specifications are proprietary and have not been viewed by anyone outside Intel. Overview of the Problem. Designing a complex system starts with a problem statement and an appropriate set of boundary conditions. A component of a computer system is initially represented abstractly as a black box, with assertions about its functionality and with some guidelines on performance, cost, and complexity. The engineering process involves iteratively refining this abstract model into lower-level models. Each lower-level model is a representation of the design at a certain level of abstraction, and it has a specific purpose. Some models are meant to evaluate tradeoffs between scope, performance, cost, and complexity. Others carry the design down to the low level of detail needed to manufacture the component. The engineering process therefore creates multiple representations of a design. Validation entails checking these multiple representations against one another. Designers of digital systems have good tools and methods for validating mid-level functional models, written in a hardware description language (HDL) like VHDL or RTL, against lower-level models such as circuit net-lists. However, they have not had as much success checking the higher-level functional representations of the design against one another, and against the initial problem statement and functional assertions. For some components, there is an intuitive correlation between the highlevel notions of correctness and the HDL model; such components tend not to be difficult to validate. Other components, like multiprocessor cache-coherence protocols, are sufficiently complex that checking the HDL model against the problem statement is quite challenging. We need formal techniques from the world of mathematics to perform this high-level validation. Although formal methods are based on mathematics, engineers view them differently from the way mathematicians do. To engineers, formal verification is simply another imperfect validation tool (albeit a powerful one). A TLA+ specification is only an abstraction of the actual system, and model checking can usually validate the specification only for a highly restricted set of system parameters. Validating the specification therefore cannot guarantee that there are no errors in the system. For engineers, formal verification is a way of finding bugs, not of proving correctness.
252
B. Batson and L. Lamport
The main benefit of applying TLA+ to engineering problems comes from the efficiency of the TLC model checker in reaching high levels of coverage and finding bugs. A secondary benefit we have encountered is the ability of TLA+ and TLC to provide good metrics for the complexity of a design. Complexity is a major consideration in evaluating design tradeoffs. However, unlike performance or cost, engineers have not historically had a good way to quantify algorithmic complexity before attempting to validate a design. TLA+ encourages designers to specify the design abstractly, suppressing lower-level details, so the length of the specification provides a measure of a design’s complexity. TLC reports the size of the reachable state space, providing another measure of complexity. Experience and intuition will always have a place in evaluating complexity, but TLA+ and TLC provide robust and impartial input to the evaluation. Having this input early in the design process is of considerable value.
Designing with TLA+. The core group at Intel started using TLA+ at Compaq while working on the Alpha EV7 and EV8 multiprocessor projects described above. From that experience, the Alpha engineers learned that multiprocessor cache-coherence protocols are an ideal candidate for formal methods because most of the protocol bugs can be found at a high level of abstraction. They also learned that the true value of TLA+ and TLC would be realized when (a) they were applied early enough in the design to provide implementation feedback, and (b) the implementation was based directly on the specification that had been verified. On the EV8 project, the TLA+ specification was completed before the design was stable, and it provided important feedback to the designers. When the engineers from the Alpha group joined Intel, they began applying their experience in writing TLA+ specifications when collaborating with other Intel engineers on cache-coherence protocols for future Intel products. Intel engineers are now using TLA+ as an integral part of the design process for the protocols that are under study. Whiteboard Phase. Designing one cache-coherence protocol from scratch provided the engineers with the opportunity to evaluate TLA+ as a prototyping platform for complex algorithms. Work on this protocol started by exploring the design space on a whiteboard for about two months. In this phase, basic message sequencing was determined, as were some coarse notions of what state had to be recorded at the protocol endpoints. A basic direction was set, based on the guidelines for functionality, performance, and cost. Because of their background, engineers tend to visualize an algorithm in terms of a particular implementation. They are better at gauging implementation complexity than at measuring algorithmic complexity. One benefit of having engineers write formal specifications is that it helps them learn how to think about a protocol abstractly, independent of implementation details. We found that, even in the whiteboard phase of the protocol design, the Intel engineers were able to make some judgments on complexity by asking themselves, “How would I code this in TLA+ ?”.
High-Level Specifications: Lessons from Industry
253
The whiteboard phase produced a general understanding of the protocol philosophy, an understanding of the constraints placed on the communication medium, the basic message flows, and coarse ideas on what state needed to be maintained. The next step was to introduce the rigor of a formal specification. TLA+ Scratchpad Phase. The TLA+ scratchpad phase of the project involved formally describing the abstract system, with appropriate state variables representing high-level components. This phase took about two months, starting with the initial design of the protocol. The difficulty lay not in the use of TLA+— engineers frequently learn new programming languages—but rather in (a) determining the layer of abstraction and (b) exploring the protocol’s corner cases. Task (a) is where TLA+ forces engineers to think about the protocol abstractly, which they often find unnatural. Their ability to think abstractly improves with experience writing TLA+ specifications. Task (b) is inevitable when documenting a protocol formally, as it forces the designers to explore the corner cases. During the scratchpad phase, the designers had to return to the whiteboard a few times when they encountered new race cases while writing the specification. The actions that formed the major blocks of the specification were chosen early; very few changes were made later. The Intel engineers adopted a methodology used in the earlier Alpha specifications, in which the decomposition of high-level named actions is based on classifying the protocol messages that they process. This methodology has led to fairly readable specifications, since it means that each action changes only a few local state variables. It encouraged the protocol specifications to be designed in a modular way, which also enabled the inter-module interfaces in the specification to be similar to their low-level counterparts in the implementation. Running TLC. The initial week or so of running TLC was spent finding and fixing typographical errors and type mismatch problems. This time could probably have been shortened by doing more syntax checking when writing the specification, which is what one often does when programming. The next four weeks saw a continuous process of running TLC, finding bugs, fixing them, and re-running TLC. During this phase of the project, many assumptions and assertions about the protocol were brought into question. This had the effect of educating the engineers about the protocol they had designed. We have found that TLC can be a useful learning tool if we use in-line assertions and global invariants to check everything we think is true. The Intel engineers were able to develop an intuitive understanding of the correctness of the protocol by developing meaningful global invariants and having TLC check them. If an assertion or invariant fails, TLC generates a counterexample that is useful for visualizing a difficult race case. These counterexamples are such a powerful teaching aid that the Intel engineers have developed tools to translate the TLC output into nicely formatted protocol flow diagrams that are easier to read. Another useful feature of the TLC model checker is its coverage checking. TLC can print the number of times each action was “executed”. This provides a simple way to identify holes in coverage. Much of the effort expended by the engineers in debugging the specification was spent eliminating each of these
254
B. Batson and L. Lamport
holes, or convincing themselves that it represented an action that could never happen. The performance of the model checker was sufficient to debug a large protocol specification. The engineers determined a base configuration that would “execute” all the actions and that displayed all interesting known cases. This configuration could be run on a four-processor machine in about a day, enabling fast turn-around on bug fixes. Larger configurations were periodically run as sanity checks on the smaller ones. The engineers would also run TLC in simulation mode, which randomly and non-exhaustively explores the state space, allowing them to check much larger configurations. Such random simulations are similar to the ones engineers typically perform on lower-level models, but it has the advantage of being several orders of magnitude faster because it is based on the abstract TLA+ model, and it provides a robust metric for coverage. Optimizing with TLC. Once the initial protocol specification was successfully checked by TLC, the Intel engineers were able to use it as a test bed for exploring optimizations. TLA+ is an ideal language to explore changes because its expressiveness usually allows the new version to be written quickly. Model checking the modified specification with TLC not only checks functional correctness, but it also measures any increase in the state space. Such an increase implies additional algorithmic complexity. The engineers spent several months exploring additions to the protocol, testing them with TLC. As a general rule, they would consider adopting only those optimizations that did not appreciably expand the state space. The insight that TLA+ and TLC gave into the complexity of modifications to the protocol was invaluable in iterating towards an optimal solution that adequately weighed algorithmic complexity along with factors like cost and performance. A significant optimization was later made to the protocol. This optimization followed the normal design cycle described above, though on a compressed schedule. With the original design yielding a good starting point, the entire cycle (whiteboard phase, TLA+ coding, and verification with TLC) was done within six weeks. This modification was accomplished by a recent college graduate with an undergraduate degree in engineering. He was able to learn TLA+ well enough within a matter of weeks to do this work. Feedback on TLA+ Syntax. The feedback we have received from engineers about the TLA+ language has been mostly positive. Engineers are usually able to pick up and understand a specification within a couple of days. One mistake we made was to present TLA+ to hardware designers as similar to a programming language. This led to some frustration. A better approach seems to be to describe TLA+ as being like a hardware description language. Engineers who design digital systems are well acquainted with methods for specifying finite-state machines, with the associated restrictions of allowing a primed variable to be assigned a value only once within a conjunction, not allowing a primed variable to appear in a conjunction before the assignment of its value, etc. To an engineer, TLA+ looks like a language for specifying finite-state machines.
High-Level Specifications: Lessons from Industry
255
While writing the protocol specification at Intel, BB was impressed by the ease of specifying complex data structures in TLA+ as sets and tuples. The part of the specification that described and manipulated data structures was a small part of the complete protocol specification. This compact specification of “bookkeeping tasks”, along with the overall expressiveness of TLA+, won over the engineers who were accustomed to using more clumsy functional languages for specifying complex algorithms. For the algorithmic specification, TLA+ naturally encourages nested disjunctions of conjunctions (known to engineers as sums of products of expressions). This method for specifying Boolean formulas has both advantages and disadvantages. One advantage is that it allows expressive comment blocks and assertions to be inserted in-line with a nested conjunct. A disadvantage is that this tends to lead to large specifications. The engineers are experimenting with the use of TLA+ operators to encode large blocks of regular Boolean disjunctions as truth tables, which engineers find more natural to work with.
5
Some Common Wisdom Examined
Based on our experience using TLA+, we now examine the following popular concepts from the world of programming: types, information hiding, object-oriented languages, component-based/compositional specifications, hierarchical description/decomposition, and hierarchical verification. Most of these concepts were mentioned in this symposium’s call for papers. We do not question the usefulness of these concepts for writing programs. But high-level specifications are not programs. We find that in the realm of high-level specifications, these ideas are not as wonderful as they appear. 5.1
Types
Very simple type systems are very restrictive. Anyone who has programmed in Pascal has written programs that were obviously type-correct, but which were not allowed by Pascal’s simple type system. Moderately complicated type systems are moderately restrictive. A popular type system is that of higher-order logic [8]. However, it does not allow subtyping. With such a type system, an integer cannot be a real number. When writing Fortran programs, one gets used to 1.0 being unequal to 1. One should not have to put up with that kind of complication in a specification language. Subtyping is provided by predicate subtyping, perhaps best known through its use in the PVS verification system [23]. We will see below a problem with PVS’s predicate subtyping. Moreover, predicate subtyping is not simple. It has led to several bugs that caused PVS to be unsound. For a typed language to be as expressive as TLA+, it will need an extremely complicated type system, such as that of Nuprl [4]. Engineers have a hard enough task dealing with the complexity of the systems that they design; they don’t want to have to master a complicated type system too.
256
B. Batson and L. Lamport
A specification consists of a large number of definitions, including many local ones in let/in expressions. Although an operator may have a simple type, it is often hard or impossible to declare the types of the operators defined locally within its definition. Even when those type declarations are possible, LL has found that they clutter a specification and make it harder to read. (Early precursors of TLA+ did include type declarations.) Any information contained in a type declaration that is helpful to the reader can be put in a comment. The main virtue of types, which makes us happy to bear the inconvenience they cause when writing programs, is that they catch errors automatically. (That advantage disappears in type systems with predicate subtyping, in which type checking can require manually guided theorem proving.) However, we have found that the errors in a TLA+ specification that could have been found by type checking are generally caught quite quickly by running TLC with very small models. The problems with types are discussed at length in [19]. We want to emphasize that we do not dispute the usefulness of types in programming languages. We prefer to program in strongly typed languages. We are questioning the use of types only in a high-level specification language. 5.2
Information Hiding
We have learned that programmers should hide irrelevant implementation details. However, a high-level specification should not contain implementation details. Such details will appear in a well-written specification only if an inexpressive language requires high-level concepts to be described by low-level implementations. TLA+ provides users with powerful mathematical objects like sets and functions; they don’t have to encode them in arrays of bits and bytes. Such “bookkeeping details” do not occur in specifications written in a truly high-level language like TLA+, so there is no need to hide them. 5.3
Object-Oriented Languages
The mathematical concept underlying object oriented programming languages can be described as follows. A program maintains identifiers of (references to) objects. There is a function Obj that maps the set ObjectId of object identifiers to a set Object of objects. An object-oriented language simply hides the function Obj , allowing the programmer to write o.field instead of Obj [o].field , where o is an object identifier. Eliminating explicit mention of Obj can make a specification look a little simpler. But it can also make it hard to express some things. For example, suppose we want to assert that a property P (obj ) holds for every object obj . (Usually, P (obj ) will assert that, if obj is a non-null object of a certain type, then it satisfies some property.) This is naturally expressed by the formula ∀ o ∈ ObjectId : P (Obj [o]) It can be difficult or impossible to express in a language that hides Obj .
High-Level Specifications: Lessons from Industry
257
Object-orientation introduces complexity. It raises the problem of aliasing. It leads to the confusing difference between equality of object identifiers and equality of objects—the difference between o1 = o2 and o1.equals(o2). You can’t find out what o1.equals(o2) means by looking it up in a math book. Object-oriented programming languages were developed for writing large programs. They are not helpful for two-thousand-line programs. Object orientation is not helpful for two-thousand-line specifications. 5.4
Component-Based/Compositional Specifications
A high-level specification describes how the entire system works. In a TLA+ specification, a component is represented by a subexpression of the next-state relation—usually by a disjunct. We can’t understand a formula by studying its subexpressions in isolation. And we can’t understand a system by studying its components in isolation. We have known for 20 years that the way to reason about a distributed system is in terms of a global invariant, not by analyzing each component separately [13]. Many tools have been developed for debugging the low-level designs of individual hardware components. Engineers need a high-level specification to catch bugs that can’t be found by looking at individual components. 5.5
Hierarchical Description/Decomposition
Hierarchical description or decomposition means specifying a system in terms of its pieces, specifying each of those pieces in terms of lower-level pieces, and so on. Mathematics provides a very simple, powerful mechanism for doing this: the definition. For example, one might define A by ∆
A = B ∨C ∨D and then define B , C , and D. (TLA+ requires that the definitions appear in the opposite order, but one can present them in a top-down order by splitting the specification into modules.) Building up a specification by a hierarchy of definitions seems simple enough. But a specification language can make it difficult in at least two ways: – It can restrict the kinds of pieces into which a definition can be broken. For example, it might require the pieces to be separate processes. There is no reason to expect that splitting the system into separate processes will be the best way to describe it. – It can use a strict type system. For example, suppose x is a variable of type real number, and we want to define an expression A by ∆
A = if x = 0 then B else C where B is defined by ∆
B = 1/x
258
B. Batson and L. Lamport
This is a perfectly reasonable definition, but PVS’s type system forbids it. PVS allows the expression 1/x only in a context in which x is different from 0. This particular example is contrived, but TLA+ specifications often contain local definitions in let/in expressions that are type-correct only in the context in which they are used, not in the context in which they are defined. How Intel engineers use and don’t use hierarchical decomposition is somewhat surprising. As we observed above, the major part of a TLA+ specification is the definition of the next-state action. Intel engineers use the common approach of decomposing this definition as a disjunction such as ∆
Next = ∃ p ∈ Proc : A1 (p) ∨ . . . ∨ An (p) where each Ai (p) describes a particular operation performed by process p. They also use local definitions to make a definition easier to read. For example, an action Ai (p) might be defined to equal let newV = . . . newW = . . . in . . . ∧ v = newV ∧ w = newW ... This allows a reader to scan the in clause to see what variables the action changes, and then read the complex definitions of newV and newW to see what the new values of v and w are. What the Intel engineers do not do is use hierarchical decomposition to hide complexity. For example, they would not eliminate common subexpressions by writing ∆ SubAction(p) = . . . ∆
A1 (p) = . . . ∧ SubAction(p) ∧ . . . ∆
A2 (p) = . . . ∧ SubAction(p) ∧ . . . if the two instances of SubAction(p) represent physically distinct components. The Intel engineers rely on the TLA+ specification to gauge the complexity of their designs, using the number of lines in the specification as a measure of a design’s complexity. This is possible because TLA+ does not introduce the extraneous details needed by lower-level languages to encode higher-level concepts. 5.6
Hierarchical Verification
Hierarchical verification works as follows: To show that a system described by the specification Sys implements the correctness properties Spec, we write an intermediate-level spec Mid and show that Sys implements Mid and Mid implements Spec. In TLA+, implementation is implication. To show that a specification ∃ x : F implements a specification ∃ y : G, we must show
High-Level Specifications: Lessons from Industry
259
(∃ ∃ x : F ) ⇒ (∃ ∃ y : G) Here, x and y are tuples of variables, which we assume for simplicity to be distinct. By simple logic, we show this by showing F ⇒ (G with y ← exp) for some tuple of expressions exp, where with denotes substitution (which is expressed in TLA+ by module instantiation). The tuple exp is called a refinement mapping [1]. To show that Sys implies Spec, we must show Sys ⇒ (ISpec with h ← exp) where ISpec is the inner specification, with internal variables h visible. To use hierarchical verification, we find an intermediate-level inner specification IMid , with internal variables m visible, and show: Sys ⇒ (IMid with m ← exp1) IMid ⇒ (ISpec with h ← exp2) When the verification is done by TLC, there is no reason for such a decomposition; TLC can verify directly that Sys implements ISpec under the refinement mapping. When the verification is done by mathematical proof, this decomposition seems reasonable. However, as LL has argued elsewhere [16], it is just one way to decompose the proof; it is not necessarily the best way. Unfortunately, the whole problem of verifying that a high-level design meets its specification is not yet one that is being addressed in the hardware community. Thus far, engineers are checking only that their TLA+ design specifications satisfy an incomplete set of invariants. Checking that they satisfy a complete specification is the next step. Engineers want to do it, but they have to learn how—which means learning how to find a correct refinement mapping. TLC has the requisite functionality to do the checking, but only doing it for real systems will tell if it works in practice. The reader can get a feeling for the nature of the task by trying to solve the Wildfire Challenge Problem [20].
6
Conclusions
Our industrial experience with TLA+ has led us to some simple, common-sense conclusions: – Buzzwords like hierarchical and object-oriented are to be viewed with suspicion. – A language for writing high-level specifications should be simple and have effective debugging tools. – Only proofs and model checking can catch concurrency bugs in systems. For the vast majority of applications, proofs are not a practical option; engineers have neither the training nor the time to write them.
260
B. Batson and L. Lamport
– A specification method cannot be deemed a success until engineers are using it by themselves. TLA+ and TLC are practical tools for catching errors in concurrent systems. They can be used very early in the design phase to catch bugs when it is relatively easy and cheap to fix them. Writing a formal specification of a design also catches conceptual errors and omissions that might otherwise not become evident until the implementation phase. TLA+ is not just for industrial use. Anyone who writes concurrent or distributed algorithms can use it. We invite the reader to give it a try.
References 1. Mart´ın Abadi and Leslie Lamport. The existence of refinement mappings. Theoretical Computer Science, 82(2):253–284, May 1991. 2. E. A. Ashcroft and Z. Manna. Formalization of properties of parallel programs. In Machine Intelligence, volume 6. Edinburgh University Press, 1970. 3. K. Mani Chandy and Jayadev Misra. Parallel Program Design. Addison-Wesley, Reading, Massachusetts, 1988. 4. R. L. Constable, S. F. Allen, H. M. Bromley, W. R. Cleaveland, J. F. Cremer, R. W. Harper, D. J. Howe, T. B. Knoblock, N. P. Mendler, P. Panagaden, J. T. Sasaki, and S. F. Smith. Implementing Mathematics with the Nuprl Proof Development System. Prentice-Hall, 1986. 5. R. W. Floyd. Assigning meanings to programs. In Proceedings of the Symposium on Applied Math., Vol. 19, pages 19–32. American Mathematical Society, 1967. 6. Eli Gafni and Leslie Lamport. Disk paxos. To appear in Distributed Computing., 2002. 7. Kourosh Gharachorloo, Madhu Sharma, Simon Steely, and Stephen Van Doren. Architecture and design of AlphaServer GS320. In Anoop Gupta, editor, Proceedings of the Ninth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS IX), pages 13–24, November 2000. 8. M. J. C. Gordon and T. F. Melham. Introduction to HOL: A Theorem Proving Environment for Higher Order Logic. Cambridge University Press, 1993. 9. C.A.R. Hoare. An axiomatic basis for computer programming. Communications of the ACM, 12(10):576–583, October 1969. 10. Gerard Holzmann. The model checker spin. IEEE Transactions on Software Engineering, 23(5):279–295, May 1997. 11. Simon S. Lam and A. Udaya Shankar. Protocol verification via projections. IEEE Transactions on Software Engineering, SE-10(4):325–342, July 1984. 12. Leslie Lamport. Proving the correctness of multiprocess programs. IEEE Transactions on Software Engineering, SE-3(2):125–143, March 1977. 13. Leslie Lamport. An assertional correctness proof of a distributed algorithm. Science of Computer Programming, 2(3):175–206, December 1982. 14. Leslie Lamport. How to write a long formula. Formal Aspects of Computing, 6:580– 584, 1994. First appeared as Research Report 119, Digital Equipment Corporation, Systems Research Center. 15. Leslie Lamport. The temporal logic of actions. ACM Transactions on Programming Languages and Systems, 16(3):872–923, May 1994.
High-Level Specifications: Lessons from Industry
261
16. Leslie Lamport. Composition: A way to make proofs harder. In Willem-Paul de Roever, Hans Langmaack, and Amir Pnueli, editors, Compositionality: The Significant Difference (Proceedings of the COMPOS’97 Symposium), volume 1536 of Lecture Notes in Computer Science, pages 402–423. Springer-Verlag, 1998. 17. Leslie Lamport. Specifying Systems. Addison-Wesley, Boston, 2002. A link to an electronic copy can be found at http://lamport.org. 18. Leslie Lamport, John Matthews, Mark Tuttle, and Yuan Yu. Specifying and verifying systems with TLA+ . In Proceedings of the Tenth ACM SIGOPS European Workshop, pages 45–48, Saint-Emilion, France, September 2002. INRIA (Institut National de Recherche en Informatique et en Automatique). 19. Leslie Lamport and Lawrence C. Paulson. Should your specification language be typed? ACM Transactions on Programming Languages and Systems, 21(3):502– 526, May 1999. 20. Leslie Lamport, Madhu Sharma, Mark Tuttle, and Yuan Yu. The wildfire verification challenge problem. At URL http://research.microsoft.com/users/ lamport/tla/wildfire-challenge.html on the World Wide Web. It can also be found by searching the Web for the 24-letter string wildfirechallengeproblem. 21. Susan Owicki and David Gries. Verifying properties of parallel programs: An axiomatic approach. Communications of the ACM, 19(5):279–284, May 1976. 22. Susan Owicki and Leslie Lamport. Proving liveness properties of concurrent programs. ACM Transactions on Programming Languages and Systems, 4(3):455–495, July 1982. 23. Sam Owre, John Rushby, Natarajan Shankar, and Friedrich von Henke. Formal verification for fault-tolerant architectures: Prolegomena to the design of PVS. IEEE Transactions on Software Engineering, 21(2):107–125, February 1995. 24. Amir Pnueli. The temporal logic of programs. In Proceedings of the 18th Annual Symposium on the Foundations of Computer Science, pages 46–57. IEEE, November 1977. 25. Serdar Tasiran, Yuan Yu, Brannot Batson, and Scott Kreider. Using formal specifications to monitor and guide simulation: Verifying the cache coherence engine of the Alpha 21364 microprocessor. In In Proceedings of the 3rd IEEE Workshop on Microprocessor Test and Verification, Common Challenges and Solutions. IEEE Computer Society, 2002. 26. Yuan Yu, Panagiotis Manolios, and Leslie Lamport. Model checking TLA+ specifications. In Laurence Pierre and Thomas Kropf, editors, Correct Hardware Design and Verification Methods, volume 1703 of Lecture Notes in Computer Science, pages 54–66, Berlin, Heidelberg, New York, September 1999. Springer-Verlag. 10th IFIP wg 10.5 Advanced Research Working Conference, CHARME ’99.
How the Design of JML Accommodates Both Runtime Assertion Checking and Formal Verification Gary T. Leavens1 , Yoonsik Cheon1 , Curtis Clifton1 , Clyde Ruby1 , and David R. Cok2 1
Department of Computer Science, Iowa State University 226 Atanasoff Hall, Ames, Iowa 50011-1041 USA {leavens,cheon,cclifton,ruby}@cs.iastate.edu phone: +1 515 294 1580, fax: +1 515 294 1580 2 Eastman Kodak Company Research & Development Laboratories 1700 Dewey Avenue, Building 65, Rochester, New York 14650-1816 USA [email protected] phone: +1 585 588 3107, fax: +1 585 588 3269
Abstract. Specifications that are used in detailed design and in the documentation of existing code are primarily written and read by programmers. However, most formal specification languages either make heavy use of symbolic mathematical operators, which discourages use by programmers, or limit assertions to expressions of the underlying programming language, which makes it difficult to write complete specifications. Moreover, using assertions that are expressions in the underlying programming language can cause problems both in runtime assertion checking and in formal verification, because such expressions can potentially contain side effects. The Java Modeling Language, JML, avoids these problems. It uses a side-effect free subset of Java’s expressions to which are added a few mathematical operators (such as the quantifiers \forall and \exists). JML also hides mathematical abstractions, such as sets and sequences, within a library of Java classes. The goal is to allow JML to serve as a common notation for both formal verification and runtime assertion checking; this gives users the benefit of several tools without the cost of changing notations.
1
Introduction
The Java Modeling Language, JML [55, 54], is the result of a cooperative, international effort aimed at providing a common notation and semantics for the specification of Java code at the detailed-design level [58]. JML is being designed cooperatively so that many different tools can use a common notation for Hoarestyle behavioral interface specifications. In this paper we explain the features of JML’s design that make its assertions easily understandable by programmers and suitable for both runtime assertion checking and formal verification. F.S. de Boer et al. (Eds.): FMCO 2002, LNCS 2852, pp. 262–284, 2003. c Springer-Verlag Berlin Heidelberg 2003
How the Design of JML Accommodates
1.1
263
Background
By a Hoare-style specification we mean one that uses pre- and postconditions to specify the behavior of methods [34, 43, 44]. A behavioral interface specification language (BISL) is a specification language that specifies both the syntactic interface of a module and its behavior [33, 48, 52, 85]. JML, the interface specification languages in the Larch family [33,48,52,85] and RESOLVE/C++ [22,73] are BISLs. Most design by contract languages and tools, such as Eiffel [70, 71] and APP [77], are also BISLs, because they place specifications inside programming language code. By contrast, neither Z [80, 79, 87] nor VDM [6, 27, 74, 43] is a BISL; they have no way to specify interface details for a particular programming language. OCL [82, 83] is a BISL for the UML, but the UML itself is language-independent; this poses problems for a Java programmer, because the UML does not have standard notations for all details of Java method signatures. For example, the UML’s syntax for specifying the signatures of operations has no standard notation for declaring that a Java method is strictfp or for declaring the exceptions that a method may throw [7, pp. 128-129] [49, p. 516]1 . Also the OCL has no standard constraints that correspond to JML’s exceptional postconditions. Because BISLs like JML specify both interface and behavior, they are good at specifying detailed designs that include such Java details. This makes JML well suited to the task of documenting reusable components, libraries, and frameworks written in Java. 1.2
Tool Support
Because BISLs are easily integrated with code, they lend themselves to tool support for activities related to detailed design, coding, testing, and maintenance. An important goal of JML is to enable a wide spectrum of such tools. Besides tools that enforce JML’s semantics (e.g., type checking), the most important JML tools help with the following tasks. Runtime checking and testing. The Iowa State group provides (from www. jmlspecs.org): – the jmlc runtime assertion checking compiler [13], which generates class files from JML-annotated Java sources2 , and – the jmlunit tool [14], which uses the runtime assertion checker to generate test oracle code for JUnit tests. Documentation. David Cok provides the jmldoc tool, also available through www.jmlspecs.org, which generates HTML documentation similar to that 1
2
Larman notes that the UML has some nonstandard ways to specify the exceptions that a method may throw, by either using Java’s own syntax directly or by using a “property string”. Besides this runtime assertion checking work at Iowa State, which relies on adding instrumentation to compiled code, Steven Edwards’s group at Virginia Tech is working on a wrapper-class based approach to runtime assertion checking that will allow instrumentation of programs for which source code is not available.
264
G.T. Leavens et al.
produced by javadoc [29], but containing specifications as well. The generated documentation is useful for browsing specifications or posting to the web. Static analysis and verification. The following tools are prepared by our partners at Compaq and the University of Nijmegen: – The ESC/Java tool [28,65,66] statically checks Java code for likely errors. ESC/Java understands a subset of JML annotations. – The LOOP tool [37,38,40,42] assists in the formal verification of the correctness of implementations from JML specifications, using the theorem prover PVS. In addition, the Daikon dynamic invariant detector [23,72] outputs invariants for Java programs in a subset of JML, and the Korat automated testing tool [8] uses the jmlunit tool to exercise the test data it derives. In this paper, we discuss how JML meets the needs of tools for runtime assertion checking, documentation, static analysis, and verification. We focus on runtime assertion checking and formal verification, which we consider to be the extremes of the spectrum of tools that a BISL might support. The tasks of runtime assertion checking and formal verification have widely differing needs: – Runtime assertion checking places a high premium on executability. Many specification languages intended for runtime assertion checking, such as Eiffel [70, 71] and APP [77], only allow assertions that are completely executable. This is sensible for a language that is intended only to support runtime assertion checking and not formal verification. – On the other hand, formal theorem proving and reasoning place a high premium on the use of standard mathematical notations. Thus, most specification languages intended for formal reasoning or verification, such as VDM, the members of the Larch family, and especially Z, feature a variety of symbolic mathematical notations. Many expressive mathematical notations, such as quantifiers, are impossible, in general, to execute at runtime. Again, including such notations is sensible for a language intended only to support formal theorem proving and reasoning and not runtime assertion checking. 1.3
Problems
We begin by describing the problems that arise when addressing the needs of the range of tools exemplified by runtime assertion checking and formal verification. 1.3.1 Notational Problem. It is often said that syntax does not matter; however, our experience with Larch/Smalltalk [11] and Larch/C++ [12, 50, 51, 53, 56] showed that programmers object to learning a specialized mathematical notation (the Larch Shared Language). This is similar to the problems found by Finney [26], who did a preliminary experiment demonstrating that the symbolic notation in Z specifications may make them hard to read. Conversely, in executable languages like Eiffel and APP, programmers feel comfortable with the use of the programming language’s expressions in assertions. Such an assertion
How the Design of JML Accommodates
265
language is therefore more appealing for purposes of documentation than highly symbolic mathematical notations. To summarize, the first problem that we address in this paper is how to provide a good syntax for specification expressions. Specification expressions are the syntactic forms that are used to denote values in assertions. By a good syntax we mean one that is close enough to programming language expressions that programmers feel comfortable with it and yet has all of the features necessary to support both runtime assertion checking and formal verification. 1.3.2 Undefinedness Problem. Expressions in a programming language may abruptly terminate (e.g., throw exceptions) and may go into infinite loops; consequently, they may have undefined values from a strictly mathematical point of view. Programming languages typically provide features to deal explicitly with such undefinedness. For example, Java provides short-circuit versions of boolean operators (such as && and ||) that allow programmers to suppress evaluation of some subexpressions. We want both programmers and mathematicians to use JML’s notations; hence, JML’s specification expressions should not only look like Java’s expressions and use Java’s semantics, but should also validate the standard laws of logic. However, because of a potential for undefinedness, Java expressions do not satisfy all the standard rules of logic; for example, in Java the conjunction E1 && E2 is not equal to E2 && E1 , although in logic they would be equal. To resolve this conflict, we are willing to accept a slightly different semantics for assertion evaluation as long as programmers are not too surprised by it. Thus, the second problem we address in this paper is how to find a semantics for expressions used in assertions that validates standard laws of logic and yet does not surprise programmers and is still useful for runtime assertion checking. 1.3.3 Side Effects Problem. Another important semantic issue is that expressions in a programming language like Java (and most others, including Eiffel) can contain side effects. Side effects have a very practical problem related to runtime assertion checking. It is generally assumed that assertions may be evaluated or skipped with no change in the outcome of a computation, but an assertion with side effects has the potential to alter the computation’s outcome. For example, an assertion with side effects might mask the presence of a bug that would otherwise be revealed or cause bugs that are not otherwise present. Because one of the principal uses of runtime assertion checking is debugging and isolating bugs, it is unacceptable for side effects from assertion checking to alter the outcome of a computation. Thus, the third problem that we address in this paper is how to prevent side effects in assertions while still retaining as much of the syntax of normal programming language expressions as possible. 1.3.4 Mathematical Library Problem. Most specification languages come with a library of mathematical concepts such as sets and sequences. Such concepts are especially helpful in specifying collection types. For example, to specify
266
G.T. Leavens et al.
a Stack type, one would use a mathematical sequence to describe, abstractly, the states that a stack object may take [35]. VDM, OCL, Z, and the interface specification languages of the Larch family all have libraries of such mathematical concepts. They also are standard in theorem provers such as PVS. However, as discussed in Section 1.3.1, we want to limit the barriers that Java programmers must overcome to use JML. Thus, the fourth problem that we address in this paper is how to provide a library of mathematical concepts in a way that does not overwhelm programmers, and yet is useful for formal verification. 1.4
Other Goals of JML
In addition to providing solutions to the preceding four problems, the design of JML is guided and constrained by several other goals. One of the most important of these goals is to allow users to write specifications that document detailed designs of existing code. This motivates the choice of making JML a BISL, as described above. Moreover, we would like JML to be useful for documenting code regardless of whether it was designed according to any particular design method or discipline. This is important because the cost of specification is high enough that it is not always justified until one knows that the design and the code have stabilized enough to make the documentation potentially useful to other people. In general, JML’s design adheres to the goal of being able to document existing designs; however, there is one significant aspect of JML’s design that departs from this goal—JML imposes the specifications of supertypes on subtypes, a property termed specification inheritance, in order to achieve behavioral subtyping [19]. JML’s use of specification inheritance is justified by another of our goals: we want JML to support modular reasoning, that is, reasoning about the behavior of a compilation unit using just the specifications of the compilation units that it references (as opposed to the details of their implementations). Modular reasoning is important because without it, the difficulty of understanding an object-oriented program increases much more rapidly than the size of the program, and thus the benefits of the abstraction mechanisms in object-oriented languages are lost. Consequently, modular reasoning is also important for formal verification, because then the scope of the verification problem is limited. Specification inheritance, and the resulting behavioral subtyping, allows modular reasoning to be sound, by allowing one to reason based on the static types of references. Subsumption in Java allows a reference to a subtype object to be substituted for a supertype reference. The requirements of behavioral subtyping [2, 3, 19, 57, 59, 63, 69] guarantee that all such substituted objects will obey the specifications inherited from the static type of the reference [19, 60, 61]. Because of the benefits of modular reasoning to programmers and verifiers, we favor specification inheritance over the conflicting goal of being able to document existing designs that do not follow behavioral subtyping. In any case, it is possible to work around the requirements of behavioral subtyping for cases in which a subtype does not obey the inherited specifications of its supertype(s). One simply underspecifies each supertype enough to allow all of the subtypes that
How the Design of JML Accommodates
267
are desired [63, 69]. Note that this work-around does not involve changing the code or the design, but only the specification, so it does not interfere with the goal of documenting existing code. 1.5
Outline
The remainder of this paper is organized as follows. The next section discusses our solution to the notational problem described above. Having described the notation in general terms, Section 3 provides more background on JML. The subsequent three sections treat the remaining problems discussed above. The paper ends with a discussion of related work and some conclusions.
2
Solving the Notational Problem
To solve the notational problem described in Section 1.3.1, JML generally follows Eiffel, basing the syntax of specification expressions on Java’s expression syntax. However, because side effects are not desired in specification expressions, JML’s specification expressions do not include Java expressions that can cause obvious side effects, i.e., assignment expressions and Java’s increment and decrement operators (++ and --). Furthermore, to make JML suitable for formal verification efforts, JML includes a number of operators that are not present in Java [55, Section 3]. The syntax of these operators comes in two flavors: those that are symbolic and those that are textual. We did not want to introduce excess notation that would cause difficulties for programmers when reading specifications, so JML adds just five symbolic operators. Four of these are logical operators: forward and reverse implication, written ==> and <==, respectively, and logical equivalence and inequivalence, written <==> and <=!=>, respectively. The inclusion of symbols for logical operators is inspired by the calculational approach to formal methods [17, 20, 31]. The other symbolic operator is <:, which is used to compare types to see if they are in a subtype relationship [65]. All the other operators added to Java and available in JML’s specification expressions use a textual notation consisting of a backslash (\) followed by an English word or phrase. For example, the logical quantifiers in JML are written as \forall and \exists [55]. Besides these quantifiers, JML also has several other operators using this backslash syntax. One of the most important is \old(), which is used in method postconditions to indicate an expression whose value is taken from the pre-state of a method call. For example, \old(i-1) denotes the value of i-1 evaluated in the pre-state of a method call. This notation is borrowed from the old operator in Eiffel. Other JML expressions using the backslash syntax include \fresh(o), which says that o was not allocated in the pre-state of a method call, but is allocated (and not null) in the post-state, and \result, which denotes the normal result returned by a method.
268
G.T. Leavens et al.
The backslashes in the syntax of these operators serve a very important purpose—they prevent the rest of the operator’s name from being interpreted as a Java identifier. This allows JML to avoid reserving Java identifiers in specification expressions. For example, result can be used as a program variable and is distinguished from \result. This trick is useful in allowing JML to specify arbitrary Java programs. Indeed, because a goal of JML is to document existing code, it cannot add new reserved words to Java.
3
Background on JML
In this section we provide additional background on JML that will be useful in understanding our solutions to the remaining problems. 3.1
Semantics of Specification Expressions
Just as JML adopts much of Java’s expression syntax, it attempts to keep JML’s semantics similar to Java’s. In particular, the semantics of specification expressions is a reference semantics. That is, when the name of a variable or field is used in an expression, it denotes either a primitive value (such as an integer) or a reference to an object. References themselves are values in the semantics, which allows one to directly express aliasing or the lack of it. For example, the expression arg != fieldVal says that arg and fieldVal are not aliased. Java also allows one to compare the states of objects using the equals method. For example, in the postcondition of a clone method, one might write the following to say that the result returned by clone is a newly allocated object that has the same state as the receiver (this): \fresh(\result) && this.equals(\result); Note that the exact meaning of the equals method for a given type is left to the designer of that type, as in Java. Thus, if one only knows that o is an Object, it is hard to conclude much about x from o.equals(x). Because JML uses this reference semantics, specifiers must show the same care as Java programmers when choosing between the == and equals equality tests. And like Eiffel, but unlike Larch-style interface specification languages, JML does not need “state functions” to be applied to extract the value of an expression from a reference. Values are implicitly extracted as needed by methods and operators. Besides being easier for programmers, this lends some succinctness to the notation. Currently, JML adopts all of the Java semantics for integer arithmetic. Thus types such as int use two’s complement arithmetic and are finite. Although Java programmers are, in theory, aware of the nature of integer arithmetic, it seems that JML’s adoption of Java’s semantics causes some misunderstandings; for example, some published JML specifications are inconsistent because of this semantics [10]. Chalin has suggested adding new primitive value types for infinite precision arithmetic to JML; in particular, he suggests a type \bigint for infinite precision integers [9, 10]. He is currently implementing and experimenting with this idea.
How the Design of JML Accommodates
3.2
269
Method and Type Specifications
To explain JML’s semantics for method specifications, we use the example in Figure 1. JML uses special comments, called annotations, to hold the specification of behavior; these are added to the interface information contained in the Java code. A specifier writes these annotation comments by inserting an at-sign (@) following the usual characters that signify the start of a comment. In multi-line annotation comments, at-signs at the beginnings of lines are ignored. Figure 1 starts with a “model import” directive, which says that JML will consider all types in the named package, org.jmlspecs.models, to be imported for purposes of the specification. This allows the JML tools to find the type JMLObjectSequence (see the third line) in that package. The type JMLObjectSequence is used as the type of the model instance field, named absVal. In this declaration, the model keyword says that the field is not part of the Java code, but is used solely for purposes of specification. The instance keyword says that the field is imagined, for purposes of specification, to be a non-static field in every class that implements this interface3 . Following the declaration of the two model instance fields is an invariant. It says that the field absVal is never null. Following the invariant are the declarations and specifications of three methods. In JML, a method’s specifications are typically written, as they are in Figure 1, before the header of the method they specify. This makes the scope of the formal parameters of a method a bit strange, because it extends backward into the method’s specification. However, it works best with Java tools, which expect comments related to a method, such as javadoc comments, to precede the method’s header. Consider the specification of the first method, push. This shows the general form of a “normal behavior” specification case. A specification case includes a precondition, indicated by the keyword requires, and some other specification clauses. A specification case is satisfied if, whenever the precondition is satisfied, the other clauses are also satisfied. Additionally, in a normal behavior specification case, the method must not throw an exception when the precondition is satisfied. The specification case given for push includes, besides the requires clause, a frame axiom, introduced by the keyword assignable, and a normal postcondition, following the keyword ensures. As with specification languages in the Larch family, a precondition that is just true can be omitted. In the Larch family, an omitted frame axiom means “assignable \nothing;”, which is a very strong specification that says that the method has no side effects. Following a suggestion of Erik Poll, we decided that such a specification was too strong for a default. So in JML, an omitted frame axiom allows assignment to all locations. This agrees with most of the defaults for omitted clauses in JML, which impose no restrictions. JML also allows specifiers to write “exceptional behavior” specification cases, which say that, when the precondition is satisfied, the method must not return 3
Omitting instance makes fields static and final, which is Java’s default for fields declared in interfaces.
270
G.T. Leavens et al.
//@ model import org.jmlspecs.models.*; public interface Stack { //@ public model instance JMLObjectSequence absVal; //@ public instance invariant absVal != null; /*@ public normal_behavior @ requires true; @ assignable absVal; @ ensures absVal.equals(\old(absVal.insertFront(x))); void push(Object x);
@*/
/*@ public normal_behavior @ requires !absVal.isEmpty(); @ assignable absVal; @ ensures absVal.equals(\old(absVal.trailer())) @ && \result == \old(absVal.first()); @ also @ public exceptional_behavior @ requires absVal.isEmpty(); @ assignable \nothing; @ signals (Exception e) e instanceof IllegalStateException; @*/ Object pop(); //@ ensures \result <==> absVal.isEmpty(); /*@ pure @*/ boolean isEmpty(); } Fig. 1. The specification and code for the interface Stack.
normally but must instead throw an exception. An example appears in the specification of the pop method. This specification has two specification cases connected with also. The meaning of the also is that the method must satisfy both of these specification cases [84, 86]. Thus, when the value of the model instance field absVal is not empty, a call to pop must return normally and must satisfy the given ensures clause. But when the value of the model instance field absVal is empty, a call to pop must throw an IllegalStateException. This kind of case analysis can be desugared into a single specification case, which can be given a semantics in the usual way [38, 41, 53, 76]. The specification cases given for push and pop are heavyweight specification cases [55, Section 1]. Such specification cases are useful when one wants to give a relatively complete specification, especially for purposes of formal verification. For runtime assertion checking or documentation, one may want to specify only part of the behavior of a method. This can be done using JML’s lightweight specification cases, which are indicated by the absence of a behavior keyword
How the Design of JML Accommodates
271
(like normal behavior). Figure 1 gives an example of a lightweight specification case in the specification of the method isEmpty.
4
Dealing with Undefinedness
As discussed in Section 1.3.2, a fundamental problem in using the underlying language for specification expressions is dealing with expressions whose value is undefined. In Java, undefinedness in expressions is typically signaled by the expression throwing an exception. For example, when one divides an integer by 0, the expression throws an ArithmeticException. Exceptions may also be thrown by methods called from within specification expressions. Specification languages have adopted several different approaches to dealing with undefinedness in expressions [4,32]. We wanted a semantics that would not be surprising to either Java programmers or to those doing formal verification. Typically, a Java programmer would try to write the specification in a way that “protects” the meaning of the expression against any source of undefinedness [62]. This can be accomplished by using the short-circuit boolean operators; for example, a specifier might write denom > 0 && num/denom > 1 to be sure that the division would be defined whenever it was carried out. However, we would like specifications to be meaningful even if they are not protective. Hence, the semantics of JML does not rely on the programmer writing protective specifications but, instead, ensures that every expression has some value. To do this, we adopted the “underspecified total functions” approach favored in the calculational style of formal methods [31,32]. That is, an expression that would not have a value in Java is given an arbitrary, but unspecified, value. For example, num/0 has some integer value, although this approach does not say what the value is, only that it must be uniformly substituted in any surrounding expression. In JML all expressions have an implicit argument of the program’s state; thus, the uniform substitution of values need only be carried out within a given assertion. An advantage of this substitution approach is that it validates the rules for standard logic. For example, in JML, E1 && E2 is equivalent to E2 && E1 . Consider what happens if E1 throws an exception; in that case, one may chose some unspecified boolean value for E1 , say b. This means that E1 && E2 equals b && E2 , which is equal to E2 && b, as can be seen by a simple case analysis on E2 ’s value. The case where E2 throws an exception is similar. Furthermore, if programmers write protective specifications, they will never be surprised by the details of this semantics. The JML assertion checking compiler takes advantage of the semantics of undefinedness to attempt, as much as possible, to detect possible assertion violations [13]. That is, assertion checking attempts to use a value that will make the overall assertion false, whenever the undefinedness of some subexpression allows it to do so. In this way, the assertion checker can both follow the rules of standard logic and detect places where specifications are not sufficiently protective. This is a good example of how JML caters to the needs of both runtime assertion checking and formal verification.
272
5
G.T. Leavens et al.
Preventing Side Effects in Assertions
As discussed in Section 1.3.3, it is important to prevent side effects in assertions, for both practical and theoretical reasons. JML is designed to prevent such side effects statically. It does this using an effect-checking type system [30, 81]. This type system is designed to be as simple as possible. Although it allows specification expressions to call Java methods and constructors, it only allows such calls if the called method or constructor is declared with the modifier pure. The semantics of JML must thus assure that pure methods and constructors are side-effect free. 5.1
JML’s Purity Restrictions
JML’s semantic restrictions on pure methods and constructors are as follows: – A pure method implicitly has a specification that includes the following specification case [55, Section 2.3.1]: assignable \nothing; This ensures that a correct implementation of the method has no side effects. – “A pure constructor implicitly has a specification that only allows it to assign to the instance fields of the class in which it appears” (including inherited instance fields) [55, Section 2.3.1]. This ensures that, if the constructor is correctly implemented, then a new expression that calls it has no side effects. – Pure methods and pure constructors may only invoke other methods and constructors that are pure. This makes the type system modular, as it allows the purity of a method or a constructor to be checked based only on its code and the specifications of the other methods and constructors that it calls. – All methods and constructors that override a pure method or constructor must also be pure. This inheritance of purity is a consequence of specification inheritance and is necessary to make the type system modular in the presence of subtyping. The first restriction implies that a pure method may not perform any input or output, nor may it assign to any non-local variables. Similarly, by the second restriction, a pure constructor may not do any I/O and may not assign to non-local storage other than the instance fields of the object the constructor is initializing. Note that, in JML, saying that a method may not assign to non-local storage means precisely that—even benevolent side effects are prohibited [55, Section 2.1.3.1]. This seems necessary for sound modular reasoning [64]. It is also a useful restriction for reasoning about supertypes from their specifications [78] and for reasoning about concurrent programs. The last two restrictions are also motivated by modularity considerations. Inheritance of purity has as a consequence that a method cannot be pure if any overriding method has side effects. In particular, a method in Object can be specified as pure only if every override of that method, in any Java class, obeys JML’s purity restrictions.
How the Design of JML Accommodates
273
The type system of JML is an important advance over languages like Eiffel, which trust programmers to avoid side effects in assertions rather than statically checking this property. However, as we will see in the following subsection, JML’s purity restrictions give rise to some practical problems. 5.2
Practical Problems with JML’s Purity Restrictions
An initial practical problem is how to decide which methods in Java’s libraries should be specified as pure. One way to start to answer this question is to use a static analysis to conservatively estimate which methods in Java’s libraries have side effects. A conservative analysis could count a method as having side effects if it assigns to non-local storage or calls native methods (which may do I/O), either directly or indirectly. All other methods can safely be specified as pure, provided they are not overridden by methods that the analysis says have side effects. Researchers from Purdue have provided a list of such methods to us, using their tools from the Open Virtual Machine project4 . We hope to integrate this technology into the JML tools eventually. Declaring a method to be pure entails a very strong specification, namely that the method and all possible overriding methods have no side effects. Thus, finding that a method, and all known methods that override it, obey JML’s purity restrictions is not the same as deciding that the method should be specified as pure. Such a decision affects not just all existing overrides of the method, but all future implementations and overrides. How is one to make such a decision? This problem is particularly vexing because there are many methods that seem intuitively to be side-effect free, but that do not obey JML’s purity restrictions. Methods with benevolent side effects are common examples. A benevolent side effect is a change in the internal state of an object in a way that is not externally visible. Two examples from the protocol of Object will illustrate the importance of this problem. First, consider computing a hash code for an instance of a class. Because this may be computationally costly, an implementation may desire to compute the hash code the first time it is asked for and then cache the result in a private field of the object. When the hash code is requested on subsequent occasions, the cached result is returned without further computation. For example, this is done in the hashCode method of Java’s String class. However, in JML, storing the computed hash code into the cache is considered to be a side effect. So String’s hashCode method cannot be specified as pure. Second, consider computing object equality. In some implementations, an object’s fields might be lazily initialized or computed only on first access. If the equals method happens to be the first such method to be called on such an object, it will trigger the delayed computation. We found such an example in our work on the MultiJava compiler [15, 16]; in this compiler, the class CClassType has such delayed computations, and its override of Object’s equals method can trigger a previously delayed computation with side effects. It seems very difficult to rewrite this method to be side-effect free, because to do so one would probably 4
See http://www.ovmj.org/.
274
G.T. Leavens et al.
need to change the compiler’s architecture. (Similar kinds of lazy initialization of fields occur in implementations of the Singleton pattern, although these usually do not affect the equals method.) We have shown two cases where methods in the protocol of Object are overridden by methods that cannot be pure. By purity and specification inheritance, these examples imply that neither hashCode nor equals can be specified as pure in Object. Object is typically used in Java as the type of the elements in a collection. Hence, in the specification of a collection type, such as a hash table, one cannot use the hashCode or equals methods on elements. Without changes, this would make JML unsuitable for specifying collection types. (This problem is mostly a problem for collection types, because one can specify many subclasses of Object with pure hashCode and equals methods. Specifications operating on instances of such subclasses can use these methods without violating JML’s type system.) 5.3
Solving the Problems
The desire to use intuitively side-effect free methods in specifications, even if they are not pure according to JML’s semantics, is strong enough that we considered changing the semantics of the assignable clause in order to allow benevolent side effects. However, we do not know how to do that and still retain sound modular reasoning [64]. In any case, the use of such methods in runtime assertion checking would still be problematic because of the side effects they might cause. In addition, we would like to prevent problems when a programmer wrongly believes that side effects are benevolent; it is not clear whether an automatic static analysis could prevent such problems, and even if so, whether such a tool could be modular. Thus far, the only viable solution we have identified is to refactor specifications by adding pure model (i.e., specification-only) methods that are to be used in specifications in place of program methods that cannot be pure. That is, whenever one has an intuitively side-effect free program method, m, that is not pure according to JML’s semantics, one should create a pure model method m , which returns the same result as m but without its side effects. Then one replaces calls to m by calls to m in assertions. We are currently experimenting with this solution. The most important part of this experiment is to replace uses of Object’s equals method, which cannot be pure, with calls to a new pure model method in Object, called isEqualTo. The specifications of these methods are shown in Figure 2. The assignable clause in the specification of the equals method permits benevolent side effects; it is also specified to return the same result as would a call to isEqualTo. Thus, whenever someone overrides equals, they should also override the isEqualTo method. When an override of equals is specified as pure, then an override of isEqualTo in the same class can be specified in terms of this pure equals method, and the implementation of the model isEqualTo method can simply call equals as well. However, an implementation of equals can never call isEqualTo, because program code cannot call model methods (since model methods can only be used in specifications). Therefore, to avoid code duplication when equals is not
How the Design of JML Accommodates
275
/*@ public normal_behavior @ assignable objectState; @ ensures \result <==> this.isEqualTo(obj); @*/ public boolean equals(Object obj); /*@ public normal_behavior @ requires obj != null; @ assignable \nothing; @ ensures (* \result is true when obj is equal to this object *); @ also @ public normal_behavior @ requires obj != null && \typeof(this) == \type(Object); @ assignable \nothing; @ ensures \result <==> this == obj; @ also @ public normal_behavior @ requires obj == null; @ assignable \nothing; @ ensures \result <==> false; public pure model boolean isEqualTo(Object obj) { return this == obj; } @*/ Fig. 2. The refactored specification for Object’s equals method and the pure model method isEqualTo. The text between (* and *) in the first specification case of isEqualTo’s specification is an “informal description”, which formally is equivalent to writing true [53].
declared to be pure but the two methods share some common implementation code, one can introduce a (non-model) pure, private method that both equals and isEqualTo can call. We have also applied this refactoring to all the collection classes in java.util (and in other packages) that we had previously specified, in order to check that the solution is viable. So far the results seem satisfactory. However, as of May 2003, this restructuring is not part of the JML release, because the JML tools are not yet able to handle some of the details of this approach. In particular, the runtime assertion checker is not yet able to compile the model methods added to Object without having all of Object’s source code available. (And we cannot legally ship Sun’s source code for Object in the JML release.) However, we working on solutions to this problem that will allow us to obtain more experience with this approach and to do more case studies.
276
5.4
G.T. Leavens et al.
Future Work on Synchronized Methods and Purity
JML currently permits synchronized methods to be declared pure if they meet all the criteria described in Section 5.1. Given that obtaining a lock is a side effect that can affect control flow in a program, does allowing synchronized methods to be pure violate the intent of JML’s purity restrictions? On the surface it would seem so, because when a synchronized method gains a lock, it may change the outcome of other concurrent threads. Furthermore, execution of such a method might block, conceivably even causing a deadlock between concurrent threads that would not occur if one was not doing assertion checking. However, since we have largely ignored concurrency thus far in JML’s design, we leave resolution of this issue for future work.
6
Mathematical Libraries
As described in Section 1.3.4, we need to provide a library of mathematical concepts with JML in a way that does not overwhelm programmers, and yet is useful for formal verification. 6.1
Hiding the Mathematics
It is sometimes convenient to use mathematical concepts such as sets and sequences in specification, particularly for collection classes [36, 68, 85]. For example, the specification of Stack in Figure 1 uses the type JMLObjectSequence, which is part of JML’s org.jmlspecs.models package. This package contains types that are intended for such mathematical modeling. Besides sequences, these include sets, bags, relations, and maps, and a few other convenience types. Most types in org.jmlspecs.models have only pure methods and constructors5 . For example, JMLObjectSequence’s insertFront method returns a sequence object that is like the receiver, but with its argument placed at the front; the receiver is not changed in any way. JMLObjectSequence’s trailer method similarly returns a sequence containing all but the first element of the receiver, without changing the receiver. Because such methods are pure, they can be used during runtime assertion checking without changing the underlying computation. JML gains two advantages from having these mathematical modeling types in a Java package, as opposed to having them be purely mathematical concepts. First, these types all have Java implementations and thus can be used during runtime assertion checking. Second, using these types in assertions avoids the introduction of special mathematical notation; instead, normal Java expressions (method calls) are used to do things like concatenating sequences or intersecting sets. This is an advantage for our main audience, which consists of programmers and not mathematicians. 5
The org.jmlspecs.models package does have some types that have non-pure methods. These are various kinds of iterators and enumerators. The methods of these iterators and enumerators that have side effects cannot be used in specification expressions.
How the Design of JML Accommodates
6.2
277
Use by Theorem Provers
The second part of the mathematical libraries problem described in Section 1.3.4 is that the library of mathematical modeling types should be useful for formal verification. The types in the org.jmlspecs.models package are intended to correspond (loosely) to the libraries of mathematical concepts found in theorem provers, such as PVS. As we gain experience, we can add additional methods to these types to improve their correspondence to these mathematical concepts. It is also possible to add new packages of such types tailored to specific theorem provers or to other notations, such as OCL. When translating specification expressions into theorem prover input, the Loop tool currently treats all methods in the same way — it does not make a special case for pure methods in the org.jmlspecs.models package. This makes the resulting proof obligations more complex than is desirable. Since the types in the models package are known, it seems that one should be able, as a special case, to replace the general semantics of such a method call with a call to some specific function from the theorem prover’s library of mathematical concepts. To facilitate this, it may be that these model types should all be declared to be final, which is currently not the case.
7
Related Work
We have already discussed how JML differs from conventional formal specification languages, such as Z [80, 79, 87], VDM [6, 27, 74, 43], the Larch family [33, 48, 52, 85] and RESOLVE [22, 73]. To summarize, the main difference is that JML’s specification expressions are based on a subset of the Java programming language, a design that is more congenial to Java programmers. The Alloy Annotation Language (AAL) offers a syntax similar to JML for annotating Java programs [46]. AAL supports extensive compile-time checking based on static analysis techniques. Unlike similar static analysis tools such as ESC/Java [18], AAL also supports method calls and relational expressions in assertions. However, AAL’s assertion language is based on a simple first-order logic with relational operators [39] and not on a subset of Java expressions. We believe that a Java-based syntax is more likely to gain acceptance among Java programmers. However, JML could adopt some of AAL’s features for specifying sets of objects using regular expressions. These would be helpful in using JML’s frame axioms, where they would allow JML to more precisely describe locations that can be assigned to in the method. (Another option that would have similar benefits would be to use the approach taken in DemeterJ [67].) We have also discussed how JML differs from design by contract languages, such as Eiffel [70, 71], and tools, such as APP [77]. Summarizing, JML provides better support for complete specifications and formal verification by – extending the set of specification expressions with more expressive mathematical constructs, such as quantifiers, – ensuring that specification expressions do not contain side effects, and – providing a library of types corresponding to mathematical concepts.
278
G.T. Leavens et al.
JML’s specification-only (model) declarations and frame axioms also contribute to its ability to specify types more completely than is easily done with design by contract tools. We know of several other design by contract tools for Java [5,21,24,45,47,75]. The approaches vary from a simple assertion mechanism similar to the assert macros of C and C++ to full-fledged contract enforcement capabilities. Jass [5], iContract [47], and JContract [75] focus on the practical use of design by contract in Java. Handshake and jContractor focus on implementation techniques such as library-based on-the-fly instrumentation of contracts [21, 45]. Contract Java focuses on properly blaming contract violations [24, 25]. These notations and tools suffer from the same problems as Eiffel. That is, none of them guarantee the lack of side effects in assertions, handle undefinedness in a way that would facilitate formal verification and reasoning, support more expressive mathematical notations such as quantifiers, or provide a set of immutable types designed for use in specifications. In sum, they all focus on runtime checking, and thus it is difficult to write complete specifications for formal verification and reasoning.
8
Conclusion
JML synthesizes the best from the worlds of design by contract and more mathematical specification languages. Because of its expressive mathematical notations, its specification-only (model) declarations, and library of mathematical modeling types, one can more easily write complete specifications in JML than in a design by contract language, such as Eiffel. These more complete specifications, along with JML’s purity checking, allow JML to be useful for formal verification. Thus, JML’s synthesis of features allows it to serve many roles in the Java formal methods community. Our experience so far is that this approach has had a modest impact. Release 3.7 of JML has been downloaded almost 400 times. JML has been used in at least 5 universities for teaching some aspects of formal methods. It is used somewhat extensively in the Java Smart Card industry and has been used in at least one company outside of that industry (Fulcrum). In the future, we would like to extend the range of tools that JML supports to include tools for model checking and specification of concurrent Java programs [1]. We invite others to join us in this effort to furnish Java programmers with a single notation that can be used by many tools.
Acknowledgments The work of Leavens, Cheon, Clifton, and Ruby was supported in part by the US National Science Foundation, under grants CCR-0097907 and CCR-0113181. Thanks to Robyn Lutz, Sharon Ryan, and Janet Leavens for comments on earlier drafts of this paper. Thanks to all who have contributed to the design and implementation of JML including Al Baker, Erik Poll, Bart Jacobs, Joe Kiniry, Rustan Leino, Raymie Stata, Michael Ernst, Gary Daugherty, Arnd PoetzschHeffter, Peter M¨ uller, and others acknowledged in [55].
How the Design of JML Accommodates
279
References 1. E. Abraham-Mumm, F.S. de Boer, W.P. de Roever, , and M. Steffen. A toolsupported proof system for mutlithreaded java. In Frank de Boer, Marcello Bonsangue, Susanne Graf, and Willem-Paul de Roever, editors, FMCO 2002: Formal Methods for Component Objects, Proceedings, Lecture Notes in Computer Science. Springer-Verlag, 2003. 2. Pierre America. Inheritance and subtyping in a parallel object-oriented language. In Jean Bezivin et al., editors, ECOOP ’87, European Conference on ObjectOriented Programming, Paris, France, pages 234–242, New York, NY, June 1987. Springer-Verlag. Lecture Notes in Computer Science, volume 276. 3. Pierre America. Designing an object-oriented programming language with behavioural subtyping. In J. W. de Bakker, W. P. de Roever, and G. Rozenberg, editors, Foundations of Object-Oriented Languages, REX School/Workshop, Noordwijkerhout, The Netherlands, May/June 1990, volume 489 of Lecture Notes in Computer Science, pages 60–90. Springer-Verlag, New York, NY, 1991. 4. H. Barringer, J. H. Cheng, and C. B. Jones. A logic covering undefinedness in program proofs. Acta Informatica, 21(3):251–269, October 1984. 5. D. Bartetzko, C. Fischer, M. Moller, and H. Wehrheim. Jass - Java with assertions. In Workshop on Runtime Verification held in conjunction with the 13th Conference on Computer Aided Verification, CAV’01, 2001. Published in Electronic Notes in Theoretical Computer Science, K. Havelund and G. Rosu (eds.), 55(2), 2001. Available from www.elsevier.nl. 6. Juan Bicarregui, John S. Fitgerald, Peter A. Lindsay, Richard Moore, and Brian Ritchie. Proof in VDM: A Practitioner’s Guide. Springer-Verlag, New York, NY, 1994. 7. Grady Booch, James Rumbaugh, and Ivar Jacobson. The Unified Modeling Language User Guide. Object Technology Series. Addison Wesley Longman, Reading, Mass., 1999. 8. Chandrasekhar Boyapati, Sarfraz Khurshid, and Darko Marinov. Korat: Automated testing based on Java predicates. In Proceedings International Symposium on Software Testing and Analysis (ISSTA), pages 123–133. ACM, July 2002. 9. Patrice Chalin. Back to basics: Language support and semantics of basic infinite integer types in JML and Larch. Technical Report CU-CS 2002-003.1, Computer Science Department, Concordia University, October 2002. 10. Patrice Chalin. Improving JML: For a safer and more effective language. Technical Report 2003-001.1, Computer Science Department, Concordia University, March 2003. 11. Yoonsik Cheon and Gary T. Leavens. The Larch/Smalltalk interface specification language. ACM Transactions on Software Engineering and Methodology, 3(3):221– 253, July 1994. 12. Yoonsik Cheon and Gary T. Leavens. A quick overview of Larch/C++. Journal of Object-Oriented Programming, 7(6):39–49, October 1994. 13. Yoonsik Cheon and Gary T. Leavens. A runtime assertion checker for the Java Modeling Language (JML). In Hamid R. Arabnia and Youngsong Mun, editors, Proceedings of the International Conference on Software Engineering Research and Practice (SERP ’02), Las Vegas, Nevada, USA, June 24-27, 2002, pages 322–328. CSREA Press, June 2002.
280
G.T. Leavens et al.
14. Yoonsik Cheon and Gary T. Leavens. A simple and practical approach to unit testing: The JML and JUnit way. In Boris Magnusson, editor, ECOOP 2002 — ObjectOriented Programming, 16th European Conference, M´ aalaga, Spain, Proceedings, volume 2374 of Lecture Notes in Computer Science, pages 231–255, Berlin, June 2002. Springer-Verlag. 15. Curtis Clifton. MultiJava: Design, implementation, and evaluation of a Javacompatible language supporting modular open classes and symmetric multiple dispatch. Technical Report 01-10, Department of Computer Science, Iowa State University, Ames, Iowa, 50011, November 2001. Available from www.multijava.org. 16. Curtis Clifton, Gary T. Leavens, Craig Chambers, and Todd Millstein. MultiJava: Modular open classes and symmetric multiple dispatch for Java. In OOPSLA 2000 Conference on Object-Oriented Programming, Systems, Languages, and Applications, volume 35(10) of ACM SIGPLAN Notices, pages 130–145, New York, October 2000. ACM. 17. Edward Cohen. Programming in the 1990s: An Introduction to the Calculation of Programs. Springer-Verlag, New York, NY, 1990. 18. David L. Detlefs, K. Rustan M. Leino, Greg Nelson, and James B. Saxe. Extended static checking. SRC Research Report 159, Compaq Systems Research Center, 130 Lytton Ave., Palo Alto, Dec 1998. 19. Krishna Kishore Dhara and Gary T. Leavens. Forcing behavioral subtyping through specification inheritance. In Proceedings of the 18th International Conference on Software Engineering, Berlin, Germany, pages 258–267. IEEE Computer Society Press, March 1996. A corrected version is Iowa State University, Dept. of Computer Science TR #95-20c. 20. Edsger W. Dijkstra and Carel S. Scholten. Predicate Calculus and program semantics. Springer-Verlag, NY, 1990. 21. Andrew Duncan and Urs Holzle. Adding contracts to Java with Handshake. Technical Report TRCS98-32, Department of Computer Science, University of California, Santa Barbara, CA, December 1998. 22. Stephen H. Edwards, Wayne D. Heym, Timothy J. Long, Murali Sitaraman, and Bruce W. Weide. Part II: Specifying components in RESOLVE. ACM SIGSOFT Software Engineering Notes, 19(4):29–39, Oct 1994. 23. Michael Ernst, Jake Cockrell, William G. Griswold, and David Notkin. Dynamically discovering likely program invariants to support program evolution. IEEE Transactions on Software Engineering, 27(2):1–25, February 2001. 24. Robert Bruce Findler and Matthias Felleisen. Contract soundness for objectoriented languages. In OOPSLA ’01 Conference Proceedings, Object-Oriented Programming, Systems, Languages, and Applications, October 14-18, 2001, Tampa Bay, Florida, USA, pages 1–15, October 2001. 25. Robert Bruce Findler, Mario Latendresse, and Matthias Felleisen. Behavioral contracts and behavioral subtyping. In Proceedings of Joint 8th European Software Engineering Conference (ESEC) and 9th ACM SIGSOFT International Symposium on the Foundations of Software Engineering (FSE), September 10-14, 2001, Vienna, Austria, September 2001. 26. Kate Finney. Mathematical notation in formal specification: Too difficult for the masses? IEEE Transactions on Software Engineering, 22(2):158–159, February 1996. 27. John Fitzgerald and Peter Gorm Larsen. Modelling Systems: Practical Tools in Software Development. Cambridge, Cambridge, UK, 1998.
How the Design of JML Accommodates
281
28. Cormac Flanagan, K. Rustan M. Leino, Mark Lillibridge, Greg Nelson, James B. Saxe, and Raymie Stata. Extended static checking for Java. In Cindy Norris and Jr. James B. Fenwick, editors, Proceedings of the ACM SIGPLAN 2002 Conference on Programming Language Design and Implementation (PLDI-02), volume 37, 5 of SIGPLAN, pages 234–245, New York, June 17–19 2002. ACM Press. 29. Lisa Friendly. The design of distributed hyperlinked programming documentation. In S. Fra¨iss`e, F. Garzotto, T. Isakowitz, J. Nanard, and M. Nanard, editors, Proceedings of the International Workshop on Hypermedia Design (IWHD’95), Montpellier, France, 1–2 June 1995, pages 151–173. Springer, 1995. 30. David K. Gifford and John M. Lucassen. Integrating functional and imperative programming. In ACM Conference on LISP and Functional Programming, pages 28–38. ACM, August 1986. 31. David Gries and Fred B. Schneider. A Logical Approach to Discrete Math. Texts and Monographs in Computer Science. Springer-Verlag, New York, NY, 1994. 32. David Gries and Fred B. Schneider. Avoiding the undefined by underspecification. In Jan van Leeuwen, editor, Computer Science Today: Recent Trends and Developments, number 1000 in Lecture Notes in Computer Science, pages 366–373. Springer-Verlag, New York, NY, 1995. 33. John V. Guttag, James J. Horning, S.J. Garland, K.D. Jones, A. Modet, and J.M. Wing. Larch: Languages and Tools for Formal Specification. Springer-Verlag, New York, NY, 1993. 34. C. A. R. Hoare. An axiomatic basis for computer programming. Communications of the ACM, 12(10):576–583, October 1969. 35. C. A. R. Hoare. Notes on data structuring. In E. Dijkstra Ole-J. Dahl and C. A. R. Hoare, editors, Structured Programming, pages 83–174. Academic Press, Inc., New York, NY, 1972. 36. C. A. R. Hoare. Proof of correctness of data representations. Acta Informatica, 1(4):271–281, 1972. 37. Marieke Huisman. Reasoning about Java Programs in higher order logic with PVS and Isabelle. Ipa dissertation series, 2001-03, University of Nijmegen, Holland, February 2001. 38. Marieke Huisman and Bart Jacobs. Java program verification via a Hoare logic with abrupt termination. In T. Maibaum, editor, Fundamental Approaches to Software Engineering (FASE 2000), volume 1783 of LNCS, pages 284–303. Springer-Verlag, 2000. An earlier version is technical report CSI-R9912. 39. Daniel Jackson. Alloy: A lightweight object modeling notation. ACM Transactions on Software Engineering and Methodology, 11(2):256–290, April 2002. 40. Bart Jacobs, Joseph Kiniry, and M. Warnier. Java program verification challenges. In Frank de Boer, Marcello Bonsangue, Susanne Graf, and Willem-Paul de Roever, editors, FMCO 2002: Formal Methods for Component Objects, Proceedings, Lecture Notes in Computer Science. Springer-Verlag, 2003. 41. Bart Jacobs and Eric Poll. A logic for the Java modeling language JML. In Fundamental Approaches to Software Engineering (FASE’2001), Genova, Italy, 2001, volume 2029 of Lecture Notes in Computer Science, pages 284–299. SpringerVerlag, 2001. 42. Bart Jacobs, Joachim van den Berg, Marieke Huisman, Martijn van Berkum, Ulrich Hensel, and Hendrik Tews. Reasoning about Java classes (preliminary report). In OOPSLA ’98 Conference Proceedings, volume 33(10) of ACM SIGPLAN Notices, pages 329–340. ACM, October 1998. 43. Cliff B. Jones. Systematic Software Development Using VDM. International Series in Computer Science. Prentice Hall, Englewood Cliffs, N.J., second edition, 1990.
282
G.T. Leavens et al.
44. H. B. M. Jonkers. Upgrading the pre- and postcondition technique. In S. Prehn and W. J. Toetenel, editors, VDM ’91 Formal Software Development Methods 4th International Symposium of VDM Europe Noordwijkerhout, The Netherlands, Volume 1: Conference Contributions, volume 551 of Lecture Notes in Computer Science, pages 428–456. Springer-Verlag, New York, NY, October 1991. 45. Murat Karaorman, Urs Holzle, and John Bruno. jContractor: A reflective Java library to support design by contract. In Pierre Cointe, editor, Meta-Level Architectures and Reflection, Second International Conference on Reflection ’99, SaintMalo, France, July 19–21, 1999, Proceedings, volume 1616 of Lecture Notes in Computer Science, pages 175–196. Springer-Verlag, July 1999. 46. Sarfraz Khurshid, Darko Marinov, and Daniel Jackson. An analyzable annotation language. In Proceedings of OOPSLA ’02 Conference on Object-Oriented Programming, Languages, Systems, and Applications, volume 37(11) of SIGPLAN Notices, pages 231–245, New York, NY, November 2002. ACM. 47. Reto Kramer. iContract – the Java design by contract tool. TOOLS 26: Technology of Object-Oriented Languages and Systems, Los Alamitos, California, pages 295– 307, 1998. 48. Leslie Lamport. A simple approach to specifying concurrent systems. Communications of the ACM, 32(1):32–45, January 1989. 49. Craig Larman. Applying UML and Patterns: An Introduction to Object-Oriented Analysis and Design and the Unified Process. Prentice Hall PTR, Upper Saddle River, NJ, second edition edition, 2002. 50. Gary T. Leavens. An overview of Larch/C++: Behavioral specifications for C++ modules. In Haim Kilov and William Harvey, editors, Specification of Behavioral Semantics in Object-Oriented Information Modeling, chapter 8, pages 121–142. Kluwer Academic Publishers, Boston, 1996. An extended version is TR #96-01d, Department of Computer Science, Iowa State University, Ames, Iowa, 50011. 51. Gary T. Leavens. Larch/C++ Reference Manual. Version 5.41. Available in ftp: //ftp.cs.iastate.edu/pub/larchc++/lcpp.ps.gz or on the World Wide Web at the URL http://www.cs.iastate.edu/˜leavens/larchc++.html, April 1999. 52. Gary T. Leavens. Larch frequently asked questions. Version 1.110. Available in http://www.cs.iastate.edu/˜leavens/larch-faq.html, May 2000. 53. Gary T. Leavens and Albert L. Baker. Enhancing the pre- and postcondition technique for more expressive specifications. In Jeannette M. Wing, Jim Woodcock, and Jim Davies, editors, FM’99 — Formal Methods: World Congress on Formal Methods in the Development of Computing Systems, Toulouse, France, September 1999, Proceedings, volume 1709 of Lecture Notes in Computer Science, pages 1087– 1106. Springer-Verlag, 1999. 54. Gary T. Leavens, Albert L. Baker, and Clyde Ruby. JML: A notation for detailed design. In Haim Kilov, Bernhard Rumpe, and Ian Simmonds, editors, Behavioral Specifications of Businesses and Systems, pages 175–188. Kluwer Academic Publishers, Boston, 1999. 55. Gary T. Leavens, Albert L. Baker, and Clyde Ruby. Preliminary design of JML: A behavioral interface specification language for Java. Technical Report 98-06v, Iowa State University, Department of Computer Science, May 2003. See www.jmlspecs.org. 56. Gary T. Leavens and Yoonsik Cheon. Preliminary design of Larch/C++. In U. Martin and J. Wing, editors, Proceedings of the First International Workshop on Larch, July, 1992, Workshops in Computing, pages 159–184. Springer-Verlag, New York, NY, 1993.
How the Design of JML Accommodates
283
57. Gary T. Leavens and Krishna Kishore Dhara. Concepts of behavioral subtyping and a sketch of their extension to component-based systems. In Gary T. Leavens and Murali Sitaraman, editors, Foundations of Component-Based Systems, chapter 6, pages 113–135. Cambridge University Press, 2000. 58. Gary T. Leavens, K. Rustan M. Leino, Erik Poll, Clyde Ruby, and Bart Jacobs. JML: notations and tools supporting detailed design in Java. In OOPSLA 2000 Companion, Minneapolis, Minnesota, pages 105–106. ACM, October 2000. 59. Gary T. Leavens and Don Pigozzi. A complete algebraic characterization of behavioral subtyping. Acta Informatica, 36:617–663, 2000. 60. Gary T. Leavens and William E. Weihl. Reasoning about object-oriented programs that use subtypes (extended abstract). In N. Meyrowitz, editor, OOPSLA ECOOP ’90 Proceedings, volume 25(10) of ACM SIGPLAN Notices, pages 212–223. ACM, October 1990. 61. Gary T. Leavens and William E. Weihl. Specification and verification of objectoriented programs using supertype abstraction. Acta Informatica, 32(8):705–778, November 1995. 62. Gary T. Leavens and Jeannette M. Wing. Protective interface specifications. Formal Aspects of Computing, 10:59–75, 1998. 63. Gary Todd Leavens. Verifying object-oriented programs that use subtypes. Technical Report 439, Massachusetts Institute of Technology, Laboratory for Computer Science, February 1989. The author’s Ph.D. thesis. 64. K. Rustan M. Leino. A myth in the modular specification of programs. Technical Report KRML 63, Digital Equipment Corporation, Systems Research Center, 130 Lytton Avenue Palo Alto, CA 94301, November 1995. Obtain from the author, at [email protected]. 65. K. Rustan M. Leino, Greg Nelson, and James B. Saxe. ESC/Java user’s manual. Technical note, Compaq Systems Research Center, October 2000. 66. K. Rustan M. Leino, James B. Saxe, and Raymie Stata. Checking Java programs via guarded commands. Technical Note 1999-002, Compaq Systems Research Center, Palo Alto, CA, May 1999. 67. Karl Lieberherr, Doug Orleans, and Johan Ovlinger. Aspect-oriented programming with adaptive methods. Communications of the ACM, 44(10):39–41, October 2001. 68. Barbara Liskov and John Guttag. Abstraction and Specification in Program Development. The MIT Press, Cambridge, Mass., 1986. 69. Barbara Liskov and Jeannette Wing. A behavioral notion of subtyping. ACM Transactions on Programming Languages and Systems, 16(6):1811–1841, November 1994. 70. Bertrand Meyer. Eiffel: The Language. Object-Oriented Series. Prentice Hall, New York, NY, 1992. 71. Bertrand Meyer. Object-oriented Software Construction. Prentice Hall, New York, NY, second edition, 1997. 72. Jeremy W. Nimmer and Michael D. Ernst. Static verification of dynamically detected program invariants: Integrating Daikon and ESC/Java. In Proceedings of RV’01, First Workshop on Runtime Verification. Elsevier, July 2001. To appear in Electronic Notes in Theoretical Computer Science. 73. William F. Ogden, Murali Sitaraman, Bruce W. Weide, and Stuart H. Zweben. Part I: The RESOLVE framework and discipline — a research synopsis. ACM SIGSOFT Software Engineering Notes, 19(4):23–28, October 1994.
284
G.T. Leavens et al.
74. International Standards Organization. Information technology – programming languages, their environments and system software interfaces – Vienna Development Method – specification language – part 1: Base language. ISO/IEC 13817-1, December 1996. 75. Parasoft Corporation. Using design by contractTM to automate JavaTM software and component testing. Available from http://www.parasoft.com/jsp/ products/tech papers.jsp?product=Jcontract, as of Feb. 2003. 76. Arun D. Raghavan and Gary T. Leavens. Desugaring JML method specifications. Technical Report 00-03c, Iowa State University, Department of Computer Science, August 2001. 77. D. S. Rosenblum. Towards a method of programming with assertions. In Proceedings of the 14th International Conference on Software Engineering, pages 92–104, May 1992. 78. Clyde Ruby and Gary T. Leavens. Safely creating correct subclasses without seeing superclass code. In OOPSLA 2000 Conference on Object-Oriented Programming, Systems, Languages, and Applications, Minneapolis, Minnesota, volume 35(10) of ACM SIGPLAN Notices, pages 208–228, October 2000. 79. J. Spivey. An introduction to Z and formal specifications. Software Engineering Journal, January 1989. 80. J. Michael Spivey. The Z Notation: A Reference Manual. International Series in Computer Science. Prentice-Hall, New York, NY, 1989. ISBN 013983768X. 81. Jean-Pierre Talpin and Pierre Jouvelot. The type and effect discipline. Information and Computation, 111(2):245–296, June 1994. 82. Jos Warmer and Anneke Kleppe. The Object Constraint Language: Precise Modeling with UML. Addison Wesley Longman, Reading, Mass., 1999. 83. Jos Warmer and Anneke Kleppe. OCL: The constraint language of the UML. Journal of Object-Oriented Programming, 12(1):10–13,28, March 1999. 84. Alan Wills. Capsules and types in Fresco: Program validation in Smalltalk. In P. America, editor, ECOOP ’91: European Conference on Object Oriented Programming, volume 512 of Lecture Notes in Computer Science, pages 59–76. Springer-Verlag, New York, NY, 1991. 85. Jeannette M. Wing. Writing Larch interface language specifications. ACM Transactions on Programming Languages and Systems, 9(1):1–24, January 1987. 86. Jeannette Marie Wing. A two-tiered approach to specifying programs. Technical Report TR-299, Massachusetts Institute of Technology, Laboratory for Computer Science, 1983. 87. Jim Woodcock and Jim Davies. Using Z: Specification, Refinement, and Proof. Prentice Hall International Series in Computer Science, 1996.
Finding Implicit Contracts in .NET Components Karine Arnout1 and Bertran Meyer1,2 1 Chair
of Software Engineering, Swiss Federal Institute of Technology (ETH) CH-8092 Zurich, Switzerland 2 Eiffel Software, 356 Storke Road, Santa Barbara CA 93117, USA [email protected] http://se.inf.ethz.ch, http://www.eiffel.com
Abstract. Are contracts inherent in reusable libraries, or just one design technique among others? To help answer this question, we performed an empirical study of library classes from the .NET Collections library, which doesn’t use Design by Contract™, to look for unexpressed contracts. This article reports on the buried contracts we have found, and discusses improvements to the architecture – especially to the libraries’ ease of learning and ease of use – that may result from making the contracts explicit. It extends previous reports [3,4,5,6] with an analysis of the benefits of an a posteriori addition of contracts for the library users. Keywords: Design by Contract™, Library design, Reuse, Implicit contracts, .NET, Metadata, Contract Wizard, Eiffel.
1 Introduction Equipping libraries with contracts has become a second nature to designers working with Eiffel. Many commonly used libraries, however, don’t show any contracts at all. The resulting style is very different, and, to someone used to Design by Contract [21, 23,25,31], deficient. Because the benefits of contracts are so clear to those who use them, it’s natural to suspect that non-Eiffel programmers omit contracts because they have no good way to express them, or haven’t even been taught the concepts, but that conceptually contracts are there all the same: that inside every contract-less specification there is a contract wildly signaling to be let out. For an Eiffel programmer this is the natural interpretation. But when you are doing something different from the rest of the world, it’s good to check your own sanity. Are we wrong in seeing contracts around libraries, and the rest of the world – including the most recent general-purpose development frameworks – right in continuing to act as if contracts had never been invented? This article is such a sanity check. The basic conjecture that it explores may be stated more precisely: Resolving the Closet Contract Conjecture is interesting for several reasons: − The answer can shed light on important issues of reusable component design, one of the keys to progress in software engineering. − An answer can help library users (application programmers) choose between competing libraries. F.S. de Boer et al. (Eds.): FMCO 2002, LNCS 2852, pp. 285–318, 2003. © Springer-Verlag Berlin Heidelberg 2003
286
K. Arnout and B. Meyer Box 1. The Closet Contract Conjecture
Eiffel libraries have contracts; most others don’t. Which is the right explanation? − The contract-rich style of Eiffel libraries is but an artefact of the support for contracts in the Eiffel method, language and tools. Remove contract mechanisms, and the contracts just go away. − Contracts are inherent in library design; if not explicitly stated, as in C++/Java/.NET libraries, they are lurking anyway under the cover, either suppressed or replaced by comments in the program, explanations in the documentation, exceptions and other ersatz techniques. − On a more specific point, the answer would help ascertain the potential usefulness of a “Contract Wizard”. Such a tool, of which a first version has been implemented by Eiffel Software [1], takes advantage of the reflection facilities of .NET — “Metadata” — to let its users work interactively on compiled classes, coming from any non-contracted language such as C#, C++, Visual Basic, Cobol or Java, and add contracts to them a posteriori. But this is only interesting if the second answer holds for the Closet Contract Conjecture. If not, the Wizard’s user wouldn’t find any interesting contracts to add. − If that second answer indeed holds, we may use it to improve our understanding of the libraries, and even to improve the library themselves by turning the implicit contracts that we have elicited into explicit elements of the software. To help answer the conjecture, we have started a study of non-contracted libraries to see if we could spot implicit contracts. The .NET collection library [27], a comprehensive set of data structure implementations, has been one of the first targets. We examined some commonly used .NET collection classes, sleuthing around for hidden contracts, and trying to uncover language or documentation techniques used to make up for the absence of proper contract mechanisms such as precondition clauses, postcondition clauses and class invariants. Where we spotted closet contracts, we proceeded to out them, by producing class variants that retain the original APIs but make the contracts explicit. We compared the result both with the originals and with some of the classes’ closest counterparts in the EiffelBase library. The rest of this presentation describes the analysis and its outcomes: − Section 2 provides more details on why we have engaged in this study and explains the method of analysis. − Section 3 recalls the principles of Design by Contract and their application to library design. − Section 4 summarizes the contributions of .NET and its Collections library. − Section 5 presents the results of analyzing an important .NET collection class: ArrayList [28]. − Section 6 introduces a variant of ArrayList where the implicit contracts detected in the official version have been made explicit, and gives measurements of properties of the contracted class.
Finding Implicit Contracts in .NET Components
287
− Section 7 extends the analysis to some other classes and interfaces to assess the consistency of the results across the library. − Section 8 compares the effect of the two design styles, with and without contracts, on the development of applications using a library: ease of use, ease of learning, bug avoidance. − Section 9 presents related work about contract extraction and evaluates the possibility of automating the extraction of hidden preconditions by analyzing the CIL code. − Section 10 concludes with an assessment of the lessons learned. The focus of this work is on design and programming methodology, in particular design methodology for the construction of libraries of reusable components; we are looking for techniques that help component producers turn out better components, help component consumers learn to use the components, and reduce the potential for errors arising from incorrect, incomplete or misunderstood specifications. We were surprised, when presenting early versions, that some listeners were mostly interested in possibilities of extracting the contracts automatically from non-contracted components. Although section 9 indeed describes possibilities in this direction, building on earlier work on contract extraction, we must warn the reader not to expect any miracles; no computer tool can divine the intent behind a programmer’s work without some help from the programmer. The potential for automatic contract inference should not obscure the concrete and immediate benefits that good design methodology can bring to both the producers and consumers of reusable software.
2 The Context 2.1 A Distinctive Design Style Applying to reusable libraries the ideas of Design by Contract [21,23,25,31] means equipping each library class with precise specifications, or “contracts”, governing its interaction with the library's clients. Contracts include the class invariant, stating general consistency conditions to be maintained by every exported routine of the class, and, for every routine, preconditions stating the clients' obligations, and postconditions stating guarantees to the clients. Systematic application of these principles leads to a distinctive design style, immediately visible in Eiffel frameworks such as EiffelBase [10] [22] covering fundamental data structures and algorithms, EiffelVision for portable graphics, and others. These Eiffel libraries have been in wide use for many years and seemingly appreciated by their users as easy to learn, convenient to use, and beneficial to the reliability of applications built with them. A recent report by the Software Engineering Institute [37] confirms that for components in general — not just classes — the use of contracts appears to be a key condition of any effort to improve “composability” and scale up the application of component-based technology. Design by Contract as it has been applied to libraries so far, mostly in Eiffel, is not an a posteriori addition to the design of a library; it is an integral part of the design process. The resulting contract-rich library APIs are markedly different from more traditional, contract-less designs. One might argue that criticism of current libraries [38] becomes partly unjustified when libraries are built according to this style. The
288
K. Arnout and B. Meyer
difference is clear, for example, in a comparison of two libraries that cover some of the same ground: EiffelBase [10] [22], which is based on Design by Contract, and the .NET Collections library [27], which is not. Most non-Eiffel libraries, such as the .NET framework’s libraries, have indeed been built without explicit consideration to the notion of contract. Three possible explanations come to mind: − The library authors do not know about Design by Contract. − They know about the concepts, but don’t find them particularly useful. − They know about the concepts and find them interesting but too cumbersome to apply without built-in Eiffel-style support in the method, language and supporting tools. Regardless of the reason, the difference in styles is so stark that we must ask what happened, in these contract-less libraries, to the properties that the Eiffel designer would have expressed in preconditions, postconditions and class invariants. It’s this question that leads to the Closet Contract Conjecture: are the contracts of Eiffel libraries a figment of the Eiffel programmer’s obsession with this mechanism? Or are they present anyway, hidden, in non-Eiffel libraries as well? The only way to find out is to search contract-less libraries for closet contracts. In performing this search we have been rummaging through interface specifications, source code when available, documentation, even – since any detective knows not to overlook the household’s final output – generated code, which in .NET and Java still retains significant high-level information. 2.2 .NET Libraries and the Contract Wizard A property of the .NET libraries that makes them particularly interesting for such a study is the flexibility of the .NET component model, which has enabled the development of a “Contract Wizard” [1], a tool that enables a user to examine a compiled module (“assembly” in .NET), typically coming from a contract-less language such as C#, Visual Basic, C++, Cobol etc., and interactively add contracts to its classes and routines, producing a proxy assembly that is contracted as if it had been written in Eiffel, but calls the original. The Contract Wizard relies on the reflection capabilities provided in .NET by the metadata that every assembly includes, providing interface information such as the signature of each routine, retained from the source code in the compiling process. By nature, however, the Contract Wizard is only interesting if the Closet Contract Conjecture holds. This observation provides one of the incentives for the present study: as we consider further developments of the Contract Wizard, we must first gather empirical evidence confirming or denying its usefulness. If we found that .NET and other non-contracted libraries do very well without contracts, thank you very much, and that there are no useful closet contracts to be added, it would be a waste of time to continue working on the Contract Wizard. 2.3 Method of Work Our library analyses have so far not relied on any automatic tools. Because we are looking for something that officially isn’t there, we have to exercise our own interpretation to claim and authenticate our finds. It’s incumbent on us to state why we think a
Finding Implicit Contracts in .NET Components
289
particular class characteristic, such as an exception, is representative of an underlying contract. Having to rely on a manual extraction process puts a natural limit on future extensions of this article’s analysis to other libraries. Beyond facilitating the analysis, automated extraction tools could help users of the Contract Wizard by suggesting possible contract additions. The results of this article indeed suggest certain patterns, in code or documentation, that point to possible contracts, as certain geological patterns point to possible oil deposits. However, the final process of contract elicitation, starting from non-contracted libraries, requires subjective decisions.
3 Building Libraries with Design by Contract The ideas of Design by Contract are inspired by commercial relationships and business contracts, which formally express the rights and obligations binding a client and a supplier. Likewise, software contracts are a way to specify the roles and constraints applying to a class as a whole (class invariants) or to the routines of the class (preconditions and postconditions). 3.1 Why Use Contracts? Many programmers who have heard of contracts think they are just a way to help test and debug programs through conditionally compiled instructions of the form if not “Some condition I expect to hold here” “Scream”
then
end where “Scream” might involve triggering an exception, or stopping execution altogether. Such a use — similar to the “assert” of C — is only a small part of the application of contracts, and wouldn’t by itself justify special language constructs. Contracts address a wider range of issues in the software process, for general application development as well as library design: − Correctness: Contracts help build software right in the first place by avoiding bugs rather than correcting once they are there. Designing with contracts encourages the designer to think about the abstract properties of each software element, and build the observance of these properties into the software. − Documentation: From contracted software, automatic tools can extract documentation that is both abstract and precise. Because the information comes from the software text, this approach saves the effort of writing documentation as a separate product, and lowers the risk of divergence between software and documentation. It underlies the basic form of Eiffel documentation for Eiffel software: the contract form, produced by tools of the Eiffel environment and retaining interface information only. − Debugging and testing: Run-time monitoring of contracts permits a coherent, focused form of quality assurance based on verifying that the run-time state of the software satisfies the properties expected by the designers.
290
K. Arnout and B. Meyer
− Inheritance control: Design by Contract principles provide a coherent approach to inheritance, limiting the extent to which new routine definitions may affect the original semantics (preconditions may only be weakened, postconditions strengthened). − Management: Contracts allow project managers and decision makers to understand the global purpose of a program without going into the depth of the code. The principles are particularly relevant to library design. Eiffel libraries are thoroughly equipped with contracts stating their abstract properties, as relevant to clients. 3.2 Kinds of Contract Elements Contracts express the semantic specifications of classes and routines. They are made of assertions: boolean expressions stating individual semantic properties, such as the property, in a class representing lists stored in a container of bounded capacity, that the number count of elements in a list must not exceed the maximum permitted, capacity. Uses of contracts include: − Preconditions: Requirements under which a routine will function properly. A precondition is binding on clients (callers); the supplier (the routine) can turn it to its advantage to simplify its algorithm by assuming the precondition. − Postconditions: Properties guaranteed by the supplier to the client on routine exit. − Class invariants: Semantic constraints characterizing the integrity of instances of a class; they must be ensured by each constructor (creation procedure) and maintained by every exported routine. − Check instructions: “Assert”-like construct, often used on the client side, before a call, to check that a precondition is satisfied as expected. − Loop variants and invariants: Correctness conditions for a loop. Check instructions, loop variants and loop invariants address implementation correctness rather than properties of library interfaces and will not be considered further here. Although preconditions and postconditions are the best known forms of library contracts, class invariants are particularly important in an object-oriented context since they express fundamental properties of the abstract data type (ADT) underlying a class, and the correctness of the ADT implementation chosen for the class (representation invariant [9]). We must make sure that our contract elicitation process doesn’t overlook them. 3.3 Contracts in Libraries Even a very simple example shows the usefulness of contracts in library design. Consider a square root function specified, in a first approach, as sqrt (x: REAL): REAL This specification tells us that the function takes a REAL argument and returns a REAL result. That is already a form of contract, specifying the type signature of the
Finding Implicit Contracts in .NET Components
291
function. We can call it a signature contract. (Regrettably, some of the Java and .NET reference documentation uses the term “contract”, without qualification, for such signature contracts, creating confusion with the well established use of the term as used in the rest of this article.) A more complete contract — semantic contract if we need to distinguish it from mere signature contracts — should also specify properties of the argument and result that can’t just be captured by type information, but is just as important to the library client. The most obvious example is what happens for a negative argument, with at least four possible answers: − The function might silently return a default value, such as zero. (Not very good!) − It might return a default value, and set a special flag that it is the caller’s responsibility to examine after a call. − It might trigger an exception, which it is the caller’s responsibility to handle (otherwise execution will probably terminate abnormally). − It might produce aberrant behavior, such as entering an infinite loop or crashing the execution. (Not good in the absence of contracts.) A fifth would be to return a COMPLEX result, but that is not permitted by statically typed languages if the specification, as above, declares the type of the result as REAL. A contract — here a precondition and a postcondition — will express which of these specifications the function implements. In Eiffel the function would appear as sqrt (x: REAL): REAL is -- Mathematical square root of x, within epsilon require non_negative: x >= 0 do ... Square root algorithm here ... ensure good_approximation: abs (Result ^2 – x) <= 2 * x * epsilon end
where epsilon is some appropriate value expressing the requested precision, abs gives the absolute value, and ^ is the power operator. The assertion tags non_negative and good_approximation are there for documentation purposes and will also appear in error messages if the contracts are checked at run time during debugging and testing. The contract form of the enclosing class, as produced by the environment tools, will show the above without the do… part and the is keyword (but with the header comment). This is the basic documentation that any application programmer wishing to use an Eiffel class will receive. That documentation is, essentially, the set of contracts associated with the class. Here we find direct support for the contract clauses — require, ensure, and the yet to be encountered invariant — in the language and the associated documentation standard, but the contract is inherent to the routine, regardless of its language of implementation. If a library is to provide a usable square root routine it must be based on such a contract. Unless you know under what conditions a square root
292
K. Arnout and B. Meyer
function will operate and what properties you may expect of its result, you couldn’t use it properly. The general questions are: − If there is no explicit contract discipline, comparable to the Eiffel practice of documenting all libraries through their contract form, where will we find the implicit contract? − Will, as in this example, a contract always exist, whether expressed or not? The rest of this study provides material for answering these questions.
4 Why .NET Libraries? .NET libraries suggested themselves for our study not only because they are one of the most recent and widely publicized collections of general-purpose reusable components, but also because the innovative .NET concept of metadata equips them with specification information that appears directly useful to contract elicitation. 4.1 The Scope of .NET The .NET libraries are part of the .NET framework [24], a major Microsoft endeavor. Overall, .NET addresses the needs of companies and the general public through advances in Web services infrastructure, new tools for business-to-business and business-to-consumer interactions (Universal Description, Discovery and Integration, MyServices, Biztalk), advanced Web mechanisms (through the ASP.NET framework), new security mechanisms and other innovations. In support of these goals, .NET includes new techniques and products of direct interest to developers. The most significant advance here is extensive support for multilanguage programming with full interoperability between languages. Beyond Microsoft’s own languages, C# and Visual Basic .NET, implementations exist for thirdparty languages, in particular Eiffel. (The Eiffel implementation on .NET includes the full language as on other platforms; in particular it supports Design by Contract, genericity and multiple inheritance without restrictions [2] [36].) All such language implementations benefit from common mechanisms including a versioning scheme, an extensive security framework, a component model considerably simpler to use than Microsoft’s previous COM technology, facilities for memory management, debugging and exception handling, a multi-language development environment (Visual Studio .NET), all supported by a Common Language Runtime. They also benefit from a set of libraries covering many areas of applications, which the interoperability infrastructure makes available to programs written in all languages supported on .NET. 4.2 The Role of Metadata The basic compilation unit in .NET, covering for example a library or a section of a library, is the assembly. The key to the framework’s support for component-based development is the presence, in every assembly, of documentary information known as metadata, making the assembly self-describing in accordance with the selfdocumentation principle [23].
Finding Implicit Contracts in .NET Components
293
The metadata of an assembly — accessible to programs through the Reflection library, to users through various tools, and to the world at large in XML — provides information on the assembly’s contents, including: − A “manifest” describing the assembly name, version, culture and public key (if the assembly is signed). − The list of dependencies on other .NET assemblies. − For each class, the list of its parents (interfaces, and at most one class) and of its features (routines, attributes, properties and events). − For each class member, the signature (including arguments and return type). In addition to these predefined categories, developers can define, assuming proper source language support, their own specific kinds of metadata in the form of custom attributes. Metadata of both kinds – predefined and custom – opens attractive new possibilities. The Contract Wizard is one of them: by relying on the metadata, it enables users to examine a class and its features interactively, and add any appropriate contracts – all without having access to the source code.
5 Analysis of a Collection Class Our first search for closet contract will target the class ArrayList [28], part of the core .NET library (mscorlib.dll). This choice of class is almost arbitrary; in particular it was not based on any a priori guess that the class would suggest more (or fewer) contracts than any other. Rather, the informal criteria were that the class, describing lists implemented through arrays: − Is of obvious practical use. − Seems typical, in its style, of the Collections library. − Has a direct counterpart, ARRAYED_LIST, in the EiffelBase library, opening the possibility of comparisons once we’ve completed the contract elicitation process. 5.1 Implicit Class Invariants Documentation comments first reveal properties of ArrayList that fall into the category of class invariants. We find our first leads in the specification of class constructors, which states that “The default initial capacity for an ArrayList is 16” This comment implies that the capacity of the created object is greater than zero. Taking up this lead, we notice that all three constructors of ArrayList set the initial list’s capacity to a positive value. This suggests an invariant, since it is part of the Design by Contract rules that an invariant property must be guaranteed by all the creation procedures of a class. To test our intuition, we examine the other key property of an invariant: that it must be preserved by every exported routine of the class. Examining all such routines
294
K. Arnout and B. Meyer
confirms this and suggests that we indeed have the germ of an invariant, which in Eiffel would be expressed by the clause invariant positive_capacity: capacity >= 0
Continuing our exploration of the documentation, we note that two of the three constructors of ArrayList “initialize a new instance of the ArrayList class that is empty”. The count of elements of an array list created in such a way must then be zero. The third constructor, which takes a collection c as parameter “initializes a new instance of the ArrayList class that contains elements copied from the specified collection” So the number of elements of the new object equals the number of elements in the collection received as parameter, expressed by the assertion count = c.count (which in Eiffel would normally appear in a postcondition). Can then c.count be negative? Most likely not. Checking the documentation further reveals that the argument c passed to the constructor may denote any non-void collection, represented through one of the many classes inheriting from the ICollection interface [29]: arrayed list, sorted list, queue etc. Without performing an exhaustive examination, we note a hint in ArrayList itself, in the specification of routine Remove: “The average execution time is proportional to Count. That is, this method is an O(n) operation, where n is Count” which implies that count must always be non-negative. This evidence is enough to let us add a clause to the above invariant: positive_count: count >= 0 These first two properties are simple but already useful. For our next insights we examine the specification of class members. Documentation on the Count property reveals interesting information: “Count is always less than or equal to Capacity”. The self-assurance of this statement indicates that this property of the class always holds, suggesting that it is a class invariant. Hence a third invariant property for class ArrayList yielding the accumulated clause invariant positive_capacity: capacity >= 0 positive_count: count >= 0 valid_count: count <= capacity
Finding Implicit Contracts in .NET Components
295
5.2 Implicit Routine Preconditions Aside from implicit class invariants, the documentation also suggests preconditions. To get our clues we may look at documented exception cases. The specification of the routine Add of class ArrayList states that Add throws an exception of type NotSupportedException if the arrayed list on which it is called is read-only or has a fixed size. This suggests that the underlying implementation of Add first checks that the call target is writable (not read-only) and extendible (does not have a fixed size) before actually adding elements to the list. Such a requirement for having the method do what it is expected to is the definition of a routine precondition in terms of Design by Contract. An Eiffel specification of Add would then include the following two preconditions: require writable: not is_read_only extendible: not is_fixed_size
is_read_only and is_fixed_size are the Eiffel counterparts of the .NET properties IsReadOnly and IsFixedSize of class ArrayList. This example — one of many to be found in the reference documentation of the .NET Framework — suggests a scheme for extracting preconditions, applicable systematically, with some possibility of tool support: − Read the exception condition; e.g. the array list is read-only. − Take the opposite; for ArrayList the condition would be not is_read_only
− Infer the underlying routine precondition; here: writable: not is_read_only
5.3 Implicit Routine Postconditions
Does the .NET documentation also reveal closet postconditions? For an answer we consider the example of the query IndexOf. More precisely, since it is an overloaded method, we choose a specific version identified by its signature: public virtual int IndexOf (Object value);
The documentation explains that the return value is “the zero-based index of the first occurrence of value within the entire ArrayList, if found; otherwise, -1”. We may rephrase this specification more explicitly: − If value appears in the list, the result is the index of the first occurrence, hence greater than or equal to zero (.NET list indexes are indexed starting at zero) and less than Count, the number of elements in the list. − If value is not found, the result is –1.
296
K. Arnout and B. Meyer
Such a property is a guarantee on routine exit, a condition incumbent on the supplier on completion of the task — the definition of a postcondition. In Eiffel we would add the corresponding clause to the routine: ensure valid_index_if_found: contains (value) implies Result>0 and Result
This simple analysis suggests that routine postconditions do exist in .NET libraries, although not explicitly expressed because of the lack of support from the underlying environment. Unlike preconditions — for which it may be possible to devise supporting tools — postconditions are likely to require case-by-case human examination since they are scattered across the reference documentation. 5.4 Contracts in Interfaces Class ArrayList implements three interfaces (completely abstract specification modules) of the .NET Collections library: IList, ICollection, and IEnumerable. It is interesting to subject such interfaces to the same analysis as we have applied to the class. This analysis (see sections 6 and 7 for general statistics about the contract rates of .NET collection classes and interfaces) indicates that IList, ICollection, IEnumerable, and IEnumerator (of which IEnumerable is a client), do have routine preconditions and postconditions similar to those of ArrayList. We have not, however, found class invariants in these interfaces. This is probably because of the more limited scope of interfaces in .NET (coming from Java) as compared to “deferred classes”, their closest counterpart in the object-oriented model embodied by Eiffel. Deferred classes may have a mix of abstract and concrete features; in particular, they may include attributes. Interfaces, for their part, are purely abstract and may not contain attributes. The Eiffel policy provides a continuous spectrum from totally deferred classes, the equivalent of .NET and Java interfaces, to fully implemented classes, supporting the aims of object-oriented development with a seamless process from analysis (which typically uses deferred classes) to design and implementation (which make the classes progressively more concrete). Class invariants in the Eiffel libraries [22] often express consistency properties binding various attributes together. One can imagine, however, finding properties that hold for all the classes implementing the interface and that would be relevant candidates for “interface invariants”. But our non-exhaustive analysis of the .NET Collections library did not reveal such a case.
Finding Implicit Contracts in .NET Components
297
6 Adding Contracts a Posteriori The discovery of closet contracts in the .NET arrayed list class suggests that we should build a “contracted variant” of this class, ARRAY_LIST, that has the same interface as the original ArrayList plus the elicited contracts. We now present a sketch of this class and compare it with its EiffelBase counterpart: ARRAYED_LIST. Rather than modifying the original class we may produce the contracted variant — here in Eiffel — as a new class whose routines call those of the original. This is the only solution anyway when one doesn’t have access to the source code. The Contract Wizard is intended to support such a process, although for this discussion we have produced the result manually. 6.1 A Contracted Form of the .NET Arrayed List Class The original class, ArrayList, has 57 features (members in the .NET terminology). In presenting the contracted version we limit ourselves to 12 features, discussed in the preceding analysis of the class. The notation require -- from MY_CLASS
or ensure -- from MY_CLASS
shows that the assertion clauses that follow (respectively preconditions or postconditions) are inherited from the parent class MY_CLASS. This is the convention applied by EiffelStudio documentation tools when displaying the assertions of a class, for the parts inherited from ancestors. The difference is that in Eiffel assertions clauses are automatically inherited from parents; in .NET there is no such convention, so the .NET documentation has to repeat the same comments and exception conditions for an ancestor class or interface and all its descendants. The feature keyword introduces a “feature clause”, which groups a set of features (members) that have a common purpose. For example, the feature clause feature -- Initialization lists all the creation procedures of class ARRAY_LIST: make, make_from_capacity and make_from_collection. indexing description: "[ Implementation of a list using an array, whose size is dynamically increased as required. ]" class interface ARRAY_LIST create -- Note for non-Eiffelists: This is the list of -- creation procedures (constructors) for the -- class; the procedures’ definitions appear below.
298
K. Arnout and B. Meyer
make, make_from_capacity, make_from_collection feature -- Initialization make -- Create empty list with capacity -- Default_capacity. ensure empty: count = 0 default_capacity_set: capacity = Default_capacity writable: not is_read_only extendible: not is_fixed_size make_from_capacity (a_capacity: INTEGER) -- Create empty list with capacity a_capacity. require positive_capacity: a_capacity >= 0 ensure empty: count = 0 positive_capacity_implies_capacity_set: a_capacity > 0 implies capacity = a_capacity capacity_is_zero_implies_default_capacity_set: a_capacity =0 implies capacity =Default_capacity writable: not is_read_only extendible: not is_fixed_size make_from_collection (c: ICOLLECTION) -- Create list containing elements copied from c -- and the corresponding capacity. require collection_not_void: c /= Void ensure capacity_set: capacity = c.count count_set: count = c.count writable: not is_read_only extendible: not is_fixed_size feature -- Access capacity: INTEGER -- Number of elements the list can store count: INTEGER
Finding Implicit Contracts in .NET Components -- Number of elements in the list Default_capacity: INTEGER is 16 -- Default list capacity index_of (value: ANY): INTEGER -- Zero-based index of the first occurrence of -- value ensure -- from ILIST not_found_implies_minus_one: not contains (value) implies Result = - 1 found_implies_valid_index: contains (value) implies Result >= 0 and Result < count found_implies_correct_index: contains (value) implies item (Result) = value item (index: INTEGER): ANY -- Entry at index require -- from ILIST valid_index: index >= 0 and index < count feature -- Status report contains (an_item: ANY): BOOLEAN -- Does list contain an_item? is_fixed_size: BOOLEAN -- Has list a fixed size? is_read_only: BOOLEAN -- Is list read-only? feature -- Status setting set_capacity (value: like capacity) -- Set list capacity to value. require valid_capacity: value >= count ensure capacity_set: value > 0 implies capacity = value default_capacity_set: value = 0 implies capacity = Default_capacity feature -- Element change add (value: ANY): INTEGER -- Add value to the end of the list (double list
299
300
K. Arnout and B. Meyer -- capacity if the list is full) and return the -- index at which value has been added. require -- from ILIST writable: not is_read_only extendible: not is_fixed_size ensure -- from ILIST value_added: contains (value) updated_count: count = old count + 1 valid_index_returned: Result = count - 1 ensure then capacity_doubled: (old count = old capacity)
implies (capacity = 2 * (old capacity)) invariant positive_capacity: capacity >= 0 positive_count: count >= 0 valid_count: count <= capacity end
6.2 Metrics Fig. 1 shows measurements of properties of the contracted class ARRAY_LIST, produced with by the Metrics Tool of EiffelStudio. The measurements apply to the full class, with all 57 features, not to the abbreviated form shown above. A feature is “immediate” if it is new in the class, as opposed to a feature inherited from a parent (and possibly redefined in the class). The Eiffel metric tool’s output uses the following terminology: − Feature is the general name for class members. Features include attributes (“fields”) and routines (“methods”). A routine is a computation (algorithm) applicable to instances of the class; an attribute is stored in memory. If a routine returns a result, it is called a procedure; otherwise, it is a function. − Eiffel also distinguishes between commands and queries: A command returns no result; a query returns a result. If a query is computed at run time, it is a function; if it is stored in memory, it is an attribute. We note the following conclusions from these measurements: − 62% of the routines now have a contract (a precondition or a postcondition, usually both): 33 out of 52. − The 33 routines with preconditions tend to have more than one precondition clause: 2.5 on the average (82 total). − The 33 routines with postconditions tend to have more than one postcondition clause: 2 on average (67 total).
Finding Implicit Contracts in .NET Components
301
Fig. 1. Metrics about the contracted arrayed list class
7 Extending to Other Classes and Interfaces Equipped with our first results on ArrayList and its contracted Eiffel counterpart ARRAY_LIST, we now perform similar transformations and measurements on a few other classes and interfaces, to probe how uniform the results appear to be across the .NET Collections library. Since the assumptions and techniques are the same, we won’t repeat the details but go directly to results and interpretations. 7.1 Interfaces First, consider Eiffel deferred classes obtained by contracting the .NET interfaces from which ArrayList inherits: − ILIST, ICOLLECTION, IENUMERABLE, from which ARRAY_LIST inherits. − IENUMERATOR, of which IENUMERABLE is a client. Table 1 shows some resulting measurements. The statistics highlight three trends: − Absence of class invariant in these .NET interfaces, as already noted. − Presence of routine contracts: both preconditions and postconditions. The figures about IENUMERABLE and ICOLLECTION involve too few routines to bring valuable information — only one for class IENUMERABLE and four for ICOLLECTION. The figures about ILIST and IENUMERATOR are more significant in that respect — class ILIST has eleven routines and IENUMERATOR has six. Both classes (ILIST and IENUMERATOR) have at least one half of their routines with contracts. − Presence of multiple routine contracts: most routines have several preconditions and postconditions.
302
K. Arnout and B. Meyer Table 1. “Contract rate” of some .NET collection interfaces ILIST
ICOLLECTION
IENUMERABLE
IENUMERATOR
Routines
11
4
1
6
Routines with preconditions
7
1
0
3
Routines with postconditions
7
1
1
3
Number of preconditions
14
7
0
4
Number of postconditions
11
1
2
3
Precondition rate
64 %
25 %
0%
50 %
Postcondition rate
64 %
25 %
100 %
50 %
0
0
0
0
Class invariants
The last two points are consistent with the properties observed for class ARRAY_LIST. 7.2 Other Classes: Stack and Queue To test the generality of our first results on ArrayList, we consider two other classes of the Collections library. We choose Stack and Queue because they: − Are concrete collection classes. − Have no relation to ArrayList, except that all three implement the .NET interfaces ICollection and IEnumerable. − Have a direct counterpart in the EiffelBase library. Three classes is still only a small sample of the library. Any absolute conclusion would require exhaustive analysis, and hence a larger effort than the present study since the analysis is manual. So we have to be careful with any generalization of the results. We may note, however, that none of the three choices has been influenced by any a priori information or guess about the classes’ likelihood of including contracts. The same approach was applied to these classes as to ArrayList and its parents. Fig. 2 shows some of the resulting measurements for classes Stack and Queue. The figures confirm the trends previously identified: − Preconditions and postconditions are present. Class STACK has a 29% precondition rate (17 routines, of which 5 have preconditions) and a 59% postcondition rate (10 postcondition-equipped out of 17); class QUEUE has 42% and 58%.
Finding Implicit Contracts in .NET Components
303
Fig. 2. Metrics about the contracted stack and queue classes
− Preconditions and postconditions usually include several assertions. For example, STACK has 16 postcondition assertions for 10 contract-equipped routines. − Concrete classes have class invariants. For example, all three classes ArrayList, Stack, and Queue have an invariant clause positive_count: count >= 0
involving one attribute: count. This case-by-case analysis of 3 concrete classes and 4 interfaces of the .NET Collections library (out of 13 concrete classes and 8 interfaces) supports the second answer of the “Closet Contract Conjecture” – that contracts are inherent. We will now explore the benefits and limitations of such an a posteriori addition of contracts.
8 Effect on Library Users To appreciate the value of the results of the preceding analysis, we should assess their effect on the only constituency that matters in the end: library users – application
304
K. Arnout and B. Meyer
developers who take advantage of library classes to build their own systems. This issue is at the core of the Closet Contract Conjecture, since it determines whether we are doing any good at all by uncovering implicit contracts in contract-less libraries. By producing new versions of the library that make the contracts explicit, are we actually helping the users? To answer this question, we may examine the effect of the different styles on the library user (in terms of ease of learning and ease of use) and on the likely quality of the applications they develop. We take arrayed lists as an example and consider three variants: − The original, non-contracted class ArrayList from the .NET Collections library. − The contracted version ARRAY_LIST discussed above. − Finally, the corresponding class in the EiffelBase library, called ARRAYED_LIST, which was built with Design by Contract right from the start, rather than contracted a posteriori, and uses some other design ideas as well. 8.1 Dealing with Abnormal Cases in a Contract-Less Style The chapter in the .NET documentation devoted to class ArrayList provides a typical example of dealing with arrayed lists in that framework: using System; using System.Collections; public class SamplesArrayList { public static void Main() { // Creates and initializes a new ArrayList. ArrayList myAL = new ArrayList(); myAL.Add("Hello"); myAL.Add("World"); myAL.Add("!"); // Displays the properties and values of the // ArrayList. Console.WriteLine("myAL"); Console.WriteLine("\tCount: {0}",myAL.Count); Console.WriteLine("\tCapacity: {0}",myAL.Capacity); Console.Write ("\tValues:"); PrintValues (myAL); } public static void PrintValues (IEnumerable myList) { System.Collections.IEnumerator myEnumerator = myList.GetEnumerator(); while (myEnumerator.MoveNext()) Console.Write("\t{0}", myEnumerator.Current); Console.WriteLine(); } }
Finding Implicit Contracts in .NET Components
305
Running this C# program produces the following output: myAL Count: 3 Capacity: 16 Values: Hello
World
!
One striking point of this example is the absence of any exception handling — not even one if instruction in the class text — although our analysis of class ArrayList (see section 5) has revealed a non-trivial number of implicit contracts. For example, we have seen that the .NET method Add can only work properly if the targeted arrayed list is writable and extendible. But there is no such check in the class text above. This is likely to be on purpose since the property always holds at this point of the method execution: the .NET constructor ensures that the created list is not read-only and does not have a fixed size (see the contracted version of class ArrayList introduced in section 6), which allows calling the method Add on it. ArrayList myAL = new ArrayList(); /* Implicit check: (!myAL.IsFixedSize) && (!myAL.IsReadOnly) */ myAL.Add ("Hello"); myAL.Add ("World"); myAL.Add ("!");
If harmless in this simple example, such code may become dangerous if part of a reusable component. As a matter of fact, a novice programmer may overlook such a subtlety and reuse this code to create and add elements to a fixed-size arrayed list, which would cause the program execution to terminate on an unhandled exception of type NotSupportedException. This becomes even clearer if we encapsulate the calls to Add in a separate method FillArrayList that would look like the following: public void FillArrayList( ArrayList AL ){ AL.Add ("Hello"); AL.Add ("World"); AL.Add ("!"); }
and use FillArrayList in the Main routine: public static void Main() { ArrayList myAL = new ArrayList(); /* Implicit check: (!myAL.IsFixedSize) && (!myAL.IsReadOnly) */ FillArrayList (myAL); }
The previous program would work; the following one would not:
306
K. Arnout and B. Meyer
public static void Main() { ArrayList myAL = new ArrayList(); ArrayList.FixedSize (myAL); // The following call would throw an exception // because myAL is now a fixed-size arrayed list, // to which no element can be added. FillArrayList (myAL); }
Having Design by Contract support would be the right solution here (as discussed in the next sections). But because the .NET Common Language Runtime does not have native knowledge of contracts, .NET users have to rely on other techniques: − Using a “defensive” style of programming: checking explicitly for the routine requirements even if it can be inferred directly from the previous method statements (relying on the motto: “better check too much than too less”), hence adding redundant checking: ArrayList myAL = new ArrayList(); if ((!myAL.IsFixedSize) && (!myAL.IsReadOnly)) FillArrayList (myAL);
with: public void FillArrayList (ArrayList AL){ if ((!myAL.IsFixedSize) && (!myAL.IsReadOnly)){ AL.Add ("Hello"); AL.Add ("World"); AL.Add ("!"); } }
This style, however, leads to needless complexity by producing duplicate errorchecking code. The Design by Contract method goes in the opposite direction by avoiding redundancy and needless (Non-Redundancy principle, [20] p 343). − Relying on the exception handling mechanism of the .NET Common Language Runtime (typically, by using try…catch...finally… clauses): public static void Main() { try { // Creates and initializes a new ArrayList. ArrayList myAL = new ArrayList(); FillArrayList (myAL); // Prints list values. } catch (NotSupportedException e) { Console.WriteLine (e.Message); } }
Finding Implicit Contracts in .NET Components
307
with: public void FillArrayList (ArrayList AL) throws NotSupportedException { AL.Add ("Hello"); AL.Add ("World"); AL.Add ("!"); }
− Adding comments in the code to make implicit checks explicit and avoid misleading the library users: public static void Main() { ArrayList myAL = new ArrayList(); /* Implicit check: (!myAL.IsFixedSize) && (!myAL.IsReadOnly) */ FillArrayList (myAL); }
with: /* This method can only be called if AL does not * have a fixed size and is not read-only. */ public void FillArrayList (ArrayList AL) { AL.Add ("Hello"); AL.Add ("World"); AL.Add ("!"); }
Such an approach is efficient in the sense that there is no redundant check, thus no performance penalty, which gets closer to the ideas of Design by Contract, but it is not enforced at run time since it just relies on comments. This suggests the next approach, a posteriori contracting of classes. 8.2 Dealing with Abnormal Cases in a Contract-Rich Style A posteriori addition of contracts to a .NET component is likely to simplify the task of clients: rather than testing for a routine’s successful completion, they can just rely on the contracts, yielding to a lighter programming style (no redundant checking): indexing description: "[ Typical use of contracted class ARRAY_LIST ]"
308
K. Arnout and B. Meyer
class ARRAY_LIST_SAMPLE create make feature -- Initialization make is -- Create an arrayed list, fill it with -- Hello World!, and print its content. local my_list: ARRAY_LIST do create my_list.make fill_array_list (my_list) print_values (my_list) end feature -- Element change fill_array_list (an_array_list: ARRAY_LIST) is -- Fill an_array_list with Hello World!. require an_array_list_not_void: an_array_list /= Void is_extendible: not an_array_list.is_fixed_size is_writable: not an_array_list.is_read_only local index: INTEGER do index := an_array_list.add ("Hello ") index := an_array_list.add ("World”) index := an_array_list.add (“!") ensure array_list_filled: an_array_list.count = 3 end feature -- Output print_values (an_array_list: ARRAY_LIST) is -- Print content of an_array_list. require an_array_list_not_void: an_array_list /= Void local my_enumerator: IENUMERATOR do from my_enumerator := an_array_list.enumerator until
Finding Implicit Contracts in .NET Components
309
not my_enumerator.move_next loop print (my_enumerator.current_element) end end end
Since we know from the postconditions is_extendible and is_writable of creation procedure make of ARRAY_LIST that the preconditions of fill_array_list will be satisfied at this point of the routine execution, we do not need to add tests before calling the procedure. For readability or to facilitate debugging — when executing the software with assertion monitoring on — we might want to use an additional check instruction: create my_list.make check non_void: my_list /= Void is_extendible: not my_list.is_fixed_size is_writable: not my_list.is_read_only end fill_array_list (my_list) print_values (my_list)
although this is not required. If the creation routine make of class ARRAY_LIST had no such postconditions as is_extendible and is_writable, an explicit if control would have been needed in the client class ARRAY_LIST_SAMPLE to guarantee that the requirements of feature fill_array_list actually hold: create my_list.make if not my_list.is_fixed_size and not my_list.is_read_only then fill_array_list (my_list) end
But this is different from the “defensive” programming style used in a contract-less environment, since the test only affects the client side, not both client and supplier; the latter simply has preconditions. We call such a use of routine preconditions the a priori scheme: the client must act beforehand — before calling the routine — and ensure that the contracts are satisfied (either by testing them directly with an if control, or by relying on the postconditions of a previously called routine or on the class invariants). With this approach, any remaining run-time failure signals a design error. Such a design may not always be applicable in practice for either of three reasons: − Performance: Testing for a precondition before a routine call may be similar to the task of the routine itself, resulting in an unacceptable performance penalty. − Lack of expressiveness of the assertion languages: The notation for assertions might not be powerful enough.
310
K. Arnout and B. Meyer
− Dependency on external events: It is impossible to test for requirements if a routine involves interaction with the outside world, for example with a human user: there is no other choice than attempting to execute it, hence no way to predict abnormal cases. To address these limitations of the a priori scheme, it is possible to apply an a posteriori scheme — try the operation first and find out how it went — if a failed attempt has no irrecoverable consequences. Performance overhead – the first case above – is not a problem when the test being repeated is checking that a number is positive or a reference is not void. But the inefficiency might be more significant. An example from numerical computation [23] is a matrix equation solver: an equation of the form AX = B, where A is a matrix, and X (the unknown) and B are vectors, has a unique solution of the form X = A•¹ B only if matrix A is not singular. (A matrix is singular if one of the rows is a linear combination of others.) Applying the a priori scheme would lead the client to write code looking like the following: if a.is_singular then -- Report error. else x := a.inverse (b) end using a function inverse with precondition non_singular_matrix: inverse (b: VECTOR): VECTOR -- Solve equation of the form ax = b. require non_singular_matrix: not is_singular
This code does the job but is inefficient since determining whether a matrix is singular is essentially the same operation as solving the associated linear equation. Hence the idea of applying the a posteriori scheme; the client code would be of the form: a.invert (b) if a.inverted then x := a.inverse else -- Process erroneous case. end
Procedure invert replaces the previous function inverse. A call to this procedure (for which a more accurate name might be attempt_to_invert) sets a boolean attribute inverted to True or False to indicate whether inverting the matrix was possible, and if it was, makes the result available through attribute inverse. (A class invariant may state that inverted = (inverse /= Void).) This technique, which splits any function that may produce errors into a procedure that attempts to perform an operation and two attributes, one reporting whether the
Finding Implicit Contracts in .NET Components
311
operation was successful and the other giving access to the result of the operation if any, is compliant with the Command-Query Separation principle ([23], p 751). This example highlights one basic engineering principle for dealing with abnormal cases: whenever available, a method for preventing failures to occur is usually preferable to methods for recovering from failures. The techniques seen so far do not, however, provide a solution in three cases: − When abnormal events — such as a numerical failure or memory exhaustion — can cause the hardware or the operating system to interrupt the program execution abruptly (which is intolerable for systems with continuous availability requirements). − When abnormal situations, although not detectable through preconditions, must be diagnosed at the earliest possible time to avoid disastrous consequences — such as destroying the integrity of a database or even endangering human lives, as in an airplane control system. (One must keep in mind that such situations can appear in a contract-rich environment as well, since the support for assertions may not be rich enough to express complex properties.) − When there is a requirement for software fault tolerance, protecting against the most dramatic consequences of any remaining errors in the software. 8.3 “A Posteriori Contracting” vs. “Contracting from the Start” We have seen that clients of a .NET library are likely to benefit from an a posteriori addition of contracts: instead of having to test whether a routine successfully went to completion (with the risk of forgetting to check and getting an exception at run time), they could just rely on the contracts. What about contracting “from the start” now? Is the EiffelBase class ARRAYED_LIST more convenient to use than the contracted class ARRAY_LIST? To help answer this question, let’s consider a variant of the previous class ARRAY_LIST_SAMPLE, representing a typical client use of arrayed lists, this time using the EiffelBase ARRAYED_LIST: indexing description: "Typical use of EiffelBase ARRAYED_LIST" class ARRAYED_LIST_SAMPLE create make feature -- Initialization make is -- Create a list with two elements and print the -- list contents. local my_list: ARRAYED_LIST [STRING] do create my_list.make my_list.extend ("Hello ")
312
K. Arnout and B. Meyer my_list.extend ("World”) my_list.extend (“!") from my_list.start until my_list.after loop io.put_string (my_list.item) my_list.forth end end
end
This example highlights three characteristics of the EiffelBase ARRAYED_LIST: − A clear separation between commands and queries: the routine extend returns no result (on the contrary to the .NET feature Add, which returns an integer, yielding a useless local variable index in the ARRAY_LIST code example). − The usefulness of genericity: we know that my_list.item is of type STRING, thus we can use a more appropriate I/O feature to print it: put_string, rather than the general print. − A user-friendly interface to traverse the list through features start, after, item, and forth, relying on an internal cursor stored in class ARRAYED_LIST. Another interesting property to look at is the easiness of switching to other list implementations. As shown on Fig. 3, ARRAYED_LIST inherits from both ARRAY and DYNAMIC_LIST. It suffices to remove the relationship with class ARRAY to obtain a LINKED_LIST. The a-posteriori-contracted class ARRAY_LIST (section 6) just mapped the original .NET hierarchy, which makes extensive use of interfaces to compensate the lack of multiple inheritance (Fig. 4). Not surprisingly in light of how it was obtained, this class does not fully benefit from the power of Eiffel, in matter of design, reusability, and extendibility. Another obvious difference between ARRAY_LIST (the contracted class) and ARRAYED_LIST (the EiffelBase class) is the use of genericity in the Eiffel version. Although this falls beyond the scope of the present discussion, we may note that the lack of genericity in the current .NET object model leads to code duplication as well as to run-time check (casts) that damage both the performance and the reliability of the software. Future versions of .NET are expected to provide genericity [17] [18]. The next difference is the use of enumerators in ARRAY_LIST whereas ARRAYED_LIST stores a cursor as an attribute of the class. − In the first approach, enumerators become irrecoverably invalidated as soon as the corresponding collection changes (addition, modification, or deletion of list elements). The approach, on the other hand, allows multiple concurrent traversals of the same list through multiple cursors. − The EiffelBase approach solves the problem of invalid cursors: addition or deletion of list elements change the cursor position, and queries before and after take care of the cursor position’s validity. But the use of internal cursors requires care to avoid endless loops.
Finding Implicit Contracts in .NET Components
313
Fig. 3. Inheritance hierarchy of the EiffelBase arrayed list class
Fig. 4. Inheritance hierarchy of the .NET arrayed list class
These differences of style should not obscure the fundamental difference between a design style that has contracts from the start and one that adds them to existing contract-less library components. The examples illustrate that the right time to put in the contracts is during design. When that is not possible, for example with a library produced by someone who didn’t trouble himself with contracts, it may still be worthwhile to add them a posteriori, as a way to understand the library better, improve its documentation, and make it safer to use.
9 Automatic Extraction of Closet Contracts Our analysis of the .NET Collections library has shown interesting “patterns” about the nature and location of hidden contracts. In particular, routine preconditions tend to be buried under exception conditions. Our goal is to estimate the highest degree of automation we can achieve in extracting closet contracts from .NET libraries. We first report on the technique of dynamic contract detection and then describe our approach of inferring preconditions from the CIL code [32] of .NET assemblies.
314
K. Arnout and B. Meyer
9.1 Dynamic Contract Inference Dynamic contract inference, working from source code, seeks to deduce assertions from captured variable traces by executing the program with various inputs, relying on a set of possible assertions to deduce contracts from the execution output. The next step is to determine whether the detected assertions are meaningful and useful to the users, typically by computing a confidence probability. Ernst’s Daikon tool discovers class invariants, loop invariants and routine pre- and postconditions. Its first version [11] was limited to finding contracts over scalars and arrays; the next one (Daikon 2) [14] enables contract discovery over collections of data, and computes conditional assertions. Daikon succeeds in finding the assertions of a formally-specified program, and can even find some more, revealing deficiencies in the formal specification. Daikon also succeeds in inferring contracts from a C program, which helps developers performing changes to the C program without introducing errors [12]. It appears that the large majority of reported invariants are correct, and that Daikon extracts more invariants from high-quality programs [11]. Daikon still needs improvement in terms of: − Performance: “Invariant detection time grows approximately quadratically with the number of variables over which invariants are checked” [12]. Some experiments using incremental processing have shown promising results in improving Daikon’s performance [14]. − Relevance of the reported invariants: Daikon still reports irrelevant — meaning useless, but not necessary incorrect — invariants. Polymorphism can help increase the number of desired invariants reported to the users [14]. − Richness of inferred invariants: Currently, most cover simple properties. Ernst et al. suggest examining the techniques and algorithms used in the research fields of artificial intelligence [12] and information retrieval [13] to improve dynamic inference of invariants for applications in software evolution. The Daikon detector is not the sole tool available to dynamically extract contracts. Some Java detectors also exist; some of them do not even require the program source code to infer contracts: they can operate directly on bytecode files (*.class). 9.2 Extracting Routine Preconditions from Exception Cases “Human analysis is sometimes more powerful than either, allowing deep and insightful reasoning that is beyond hope for automation” [16]. When comparing static and dynamic techniques of program analysis, Ernst et al. admit that automatic tools fail in some cases where a human being would succeed. Does extraction of closet contracts fall into the category of processes that cannot be fully automated? If so, can we at least automate part of the effort? The analysis reported in this article has shown some regularity in the form and location of the closet contracts we can find in existing .NET components. In particular, preconditions tend to be buried in exception cases. Since method exception cases are not kept into the assembly metadata, we are currently exploring another approach: inferring routine preconditions from a systematic analysis of the CIL (Common Intermediate Language) code [32] of the .NET assemblies provided as input. More precisely, we are parsing the CIL code of .NET libraries — using Gobo Eiffel Lex and
Finding Implicit Contracts in .NET Components
315
Gobo Eiffel Yacc [9] — to list the exceptions a method or a property may throw to infer the corresponding routine preconditions. The first results are promising.
10 Conclusion This discussion has examined some evidence from the .NET libraries relevant to our basic conjecture: do existing libraries designed without a clear notion of contract contain some “contracts” anyway? This analysis provides initial support for the conjecture. The contracts are there, expressed in other forms. Preconditions find their way into exceptions; postconditions and class invariants into remarks scattered across the documentation, hence more difficult to extract automatically. The analysis reported here provides a first step in a broader research plan, which we expect to expand in the following directions: − Applying the same approach to other .NET and non-.NET libraries, such as C++ STL (a first informal look at [26] suggests that there are contracts lurking there too). − Investigating more closely the patterns that help discover each type of contract — class invariants, routine preconditions and postconditions — to facilitate the work of programmers interested in adding contracts a posteriori to existing libraries, with a view to providing an interactive tool that would support this process. − Turning the Eiffel Contract Wizard into a Web service to allow any programmers to contribute contracts to .NET components. This area of research opens up the possibility of various generalizations of this work in a broad investigation of applications of Design by Contract. (We are looking forward to seeing the evolution of the project conducted by Kevin McFarlane and aiming at providing a Design by Contract framework for use in .NET projects [19], of a current project at Microsoft Research about adding contracts into the C# language — see the “Assertions” section of [30] — and also of the new eXtensible C#© [35]; the outcome of these projects are likely to influence our research direction.)
Acknowledgements This paper takes advantage of extremely valuable comments and insights from Éric Bezault (Axa Rosenberg), Michael D. Ernst (MIT), Tony Hoare (Microsoft) and Emmanuel Stapf (Eiffel Software). Opinions expressed are of course our own. References [3] to [6] are previous versions of this work. Reference [7] is a summary version.
316
K. Arnout and B. Meyer
References 1. Karine Arnout, and Raphaël Simon. “The .NET Contract Wizard: Adding Design by Contract to languages other than Eiffel”. TOOLS 39 (39th International Conference and Exhibition on Technology of Object-Oriented Languages and Systems). IEEE Computer Society, July 2001, p 14-23. 2. Karine Arnout. “Eiffel for .NET: An Introduction”. Component Developer Magazine, September-October 2002. Available from http://www.devx.com/codemag/Article/8500. Accessed October 2002. 3. Karine Arnout, and Bertrand Meyer. “Extracting implicit contracts from .NET components”. Microsoft Research Summer Workshop 2002, Cambridge, UK, 9-11 September 2002. Available from http://se.inf.ethz.ch/publications/arnout/workshops/microsoft_summer_research_workshop_2002/contract_extraction.pdf. Accessed September 2002. 4. Karine Arnout. “Extracting Implicit Contracts from .NET Libraries”. 4th European GCSE Young Researchers Workshop 2002, in conjunction with NET.OBJECT DAYS 2002. Erfurt, Germany, 7-10 October 2002. IESE-Report No. 053.02/E, 21 October 2002, p 20-24. Available from http://www.cs.uni-essen.de/dawis/conferences/Node_YRW2002/papers/ karine_arnout_gcse_final_copy.pdf. Accessed October 2002. 5. Karine Arnout. “Extracting Implicit Contracts from .NET Libraries”. OOPSLA 2002 (17th ACM Conference on Object-Oriented Programming, Systems, Languages, and Applications), Posters. Seattle USA, 4-8 November 2002. OOPSLA'02 Companion, ACM, p 104105. 6. Karine Arnout, and Bertrand Meyer. “Contrats cachés en .NET: Mise au jour et ajout de contrats a posteriori”. LMO 2003 (Langages et Modèles à Objets). Vannes, France, 3-5 February 2003. 7. Karine Arnout, and Bertrand Meyer. “Spotting hidden contracts: the .NET example”. Submitted for publication. 8. Mike Barnett, and Wolfram Schulte. “Contracts, Components, and their Runtime Verification on the .NET Platform”. Microsoft Research Technical Report TR 2002-38, April 2002. Available from ftp://ftp.research.microsoft.com/pub/tr/tr-2002-38.pdf. Accessed April 2002. 9. Éric Bezault. Gobo Eiffel Lex and Gobo Eiffel Yacc. Retrieved September 2002 from http://www.gobosoft.com. 10. Eiffel Software Inc. EiffelBase. Retrieved October 2002 from http://docs.eiffel.com/libraries/base/index.html. 11. Michael D. Ernst. “Dynamically Detecting Likely Program Invariants”. Ph.D. dissertation, University of Washington, 2000. Available from http://pag.lcs.mit.edu/~mernst/pubs/invariants-thesis.pdf. Accessed August 2002. 12. Michael D. Ernst, Jake Cockrell, William G. Griswold, and David Notkin. “Dynamically Discovering Likely Program Invariants to Support Program Evolution”. IEEE TSE (Transactions on Software Engineering), Vol.27, No.2, February 2001, p: 1-25. Available from http://pag.lcs.mit.edu/~mernst/pubs/invariants-tse.pdf. Accessed August 2002. 13. Michael D. Ernst, Adam Czeisler, William G. Griswold, and David Notkin. “Quickly Detecting Relevant Program Invariants”. ICSE 2000 (International Conference on Software Engineering), Limerick, Ireland, 4-11 June 2000; Available from http://pag.lcs.mit.edu/~mernst/pubs/invariants-icse2000.pdf. Accessed August 2002. 14. Michael D. Ernst, William G. Griswold, Yoshio Kataoka, and David Notkin. “Dynamically Discovering Program Invariants Involving Collections”, Technical Report, University of Washington, 2000. Available from http://pag.lcs.mit.edu/~mernst/pubs/invariantspointers.pdf. Accessed August 2002. 15. C.A.R. Hoare. “Proof of Correctness of Data Representations”. Acta Infomatica, Vol. 1, 1973, p: 271-281.
Finding Implicit Contracts in .NET Components
317
16. Yoshio Kataoka, Michael D. Ernst, William G. Griswold, and David Notkin: “Automated Support for Program Refactoring using Invariants”. ICSM 2001 (International Conference on Software Maintenance), Florence, Italy, 6-10 November 2001. Available from: http://pag.lcs.mit.edu/~mernst/pubs/invariants-refactor.pdf. Accessed August 2002. 17. Andrew Kennedy, and Don Syme. “Design and Implementation of Generics for the .NET Common Language Runtime”. PLDI 2001 (Conference on Programming Language Design and Implementation). Snowbird, Utah, USA, 20-22 June 2001. Available from http://research.microsoft.com/projects/clrgen/generics.pdf. Accessed September 2002. 18. Andrew Kennedy, and Don Syme. Generics for C# and .NET CLR, September 2002. Retrieved September 2002 from http://research.microsoft.com/projects/clrgen/. 19. Kevin McFarlane. Design by Contract Framework for .Net. February 2002. Retrieved October 2002 from http://www.codeproject.com/csharp/designbycontract.asp and http://www.codeguru.com/net_general/designbycontract.html. st 20. Bertrand Meyer: Object-Oriented Software Construction (1 edition). Prentice Hall International, 1988. 21. Bertrand Meyer. “Applying ‘Design by Contract’”. Technical Report TR-EI-12/CO, Interactive Software Engineering Inc., 1986. Published in IEEE Computer, Vol. 25, No. 10, October 1992, p 40-51. Also published as “Design by Contract” in Advances in ObjectOriented Software Engineering, eds. D. Mandrioli and B. Meyer, Prentice Hall, 1991, p 150. Available from http://www.inf.ethz.ch/personal/meyer/publications/computer/contract. pdf. Accessed April 2002. 22. Bertrand Meyer: Reusable Software: The Base Object-Oriented Component Libraries. Prentice Hall, 1994. 23. Bertrand Meyer: Object-Oriented Software Construction, second edition. Prentice Hall, 1997. 24. Bertrand Meyer, Raphaël Simon, and Emmanuel Stapf: Instant .NET. Prentice Hall (in preparation). 25. Bertrand Meyer: Design by Contract. Prentice Hall (in preparation). 26. Scott Meyers: Effective STL. Addison Wesley, July 2001. 27. Microsoft. .NET Collections library. Retrieved June 2002 from http://msdn.microsoft.com/library/default.asp?url=/library/enus/cpref/html/frlrfsystemcollections.asp. 28. Microsoft. .NET ArrayList class. Retrieved June 2002 from http://msdn.microsoft.com/library/default.asp?url=/library/enus/cpref/html/frlrfsystemcollectionsarraylistclasstopic.asp. 29. Microsoft. .NET ICollection interface. Retrieved October 2002 from http://msdn.microsoft.com/library/default.asp?url=/library/enus/cpref/html/frlrfsystemcollectionsicollectionclasstopic.asp. 30. Microsoft Research. Current research, Programming Principles and Tools. Retrieved November 2002 from http://research.microsoft.com/research/ppt/. 31. Richard Mitchell, and Jim McKim: Design by Contract, by example. Addison-Wesley, 2002. 32. NET Experts. ECMA TC39 TG2 and TG3 working documents. Retrieved September 2002 from http://www.dotnetexperts.com/ecma/index.html. 33. Jeremy W. Nimmer, and Michael D. Ernst. “Invariant Inference for Static Checking: An Empirical Evaluation”. FSE ’02 (10th International Symposium on the Foundations of Software Engineering). Charleston, SC, USA. November 20-22, 2002. Proceedings of the ACM SIGSOFT. Available from http://pag.lcs.mit.edu/~mernst/pubs/esc-annotate.pdf. Accessed October 2002. 34. Jeremy W. Nimmer, and Michael D. Ernst. “Automatic generation of program specifications”. ISSTA 2002 (International Symposium on Software Testing and Analysis). Rome, Italy, 22-24 July 2002. Available from http://pag.lcs.mit.edu/~mernst/pubs/invariantsspecs.pdf. Accessed October 2002.
318
K. Arnout and B. Meyer
35. ResolveCorp. eXtensible C#© is here! Retrieved May 2003 from http://www.resolvecorp.com/products.htm. 36. Raphaël Simon, Emmanuel Stapf, and Bertrand Meyer. “Full Eiffel on .NET”. MSDN, July 2002. Available from http://msdn.microsoft.com/library/default.asp?url=/library/enus/dndotnet/html/pdc_eiffel.asp. Accessed October 2002. 37. Software Engineering Institute. “Volume II: Technical Concepts of Component-Based Software Engineering”. CMU/SEI-2000-TR-008, 2000. Available from http://www.sei.cmu.edu/publications/documents/00.reports/00tr008.html. Accessed June 2002. 38. Dave Thomas. “The Deplorable State of Class Libraries”. Journal of Object Technology (JOT), Vol.1, No.1, May-June 2002. Available from http://www.jot.fm/issues/issue_2002_05/column2. Accessed June 2002.
From Co-algebraic Specifications to Implementation: The Mihda Toolkit Gianluigi Ferrari, Ugo Montanari, Roberto Raggi, and Emilio Tuosto Dipartimento di Informatica, Universit´ a di Pisa, Italy. {giangi,ugo,raggi,etuosto}@di.unipi.it
Abstract. This paper describes the architecture of a toolkit, called Mihda, providing facilities to minimize labelled transition systems for name passing calculi. The structure of the toolkit is derived from the co-algebraic formulation of the partition-refinement minimization algorithm for HD-automata. HD-automata have been specifically designed to allocate and garbage collect names and they provide faithful finite state representations of the behaviours of π-calculus processes. The direct correspondence between the coalgebraic specification and the implementation structure facilitates the proof of correctness of the implementation. We evaluate the usefulness of Mihda in practice by performing finite state verification of π-calculus specifications.
1
Introduction
Finite state automata (e.g. labelled transition systems) provide a foundational model underlying effective verification techniques of concurrent and distributed systems. From a theoretical point of view, many behavioural properties of concurrent and distributed systems can be naturally defined directly as properties over automata. From a practical point of view, efficient algorithms and verification techniques have been developed and widely applied in practice to case studies of substantial complexity in several areas of computing such as hardware, compilers, and communication protocols. We refer to [2] for a review. A fundamental property of automata is the possibility, given an automaton, to construct its canonical form: The minimal automaton. The theoretical foundations guarantee that the minimal automaton is indistinguishable from the original one with respect to many behavioural properties (e.g., bisimilarity of automata and behavioural properties expressed in suitable modal or temporal logics). Minimal automata are very important also in practice. For instance, the problem of deciding bisimilarity is reduced to the problem of computing the minimal transition system [8]. Moreover, it is often convenient, from a computational point of view, to verify properties on the minimal automaton rather than on the original one. Indeed, minimization algorithms can be used to attack state explosion: They yield a small state space, but still retain all the relevant information for the verification.
This work has been supported by EU-FET project PROFUNDIS IST-2001-33100 and by MIUR project NAPOLI.
F.S. de Boer et al. (Eds.): FMCO 2002, LNCS 2852, pp. 319–338, 2003. c Springer-Verlag Berlin Heidelberg 2003
320
G. Ferrari et al.
Global computing systems consists of networks of stationary and mobile components. The primary features of a global computing systems is that components are autonomous, software versioning is highly dynamic, the network coverage is variable and often components reside over the nodes of the network (WEB services), membership is dynamic and often ad hoc without a centralized authority. Global computing systems must be made very robust since they are intended to operate in potentially hostile environments. Moreover, they are hard to construct correctly and very difficult to test in a controlled way. Although significant progresses have been made in providing formal models and effective verification techniques to support verification of global computing systems, current software engineering technologies provide limited solutions to some of the issues discussed above. The problem of formal verification of global computing systems still requires considerable research and dissemination efforts. History Dependent automata (HD-automata shortly) have been proposed in [14, 11, 4] as a new effective model for name passing calculi. Name passing calculi (e.g. the π-calculus [10, 9, 16]) are basically the best known and probably the most acknowledged models of mobility. Moreover, they provide a rich set of techniques for reasoning about mobile systems. Similarly to ordinary automata, HD-automata are made out of states and labelled transitions; their peculiarity resides in the fact that states and transitions are equipped with names which are no longer dealt with as syntactic components of labels, but become explicit part of the operational model. This allows one to model explicitly name creation/deallocation or name extrusion: These are the distinguished mechanisms of name passing calculi. HD-automata have been abstractly understood as automata over a permutation model, whose ingredients are a set of names and an action of its group of permutations (renaming substitutions) on an abstract set. This framework is sufficient to describe and reason about formalisms with name-binding operations. It has been incorporated into various kinds of transition systems that aim at providing syntax-free models of name-passing calculi [5, 6, 12, 15]. It is important to emphasize that names of states of HD-automata have local meaning. For instance, assume that A(x, y, z) denotes an agent having three (free) names x, y and z. Then agent A(y, x, z), obtained through the transformation which swaps names x and y, is syntactically different from A(x, y, z). However, these two agents can be semantically represented by means of a single state q of a HD-automaton simply by considering a “swapping” operation on the local names corresponding to names x and y. More generally, states that differs only for renaming of their local names are identified in HD-automata. This property allows for a very compact representation of name passing calculi. Local meaning of names requires a mechanism for describing how names correspond each other along state transitions. Graphically, we can represent such correspondences using “wires” that connect names of labels, source and target states of transitions. For instance, Figure 1 depicts a transition from source state s to destination state d. The transition exposes two names: Name 2 of s and a fresh name 0. State s has three names, 1, 2 and 3 while state d has two names
From Co-algebraic Specifications to Implementation: The Mihda Toolkit
lab 2
3
321
0 5
2 1
4 s
σ
d
Fig. 1. A HD-automaton transition
4 and 5 which correspond to the old name 1 of s and to the fresh name 0, respectively. Notice that names 3 is discharged along such state transition. HD-automata have a natural representation as coalgebras on a category of named sets and named functions. Elements of named sets are equipped with names which are defined up to groups of name permutations called symmetries [12]. General results concerning coalgebras guarantees the existence of the minimal HD-automata up to bisimilarity. In [4] two of the authors describe a declarative coalgebraic procedure to perform minimization of finite state HDautomata. In this paper, we review the coalgebraic description of the minimization algorithm for HD-automata, and we illustrate its prototype implementation. This yields a toolkit, called Mihda, providing general facilities to minimize labelled transition systems for name passing calculi. The usefulness of the Mihda toolkit will be shown by performing finite state verification of π-calculus specifications. The minimization algorithm has been specified by exploiting a type-theoretic notation rather than standard coalgebraic notations. The implementation data structures have been obtained by refining with efficiency considerations the typetheoretic formulation. The direct correspondence between the semantical structures and the implementation structures facilitates the design and the implementation of the toolkit. Moreover, it provides the formal machinery to perform the proof of correctness of the implementation. Recently, several software engineering technologies have been introduced to support a programming paradigm where the web is exploited as a service distributor. By service we do not mean a monolithic web server but rather a component available over the web that others might use to develop other services. Conceptually, web services are stand-alone components that reside over the nodes of the network. Each web service has an interface which is network accessible through standard network protocols and describes the interaction capabilities of the service. The Mihda toolkit have been designed and made available as a WEB service. By a few clicks in a browser at the URL http://jordie.di.unipi.it:8080/pweb/ the Mihda toolkit can be accessed remotely and its facilities can be evaluated directly over the WEB.
2
Preliminaries
This section sketches the main concepts on the coalgebraic representation of automata as a basis for finite state verification by semantic minimization. We
322
G. Ferrari et al.
Set idA
/ F(A) t
,A f
f ;g
+3 Set
F
F (f )
/ F(B)
B g
F(idA ) = idF(A)
F(f ; g) = F(f ); F(g)
F (g)
/ F(C)
C
Fig. 2. Functor over Set
illustrate the approach by providing the coalgebraic specification of the minimization algorithm for ordinary labelled transition systems. Hereafter, we will use terms ’automaton’ and ’labelled transition system’ interchangeably. An automaton A is a triple (S, L, →) where S is the set of states, L is the set of actions or labels and →⊆ S × L × S is the transition relation. Usually, one writes s − → d to indicate (s, , d) ∈− →; s is the source state and d is the destination or target state. Let idA denote the identity function on set A and f ; g be the composition of functions f and g (when it is defined). An endo-functor F (over category Set) is a mapping from sets to sets and from function to functions that preserves identity functions and function composition. Figure 2 gives a graphical representation of how a functor acts on sets and functions. If A is mapped on F (A) then idA is associated to idF (A) and, if f and g can be composed, then F(f ) and F (g) can be composed as well. Moreover, the image through F of the function composition f ; g is obtained by composing the images of functions f and g. Definition 1 (F -coalgebra). Let F be an endo-functor on the category Set. A F-coalgebra consists of a pair (A, α) such that α : A → F (A). The duality between F -coalgebras and F -algebras (a function F(A) → A) consists in the fact that domain and codomain are “reversed”, namely, are arrows between the same objects but with opposite directions. Different directions can be interpreted as “construction” (induction) and “observation” (coinduction). The interested reader is referred to [7, 1]. Before specifying the coalgebraic description of the minimization algorithm we introduce some notations. – Expression Q : Set denotes a set and q : Q is synonym of q ∈ Q; – Fun is the collection of functions among sets (the arrows of category Set). The function space over sets has the following structure: Fun = {H | H = S : Set , D : Set , h : S − → D}.
From Co-algebraic Specifications to Implementation: The Mihda Toolkit bij
323
inj
– h : A − → B (h : A − → B) explicitly states that function h is bijective (injective). We shall use SH , DH and hH to denote domain, codomain and mapping of an element of Fun,respectively. A similar convention will be used throughout the paper to denote components of tuples. Let H, K ∈ Fun be two functions, then the composition of H and K (H; K) is defined provided that SK = DH and it is the function such that SH;K = SH , DH;K = DK , and hH;K = hK ◦ hH . Sometimes, we shall need to work with be the function given by S = SH , D = surjective functions. Hence we let H H H {q : DH | ∃q : SH , hH (q) = q } and hH = hH . Finite-state transition systems have been coalgebraically described by employing two ingredients: A set Q, that represents the state space, together with a function K : Q − → ℘fin (L × Q) giving the transition relation; K(q) is the set of
→ q . pairs (, q ) such that q − In this paper, we shall work on a more concrete representation. In particular, we introduce a mathematical structure, called bundle, whose rˆole is to provide a declarative specification of the concrete data structure storing all the transitions out of a given state. Indeed, each bundle details which states are reachable by performing certain actions. Definition 2 (Bundles). Let L be the set of labels, then a bundle β over L is a structure D : Set, Step : ℘fin (L × D). Set D is the support of β. Given a fixed set of labels L, B L denotes the collection of bundles and β : B L indicates that β is a bundle over L. We now introduce the functor A over the universe of sets and functions. The following clauses define A: – A(Q) = {β : B L | Dβ = Q}, for each Q : Set; – For each H : Fun, A(H) is defined as follows: • SA(H) = A(SH ) and DA(H) = A(DH ); • hA(H) : β → DH , {, hH (q) | , q : Stepβ }. Definition 3 (Transition systems as coalgebras). Let L be a set of labels. Then a labelled transition system over L is a coalgebra for functor A, namely it is a function K such that DK = A(SK ). Example 1. A coalgebra K for functor A represents a transition system where SK is the set of states, and hK (q) = β, with Dβ = SK . Let us consider a finite-state automaton and its coalgebraic formulation via the mapping hK . 0> >> >> b
a
a
/1i >> >> a > b b 4 3> >> > > c c 5
) b
2
Sk , {a, 1, b, 3} hK (0) = hK (1) = Sk , {a, 2, b, 3, b, 4} hK (2) = Sk , {a, 1, b, 4} hK (3) = Sk , {c, 5} hK (4) = Sk , {c, 5} hK (5) = Sk , ∅
324
G. Ferrari et al.
Note how, for each state q ∈ {0, ..., 5}, hK (q) yields all the immediate successor states of q and the corresponding labels. In other words, (, q ) ∈ StephK (q) if,
→ q . and only if, q − General results on coalgebras ensure the existence of the final coalgebra for a large class of functors. These results apply to our formulation of labelled transition systems. In particular, it is interesting to see the result of the iteration along the terminal sequence of functor A. Let K be a transition system, and let H0 , H1 , . . . , Hi+1 , . . . be the sequence of functions computed by Hi+1 = K; A(Hi ), where H0 is the unique function from SK to the one-element set {∗} given by SH0 = SK ; DH0 = {∗}; and hH0 (q : SH0 ) = ∗. Finiteness of ℘fin ensures convergence of the iteration along the terminal sequence. We can say much more if the transition system is finite state. Indeed, if K is a finite-state transition system, then – The iteration along the terminal sequence converges in a finite number of steps, i.e. DHn+1 ≡ DHn (for some natural number n), – The isomorphism mapping F : DHn − → DHn+1 yields the minimal realization of transition system K. Comparing the co-algebraic construction with the standard algorithm [8, 3] which constructs the minimal labelled transition system we can observe: – at each iteration i the elements of DHi are the blocks of the minimization algorithm (i.e. the i-th partition). Notice that the initial approximation DH0 contains a single block: in fact H0 maps all the states of the transition system into {∗}. – at each step the algorithm creates a new partition by identifying the splitters for states q and q . This corresponds in our co-algebraic setting to the fact that Hi (q) = Hi (q ) but Hi+1 (q) = Hi+1 (q ). – the iteration proceeds until a stable partition of blocks is reached: then the iteration along the terminal sequence converges. We now apply the iteration along the terminal sequence to the coalgebraic formulation of the transition system of Example 1. The initial approximation is the function H0 defined as follows H0 = SH0 = SK , DH0 = {∗}, hH0 (q) = ∗ and
→ q } the first approximation H1 is the map hH1 : q → DH0 , {, hH0 (q ) : q − obtained by T (H0 ){1, 2, 3, 4, 5}, {a, 2, b, 3, b, 4} = {∗}, {a, ∗, b, ∗}
From Co-algebraic Specifications to Implementation: The Mihda Toolkit
325
We obtain the function hH1 and the destination state DH1 = {β1 , β2 , β3 } as detailed below. hH1 (0) hH1 (1) hH2 (2) hH1 (3) hH1 (4) hH1 (5)
= {∗}, {a, ∗, b, ∗} = {∗}, {a, ∗, b, ∗} β1 = {∗}, {a, ∗, b, ∗} = {∗}, {a, ∗, b, ∗} {∗}, {c, ∗} β2 = = {∗}, {c, ∗} β3 = ∅ = {∗}, {c, ∗} = {∗}, ∅
A further iteration yields: hH2 (0) hH2 (1) hH2 (2) hH2 (3) hH2 (4) hH2 (5)
= DH1 , {a, β1 , b, β2 } = DH1 , {a, β1 , b, β2 } = DH1 , {a, β1 , b, β2 } = DH1 , {c, ∅} = DH1 , {c, ∅} = DH1 , ∅
Since DH2 ≡ DH1 the iterative construction converges, thus providing the minimal labelled transition system depicted as
a
•M 1
b
/ •2
c
/ •3
where •1 = {0, 1, 2}, •2 = {3, 4} and •3 = {5}.
3
HD-Automata for π-Agents
This section outlines the representation of HD-automata as coalgebras over the concrete permutation algebra of named sets. Let N be an infinite countable set of names ranged over by v and let N be the set N ∪ ∗ where ∗ ∈ N is a distinguished name. The distinguished name ∗ will be used to model name creation. We also assume ta total order < on N (for instance, < can be the lexicographic order on N and ∀v ∈ N : ∗ < v). Table 1 displays the definitions of named sets, named functions, and composition of named functions. In Table 1, the general product is employed (as usual in type theory) to type functions f such that the type of f (q) is dependent on q. Intuitively, a named set represents a set of states equipped with a mechanism to give local meaning to names occurring in each state. In particular, function | | yields the number of local names of states. Moreover, the permutation group GA (q) allows one to describe directly the renamings that do not affect the behaviour of q, i.e., symmetries on the local names of q. For technical reasons, we assume that states are totally ordered. By convention we write {q : QA }A to indicate the set {v1 , ..., v|q|A } and we use NSet to denote the universe of named sets. As in the case of standard transition systems, named functions are used to determine the possible transitions of a given state. Intuitively, hH (q) yields the
326
G. Ferrari et al.
Table 1. Named sets, Named Functions and Composition of Named Functions Named set A named set A is a structure A = Q : Set, | |: Q −→ ω, ≤: ℘(Q × Q), G :
bij
℘({v1 ..v|q| } −→ {v1 ..v|q| })
q∈Q
where ∀q : QA , GA (q) is a permutation group and ≤A is a total ordering.
Named function A named function H is a structure inj
H = S : NSet, D : NSet, h : QS −→ QD , Σ : QS −→ ℘({h(q)}D −→ {q}S ) where ∀q : QSH , ∀σ : ΣH (q), 1. GDH (hH (q)); σ = ΣH (q) and 2. σ; GSH (q) ⊆ ΣH (q). Composition of named functions Named functions can be composed in the obvious way. Let H and K be named functions. Then H; K is defined only if DH = SK , and SH;K = SH ,
DH;K = DK ,
hH;K : QSH −→ QDK = hH ; hK ,
ΣH;K (q : QSH ) = ΣK (hH (q)); ΣH (q)
denotes the surjective component of H: Let H be a named function, H = {q : QDH | ∃q : QSH .hH (q) = q }, – SH = SH and QDH – |q|D = |q|DH , GD (q) = GDH (q), hH (q) = hH (q) and ΣH (q) = ΣH (q)
H
H
behaviour of state q : SH , i.e. the transitions departing from q. Since states are equipped with local names, a name correspondence is needed to describe how names in the destination state are mapped into names of the source state, therefore we must equip H with a set ΣH (q) of injective functions. However, names of corresponding states (q, hH (q)) in hH are defined up to permutation groups and name correspondence must not be “sensible” to the local meaning of names. Therefore, the whole set ΣH (q) must be generated by saturating any of its elements by the permutation group of hH (q), and the result must be invariant with respect to the permutation group of q. Condition (1) in Table 1 states that the group of hH (q) does not change the meaning of names in hH (q), while Condition (2) states that the group of q does not “generate meanings” for local names of q that are outside hH (q).
From Co-algebraic Specifications to Implementation: The Mihda Toolkit
327
Table 2. Bundles: the π-calculus case Bundles A bundle β consists of the structure β = D : NSet, Step : ℘(qd D) where qd D is the set of quadruples of the form , π, σ, q given by inj
qd D = { : Lπ , π : N −→ {v1 ..}, σ :
inj
{q}D → Q, q : QD }.
∈Lπ
and Q =
N if ∈ {BOU T, BIN } N if ∈ {BOU T, BIN }
under the constraint that GDβ (q); Sq = Sq , where Sq = {, π, σ, q ∈ Stepβ } and ρ; , π, σ, q = , π, ρ; σ, q.
Bundle names Let β be a bundle. Function {| |} : B → N , mapping each bundle to the set of its names, is defined by {| β |} =
rng(π) ∪ rng(σ) \ {∗}
,π,σ,q∈Stepβ
where rng yields the range of functions. We only consider bundles β such that {| β |} is finite and we let β to indicate the number of names which occur in the bundle β (i.e. β = |{| β |}|).
3.1
Bundles over π-Calculus Actions
To represent the minimization algorithm for the early semantics of π-calculus [10], the notion of bundle must be enriched. Labels of transitions must distinguish among the different meanings of names occurring in π-calculus actions, namely synchronization, bound/free output and bound/free input. The set of π-calculus labels Lπ is {T AU, BOU T, OU T, BIN, IN }. We specify two different labels for input actions: Label BIN is used when the input transition exposes a fresh name, while label IN handles the case of an input transition that exposes a name of the source state of the transition. Labels in Lπ have weights. The weight map | | : Lπ → {∅, {1}, {1, 2}} is defined as: |T AU | = ∅
|BOU T | = |BIN | = {1}
|OU T | = |IN | = {1, 2}
and associates the set of indexes of distinct names the label refers to. A bundle on π-labels is defined as in Table 2. Table 2 illustrates definitions of bundles and names of bundles. As it is the case for ordinary automata, the Step component of a bundle specifies the data structure that contains the set of successor states for a given source state. More precisely, if , π, σ, q ∈ qd D,
328
G. Ferrari et al.
then q is the destination state; is the label of the transition; π associates to the label the names observed in the transition; and σ states how names in the destination state are related to the names in the source state. According to the definition of σ in Table 2, a name in a destination state of a quadruple is mapped on the distinguished name ∗ only on transitions where a new name is created (i.e. transitions labelled by BOU T or BIN ). In order to exploit named functions for representing HD-automata it is necessary to equip the set of bundles B with a named set structure. In other words we must define – a total order on bundles, – a function that maps a bundle to its number of names, – a group of permutations over those names. The names of a bundle are the names (different from ∗) that appear either in the labels or in the range of σ’s of the quadruples of the bundle. Without loss a generality, we can assume that a total order on states and labels exist. Hence, quadruples are totally ordered1 . The order over quadruples yields an ordering over bundles. bij The group of β : B Lπ is the set of permutations θ : {| β |} −→ {| β |} such that β; θ = β where β; θ is defined as Dβ , {, π; θ, σ; θ, q | , π, σ, q : β}. 3.2
Normalizing Bundles
In the minimization algorithm two states belong to the same block (partition) whenever they have the “same” bundles. Hence, the most crucial construction on bundles is the normalization operation. This operation is necessary for two different reasons. The first reason is that there are different equivalent ways for picking up the step components (i.e. quadruples , π, σ, q) of a bundle. The second (more important) reason consists of removing from the step component of a bundle all the redundant input transitions. Indeed, redundant transitions occur when a HD-automaton is built from a π-calculus agent. During this phase, it is not possible to decide which free input transitions are required, and which transitions are irredundant2 . The solution to this problem consists of adding a superset of the required free input transitions and to exploit a reduction function to remove the unnecessary ones during the minimization phase. Consider for instance the case of a state q having only one name v1 and assume that the following two tuples appear in a bundle: IN, xy, {v1 → y}, q
and
BIN, x, {v1 → ∗}, q.
Then, the IN transition is redundant if y is not active in q as it expresses exactly the same behaviour of the second tuple, except that a “free” input transition 1 2
For instance, we can assume the lexicographic order of labels, states and names. In the general case, to decide whether a free input transition is required it is as difficult as to decide the bisimilarity of two π-calculus agents.
From Co-algebraic Specifications to Implementation: The Mihda Toolkit
329
is used rather than a “bound” one. Hence, the transformation removes the first tuple from the bundle. During the iterative execution of the minimization algorithm, bundles are split; hence the set of redundant transitions of bundles decreases. Hence, when the iterative construction terminates, only those free inputs that are really redundant have been removed from the bundles. The normalization of a bundle β is done in different steps. First, the bundle is reduced by removing all the possibly redundant input transitions. Reduction function red(β) on bundles is defined as follows: – Dred(β) = Dβ , – Stepred(β) = Stepβ \ {IN, xy, σ, q | BIN, x, σ , q : Stepβ ∧ σ = σ; {y → ∗}}.
where σ; {y → ∗} is the function equal to σ on any name different from y and that assigns ∗ to y. Second, the normalization function norm(β) is defined as follows: – Dnorm(β) = Dβ – Stepnorm(β) = min Stepβ \ {IN, xy, σ, q | y ∈ anβ } ,
where anβ = {| red(β) |} is the active names of β and min is the function that, when applied to Stepβ , returns the step of the minimal bundle (with respect to order ) among those obtained by permuting names of β in all possible ways. More precisely, given a bundle β, min β is the minimal bundle in the set bij
{β; θ | θ : {| β |} −→ {| β |}}, with respect to the total ordering of bundles over D. The order relation is used to define the canonical representatives of bundles and relies on the order of quadruples. Hereafter, we use perm(β) to denote the canonical permutation that associates Stepnorm(β) and Stepβ \ {IN, xy, σ, q | y ∈ anβ }. We remark that, while all IN transitions covered by BIN transitions are removed in the definition of red(β), only those corresponding to the reception of non-active names are removed in the definition of norm(β). In fact, even if an input transition is redundant, it might be the case that it corresponds to the reception of a name that is active due to some other transitions. Finally, we need a construction which extracts in a canonical way a group of permutations out of a bundle. Let β be a bundle, define Gr β to be the set {ρ | Stepβ ; (ρ[∗ /∗ ]) = Stepβ }. It can be proved that Gr β is a group of permutations. 3.3
The Minimization Algorithm
We are now ready to give the definition of the functor T that states the coalgebras for HD-automata. The action of functor T over named sets is given by: – – – –
QT (A) = {β : Bundle | Dβ = A, β normalized}, |β|T (A) = β, GT (A) (β) = Gr β, β1 ≤T (A) β2 iff Stepβ1 Stepβ2 ,
while the action of functor T over named functions is given by:
330
G. Ferrari et al.
– ST (H) = T (SH ), DT (H) = T (DH ), – hT (H) (β : QT (SH ) ) : QT (DH ) = norm(β ), : → {| norm(β ) |}{β}T (SH ) – ΣT (H)(β : QT (SH ) ) = Gr(norm(β ));(perm(β ))−1;inj − where β = DH , {, π, σ ; σ, hH (q) | , π, σ, q : Stepβ , σ : ΣH (q)}.
Notice that functor T maps every named set A into the named set T (A) of its normalized bundles. A named function H is mapped into a named function T (H) in such a way that every corresponding pair (q, hH (q)) in hH is mapped into a set of corresponding pairs (β, norm(β )) of bundles in hT (H) . The quadruples of bundle β are obtained from those of β by replacing q with hH (q) and by saturating with respect to the set of name mappings in ΣH (q). The name mappings in ΣT (H) (β) are obtained by transforming the permutation group of bundle norm(β ) with the inverse of the canonical permutation of β and with a fixed injective function inj mapping the set of names of norm(β ) into the set of names of β, defined as i < j, inj(vi ) = vi and inj(vj ) = vj implies i < j . Without bundle normalization, the choice of β among those in β ; θ would have been arbitrary and not canonical with the consequence of mapping together fewer bundles than needed. Definition 4 (Transition systems for π-agents). A transition system over named sets and π-actions is a named function K such that DK = T (SK ). HD-automata are particular transition systems over named sets. An HD- automaton A is given by: – – – –
the elements of QA are π-agents and ≤A is the lexicographic order on QA ; |p(v1 , ..., vn )|A = n; GA (q) = {id : {q}A −→ {q}A }, where id denotes the identity function, h : QA −→ {β | Dβ = A} is such that , π, σ, q ∈ Steph(q) represent the π-calculus transitions from agent q. ,π,σ
We will often use the notation q −−→ q to denote the “representative” transitions from agent q that are used in the construction of the HD-automaton. We can now define the function K. – SK = A, – hK (q) = norm(h(q)), – ΣK (q) = Gr(hK (q)); (perm(h(q)))−1 ; inj : {| h(q) |} −→ {q}A The minimal HD-automata is built by an iterative procedure on K: the iteration along the terminal sequence. The formula which details the iterative construction is given by Hi+1 = K;T (Hi ). If K is a finite state HD-automaton. Then The initial approximation, H0 , is defined as follows:
From Co-algebraic Specifications to Implementation: The Mihda Toolkit
331
Block
Domination
Bundle
Automaton
Transitions
Labels
States Fig. 3. Mihda Software Architecture
– SH0 = SK , DH0 = unit where Qunit = {∗}, |∗|unit = 0 (and hence {∗} = φ), Gunit ∗ = φ, and ∗ ≤unit ∗, – hH0 (q : QsH0 ) = ∗, – ΣH0 (q) = {φ} We recall that the iteration along the terminal sequence converges in a finite number of steps: i exists such that DHi+1 ≡ DHi , and the isomorphism mapping F : DHi → DHi+1 yields the minimal realization of the transition system K up to strong early bisimilarity.
4
The Mihda Toolkit
The previous sections outlined the coalgebraic foundation for the finite state verification of name passing process calculi. It remains to show that this theory can be effectively used as a basis for the design and development of effective and usable verification toolkits. This section and the one following explore this issue by describing our experience in designing, implementing and experimenting a minimization toolkit, called Mihda, for verifying finite state mobile systems represented in the π-calculus. The Mihda toolkit cleanly separates facilities that are language-specific (parsing, transition system calculation) from those that are independent from the calculus notation (bisimulation) in order to ease modifications. The toolkit has been implemented in ocaml. Indeed, the partition refinement algorithm has been specified in a “type-theoretic” style and the underlying type system makes use of parametric polymorphism. The type system of ocaml offers all the necessary features for handling these kind of types. Figure 3 illustrates the modules of Mihda and their dependencies.
332
G. Ferrari et al.
For instance, State is the module which provides all the structures for handling states and its main type defines the type of the states of the automata. Domination is the module containing the structures underlying bundle normalization. The connections express typing relationships among the modules. For instance, since states in bundles and transitions must have the same type, then a connection exists between modules Bundle and Transitions. Notice that the iterative construction of the minimal automaton is parameterized with respect to the modules of Figure 3. Indeed, the same algorithm can be applied to different kind of automata and bisimulation, provided that these automata match the constraints on types imposed by the software architecture. For instance, the architecture of Mihda has been exploited to provide minimization of both HD-automata and ordinary automata (up to strong bisimilarity). 4.1
The Main Cycle
We have already pointed out that the iterative step of the minimization algorithm can be represented in a functional style form as follows: ,π,σ
hHi+1 (q) = norm DHi , {, π, σ ; σ, hHi (q ) | q −−→ q , σ : ΣHi (q )}.
(1)
We compute hHi+1 (q) through the following steps: (a) determine the bundle of state q; (b) for each quadruple , π, σ, q in this bundle, apply hHi to q , the target state of the quadruple (yielding the bundle of q in the previous iteration of the algorithm); (c) left-compose symmetry σ ∈ Σ(q ) with σ; (d) normalize the resulting bundle. In the Mihda implementation the value of the i-th iteration (i.e. hHi ) is stored in a list of blocks which are the crucial data structures of Mihda. Blocks implements the action of the functor on states of the automata and contain all those information needed to compute the iteration steps of the algorithm expressed in a set theoretic framework. Blocks represent both (finite) named functions and partitions of an automaton (at each iteration of the algorithm). When the algorithm terminates, each block will correspond to a state of the minimal automaton. A block has the following structure: type Block t = Block of id : string ∗ states : State t list ∗ norm : Bundle t ∗ names : int list ∗ group : int list list ∗ Σ : (State t → (int ∗ int) list list) ∗ Θ −1 : (State t → (int ∗ int) list)
From Co-algebraic Specifications to Implementation: The Mihda Toolkit
333
q
θq
x
Fig. 4. Graphical representation of a block
Field id is the name of the block and is used to identify the block in order to construct the minimal automaton at the end of the algorithm. Field states contains the states which are considered equivalent. The remaining fields represent – – – –
the normalized bundle with respect to the block considered as state (norm), names is the list of names of the bundle in norm, group is its group, function Θ −1 , given a state q, maps the names appearing in norm into the name of q. Basically, Θ−1 (q) is the function which establishes a correspondence between the bundle of q and the bundle of the corresponding representative element in the equivalence class of the minimal automaton.
We pictorially represent (some components of) a block as in Figure 4: The upper elements are the states in the block, while the element x is the “representative state”, namely it is a graphical representation of the block as a state. For each state q a function θq maps names of x into the names of q. Function θq describes “how” the block approximates the state q at a given iteration. The circled arrow on x aims at recording that a block also has symmetries on its names. Bundle norm of block x is computed by exploiting the ordering relations over names, labels and states. A graphical representation of steps (a)-(d) above in terms of blocks is illustrated in Figure 5. Step (a) is computed by the facility Automaton.bundle that filters all transitions of the automaton whose source corresponds to q. Figure 5(a) shows that a state q is taken from a block and its bundle is computed. Step (b) is obtained by applying facility Block.next to the bundle of q. The operation Block.next substitutes all target states of the quadruples with the corresponding current block and computes the new mappings (see Figure 5(b)). Step (c) does not seem to correctly adhere to the corresponding step of equation 1. However, if we consider that θ functions are computed at each step by composing symmetries σ’s we can easily see that θ functions exactly play the rˆole of σ’s. Finally, step (d) is represented in Figure 5(d) and is obtained via the function Bundle.normalize.
334
G. Ferrari et al.
The main step of the minimization algorithm is the function split that computes, at each iteration, the current partition (the list of blocks). let split blocks block = try let minimal = (Bundle.minimize red (Block.next (h n blocks) (state of blocks) (Automaton.bundle aut (List.hd (Block.states block))))) in Some (Block.split minimal (fun q → let normal = (Bundle.normalize red (Block.next (h n blocks) (state of blocks) (Automaton.bundle aut q))) in Bisimulation.bisimilar minimal normal) block) with Failure e → None
Fig. 5. Computing hHi+1
From Co-algebraic Specifications to Implementation: The Mihda Toolkit
335
let blocks = ref [ (Block.from states states) ] in let stop = ref false in while not ( !stop ) do begin let oldblocks = !blocks in let buckets = split iter (split oldblocks) oldblocks in begin blocks := (List.map (Block.close block (h n oldblocks)) buckets); stop := (List.length !blocks) = (List.length oldblocks) && (List.for all2 (fun x y → (Block.compare x y) == 0) !blocks oldblocks) end end done ; !blocks Fig. 6. The main cycle of Mihda
Let block be a block in the list blocks, function split computes minimal by minimizing the reduced bundle of the first state of block. The choice of the state for computing minimal is not important: Without loss of generality, given two equivalent states q and q’, it is possible to map names of q into names of q’ preserving their associated normalized bundle if, and only if, a similar map from names of q’ into names of q exists. Once minimal has been computed, split invokes Block.split with parameters minimal and block, while the second argument of Block.split is a function that computes the current normalized bundle of each state in block and checks whether or not it is bisimilar to minimal. This computation is performed by function Bisimulation.bisimilar. If bisimilarity holds through θq then Some θq is returned, otherwise None is returned. We are now ready to comment on the main cycle of Mihda reported in Figure 6. Let k = (start, states, arrows) be a HD-automaton. When the algorithm starts, blocks is the list that contains a single block collecting all the states of the automata k. At each iteration, the list of blocks is splitted, as much as, possible by split iter that returns a list of buckets which have the same fields of a block apart from the name, symmetries and the functions mapping names of destination states into names of source states. Essentially, the split operation checks if two states in a block are equivalent or not. States which are no longer equivalent to the representative element of the block are removed and inserted into a bucket. Then, by means of Block.close block, all buckets are turned into blocks which are assigned to blocks. Finally, the termination condition stop is evaluated. This condition is equivalent to say that a bijection can be established between oldblocks (that corresponds to Di ) and blocks (corresponding to Di+1 ). This
336
G. Ferrari et al.
condition reduces to test whether blocks and oldblocks have the same length and that blocks at corresponding positions are equal.
5
Verifying Mobile Systems with Mihda
In this section we discuss some experimental results of Mihda in the analysis of mobile systems. In particular, we consider the π-calculus specification of the Handover Protocol for Mobile Telephones borrowed from that given in [17] (which has been in turn derived from that in [13]). The π-calculus specification of the GSM is define GSM(in,out) = (tca)(ta)(ga)(sa)(aa)(tcp)(tp)(gp)(sp)(ap) |( Car(ta,sa,out), Base(tca,ta,ga,sa,aa), IdleBase(tcp,tp,gp,sp,ap), Centre(in,tca,ta,ga,sa,aa,tcp,tp,gp,sp,ap))
Centre receives messages from the environment on channel in; these input actions are the only observable actions performed by Centre. Module Car sends the messages to the end user along the channel out; these outputs are the only visible actions performed by the Car. Modules Centre and Car interact via the base corresponding to the cell in which the car is located. The specification of modules Car, Base, IdleBase and Centre is reported in Table 3. The behaviour of the four modules is briefly summarized below: – Car brings a MobileStation and travels across two different geographical areas that provide services to end users; – Base and IdleBase are Base Station modules; they interconnect the MobileStation and the MobileSwitching Centre. – Centre is a MobileSwitching centre which controls radio communications within the whole area composed by the two cells; The protocol starts when Car moves from one cell to the other. Indeed, Centre communicates to the MobileStation the name of the base corresponding to the new cell. The communication of the new channel name to the MobileStation is performed via the current base. All the communications between the MobileSwitching centre and the MobileStation are suspended until the MobileStation receives the names of the new transmission channels. Then the base corresponding to the new cell is activated, and the communications between the MobileSwitching centre and the MobileStation continue through the new base. In Table 4 we report the results of Mihda on two different versions of the protocols. The first row of the table corresponds to the version discussed above. The second line gives the figures on a version of the GSM protocol that models the MobileSwitching and MobileStation modules in a more realistic way. Indeed, the ’full’ version exploits a protocol for establishing whether or not the car is crossing the boundary of a cell and entering the other cell.
From Co-algebraic Specifications to Implementation: The Mihda Toolkit
337
Table 3. π-calculus specification of GSM modules define Car(talk,switch,out) = talk?(msg).out!msg.Car(talk,switch,out) + switch?(t).switch?(s).Car(t,s,out) define Base(talkcentre,talkcar,give,switch,alert) = talkcentre?(msg).talkcar!msg. Base(talkcentre,talkcar,give,switch,alert) + give?(t).give?(s).switch!t.switch!s.give!give. IdleBase(talkcentre,talkcar,give,switch,alert) define IdleBase(talkcentre,talkcar,give,switch,alert) = alert?(empty).Base(talkcentre,talkcar,give,switch,alert) define Centre(in,tca,ta,ga,sa,aa,tcp,tp,gp,sp,ap) = in?(msg).tca!msg.Centre(in,tca,ta,ga,sa,aa,tcp,tp,gp,sp,ap) + tau.ga!tp.ga!sp.ga?(empty).ap!ap. Centre(in,tcp,tp,gp,sp,ap,tca,ta,ga,sa,aa)
Table 4. Mihda at work Protocol Time to compile States Transitions Time to minimize States Transitions GSM small 0m 0.931s 211 398 0m 4.193s 105 197 GSM full 0m 8.186s 964 1778 0m 54.690s 137 253
The results are obtained by running Mihda on an AMD AthlonTM XP 1800+ dual processor with 1Giga RAM. The time for minimizing the automata is very contained. The results on the GSM seem very promising. Indeed, the size of the minimal automata in terms of states and transitions is smaller than their nonminimized version. In the case of GSM small the size of the minimal automaton is the half or the automaton obtained by compiling the original specification; while, in version GSM full, states and transitions are reduced of a factor 8.
6
Conclusion
This paper has provided an overview of a foundational model for the finite state verification of global computing systems and has showed how efficient tool supports can be derived from it. We are currently extending the Mihda toolkit with facilities to handle other notions of equivalences (e.g. open bisimilarity) and other foundational calculi for global computing (e.g. the asynchronous π-calculus, the fusion calculus). To improve efficiency, we plan to incorporate software supports for symbolic approaches based on Binary Decision Diagrams.
338
G. Ferrari et al.
References 1. Peter Aczel. Algebras and coalgebras. In Roy Backhouse, Roland Crole and Jeremy Gibbons, editors, Algebraic and Coalgebraic Methods in the Mathematics of Program Construction, volume 2297 of LNCS, chapter 3, pages 79–88. Springer Verlag, April 2002. Revised Lectures of the Int. Summer School and Workshop. 2. Edmund M. Clarke and Jeanette M. Wing. Formal methods: state of the art and future directions. ACM Computing Surveys, 28(4):626–643, December 1996. 3. Jean Claude Fernandez. An implementation of an efficient algorithm for bisimulation equivalence. Science of Computer Programming, 13:219–236, 1990. 4. GianLuigi Ferrari, Ugo Montanari, and Marco Pistore. Minimizing transition systems for name passing calculi: A co-algebraic formulation. In Mogens Nielsen and Uffe Engberg, editors, FOSSACS 2002, volume LNCS 2303, pages 129–143. Springer Verlag, 2002. 5. Marcelo Fiore, Gordon G. Plotkin, and Daniele Turi. Abstract syntax and variable binding. In 14th Annual Symposium on Logic in Computer Science. IEEE Computer Society Press, 1999. 6. Murdoch J. Gabbay and Andrew M. Pitts. A new approach to abstract syntax involving binders. In 14th Annual Symposium on Logic in Computer Science. IEEE Computer Society Press, 1999. 7. Bart Jacobs and Jan Rutten. A tutorial on (co)algebras and (co)induction. Bulletin of the EATCS, 62:222–259, 1996. 8. Paris C. Kanellakis and Scott A. Smolka. Ccs expressions, finite state processes and three problem of equivalence. Information and Computation, 86(1):272–302, 1990. 9. Robin Milner. Commuticating and Mobile Systems: the π-calculus. Cambridge University Press, 1999. 10. Robin Milner, Joachim Parrow, and David Walker. A calculus of mobile processes, I and II. Information and Computation, 100(1):1–40,41–77, September 1992. 11. Ugo Montanari and Marco Pistore. History dependent automata. Technical report, Computer Science Department, Universit` a di Pisa, 1998. TR-11-98. 12. Ugo Montanari and Marco Pistore. π-calculus, structured coalgebras and minimal hd-automata. In Mathematical Foundations of Computer Science 2000, volume 1893. Springer, 2000. 13. Fredrik Orava and Joachim Parrow. An algebraic verification of a mobile network. Formal Aspects of Computing, 4(5):497–543, 1992. 14. Marco Pistore. History dependent automata. PhD thesis, Computer Science Department, Universit` a di Pisa, 1999. 15. Andrew M. Pitts and Murdoch J. Gabbay. A metalanguage for programming with bound names modulo renaming. In Mathematics of Program Construction, 5th International Conference, MPC2000, volume 1837. Springer, 2000. 16. Davide Sangiorgi and David Walker. The π-calculus: a Theory of Mobile Processes. Cambridge University Press, 2002. 17. Bj¨ orn Victor and Faron Moller. The Mobility Workbench — a tool for the πcalculus. In David Dill, editor, Proceedings of CAV ’94, volume 818 of Lecture Notes in Computer Science, pages 428–440. Springer-Verlag, 1994.
A Calculus for Modeling Software Components Oscar Nierstrasz and Franz Achermann Software Composition Group University of Bern, Switzerland http://www.iam.unibe.ch/∼scg
Abstract. Many competing definitions of software components have been proposed over the years, but still today there is only partial agreement over such basic issues as granularity (are components bigger or smaller than objects, packages, or application?), instantiation (do components exist at run-time or only at compile-time?), and state (should we distinguish between components and “instances” of components?). We adopt a minimalist view in which components can be distinguished by composable interfaces. We have identified a number of key features and mechanisms for expressing composable software, and propose a calculus for modeling components, based on the asynchronous π calculus extended with explicit namespaces, or “forms”. This calculus serves as a semantic foundation and an executable abstract machine for Piccola, an experimental composition language. The calculus also enables reasoning about compositional styles and evaluation strategies for Piccola. We present the design rationale for the Piccola calculus, and briefly outline some of the results obtained.
1
Introduction
What is a software component? What are the essential aspects of ComponentBased Software Development? What is a suitable foundation for modeling and reasoning about CBSD? To the first question, one of the most robust and appealing answers has been: “A software component is a unit of independent deployment without state.” [43] This simple definition captures much that is important, though it leaves some very important aspects implicit. First, CBSD attempts to streamline software development and evolution by separating what is stable from what is not. That is, components are not just “independently deployable”, but they must encapsulate a stable unit of functionality. This, of course, begs the question, “If components are the stable stuff, what makes up the rest?” Second, “independent deployment” of components actually entails compliance with some well-defined component model in which components present their services as a set of interfaces or “plugs”: “A software component is a static abstraction with plugs.” [31] F.S. de Boer et al. (Eds.): FMCO 2002, LNCS 2852, pp. 339–360, 2003. c Springer-Verlag Berlin Heidelberg 2003
340
O. Nierstrasz and F. Achermann
This leads us to answer the question, “What makes up the rest?” as follows: Applications = Components + Scripts [6] that is, component-based applications are (ideally) made up of stable, off-theshelf components, and scripts that plug them together. Scripts (ideally) make use of high-level connectors that coordinate the services of various components [3,29,42]. Furthermore, complex applications may need services of components that depend on very different architectural assumptions [16]. In these cases, glue code is needed to adapt components to different architectural styles [40,41]. Returning to our original questions, then, we conclude that it is not really possible to define software components without taking these complementary aspects of CBSD into account. At a purely technical level, i.e., ignoring methodological and software process aspects, these aspects include styles (plugs and connectors), scripts, coordination and glue code. A formal foundation for any reasonable notion of software components must address these aspects. We claim that most of these aspects can be adequately addressed by the notion of forms — first-class, extensible namespaces. The missing aspect (coordination) can be addressed by agents and channels. We propose, therefore, a calculus for modeling composable software which is based on the asynchronous π calculus [25,36] extended with first-class namespaces [5]. This calculus serves both as the semantic target and as an executable abstract machine for Piccola, an experimental composition language for implementing styles, scripts, coordination abstractions and glue code [4,6]. The Piccola calculus is described in greater detail in Achermann’s PhD dissertation [2]. In this paper we first motivate the calculus by establishing a set of requirements for modeling composition of software components in section 2. Next, we address these requirements by presenting the syntax and semantics of the Piccola calculus in section 3. In section 4 we provide a brief overview of Piccola, and summarize how the calculus helps us to define its semantics, reason about composition, and optimize the language bridge by partial evaluation while preserving its semantics. Finally, we conclude with a few remarks about related and ongoing work in sections 5 and 6.
2
Modeling Software Composition
As we have seen, a foundation for modeling software components must also be suitable for expressing compositional styles, scripts, coordination abstractions and glue code. Let us examine each of these in turn to see which requirements they pose. Figure 1 summarizes these requirements, and illustrates how Piccola and the Piccola calculus support them.
A Calculus for Modeling Software Components
341
Glue Piccola • extensible, immutable records • first-class, monadic services • language bridging • introspection • explicit namespaces • services as operators • dynamic scoping on demand • agents & channels
• generic wrappers • component packaging • generic adaptors
Styles • primitive neutral object model • meta-objects • HO plugs & connectors • default arguments • encapsulation • component algebras
Scripts Coordination • coordination abstractions
• sandboxes • composition expressions • context-dependent policies
Fig. 1. How Piccola supports composition.
2.1
Compositional Styles
A compositional style allows us to express the structure of a software application in terms of components, connectors and rules governing their composition (cf. “architectural style” [42]). – Neutral object model: There exists a wide variety of different object and component models. Components may also be bigger or smaller than objects. As a consequence, a general foundation for modeling components should make as few assumptions about objects, classes and inheritance as possible, namely, objects provide services, they may be instantiated, and their internal structure is hidden. – Meta-objects: On the other hand, many component models depend on runtime reflection, so it must be possible to express dynamic generation of metaobjects. – Higher-order plugs and connectors: In general, connectors can be seen as higher-order operators over components and other connectors. – Default arguments: Flexibility in plugging together components is achieved if interface dependencies are minimized. Keyword-based rather than positional arguments to services enable both flexibility and extensibility. – Encapsulation: Components are black-box entities that, like objects, provide services, without exposing their structure. At the same time, the components and connectors of a particular style can be encapsulated as a module, or namespace within which components may be scripted.
342
O. Nierstrasz and F. Achermann
Fig. 2. Evaluating the helloButton script.
– Component algebras: Compositional styles are most expressive when compositions of components and connectors again yield components (or connectors). (The composition of two filters is again a filter.) Based on these requirements, we conclude that we need (at least) records (to model objects and components), higher-order functions, reflection, and (at some level) overloading of operators. Services may be monadic, taking records as arguments, rather than polyadic. To invoke a service, we just apply it to a record which bundles together all the required arguments, and possibly some optional ones. These same records can serve as first-class namespaces which encapsulate the plugs and connectors of a given style. For this reason we unify records and namespaces, and call them “forms”, to emphasize their special role. A “form” is essentially a nested record, which binds labels to values. Consider, for example, the following JPiccola script [30]: makeFrame title = "AWT Demo" x = 200 y = 100 hello = "hello world" sayHello: println hello component = Button.new(text=hello) ? ActionPerformed sayHello This script invokes an abstraction makeFrame, passing it a form containing bindings for the labels title, x, and so on. The script makes use of a compositional style in which GUI components (i.e., the Button) can be bound to events (i.e., ActionPerformed) and actions (i.e., sayHello) by means of the ? connector. When we evaluate this code, it generates the button we see in figure 2. When we click on the button, hello world is printed on the Java console. 2.2
Glue
Glue code is needed to package, wrap or adapt code to fit into a compositional style. – Generic wrappers: Wrappers are often needed to introduce specific policies (such as thread-safe synchronization). Generic wrappers are hard to specify for general, polyadic services, but are relatively straightforward if all services are monadic.
A Calculus for Modeling Software Components
343
– Component packaging: Glue code is sometimes needed to package existing code to conform to a particular component model or style. For this purpose, a language bridge is needed to map existing language constructs to the formal component model. – Generic adaptors: Adaptation of interfaces can also be specified generically with the help of reflective or introspective features, which allow components to be inspected before they are adapted. The JPiccola helloButton script only works because Java GUI components are wrapped to fit into our compositional style. In addition to records and higher-order functions over records, we see that some form of language bridging will be needed, perhaps not at the level of the formal model, but certainly for a practical language or system based on the model. 2.3
Scripts
Scripts configure and compose components using the connectors defined for a style. – Sandboxes: For various reasons we may wish to instantiate components only in a controlled environment. We do not necessarily trust third-party components. Sometimes we would like to adapt components only within a local context. For these and other reasons it is convenient to be able to instantiate and compose namespaces which serve as sandboxes for executing scripts. – Composition expressions: Scripts instantiate and connect components. A practical language might conveniently represent connectors as operators. Pipes and filters connections are well-known, but this idea extends well to other domains. – Context-dependent policies: Very often, components must be prepared to employ services of the dynamic context. Transaction services, synchronization or communication primitives may depend on the context. For this reason, pure static scoping may not be enough, and dynamic scoping on demand will be needed for certain kinds of component models. So, we see that explicit, manipulable namespaces become more important. 2.4
Coordination
CBSD is especially relevant in concurrent and distributed contexts. For this reason, a foundation for composition must be able to express coordination of interdependent tasks. – Coordination abstractions: Both connectors and glue code may need to express coordination of concurrent activities. Consider a readers/writers synchronization policy as a generic wrapper. We conclude that we not only need higher-order functions over first-class namespaces (with introspection), but also a way of expressing concurrency and communication [40].
344
O. Nierstrasz and F. Achermann Table 1. Syntax of the Piccola Calculus. A, B, C ::= | | | | | |
A; B → x L λx.A νc.A c?
empty form sandbox bind inspect abstraction restriction input
| | | | | | |
F, G, H ::= →F | x
empty form binding
|S service | F · G extension
S ::= F ; λx.A closure → | x bind | c output
3
R x hide x A·B AB A|B c
current root variable hide extension application parallel output
|L inspect | hide x hide
The Piccola Calculus
As a consequence of the requirements we have identified above, we propose as a foundation a process calculus based on the higher-order asynchronous π calculus [25,36] in which tuple-based communication is replaced by communication of extensible records, or forms [5]. Furthermore, forms serve as first-class namespaces and support a simple kind of introspection. The design of the Piccola calculus strikes a balance between minimalism and expressiveness. As a calculus it is rather large. In fact, it would be possible to express everything we want with the π calculus alone, but the semantic gap between concepts we wish to model and the terms of the calculus would be rather large. With the Piccola calculus we are aiming for the smallest calculus with which we can conveniently express components, connectors and scripts. 3.1
Syntax
The Piccola calculus is given by agents A, B, C that range over the set of agents A in 1. There are two categories of identifiers: labels and channels. The set of labels L is ranged over by x, y, z. (We use the term “variables” and “labels” interchangeably.) Specific labels are also written in the italic text font. Channels are denoted by a, b, c, d ∈ N . Labels are bound with bindings and λ-abstractions, and channels are bound by ν-restrictions. The operators have the following precedence: application > extension > restriction, abstraction > sandbox > parallel Agent expressions are reduced to static form values or simply forms. Forms are ranged over by F, G, H. Notice that the set of forms is a subset of all agents. Forms are the first-class citizens of the Piccola calculus, i.e., they
A Calculus for Modeling Software Components
345
are the values that get communicated between agents and are used to invoke services. Forms are sets of bindings and services. The set of forms is denoted by F. Certain forms play the role of services. We use S to range over services. User-defined services are closures. Primitive services are inspect, the bind and hide primitives, and the output service. Before considering the formal reduction relation, we first give an informal description of the different agent expressions and how they reduce. – The empty form, , does not reduce further. It denotes a form without any binding. – The current root agent, R, denotes the current lexical scope. – A sandbox A; B evaluates the agent B in the root context given by A. A binds all free labels in B. If B is a label x, we say that A; x is a projection on x in A. – A label, x, denotes the value bound by x in the current root context. →A – The primitive service bind creates bindings. If A reduces to F , then x →F . reduces to the binding x → · y →) reduces – The primitive service hide x removes bindings. So, hide x (x →. to y – The inspect service, L, can be used to interate over the bindings and services of an arbitrary form F . The result of LF is a service that takes as its argument a form that binds the labels isEmpty, isService and isLabel to services. One of these three services will then be selected, depending on whether F is , contains some bindings, or is only a service. – The values of two agents are concatenated by extension. In the value of A · B the bindings of B override those for the same label in A. – An abstraction λx.A abstracts x in A. – The application AB denotes the result of applying A to B. Piccola uses a call-by-value reduction order. In order to reduce AB, A must reduce to a service and B to a form. – The expression νc.A restricts the visibility of the channel name c to the agent expression A, as in the π calculus. – A | B spawns off the agent A asynchronously and yields the value of B. Unlike in the π calculus, the parallel composition operator is not commutative, since we do not wish parallel agents to reduce to non-deterministic values. – The agent c? inputs a form from channel c and reduces to that value. The reader familiar with the π-calculus will notice a difference with the input prefix. Since we have explicit substitution in our calculus it is simpler to specify the input by c? and use the context to bind the received value instead of defining a prefix syntax c(X).A as in the π-calculus. – The channel c is a primitive output service. If A reduces to F , then cA reduces to the message cF . The value of a message is the empty form . (The value F is only obtained by a corresponding input c? in another agent.)
346
O. Nierstrasz and F. Achermann Table 2. Free Channels. fc() fc(x) →) fc(x fc(A; B) fc(λx.A) fc(νc.A) fc(c?)
3.2
= = = = = = =
∅ ∅ ∅ fc(A) ∪ fc(B) fc(A) fc(A)\{c} {c}
fc(R) fc(L) fc(hide x ) fc(A · B) fc(AB) fc(A | B) fc(c)
= = = = = = =
∅ ∅ ∅ fc(A) ∪ fc(B) fc(A) ∪ fc(B) fc(A) ∪ fc(B) {c}
Free Channels and Closed Agents
As in the π-calculus, forms may contain free channel names. An agent may create a new channel, and communicate this new name to another agent in a separate lexical scope. The free channels fc(A) of an agent A are defined inductively in table 2. αconversion (of channels) is defined in the usual way. We identify agent expressions up to α-conversion. We omit a definition of free variables. Since Piccola is a calculus with explicit environments, we cannot easily define α-conversion on variables. Such a definition would have to include the special nature of R. Instead, we define a closed agent where all variables, root expressions, and abstractions occur beneath a sandbox: Definition 1. The following agents A are closed: →, hide x , L, c and c? are closed. – , x – If A and B are closed then also A · B, AB, A | B and νc.A are closed. – If A is closed, then also A; B is also closed for any agent B.
Observe that any form F is closed by the above definition. An agent is open if it is not closed. Open agents are R, variables x, abstractions λx.A and compositions thereof. Any agent can be closed by putting it into a sandbox with a closed context. Sandbox agents are closed if the root context is closed. In lemma 1 we show that the property of being closed is preserved by reduction. 3.3
Congruence and Pre-forms
As in the π calculus, we introduce structural congruence over agent expressions to simplify the reduction relation. The congruence allows us to rewrite agent expressions to bring communicating agents into juxtapositions, as in the Chemical Abstract Machine of Berry and Boudol [8]. The congruence rules constitute three groups (see table 3). The first group (from ext empty right to single service) deals with congruence over forms. It specifies that extension is idempotent and associative on forms. The rules single service and single binding specify that extension overwrites services and bindings with the same label. We define labels(F ) as follows:
A Calculus for Modeling Software Components
347
Table 3. Congruences. ≡ is the smallest congruence satisfying the following axioms:
x =y
implies
x =y
implies
c∈ / fc(A) c∈ / fc(A) c∈ / fc(A) c∈ / fc(A) c∈ / fc(A) c∈ / fc(A) c∈ / fc(A) c∈ / fc(A)
implies implies implies implies implies implies implies implies
F · ·F (F · G) · H →F ) S · (x →F · y →G x →F · x →G x S · S F;A · B F ; AB A; (B; C) F;G F;R →G) hide x (F · x →G) hide y (F · x hide x hide x S (F · S)G (A | B) | C (A | B) | C (A | B) · C F · (A | B) (A | B)C F (A | B) (A | B); C F ; (A | B) F |A cF νcd.A A | νc.B (νc.B) | A (νc.B) · A A · νc.B A; νc.B (νc.B); A (νc.B)A A(νc.B)
≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡
F (ext empty right) F (ext empty left) F · (G · H) (ext assoc) →F ) · S (x (ext service commute) →G · x →F y (ext bind commute) →G x (single binding) S (single service) (F ; A) · (F ; B) (sandbox ext) (F ; A)(F ; B) (sandbox app) (A; B); C (sandbox assoc) G (sandbox value) F (sandbox root) hide x F (hide select) →G hide y F · x (hide over) (hide empty) S (hide service) SG (use service) A | (B | C) (par assoc) (B | A) | C (par left commute) A | B·C (par ext left) A | F ·B (par ext right) A | BC (par app left) A | FB (par app right) A | B; C (par sandbox left) F;A | F;B (par sandbox right) A (discard zombie) cF | (emit) νdc.A (commute channels) νc.(A | B) (scope par left) νc.(B | A) (scope par right) νc.(B · A) (scope ext left) νc.(A · B) (scope ext right) νc.(A; B) (scope sandbox left) νc.(B; A) (scope sandbox right) νc.BA (scope app left) νc.AB (scope app right)
Definition 2. For each form F , the set of labels(F ) ⊂ L is given by: labels() = ∅ →G) = {x} labels(x
labels(S) = ∅ labels(F · G) = labels(F ) ∪ labels(G)
348
O. Nierstrasz and F. Achermann
Using the form congruences, we can rewrite any form F into one of the following three cases: F ≡ F ≡S →G F ≡ F · x
where x ∈ labels(F )
This is proved by structural induction over forms [2]. This formalizes our idea that forms are extensible records unified with services. A form has at most one binding for a given label. The second group (from sandbox ext to use service) defines preforms. These are agent expressions that are congruent to a form. For instance, the agent hide x is equivalent to the empty form . The set of all preforms is defined by: F ≡ = {A|∃F ∈ F with F ≡ A} Clearly, all forms are preforms. The last group (from par assoc to scope app right) defines the semantics of parallel composition and communication for agents. Note how these rules always preserve the position of the rightmost agent in a parallel composition, since this agent, when reduced to a form, will represent the value of the composition. In particular, the rule discard zombie garbage-collects form values appearing to the left of this position. The rule emit, on the other hand, spawns an empty form as the value, thus enabling the message to move around freely. For instance in →c() ≡ x →(c() | ) x → ≡ c() | x
by emit by par ext right
→. the message c() escapes the binding x
3.4
Reduction
We define the reduction relation → on agent expressions to reduce applications, communications and projections (see table 4). ⇒ is the reflexive and transitive closure of →. Especially noteworthy is the rule reduce beta. This rule does not substitute G for x in the agent A as in the classical λ-calculus. Instead, it extends the environment in which A is evaluated. This is essentially the beta-reduction rule found in calculi for explicit substitution [1,32]: →G; A (F ; λx.A)G → F · x
The application of the closure F ; λx.A to the argument G reduces to a sandbox →G. Free expression in which the agent A is evaluated in the environment F · x occurrences of x in A will therefore be bound to G. The property of being closed is respected by reduction: Lemma 1. If A is a closed agent and A → B or A ≡ B then B is closed as well. Proof. Easily checked by induction over the formal proof for A → B.
A Calculus for Modeling Software Components
349
Table 4. Reduction rules.
→G; A (F ; λx.A) G → F · x
(reduce beta)
cF | c? → F
(reduce comm)
→G; x → G F · x
(reduce project)
L → ; λx.(x; isEmpty)
(reduce inspect empty)
LS → ; λx.(x; isService)
(reduce inspect service)
→G) → ; λx.(x; isLabel )label x L(F · x
(reduce inspect label)
A≡A
A →B
A→B
B ≡B A→B
E[A] → E[B]
(reduce struct) (reduce propagate)
→(; λx.(x; x)) · hide →hide x · bind →(x →) and E is an evaluation where label x = project context defined by the grammar: E ::= [ ] E · A F · E E; A F ; E EA F E A|E E|A νc.E
3.5
Encoding Booleans
The following toy example actually illustrates many of the principles at stake when we model components with the Piccola calculus. We can encode booleans by services that either project on the labels true or false depending on which boolean value they are supposed to model (cf. [13]). (This same idea is used by the primitive service L to reflect over the bindings and services of a form.) def
True = ; λx.(x; true) def
False = ; λx.(x; false)
(1) (2)
Consider now: →1 · false →2) = (; λx.(x; true))(true →1 · false →2) True(true →(true →1 · false →2); (x; true) → · x
by reduce beta →(true →1 · false →2); x); true by sandbox assoc ≡ ( · x →1 · false →2); true → (true by reduce project →2 · true →1); true ≡ (false by ext bind commute →1
by reduce project
350
O. Nierstrasz and F. Achermann
Note how the bindings are swapped to project on true in the last step. A →1 · false →2) ⇒ 2. similar reduction would show False(true One of the key points of forms is that a client can provide additional bindings which are ignored when they are not used (cf. [13]). This same principle is applied to good effect in various scripting languages, such as Python [22]. For instance →F for arbitrary we can use True and provide an additional binding notused form F : →1 · false →2 · notused →F ) True(true →1 · false →2 · notused →F ); true ⇒ (true →2 · true →1 · notused →F ); true ≡ (false →2 · notused →F · true →1); true ≡ (false →1
by ext bind commute by ext bind commute by reduce project
Extending forms can also be used to overwrite existing bindings. For instance instead of binding the variable notused a client may override true: →1 · false →2 · true →3) ⇒ 3 True(true
A conditional expression is encoded as a curried service that takes a boolean and a case form. When invoked, it selects and evaluates the appropriate service in the case form: def
→(v; then) · false →(v; else)) if = ; λuv.u(true
Now consider: →(F ; λx.A) · else →(G; λx.B)) if True (then →; A ⇒ F · x
The expression if True has triggered the evaluation of agent A in the envi→. ronment F · x The contract supported by if requires that the cases provided bind the labels then and else. We can relax this contract and provide default services if those bindings are not provided by the client. To do so, we replace in the definition of if the sandbox expression v; else with a default service. This service gets triggered when the case form does not contain an else binding: def
→(v; then) · false →(else →(λx.) · v; else)) ifd = ; λuv.b(true (F ; λx.A)) ⇒ . Now ifd False(then →
3.6
Equivalence for Agents
Two agents are equivalent if they exhibit the same behaviour, i.e., they enjoy the same reductions. We adopt Milner and Sangiorgi’s notion of barbed bisimulation
A Calculus for Modeling Software Components
351
[26]. The idea is that an agent A is barbed similar to B if A can exhibit any reduction that B does and if B is a barb, then A is a barb, too. If A and B are similar to each other they are bisimilar. The advantage of this bisimulation is that it can be readily be given for any calculus that contains barbs or values. For the asynchronous π-calculus, barbs are usually defined as having the capability of doing an output on a channel. A Piccola agent reduces to a barb, i.e., it returns a form. During evaluation the agent may spawn off new subthreads which could be blocked or still be running. We consequently define barbs as follows: Definition 3. A barb V is an agent expression A that is congruent to an agent generated by the following grammar: V ::= F A|V νc.V We write A ↓ for the fact that A is a barb, and A ⇓ when a barb V exists such that A ⇒ V . The following lemma relates forms, barbs and agents: Lemma 2. The following inclusion holds and is strict: F ⊂ F ≡ ⊂ {A|A ↓} ⊂ A Proof. The inclusions hold by definition. To see that the inclusion are strict, consider the empty form , the agent hide x , the barb 0 | hide x and the agent 0 (where 0 = νc.c? is the deadlocked null agent). The following lemma gives a syntactical characterization of barbs. Lemma 3. For any form F , agent A, and label x, the following terms are barbs, given V1 and V2 are barbs. V 1 · V2 V1 ; V 2 →V1 x
νc.V1 A | V1
Proof. By definition we have V ≡ ν˜ c.A | F . The claim follows by induction over F. We now define barbed bisimulation and the induced congruence: Definition 4. A relation R is a (weak) barbed bisimulation, if A R B, i.e., (A, B) ∈ R implies: – – – –
If If If If
A → A then there exists an agent B with B ⇒ B and A R B . B → B then there exists an agent A with A ⇒ A and A R B . A ↓ then B ⇓. B ↓ then A ⇓.
352
O. Nierstrasz and F. Achermann
˙ B, if there is some (weak) Two agents are (weakly) barbed bisimilar, written A ≈ barbed bisimulation R with A R B. Two agents are (weakly) barbed congruent, ˙ C[B]. written A ≈ B, if for all contexts C we have C[A] ≈ We define behavioural equality using the notion of barbed congruence. As usual we can define strong and weak versions of barbed bisimulation. The strong versions are obtained in the standard way by replacing ⇒ with → and ⇓ with ↓ in Definition 4. We only concentrate on the weak case since it abstracts internal computation. 3.7
Erroneous Reductions
Not all agents reduce to forms. Some agents enjoy an infinite reduction [2]. Other agents are stuck. An agent is stuck if it is not a barb and can reduce no further. Definition 5. An agent A is stuck, written A ↑, if A is not a barb and there is no agent B such that A → B. Clearly it holds that 0 ↑ and R ↑. The property of being stuck is not compositional. For instance c? ↑ but obviously, c() | c? can reduce to . We can put R into a context so that it becomes a barb, for instance F ; R ≡ F . Note that if an agent is stuck it is not a preform: F ≡ ∩ {A|A ↑} = ∅ by definition. Although 0 is arguably stuck by intention, in general a stuck agent can be interpreted as an error. The two typical cases which may lead to errors are (i) projection on an unbound label, e.g., ; x, and (ii) application of a non-service, e.g., . 3.8
π-Calculus Encoding
One may well ask what exactly the Piccola calculus adds over and above the asynchronous π calculus. In Achermann’s thesis it is shown that the Piccola calculus can be faithfully embedded into the localized π-calculus Lπ of Merro and Sangiorgi [23,36]. The mapping [[]]a encodes Piccola calculus agents as π-calculus processes. The process [[A]]a evaluates A in the environment given by the empty form, and sends the resulting value along the channel a. A form (value) is encoded as a 4-tuple of channels representing projection, invocation, hiding and selection. The main result is that the encoding is sound and preserves reductions. We do not require a fully abstract encoding since that would mean that equivalent Piccola agents translated into the π-calculus could not be distinguished by any π-processes. Our milder requirement means that we consider only π-processes which are translations of Piccola agents themselves and state that they cannot distinguish similar agents: Proposition 1 (Soundness). For closed agents A, B and channel a the congruence [[A]]a ≈ [[B]]a implies A ≈ B.
A Calculus for Modeling Software Components
353
Although it is comforting to learn that the π-calculus can serve as a foundation for modeling components, it is also clear from the complexity of the encoding that it is very distant from the kinds of abstractions we need to conveniently model software composition. For this reason we find that a richer calculus is more convenient to express components and connectors.
4
From the Piccola Calculus to Piccola
Piccola is a small composition language that supports the requirements summarized in figure 1, and whose denotational semantics is defined in terms of the Piccola calculus [2]. Piccola is designed in layered fashion. At the lowest level we have an abstract machine that implements the Piccola calculus. At the next level, we have the Piccola language, which is implemented by translation to the abstract machine, following the specification of the denotational semantics.
Applications: Composition styles: Standard libraries:
Piccola language:
Piccola calculus:
Piccola Layers Components + Scripts Streams, GUI composition, ... Coordination abstractions, control structures, basic object model ... Host components, user-defined operators, dynamic namespaces Forms, agents and channels
Piccola provides a more convenient, Python-like syntax for programming than does the calculus, including overloaded operators to support algebraic component composition. It also provides a bridge to the host language (currently Java or Squeak). Piccola provides no basic data types other than forms and channels. Booleans, integers, floating point numbers and strings, for example, must be provided by the host language through the language bridge. Curiously, the syntax of the Piccola calculus is actually larger than that of Piccola itself. This is because we need to represent all semantic entities, including agents and channels, as syntactic constructs in the calculus. In the Piccola language, however, these are represented only by standard library services, such as run and newChannel. The next layer provides a set of standard libraries to simplify the task of programming with Piccola. Not only does the Piccola language provide no built-in data types, it does not even offer any control structures of its own. These, however, are provided as standard services implemented in Piccola. Exceptions and
354
O. Nierstrasz and F. Achermann
try-catch clauses are implemented using agents, channels, and dynamic namespaces [5]. The first three layers constitute the standard Piccola distribution. The fourth layer is provided by the component framework designer. At this level, a domain expert encodes a compositional styles as a library of components, connectors, adaptors, coordination abstractions, and so on. Finally, at the top level, an application programmer may script together components using the abstractions provided by the lower layers [3,29]. 4.1
Reasoning about Styles
We have also explored how to reason about Piccola programs at the language level [2]. We have studied two extended examples. First, we considered synchronization wrappers that express the synchronization constraints assumed by a component. We can use synchronization wrappers to make components safe in a multithreaded environment. The wrappers separate the functionality of the component from their synchronization aspects. If the constraints assumed by the component hold in a particular composition the wrapper is not needed. In particular the wrapper is not necessary when the component is already wrapped. This property is formally expressed by requiring that the wrappers be idempotent. The second study compares push- and pull-flow filters. We demonstrate how to adapt pull-filters so that they work in a push-style. We have constructed a generic adapter for this task in two iterations. The first version contains a racecondition that may lead to data being lost. The formal model of Piccola is used to analyze the traces of an adapted filter and helps to detect the error. To fix the problem we specify the dynamics of a push-style filter, namely that push and close calls be mutually exclusive, that no further push calls may be attempted after a close, and that no “air-bubble” elements (filter slots holding an empty form) may be pushed downstream. Having clarified the interaction protocol as a wrapper, we present an improved version of the generic adapter. We show that the adapter ensures these invariants. 4.2
Partial Evaluation
Another interesting application of the calculus was to enable the efficient implementation of the language bridge. Since Piccola is a pure composition language, evaluating scripts requires intensive upping and downing [24] between the “down” level of the host language and the “up” level of Piccola. If the language bridge were implemented na¨ıvely, it would be hopelessly inefficient. Instead, Piccola achieves acceptable performance by adopting a partial evaluation scheme [2,38,39]. Since the language has a denotational semantics, we can implement it efficiently while proving that we preserve the intended semantics. The partial evaluation algorithm uses the fact that forms are immutable. We replace references
A Calculus for Modeling Software Components
355
to forms by the forms referred to. We can then specialize projections and replace applications of referentially transparent services by their results. However, most services in Piccola are not referentially transparent and cannot be inlined since that would change the order in which side-effects are executed. We need to separate the referentially transparent part from the non-transparent part in order to replace an application with its result and to ensure that the order in which the side-effects are evaluated is preserved. At the heart of the proof is that we can separate form expressions into sideeffects and referentially transparent forms [2].
5
Related Work
The Piccola calculus extends the asynchronous π-calculus with higher-order abstractions and first-class environments. π-calculus. The π-calculus [25] is a calculus of communicating systems in which one can naturally express processes with a changing structure. Its theory has been thoroughly studied and many results relate other formalisms or implementations to it. The affinity between objects and processes, for example, has been treated by various authors in the context of the π-calculus [18,44]. The Pict experiment has shown that the π-calculus is a suitable basis for programming many high-level construct by encodings [33]. For programming and implementation purposes, synchronous communication seems uncommon and can generally be encoded by using explicit acknowledgments (cf. [18]). Moreover, asynchronous communication has a closer correspondence to distributed computing [45]. Furthermore, in the π-calculus the asynchronous variant has the pleasant property that equivalences are simpler than for the synchronous case [14]. Input-guarded choice can be encoded and is fully abstract [27]. For these reasons we adopt asynchronous channels in Piccola. Higher-order abstractions. Programming directly in the π-calculus is often considered like programming a concurrent assembler. When comparing programs written in the π-calculus with the lambda-calculus it seems like lambda abstractions scale up, whereas sending and receiving messages does not scale well. There are two possible solutions proposed to this problem: We can change the metaphor of communication or we can introduce abstractions as first-class values. The first approach is advocated by the Join-calculus [15]. Communication does not happen between a sender and a receiver, instead a join pattern triggers a process on consumption of several pending messages. The Blue calculus of Boudol [9] changes the receive primitive into a definition which is defined for a scope. By that change, the Blue calculus is more closely related to functions and provides a better notion for higher-order abstraction. Boudol calls it a continuation-passing calculus. The other approach is adopted by Sangiorgi in the HOπ-calculus. Instead of communicating channels or tuples of channels, processes can be communicated
356
O. Nierstrasz and F. Achermann
as well. Surprisingly, the higher-order case has the same expressive power as the first-order version [35,36]. In Piccola we take the second approach and reuse existing encodings of functions into the π-calculus as in Pict. The motivation for this comes from the fact that the HOπ-calculus itself can be encoded in the first-order case. Asymmetric parallel composition. The semantics of asynchronous parallel composition is used in the concurrent object calculus of Gordon and Hankin [17] or the (asymmetric) Blue calculus studied by Dal-Zilio [12]. In the higher-order π-calculus the evaluation order is orthogonal to the communication semantics [36]. In Piccola, evaluation strategy interferes with communication, therefore we have to fix one for meaningful terms. For Piccola, we define strict evaluation which seems appropriate and more common for concurrent computing. Record calculus. When modeling components and interfaces, a record-based approach is the obvious choice. We use forms [20,21] as an explicit notion for extensible records. Record calculi are studied in more detail for example in [11,34]. In the λ-calculus with names of Dami [13] arguments to functions are named. The resulting system supports records as arguments instead of tuples as in the classical calculus. The λN -calculus was one of the main inspiration for our work on forms without introspection. An issue omitted in our approach is record typing. It is not clear how far record types with subtyping and the runtime acquisition can be combined. An overview of record typing and the problems involved can be found for example in [11]. Explicit environments. An explicit environment generalizes the concept of explicit substitution [1] by using a record like structure for the environment. In the environment calculus of Nishizaki, there is an operation to get the current environment as a record and an operator to evaluate an expression using a record as environment [32,37]. Projection of a label x in a record R then corresponds to evaluating the script x in an environment denoted by R. The reader may note that explicit environments subsume records. This is the reason why we call them forms in Piccola instead of just records. Handling the environment as a first-class entity allows us to define concepts like modules, interfaces and implementation for programming in the large within the framework. To our knowledge, the language Pebble of Burstall and Lampson was the first to formally show how to build modules, interfaces and implementation, abstract data types and generics on a typed lambda calculus with bindings, declarations and types as first-class values [10]. Other approaches A very different model is offered by ρω (AKA Reo) [7], a calculus of component connectors. Reo is algebraic in flavour, and provides various connectors that coordinate and compose streams of data. Primitives connectors can be composed using the Reo operators to build hiigher-level connectors. In contrast to process calculi, Reo is well-suited to compositional reasoning, since connectors can be composed to yield new connectors, and properties of
A Calculus for Modeling Software Components
357
connectors can be shown to compose. Data communicated along streams are uninterpreted in Reo, so it would be natural to explore the application of Reo to streams of forms.
6
Concluding Remarks
We have presented the Piccola calculus, a high-level calculus for modeling software components that extends the asynchronous π-calculus with explicit namespaces, or forms. The calculus serves as the semantic target for Piccola, a language for composing software components that conform to a particular compositional style. JPiccola, the Java implementation of Piccola, is realized by translation to an abstract machine that implements the Piccola calculus. The Piccola calculus is not only helpful for modeling components and connectors, but it also helps to reason about the Piccola language implementation and about compositional styles. Efficient language bridging between Piccola and the host language (Java or Squeak) is achieved by means of partial evaluation of language wrappers. The partial evaluation algorithm can be proved correct with the help of the Piccola calculus. Different compositional styles make different assumptions about software components. Mixing incompatible components can lead to compositional mismatches. We have outlined how the Piccola calculus can help to bridge mismatches by supporting reasoning about wrappers that adapt component contracts from one style to another. One shortcoming of our work so far is the lack of a type system. We have been experimenting with a system of contractual types [28] that expresses both the provided as well as the required services of a software component. Contractual types are formalized in the context of the form calculus, which can be seen as the Piccola calculus minus agents and channels. Contractual types have been integrated into the most recent distribution of JPiccola [19].
Acknowledgments We gratefully acknowledge the financial support of the Swiss National Science Foundation for projects No. 20-61655.00, “Meta-models and Tools for Evolution Towards Component Systems”, and 2000-067855.02, “Tools and Techniques for Decomposing and Composing Software”.
References 1. Mart´ın Abadi, Luca Cardelli, Pierre-Louis Curien, and Jean-Jacques L´evy. Explicit substitutions. Journal of Functional Programming, 1(4):375–416, October 1991. 2. Franz Achermann. Forms, Agents and Channels - Defining Composition Abstraction with Style. PhD thesis, University of Berne, January 2002.
358
O. Nierstrasz and F. Achermann
3. Franz Achermann, Stefan Kneubuehl, and Oscar Nierstrasz. Scripting coordination styles. In Ant´ onio Porto and Gruia-Catalin Roman, editors, Coordination ’2000, volume 1906 of LNCS, pages 19–35, Limassol, Cyprus, September 2000. SpringerVerlag. 4. Franz Achermann, Markus Lumpe, Jean-Guy Schneider, and Oscar Nierstrasz. Piccola – a small composition language. In Howard Bowman and John Derrick, editors, Formal Methods for Distributed Processing – A Survey of Object-Oriented Approaches, pages 403–426. Cambridge University Press, 2001. 5. Franz Achermann and Oscar Nierstrasz. Explicit Namespaces. In J¨ urg Gutknecht and Wolfgang Weck, editors, Modular Programming Languages, volume 1897 of LNCS, pages 77–89, Z¨ urich, Switzerland, September 2000. Springer-Verlag. 6. Franz Achermann and Oscar Nierstrasz. Applications = Components + Scripts – A Tour of Piccola. In Mehmet Aksit, editor, Software Architectures and Component Technology, pages 261–292. Kluwer, 2001. 7. Farhad Arbab and Farhad Mavaddat. Coordination through channel composition. In F. Arbab and C. Talcott, editors, Coordination Languages and Models: Proc. Coordination 2002, volume 2315 of Lecture Notes in Computer Science, pages 21– 38. Springer-Verlag, April 2002. 8. G´erard Berry and G´erard Boudol. The chemical abstract machine. Theoretical Computer Science, 96:217–248, 1992. 9. G´erard Boudol. The pi-calculus in direct style. In Conference Record of POPL ’97, pages 228–241, 1997. 10. Rod Burstall and Butler Lampson. A kernel language for abstract data types and modules. Information and Computation, 76(2/3), 1984. Also appeared in Proceedings of the International Symposium on Semantics of Data Types, Springer, LNCS (1984), and as SRC Research Report 1. 11. Luca Cardelli and John C. Mitchell. Operations on records. In Carl A. Gunter and John C. Mitchell, editors, Theoretical Aspects of Object-Oriented Programming. Types, Semantics and Language Design, pages 295–350. MIT Press, 1993. 12. Silvano Dal-Zilio. Le calcul bleu: types et objects. Ph.D. thesis, Universit´e de Nice - Sophia Antipolis, July 1999. In french. 13. Laurent Dami. Software Composition: Towards an Integration of Functional and Object-Oriented Approaches. Ph.D. thesis, University of Geneva, 1994. 14. C´edric Fournet and Georges Gonthier. A hierarchy of equivalences for asynchronous calculi. In Proceedings of ICALP ’98, pages 844–855, 1998. 15. C´edric Fournet, Georges Gonthier, Jean-Jacques L´evy, Luc Maranget, and Didier R´emy. A calculus of mobile agents. In Proceedings of the 7th International Conference on Concurrency Theory (CONCUR ’96), volume 1119 of LNCS, pages 406–421. Springer-Verlag, August 1996. 16. David Garlan, Robert Allen, and John Ockerbloom. Architectural mismatch: Why reuse is so hard. IEEE Software, 12(6):17–26, November 1995. 17. Andrew D. Gordon and Paul D. Hankin. A concurrent object calculus: Reduction and typing. In Proceedings HLCL ’98. Elsevier ENTCS, 1998. 18. Kohei Honda and Mario Tokoro. An object calculus for asynchronous communication. In Pierre America, editor, Proceedings ECOOP ’91, volume 512 of LNCS, pages 133–147, Geneva, Switzerland, July 15–19 1991. Springer-Verlag. 19. Stefan Kneubuehl. Typeful compositional styles. Diploma thesis, University of Bern, April 2003. 20. Markus Lumpe. A Pi-Calculus Based Approach to Software Composition. Ph.D. thesis, University of Bern, Institute of Computer Science and Applied Mathematics, January 1999.
A Calculus for Modeling Software Components
359
21. Markus Lumpe, Franz Achermann, and Oscar Nierstrasz. A Formal Language for Composition. In Gary Leavens and Murali Sitaraman, editors, Foundations of Component Based Systems, pages 69–90. Cambridge University Press, 2000. 22. Mark Lutz. Programming Python. O’Reilly & Associates, Inc., 1996. 23. Massimo Merro and Davide Sangiorgi. On asynchrony in name-passing calculi. In Kim G. Larsen, Sven Skyum, and Glynn Winskel, editors, 25th Colloquium on Automata, Languages and Programming (ICALP) (Aalborg, Denmark), volume 1443 of LNCS, pages 856–867. Springer-Verlag, July 1998. 24. Wolfgang De Meuter. Agora: The story of the simplest MOP in the world — or — the scheme of object–orientation. In J. Noble, I. Moore, and A. Taivalsaari, editors, Prototype-based Programming. Springer-Verlag, 1998. 25. Robin Milner, Joachim Parrow, and David Walker. A calculus of mobile processes, part I/II. Information and Computation, 100:1–77, 1992. 26. Robin Milner and Davide Sangiorgi. Barbed bisimulation. In Proceedings ICALP ’92, volume 623 of LNCS, pages 685–695, Vienna, July 1992. Springer-Verlag. 27. Uwe Nestmann and Benjamin C. Pierce. Decoding choice encodings. In Ugo Montanari and Vladimiro Sassone, editors, CONCUR ’96: Concurrency Theory, 7th International Conference, volume 1119 of LNCS, pages 179–194, Pisa, Italy, August 1996. Springer-Verlag. 28. Oscar Nierstrasz. Contractual types. submitted for publication, 2003. 29. Oscar Nierstrasz and Franz Achermann. Supporting Compositional Styles for Software Evolution. In Proceedings International Symposium on Principles of Software Evolution (ISPSE 2000), pages 11–19, Kanazawa, Japan, Nov 1-2 2000. IEEE. 30. Oscar Nierstrasz, Franz Achermann, and Stefan Kneubuehl. A guide to jpiccola. Technical report, Institut f¨ ur Informatik, Universit¨ at Bern, Switzerland, 2003. Available from www.iam.unibe.ch/∼scg/Research/Piccola. 31. Oscar Nierstrasz and Laurent Dami. Component-oriented software technology. In Oscar Nierstrasz and Dennis Tsichritzis, editors, Object-Oriented Software Composition, pages 3–28. Prentice-Hall, 1995. 32. Shin-ya Nishizaki. Programmable environment calculus as theory of dynamic software evolution. In Proceedings ISPSE 2000. IEEE Computer Society Press, 2000. 33. Benjamin C. Pierce and David N. Turner. Pict: A programming language based on the pi-calculus. In G. Plotkin, C. Stirling, and M. Tofte, editors, Proof, Language and Interaction: Essays in Honour of Robin Milner. MIT Press, May 2000. 34. Didier R´emy. Typing Record Concatenation for Free, chapter 10, pages 351–372. MIT Press, April 1994. 35. Davide Sangiorgi. Expressing Mobility in Process Algebras: First-Order and HigherOrder Paradigms. Ph.D. thesis, Computer Science Dept., University of Edinburgh, May 1993. 36. Davide Sangiorgi. Asynchronous process calculi: the first-order and higher-order paradigms (tutorial). Theoretical Computer Science, 253, 2001. 37. Masahiko Sato, Takafumi Sakurai, and Rod M. Burstall. Explicit environments. In Jean-Yves Girard, editor, Typed Lambda Calculi and Applications, volume 1581 of LNCS, pages 340–354, L’Aquila, Italy, April 1999. Springer-Verlag. 38. Nathanael Sch¨ arli. Supporting pure composition by inter-language bridging on the meta-level. Diploma thesis, University of Bern, September 2001. 39. Nathanael Sch¨ arli and Franz Achermann. Partial evaluation of inter-language wrappers. In Workshop on Composition Languages, WCL ’01, September 2001. 40. Jean-Guy Schneider. Components, Scripts, and Glue: A conceptual framework for software composition. Ph.D. thesis, University of Bern, Institute of Computer Science and Applied Mathematics, October 1999.
360
O. Nierstrasz and F. Achermann
41. Jean-Guy Schneider and Oscar Nierstrasz. Components, scripts and glue. In Leonor Barroca, Jon Hall, and Patrick Hall, editors, Software Architectures – Advances and Applications, pages 13–25. Springer-Verlag, 1999. 42. Mary Shaw and David Garlan. Software Architecture: Perspectives on an Emerging Discipline. Prentice-Hall, 1996. 43. Clemens A. Szyperski. Component Software. Addison Wesley, 1998. 44. David Walker. Objects in the π-calculus. Information and Computation, 116(2):253–271, February 1995. 45. Pawel T. Wojciechowski. Nomadic Pict: Language and Infrastructure Design for Mobile Computation. PhD thesis, Wolfson College, University of Cambridge, March 2000.
Specification and Inheritance in CSP-OZ Ernst-R¨ udiger Olderog and Heike Wehrheim Department of Computing Science University of Oldenburg 26111 Oldenburg, Germany {olderog,wehrheim}@informatik.uni-oldenburg.de
Abstract. CSP-OZ [16,18] is a combination of Communicating Sequential Processes (CSP) and Object-Z (OZ). It enables the specification of systems having both a state-based and a behaviour-oriented view using the object-oriented concepts of classes, instantiation and inheritance. CSP-OZ has a process semantics in the failures divergence model of CSP. In this paper we explain CSP-OZ and investigate the notion of inheritance. Behavioural subtyping relations between classes introduced in [50] guarantee the inheritance of safety and “liveness” properties. Keywords: CSP, Object-Z, failure divergence semantics, inheritance, safety and “liveness” properties, model-checking, FDR
1
Introduction
In contrast to the wide-spread use of object-oriented programming and specification languages, little is known about the properties enjoyed by systems constructed in the object-oriented style. Research on verification of object-oriented descriptions takes often place in the setting of object-oriented programming languages, for instance Java. The methods range from Hoare-style verification supported by theorem provers [25,36] via static checkers [30] to model-checking techniques [20]. Verification of object-oriented modeling languages focusses on UML (e.g. [28,40]). These approaches check properties of UML state machines by translating them into existing model-checkers. Although UML is an integrated formalism allowing the specification of data and behaviour aspects, existing model-checking techniques most often focus on the behavioural view. For a semantics of UML integrating different types of diagrams (including dynamic object creation and dynamically changing communication topologies) and its verification see [8,9]. Reasoning about object-oriented specifications represents a challenge in its own right. To describe the preservation of behavioural properties of classes under change the concept of subtyping has been lifted from data types to objects by [1,31]. Whereas these approaches are restricted to state-based specifications (using e.g. Object-Z) [34] proposed definitions suitable to deal with behaviouroriented specifications (using e.g. CSP). A first systematic study of subtyping for specifications integrating state-based and behaviour-oriented views is [50].
This research is partially supported by the DFG under grant Ol/98-3.
F.S. de Boer et al. (Eds.): FMCO 2002, LNCS 2852, pp. 361–379, 2003. c Springer-Verlag Berlin Heidelberg 2003
362
E.-R. Olderog and H. Wehrheim
Given a proof made with respect to one data-model, its reuse in another data-model extended by inheritance represents a major problem that must be overcome in order to build up libraries that support proofs about non-trivial applications. Existing approaches, based on shallow representations of subtyping via parametric polymorphism, are either geared towards abstractions of programs (like Eiffel, e.g. [21]) or specific data-models (like UML/OCL, [5]). So far, these techniques have not been applied to the combination of data-oriented and behavioral specifications. In this paper we study specification and inheritance in the language CSP-OZ [16,18] combining two existing specification languages for processes and data. Specification of processes. Communicating Sequential Processes (CSP) were introduced by Hoare [22,23]. The central concepts of CSP are synchronous communication via channels between different processes, parallel composition and hiding of internal communication. c!e
c
c?x
For CSP a rich mathematical theory comprising operational, denotational and algebraic semantics with consistency proofs has been developed [4,35,38]. Tool support comes through the FDR model-checker [37]. The name stands for Failure Divergence Refinement and refers to the standard semantic model of CSP, the failures divergence model, and its notion of process refinement. Specification of data. Z was introduced in the early 80’s in Oxford by Abrial as a set-theoretic and predicate language for the specification of data, state spaces and state transformations. It comprises of the mathematical tool kit, a collection of convenient notations and definitions, and the schema calculus for structuring large state spaces and their transformations. A Z schema has a name, say S, and consists of variable declarations and a predicate constraining the values of these variables. It is denoted as follows: S declarations predicate The first systematic description of Z is [46]. Since then the language has been published extensively (e.g. [52]) and used in many case studies and industrial projects. Object-Z is an object-oriented extension of Z [11,42,44]. It comprises the concepts of classes, instantiation and inheritance. Z and Object-Z come with the concept of data refinement. For Z there exist proof systems for establishing properties of specifications and refinements such as Z/EVES [39] or HOL-Z based on Isabelle [27]. For Object-Z type checkers exist. Verification support is less developed except for an extension of HOL-Z [41].
Specification and Inheritance in CSP-OZ
363
Combination. CSP-OZ, developed by C. Fischer at the University of Oldenburg, is a combination of CSP and Object-Z. Object-Z is used to describe all data-dependent aspects of classes, viz. attributes and methods. The dynamic behaviour of classes, i.e. their protocols of interaction, are specified within CSP. CSP-OZ has been used in a number of case studies from the area of telecommunication systems, production automatisation and satellite technology [3,49,33]. Verification of CSP-OZ specifications is supported by the FDR model-checker for CSP [14]. Structure of this paper. In Section 2 the combination CSP-OZ is introduced by way of an example. In Section 3 the semantics of CSP-OZ is reviewed. Section 4 is devoted to the inheritance operator in CSP-OZ and its semantics. In Section 5 inheritance of properties is studied in depth, and Section 6 concludes the paper.
2
The Combination CSP-OZ
There are various specialised techniques for describing individual aspects of system behaviour. But complex systems exhibit various behavioural aspects. This observation has led to research into the combination and semantic integration of specification techniques. One such combination is CSP-OZ [16,13,14,18] integrating CSP and ObjectZ. Central to CSP-OZ is the notion of a class. The specification of a class C has the following format: C I P Z
[interface] [CSP part] [OZ part]
The interface I declares channels names and types to be used by the class. The CSP part uses a CSP process P to describe the desired sequencing behaviour on these channels. The OZ part is of the form Z st : State Init(st) com c(st, in?, out!, st )
[state space] [initial condition] [communication schemas]
where the state space of C and its transformation is specified in the style of Object-Z. The state space itself is given by a schema State, here with the symbolic variable st ranging over State. The initial state of C is described by the Init schema that restricts the value of st. For the channels c declared in the interface communication schemas com c describe the state transformation induced by communicating along c with input values in? and output values out!. In Z and hence Object-Z the prime decoration is used for denoting the new values after the transformation. Thus com c depends on st, in?, out!, st .
364
E.-R. Olderog and H. Wehrheim
Example 1. To illustrate the use of CSP-OZ we present here part of the specification of a till [50] for the problem “cash point service” defined in [10]. Informally, the till is used by inserting a card and typing in a PIN which is compared with the PIN stored on the card. In case of a successful identification, the customer may withdraw money from the account. The till is only one component of a larger system including banks, cards, customers, and money dispensers. Global definitions. The CSP-OZ class Till makes use of two basic Z types: [Pin, CardID] Pin represents the set of personal identification numbers and CardID the set of card identification numbers. Interface. The Till is connected to its environment by several typed channels: chan chan chan chan
getCustPin : [ p? : Pin ] getCardPin : [ p? : Pin ] pay : [ a! : N ] updateBalance : [ a! : N; c! : CardID ]
The channel getCustPin expects a value p? of type Pin as its input, the channel getCardPin inputs a value p? of type Pin and outputs a value c! of type CardID, the channel pay outputs a natural number a! (the amount of money to be paid out to the customer), the channel updateBalance is intended for communication with the bank and outputs an amount a! of money to be withdrawn form the account belonging to card with identity c!. The full definition of the till comprises of more interface channels (see [50]). CSP part. This part specifies the order in which communications along the interface channels can occur. To this end, a set of recursive process equations is c given. The main equation starts with the process identifier main. The symbol = is used instead of an ordinary equals symbol to distinguish between CSP process equations and Z equations. c
main = insertCard → Ident c Ident = getCustPin → getCardPin → (idFail → Eject 2 idSucc → Service) c
Eject = eject → main c Service = (stop → Eject 2 withdraw → getAmount → pay → updateBalance → Eject)
Specification and Inheritance in CSP-OZ
365
This process specifies the following behaviour. First the till gets the CardID via a communication along channel insertCard . Then the till checks the customers identity by getting the customer’s PIN via getCustPin, retrieving the PIN stored on the card via getCardPin, and comparing both. The communications idFail signals that this comparison failed, the communication idSucc signals that it was successful. In case of failure the card is ejected. In case of success that service mode is entered where the customer has the choice of stopping the interaction or withdrawing money. In the latter case the communication getAmount inputs the amount of money the customer wishes to withdraw, the communication pay initiates the money dispenser to pay out this amount, and the communication updateBalance informs the bank about the change. Note that in this particular CSP process no communication values are specified. They will be dealt with in the OZ part. OZ part. This part specifies the state of the till and the effect of the communications on this state. The state consists of the following typed variables:
currCard : CardID currPin, typedPin : Pin amount : N
[state space]
The effect of communications along the interface channels are specified using communication schemas. Here we give only some examples. The schema com insertCard ∆(currCard ) c? : CardId currCard = c? specifies that a communication along insertCard may only change the state variable currCard . This is the meaning of the ∆-list in the first line of the declaration part of this schema. The second line specifies that a communication along insertCard has an input value c? of type CardID. The predicate of the schema specifies that the effect of the schema is an assignment of this input value to the variable currCard . The schema com idSucc ∆() currPin = typedPin specifies that a communication idSucc does not change any variable of the state (empty ∆-list) and is enabled only if the current PIN of the card is identical to the PIN typed in by the customer. The schema
366
E.-R. Olderog and H. Wehrheim
com updateBalance ∆() a! : N c! : CardId a! = amount ∧ c! = CardId specifies that the amount a! of money to withdrawn and the identity c! of the card are sent to the bank where the balance of the account is updated. Instances. An instance (object) t of the class Till is specified by a declaration t : Till . The instance t behaves like the class Till but with all channels ch renamed to ch.t with its own identity t. A customer using till t might perform the following interaction with it expressed as a CSP process: c
Customer = insertCard .t.765 → getCustPin.t.4711 → withdraw .t → getAmount.t.100 → SKIP To model the behaviour of several instances t1 , ..., tn of the class Till the interleaving operator ||| of CSP can be used: t1 ||| ... ||| tn
or
|||i=1,...,n ti
To be able to be connected to a finite sets of tills the class Bank will have a parameter adr of type F Till for the addresses of the different tills: Bank [adr : F Till ] chan updateBalance : [ t : adr , a! : N; c! : CardID ] ........................... For example, a system comprising one bank b connected to tills t1 , ..., tn can be specified by the process expression (b : Bank ; t1 , ..., tn : Till • b({t1 , ..., tn }) ||{updateBalance} (|||i=1,...,n ti )) where b is instantiated with the set {t1 , ..., tn } and ||{updateBalance} is the parallel composition enforcing synchronisation on the channel updateBalance between the bank b and the tills t1 , ..., tn .
3
Semantics
Each class of a CSP-OZ specification denotes a process obtained by transforming the OZ part into a process that runs in parallel with the CSP part. First we briefly review the semantics of CSP and Object-Z.
Specification and Inheritance in CSP-OZ
3.1
367
Semantics of CSP
The standard semantics of CSP is the FD-semantics based on failures and divergence [38]. A failure is a pair (s, X ) consisting of a finite sequence or trace s ∈ seq Comm over a set Comm of communications and a so-called refusal set X ∈ P Comm. Intuitively, a failure (s, X ) describes that after engaging in the trace s the process can refuse to engage in any of the communications in X . Refusal sets allow us to make fine distinctions between different nondeterministic process behaviour; they are essential for obtaining a compositional definition of parallel composition in the CSP setting of synchronous communication when we want to observe deadlocks. Formally, we need the following sets of observations about process behaviour: Traces = seq Comm, Refusals = P Comm, Failures = Traces × Refusals. A divergence is a trace after which the process can engage in an infinite sequence of internal actions. The FD-semantics of CSP is then given by two mappings F : CSP → P Failures and D : CSP → P Traces. For a CSP process P we write FD[[P ]] = (F[[P ]] , D[[P ]]). Certain well-formedness conditions relate the values of F and D (see [38], p.192). The FD-semantics induces a notion of process refinement denoted by F D . For CSP processes P and Q this relation is defined as follows: P F D Q iff F[[P ]] ⊇ F[[Q]] and D[[P ]] ⊇ D[[Q]] Intuitively, P F D Q means that Q refines P , i.e. Q is more deterministic and more defined than P . 3.2
Semantics of Z and Object-Z
In Z an operation schema Op x : Dx x : Dx P( x, x
)
describes a transformation on state space x specified by the predicate P ( x , x ). Z comes with the usual notion of data refinement and operation refinement [52]. Given a relation ρ between an abstract and a concrete data domain, a concrete operation schema C Op refines an abstract operation schema A Op, denoted by A Op
ρ C Op,
368
E.-R. Olderog and H. Wehrheim
if C Op is more defined and more deterministic than A Op. For Object-Z a history semantics comprising of sequences of states and events (operation calls) as well as some more abstract semantics are defined [42]. We do not need these semantics here because we only use the state transformation view of the communication schemas in the OZ part. 3.3
Semantics of CSP-OZ
The semantics of CSP-OZ is defined in [16,18]. Each CSP-OZ class denotes a process in the failures divergence model of CSP. This is achieved by transforming the OZ part of such a class into a CSP process that runs in parallel and communicates with the CSP part of the class. The OZ part can hence be seen as describing all data-dependent orderings of communications. The process successively chooses one of the enabled methods, executes it (thereby changing the state space) and afterwards starts again. Consider a CSP-OZ class C I P Z
[interface] [CSP part] [OZ part]
also written horizontally as U = spec I L P Z end with an OZ part Z st : State Init(st) ...com c(st, in?, out!, st )...
[state space] [initial condition] [one communication schema for each c in I]
where the notation com c(st, in?, out!, st ) indicates that this communication schema for c relates the state st to the successor state st and has input parameters in? and output parameters out!. Transformation. The OZ part of the class is transformed into a CSP process OZMain defined by the following system of (parametrised) recursive equations for OZpart using the (indexed) CSP operators for internal nondeterministic choice () and alternative composition (2): OZMain = OZPart(st) =
st OZPart(st) 2c, in? out!, st
c.in?.out! → OZPart(st )
where st ranges over all states in State : Exp satisfying Init(st). Thus the process OZMain can nondeterministically choose any state st satisfying Init(st) to start
Specification and Inheritance in CSP-OZ
369
with. Further on, c ranges over all channels declared in I , and in? ranges over the set Inputs(c) such that the precondition of the communication schema for c holds, i.e. ∃ out! : Outputs(c); st : State • com c(st, in?, out!, st ). Finally, for any chosen c and in?, the value out! ranges over the set Outputs(c), and st ranges over State such that com c(st, in?, out!, st ) holds. So the OZPart(st) is ready for every communication event c.in?.out! along a channel c in I where for the input values in? the communication schema com c(st, in?, out!, st ) is satisfiable for some output values out! and successor state st . For given input values in? any such out! and st can be nondeterministically chosen to yield c.in?.out! and the next recursive call OZPart(st ). Thus input and output along channels c are modelled by a subtle interplay of the CSP alternative and nondeterministic choice. Semantics of a class. A CSP-OZ class defines a template for all of its instances. The semantics of an instance is derived from the semantics of its class by filling in an instance name. Using parallel composition we define the semantics of a class in the failures divergence model: FD[[C ]] = FD[[ P ||{commonEvents} ZMain ]] D = FD[[P ]] ||F {commonEvents} FD[[Main]] D where ||F denotes the semantic counterpart in the failure divergence model A of the parallel composition ||A with synchronisation on the event set A (see e.g. [38]).
Semantics of an instance. Suppose an instance o of class C is declared by o : C . Then o denotes a process which is obtained from FD[[C ]] by applying a renaming operator that renames every channel ch in the interface I of C into ch.o. Thus events ch.d for some data element d become now ch.o.d as described by the following definition due to [12]: FD[[o]] = FD[[ C [{ch : Chans; d : Data | ch.d ∈ I • ch.d → ch.o.d }] ]] Refinement compositionality. By the above process semantics of CSP-OZ, the refinement notion F D is immediately available for CSP-OZ. In [18] it has been shown that CSP-OZ satisfies the principle of refinement compositionality, i.e. refinement of the parts implies refinement of the whole. Formally: – Process refinement P1 F D P2 implies refinement in CSP-OZ: spec I P1 Z end F D spec I P2 Z end – Data refinement Z1 ρ Z2 for a refinement relation ρ implies refinement in CSP-OZ: spec I P Z1 end F D spec I P Z2 end
370
4
E.-R. Olderog and H. Wehrheim
Inheritance
Process refinement P F D Q in CSP stipulates that P and Q have the same interface. Often one wishes to extend communication capabilities of a process or the operation capabilities of a class. This can be specified using the notion of inheritance, a syntactic relationship on classes: a superclass (or abstract class) A is extended to a subclass (or concrete class) C . The subclass should inherit the parts of the superclass. In CSP-OZ this is denoted as follows. Given a superclass A of the form A IA PA ZA
[interface] [CSP part] [OZ part]
we obtain a subclass C of A by referring to A using the inherit clause: C inherit A IC PC ZC
[superclass] [interface] [CSP part] [OZ part]
The semantics of the inherit operator is defined in a transformational way, i.e. by incorporating the superclass A into C yielding the following expanded version of C : C I P Z
[interface] [CSP part] [OZ part]
where I = IA ∪ IC and P is obtained from PA and PC by parallel composition and Z is obtained from ZA and ZC by schema conjunction. More precisely, to obtain the CSP part P we first replace in PA and PC the process identifiers main by new identifiers mainA and mainC respectively, then collect the resulting set of process equations, and add the equation c
main = mainA ||{commonEvents} mainC modelling parallel composition of PA and PC . To obtain the OZ part Z the corresponding schema of ZA and ZC are conjoined: State = StateA ∧ StateC , Init = InitA ∧ InitC , com c = com cA ∧ com cC for all channels c in IA ∩ IC , com c = com cA for all channels c in IA \ IC , com c = com cC for all channels c in IC \ IA .
Specification and Inheritance in CSP-OZ
371
Note that a subclass constructed in this way is not necessarily a refinement of its superclass.
5
Inheritance of Properties
In this section we study preservation of properties under inheritance. The scenario we are interested in is the following: suppose we have a superclass A for which we have already verified that a certain property P holds. Now we extend A to a subclass C and like to know under which conditions P also holds for C . This would allow us to check the conditions on the subclass and avoid re-verification. In principle we are thus interested in a reasoning which is similar to that used for refinement in CSP. In CSP, properties are preserved under refinement, i.e. P F D A ∧ A F D C ⇒ P F D C holds. If inheritance is employed instead of refinement this is in general not true. Inheritance allows to modify essential aspects of a class and thus may destroy every property proven for the superclass. We thus have to require a closer relationship between super- and subclass in order to achieve inheritance of properties. A relationship which guarantees a certain form of property inheritance is behavioural subtyping [31]. Originally studied in state-based contexts, this concept has recently been extended to behaviour-oriented formalisms (see [15]) and is thus adequate for CSP-OZ with its failure divergence semantics. Behavioural subtyping guarantees substitutability while also allowing extension of functionality as introduced by inheritance. In the following we will look at two forms of behavioural subtyping and show that one of them preserves safety and the other also a form of liveness properties. 5.1
Safety: Trace Properties
Since CSP-OZ has a failure-divergence semantics we use the CSP way of property specification. In CSP, properties are formalised by CSP processes. A property holds for a class A if the class refines the property. Since CSP offers different forms of refinement there are also different forms of satisfaction. We say that a class satisfies a property with respect to safety issues if it is a trace refinement of the property; when failure-divergence refinement is used a (limited form of) liveness is checked (see next section). Definition 2. Let A be a class and P a CSP property (process). A satisfies the trace property P (or A satisfies P in the trace semantics) iff traces(A) ⊆ traces(P ) (or equally P T A). We illustrate this by means of the cash point example. Consider the following class A0 with a behaviour as specified in Figure 1. For reasons of readability we only consider a very simple form of till.
372
E.-R. Olderog and H. Wehrheim A0
R idSucc
stop
? withdraw
pay
R
Fig. 1. The simple till A0 .
We want to specify that money is only paid out after the correct PIN code has been entered. As a CSP property process this gives: Seq = idSucc → pay → Seq Safe = Seq ||| CHAOS ( Σ\{idSucc, pay} ) Here the process CHAOS (S ), where S is a set of events, is the chaotic process which can always choose to communicate as well as refuse events of S . It is ev → CHAOS (S )). A0 satisfies the defined by CHAOS (S ) = STOP ( ev ∈S trace property Safe since Safe T A0 . Next we like to know whether such a trace property P can be inherited to a subclass (or more specifically, a subtype) C . As a first observation we notice that C potentially has traces over a larger alphabet than A since it can have a larger interface. This might immediately destroy the holding of a trace property. Nevertheless, a trace property might still hold in the sense that, as far as the operations of A are concerned, the ordering of operations as specified in P also hold in C . Definition 3. Let A, C be classes with α(A) ⊆ α(C ). C satisfies a trace property P w.r.t. α(A) iff traces(C ) ↓ α(A) ⊆ traces(P ) ↓ α(A). Here α(A) denotes the alphabet of A, i.e. the set of events A may communicate, and ↓ denotes projection. The question we ultimately like to answer can now be precisely formulated: if A satisfies P , does C satisfy P wrt. α(A)? This is in fact the case when C is a trace subtype of A. Definition 4. Let A, C be classes with α(A) ⊆ α(C ) and N = α(C )\α(A). Then C is a trace subtype of A, abbreviated A tr −st C , iff A ||| CHAOS (N ) T C .
Specification and Inheritance in CSP-OZ
373
Intuitively, parallel composition with the chaotic process over the set of methods N says that C may have new methods in addition to A and these can at any time be executed as well as refused but have to be independent from (interleaved with) the A-part. Safety properties are inherited to trace subtypes. Theorem 5. Let A, C be processes with A tr −st C , let P be a process formalising a trace property. If A satisfies the trace property P then C satisfies P w.r.t. α(A). As an example we look at an extension of class A0 . The till C0 depicted in Figure 2 extends A0 with a facility of viewing the current balance of the account. C0 is a trace subtype of A0 and thus inherits property Safe, i.e. C0 satisfies Safe wrt. α(A0 ). A0
C0
R
R
idSucc
idSucc
stop display
? withdraw
pay
R
stop
-?
view
withdraw
pay
R
Fig. 2. A class and a trace subtype.
This completes the section on inheritance of safety properties. Next we study liveness properties. 5.2
“Liveness”: F D Properties
Liveness properties are checked in CSP by comparing the property process and the class with respect to their failure divergence set, i.e. checking whether the class is a failure divergence refinement of the property. This yields a form of bounded liveness check: it can be proven that methods can be refused or conversely, are always enabled after certain traces. Unbounded liveness, as expressible in temporal logic, cannot be specified in CSP. Definition 6. Let A be a class and P a CSP property (process). A satisfies the liveness property P (or A satisfies P in the FD semantics) iff FD[[A]] ⊆ FD[[P]] (or equally P F D A).
374
E.-R. Olderog and H. Wehrheim
We illustrate this again on our till example. The property we like to prove for class A0 concerns service availability: Money can be withdrawn immediately after the PIN code has been verified. Formalised as a CSP property this is: Live = idSucc → withdraw → Live pay → Live stop → Live Class A0 satisfies the liveness property Live since Live F D A0 . Analogous to trace properties we now first have to define what preservation of a liveness property to a subclass should mean. Again, we have to face the fact that the subclass has additional functionality and thus failures and divergences range over a larger alphabet. In principle we apply the same technique as for trace properties. We project the failures and divergences of the subclass down to the alphabet of the superclass and on this projection carry out the comparison with the property P . Definition 7. Let A, C be classes with α(A) ⊆ α(C ). C satisfies a liveness property P w.r.t. α(A) iff ∀(s, X ) ∈ failures(C ) ∃(t, Y ) ∈ failures(P ) : s ↓ α(A) = t ↓ α(A) ∧ X ∩ α(A) = Y ∩ α(A) and ∀ s ∈ divergences(C ) ∃ t ∈ divergences(P ) : s ↓ α(A) = t ↓ α(A) . Liveness properties can be shown to be inherited to classes which are optimal subtypes of the superclass. Definition 8. Let A, C be classes with α(A) ⊆ α(C ) and N = α(C )\α(A). Then C is an optimal substype of A, abbreviated A ost C , iff A ||| CHAOS (N ) F D C . This definition lifts the idea of trace subtyping to the failure divergence semantics: in class C anything ”new” is allowed in parallel with the behaviour of A as long as it does not interfere with this ”old” part. Looking at class C0 in comparison with A0 we find that C0 is not an optimal subtype of A0 . For instance, C0 has the pair (idSucc view , {Σ \ {display}) in its failure set for which projected down to the alphabet of A0 no corresponding pair can be found in failures(A0 ) (the crucial point is refusal of withdraw ). Class C1 as depicted in Figure 3 on the other hand is an optimal subtype of class A0 and it indeed inherits property Live. Theorem 9 (Wehrheim). Let A, C be classes with A ost C , let P be a process formalising a liveness property. If A satisfies P in the FD semantics, then C satisfies P w.r.t. α(A) in the FD semantics.
Specification and Inheritance in CSP-OZ A0
375
C1
R
R
idSucc
idSucc
stop
?
?
test withdraw
stop
pay
withdraw
R
pay
R Fig. 3. A class and an optimal subtype.
stop
english
idSucc
? pay
withdraw
C2
german
german
R idSucc
stop
?
english
withdraw
pay
R
Fig. 4. Another optimal subtype.
A more complex extension of the simple till which is also an optimal subtype is shown in Figure 4. The till allows to switch between different languages but switching does not effect the basic functionality. Since C2 is an optimal subtype of A0 we get by the previous theorem that C2 satisfies Live.
6
Conclusion
In this paper we took the combined specification formalism CSP-OZ to define and study the inheritance of properties from superclasses to subclasses. Semantically, classes, instances, and systems in CSP-OZ denote processes in the standard failures divergence model of CSP. This allowed us to make full use of the well established mathematical theory of CSP. In case of systems with finite state CSP parts and finite data types in the OZ parts the FDR model-checker for CSP can be applied to automatically verify refinement relations between CSPOZ specifications [14] and subtyping relations [51]. Optimal subtyping is a strong requirement for subclasses to satisfy. As future work it would be interesting to study conditions under which specific properties are inherited. Related Work. A number of other combinations of process algebra with formalisms for describing data exist today. A comparison of approaches for com-
376
E.-R. Olderog and H. Wehrheim
bining Z with a process algebra can be found in [17]. Such integrations include Timed CSP and Object-Z (TCOZ) [32], B and CSP [6] and Z and CCS [47,19]. A similar type of integration which has been developed much earlier than these approaches is LOTOS [2]. LOTOS adopts a number of operators from CCS and CSP. The first design of LOTOS contained ACT-One for describing data, a language for specifying abstract data types. This proved to be unacceptable for industrial users, and the newest development called E-Lotos [26] (standardised by ISO) uses instead elements from functional languages. Closest to the combination CSP-OZ is Object-Z/CSP due to Smith [43]. There, CSP operators serve to combine Object-Z classes and instances. Thus to Object-Z classes a semantics in the failures divergence model of CSP is assigned just like done here for CSP-OZ. This semantics is obtained as an abstraction of the more detailed history semantics of Object-Z [42]. In contrast to CSP-OZ there is no CSP-part inside of classes. As we have seen in the example, the CSP part is convenient for specifying sequencing constraints on the communications events. Both CSP-OZ and Object-Z/CSP have been extended to deal with realtime [24,45]. The issue of inheritance of properties to subtypes has been treated by van der Aalst and Basten [48]. They deal with net-specific properties like safety (of nets), deadlock freedom and free choice. Leavens and Weihl [29] show how to verify object-oriented programs using a technique called “supertype abstraction”. This technique is based on the idea that subtypes need not to be re-verified once a property has been proven for their supertypes. In their study they have to take particular care about aliasing since in object-oriented programs several references may point to the same object, and thus an object may be manipulated in several ways. Subtyping for objectoriented programs has to avoid references which are local to the supertype but accessible in the subtype. Preservation of properties is also an issue in transformations within the language UNITY proposed by Chandy and Misra [7]. The superposition operator in Unity is a form of parallel composition which requires that the new part does not make assignments to underlying (old) variables. This is close to the non-modification condition we used in one pattern. Superposition preserves all properties of the original program.
References 1. P. America. Designing an object-oriented programming language with behavioural subtyping. In J.W. de Bakker, W.P. de Roever, and G. Rozenberg, editors, REX Workshop: Foundations of Object-Oriented Languages, number 489 in LNCS. Springer, 1991. 2. T. Bolognesi and E. Brinksma. Introduction to the ISO specification language LOTOS. Computer Networks and ISDN Systems, 14:25–59, 1987. 3. J. Bredereke. Maintaining telephone switching software requirements. IEEE Communications Magazine, 40(11):104–109, 2002.
Specification and Inheritance in CSP-OZ
377
4. S.D. Brookes, C.A.R. Hoare, and A.W. Roscoe. A theory of communicating sequential processes. Journal of the ACM, 31:560–599, 1984. 5. A. Brucker and B. Wolff. A proposal for a formal OCL semantics in Isabelle/HOL. In Proc. International Conference on Theorem Proving in Higher Order Logics (TPHOLs), LNCS. Springer, 2002. 6. M. Butler. csp2B: A practical approach to combining CSP and B. In J. Wing, J. Woodcock, and J. Davies, editors, FM’99: Formal Methods, number 1708 in Lecture Notes in Computer Science, pages 490–508. Springer, 1999. 7. K.M. Chandy and J. Misra. Parallel Program Design – A Foundation. AddisonWesley, 1988. 8. W. Damm, B. Josko, A. Pnueli, and A. Votintseva. Understanding UML: A Formal Semantics of Concurrency and Commu nication in Real-Time UML. In F.S. de Boer, M. Bonsangue, S. Graf, and W.P. de Roever, editors, Formal Methods of Components and Objects (FMCO’02), LNCS. Springer, 2003. (this volume). 9. W. Damm and B. Westphal. Live and Let Die: LSC-based Verification of UMLModels. In F.S. de Boer, M. Bonsangue, S. Graf, and W.P. de Roever, editors, Formal Methods of Components and Objects (FMCO’02), LNCS. Springer, 2003. (this volume). 10. B. T. Denvir, J. Oliveira, and N. Plat. The Cash-Point (ATM) ‘Problem’. Formal Aspects of Computing, 12(4):211–215, 2000. 11. R. Duke, G. Rose, and G. Smith. Object-Z: A specification language advocated for the description of standards. Computer Standards and Interfaces, 17:511–533, 1995. 12. C. Fischer, E.-R. Olderog, and H. Wehrheim. A CSP view on UML-RT structure diagrams. In H. Husmann, editor, Fundamental Approaches to Software Engineering, volume 2029 of LNCS, pages 91–108. Springer, 2001. 13. C. Fischer and G. Smith. Combining CSP and Object-Z: Finite or infinite tracesemantics? In T. Mizuno, N. Shiratori, T. Higashino, and A. Togashi, editors, Proceedings of FORTE/PSTV’97, pages 503–518. Chapmann & Hall, 1997. 14. C. Fischer and H. Wehrheim. Model-checking CSP-OZ specifications with FDR. In K. Araki, A. Galloway, and K. Taguchi, editors, Integrated Formal Methods, pages 315–334. Springer, 1999. 15. C. Fischer and H. Wehrheim. Behavioural subtyping relations for object-oriented formalisms. In T. Rus, editor, AMAST 2000: International Conference on Algebraic Methodology And Software Technology, number 1816 in Lecture Notes in Computer Science, pages 469–483. Springer, 2000. 16. C. Fischer. CSP-OZ: A combination of Object-Z and CSP. In H. Bowman and J. Derrick, editors, Formal Methods for Open Object-Based Distributed Systems (FMOODS’97), volume 2, pages 423–438. Chapman & Hall, 1997. 17. C. Fischer. How to combine Z with a process algebra. In J. Bowen, A. Fett, and M. Hinchey, editors, ZUM’98 The Z Formal Specification Notation, volume 1493 of LNCS, pages 5–23. Springer, 1998. 18. C. Fischer. Combination and Implementation of Processes and Data: From CSPOZ to Java. PhD thesis, Bericht Nr. 2/2000, University of Oldenburg, April 2000. 19. A. J. Galloway and W. Stoddart. An operational semantics for ZCCS. In M. Hinchey and Shaoying Liu, editors, Int. Conf. of Formal Engineering Methods (ICFEM). IEEE, 1997. 20. J. Hatcliff and M. Dwyer. Using the Bandera tool set to model-check properties of concurrent Java software. In K.G. Larsen, editor, CONCUR 2001, LNCS. Springer, 2001.
378
E.-R. Olderog and H. Wehrheim
21. S. Helke and T. Santen. Mechanized analysis of behavioral conformance in the Eiffel base libraries. In M. Butler, L. Petre, and K. Sere, editors, Proceedings of the FME 2001, LNCS. Springer, 2001. 22. C.A.R. Hoare. Communicating sequential processes. CACM, 21:666–677, 1978. 23. C.A.R. Hoare. Communicating Sequential Processes. Prentice Hall, 1985. 24. J. Hoenicke and E.-R. Olderog. Combining specification techniques for processes, data and time. In M. Butler, L. Petre, and K. Sere, editors, Integrated Formal Methods (IFM 2002), volume 2335 of LNCS, pages 245–266. Springer, 2002. 25. M. Huisman and B. Jacobs. Java Program Verification via a Hoare Logic with Abrupt Termination. In T. Maibaum, editor, Fundamental Approaches to Software Engineering (FASE 2000), volume 1783 of LNCS, pages 284–303. Springer, 2000. 26. ISO. Final comittee draft on enhancements to LOTOS. ISO/IEC JTC1/SC21, WG7 Enhancements to LOTOS, 1998. ftp://ftp.dit.upm.es/pub/lotos/elotos/Working.Docs/. 27. Kolyang. HOL-Z – An Integrated Formal Support Environment for Z in Isabelle/HOL. PhD thesis, Univ. Bremen, 1997. Shaker Verlag, Aachen, 1999. 28. D. Latella, I. Majzik, and M. Massink. Automatic verification of a behavioural subset of UML statechart diagrams using the SPIN model-checker. Formal Aspects of Computing, 11:430–445, 1999. 29. G.T. Leavens and W.E. Weihl. Specification and verification of object-oriented programs using supertype abstraction. Acta Informatica, 32:705–778, 1995. 30. K. R. M. Leino. Extended static checking: A ten-year perspective. In R. Wilhelm, editor, Informatics – 10 Years Back, 10 Years Ahead, volume 2000 of LNCS, pages 157–175. Springer, 2001. 31. B. Liskov and J. Wing. A behavioural notion of subtyping. ACM Transactions on Programming Languages and Systems, 16(6):1811 – 1841, 1994. 32. B. P. Mahony and J.S. Dong. Blending Object-Z and Timed CSP: An introduction to TCOZ. In The 20th International Conference on Software Engineering (ICSE’98), pages 95–104. IEEE Computer Society Press, April 1998. 33. A. Mota and A. Sampaio. Model-checking CSP-Z: strategy, tool support and industrial application. Science of Computer Programming, 40(1), 2001. 34. O. Nierstrasz. Regular types for active objects. In O. Nierstrasz and D. Tsichritzis, editors, Object-oriented software composition, pages 99 – 121. Prentice Hall, 1995. 35. E.-R. Olderog and C.A.R. Hoare. Specification-oriented semantics for communicating processes. Acta Inform., 23:9–66, 1986. 36. A. Poetzsch-Heffter and J. Meyer. Interactive verification environments for objectoriented languages. Journal of Universal Computer Science, 5(3):208–225, 1999. 37. A.W. Roscoe. Model-checking CSP. In A.W. Roscoe, editor, A Classical Mind — Essays in Honour of C.A.R.Hoare, pages 353–378. Prentice-Hall, 1994. 38. A.W. Roscoe. The Theory and Practice of Concurrency. Prentice-Hall, 1997. 39. M. Saaltink. The Z/EVES system. In J. Bowen, M. Hinchey, and D. Till, editors, ZUM’97, volume 1212 of LNCS, pages 72–88. Springer, 1997. 40. T. Sch¨ afer, A. Knapp, and S. Merz. Model Checking UML State Machines and Collaborations. In Workshop on Software Model Checking, volume 55 of ENTCS, 2001. 41. G. Smith, F. Kamm¨ uller, and T. Santen. Encoding Object-Z in Isabelle/HOL. In D. Bert, J.P. Bowen, M.C. Henson, and K. Robinson, editors, ZB 2002: Formal Specification and Development in Z and B, volume 2272 of LNCS, pages 82–99. Springer, 2002. 42. G. Smith. A fully abstract semantics of classes for Object-Z. Formal Aspects of Computing, 7:289–313, 1995.
Specification and Inheritance in CSP-OZ
379
43. G. Smith. A semantic integration of Object-Z and CSP for the specification of cocurrent systems. In J. Fitsgerald, C.B. Jones, and P. Lucas, editors, Foraml Methods Europe (FME’97), volume 1313 of LNCS, pages 62–81. Springer, 1997. 44. G. Smith. The Object-Z Specification Language. Kluwer Academic Publisher, 2000. 45. G. Smith. An integration of real-time Object-Z and CSP for specifying concurrent real-time systems. In M. Butler, L. Petre, and K. Sere, editors, Integrated Formal Methods (IFM 2002), volume 2335 of LNCS, pages 267–285. Springer, 2002. 46. J.M. Spivey. The Z Notation: A Reference Manual. Prentice-Hall International Series in Computer Science, 2nd edition, 1992. 47. K. Taguchi and K. Araki. Specifying concurrent systems by Z + CCS. In International Symposium on Future Software Technology (ISFST), pages 101–108, 1997. 48. W.M.P. van der Aalst and T. Basten. Inheritance of Workflows – An approach to tackling problems related to change. Theoretical Computer Science, 270(1-2):125– 203, 2002. 49. H. Wehrheim. Specification of an automatic manufacturing system – a case study in using integrated formal methods. In T. Maibaum, editor, FASE 2000: Fundamental Aspects of Software Engineering, number 1783 in LNCS, pages 334–348. Springer, 2000. 50. H. Wehrheim. Behavioural subtyping in object-oriented specification formalisms. University of Oldenburg, Habilitation Thesis, 2002. 51. H. Wehrheim. Checking behavioural subtypes via refinement. In A. Rensink B. Jacobs, editor, FMOODS 2002: Formal Methods for Open Object-Based Distributed Systems, pages 79–93. Kluwer, 2002. 52. J. Woodcock and J. Davies. Using Z — Specification, Refinement, and Proof. Prentice-Hall, 1996.
Model-Based Testing of Object-Oriented Systems Bernhard Rumpe IRISA-Université de Rennes 1, Campus de Beaulieu, Rennes, France and Software & Systems Engineering, TU München, Germany
Abstract. This paper discusses a model-based approach to testing as a vital part of software development. It argues that an approach using models as central development artifact needs to be added to the portfolio of software engineering techniques, to further increase efficiency and flexibility of the development as well as quality and reusability of results. Then test case modeling is examined in depth and related to an evolutionary approach to model transformation. A number of test patterns is proposed that have proven helpful to the design of testable object-oriented systems. In contrast to other approaches, this approach uses explicit models for test cases instead of trying to derive (many) test cases from a single model.
1 Portfolio of Software Engineering Techniques Software has become a vital, but often invisible part of our lives. Embedded forms of software are part of almost any technical device. The average household uses several computers, and the internet and telecommunication world has considerably changed our lives. Software is used for a variety of jobs. It can be as small as a simple script or as complex as an entire operating or enterprise resource planning system. For the near future, we can be rather sure that we will not have a single notation or process that can cover the diversity of today’s development projects. Projects are too different in their application domain, size, need for reliability, time-to-market pressure, and the skills and demands of the project participants. Even the UML [OMG02], which is regarded as a de-facto standard, is seen as a family of languages rather than a single notation and by far doesn’t cover all needs. This leads to an ongoing proliferation of methods, notations, principles, techniques and tools in the software engineering domain that is at least partly influenced from practical applications of Formal Methods. On the one hand, methods like Extreme Programming [Bec99] and Agile Software Development [Coc02] even discourage the long well known distinction between analysis, design and implementation activities and abandon all documentation activities in favor of rigorous test suites. On the other hand, upcoming development tools allow to generate increasing amounts of code from UML models, thus supporting the OMG’s initiative on “Model Driven Architecture” (MDA) [OMG01]. MDA’s primary purpose is to decouple platform-independent models from platform-specific, F.S. de Boer et al. (Eds.): FMCO 2002, LNCS 2852, pp. 380–402, 2003. © Springer-Verlag Berlin Heidelberg 2003
Model-Based Testing of Object-Oriented Systems
381
technical information. This should increase the reusability of both. Code generation, however, is today focusing pretty much on the generation of the productive system. Generation of test code from models is still a side issue. In particular the question, what a good test model should look like, is to be examined in this paper. In general, we can observe that in the foreseeable future we will have a portfolio of software engineering techniques that enables developers and managers to select appropriate processes and tools for their projects. Today, however, it is not quite clear which elements the portfolio should have, how they relate, when they are applicable, and what their benefits and drawbacks are. The software and systems engineering community therefore must reconsider and extend its portfolio of software engineering techniques incorporating new ideas and concepts, but also try to scientifically assess the benefits and limits of new approaches. For example: • Lightweight projects that don’t produce requirement and design documentation need intensive communications and can hardly be split into independent subprojects. Thus they don’t scale up to large projects. But where are the limits? A guess is, around 10 people, but there have been larger projects reportedly “successful” [RS02]. • Formal methods have built a large body of knowledge (see for example [ABRS03,JKW03,LCCRC03]), but how can this knowledge successfully and in a goal-oriented way be applied in today’s projects? A guess seems to be, formal methods apply best, if embodied in practical tools, using practical and well known notations without exposing the user directly to the formal method. • Product reliability often need not be 100% for all developments and in the first iteration already. But how to predict reliability from project metrics and how to adapt the project to increase reliability and accuracy to the desired level while minimizing the project/product costs? Thus in contrast to applying formal methods for verification purposes, the use of formal techniques for test case specification and metrics of test coverage does not give 100% reliability, but in practice has a much better cost/benefit ratio. Based on this observation we will in the following examine the modeling of test cases using several of the UML-like notations, arguing that this technique should be a new item in the broad portfolio of SE techniques. For this purpose, we develop our interpretation of the used UML notation in the context of test case modeling, which should give us the justification to regard the used notation as being backed up by a formal technique (without explicitly referring to that formal method). Section 2 discusses synergies and problems of using models for a variety of activities, including programming. Section 3 establishes the general needs of successful testing strategies. In Section 4 the scenario of a model-based test approach is discussed. Sections 5 and 6 present several test patterns that are useful to make an object-oriented design testable. While Section 5 concentrates on basic test patterns, Section 6 presents more specific test patterns for distributed systems. Section 7 finally discusses the benefits of an evolutionary approach to modeling in combination with an intensive, model-based test approach. In particular, the usability of tests as invariant observations for model-transformations is explored. For sake of conceptual discussions, technical details are omitted, but can be found in [Rum03].
382
B. Rumpe
2 Modeling Meets Programming UML [OMG02] undoubtedly has become the most popular modeling language for software intensive systems used today. Models can be used for quite a variety of purposes. Besides informal sketches that are used for communication, e.g. by being drawn on paper and posted on a wall the most common are: • Semi-precisely defined diagrams are used for documentation of that part of the requirements that is not written in plain English. • Architecture and designs are captured and documented with models. In practice, these models are increasingly often used for code generation. More sophisticated and therefore less widespread uses of models are analysis of certain quality attributes (such as message throughput, responsiveness or failure likelihood) or development of tests from models. Many UML-based tools today offer functionality to directly simulate models or generate at least parts of the code. As tool vendors work hard on continuous improvement of this feature, this means a sublanguage of UML will become a high-level programming language and modeling at this level becomes identical to programming. This raises a number of interesting questions: • Is it critical for a modeling language to be also used as programming language? For example analysis and design models may become overloaded with details that are not of interest yet, because modelers are addicted to executability. • Is a future version of the UML expressive enough to describe systems completely or will it be accompanied by conventional languages? How well are these integrated? • How will the toolset of the future look like and how will it overcome round trip engineering (i.e. mapping code and diagrams in both directions)? • What implications does an executable UML have on the development process? In [Rum03,Rum02] we have discussed these issues and have demonstrated, how the UML in combination with Java may be used as a high-level programming language. But, UML cannot only be used for modeling the application, but more importantly for modeling tests on various levels (class, integration, and system tests) as well. For this purpose we need executable test models, as testing is in general the process of executing a program with the intention to identify faults [Mye79,Bin99]. Executable models are usually less abstract than design models, but they are still more compact and abstract than the implementation. The same holds for test models versus manually implemented tests. One advantage of using models for test case description is that application specific parts are modeled with UML-diagrams and technical issues, such as connection to frameworks, error handling, persistence, or communication are handled by the parameterized code generator. This basically allows us to develop models that are independent of any technology and platform, as for example proposed in [SD00]. Only during the generation process platform dependent elements are added. When the technology changes, we only need to update the generator, but the application defining models as well as test models can directly be reused. This concept also directly
Model-Based Testing of Object-Oriented Systems
383
supports the above mentioned MDA-Approach [Coc02] of the OMG. Another important advantage is that both, the production code and automatically executable tests at any level, are modeled by the same UML diagrams. Therefore, developers use a single homogeneous language to describe implementation and tests. This will enhance the availability of tests already at the beginning of the coding activities and leads to a development method similar to the “test first approach” [Bec01,LF02]. Some of the UML-models (mainly class diagrams and statecharts) are used constructively, others are used for test case definition (mainly OCL, sequence and enhanced object diagrams). Fig. 1 illustrates the key mappings. object diagrams
statecharts
__:
class diagrams
__: sequence diagrams
Java
OCL
__:
WKLFNQHVVRIDUURZ LQGLFDWHVLPSRUWDQFH RIJHQHUDWLRQ
FRGHDQG WHVWFDVH JHQHUDWLRQ production code
test code
Fig. 1. Mapping of UML-models to code and test code.
As a consequence of the various possible forms of model use, we will identify the notion of diagram and model. Thus a model is a coherent piece of information, denoted in a diagrammatic or textual notation that describes an abstraction of the desired system. Multiple models can describe various aspects of the system. We also allow models to be composed of sub-models, as is the case with test models. This is in slight contrast to approaches, where a model is a virtual something in a tool and the user manipulates it indirectly through (diagrammatic) views. The latter approach has shown some difficulties when using models for test case definition.
3 Testing Strategies Not only [Bin99,Mye79] show that there is a huge variety of testing strategies and testing goals. While tests in the small usually try to identify faults, tests suites and coverage metrics can be used to estimate the quality of the system in terms of absence of faults. The “success” of a test can therefore be seen twofold, but we follow the
384
B. Rumpe
general line and define a test to fail, when an abnormal behavior (failure) shows that there is at least one fault in the system. This means the test and the system do not fit together. Testing can be done manually or automated. The widespread use of JUnit [BG99] shows that the use of automated tests has gained considerable attention in recent years, because it allows to “reuse” tests in form of regression tests on evolving systems without actually knowing what the test does. This allows very small iterations with continuous integration and the use of refactoring techniques [Fow99] to improve the code structure. Automated tests ensure a low defect rate and continuous progress, whereas manual tests would very rapidly lead to exhausted testers. To summarize the characteristics of tests we are aiming at: • Tests run the system – in contrast to static analyses. • Tests are automatic to prevent project members to get bored with tests (or alternatively to prevent a system that isn’t tested enough) • Automated tests build the test data, run the test and examine the result automatically. Success resp. failure of the test are automatically observed during the test run. • A test suite also defines a system that is running together with the tested production system. The purpose of this extended system is to run the tests in an automated form. • A test is exemplaric. A test uses particular values for the input data, the test data. • A test is repeatable and determined. For the same setup the same results are produced. In particular the last point is tricky to achieve, when the system under test (SUT) is distributed or has side effects. Specific care has to be taken to deal with these situations. Faults may also occur without being observable, because they are hidden in the SUT or cannot be traced in a distributed system. That is why the TTCN standard [TTCN92] also allows “inconclusive” and “none” as test results. We instead strongly demand that systems need to be built in a way that testing can be properly achieved. At least in the domain of object-oriented software systems this is a realistic demand. After discussing the structure of a model-based test in the following section, we will discuss, how object-oriented systems should be designed to assist automated testing.
4 Model-Based Testing The use of models for the definition of tests and production code can be manifold: • Code or at least code frames can be generated from a design model. • Test cases can be derived from an analysis or design model that is not used/usable for constructive generation of production code. For example behavioral models, such as statecharts, can be used to derive test cases that cover states, transitions or even larger subsets of its paths. • The models itself can be used to describe test cases or at least some part thereof.
Model-Based Testing of Object-Oriented Systems
test data
o1 o3
expected result and/or OCL-contract as test oracle
test driver
o2
OD
SD or method call o2
o1
objects under test o5
o4
385
o3
OD
o4
+
OCL
Fig. 2. Structure of a test modeled with object diagrams (OD), sequence diagram (SD) and the Object Constraint Language (OCL).
The first two uses are already discussed e.g. in [Rum02] and [BL01]. Therefore, in this section we concentrate on the development of models that define tests. A typical test, as shown in Fig. 2 consists of a description of the test data, the test driver and an oracle characterizing the expected test result. In object-oriented environments, test data can usually be described by an object diagram (OD). The object diagram in Fig. 3 shows the necessary objects as well as concrete values for their attributes and the linking structure. OD Ready a1213:Auction
…
long auctionIdent = 1213 String title= “Copper 413tons“ /int numberOfBids = 0
timePol:ConstantTimingPolicy int status = TimingPolicy.READY_TO_GO boolean isInExtension = false int extensionTime = 180
Fig. 3. Object diagram (OD) describing the test data as a particular situation in an online auction, that has not yet started (see the full example in [Rum03,Rum03b]).
The test driver can be defined using a simple method call or, if more complex, modeled by a sequence diagram (SD). An SD has the considerable advantage that not only the triggering method calls can be described, but it is possible to model desired interactions and check object states during the test run. For this purpose, the Object Constraint Language (OCL, [WK98]) is used. In the sequence diagram in Fig. 4, an OCL constraint at the bottom ensures that the new closing time of the auction is set to the time when the bid was submitted (bid.time) plus the extension time to allow competitors to react (the auction system using this structure is partly described in [Rum03, Rum03b]). It is effective to model the test oracle using a combination of an object diagram and reuse globally valid OCL properties. The object diagram in this case serves as a property description and can therefore be rather incomplete, just focusing on desired effects. The OCL constraints can also be taken from the set of general invariants or can be defined as specific properties. In practice, it turns out that there is a high amount of reuse possible through the following techniques:
386
B. Rumpe SD copper: Auction «trigger» handleBid(bid)
:BidPolicy
:TimingPolicy
validateBid(bid) return OK
test driver getNewClosingTime(bid) return t
OCL constraints describe properties during the test run
t.time == bid.time + extensionTime
Fig. 4. A sequence diagram (SD) describing the trigger of a test driver and predicts some interactions as well as an OCL property that holds at that point in the test.
•
Well prepared test data can be reused for many tests. There is often only a handful of basic test structures necessary. From those the specifically desired test data can be derived by small adaptations, e.g. replacing a single attribute value or adding a certain object. Having an explicit, graphically depicted basic model of the data structure at hand increases its reusability for specific adaptations. • Test data can be composed of several object diagrams, describing different parts of the data. • Test oracles can be defined using a combination of an object diagram and reuse globally valid OCL properties. The resulting object diagram can be rather small, describing deltas only and can be derived from the test data diagram. The OCL properties can be reused as they are usually globally valid invariants. As already mentioned, being able to use the same, coherent language to model the production system and the tests allows for a good integration between both tasks. It allows the developer to immediately define tests for the constructive model developed. It is therefore feasible that in a kind of “test-first modeling approach” the test data in form of possible object structures is developed before the actual implementation.
5 Test Pattern In the last few years a number of Agile Methods have been defined that share a certain kind of characteristics, described in [AM03]. Among these Extreme Programming (XP) [Bec99] is the most widely used and discussed method. One of the most important XP characteristics is that it uses automated tests at all stages. Practical experience shows that when this is properly done, the defect rate is considerably low [RS02]. Furthermore, automation allows to repeat tests continuously in form of regression tests. Thus the quality of the result is ensured through strong emphasis on testing activities, ideally on development of the tests before the production code (“test
Model-Based Testing of Object-Oriented Systems
387
first approach” [LF02]). When using UML models for test design the development project should become even more efficient. However, practical experience shows that there are a number of obstacles that need to be overcome to enable model-based testing. In particular, there are object-oriented architectures that exhibit serious problems that prevent tests. It is therefore important to identify those problems and offer appropriate and effective solutions. In the remainder of this section, we provide several solutions for a number of problems that typically occur and that we also experienced e.g. in the online auction system. These solutions are defined in form of applicable test patterns similar to the design patterns of [GHJV94]. Indeed, some of the test patterns are based on design patterns, such as singleton, adapter or factory to achieve a testable design. Unlike [Bin99] we only provide the essential structure and a short explanation of the patterns in this article and refer to [Rum03] to a more detailed description. A test pattern description typically consists of several parts, describing intention, how to apply, the resulting structure, example implementations and a discussion of the pros and cons. Often the structure itself appears as a simple concept and it’s the method part, describing practical knowledge of its applicability that makes a pattern useful. auctions Auction *
1 «interface» BiddingPolicy
BidPolDummy
participants
1 «interface» TimingPolicy
TimePolDummy
CD
bidder Person *
PersonDummy
dummies replace ordinary objects through subclassing
Fig. 5. Class diagram showing how dummies are added to the system.
Dummies for the Test Context It has become a primary testing technique to use dummies (also “stubs” or “mocks”) to replace parts of the system and thus better expose the tested part to the test driver. Only object-oriented concepts, namely inheritance and dynamic binding (also known as polymorphic replacement of objects), comfortably allows to build object structures that are testable with dummies. Fig. 5 shows the principle in one class diagram that allows to isolate an auction object, by replacing its context completely by dummies. Sometimes a dummy just does nothing, but often it is also necessary to feed back specific values to keep the test going in the desired direction. Practical experience shows that this should normally not be achieved through various dummy-subclasses,
388
B. Rumpe
but through parameterized dummies, whose behavior during the test can be determined via constructor parameters. This for example allows to predefine and store results of queries given back to the calling SUT just to see what the SUTs reaction will be on that data. Remembering Interaction and Results A typical application of a dummy is to prevent side effects that a system otherwise has on the environment. Such side effects may affect files, the data base, the graphical user interface etc. An object responsible for logging activities that provides a method “write” may be replaced by a subclass object (say “LogDummy”) where the “write” method simply stores the line to write in a local attribute where it can be examined after the test. Sequence diagrams, however, already allow access to this kind of values during the test. Fig. 6 describes the effect of a test on the log object directly.
«Testclass» :AuctionTest
kupfer912: Auction
SD :ProtocolSimpleDummy
«trigger» start() «trigger» handleBid(bid1) «trigger» handleBid(bid2) «trigger» finish()
write(“Auction 912 opened at 14:00:00“)
write(“Bid 55.200 $US in Auction 912 from Theo accepted“)
effects of the test are captured in the method arguments
write(“Bid 55150 $US in Auction 912 from Joe accepted“)
write(“Auction 912 closed at 15:30:00“)
Fig. 6. Describing the effects for the log in a sequence diagram.
Static Elements As explained two concepts of object-oriented languages, namely inheritance and dynamic binding, allow to setup tests in a form that was not possible in procedural and functional languages. However, typical OO languages also provide concepts that make testing difficult. These are in particular static concepts, such as static attributes, static methods and constructors. Static attributes are mainly used to allow easy sharing of some global resources, such as the log object or database access. Static attributes, however, should be avoided anyway. If necessary e.g. for access to generally known objects, a static attribute can at least be encapsulated by a static method. The problem with a static method results from the inability to redefine it for testing purposes. For example if the static method “write” does have side effects, these cannot be prevented during a test. Unfortunately, there are often at least some static
Model-Based Testing of Object-Oriented Systems
389
methods necessary. We have therefore used the technique shown in Fig. 7 to provide a static interface to the customer and at the same time to allow the effect of the static method to be adaptable, through using an internal delegation to a singleton object that is stored in a static attribute. With proper encapsulation of the initialization of that attribute this is a safe and still efficient technique to make static methods replaceable by dummies without changing their signature. Thus the side effects of static methods can be prevented. CD
Singleton #Singleton singleton = null +initialize() #initialize(Singleton s) +method(Arguments) #doMethod(Arguments)
SingletonDummy #doMethod(Arguments)
singleton initialisation (default resp. with an object of a subclass) the (underlined) static method only calls the dynamic method „doMethod“ dynamic method contains the real functionality and can be redefined.
Fig. 7. Singleton object behind a static method.
Constructors With respect to testing purposes a constructor shares a number of similarities with static methods, because a constructor is not dynamically bound and can therefore not be redefined. Furthermore, a constructor creates new objects and thus allows the object under test to change its own environment. For example the tested object may have the idea to create its own log object and write the log through it. The standard solution for this problem is to force any object creation to be done by a factory. So instead “new class(arg)” a method “getClass(arg)” might be called. This method may be static using the approach above to encapsulate the factory object, but still make it replaceable for tests. A factory dummy can then create objects of appropriate subclasses that serve as dummies with certain predefined test behavior. In practice, we found it useful to model the newly created objects using object diagrams. The factory dummy that replaces the factory then doesn’t really create new objects, but returns one of the predefined objects each time it is called. Fig. 8 shows the factory part of a test data structure where three different person objects shall be “created” during the test. The data structure and the attribute values can be described using the same object diagram as is used when describing the basic test data structure. Further advantages are (1) that the newly “created” objects are known by name and can thus easily be checked after the test, even if the objects were disposed during the test and (2) the order of creation can also be checked.
390
B. Rumpe OD
:TestFactoryDummy index=0
:PersonBidderDummy
…
personIdent = 1783 name = “Theo Smith“ isActive = true
index=1
index=2
:PersonGuestDummy personIdent = 20544 name = “Otto Mair“ isActive = false
…
:Person
…
personIdent = 19227 name = “Jo Miller“ isActive = false
Fig. 8. Replacing a constructor by a factory and modeling the factory behavior through a series of objects to be “created” during a test.
Frameworks and Components Frameworks are commonly used in object-oriented systems. Most prominent examples are the AWT and Swing from Java, but basic classes, such as the containers partly also belong to that category. Software that uses predefined frameworks is particularly hard to test: • The control flow of the framework makes tests tricky to run. • Form and class of newly created objects within the framework is predefined through the used constructors. • Static variables and methods of the framework cannot be controlled, in particular if they are encapsulated. • Encapsulation prevents to check the test result. • Sometimes subclasses cannot be built properly, because the class or methods are declared “final”, there is no public constructor, the constructor does have side effects, or the internal control flow is unknown. Today there doesn’t exist a single framework that is directly suited for tests. Such a framework should allow to replace framework objects by self-defined subclass objects and should provide its own default dummy subclasses. Furthermore, the framework should use factories for object creation and give the test developer a possibility to replace these factories. The white-box adaptation principles that frameworks usually provide through subclassing are indeed helpful and sometimes sufficient, but if not, a more general technique, the adapter, is needed to separate application and framework. This is a recommended technique for application developers anyway to decouple application and framework and can be reused for improvement of testability as well. Fig. 9 shows how a JSP “ServletRequest” class is adapted. A “ServletRequest” basically contains the contents of a web form filled by the user in form of pairs (parametername, content). Unfortunately “ServletRequest”-objects can only be created by handling actual requests through the web. Therefore, an adapter is used, which is called “OwnServletRequest”. In an adapter normally simple delegation is used. But, as framework classes are strongly interconnected method calls often require other framework objects as parameters or reveal access to other framework objects. For example the method “get-
Model-Based Testing of Object-Oriented Systems
391
Session()” needs an additional wrapping to return the proper object of class “OwnSession”. This adapter technique allows us to completely decouple application and framework and even to run the application part without the framework as it may be desired in tests. For testing purposes “OwnServletRequestDummy” may now overwrite all methods and use a “Map” to store a predefined set of “user” inputs. However, there must be noted that this kind of wrapping may need additional infrastructure to ensure that each time “getSession()” is called on the same “ServletRequest”, the same corresponding session object is returned. This can be solved through an internal Map from Session to OwnSession that keeps track.
«Adapter» OwnServletRequest
…
OwnServletRequest() OwnServletRequest(HttpServletRequest hsr) +Enumeration getParameterNames() +String getParameter(String name) +OwnSession getSession()
OwnServletRequestDummy
HttpServletRequest 0..1
…
CD
+Enumeration getParameterNames() +String getParameter(String name) +HttpSession getSession()
result with the type of a framework class are also wrapped in an adapter
…
Map/*String,String*/ parameter OwnServletRequestDummy(Map /*String,String*/ p) +Enumeration getParameterNames() +String getParameter(String name) +OwnSession getSession()
…
0..1
«Adapter» OwnSession
… HttpSession
… OwnSessionDummy
Fig. 9. Adapters for framework classes.
So far we have discussed a number of easy to apply, but effective techniques to make object-oriented systems testable. We have used class, sequence and object diagrams to model the test pattern and demonstrated how to use these diagrams to model the test data and dummies. It is an interesting question, how to present such methodological knowledge and its individual parts. Basically the technical principles, such as pattern structure can be formally defined. This has the advantage that at least the structural part of such a pattern can automatically be applied using an appropriate tool. However, the most important part of a test pattern, namely the methodical experience cannot be formalized, but needs to be presented to the user in an understandable way. The user needs then to adapt a pattern to his specific situation. Therefore, we have chosen to largely use examples instead of precise descriptions for patterns.
6 Test Pattern for Distributed Systems Testing a distributed system may become hard, as distribution naturally involves concurrent processes with interactions and timeouts thus leading to nondeterministic behavior. Furthermore, it is normally not possible to stop the system and obtain a global system state for consistency checking. There exists a variety of approaches to
392
B. Rumpe
deal with these problems, in particular in the hardware and embedded systems area. Through the distribution of web services in particular in the E-Commerce domain, it becomes increasingly important to be able to deal with distributed object systems in this domain as well. In our example application, the online auction system, timing and distribution are very important, as auctions last only a very restricted time (e.g. one hour) and in the final phase bids are submitted within seconds. Therefore, it is necessary that auctions are handled synchronously over the web. The test patterns discussed in this section tackle four occurring problems: (1) simulation of time and progress, (2) handling concurrency through threads, (3) dealing with distribution and (4) communication. As already mentioned, the test patterns concentrate on functional tests. Additional effort is necessary to test quality of service attributes, such as throughput, mean uptime, etc. The proposed techniques have already been used in other approaches, the novelty basically comes from the combination of modeling techniques and these concepts in form of methodical test patterns. Handling Time and Progress An online auction usually takes about one hour. However, a single test may not take that time, but needs to be run in milliseconds, as hundreds of tests shall finish quickly. So it is necessary to simulate time. This becomes even more important, when distributed processes come into play that do not agree on a global time as is usually the case in the internet. Thus instead of calling the time routine of the operating system directly, an adapter is used. The adapter can be replaced by a parameterized dummy that allows us to freely set time. For many tests, a fixed time is sufficient, for tests of larger series of behaviors, however, it is also necessary that progress happens. Thus in the time pattern two more concepts can be established: First, each query of the current time increases time for one tick. Second, we use explicit time stamps on sequence diagrams to adapt time during the test. The time stamps as shown in Fig. 10 therefore correspond to statements that update the timing dummy. This active use of time stamps contrasts other approaches, where a passive interpretation regards a time stamp as maximum durations th