December)

from the editor Editor in Chief: Steve McConnell

■

Construx Software

■

[email protected]

Raising Your Software Consciousness Steve McConnell

I

n 1970, Charles Reich published a bestselling book called The Greening of America.1 In it, Reich identifies three kinds of awareness or consciousness, which he calls Consciousness I, Consciousness II, and Consciousness III. Consciousness I (“Con I”) is the pioneer mentality. People who operate at Con I place great value on independence and self-satisfaction. They don’t easily tolerate other people telling them what to do. They are highly self-reliant and self-sufficient. Reich believes that Con I dominated the American psyche during America’s first centuries and that this focus on selfreliance was a significant factor in America’s development. Consciousness II is the gray flannel suit mentality—corporation man. People who operate at Con II understand the importance of getting along with others and playing by the rules. They believe rules are good for society, and they think everyone should follow them. Reich believes that Con II became more dominant than Con I in the mid-twentieth century. Consciousness III is the mentality of enlightened independence. The Con III person operates on the basis of principles, with little regard for the rules that predominate in Con II and without the selfishness that predominates in Con I. By the time Greening was published, Reich argued that Con II’s time was over. He believed Con III was in its ascendancy and would soon replace Con II. Although The Greening of America struck a resonant chord when it was published, hisCopyright © 2001 Steven C. McConnell. All Rights Reserved.

tory has not been kind to the book. In 1999, Slate magazine’s readers voted it the silliest book of the 20th century. Reich’s Con III was a hippie nirvana, and the “greening” Reich predicted was a nationwide movement toward the hippie culture of the 1960s and 1970s—psychedelic drugs, bell-bottom pants, and all. As the hippie culture faded into obscurity in the 1980s, so did the credibility of Reich’s predictions.

Can’t get no satisfaction Reich’s political predictions may not have withstood the test of time, but his classification of Con I, Con II, and Con III provides a useful model for the software industry today. Con I in software is associated with a focus on self-reliance. Software experts often refer to software developers operating at this level of awareness as mavericks, cowboy programmers, Lone Rangers, and prima donnas. Software developers at this level tend to have little tolerance for other people’s ideas. They like to work alone. They don’t like following standards. The “Not Invented Here” syndrome thrives. Con I’s advantage is that little training is needed, and the lone-wolf approach works adequately in environments that employ only small numbers of programmers who work independently on small projects. Con I’s disadvantage is that it scales poorly to projects that need teams of programmers rather than isolated individuals. Con II in software is associated with a focus on rules. Many software developers eventually discover the limitations of Con I’s selfreliant development style and see the November/ December 2001

IEEE SOFTWARE

7

FROM THE EDITOR

D E PA R T M E N T E D I T O R S

Bookshelf: Warren Keuffel, [email protected] Country Report: Deependra Moitra, Lucent Technologies [email protected] Design: Martin Fowler, ThoughtWorks, [email protected] Loyal Opposition: Robert Glass, Computing Trends, [email protected] Manager: Don Reifer, Reifer Consultants, [email protected] Quality Time: Jeffrey Voas, Cigital, [email protected] STAFF

Senior Lead Editor Dale C. Strok [email protected] Group Managing Editor Crystal Chweh Associate Editors Jenny Ferrero and Dennis Taylor Staff Editors Shani Murray, Scott L. Andresen, and Kathy Clark-Fisher Magazine Assistants Dawn Craig [email protected] Pauline Hosillos Art Director Toni Van Buskirk Cover Illustration Dirk Hagner Technical Illustrator Alex Torres Production Artists Carmen Flores-Garvey and Larry Bauer Acting Executive Director Anne Marie Kelly Publisher Angela Burgess Assistant Publisher Dick Price Membership/Circulation Marketing Manager Georgann Carter Advertising Assistant Debbie Sims CONTRIBUTING EDITORS

Greg Goth, Denise Hurst, Anne Lear, Keri Schreiner, and Margaret Weatherford

Editorial: All submissions are subject to editing for clarity, style, and space. Unless otherwise stated, bylined articles and departments, as well as product and service descriptions, reflect the author’s or firm’s opinion. Inclusion in IEEE Software does not necessarily constitute endorsement by the IEEE or the IEEE Computer Society. To Submit: Send 2 electronic versions (1 word-processed and 1 postscript or PDF) of articles to Magazine Assistant, IEEE Software, 10662 Los Vaqueros Circle, PO Box 3014, Los Alamitos, CA 90720-1314; [email protected]. Articles must be original and not exceed 5,400 words including figures and tables, which count for 200 words each.

8

IEEE SOFTWARE

November/ December 2001

advantages of working in groups. Over time, they learn rules that allow them to coordinate their work with others. Some groups of developers create their own informal rules through trial and error, and these groups can be highly effective. Other groups buy a prebuilt methodology. Sometimes the rules are provided by consultants, as in the classic “17 three-ring binders” methodologies. Other times, the rules are taken from books, such as The Rational Unified Process,2 the Extreme Programming series,3 or my own Software Project Survival Guide.4 Developers at this level of awareness tend to focus on the details of adhering to the rules. They argue about which interpretations of the rules are correct and focus on “following the methodology.” The advantage of Con II is that a developer needs to be trained to use only a single approach. If a good approach is chosen, the developer can leverage a relatively small amount of training across many projects. The disadvantage is that a Con II developer is ill-equipped to succeed on projects that fall outside the specific methodology in which the developer was trained. Con III in software is associated with a focus on principles. At this level of awareness, developers understand that the rules of any prepackaged methodology are, at best, approximations of principles. Those approximations might apply most of the time, but they won’t apply all of the time. The disadvantage of Con III is that extensive education and training are needed to introduce a developer to the principles underlying effective software development, and that training is not easily obtained. Con III’s advantage is that, once that training has been obtained, the developer is equipped with a full range of software engineering tools that support success on a wide range of projects.

Love the one you’re with The software industry has a long history of trying and ultimately rejecting “one size fits all” methodologies. These methodologies are Con II soft-

ware approaches, and they fail outside narrowly defined areas of applicability —predictably—precisely because they are Con II. The world of software is far too varied to be addressed by a single set of rules. For example, compare the practices you would use to develop a heart pacemaker control to those you would use to develop a video store management program. If a software malfunction caused you to lose one video out of 1,000, it might affect the store’s profitability by a fraction of a percent, but the impact is negligible. If a malfunction caused one pacemaker out of 1,000 to fail, however, you’ve got a real problem. Generally speaking, widely distributed products must be developed more carefully than narrowly distributed ones. Products whose reliability is important must be developed more carefully than products whose reliability doesn’t much matter. These different kinds of software require different development practices. Practices that would be considered to be overly rigorous, bureaucratic, and cumbersome for video store management software might be considered irresponsibly quick and dirty—or reckless—for an embedded pacemaker control. The Con III developer will use different practices to develop a heart pacemaker control than to develop an video inventory tracking system. The Con II developer will try to apply a one-size-fits-all methodology to both projects, with the likelihood that the methodology won’t work particularly well for either one.

Are you experienced? Reich identified the three levels of consciousness as the zeitgeists of different eras, but I see Con I, Con II, and Con III as three distinct steps along a path of personal software engineering maturity. Most software developers begin their careers at Con I and eventually journey to Con II. In many environments, Con II supports effective work, and no further development is needed. In some environments, however, a further progression toward Con III is needed.

FROM THE EDITOR

The by-the-book methodologies of Con II seem to be a reasonable learning path for developers at Con I who are not yet well versed in a wide range of software practices. The specific details of the rules-based practices probably don’t matter all that much. People who are trying to raise themselves from Con I to Con II simply need to take a first step away from the chaos of a completely unmanaged project. They must learn a set of rules and get some experience applying those rules before they can advance to the Con III level, where they understand software project dynamics well enough to break the rules when needed. This whole process is part of the natural progression from apprentice to journeyman to master.

References 1. C. Reich, The Greening of America, Random House, New York, 1970. 2. P. Kruchten, The Rational Unified Process: An Introduction, 2nd ed., Addison-Wesley, Reading, Mass., 2000. 3. K. Beck, Extreme Programming: Embrace Change, Addison-Wesley, Reading, Mass., 2000. 4. S. McConnell, Software Project Survival Guide, Microsoft Press, Redmond, Wash., 1998.

EDITOR IN CHIEF: Steve McConnell 10662 Los Vaqueros Circle Los Alamitos, CA 90720-1314 [email protected] EDITOR IN CHIEF EMERITUS: Alan M. Davis, Omni-Vista A S S O C I AT E E D I T O R S I N C H I E F

Design: Maarten Boasson, Quaerendo Invenietis [email protected] Construction: Terry Bollinger, Mitre Corp. [email protected] Requirements: Christof Ebert, Alcatel Telecom [email protected] Management: Ann Miller, University of Missouri, Rolla [email protected] Quality: Jeffrey Voas, Cigital [email protected] Experience Reports: Wolfgang Strigel, Software Productivity Center; [email protected] EDITORIAL BOARD

Don Bagert, Texas Tech University Richard Fairley, Oregon Graduate Institute Martin Fowler, ThoughtWorks Robert Glass, Computing Trends Andy Hunt, Pragmatic Programmers Warren Keuffel, independent consultant Brian Lawrence, Coyote Valley Software Karen Mackey, Cisco Systems Deependra Moitra, Lucent Technologies, India Don Reifer, Reifer Consultants Suzanne Robertson, Altantic Systems Guild Dave Thomas, Pragmatic Programmers Karl Wiegers, Process Impact INDUSTRY ADVISORY BOARD

Upcoming

Topics

January/February ’02: Building Systems Securely from the Ground Up March/April ’02: The Software Engineering of Internet Software May/June ’02: Knowledge Management in Software Engineering July/August ’02: The Business of Software Engineering

Robert Cochran, Catalyst Software (chair) Annie Kuntzmann-Combelles, Q-Labs Enrique Draier, PSINet Eric Horvitz, Microsoft Research David Hsiao, Cisco Systems Takaya Ishida, Mitsubishi Electric Corp. Dehua Ju, ASTI Shanghai Donna Kasperson, Science Applications International Pavle Knaflic, Hermes SoftLab Günter Koch, Austrian Research Centers Wojtek Kozaczynski, Rational Software Corp. Tomoo Matsubara, Matsubara Consulting Masao Matsumoto, Univ. of Tsukuba Dorothy McKinney, Lockheed Martin Space Systems Nancy Mead, Software Engineering Institute Stephen Mellor, Project Technology Susan Mickel, AgileTV Dave Moore, Vulcan Northwest Melissa Murphy, Sandia National Laboratories Kiyoh Nakamura, Fujitsu Grant Rule, Software Measurement Services Girish Seshagiri, Advanced Information Services Chandra Shekaran, Microsoft Martyn Thomas, Praxis Rob Thomsett, The Thomsett Company John Vu, The Boeing Company Simon Wright, Integrated Chipware Tsuneo Yamaura, Hitachi Software Engineering M A G A Z I N E O P E R AT I O N S C O M M I T T E E

Sorel Reisman (chair), James H. Aylor, Jean Bacon, Thomas J. Bergin, Wushow Chou, William I. Grosky, Steve McConnell, Ken Sakamura, Nigel Shadbolt, Munindar P. Singh, Francis Sullivan, James J. Thomas, Yervant Zorian P U B L I C AT I O N S B O A R D

Rangachar Kasturi (chair), Angela Burgess (publisher), Jake Aggarwal, Laxmi Bhuyan, Mark Christensen, Lori Clarke, Mike T. Liu, Sorel Reisman, Gabriella Sannitti di Baja, Sallie Sheppard, Mike Williams, Zhiwei Xu

November/ December 2001

IEEE SOFTWARE

9

design Editor: Martin Fowler

■

T h o u g h t Wo r k s

■

[email protected]

To Be Explicit Martin Fowler

S

oftware is an odd medium in which to construct something. Because few physical forces make you design one way or another, many design decisions sadly resist any form of objective analysis. Where design counts is often not in how the software runs but in how easy it is to change. When how it runs is important, ease of change can be the biggest factor in ensuring good performance. This drive toward changeability is why it’s so important for a design to clearly show what the program does—and how it does it. After all, it’s hard to change something when you can’t see what it does. An interesting corollary of this is that people often use specific designs because they are easy to change, but when they make the program difficult to understand, the effect is the reverse of what was intended.

Attributes and dictionaries Let’s say we want a person data structure. We can accomplish this by having specific fields, as Figure 1 shows. Of course, to make this work, we must define the variables in the person class. Like many modern languages, Ruby provides a dictionary data structure (also knows as a map, associative array, or hash table). We could use Ruby instead to define the person class, using the approach in Figure 2. (This is slower, but let’s assume this section of code isn’t performance critical.) Using a dictionary is appealing because it lets you change what you store in the person without changing the person class. If you want to add a telephone number, you can do it without altering the original code. Despite this, the dictionary doesn’t make it easier to modify the code. If I’m trying to use the person structure, I can’t tell what is in it. To learn that someone’s storing the number of dependents, I must review the entire system. If the number of dependents is declared in the

class Person attr_accessor :lastName, :firstName, :numberOfDependents end def frag1 martin = Person.new martin.firstName = “Martin” martin.lastName = “Fowler” martin.numberOfDependents = 1 print (martin.firstName, “ “, martin.lastName, “ has “, martin.numberOfDependents, “ dependents”) Figure 1. Explicit fields (using Ruby). 10

IEEE SOFTWARE

end

November/December 2001

0740-7459/01/$10.00 © 2001 IEEE

Figure 2. Dictionary fields (using Ruby).

class, then I only have to look in the person class to see what it supports. The key principle is that explicit code is easier to understand—which makes the code easier to modify. As Kent Beck puts is, the explicit code is intention revealing. This dictionary example is small in scale, but the principle holds at almost every scale of software design. Events and explicit calls Here’s another example, on a slightly bigger scale. Many platforms support the notion of events to communicate between modules. Say we have a reservation module that, when canceled, needs to get a person module to send email to that person. We can do this using an event, as Figure 3 shows. We can define interesting

class Person attr_accessor :data def initialize() @data = {} end end def frag2 martin = Person.new martin.data[“firstName”] = “Martin” martin.data[“lastName”] = “Fowler” martin.data[“numberOfDependents”] = 1 print (martin.data[“firstName”],“ “, martin.data[“lastName”], “ has “, martin.data[“numberOfDependents”], “ dependents”) end

events that affect a reservation, and any object that wants to do anything when an event occurs can build a handler to react when it occurs. This approach is appealing because you need not modify

the reservation class to get something else to happen when you cancel a reservation. As long as other objects put handlers on the event, it’s easy to extend the behavior at these points.

public delegate void ReservationHandler (IReservation source); public class Reservation ... public String Id; public event ReservationHandler Cancelled; public Person client { get { return client; } set { value.AddReservation(this); } } public void Cancel(){ Cancelled (this); } public class Person ... public String EmailAddress; public readonly ArrayList reservations; public void SendCancellationMessage(Reservation arg) { // send a message } public void AddReservation(Reservation arg) { //invoke SendCancellationMessage when the cancelled event occurs on arg arg.Cancelled += new ReservationHandler(SendCancellationMessage); } Figure 3. Cancellation using events (using C#). November/December 2001

IEEE SOFTWARE

11

DESIGN

However, there is a cost to using events—I can’t see what happens at cancellation by reading the code in the cancellation method. To find out what happens, I have to search for all the code that has a handler for the event. The explicit code for this (see Figure 4) clearly shows in the cancel method the consequences of cancellation, at the cost of modifying the reservation class when I need to change the behavior. I’ve seen a few code examples that use events heavily, and the problem is

that it’s hard to determine what the program does when you call a method. This becomes particularly awkward when you’re debugging, because behavior pops up suddenly in places you don’t expect. I’m not saying that you shouldn’t use events. They let you carry out behavior without changing a class, which makes them useful when working with library classes you can’t modify. They are also valuable because they don’t create a depen-

dency from the class triggering the event to the one that needs to react. This lack of a dependency is valuable when the two classes are in different packages and you don’t want to add a dependency. The class case of this is when you want to modify a window in a presentation when some domain object changes. Events let you do this while preserving the vital separation of the presentation and domain. Those forces both suggest events, but in their absence, the lack of explicitness of events becomes more dominant. So, I would be reluctant to use events between two application classes that can be aware of each other. As you can see, explicitness is not always the dominant force in design decisions. In this example, packaging and dependency forces are also important. People often underestimate the value of explicitness. There are times when I would add a dependency to make code more explicit, but, as always with design, each situation has its own trade-offs to consider. Data-driven code and explicit subclasses My final example is on yet a bigger scale. Consider a discounting scheme for orders that uses different discounting plans. The blue plan gives you a fixed discount of 150 if you buy goods from a particular group of suppliers and the value of your order is over a certain threshold. The red plan gives you a 10 percent discount when delivering to certain US states. Figure 5 presents explicit code for this. The order has a discounter with specific subclasses for the blue and red cases. The data-driven version in Figure 6 uses a generic discounter that is set up with data when the order is created. The generic discounter’s advantage is that you can create new kinds of discounters without making new classes by writing code—if the new classes fit in with the generic behavior. For the sake of argument, let’s assume they can. Is the generic case always the best choice? No, again because of explicitness. The explicit subclasses are easier to read and they make it easier to

Figure 4. An explicit reaction to cancel (using C#).

public class Reservation ... public String Id; public Person client;

Figure 5. Explicitly programmed discount logic (using C#).

public class public public public public

public void Cancel(){ client.SendCancellationMessage(this); }

Order ... Decimal BaseAmount; String Supplier; String DeliveryState; Discounter Discounter;

public virtual Decimal Discount { get { return Discounter.Value(this); } } } abstract public class Discounter { abstract public Decimal Value (Order order); } public class BlueDiscounter : Discounter { public readonly IList DiscountedSuppliers = new ArrayList(); public Decimal Threshold = 500m; public void AddDiscountedSupplier(String arg) { DiscountedSuppliers.Add(arg); } public override Decimal Value (Order order) { return (DiscountApplies(order)) ? 150 : 0; } private Boolean DiscountApplies(Order order) { return DiscountedSuppliers.Contains(order.Supplier) && (order.BaseAmount > Threshold); } } public class RedDiscounter : Discounter { public readonly IList DiscountedStates = new ArrayList(); public void AddDiscountedState (String arg) { DiscountedStates.Add(arg); } public override Decimal Value (Order order) { return (DiscountedStates.Contains(order.DeliveryState)) ? order.BaseAmount * 0.1m : 0; } } // to set up a blue order BlueDiscounter bluePlan = new BlueDiscounter(); bluePlan.AddDiscountedSupplier(“ieee”); blue = new Order(); blue.Discounter = bluePlan; blue.BaseAmount = 500; blue.Supplier = “ieee”;

November/December 2001

IEEE SOFTWARE

13

DESIGN

public class GenericOrder : Order ... public Discounter Discounter; public override Decimal Discount { get { return Discounter.Value(this); } } public enum DiscountType {constant, proportional}; public class public public public public public public

Discounter ... DiscountType Type; IList DiscountedValues; String PropertyNameForInclude; String PropertyNameForCompare; Decimal CompareThreshold; Decimal Amount;

public Decimal Value(GenericOrder order) { if (ShouldApplyDiscount(order)) { if (Type == DiscountType.constant) return Amount; if (Type == DiscountType.proportional) return Amount * order.BaseAmount; throw new Exception (“Unreachable Code reached”); } else return 0; } private Boolean ShouldApplyDiscount(Order order) { return PassesContainTest(order) && PassesCompareTest(order); } private Boolean PassesContainTest(Order order) { return DiscountedValues.Contains (GetPropertyValue(order, PropertyNameForInclude)); } private Boolean PassesCompareTest(Order order){ if (PropertyNameForCompare == null) return true; else { Decimal compareValue = (Decimal) GetPropertyValue(order, PropertyNameForCompare); return compareValue > CompareThreshold; } } private Object GetPropertyValue (Order order, String propertyName) { FieldInfo fi = typeof(Order).GetField(propertyName); if (fi == null) throw new Exception(“unable to find field for “ + propertyName); return fi.GetValue(order); } }

Figure 6. Data-programmed discount logic (using C#). 14

IEEE SOFTWARE

November/December 2001

DESIGN

//to set up a blue order GenericDiscounter blueDiscounter = new GenericDiscounter(); String[] suppliers = {“ieee”}; blueDiscounter.DiscountedValues = suppliers; blueDiscounter.PropertyNameForInclude = “Supplier”; blueDiscounter.Amount = 150; blueDiscounter.PropertyNameForCompare = “BaseAmount”; blueDiscounter.CompareThreshold = 500m; blueDiscounter.Type = DiscountType.constant; blue = new Order(); blue.BaseAmount = 500; blue.Discounter = blueDiscounter;

understand the behavior. With the generic case, you must look at the generic code and setup code, and it’s harder to see what’s happening—and even harder for more complicated bits of behavior. Of course, we can extend the generic order without “programming,” but I’d argue that configuring that data is a form of programming. Debugging and testing are often both difficult and overlooked with data-driven behavior. The generic case works when you have dozens of discounters. In such cases, the volume of code becomes a problem, while greater volumes of data are less problematic. Sometimes a well-chosen data-driven abstraction can make the logic collapse into a much smaller and easier-to-maintain piece of code. Ease of deploying new code is also a factor. If you can easily add new subclasses to an existing system, explicit behavior works well. However, generic behavior is a necessity if new code means long and awkward compile and link cycles. There’s also the option of combining the two, using a data-driven generic design for most of the cases and explicit subclasses for a few hard cases. I like this approach because it keeps the generic design much simpler, but the subclasses give you a lot of flexibility when you need it.

E

xplicitness is not an absolute in design, but clever designs often become hard to use because they aren’t explicit enough. In some cases, the cost is worth it, but it’s always something to consider. In the last few years, I’ve tended to choose explicit designs more often because my views of

what makes good design have evolved (hopefully in the right direction).

Martin Fowler is the chief scientist for ThoughtWorks, an Internet systems delivery and consulting company. Contact him at [email protected].

Boston University If you have significant academic or industrial experience in developing large software systems and you are committed to improving the practice of software engineering, we want to talk to you about a faculty appointment at Boston University in Computer Systems Engineering. Our graduate program teaches the engineering skills necessary for the effective development of large-scale computer systems in which software provides essential functionality. We teach students to apply engineering principles to the design of a full range of computer products from embedded systems, to data communication networks, to software products. Three types of appointments are available in the Department of Electrical and Computer Engineering (ECE) starting in September 2002: • Research oriented tenure-track and tenured appointments. • Non-tenure track positions, which require extensive experience in practicing software engineering. • Adjunct (part-time) positions for Boston-area experts who are interested in teaching their specialty at the graduate level. All positions require a commitment to excellence in teaching at the undergraduate and graduate levels. For additional information on the College of Engineering and ECE Department visit the College’s homepage at http://www.bu.edu/eng/. To learn more about opportunities for a non-tenure track OR adjunct positions, please e-mail: [email protected] and a faculty member will call to discuss our opportunities. For tenure-track OR tenured appointments, send your Curriculum Vita to: Professor Bahaa E. A. Saleh, Chair, Department of Electrical and Computer Engineering, Boston University, 8 Saint Mary’s Street, Boston, MA 02215.

November/December 2001

IEEE SOFTWARE

15

focus

guest editor’s introduction

Reports from the Field Using Extreme Programming and Other Experiences Wolfgang Strigel, Software Productivity Center

L

earning from the successes and failures of others is a quick way to learn and enlarge our horizon. Our own experience can only cover a narrow path though the wealth of existing knowledge. Last June, the IEEE Software Editorial Board decided to make more room for

experience reports and give our readers a forum to share their own learning experiences with others. If you are interested in submitting

0740-7459/01/$10.00 © 2001 IEEE

an experience report, please refer to www. computer.org/software/genres.htm for author guidelines.

November/December 2001

IEEE SOFTWARE

17

Extreme Programming and Agile Methods: Web Resources Martin Fowler, ThoughtWorks Short introduction to XP: www.cutter.com/ead/ead0002.html www.rolemodelsoft.com/articles/xpCorner/xpDistilled.htm XP portals: www.xprogramming.com www.extremeprogramming.org www.rolemodelsoft.com/xp/index.htm www.jera.com/techinfo/xpfaq.html XP mailing lists: http://groups.yahoo.com/group/extremeprogramming/ news:comp.software.extreme-programming Introduction to agile methods: http://martinfowler.com/articles/newMethodology.html (All URLs current 15 Oct. 2001)

By lucky coincidence, we had a large backlog of experience reports and were able to include six of them in this issue. On an ongoing basis, we hope to publish two or three shorter experience reports per issue. I think you’ll enjoy these interesting stories that are typical of the challenges we all face in this industry. Even if you were to pick only one gem from the experience of others, it might help you, your project, and your company. The first four articles address the topic of Extreme Programming; the final two address a different set of experiences from the field. Extreme Programming in the real world Many methodologies have come and gone. Only time will tell if one of the more recent methodology innovations, Extreme Programming, will have a lasting impact on our way to build software systems. Like other methodologies, XP is not the ultimate silver bullet that offers an answer to all development problems. But it has gained significant momentum and an increasing number of software teams are ready to give it a try. Our first article is not really an experience report but an interesting comparison of XP with the more established Capability 18

IEEE SOFTWARE

November/December 2001

Maturity Model. As one of the foremost experts on CMM, Mark Paulk offers an opinion on XP as a lightweight methodology from the perspective of the heavyweight CMM. From my perspective, the difference is not so much the “weight” of the methodology than the way they are introduced in an organization. XP tends to be a grassroots methodology. Developers and development teams typically drive its introduction. This becomes quite clear from reading the subsequent experience reports. CMM, on the other hand, is typically introduced at the corporate level and then deployed to development teams. As in past “methodology wars,” there are heated debates about the pros and cons of the respective approaches. I agree with Paulk that CMM and XP can be considered complementary. To establish lasting success, methodologies need buyin from management as well as from the developers. Martin Fowler offers a few links to further information about XP and agile methods in the “Web Resources” sidebar. Two more reports The last two articles in the set cover dissimilar experiences, but they have one thing in common: an account of our continuous struggle to make software development more efficient. The first article presents a typical example of survival struggles in a rapidly growing company and its attempts to use process to get development activities under control. The second article describes a technique, called defect logging and defect data analysis, that aims to decrease programmers’ repetitive errors. The author picked one element of the Personal Software Process and made it easier to apply.

About the Author Wolfgang Strigel is the founder and president of Software Productivity Center, a consulting and products company, and of QA Labs, a contract testing company. His interests include collaborative software development, process improvement, project estimation, testing, and software engineering economics. He has a BSc in mathematics from the Technical University, Munich, Germany, an MSc in computer science from McGill University, and an MBA from Simon Fraser University. Contact him at [email protected].

focus

reports from the field

Extreme Programming from a CMM Perspective Mark C. Paulk, Software Engineering Institute

xtreme Programming is an “agile methodology” that some people advocate for the high-speed, volatile world of Internet and Web software development. Although XP is a disciplined process, some have used it in arguments against rigorous software process improvement models such as the Software Capability Maturity Model.1

E XP has good engineering practices that can work well with the CMM and other highly structured methods. The key is to carefully consider XP practices and implement them in the right environment. 0740-7459/01/$10.00 © 2001 IEEE

In this article, I summarize both XP and the SW-CMM, show how XP can help organizations realize the SW-CMM goals, and then critique XP from a SW-CMM perspective. The Software CMM The Software Engineering Institute at Carnegie Mellon University developed the SW-CMM as a model for building organizational capability, and it has been widely adopted in the software community and beyond. As Table 1 shows, the SW-CMM is a five-level model that describes good engineering and management practices and prescribes improvement priorities for software organizations. Although the SW-CMM is described in a book of nearly 500 pages, the requirements for becoming a Level 5 organization are concisely stated in 52 sentences—the 52 goals of the model’s 18 key process areas (KPAs). The practices, subpractices, and examples that flesh out the model can guide

software professionals in making reasonable, informed decisions about a broad range of process implementations. The SW-CMM informative materials focus primarily on large projects and large organizations. With minor tailoring and common sense, however, the model can be applied in radically different environments, ranging from two- to three-person projects in small start-up companies to 500-person projects building hard real-time, life-critical systems.2,3 The SW-CMM’s rating components are intentionally abstract, capturing “universal truths” about high-performance software organizations. As a glance at Table 2 shows, the KPAs are clearly important to all types of software organizations. With the exception of software subcontract management, which applies only to organizations that do subcontracting, the KPAs and their goals can apply to any software organization. Companies that focus on innovation more than operational excellence might downplay the role of consisNovember/December 2001

IEEE SOFTWARE

19

Table 1 An overview of the Software CMM Level

Focus

Key process areas

5: Optimizing

Continual process improvement

4: Managed

Product and process quality

3: Defined

Engineering processes and organizational support

2: Repeatable

Project management processes

Defect prevention Technology change management Process change management Quantitative process management Software quality management Organization process focus Organization process definition Training program Integrated software management Software product engineering Intergroup coordination Peer reviews Requirements management Software project planning Software project tracking and oversight Software subcontract management Software quality assurance Software configuration management

1: Initial

Competent people (and heroics)

tency, predictability, and reliability, but performance excellence is important even in highly innovative environments. Extreme Programming The XP method is typically attributed to Kent Beck, Ron Jeffries, and Ward Cunningham.4,5 XP’s target is small to mediumsized teams building software with vague or rapidly changing requirements. XP teams are typically colocated and have fewer than 10 members. XP’s critical underlying assumption is that developers can obviate the traditional high cost of change using technologies such as objects, patterns, and relational databases, resulting in a highly dynamic XP process. Beck’s book is subtitled Embrace Change, and XP teams typically deal with requirements changes through an iterative life cycle with short cycles. The XP life cycle has four basic activities: coding, testing, listening, and designing. Dynamism is demonstrated through four values: ■ ■ ■ ■

20

IEEE SOFTWARE

continual communication with the customer and within the team; simplicity, achieved by a constant focus on minimalist solutions; rapid feedback through mechanisms such as unit and functional testing; and the courage to deal with problems proactively.

November/December 2001

Principles in practice Most of XP’s principles—minimalism, simplicity, an evolutionary life cycle, user involvement, and so forth—are commonsense practices that are part of any disciplined process. As Table 3 summarizes, the “extreme” in XP comes from taking commonsense practices to extreme levels. Although some people may interpret practices such as “focusing on a minimalist solution” as hacking, XP is actually a highly disciplined process. Simplicity in XP terms means focusing on the highest-priority, most valuable system parts that are currently identified rather than designing solutions to problems that are not yet relevant (and might never be, given that requirements and operating environments change). Although developers might use many different XP practices, the method typically consists of 12 basic elements: ■

■

■

Planning game: Quickly determine the next release’s scope, combining business priorities and technical estimates. The customer decides scope, priority, and dates from a business perspective, whereas technical people estimate and track progress. Small releases: Put a simple system into production quickly. Release new versions on a very short (two-week) cycle. Metaphor: Guide all development with a simple, shared story of how the overall system works.

Table 2 The Software CMM key process areas and their purposes Key process area

Purpose Maturity Level 2: Repeatable

Requirements management Software project planning Software project tracking and oversight Software subcontract management Software quality assurance Software configuration management

Establish a common understanding between the customer and software project team about the customer’s requirements. Establish reasonable plans for software engineering and overall project management. Provide adequate visibility into actual progress so that management can act effectively when the software project’s performance deviates significantly from the software plans. Select qualified software subcontractors and manage them effectively. Provide management with appropriate visibility into the product and the software process. Establish and maintain the integrity of software products throughout the project’s software life cycle. Maturity Level 3: Defined

Organization process focus Organization process definition Training program Integrated software management Software product engineering Intergroup coordination Peer reviews

Establish organizational responsibility for software process activities that improve the organization’s overall software process capability. Develop and maintain a usable set of software process assets that improve process performance across the projects and provide a basis for cumulative, long-term organizational benefits. Develop individuals’ skills and knowledge so that they can perform their roles effectively and efficiently. Integrate the software engineering and management activities into a coherent, defined software process based on the organization’s standard software process and related process assets. Consistently use a well-defined engineering process that integrates all the software engineering activities to produce correct, consistent software products effectively and efficiently. Establish a way for the software engineering group to participate actively with other engineering groups so that the project can effectively and efficiently satisfy customer needs. Remove defects from the software work products early and efficiently. An important corollary effect is to develop a better understanding of the software products and the preventable defects. Maturity Level 4: Managed

Quantitative process management Software quality management

Quantitatively control the performance of the software project’s process. Software process performance represents the actual results achieved from following a software process. Quantify the quality of the project’s software products and achieve specific quality goals. Maturity Level 5: Optimizing

Defect prevention Technology change management Process change management

■ ■

■

■

■

Identify the cause of defects and prevent them from recurring. Identify new technologies (such as tools, methods, and processes) and introduce them into the organization in an orderly manner. Continually improve the organization’s software processes with the goal of improving software quality, increasing productivity, and decreasing the product-development cycle time.

Simple design: Design as simply as possible at any given moment. Testing: Developers continually write unit tests that must run flawlessly; customers write tests to demonstrate that functions are finished. “Test, then code” means that a failed test case is an entry criterion for writing code. Refactoring: Restructure the system without changing its behavior to remove duplication, improve communication, simplify, or add flexibility. Pair programming: All production code is written by two programmers at one machine. Collective ownership: Anyone can im-

■

■

■ ■

prove any system code anywhere at any time. Continuous integration: Integrate and build the system many times a day (every time a task is finished). Continual regression testing prevents functionality regressions when requirements change. 40-hour weeks: Work no more than 40 hours per week whenever possible; never work overtime two weeks in a row. On-site customer: Have an actual user on the team full-time to answer questions. Coding standards: Have rules that emphasize communication throughout the code. These basic practices work together to creNovember/December 2001

IEEE SOFTWARE

21

Table 3 The “extreme” in Extreme Programming Commonsense

XP extreme

XP implementation practice

Code reviews Testing Design Simplicity

Review code all the time Test all the time, even by customers Make design part of everybody’s daily business Always work with the simplest design that supports the system’s current functionality Everybody works to refine the architecture all the time Integrate and test several times a day Make iterations extremely short—seconds, minutes, and hours rather than weeks, months, and years

Pair programming Unit testing, functional testing Refactoring The simplest thing that could possibly work

Architecture Integration testing Short iterations

ate a coherent method. XP characterizes the full system functionality using a pool of “stories,” or short feature descriptions. For the planning game and small releases, the customer must select a subset of stories that characterize the most desirable work for developers to implement in the upcoming release. Because the customer can add new stories to the pool at any time, requirements are highly volatile. However, volatility is managed by implementing functionality in two-week chunks. Having a customer onsite supports this ongoing cycle of two-week releases. XP developers generate a metaphor to provide the project’s overarching vision. Although you could view this as a high-level architecture, XP emphasizes design, while at the same time minimizing design documentation. Some people have characterized XP as not allowing documentation outside code, but that is not quite accurate. Because XP emphasizes continual redesign—using refactoring whenever necessary—there is little value to detailed design documentation (and maintainers rarely trust anything other than the code anyway). XP developers typically throw away design documentation after the code is written, although they will keep it if it’s useful. They also keep design documentation when the customer stops coming up with new stories. At that point, it’s time to put the system in mothballs and write a five- to 10-page “mothball tour” of the system. A natural corollary of the refactoring emphasis is to always implement the simplest solution that satisfies the immediate need. Requirements changes are likely to supersede “general solutions” anyway. Pair programming is one of XP’s more controversial practices, mainly because it has resource consequences for the very managers who decide whether or not to let a project use XP. Although it might appear 22

IEEE SOFTWARE

November/December 2001

Metaphor Continuous integration Planning game

that pair programming consumes twice the resources, research has shown that it leads to fewer defects and decreased cycle time.6 For a jelled team, the effort increase can be as little as 15 percent, while cycle time is reduced by 40 to 50 percent. For Internet-time environments, the increased speed to market may be well worth the increased effort. Also, collaboration improves problem solving, and increased quality can significantly reduce maintenance costs. When considered over the total life cycle, the benefits of pair programming often more than pay for added resource costs. Because XP encourages collective ownership, anyone can change any piece of code in the system at any time. The XP emphasis on continuous integration, continual regression testing, and pair programming protects against a potential loss of configuration control. XP’s emphasis on testing is expressed in the phrase “test, then code.” It captures the principle that developers should plan testing early and develop test cases in parallel with requirements analysis, although the traditional emphasis is on black-box testing. Thinking about testing early in the life cycle is standard practice for good software engineering, though it is too rarely practiced. The basic XP management tool is the metric, and the metric’s medium is the “big visible chart.” In the XP style, three or four measures are typically all a team can stand at one time, and those should be actively used and visible. One recommended XP metric is “project velocity”—the number of stories of a given size that developers can implement in an iteration. Adoption strategies XP is an intensely social activity, and not everyone can learn it. There are two conflicting attitudes toward XP adoption. XP is gen-

erally viewed as a system that demonstrates emergent properties when adopted as a whole. As the discussion thus far shows, there are strong dependencies between many XP practices, such as collective ownership and continuous integration. Nonetheless, some people recommend adopting XP one practice at a time, focusing on the team’s most pressing current problem. This is consistent with the attitude toward change that XP is “just rules” and the team can change the rules anytime as long as they agree on how to assess the change’s effects. Beck, for example, describes XP practices as “etudes”: They help developers master the techniques, but experienced users can modify them as necessary. XP and the CMM The SW-CMM focuses on both the management issues involved in implementing effective and efficient processes and on systematic process improvement. XP, on the other hand, is a specific set of practices—a “methodology”—that is effective in the context of small, colocated teams with rapidly changing requirements. Taken together, the two methods can create synergy, particularly in conjunction with other good engineering and management practices. I’ll now illustrate this by discussing XP practices in relation to the CMM KPAs and goals outlined in Table 2. XP and Level 2 practices XP addresses Level 2’s requirements management KPA through its use of stories, an onsite customer, and continuous integration. Although system requirements might evolve dramatically over time, XP integrates feedback on customer expectations and needs by emphasizing short release cycles and continual customer involvement. “Common understanding” is established and maintained through the customer’s continual involvement in building stories and selecting them for the next release (in effect, prioritizing customer requirements). XP addresses software project planning in the planning game and small releases. XP’s planning strategy embodies Watts Humphrey’s advice, “If you can’t plan well, plan often.” The first three activities of this KPA deal with getting the software team involved in early planning. XP integrates the software team into the commitment process

by having it estimate the effort involved to implement customer stories; at the level of two-week releases, such estimates are typically quite accurate. The customer maintains control of business priorities by choosing which stories to implement in the next release with the given resources. By definition, the XP life cycle is both incremental and evolutionary. The project plan is not detailed for the project’s whole life cycle, although the system metaphor does establish a vision for project direction. As a result, developers can identify and manage risks efficiently. XP addresses software project tracking and oversight with the “big visual chart,” project velocity, and commitments (stories) for small releases. XP’s commitment process sets clear expectations for both the customer and the XP team at the tactical level and maximizes flexibility at the project’s strategic level. The emphasis on 40-hour weeks is a general human factors concern; although CMM does not address it, having “rational work hours” is usually considered a best practice. XP also emphasizes open workspaces, a similar “people issue” that is outside CMM’s scope. XP does not address software subcontract management, which is unlikely to apply in XP’s target environment. While an independent software quality assurance group is unlikely in an XP culture, SQA could be addressed by the pairprogramming culture. Peer pressure in an XP environment can achieve SQA’s aim of assuring conformance to standards, though it does not necessarily give management visibility into nonconformance issues. Dealing with process and product assurance using peer pressure can be extraordinarily effective in a small team environment. However, larger teams typically require more formal mechanisms for objectively verifying adherence to requirements, standards, and procedures. Also, peer pressure might be ineffective when the entire team is being pushed, just as a software manager might be vulnerable to external pressure. This vulnerability should be addressed at the organizational level when considering SQA. Although not completely and explicitly addressed, software configuration management is implied in XP’s collective ownership, small releases, and continuous integration. Collective ownership might be problematic for large systems, where more formal com-

Taken together, the two methods can create synergy, particularly in conjunction with other good engineering and management practices.

November/December 2001

IEEE SOFTWARE

23

Table 4 XP satisfaction of key process areas, given the appropriate environment Level

2 2 2 2 2 2 3 3 3 3 3 3 3 4 4 5 5 5 + ++ —

Key process area

Requirements management Software project planning Software project tracking and oversight Software subcontract management Software quality assurance Software configuration management Organization process focus Organization process definition Training program Integrated software management Software product engineering Intergroup coordination Peer reviews Quantitative process management Software quality management Defect prevention Technology change management Process change management

Satisfaction

++ ++ ++ — + + + + — — ++ ++ ++ — — + — —

Partially addressed in XP Largely addressed in XP (perhaps by inference) Not addressed in XP

munication channels are necessary to prevent configuration management failures. XP and Level 3 practices At Level 3, XP addresses organization process focus at the team rather than organizational level. A focus on process issues is nonetheless implied in adopting XP one practice at a time, as well as in the “just rules” philosophy. Because XP focuses on the software engineering process rather than organizational infrastructure issues, organizations adopting XP must address this and other organization-level processes, whether in a CMM-based context or not. Similarly, the various XP-related books, articles, courses, and Web sites partially address the organization process definition and training program KPAs, but organizational assets are outside the scope of the XP method itself. As a consequence, XP cannot address integrated software management because there may not be any organizational assets to tailor. Several XP practices effectively address software product engineering: metaphor, simple design, refactoring, the “mothball” tour, coding standards, unit testing, and functional testing. XP’s de-emphasis of design documentation is a concern in many environments, such as hard real-time systems, 24

IEEE SOFTWARE

November/December 2001

large systems, or virtual teams. In such environments, good designs are crucial to success, and using the refactoring strategy would be high-risk. For example, if developers performed refactoring after a technique such as rate-monotonic analysis proved that a system satisfied hard real-time requirements, they’d have to redo the analysis. Such an environment invalidates XP’s fundamental assumption about the low cost of change. XP’s emphasis on communication— through onsite customers and pair programming—appears to provide as comprehensive a solution to intergroup coordination as integrated product and process development. In fact, XP’s method might be considered an effective IPPD approach, although the software-only context ignores multidiscipline environments. Pair programming addresses peer reviews, and is arguably more powerful than many peer review techniques because it adopts preventive concepts found in code reading and literate programming. However, pair programming’s relative lack of structure can lessen its effectiveness. Empirical data on pair programming is currently sparse but promising.6 To make informed trade-off decisions, we’ll need more empirical research that contrasts and compares pair programming and peer review techniques, especially more rigorous techniques such as inspections. Beyond Level 3 XP addresses few of the Level 4 and 5 KPAs in a rigorous statistical sense, although feedback during rapid cycles might partially address defect prevention. Table 4 summarizes XP’s potential to satisfy CMM KPAs, given the appropriate domain. Many of the KPAs that XP either ignores or only partially covers are undoubtedly addressed in real projects. XP needs management and infrastructure support, even if it does not specifically call for it. Discussion As the earlier comparison shows, XP generally focuses on technical work, whereas the CMM generally focuses on management issues. Both methods are concerned with “culture.” The element that XP lacks that is crucial for the SW-CMM is the concept of “institutionalization”—that is, establishing a culture of “this is the way we do things around here.”

Table 5 XP and institutionalization practices Although implicit in some practices, such as the peer pressure arising from pair programming, XP largely ignores the infrastructure that the CMM identifies as key to institutionalizing good engineering and management practices. Table 5 summarizes XP’s coverage of institutionalization in its domain. The CMM’s KPAs share common features that implement and institutionalize processes. Each KPA’s institutionalization practices map to the area’s goals; a naïve XP implementation that ignored these infrastructure issues would fail to satisfy any KPA. XP ignores some of these practices, such as policies. XP addresses others, such as training and SQA, by inference. It addresses still others—project-specific practices such as management oversight and measurement—to a limited degree. As an implementation model focused on the development process, these issues are largely outside XP’s focus, but they are arguably crucial for its successful adoption. Size matters Much of the formalism that characterizes most CMM-based process improvement is an artifact of large projects and severe reliability requirements, especially for life-critical systems. The SW-CMM’s hierarchical structure, however, is intended to support a range of implementations through the 18 KPAs and 52 goals that comprise the requirements for a fully mature software process. As systems grow, some XP practices become more difficult to implement. XP is, after all, targeted toward small teams working on small to medium-sized projects. As projects become larger, emphasizing a good architectural “philosophy” becomes increasingly critical to project success. Major investment in product architecture design is one of the practices that characterizes successful Internet companies.7 Architecture-based design, designing for change, refactoring, and similar design philosophies emphasize the need to manage change systematically. Variants of the XP bottom-up design practices, such as architecture-based design, might be more appropriate in large-project contexts. In a sense, architectural design that emphasizes flexibility is the goal of any good object-oriented methodology, so XP and object orientation are well suited to one another. Finally, large

Common feature (in each KPA)

Practice

Commitment to perform

Policy Leadership and sponsorship Organizational structures Resources and funding Training Measurement Senior management oversight Project management oversight Software quality assurance

Ability to perform

Measurement and analysis Verifying implementation

+ ++ —

Satisfaction

— — + + + + — ++ +

Partially addressed in XP Largely addressed in XP (perhaps by inference) Not addressed in XP

projects tend to be multidisciplinary, which can be problematic given that XP is aimed at software-only projects. Why explore XP? Modern software projects should capture XP values, regardless of how radically their implementation differs from XP’s. Organizations might call communication and simplicity by other names, such as coordination and elegance, but without these values, nontrivial projects face almost insurmountable odds. XP’s principles of communication and simplicity are also fundamental for organizations using the SW-CMM. When defining processes, organizations should capture the minimum essential information needed, structure definitions using good software design principles (such as information hiding and abstraction), and emphasize usefulness and usability.2 For real-time process control, rapid feedback is crucial. Previous eras have captured this idea in aphorisms such as “don’t throw good money after bad”; in a quantitative sense, we can view this as the soul of the CMM’s Level 4. One of the consequences of the cultural shift between Levels 1 and 2 is the need to demonstrate the courage of our convictions by being realistic about estimates, plans, and commitments. False opposition The main objection to using XP for process improvement is that it barely touches the management and organizational issues that the SW-CMM emphasizes. Implementing the kind of highly collaborative environment that XP assumes requires enlightened management and appropriate organizational infrastructure. November/December 2001

IEEE SOFTWARE

25

The argument that CMM’s ideal of a rigorous, statistically stable process is antithetical to XP is unconvincing. XP has disciplined processes, and the XP process itself is clearly well defined. We can thus consider CMM and XP complementary. The SWCMM tells organizations what to do in general terms, but does not say how to do it. XP is a set of best practices that contains fairly specific how-to information—an implementation model—for a particular type of environment. XP practices can be compatible with CMM practices (goals or KPAs), even if they do not completely address them.

M

ost of XP consists of good practices that all organizations should consider. While we can debate the merits of any one practice in relation to other options, to arbitrarily reject any of them is to blind ourselves to new and potentially beneficial ideas. To put XP practices together as a methodology can be a paradigm shift similar to that required for concurrent engineering. Although its concepts have been around for decades, adopting concurrent engineering practices changes your product-building paradigm. XP provides a systems perspective on programming, just as the SW-CMM provides a systems perspective on organizational process improvement. Organizations that want to improve their capability should take advantage of the good ideas in both, and exercise common sense in selecting and implementing those ideas. Should organizations use XP, as published, for life-critical or high-reliability systems? Probably not. XP’s lack of design documentation and de-emphasis on architecture are risky. However, one of XP’s virtues is that you can change and improve it for different environments. That said, when you change XP, you risk losing the emergent properties that provide value in the proper context. Ultimately, when you choose and improve software processes, your emphasis should be to let common sense prevail—and to use data whenever possible to offer insight on challenging questions.

26

IEEE SOFTWARE

November/December 2001

Acknowledgments I gratefully acknowledge Kent Beck, Steve McConnell, and Laurie Williams for their comments. I presented an earlier version of this article at XP Universe in July 2001.

References 1. M.C. Paulk et al., The Capability Maturity Model: Guidelines for Improving the Software Process, Addison-Wesley, Reading, Mass., 1995. 2. M.C. Paulk, “Using the Software CMM with Good Judgment,” ASQ Software Quality Professional, vol. 1, no. 3, June 1999, pp. 19–29. 3. D.L. Johnson and J.G. Brodman, “Applying CMM Project Planning Practices to Diverse Environments,” IEEE Software, vol. 17, no. 4, July/Aug. 2000, pp. 40–47. 4. K. Beck, Extreme Programming Explained: Embrace Change, Addison-Wesley, Reading, Mass., 1999. 5. “eXtreme Programming Pros and Cons: What Questions Remain?” IEEE Computer Soc. Dynabook, J. Siddiqi, ed., Nov. 2000; www.computer.org/seweb/ dynabook/index.htm (current 24 Sept. 2001). 6. L. Williams et al., “Strengthening the Case for Pair Programming,” IEEE Software, vol. 17, no. 4, July/Aug. 2000, pp. 19–25. 7. A. MacCormack, “Product-Development Practices that Work: How Internet Companies Build Software,” MIT Sloan Management Rev., no. 42, vol. 2, Winter 2001, pp. 75–84.

For more information on this or any other computing topic, please visit our Digital Library at http://computer.org/publications/dlib.

About the Author Mark C. Paulk is a senior member of the technical staff at the Software Engineering Institute. His current interests include high-maturity practices and statistical control for software processes. He was “book boss” for Version 1.0 of the Capability Maturity Model for Software and project leader during the development of Software CMM Version 1.1. He is also involved with software engineering standards, including ISO 15504, ISO 12207, and ISO 15288. He received his bachelor’s degree in mathematics and computer science from the University of Alabama in Huntsville and his master’s degree in computer science from Vanderbilt University. Contact him at the Software Engineering Inst., Carnegie Mellon Univ., Pittsburgh, PA 15213; [email protected].

focus

reports from the field

Launching Extreme Programming at a ProcessIntensive Company James Grenning, Object Mentor

A company that has traditional formal processes launched a project using many Extreme Programming practices. The author covers how XP was proposed to management, how the project seed began and grew, and some of the issues the team faced during its first six months. 0740-7459/01/$10.00 © 2001 IEEE

his is a story about starting a project using an adaptation of XP in a company with a large formal software development process. Defined Process Systems is a big company (the name is fictitious). The division I worked with was developing safety-critical systems and was building a new system to replace an existing legacy product. The project was an embedded-systems application running on Windows NT and was part of a network of machines that had to collaborate to provide services.

T

Through a systems engineering effort, DPS divided the system into subsystems, each of which was later designed by a development team. I was brought in to help one of the teams start its part of the project using iterative development, use cases, and objectoriented design techniques. Shortly after starting this project, I spent a week in XP Immersion I, a one-week intensive training class in the techniques and philosophy of Extreme Programming.1 Enthused by XP, I talked to the DPS team about applying some XP practices to our work. The company had already decided to try iterative development and OO techniques, departing from the division’s standard process. They were game for something different, but how different? We decided to take the idea of incorporating XP practices to the director of engineering.

What we were facing The standard practice in the division was created in response to past and current problems. Most of the developers had one to three years’ experience. The more senior developers had the role of reviewing and approving the work of the less experienced developers. To their credit, the review team took the formal review process very seriously. They were good at it: they captured issues and defects on the Web, prepared reviewers, and held crisp review meetings. In the world of Big Design Up Front and phase containment, these guys were good. I could end this article right now except for one problem: all this process added overhead to the development of software. To design something, they needed to figure out the design, document it in Rose, schedule a review meeting, and distribute materials to November/December 2001

IEEE SOFTWARE

27

The need for documentation was ingrained in the culture, so we expected concern over XP’s lack of formal documentation.

review. Then reviewers had to review the materials and enter issues on the Web, have a review meeting, document the meeting outcome, repair the defects and then close them on the Web, fix documents, and have the changes reviewed again. However, all this process work was not keeping bugs out of the product. Unrealistic deadlines and surprises late in the project were taking their toll. Products were delivered late. Engineers were just getting their skills to a decent technical depth, but they were also burning out and heading for systems engineering or management. The team struggled with how to begin the new project. Its requirements were in prose format and fully understood only by the person who wrote them. To summarize: the existing process had a lot of overhead, deadlines were tight, engineers were running away, requirements were partially defined, and they had to get a project started. With all these issues, something had to change. Did they need something extreme like XP? I believed and hoped XP would help. Choosing your battles The DPS culture values up-front requirements documents, up-front designs, reviews, and approvals. Writing the product features on note cards, not doing any up-front design, and jumping into coding were not going to be popular ideas. To people unfamiliar with XP, this sounded a lot like hacking. How did we get by these objections? Having been introduced to XP, the group understood what the main objections would be as we tried to sell XP to the management team. Like good lawyers, we prepared anticipated questions along with their answers for our presentation. We expected that the decision makers would consider some of the practices dangerous and unworkable at DPS. The need for documentation was ingrained in the culture, so we expected concern over XP’s lack of formal documentation. Can the code be the design? Can we really build a product without up-front design? What if there is thrashing while refactoring? What about design reviews? To paraphrase Kent Beck, one of XP’s originators, “do all of XP before trying to customize it.” I think that is great advice, but for this environment we would never have gotten the okay to mark up the first in-

28

IEEE SOFTWARE

November/December 2001

dex card. We decided to choose our battles. We needed to get some of the beneficial practices into the project and not get hurt by leaving other practices behind. We did not omit practices that we didn’t feel like doing; we tried to do as many as we could. We used the practices and their interactions as ways to sell around the objections. We started by identifying the project’s goals—to build a working product with reliable operation and timely delivery, with enough documentation to enable effective maintenance (no more, no less), and with understandable source code. This was as objectionable as motherhood and apple pie. The standard process would identify the same objectives. We all agreed that a reliable working product was a critical output of the project. This was particularly important, as this was a safety-critical system. A tougher question was, what was enough documentation? This was where it got interesting. This application was not your typical XP target application—it was part of a larger system that multiple groups were developing at multiple sites. These other groups were using the standard, waterfall-style DPS process, not XP or short-iteration development. We had a potential impedance mismatch between the XP team and the rest of the project. How much documentation? Proposing no documentation would end the conversation. We decided to keep the conversation going and answer a question with a question. What did we want from our documentation? We needed ■

■ ■

enough documentation to define the product requirements, sustain technical reviews, and support the system’s maintainers; clean and understandable source code; and some form of interface documentation, due to the impedance mismatch between groups.

These answers did not all align with XP out of the book, but they kept the conversation going. XP is not antidocumentation; it recognizes that documentation has a cost and that not creating it might be more cost-effective. This, of course, violates conventional wisdom.

After acknowledging and addressing the project’s objectives, I led the team through the cost-of-change pitch from Beck’s Extreme Programming Explained.1 The director, the manager, and some senior technologists agreed that XP addressed many of their current development problems. They also thought XP right out of the book would not work for them. What did we want to do differently? Documentation and reviews were going to be the big roadblocks. We heard, “Requirements on note cards!?!” “I can’t give a stack of note cards to the test team.” “Bob in firmware needs the cards too.” “Someone will lose the cards.” I noticed that the company’s “standard” process allowed use cases in the form described by Alistair Cockburn.2 This is a text-based method, similar to user stories but with more details. Other DPS groups needed to look at the use cases, so we decided not to fight that battle—we had enough lined up already. We decided to use use cases. Another objection was “We need documentation for the future generations of engineers that will maintain the product we are building. We need our senior people to look at the design to make sure your design will work.” Our answer was that a good way to protect future software maintainers is to provide them with clean and simple source code, not binders full of out-of-date paper. Maintainers always go to the source code; it cannot lie, as documents can. XP relies on the source code being simple and expressive and uses refactoring to keep it that way.3 The source code is the most important part of the design. At some point, the maintainers will need a high-level document to navigate the system. A follow-on objection was that what one person thinks is readable source code is not to another. XP addresses this through pair programming. If a pair works hard at making the source readable, there is a really good chance that a third programmer who sees the code will find it readable, too. Plus, with collective code ownership, anyone can change the source code if need be. Another expected objection was that code was not enough—we needed a documentation or transition strategy for whenever a project is put on the shelf or transferred to another team. Ideally, the documentation

given to the maintainers describes the state of the software at the time it was shelved or transferred. This document can and should be written in a way that avoids needing frequent modification. As maintenance proceeds, the document will likely be neglected. Make the document high-level enough so that the usual maintenance changes and bug fixes do not affect it. Let the documentation guide the future developers to the right part of the code—then they can use the highquality, readable, simple source code to work out the details. Following this strategy will not result in a huge document. Remember, you have some of the best detailed documentation available in the form of automated unit tests—working code examples of exactly how to use each object in the system. XP does not prohibit documentation; just realize it has a cost and make sure it is worth it. You can plan documentation tasks into any iteration. The advice here is to document what you have built, not what you anticipate building. The next follow-on objection was that we’d never do the document at the end of the project. My reply: So, you would rather do a little bit at a time, and have to keep changing and rewriting it? Doesn’t that sound like it would take a lot of time? It does! So the management team must stick to its guns and do the high-level documentation task at the end of the project. Pay for it with less time wasted during development.

Documentation and reviews were going to be the big roadblocks.

Reviews The director did not completely buy into documenting only at the end of the project and hence made a final objection: “I still have my concerns. What if the design is no good? Do I have to wait until the end of the project to find out?” Pair programming was not reason enough for the management team to give up its review process. The big problem with DPS’s development process was that it guaranteed that development would go slowly. The process went something like this: create a design, document it in Rose, schedule a review meeting, distribute the materials, have everyone review the materials, have the review meeting, collect the issues, fix the issues, maybe do another review, then finally write some code to see if the design works. This made sure the cost of change was high. Because of the safety-critical nature of the November/December 2001

IEEE SOFTWARE

29

At the beginning of a project, you need to believe that the design can and will evolve.

application, the management team was not willing to give up on reviews. I proposed a more efficient way to do the reviews: the ask-for-forgiveness (rather than ask-for-permission) design process. We let the development team work for a month at a time on the system. At the end of the month, they assembled and reviewed a design-as-built review package. This took the review off the critical path, so the review process did not slow down the team. We agreed to document the significant designs within the iteration and review them with the review team. We gave the remaining issues that the reviewers found to the next iteration as stories. The idea here was to spend a small amount of time in the iteration documenting the design decisions that month. As it turned out, we really did not have to ask for forgiveness at all. It’s not about XP It’s about building better software predictably and faster. We compromised on a number of issues, but we did have agreement to use most of the XP practices: test-first programming, pair programming, short iterations, continuous integration, refactoring, planning, and team membership for the customer. Table 1 describes the XP practices we used and how we modified them to suit our needs. We added some process and formality: use cases, monthly design reviews, and some documentation. This adaptation of XP was a significant change in how DPS developed software, and we hoped to prove it was an improvement. The director thought XP offered a lot of promise for a better way to work that could lead to improved quality, faster development, better predictability, and more on-the-job satisfaction (“smiles per hour”). He said we were “making footprints in the sand.” If we could improve one development factor—quality, job satisfaction, productivity, or predictability—the director thought it might be worth doing XP. If we could improve any two of those factors, he thought there would be a big payoff. The problems of poor quality, delivery delays, long delivery cycles, and burned-out engineers plague the software industry. XP struck a cord with our team leaders. The techniques appeared to address some of the problems the team was facing, and the focus on testing and pair programming could help it

30

IEEE SOFTWARE

November/December 2001

build a high-quality product. XP’s iterative nature could help the team determine how fast it could go and give management the feedback it needed. So, it’s about getting the job done. XP is a set of techniques that seemed promising. Projects can get stalled in the fuzzy front end.4 This is a problem, especially in waterfall projects, where you must define all the requirements prior to starting the design process. In XP, as soon as you have defined a couple weeks’ worth of user stories, development can start. Think of how hard it is to shave a month off the end of a project. Now think of how easy it would be to save that month just by starting development as soon as you have identified a month’s worth of stories. Throughout the rest of the project, story development occurs concurrently with story implementation. The first iteration The team had three people—a customer and two developers, including me. We started by getting the unit test tool CppUnit set up and integrated with our development environment, VC++. This did not take long—the tools are relatively easy to use. The project’s customer (our systems engineer) gave us a requirements document. As we identified a functional requirement, we wrote it on a card. Each card named a use case. We did not bother to elaborate the use cases, just name them; in a few days, we had identified 125 use cases. Picking the most important ones was relatively easy using this list. In XP, the customer chooses the most valuable user stories and discusses them with the programmers. We were using use cases, a similar idea; our customer chose the most valuable use cases and elaborated them. For the early iterations, we decided to ignore use case extensions (which hold the special cases or variations) and keep the product definition simple.2 We just assumed there were no special cases or error cases; because we were using XP, we believed we could ignore the details and not be penalized later. We also did not bother using use case diagrams, because they did not add any value to the development team. Our main goal in the first iteration was to build a little bit of the product, get some experience, and build some skill and confidence. At the beginning of a project, you need to believe that the design can and will evolve.

Table 1 Summary of XP practices used XP practice

Adoption status

Our experience

Planning game

Partially adopted

Small releases Metaphor

Adopted Not adopted

Simple design

Adopted

Functional testing

Adopted

Test-first design

Adopted

Refactoring Pair programming

Adopted Adopted

Collective ownership

Adopted

Continuous integration

Adopted

40-hour week (also known as sustainable pace) On-site customer

Adopted

Coding standards

Adopted

Open workspace

Not adopted

The team practiced scope limiting, task breakdown, and task sign-up techniques. We used use cases rather than user stories. We wrote the use cases from an existing requirements document and stored them in a database. Iterations were one month long each. A metaphor had not yet evolved, and we didn’t develop one. Instead, a high-level design evolved and was recorded in a small set of UML diagrams and explanatory text. It played the role of our metaphor. The design did not have anticipatory elements. A monthly design-as-built review let the senior people monitoring the project see the team’s design decisions. We developed functional tests in a custom scripting language. The tests demonstrated that the application logic met the customer’s need. However, the team got behind on automated acceptance tests. This is not recommended. We wrote the first line of production code using test-first design. We wrote the program in C++ and used CppUnit 1.5 as our unit test tool. We refactored regularly, and the design evolved smoothly. We could develop tests, interfaces, and simulations on our own but used pair programming to create production code. We collectively owned the code. On one occasion, new code that required special knowledge resulted in a module owned by one programmer. Development slowed on that part of the system. During our first iteration, continuous integration was no problem. As soon as we added a second pair to the project, the team had integration problems. We quickly learned how to avoid collisions and do merges. The team worked at a sustainable pace. A couple times, we put in overtime to meet the iteration goals. A systems engineer acted as the on-site customer. We derived our acceptance tests from the use cases. The customer was not directly responsible for these tests. The team scared the customer a couple times by delivering his use cases in less time than it took him to define them. The main coding standard was “make the code look like the code that is already there.” The team used header and C++ file source templates to provide the company-required comment blocks. The comment blocks were mainly noise that hid the code. There was no open workspace. Workstations were in the corners, making pair programming awkward. The roadblocks to building a team workspace were political.

Partially adopted

Otherwise, the desire to do up-front specification work will put the team into analysis paralysis. Knowing that the design can evolve sets the team free to start building the system as soon as some functionality is identified. Simplifying assumptions keep complexity out of the code, at least temporarily. John Gall wrote, “A complex system that works is invariably found to have evolved from a simple systems that works.”5 Thus, it really helps to make these simplifying and scope-limiting decisions in each iteration. This lets the core features drive the design. We had a planning meeting and discussed the features that were to be included in the first iteration. My partner and I had OOD experience but no real XP experience (except for the week I spent in class). We wanted a guide to our first iteration, so we spent

about half a day with a whiteboard looking at design ideas. Because we were unsure of XP, we were not sure if we could really start coding without doing some design. We did a little bit of design, or as we called it a Little Design Up Front (LDUF). Ron Jeffries calls this a Quick design session.6 In our LDUF session, we found a group of collaborating objects we thought would meet the needs of our first stories. We worked from hand-drawn copies, not bothering with a diagramming or CASE tool. We knew things would change, and we did not want to spend our time making it look pretty. It gave us confidence and a vision of where we were going. During the iteration planning meeting and our LDUF session, we identified some of the interfaces we needed to support the iterNovember/December 2001

IEEE SOFTWARE

31

Figure 1. Simulating the interface by creating a Mock Object, shown in Unified Modeling Language.

Application logic

Service Interface class

<<extends>> Service implementation

Simulated (mock) service implementation

Hardware API

Figure 2. The test-first design process.

Write a test for new feature Refactor as needed

Compile

Run the test and see it pass

Fix compile errors Write the code

Run the test and see it fail

ation 1 features. These acted as placeholders for the real hardware. We then added a simulation of the interface by creating a Mock Object (see Figure 1).7 This facilitated development and also kept volatile entities such as the database schema, GUI, hardware dependencies, and protocol from creeping into the application logic. We witnessed an example of designing for testability leading to reduced coupling in the application. So we sat down to write our first line of code. We picked a candidate class from our hand-drawn LDUF diagram and followed the test-first design process. Figure 2 represents the test-first design process, XP’s innermost feedback loop. Our first line of code was a test. The test did not compile. We fixed the compile. We ran the test. It failed. We fixed the test. We were finally in maintenance! We also established pair-programming guidelines. We could develop test cases, simulators, and interface classes on our own but had to do all other production code in pairs. The first iteration had significant downtime, considering only one pair was 32

IEEE SOFTWARE

November/December 2001

working on the project. Meetings were a real productivity killer. Pair programming was fun and intense; we were able to stay on task. If one partner lost track of where we were going, the other partner quickly resynched their efforts. We taught each other about the tools and learned new skills. The coding standard is a team agreement to make the code look the same—it is the team’s code, so it should all look the same. It turned out that my partner and I had a compatible coding style. As more people joined the team, we established a self-documenting coding standard: “Make the new or modified code look like the code that is already there.” Cool, a one-line coding standard! However, there was pressure to use the division’s coding standard. From a “choosing your battles” point of view, we gave into using the standard comment blocks in front of each function. Later iterations We planted a seed of functionality at the center of this subsystem application and simulated its interactions with its environment. We brought in new stories that made the seed grow, incrementally adding new functionality and complexity. From its simple beginning of a half dozen classes and a few simulations, the clean, loosely coupled design evolved to about 100 classes. Unit test volume grew. The unit tests saved us numerous times from unexpected side-effect defects. We could fix the defects immediately because we just added the code that broke the tests. We created our own acceptance test scripting language to drive transactions into the system and used text file comparisons of simulation output to confirm system operation. We were able to design simulations that stressed the system beyond the limits expected in the field. Unfortunately, the team got behind in acceptance testing. I do not recommend this. Evolutionary design Evolutionary design relieved a lot of pressure from the team. We didn’t have to create the best design of all time for things we were not quite sure about—only the best design for what we knew about at that moment. We made good design decisions one at a time. Our automated tests and refactoring gave us the confidence that we could continue to evolve the system.

Object-oriented design is an important supporting practice of XP. OO programming languages let you build software in small independent pieces, a practice that test-first programming promotes.8 Project manager surprises Not only were the programmers happy with their creation, but the project manager was as well. After the fourth iteration, he said, “I’d only have three documents by now! Instead I have a piece of the system that works!” The manager discovered another benefit. Usually a project manager coordinating a team’s work with other teams spends a lot of time juggling priorities and figuring out task dependencies. On the XP team, dependencies between features were almost nonexistent. We built features in the order of customer priority, not internal software framework order dictated by a BDUF (Big Design Up Front). The team was agile and able to adapt to the other subsystems’ changing needs. Building the team We built the team slowly, while we were developing skills. We felt we could absorb one or two people per iteration. We did not let newcomers take tasks right away, but used them mainly as pair partners during their first iteration. Then, as they got to know the system and our practices, they started to take on tasks at the iteration planning meetings. We didn’t assume that team velocity would increase when we added a new person to the team—we measure velocity, not predict it. The DPS way of developing software made up for programmer inexperience by having senior engineers review the less experienced engineers’ work. In XP projects, you must still address the spread of expertise; for instance, it is critical to have at least one senior engineer on the team. We don’t give senior people a big title or special role, but we need them. They help spread the wealth of knowledge, and both they and their pair partners learn. The end of the story Unfortunately, I cannot present the story of how the project completed, because it was mothballed due to changing market needs. This is in the spirit of one of XP’s mantras: Work on the most important thing first. Nevertheless, the team and the managers were

impressed with our results in terms of productivity and quality. Because of this project, two other pilot projects were started.

I

n my experience, when the engineers want XP, the management doesn’t, and if management wants XP, the engineers don’t. Where is the trust between management and engineering? To managers: Try XP on a team with open-minded leaders. Make it okay to try new things. Encourage the XP practices. Provide good coaching. Challenge your team to go against the status quo. Recruit a team that wants to try XP rather than force a team to use XP. Make sure the organization sees that no one will be punished for trying something different. When hand-picking XP practices, you might compromise the self-supporting nature of XP. Try as much of XP as you can. Iterations are short. Feedback comes often. To engineers: Develop a sales pitch. Identify problems that you might solve. Identify the benefits, identify the risks. Do a pilot project. Iterations are short. Feedback comes often.

About the Author James Grenning

is the director of consulting at Object Mentor and is currently practicing and coaching Extreme Programming. Areas of interest are organizational change, software process improvement, and the business impact of agile software development. He helped create the Manifesto for Agile Software Development (http://AgileAlliance.org) and is a signatory. He is also a member of the ACM. He earned a BS in electrical engineering and computer science from the University of Illinois, Chicago. Contact him at Object Mentor, 565 Lakeview Parkway, Ste. 135, Vernon Hills, IL 60061; [email protected].

Acknowledgments I thank the real client that provided the experience to write this article (who wished to remain anonymous). I also want to thank Chris Biegay and Jennifer Kohnke for a job well done in helping me prepare this article. I presented a earlier version of this article at XP Universe in July 2001.

References 1. K. Beck, Extreme Programming Explained, AddisonWesley, Reading, Mass., 1999. 2. A. Cockburn, Writing Effective Use Cases, the Crystal Collection for Software Professionals, Addison-Wesley, Reading, Mass., 2000. 3. M. Fowler et al., Refactoring: Improving the Design of Existing Code, Addison-Wesley, Reading, Mass., 1999. 4. S. McConnell, Rapid Development, Microsoft Press, Redmond, Wash., 1996. 5. J. Gall, Systemantics: How Systems Really Work and How They Fail, 2nd ed., General Systemantics Press, Ann Arbor, Mich., 1986. 6. R.E. Jeffries, Extreme Programming Installed, AddisonWesley, Reading, Mass., 2001. 7. T. Mackinnon, S. Freeman, and P. Craig, Endo-Testing: Unit Testing with Mock Objects, XP Examined, Addison-Wesley, Reading, Mass., 2001. 8. R. Martin, “Design Principles and Design Patterns,” www.objectmentor.com/publications/ Principles%20and%20Patterns.PDF (current 12 Oct. 2001).

For more information on this or any other computing topic, please visit our Digital Library at http://computer.org/publications/dlib.

November/December 2001

IEEE SOFTWARE

33

focus

reports from the field

Recovery, Redemption, and Extreme Programming Peter Schuh, ThoughtWorks

n the autumn of the New Economy, my employer, ThoughtWorks, recognized that one of its projects had maneuvered itself into a bit of a jam. In the software development industry it is well understood— and occasionally admitted—that such situations occur. What is less well-known, however, is how to address them. For this particular project, the prospects were not good. On one side (the rock), was a development team both behind schedule and coding to more than a year’s worth of

I The author recounts the tale of a development project’s rejuvenation through XP: the successes, the shortcomings, and, ultimately, the lessons learned. 34

IEEE SOFTWARE

requirements—gathered by the previous consultancy—that had outlived their build-by date. On the other side (the hard place), was a client that, acknowledging that it had switched horses midstream, was willing to scale back on functionality but refused to budge on delivery date. In short, ThoughtWorks found itself with a mere two months to both turn the project around and deliver a business-ready application. For the purposes of this article, we’ll say that the delivery date was February 2nd, Groundhog Day. The application under construction was Web-enabled and powered by Enterprise Java Beans. Although the system had a smart ntier architecture, its innards were seriously ill. The business objects served as little more than mappers to the database. The session bean methods—the application’s brain trust—were superlong tendrils of procedural code. The written code—its style, semantics,

November/December 2001

and legibility—varied wildly depending on the area of the application one happened to be nosing about in. Tests, where they existed, were expected to fail. Building and deployment were a near mystery to more than half the development staff. Possibly because of the problem’s sheer size and pervasiveness, ThoughtWorks was slow to realize how fundamentally bad things were. Even when the danger became apparent to the consultants on site, the team lead and project manager had to compete with several other projects for overstretched resources. However, once the situation was clearly stated and understood, ThoughtWorks responded with a swift and sustained effort. The problem areas in the application were identified, and developers with relevant expertise were brought in. After two long months, the project delivered on time. But we didn’t stop there. 0740-7459/01/$10.00 © 2001 IEEE

Damage control I was one of four “ThoughtWorkers” who began flying to the client site in early December—just two months prior to go-live—to assist the six developers already on site. Our mandate was to slap the application into shape no matter the cost. Full of vigor and with little appreciation for the hundred miles of bad road ahead of us, we saw this as an opportunity to test-drive Extreme Programming in a project recovery setting. By mid-January, conference-room dinners were the norm, all-nighters customary, and weekends worked through. Although we bought, distributed, and even read and quoted Extreme Programming Explained,1 we disregarded practically every XP principle—from the 40-hour workweek, to testing, to continuous integration. As the severity of the matter quickly became apparent, we were forced to concede that the impending deadline did not allow us the luxury of XP. Instead, we cut up the application and parceled it out by areas of expertise: servlets, documentation generation, business engines, database, and so on. Deemed untenable, the servlet package was all but rewritten. Elsewhere, refactoring became an opportunist’s endeavor. If a developer decided that refactoring would take less time than complementing ugly code with more of the same, then that developer refactored. Otherwise, we sucked it up and coded ugly. This was a humbling exercise that engendered such memorable comment tags as “It’s 3:30 in the morning. Please forgive me.” This was found atop a 150-line switch statement. Coding standards and test writing were goals we aspired to and occasionally met. In lieu of an object model or static documentation, one developer scrubbed the database clean and reverse-engineered it into a data model. The model was maintained regularly, and it served as the application’s most reliable high-level specifications. The 70-plus-hour workweeks would not have persisted had developer pampering not been administered by both the project manager and the top levels of ThoughtWorks management. Conference-room dinners featured catered sushi, the project manager graciously made 2 a.m. ice cream runs, and the company paid for last-minute, full-fare tickets for hurried trips home. ThoughtWorks’ management made it clear, through words, acts, and expen-

ditures, that it understood and genuinely appreciated the sacrifices the team was making. In the end, it all came together the last week of January. The application did its part by bursting nearly every seam on the 30th, forcing a 24-hour go/no-go decision. The team redoubled its efforts. Things improved. Functional tests passed. We performed the last build six hours prior to go-live. Four hours later, 5,000 rows of data were sneaked into the database. After eight weeks, we delivered the system—minus some requirements and plus a few too many bugs—on time. One member of the team best expressed the general consensus: “I’m glad I did that once. I never want to do it again.” The nuts and bolts of code reform On 3 February, the system was live, and we were all well aware of its shortcomings. The client had a long list of functionality for us to add, not to mention the requirements it had dropped to meet go-live. Meanwhile, the team was determined to refactor the application into something we weren’t embarrassed to leave our names on. The team lead and project manager negotiated with the client for time to apply “ease of maintenance” to the system (a senior developer proposed the term in response to the suggestion that “refactoring” not appear on timesheets). With the worst behind us and an easier road ahead, several team members (but by no means all) decided that this refactoring phase would be an opportune time to begin adopting XP. Walking out of the starting gate The switch to XP was a slow, unsteady process. Not only was the current code base a reluctant conspirator, but only perhaps a third of the team really supported adopting XP. Another third was impartial, and, as might be expected, the final third was quietly but vehemently wishing that all this extreme nonsense would just go away. What began as lone developers and pairs taking some first steps toward agile processes—with patience and determination—became a teamwide push to adopt XP. Testing and building both benefited from single-handed accomplishments. The few existing tests were unreliable, largely because they depended on nonrestorable data that had either been altered or dropped from the database. One developer set out to base the existNovember/December 2001

IEEE SOFTWARE

35

We attacked the buggiest sections of the application first.

ing tests on data that could be replenished, then bundled those tests into a JUnit-based test suite (see the JUnit Web site, www.junit.org). Another developer streamlined the build process, reducing it to a few simple steps that could be quickly learned, letting every developer on the team perform a build before checking in—our first step toward continuous integration. Some developers paired up to take on more traditional development tasks. We attacked the buggiest sections of the application first. Because we knew new functional requests were not a long away off, refactoring, for the most part, was pursued gently. Mediocre code was improved upon when convenient, while truly untenable code was gutted and rewritten. From a teamwide perspective, the senior developers advocated a unified coding standard, JavaDoc-based commenting (see the JavaDoc Tool homepage, http://java.sun.com/j2se/javadoc/index.html), and unit tests for all refactored or new code. Learning from successes and building on momentum Shortly after Groundhog Day, two developers began applying a constants pattern to the application. Because the constants, as they are wont to be, were used throughout the application, the switch was neither smooth nor painless. The general consensus was that refactoring was necessary and that the pattern was solid for both current use and extensibility. The team agreed, however, that we needed better communication for future refactoring. The result was an increase in email “advisories,” pick-up development discussions, and regularly scheduled code reviews. The team’s analysts had readily accepted the story card as their new document, both as a way to distribute functionality requests to the developers and as a basis on which to negotiate with the client. When the analysts handed the first batch of cards to development, some pairs and some individuals began cleaning up the portions of the application associated with their cards. Tests began to appear in the newly refactored areas of code, and these were added to the main suite. One developer made the build process portable, so everyone could build locally prior to checking in. The build machine was moved from an occupied developer’s space to an otherwise empty cube. The number of builds per day in-

36

IEEE SOFTWARE

November/December 2001

creased, and the number of broken builds plummeted. The one insurmountable impediment we encountered after Groundhog Day was the client’s lack of cooperation in XP. Without objecting to the shift, the client nonetheless refused to participate in the process in any meaningful way. The team’s project manager and analyst (both ThoughtWorkers) assumed the role of the XP customer by writing story cards, providing domain knowledge on demand, and communicating decisions. However, the client’s absence in these activities certainly affected us. Without a decision maker to sit in the same room with the team, we could never fully implement the planning game, nor could we convince the customer to make official iterations more frequent than every three months. The irony of all this was that ThoughtWorks was doing development on site. The client was paying to fly multiple ThoughtWorkers out every week, only to house them several office buildings down the road from any potential users. Making it up as you go along By April, nearly all the functionality we had originally promised the client had been coded into the application and had passed user acceptance testing. Once it was clear the project was no longer in imminent danger, a few team members turned their attention to the more fundamental aspects of development. Innovations in these areas furthered our adoption of XP. The first crucial change was the mandate of a nightly greenbar. (A greenbar is the graphically displayed result of a successful test run in JUnit; conversely, a redbar is the result of any test that fails.) It took weeks to whittle the error log down and see our first greenbar. When this occurred, we tacked a calendar to the wall alongside the build machine and recorded the result of each day’s last build with a red or green sticky note. With this highly visible performance measure serving as a reminder, developers began to work toward nightly greenbars. Then, about a month after the calendar was posted, it veered dangerously into the red, just as a major delivery date was approaching. After five days of consistent redbars, one developer sounded the alarm, emailing a teamwide plea for a greenbar. Having been alerted to the situation, the analysts and project manager be-

gan pressuring development to promote only builds that greenbarred, and the calendar moved back into the green. In the end, the calendar served two purposes. First, by providing a simple, straightforward metric, it gave development a clear and attainable performance goal. Second, because it was viewable and easily understood by the rest of the team, it served as a failsafe mechanism for development. When the developers—in a heads-down coding frenzy— failed to follow their own rules, the other team members could push them back in line. Once again, we improved the build process, this time simplifying it to a push-ofa-button procedure that eased the adoption of continuous integration.2 Possibly as important, we automated build promotion—from the development environment, through internal testing, to user acceptance testing (UAT). Automation started with code checkout and ended with the emailing of unit test results. This meant that even analysts could promote builds, and they did. Guided by the automated test results, which ran on every build, an analyst could promote the latest greenbar build into a testing environment. This saved development the hassle of being on call to perform the task and resulted in quicker feedback on new functionality and bug fixes. Finally, several developers teamed up to devise and code a test-data generator, dubbed ObjectMother.3 Through a handful of simple method calls, this utility provided a complete, valid, and customizable structure of business objects (think of an invoice: its lines, all related charges, remit to, bill to). ObjectMother yielded numerous benefits. Because it made the test suite database-independent, we could swap UAT and even production databases in and out of the development environment, letting developers code and debug against real data. Moreover, ObjectMother drastically simplified the creation of test data within code, resulting in three major benefits. First, developers were much less likely to cheat and base their tests on supposedly persistent data in the test database. Second, it became much easier to convert existing tests that did rely on persistent data. Finally, developers began writing more tests. Past the finish line and still running To reduce travel costs, the project began rolling off some of its more experienced personnel in late March, little more than three

months after the cavalry’s arrival. Junior developers who had proven their mettle now took greater responsibility, and fresh, impressionable recruits came on board. Good practices were passed along, and XP-based development gained more momentum. Within six months, the project had metamorphosed its well-deserved infamy within ThoughtWorks into high repute. If we had to do it all over again Notwithstanding everything I’ve said, what saved the project was not XP. Instead, it was a well-financed and tremendously successful death march. The client’s refusal to budge on the delivery date was the single greatest contributing factor to this outcome. Groundhog Day meant that the team could not step back and reassess the situation. It meant that we could adjust the course of development only by degrees, not by turning it on its head. The developers new to agile processes had no time to adopt a coding standard or collective ownership, build a continuous integration process, or learn to program in pairs. More often than not, we didn’t have time to refactor bad code, so the hack frequently won out over the simplest thing that could possibly work. The irony of it, however, was that Groundhog Day took the code live, and XPers prefer to work with live code. Although XP wasn’t the team’s immediate salvation, it did, in the end, make the application sustainable beyond Groundhog Day. Our gradual adoption of XP recovered, retooled, and rejuvenated the code base. Because of the project’s nature, I believe we were correct to put many agile processes on hold during the first months of rehabilitation. However, we could have introduced some principles—such as improving the build and test processes—much earlier. This section distills our experiences with XP into a list of steps that might have best served our team (or any project in similar or less dire straits). Table 1 details our overall experience with each XP practice.

Within six months, the project had metamorphosed its welldeserved infamy within ThoughtWorks into high repute.

What you should do right away So, it’s two months to go-live, the team methodology to date has been waterfall, the project is a month and a half behind schedule, and Beelzebub is banging at the front door. What do you do? Lock the door. Okay, what next? November/December 2001

IEEE SOFTWARE

37

Table 1 Adoption of XP practices Practice

Adoption status

The experience

Planning game

Partially adopted

Small releases Metaphor Simple design

Not adopted Not adopted Fully adopted

Testing

Fully adopted

Refactoring

Fully adopted

Pair programming Collective ownership

Partially adopted Fully adopted

Continuous integration 40-hour week

Fully adopted Not adopted

On-site customer

Partially adopted

Coding standards

Partially adopted

The client never really got involved in the planning game (similar to the absence of an on-site customer; see below). Nor did the entire development team participate. We could never get client buy-in. The one XP practice we overlooked. We did not attempt to implement metaphor. This went hand-in-hand with refactoring. It became a serious endeavor and permeated its way through the application. Acceptance tests were written and used. There was, however, some kicking and screaming regarding unit tests. Some developers were behind them, some neutral, and some had to be shamed into writing them. In the end, even unit-testing gained team-wide support. At least half the team had refactoring in mind moments after they landed on the project, but serious projects were put off until after Groundhog Day. There was no difficulty mustering team-wide support after that. We began pairing seriously, but slacked off and settled in at perhaps 30 percent of all developer time. Easily accepted by the team. Developers still had their favored areas, but moved about frequently, and anyone could change any portion of the code. The process arose from our automated build scripts; the team quickly embraced it. After Groundhog Day, some weeks may have been 40, but some were 60. This depended on where we were in the iteration cycle. Although we were on site, the client never provided an XP-like customer. The project manager and team analyst stood in, with some success, in the role of the customer. A single standard was proposed and influenced some team members. Peer pressure had an effect at times. JavaDoc was incorporated into the build process.

Before considering XP or any best practices. For starters, I cannot stress enough how essential ThoughtWorks’ support was to the project’s initial success. First, I do not believe the project would have ever met its original goal without serious moral and financial commitments. Employees simply will not give up their lives for two months and deliver the near impossible, gift-wrapped, if they don’t have constant reminders of how important the matter is and how valuable they are. Second, ThoughtWorks contributed to our initial success by making intelligent staffing decisions. When a project is in danger and requires extra developers, additional resources must target the project’s specific needs. Finally, reputation matters. ThoughtWorks has long asserted that its employees are its greatest asset, and it has continuously backed those words with deeds. (Forgive me if this comes off as a shameless corporate pitch—it is not.) Target the build process. This is one of the first things we should have done. Developers are keen to things that yield the greatest benefit from the least effort. A long, arduous build process discourages developers from taking responsibility for the code they check in. Conversely, the simpler it is to perform a build—the less time it takes—the more likely it is that a developer will want to know that 38

IEEE SOFTWARE

November/December 2001

new code integrates successfully. Making the build process portable, so it can run on developer machines, further encourages responsible check-ins. Assuming the process is swift and easy, what developer would not check out the latest code, perform a clean compile, and check in with confidence? The build process doesn’t have to be true push-of-a-button at this stage, but it must be streamlined to a handful of steps. The benefit to the team is obvious and measurable: few things in software development are more discouraging than spending an entire day trying to make a clean build. Organize a test suite. Even if it is the first working test class to go into the code base, write an AllTests class and run it at the end of every build. Add any existing tests that do pass, or can easily be made to pass, to AllTests. Treat a build that redbars no differently than one that fails to compile. (You’ll have to sell this idea to the analysts—or the client—as well.) Encourage developers to write tests and add them to the test suite, but don’t shove test writing down their throats (at least not yet). Finally, I recommend against including old tests that fail, even if the team intends to get them working some time in the future. Seeing a greenbar has a particular psychological value. A redbar that you consider a virtual

greenbar because “those test never pass anyway” isn’t the same. Write an ObjectMother. Writing an object generator is not a trivial task, but it pays for itself by shortening the time spent writing new tests and fixing broken ones. The utility reduces the effort-versus-benefit ratio for test writing. Developers are much more likely to write a test when they can acquire an invoice and all its associated objects from one simple method call; they are much less likely to write the same test when the invoice, its lines and charges, the customer, the bill-to address, and perhaps the associated assets all have to be instantiated and bound together before calling get-total. An ObjectMother also makes it easier to maintain tests when their associated business objects change, because it consolidates the instantiations into one space instead of letting them be spread across the application. Practices to phase in early So you’ve taken care of the most crucial elements. Now you can move on to phase two. Continuous integration. If a project has a simple build process and a serviceable test suite, the next step is to combine these and tackle continuous integration. There are two ways to approach this. The first is the standard serialized XP practice based on a build machine or build token, where developers queue up one at a time to build, test, and integrate their changes.1 The second alternative, a variation we use at ThoughtWorks, is for developers to build and test on their own machines before checking in. Backed up by multiple builds per day (to keep everybody honest), this has proved a successful practice.2 Either way, continuous integration drastically reduces the time developers would otherwise waste converging code. It had a noticeable effect on our team as soon as we implemented it. Gentle refactoring. Refactoring is good, but at this stage in a project’s recovery, it must be tempered for several reasons. First, it may be difficult to get client buy-in. Second, the worse the code base is and the less likely it is to follow the principles of object-oriented abstraction, the more difficult it will be to isolate portions for retooling. Third, at least in the beginning, you probably won’t have either a quick build process or dependable test results as success in-

dicators. Nonetheless, do pursue gentle refactoring from the start. Remove insufferable portions of code and undertake any refactoring task that offers low risk and high value. Simply simplicity. Simple design is straightforward to those in the know and aggravatingly intangible for anyone anywhere else. For our team, as we wended our way along trails of bad code in the first months of project recovery, we could easily pick out issues such as overdesign and logic duplication; however, before Groundhog Day, we had no time to address these issues. After Groundhog Day, as refactoring and new functionality became everyday business, we remembered the squalor we had wallowed through and consistently strove for simplicity. We pushed logic that permeated through the session beans back into the business layer and again refactored the servlets. Weekly code reviews were initiated, and simplicity regularly rolled off nearly every tongue.

Refactoring is good, but at this stage in a project’s recovery, it must be tempered for several reasons.

Coding standard. This is easy to encourage without spending too much time or effort. In our case, we were fortunate to have two standards to fall back on. First, we had a proposed in-house coding standard that one team member had helped author several months earlier. This document was both a good primer for the greener developers and a good starting point for further discussion and debate. Second, we relied on the JavaDoc standard for code commenting, which we further leveraged by incorporating the generation of API docs directly into our build process. This practice is easiest to encourage when you have an in-house or open standard (such as JavaDoc) that you can simply pass or email about and then occasionally refer to during discussions. Stand-up meetings. We never introduced these, and I believe it was a major mistake. Quick, daily, face-to-face meetings keep developers informed about what others on the team are doing. They help stop people from stepping on each other’s toes. They keep the team lead informed about who’s ahead and who’s behind. They air new ideas and prevent people from duplicating work. Mind the database. Okay, this isn’t an XP process, but it’s definitely as important as one. The database is an essential component of nearly every business application—neglect it November/December 2001

IEEE SOFTWARE

39

Quick, daily, face-to-face meetings keep developers informed about what others on the team are doing.

at your peril. Nothing good ever comes of a database architected without a mind to conversion or reporting. Similarly, a database that updates schema for new attributes and entities but not deleted ones—a database that lets test data pile up and atrophy—will be cantankerous to develop with, hard to test on, and difficult to alter. Conversely, a well-architected and maintained database, through intelligent and efficient data organization, can guide good development. Finally, as mentioned earlier, in lieu of an object model or other documentation, a data model can provide an extremely handy overview of the application. What to do when the pressure lets up Groundhog Day has come and gone, and you’ve phased in several XP practices. So what’s next? Step back and relax. Once the project has met some of its immediate goals (a major delivery or go-live date), the development team should step back and get everything into perspective. If the time has come for serious refactoring, what parts of the application should be put through the grinder first? How is the adoption of XP coming along—who’s resisting and who’s welcoming it? How is the team’s mental health? If the last couple months have been a bloodbath, should some exhausted team members be rolled off the project? Would the project benefit from new recruits and fresh perspective? Iterations and small releases. I didn’t list this practice earlier because the principles in the first two groups weigh in as either more important or easier to accomplish. Why? Because a project plan is already in place, and negotiating for a formal change to that plan expends the scarce resources of both time and political capital. However, even without formal customer consent, a team can institute internal iterations. We did this. They were not the same as true iterations because the customer wasn’t bearing down on us, but breaking the work out internally was definitely better than delivering once every three months. Get an on-site customer. This is also on my wish list of XP practices to phase in early. I fear, however, that any project that falls to the level of depravity that ours did does so partly because there has been no real participation

40

IEEE SOFTWARE

November/December 2001

on the customer’s part. Therefore, attempting to foster customer input early in the recovery process may well be a Sisyphean endeavor. If your project can actually produce a real customer earlier, then Tyche has shined on you. Roll in the rest of XP. As the pace of the project returns to something akin to normal, the team should address the remaining elements of XP. As a rule, when you add functionality to poorly written areas of the application, refactor the code. Start looking at patterns. What parts of the application might benefit from their use? Encourage pair programming and make changes to the workspace to facilitate it if necessary. Story cards and the planning game should become the means by which the functionality is proposed, deliberated, and built into the application. We never got serious about metaphor or the 40-hour workweek; they should, however, be given serious consideration. Communicate, communicate, communicate. If you have so far managed to avoid instituting stand-up meetings, put them in place now. Whenever possible, XP principles should propagate from the bottom up rather than being imposed from the top down. Ideally, this means that the team as a whole decides what XP principles it will introduce and get serious about first. Involve the entire team in estimation. All these things foster a sense of collective ownership, not only of the code but also of the project’s general well-being.

H

ad our client been willing, many members of our team would have razed the code base and started again from step one. But few clients are so giving and few projects so fortunate. Furthermore, who is to say a project will not falter again—for similar or wholly different reasons? In such situations, little is achieved without a lot of hard work. XP had no hand in recovering our project but weighed in heavily on its redemption. Before Groundhog Day, although we discussed XP and made occasional, nearly clandestine incursions into agile development, we simply could see no way to rewrite our development process in the time we had. Rather, we stepped in to shore up and compensate for the

current process until we had the time to address the many issues in earnest. After Groundhog Day, we made a sincere and sustained effort to go XP, which was successful on many levels. In the end, we landed somewhere within the realm of agile development without quite having achieved XP—but it was quite a good place to be. I cannot stress enough how the team composition factored into our success. If we did not have so many stars and suckers—those capable of great things and those willing to put their lives on hold to make great things happen—the project would have never attained the success it did. Ultimately, our project’s redemption rested on two strong foundations: the processes drawn from Extreme Programming and the team that applied them. Acknowledgments First and foremost, credit goes to the fellow ThoughtWorkers with whom I endured the worst death march I ever wish to be a party to. I am quite grateful to have worked with such a team. Many thanks, also, go to Martin Fowler for suggesting the topic for the article and providing the guidance necessary to ensure that it was written. I presented an earlier version of this article at XP2001 in May 2001.

FOR CALL LES: ARTIC

re Softwa ction u Constr

References 1. K. Beck, Extreme Programming Explained, AddisonWesley, Reading, Mass., 1999. 2. M. Fowler and M. Foemmel, “Continuous Integration,” www.martinfowler.com/articles/continuousIntegration. html (current 16 Oct. 2001). 3. P. Schuh and S. Punke, “ObjectMother: Easing Test Object Creation in XP,” www.thoughtworks.com/cl_ library.html (current 24 Oct. 2001).

For more information on this or any other computing topic, please visit our Digital Library at http://computer.org/publications/dlib.

About the Author Peter Schuh is a team lead at ThoughtWorks,

Inc., a provider of custom e-business application development and advanced system integration services to Global 1000 companies. He has written and spoken about XP, the adoption of agile processes, agile development’s impacts upon database administration, and the ObjectMother pattern. With a BA in English literature and an MA in international relations, he is an unapologetic interloper in the software development industry. Contact him at ThoughtWorks, Ste. 600, 651 W. Washington, Chicago, IL 60661; [email protected].

HAVE YOU EVER HAD AN EXPERIENCE in constructing software that gave you unexpected insights into the larger problem of software engineering and development of high-quality software? If so, IEEE Software encourages you to submit your experiences, insights, and observations so that others can also benefit from them. A unique aspect of software engineering is that techniques used at the smallest scale of individual software construction are often relevant even for the largest software systems—and vice-versa. We are looking for articles that encourage a better understanding of this commonality between programming in the small and programming in the large, and especially ones that explore the larger implications of hand-on software construction experiences.

Possible topics include but are not limited to the following: • Coding for network (grid-based) applications • Coding for high-availability applications • Coding for compatibility and extensibility • Coding for network interoperability • Effective use of standards by programmers • Lessons learned from game programming • Techniques for writing virus-proof software • Simple and commonsense techniques for making multithreading and concurrency easier and more reliable • Coding implications of continually increasing processing and memory resources • Agents: When, where, and how to use them • PDAs and the future of “wearable” software • Is “agile” programming fragile programming? • Prestructuring versus restructuring of code • Integration of testing and construction

Construction Editor: Terry Bollinger, [email protected]

• • • • • • • • • • • • • •

Aspect-oriented programming Impacts of language choice on application cost, stability, and durability Proliferation of Internet-based computing languages (for instance, Perl, Python, Ruby) “Throw-away” programming and languages: Good, bad, or just ugly? Code ownership in an Internet economy When, how, and where to use open source choosing languages for your applications Lessons learned in buying and using tools XML: Where to next? What will follow it? How to keep documentation from diverging Personal-level configuration management Personal-level defect analysis Build processes for the new millennium Deciding when to branch code bases

Submissions are accepted at any time; contact [email protected]

focus

reports from the field

Using Extreme Programming in a Maintenance Environment Charles Poole and Jan Willem Huisman, Iona Technologies

uring the 1990s, Iona Technologies’ flagship product was a Corbabased middleware product called Orbix. We have since created a newer version of Orbix, but here we discuss how we developed and, more importantly, now maintain the older version, which represents the Corba specification’s early evolution. We also review the inherent problems of code entropy due to specification instability and time-to-market pressures. Over the years, because of these pressures and the rapid growth

D The authors review how efforts to introduce industry-level best practices led to Extreme Programming, which improved the team’s ability to deliver quality support and product enhancements. 42

IEEE SOFTWARE

in Iona’s engineering team, we often ignored good engineering processes and practices. As a result, the code base’s overall health degenerated and was salvaged only through two successful reengineering efforts and a series of infrastructure projects designed to improve overall engineering practices. It wasn’t until later that we realized how closely our efforts were tied to Extreme Programming. The problems When Iona moved from a start-up to a market-leading provider of middleware products in 1999, it faced four major problem areas: processes and practices, testing, code entropy, and team morale. An initial reengineering project that started in 1997 was concluding, and although it focused a tremendous amount of resources on reengineering, improving testing, and making nec-

November/December 2001

essary changes to the code base to comply with the latest version of the Corba specification, it did not sufficiently address the four problem areas; it merely skimmed the surface of what needed to be done. Processes and practices In early 1999, you could have asked two different engineers from the same Orbix team how they did their jobs, and each would have replied differently. This reflected the team’s lack of process documentation and visibility and its failures to make process a part of each engineer’s personal software practices. Also, there was no focus on process improvement. The junior engineers assigned to maintenance and enhancement had no experience with good engineering practices and had only a rudimentary understanding of process. To compound the problems, the maintenance and 0740-7459/01/$10.00 © 2001 IEEE

enhancement team used disparate source control elements across two globally distributed development sites and lacked a welldefined and tested interface between configuration units. Dependency management was a nightmare. Testing In general, quality was never one of Orbix’s strong points. We didn’t document test coverage very well, so good metrics were not available. Our test suites were cumbersome, difficult to run, and impossible to accurately report. Also, the test suite used to provide interoperability testing with other Iona products and product components—or against other Corba middleware products—was not automated across the platform set to which we delivered our product. Thus, system testing our product releases was difficult. Furthermore, instead of having the development and maintenance teams monitor quality, a team that was detached from the engineering effort monitored it through checklists and forms. Code entropy By the end of 1997, we had already patched and repatched the Orbix code hundreds of times to address numerous customer issues as well as the changing Corba specification. These patches often used band-aid approaches to resolve problems or add functionality and were a major factor in the code’s rapid entropy. Many of these unreviewed changes were made to a code base that was never designed to withstand the punishment meted out by many of Iona’s larger customer deployment environments. Customers demanded fixes and faster resolve times. The code’s degradation further accelerated when Iona removed most of the senior development staff from the maintenance team so that they could develop new products. Finally, there was limited acceptance of the well-documented style guides that Iona’s chief architect specified for all code development, making it difficult to read the code. Poor structuring of the source code’s directory space also made it difficult to quickly become familiar with the overall code structure. Team morale People indicated in reviews that they didn’t feel cohesiveness in the team. Many reported

poor visibility of the projects on which people worked. In general, we felt underappreciated and overworked. Initial history of change In 1999, prior to the release of Kent Beck’s Extreme Programming Explained,1 Iona undertook several projects to address its problem areas. It is in the context of these projects that it introduced Extreme Programming, realizing that many of the things it was implementing were elements of XP. Reengineering The second reengineering effort initially focused on resolving problems—which many customers reported—related to poorly implemented modules. Bug clusters highlighted in our defect-tracking system clearly identified these problem areas. Today, this reengineering effort continues as a part of our XP effort, which helps engineers resolve problems. An additional outcome of the initial effort was reducing the code’s complexity by stripping out unused code and implementing several patterns that have made it easier to maintain, test, and understand the code. We’ve reduced code size by over 40 percent. Improving engineering practices Jim Watson, one of Iona’s distinguished engineers, led the reengineering project. In addition to providing strong technical insight into coding problems, he was also instrumental in promoting stronger engineering practices. He initiated several practices to encourage growth in the engineering team, including weekly presentations that gave everyone an opportunity to present technical topics of relevance or interest such as merge strategies, patterns, and estimation. He emphasized code reviews, adherence to source-code management guidelines, and ownership of and responsibility for code style. He made engineers consider how they could constantly improve the quality of their work in areas such as test coverage, and he established a more proactive approach to problem solving. His practices still strongly influence the team. Automating everything We established a separate project to clearly understand the build and test dependencies across the older product set’s various November/December 2001

IEEE SOFTWARE

43

We were suddenly on the road to a consistent and automated build, test, and release infrastructure for nightly product builds and tests.

elements and to fully automate the build and test process. This is an ongoing project, but so far we’ve been able to build and unit test the complete product set on a nightly basis. Efforts continue to better automate the interoperability and system tests, which still require some manual intervention. Team consolidation Before starting the reengineering projects, over 70 engineers maintained and enhanced the Orbix products. By the time we completed the major portion of those projects, we had reduced the team to around 40 but were servicing three times the number of customers. Today, the team size is down to 25. In addition to the personnel consolidation, the team managed a single mainline of code using a well-defined set of common rules that govern how to merge fixes and enhancements into the consolidated mainline. Extreme maintenance The second reengineering effort resulted in a remarkable transformation—we were suddenly on the road to a consistent and automated build, test, and release infrastructure for nightly product builds and tests. The reengineering and refactoring efforts eliminated much of the code complexity and stagnation, resulting in a clean, well-structured code base that conformed to Iona code standards. So, if we saw so much improvement, why consider XP? Despite our progress, we had yet to resolve issues of testing, visibility, morale, and personal work practices. From a management standpoint, we wanted to do more with less: higher productivity, coupled with increased quality, decreased team size, and improved customer satisfaction. The engineers wanted more time to do a good job instead of always feeling pressured to deliver fix after fix. So we started looking at XP and learned that many of its elements come naturally to teams working the maintenance game. According to Kent Beck, “Maintenance is really the normal state of an XP project.”1 In our earlier projects, we created a set of maintenance processes that describe how a bug is created, prioritized, analyzed, fixed, and delivered to the customer. The Generation 3 team also had a set of processes that explain how to request, describe, prioritize, and implement incremental enhancements

44

IEEE SOFTWARE

November/December 2001

and deliver them to customers. Clearly, these processes already incorporated important XP elements—extreme maintenance means following the XP model while a product is in the mainstream of its product life cycle.2 The synergy was so great that we started to use the term XP to describe the set of processes and practices the Generation 3 element and product teams used. We then started the XP experiment in earnest in an attempt to resolve our outstanding issues. Table 1 presents all of the practices presented in Extreme Programming Explained1 and describes how we approached them. Metaphor When customers report a bug, they describe it in the form of a short story that explains what they are trying to do and how the problem manifests itself. We then note this story in our defect-tracking system and make it available to the customer and everyone working with that customer. When we receive an enhancement request or functional requirement, we present it as a specification on our requirements Web page. Because of the need for global visibility, and because customer service representatives and engineers must be able to update the system, unlike the bug reports, these requirements are not presented as index cards and don’t read as stories (yet). However, they do contain essentially the same information as traditional requirements. For the purpose of our XP experiment, we left the enhancement request until a later improvement phase and instead focused on tracking the bug-fixing stories in a wellstructured Web-based queuing system, visible to the internal customer (or Customer Service), engineers, and engineering managers. We liked the idea of using index cards to track tasks and decided to implement a storyboard for each of the element teams comprising our consolidated Generation 3 team. Our goal was to improve the poor visibility of each engineer’s work effort. We wrote each task, regardless of whether it was a customer issue or internal project task, on a color-coded card (see Figure 1). Nirmalya Sengupta, our Generation 3 operations lead, then sat down with our customer (or Customer Service representative), so he could prioritize his list of bug and internal tasks. The customer could also view internal tasks and ask to have his tasks escalated above internal tasks. Cur-

Table 1 Extreme Programming practice adoption Extreme Programming

Adoption status

The Orbix Generation 3 experience

The planning game

Partial adoption

Small releases

Followed religiously

Metaphor

Adopted from start

Simple design

Partial adoption

Testing

Adopted from start

Refactoring

Adopted from start

Pair programming

Sparingly adopted

Collective ownership

Adopted from start

Continuous integration

Adopted from start

40-hour week On-site customer

Not adopted Adopted from start

Coding standards

Adopted from start

We haven’t found it necessary to follow a strict iteration planning process. However, to understand and manage capacity, it is still critical to monitor estimates of ideal time and actual time to completion. Most bugs in our product take about two weeks to fix, so we fell into twoweek patch release iterations. Our customer service and sales organizations act as our customer proxies, providing a coordinated twice-weekly prioritization of bug issues for each patch. They also run the test cases that are our acceptance tests for each customer story against each patch delivered to a customer. We still get interruptions and midcycle reprioritization, but it’s unavoidable. Short release cycle requirements drive our support efforts, so our customer patch cycle is two weeks. Our point release cycle has moved from months down to weeks. Without automated nightly regression and unit testing, we wouldn’t have come this far. Without a single button integration test capability, engineers wouldn’t have been able to ensure that changes had not broken code. Without optimizing our test suites around test time, we wouldn’t have been able to provide the necessary timely feedback. Our metaphor is the customer story as detailed in our bug-tracking system. Our customer supplies our acceptance test for each story. There hasn’t been as much focus on simple design, because the product is complex, and we continue to use high to midlevel design documents to help people quickly understand the over all system and some system components. In our maintenance environment, there is a clear story on which to focus. We tend not to generate design documents for fixes but instead focus on implementing the story and passing the acceptance and regression tests. Test first is naturally a part of the maintenance process in the Generation 3 team. We don’t work on bugs unless a test case is developed and all test cases are run nightly. We also use code reviews to back up the lack of great test coverage. We took this on as one of the cornerstones of our efforts in improving code maintainability and stability. More often than not, the call is to focus on refactoring as a part of an engineer’s personal practices. Sometimes in refactoring, we have found ourselves doing wholesale reengineering, but this doesn’t happen often. Although we’ve tried it, it is not something the team widely practices. We’ve used code reviews to try and address code standards, design of fixes, and so forth. We have people changing code everywhere. Some have a stronger knowledge of certain areas, but we have fewer people maintaining the code and we can’t afford not to have people working on all parts of the code. For the most part, we’ve used pairing to gain an edge. We also have a strong set of code standards we enforce not only at code reviews but also as a part of the check-in process in our source-control system. It took us 18 months to build an automated testing system that was comprehensive but also easy to use and fast. Also, we don’t use integration stations. Instead we use a mutex file (requiring check in and check out) in our source control system to ensure that no one else is integrating while we are merging into the mainline. Our merge process is strong and followed quite well. We haven’t felt courageous enough to tackle this. We use our customer service team and sales force to act in the role as a customer. They set priorities and generate acceptance tests. We had already started this one when we began using XP.

rently, only customer issues are rigidly prioritized, but we intend to integrate internal tasks into the prioritization process as well. (Kent Beck suggests having a Web-accessible digital camera positioned so that people can regularly look at and zoom in on the storyboards—but that’s for a future project.) An engineer estimates the completion date after 24 hours of initial analysis. When the

task is completed, the engineer records the actual date of completion, and if there were delays, she notes reasons for the delays, observations, and lessons learned on the back of the task card. We remove all closed tasks from the board, extract the data, record it in a spreadsheet, and store the task card in a task log. In general, bug-fixing tasks should be completed as two-week pieces of effort. If, after November/December 2001

IEEE SOFTWARE

45

Pair programming in XP is critical to improving collaboration and greatly facilitates mentoring and improvements in engineering practice.

analyzing an issue, an engineer estimates more than two weeks’ worth of work, she splits the story into several tasks. Anything more is a refactoring project and should be structured more along the lines of an enhancement with clear incremental delivery of functionality based on tasks extracted from the original customer story. We also incorporated the daily stand-up meeting into the team (see Figure 2)—an XP element. Each team took 15 to 30 minutes to review their progress, and each person took a couple of minutes to talk about what he or she was working on that day and the previous day. The focus was not only on improving visibility but also on encouraging communication between team members. The qualitative results were immediate. Some engineers found it difficult to explain why they spent so much time on issues or why they looked at things that weren’t high priorities on the team’s list of tasks. Testing Automation is essential, and one of the first initiatives we undertook in 1999 was to automate our test suites and speed them up. We completed an extension of that effort in 2000, which lets developers build and test the entire product set on any platform combination we support by clicking a button from a Web-based interface or using the workstation command line. These initiatives let us test the entire product set against the Corba specification, the Orbix API, and customer-supplied test cases on a nightly basis or at the engineer’s discretion on the 17 platforms we support. Both the stories describing the functionality that the software supports and those describing the bugs that customers encountered have tests associated with them. Engineering requires each customer to provide a test case for each reported bug prior to doing any work on the issue. This test case is automatically included in the test suite’s nightly runs. Task: Area:

Details:

Customer: Engineer:

Figure 1. The colorcoded task card. 46

IEEE SOFTWARE

Date queue Date active

November/December 2001

Expect finish Date closed

Any engineer must be able to build the entire product or system and run the test suite to get immediate feedback on changes made. Nightly builds are a reasonable intermediate step to this test-on-demand requirement, and again the processes we decided to follow map well to the testing idea at the XP model’s foundation—test before you implement, and test everything often. Pairing Pair programming in XP is critical to improving collaboration and greatly facilitates mentoring and improvements in engineering practice. However, convincing engineers that it is useful and should be incorporated into their work practices is extremely difficult. Our pair programming experiment came about accidentally in late 2000. The Generation 3 team worked on a particularly difficult set of customer issues. We had a customer engineer working onsite with us. (Having customers onsite is another XP principle, although normally we treat our onsite product managers and customer service representatives as our customers.) At different times, the customer teamed up with various members of the Generation 3 team to do some pair programming. Likewise, members of the team gravitated toward a pair-programming approach as they progressed with the issues on which they were working for this particular customer. In the end, the qualitative feedback from the team was positive. They felt that they worked more effectively and with a higher level of communication as opposed to working independently on individual issues. Also, the senior staff enjoyed some of the mentoring that naturally occurred as they interacted with the junior members of the group. The overall productivity during this period was also high and morale improved. Engineers got a lot of positive reenforcement from each other, and engineering practices and quality improved as we actively noted coaching interactions between engineers. We also got no patch rejections for the work these engineers did. We are currently trying to formally implement pair programming. Until we do, we have asked each engineer to take on one or two tasks each quarter in which they pair with another team member. We’ll note the tasks in which they worked in pairs and will show them the data at the end of the quarter to

(hopefully) reinforce the results we got with our initial pair-programming experiment. Small releases In addition to describing functionality, XP focuses on increments of functionality that engineers can quickly merge into the mainline. They merge in bug fixes quite rapidly, and with the improved nightly build and test and the one-button test, they can make systems available as a temporary or permanent patch within one day. Any initial effort should focus on releasing a patch every iteration for a large, complex system—and automation and improved testing enable that end. For projects approaching XP at midlife, we encourage focusing on automation at any costs. Rapid turnaround time can only improve customer satisfaction. Unfortunately, because we just recently started collecting metrics on mainline breakage, we don’t have statistics to show what is currently a qualitative analysis of Iona’s improvement. In addition, because the code base we were working with was a legacy code base, we did not have much to say about how it was put together. However, we needed to make it maintainable and add functionality if required. As mentioned earlier, one of the improvement efforts we undertook was a major reengineering of elements of the Orbix product. This effort incorporated stories that typically took three to four weeks to implement. By keeping these increments relatively short, we could more effectively manage the delivery schedule of our last minor point release by modifying the project’s scope but not its delivery date, because we were almost always at a point where we could deliver a functional product.

views have revealed that refactoring has become part of the engineers toolkit when approaching a problem. Along with analyzing a problem, they now identify whether the area of code they are working on is a candidate for refactoring, and they follow up after delivery with a refactoring task on the storyboard. Sometimes this extends into a reengineering effort if the scope of improvement is increased, and for some customer issues, we can’t afford the time to do the refactoring effort due to commitments to restore times. However, for the most part, we’ve seen fewer problems in our new releases, a significant decrease in code entropy (as measured by comments from code reviewers), no patch rejections over the last few months, and reduced code complexity (again, as measured by comments from code reviewers). Unfortunately, we have not been able to collect complexity metrics to verify this. Facilities strategy A maintenance team environment should combine personal and team space to enhance interactions and collaborations that should occur in the team. Although not an explicit XP principle, our facilities strategy continues to be key to XP’s success. The pair programming could not have happened without changing the engineers’ workspace, and the collective ownership, testing, and continuous integration would have suffered. Prior to initiating our XP project, Iona’s engineering workspace consisted of the typical Figure 2. A stand-up floor plan—it had multiple bays, each con- meeting at the storyboard.

Refactoring With regards to refactoring, analyzing bug statistics is important in identifying areas of code that might need refactoring or reengineering. This form of customer feedback is an important part of targeting improvements and stopping code entropy. Refactoring should always be on an engineer’s mind when analyzing a problem or enhancement request: “How can I make this area of the code work better?” as opposed to “How can I quickly fix this problem (using the band-aid approach)?” We addressed refactoring in our effort to improve personal practices. Recent code reNovember/December 2001

IEEE SOFTWARE

47

Figure 3. Orbix Generation 3 cubicles (a) prior to initiating the Extreme Programming project and (b) after.

(a)

(b)

collaborations that occur. These changes to the environment were instrumental to our success in improving overall team morale. We also have had a visible increase in the number of team interactions—not only on code reviews but also regarding ongoing tasks.

taining six shoulder-height cubicles (see Figure 3a). The engineers and engineering managers initiated a project to restructure the space in which they worked to enhance team communication. They created a large central area by eliminating 20 percent of the individual cubicles and establishing nondelineated group workspaces. The group workspaces consist of either a round or rectangular meeting table or workstation area. In addition, white boards and flipcharts on tripods were made available in each work area. They also purchased a couch, and a catering service now delivers snacks on a weekly basis (see Figure 3b). The results are noticeable. Code reviews happen in the group area; pair programming happens at the workstations. People discuss their ideas on the whiteboards and large flipcharts. There is a measurable increase in the visibility of what everyone is doing on the team and the number of conversations and

Figure 4. Team bugfixing productivity.

Qualitative improvements Our metrics (although not particularly extensive) seem to show some quantitative improvements in productivity during the period in which we used pair programming to address specific deliverables or bug fixes for a major customer. The productivity is based on a constant work force with no change in overall work habits in terms of hours spent fixing bugs. This productivity increase has continued beyond the initial experiment and is perhaps attributable to more people using pair programming and open collaborative spaces created during that period.

20 18 18

New bugs

16

Closed bugs

14

8 week moving average (closed) 11 10 8

8 7

7 5

22

IEEE SOFTWARE

November/December 2001

5

5

5 4

2

2 1

6-May-01

29-Apr-01

20-Apr-01

13-Apr-01

5-Apr-01

29-Mar-01

22-Mar-01

8-Feb-01

15-Feb-01

0 1-Feb-01

0

Week ending

48

6

3 3

2 1

25-Jan-01

00

4-Jan-01

0

11-Jan-01

0

7-Dec-00

26-Oct-00

19-Oct-00

5-Oct-00

12-Oct-00

21-Sept-00

14-Sept-00

7-Sept-00

28-Sept-00

0

3 2

1

30-Nov-00

1

1

0

3

2

2

23-Nov-00

2 1

16-Nov-00

2

3

9-Nov-00

2

3

4 4 3

18-Jan-01

3

3

2-Nov-00

4

4

6

5 5

5 5

15-Mar-01

4

4

7 6

8-Mar-01

6

7

6

6

1-Mar-01

8

2

9

9

9

22-Feb-01

10

14-Dec-00

Number of bugs

12 12

70

Implemented Active 9

9

12

10

10

10

Verified 3 6

3

8

8

Submitted

10 10

7

30

7

12 10

12 22 8

20

19

18

12

17 33

33

32

32

10 23

10

11 1-Mar-01

22-Feb-01

19-Feb-01

12-Feb-01

5-Feb-01

29-Jan-01

22-Jan-01

15-Jan-01

8-Jan-01

0

13

13

15

6

8

5

10 9

11

11

11

14

13 6

5

2

4

5 20-Apr-01

33

8

11

13-Apr-01

32

7

11

5-Apr-01

31

13 13

15

29-Mar-01

20

10

6-May-01

11

2

10

29-Apr-01

7

3

10

22-Mar-01

40

6

2

15-Mar-01

50

4

8-Mar-01

Total number of issues

60

14 9

Week ending

As Figure 4 shows, the month of November (the period of this effort) saw a measurable peak in productivity over and above the previous several months. This was an improvement of 67 percent, based on the next most productive five-week period, even though the total number of people-weeks missed due to holidays during the two most productive periods is relatively equivalent (Christmas in the later half of December and the Indian holiday Diwali at the end of October and early November). Looking at the trend line, we see continued improvement, even though the overall team size was reduced in March 2001 from 36 to 25. Figure 4 shows several things. First, the new bugs represent issues that the team must address. An eight-week running average trend line would show that this averages out to four issues a week. Second, the number of closed issues represents the number of issues that are resolved in a patch to the customer. This patching cycle represents our XP iteration and is on the order of two to three weeks (note the closure peaks). One of the greatest success stories is improvements in visibility. This is the greatest benefit to the team. Having a storyboard on which to daily prioritize tasks and discuss progress encourages best practices, lets people see what others are doing, and lets management gauge progress. Figure 5 shows the dramatic improvements in our workflow queues as a result of using the storyboard. When we installed the board in February 2001, it focused people’s attention on issues that went unverified for significant amounts of time. We also saw a dramatic increase in the number of issues that people started to

Figure 5. Workflow actively work on. Visibility alone was a queues.

strong motivational factor in this turnaround. In the future, we hope to scale this practice across multiple development sites.

H

ow can XP help us further improve? First, improving the pair programming initiative can improve our lack of cross-training among the code base’s many modules. It is not a practice for which this is intended but is a useful side benefit to a real problem as the team grows smaller. Engineers have accepted the benefits, but we are still in the process of structuring a more general approach to ensuring that the practice of pairing becomes part of the Iona culture. Earlier this year, we proposed having an individual sign up for each new bug, enhancement request, or refactoring effort, making that individual responsible for grabbing a partner and working through the issue. Our technical leads felt comfortable with this approach and they are now using it as a part of the standard process. Second, metrics are critical to the replanning game, but getting engineers to contribute is difficult. To plan how long it will take to complete a story, you must know how long it took to complete a similar piece of work based on the estimates of the engineer to whom the work is assigned. “As the basis for your planning, assume you’ll do as much this week as you did last week.”3 Kent Beck and Martin Fowler call this rule Yesterday’s Weather, and we think it is an appropriate analogy. We are currently running an internal project to improve our ability to enter metrics into our defect tracking system and report on them November/December 2001

IEEE SOFTWARE

49

ADVERTISER INDEX NOVEMBER

/

DECEMBER

2001

automatically. The volume of issues makes it difficult to track using a simple spreadsheet, but we do it anyway. Our Operations Lead speaks to each developer and notes how they have progressed on the work they are doing. The limited metrics we have indicate that it takes an engineer on average two weeks to fix a bug.3 This matches the suggested length of time to implement a story in the XP model. By measuring the calendar time with the defect tracking system, we can measure the calendar time it took to resolve an issue and ultimately the team’s velocity.3 This information is invaluable in planning what we will do in the next patch or release iteration.

Advertising Personnel

James A. Vick IEEE Staff Director, Advertising Businesses Phone: +1 212 419 7767 Fax: +1 212 419 7589 Email: [email protected] Marion Delaney IEEE Media, Advertising Director Phone: +1 212 419 7766 Fax: +1 212 419 7589 Email: [email protected] Marian Anderson Advertising Coordinator Phone: +1 714 821 8380 Fax: +1 714 821 4010 Email: [email protected] Sandy Brown IEEE Computer Society, Business Development Manager Phone: +1 714 821 8380 Fax: +1 714 821 4010 Email: [email protected] Debbie Sims Assistant Advertising Coordinator Phone: +1 714 821 8380 Fax: +1 714 821 4010 Email: [email protected] Atlanta, GA C. William Bentz III Email: [email protected] Gregory Maddock Email: [email protected] Sarah K. Huey Email: [email protected] Phone: +1 404 256 3800 Fax: +1 404 255 7942

San Francisco, CA Matt Lane Email: [email protected] Telina Martinez-Barrientos Email: [email protected] Phone: +1 408 879 6666 Fax: +1 408 879 6669 Chicago, IL (product) David Kovacs Email: [email protected] Phone: +1 847 705 6867 Fax: +1 847 705 6878 Chicago, IL (recruitment) Tom Wilcoxen Email: [email protected] Phone: +1 847 498 4520 Fax: +1 847 498 5911 New York, NY Dawn Becker Email: [email protected] Phone: +1 732 772 0160 Fax: +1 732 772 0161

Acknowledgments The authors thank Kent Beck for working with the Orbix Generation 3 team and the invaluable insights into the development process he provided that have motivated our teams to adopt XP. We also acknowledge the influence of ideas presented in Refactoring: Improving the Design of Existing Code (M. Fowler, Addison-Wesley, 1999) and Extreme Programming Installed (R. Jeffries, A. Anderson, and C. Hendrickson, Addison-Wesley, 2000). We presented an earlier version of this article at XP Universe in July 2001.

Boston, MA David Schissler Email: [email protected] Phone: +1 508 394 4026 Fax: +1 508 394 4926 Dallas, TX Royce House Email: [email protected] Phone: +1 713 668 1007 Fax: +1 713 668 1176 Japan German Tajiri Email: [email protected] Phone: +81 42 501 9551 Fax: +81 42 501 9552

References 1. K. Beck, Extreme Programming Explained: Embrace Change, Addison-Wesley, Reading, Mass., 1999. 2. G.A. Moore, Crossing the Chasm: Marketing and Selling High Tech Products to Mainstream Customers, Harper Collins, New York, 1999, p. 47. 3. K. Beck and M. Fowler, Planning Extreme Programming, Addison-Wesley, Reading, Mass., 2000.

Europe Glesni Evans Email: [email protected] Phone: +44 193 256 4999 Fax: +44 193 256 4998

Advertiser

Page Number

Boston University

15

California State University Northridge

6

Compaq

Cover 4

Eaton

12

John Wiley

For more information on this or any other computing topic, please visit our Digital Library at http://computer.org/publications/dlib.

Inside Front Cover

Monmouth University

16

ParaSoft

5

Requirements Engineering Conference 2002

1

About the Authors Charles Poole is a senior engineering manager at Iona Technologies. His research interests include Extreme Programming in large distributed development environments and selfassembling software paradigms. He received his BSE in aerospace engineering from the University of Michigan and his MSE in astronautical engineering from the Air Force Institute of Technology. Contact him at [email protected]. Jan Willem Huisman

IEEE Computer Society 10662 Los Vaqueros Circle Los Alamitos, California 90720-1314 Phone: +1 714 821 8380 Fax: +1 714 821 4010 http://computer.org [email protected]

is currently working with Iona Technologies in Dublin as an engineering manager responsible for the maintenance team for the Orbix Generation 3 product. He has extensive experience in software quality control and software development. He has also been involved in the entire software development cycle in the areas of programming, technical design, testing, project management, and maintenance. Contact him at [email protected].

IEEE SOFTWARE

November/December 2001

focus

reports from the field

Survival Patterns in Fast-Moving Software Organizations Lena Holmberg, Sydney Systems Lars Mathiassen, Aalborg University

oftware practices change. Many managers adopt software process improvement initiatives to increase their organizations’ ability to develop high-quality services and institutionalize state-of-the-art disciplines.1–3 At the same time, approaches such as open source4,5 and Extreme Programming6 introduce new and innovative ways to develop software and force most organizations to choose between improving present practices and supporting innovation.

S Fast-moving software organizations must respond quickly to changing technological options and market needs. They must also deliver highquality products and services at competitive prices. The authors describe how to deal effectively with such dilemmas and opportunities. 0740-7459/01/$10.00 © 2001 IEEE

This article reports our work with improvement initiatives in a fast-moving software organization called Linq. Since the company’s start in 1996, it has grown from five to 340 employees and undergone major changes in organization, technology, and strategy. Adapting improvement ideas was challenging because commitment and responsiveness to improvement fluctuated depending on the organization’s preoccupation with other challenges. The key to addressing this issue lies in the emerging cultures of such organizations. The culture is the result of the organization’s attempts to deal effectively with its environment;7 it is not explicitly created. Rather, it emerges through behavioral responses to challenges and problems. We can express such behaviors as survival patterns.8 These patterns are activated in our daily work, and they help us make priorities, solve problems, and do things, but they can collide when

new work practices challenge traditions. From this context, we examine how to understand and facilitate improvement in dynamic software organizations while preserving their capacity for innovation. A fast-moving software company Linq sprang from the idea of using collaborative software to support workflow and projects in knowledge companies.9 Although the company changed from consulting to software product development, the basic business idea remained the same—and it profoundly affected the way the company conducted SPI. Creation In January 1996, Michael Mandahl and Jan Morath founded Linq. They wanted to start a company that would help its customers make their employees contribute more value to the organization by working together. The company had a simple structure,10 with ManNovember/December 2001

IEEE SOFTWARE

51

This approach to improved project practices— based on the Linq business idea—was named LinQing.

dahl as CEO and the rest working on projects. Linq grew steadily, and although most employees came fresh from universities, experienced IT consultants also joined. A customer-specific solution turned into a product, although the major part of the business still focused on consulting. The founders soon discovered that although customer satisfaction was high, efficiency was too low. The organization changed from a simple structure into one composed of four teams headed by a team manager,10 but employees still conducted projects in an ad hoc fashion, and learning from experience was difficult. Professionalization During the summer of 1997, the founders realized that they needed help to accomplish process improvement, so they hired an SPI consultant. Top management committed to SPI, forming four task forces to improve project start, requirements management, testing, and customer management and appointing an SPI manager to coordinate the groups. Although all the task forces produced results, implementation was slow. To make the improvements more visible, management set a clear objective: Linq would perform at Software Capability Maturity Model (CMM) Level 2 by September 2000. An internal assessment in September 1998 started the initiative. The results, although devastating, encouraged new commitments. Four new task forces formed (after the initial four completed their missions): project method development, formal reviews, electronic project room, and training and diffusion. Carefully selected project members from all parts of the organization joined the groups to ensure a broad reach. This approach to improved project practices—based on the Linq business idea—was named LinQing. Linq internally developed the LinQing framework for cooperation in software projects. The purpose was to create a collaborative space for innovation and learning through joint use of simple control mechanisms. One of this article’s authors, Lars Mathiassen of Aalborg University, helped develop it. The process started with Steve McConnell’s Software Project Survival Guide.11 The task forces presented the resulting templates and instructions in Lotus Notes databases, and training started in spring 1999. From the start,

52

IEEE SOFTWARE

November/December 2001

two emphases characterized LinQing: ■ ■

collaboration and competence transfer between Linq and the customer, and using the customer and IT to support the project process.

Implementing LinQing was never mandatory under the SPI recommendations, but performing in accordance to CMM Level 2 was an objective. Unfortunately, the innovation-oriented founders did not always use LinQing in their own projects, which created a mixed message. In spring 1999, many employees expressed their frustration with the way projects were accomplished and demanded more structure, which resulted in further diffusion and finalizing of LinQing. From September 1997 to June 1999, Linq grew from 30 to 100 employees. The projects involved larger and more demanding customers, and the company reached a higher level of professionalism—with formal contracts, formal project plans, and systematic tracking and oversight. Transformation In spring 1999, a window of opportunity opened for Linq—namely, to produce LinqPortal, a corporate portal product based on Microsoft technology. In June 1999, the company reorganized, separating product development from consulting.10 Simultaneously, plans for a larger and faster European expansion emerged, and a search for investors began. The product, the CEO’s entrepreneurship, the tight upper-management team, and the company’s performance impressed investors. The investors also stressed the existence and practical use of LinQing as one of the organization’s key assets. The company grew from 100 employees to 340, and new offices opened in several countries. Although the consultants still worked with customers, the company focused more on designing and delivering a product for a perceived market need and on building a sales force. A major R&D project started in the summer: developing a mobile version of LinqPortal. When starting the new product division, the chief technology officer decided that all projects should use LinQing. The SPI manager formed and headed a formal SPI unit in the product division. The team’s five mem-

Table 1 Two complementary survival patterns Survival pattern

bers worked part-time in product development or as consultants to ensure diffusion of the results and development of the right relevance criteria. Because LinQing was designed for consulting, the SPI unit started working on special editions for product development. The team incorporated training into new-employee orientation, and Linq initiated a simple metrics program that emphasized the packaging of relevant LinQing features. The SPI unit produced product information sheets, put together physical folders in addition to the information presented on the intranet, and introduced a special strategy for corporate portal projects: instant deployment. The SPI initiative was thus quickly tailored to the organization’s specific needs—to support product development and sales of LinqPortal. In early spring 2000, the SPI unit was dissolved and diffused on the SPI manager’s initiative into the rest of the organization. It had delivered special editions of LinQing, and the organization needed to focus on applying them. Members of the former SPI group continued diffusion work by arranging training courses and presenting information at meetings. Epilogue The focus then changed to developing and selling a new product. Management considered producing customer satisfaction and delivering a product that could meet market demand to be vital. Major changes occurred in the organization. Sales separated from consulting, product management separated from product development, and the number of consultants decreased to reduce costs (which was necessary to attract new investors).10 The business and product divisions started deciding how to best use legacy practices to improve production. At this point, Linq had the infrastructure and competence needed to perform at CMM Level 2, but actually using it would require increased commitment throughout the organization. On 23 April 2001, Linq filed for liquidation in Sweden. Although LinqPortal received recognition as one of the best of its kind, the market had not evolved as predicted. The investors quickly decided not to go through with their long-term plans, and liquidation was the only alternative. The business was split into parts and sold to other companies. The

Level Operational (behavior) Tactical (organization) Strategic (environment)

Innovation Network Flexible Dynamic

Improvement Deliver Supportive Stable

founders and approximately 50 other employees now work in a new software company that focuses on LinqPortal’s mobilility. Survival patterns Two survival patterns drove Linq’s behavior and management priorities: innovation and improvement (see Table 1). Each one is characterized by the behavior of the employees, the organization’s requirements, and assumptions about the nature of the environment. Innovation The innovation pattern is strategy-driven. A fast-moving software organization’s environment is extremely dynamic: technology and market conditions change constantly, inviting or forcing the organization to adapt or change its behavior. Investing in infrastructures does not pay for the organization because they make it difficult to respond effectively to new environmental conditions. To facilitate learning, foster new ideas, and create the dynamics needed to respond quickly to new opportunities and demands, all members of the organization must interact with each other, customers, and external players with relevant knowledge and experience. In other words, to create innovations at a reasonable speed, networking is important. Throughout Linq’s rather short history, it underwent major changes as a result of responses to internal and external opportunities and challenges. The shift from Lotus Notes to Microsoft-based solutions was one such example of market-driven considerations. Similarly, moving from focusing on projects for specific customers to emphasizing internally developed products for corporate portal solutions was another major change. The company needed many major innovations to develop new management practices in response to its fast growth, gradually transform into an international rather than a national player, and successfully develop LinQing. The innovation culture emerged from the start, with the behavior of the two founders, and it flourished and continued to develop in response to a highly dynamic environment. November/December 2001

IEEE SOFTWARE

53

Table 2 The dynamics of Linq’s survival patterns Pattern

Innovation Improvement

Creation

Professionalization

Transformation

95% 5%

30% 70%

80% 20%

Improvement Software people want to do a good job—as professionals, they want to deliver high-quality solutions in response to customer or market needs. The organization must develop solutions that satisfy its customers and generate sufficient revenue—or it won’t survive. At the operational level, a mission to deliver satisfactory solutions drives this pattern. To achieve this, the organization must offer a supportive infrastructure that makes it easy (and possible) to reuse successes from one project to the next and a management tradition that encourages (rather than hinders) professional practices. To build such a supportive infrastructure, you must make certain assumptions about the types of projects, technologies, and solutions to support. In this way, we see certain parts of the environment as being stable. The founders imported an improvement culture in response to problems experienced with projects. It also had to change from a simple structure to one composed of teams and team managers, and an infrastructure developed to make better projects. This improvement initiative combined Linq’s collaboration and networking techniques with state-of-the-art ideas on SPI. Initiated by design, the improvement pattern grew to become an integral part of Linq’s culture. Dynamics The innovation and improvement patterns are complementary, but tensions easily arise between them. The innovation pattern generates a pull toward minimal and highly flexible infrastructures; the improvement pattern generates a contradictory pull toward supportive and more elaborate infrastructures. The innovation culture naturally dominates in the beginning with its ad hoc structures and mutual adjustment as key coordination mechanisms.10 As the software organization grows and matures, more elaborate structures develop and different forms of standardization occur to exploit past successes and increase management control.10 The defining property of quickly evolving software organizations is, however, their strategic drive to respond effectively to the opportunities and challenges gen54

IEEE SOFTWARE

November/December 2001

erated through their environments. We should therefore expect a constant struggle between the innovation and improvement cultures, with changing patterns of domination but the innovation paradigm having the upper hand. Although Linq experienced both patterns, their role and relationship changed (see Table 2). During the creation phase, innovation values nearly exclusively drove the behavior. The Linq concept was developed and implemented through intensive collaborations with customers, but little attention was paid to improvement values (beyond each individual project) and few resources were used to develop organizational infrastructures. Driven by the company’s experiences and pressure to improve, this picture changed dramatically as Linq moved into its professionalization phase. During this period, management initiated and heavily supported improvement activities, and most members of the organization took an active part in attempts to build supportive infrastructures. In response to new business opportunities, Linq entered the transformation phase to pursue corporate portal technologies and emphasize product development. Management heavily downsized the improvement efforts, new SPI processes were not developed, and the emphasis was solely on maintaining the current position. Lessons learned Each software organization has its own history and needs to make strategic decisions that fit its unique environment. Linq’s lessons are therefore not directly transferable to other software organizations. We have, however, learned certain lessons that might inspire other fast-moving software organizations in their ongoing struggle to cope with a dynamic environment while simultaneously trying to improve professional practices. Appreciate the survival game Everyone in a dynamic software organization must realize the reciprocal relationship between innovation and improvement. Both values and practices must be actively supported and cultivated to create a sustainable software business. Both need top management support in terms of resources and recognition, and the different talents and disciplines involved must constantly be developed and maintained.

About the Authors Protect the improvement culture Fast-moving software organizations are constantly on the move—not because they find this behavior particularly attractive, but simply because their raison d’être is to constantly adapt to an extremely turbulent environment. Recruiting resources to work with improvement and creating the necessary commitment toward improvement is therefore difficult. Innovation always receives more hype, and the urgency and energy involved in innovative activities easily become an excuse for giving low priority to improvements. When innovation dominates, protecting and maintaining the improvement culture is particularly important. Create innovative improvements To keep up with the organization’s innovation, the people working with SPI must be agile and creative. They must anticipate the possible next steps in technology, software development, and customer relations and constantly evaluate the consequences these might have for the organization. SPI activities should adapt to changing requirements, and the SPI organization should be minimal and adaptive. Key practices should be based on active networking in which software developers, managers, and customers participate actively in creating and implementing new improvements. Improve the ability to innovate Improvements must be conceived as relevant and useful in the software organization. A conventional approach to SPI that starts by addressing the six key process areas on CMM Level 2 will have little chance of creating the necessary commitment in dynamic software organizations. Classical key process areas should be considered, but they must be complemented with other ideas that focus on the needs and practices of an innovative software culture. Otherwise there is little chance of success with SPI. This is why Linq used LinQing as the framing device for SPI. LinQing unifies the basic business idea of supporting collaboration between professionals using modern information technology with state-of-the-art disciplines in software project management. Don’t specialize For the SPI organization, understanding the business is vital, so those involved in SPI

Lena Holmberg is the managing director of Sydney Systems, a family enterprise focusing on knowledge management. After her PhD in educational research at Göteborg University, she joined Linq. Over five years, she held various positions such as Chief Knowledge Officer, HR Director, and consultant, and was responsible for the Software Process Improvement initiative. Contact her at Sydney Systems, N. Skattegård, Upplid 330 17 Rydaholm, Sweden; lena_ [email protected].

Lars Mathiassen is a professor of computer science at Aalborg University, Denmark.

His research is in software engineering and information systems, most of it based on close collaboration with industry. He has coauthored many books including Computers in Context (Blackwell, 1993), Object Oriented Analysis and Design (Marko Publishing, 2000), and Improving Software Organizations: From Principles to Practice (Addison-Wesley, 2001). Contact him at the Dept. of Computer Science, Aalborg Univ., Fredrik Bajers Vej 7E, 9220 Aalborg Øst, Denmark; [email protected]; www.cs.auc.dk/~larsm.

must actively take part in the core processes. SPI people should develop double careers: one in SPI and one in software development or management. In that way, they build a good sense of what it takes to be fast moving. Management is well advised to make participation in SPI activities an important career step and to avoid having a small group of professionals specialize in SPI.

T

hese basic lessons can help dynamic software organizations face their basic paradox. SPI is particularly important in such organizations—otherwise, they have little chance of surviving. At the same time, however, fast-moving organizations are the most difficult ones to improve in a sustainable way. References 1. B. Fitzgerald and T. O’Kane, “A Longitudinal Study of Software Process Improvement,” IEEE Software, vol. 16, no. 3, May/June 1999, pp. 37–45. 2. K. Wiegers, “Software Process Improvement in Web Time,” IEEE Software, vol. 16, no. 4, July/Aug. 1999, pp. 78–86. 3. K. Kautz, “Making Sense of Measurement for Small Organizations,” IEEE Software, vol. 16, no. 2, Mar./Apr. 1999, pp. 14–20. 4. E.S. Raymond, The Cathedral & the Bazaar, O’Reilly, Sebastopol, Calif., 1999. 5. J. Ljungberg, “Open Source Movements as a Model for Organizing,” European J. Information Systems, no. 9, 2000, pp. 208–216. 6. K. Beck, Extreme Programming Explained: Embrace Change, Addison-Wesley, Reading, Mass., 1999. 7. E.K. Schein, Organizational Culture and Leadership: A Dynamic View, Jossey-Bass, San Francisco, 1985. 8. G.M. Weinberg, Becoming a Technical Leader: An Organic Problem-Solving Approach, Dorset House, New York, 1986. 9. T.H. Davenport and L. Prusak, Working Knowledge: How Organizations Manage What They Know, Harvard Business School Press, Boston, 1998. 10. H. Mintzberg, Structure in Fives: Designing Effective Organizations, Prentice-Hall, Upper Saddle River, N.J., 1983. 11. S. McConnell, Software Project Survival Guide, Microsoft Press, Redmond, Wash., 1998.

November/December 2001

IEEE SOFTWARE

55

focus

reports from the field

Accelerating Learning from Experience: Avoiding Defects Faster Lutz Prechelt, Universität Karlsruhe

ll programmers learn from experience. A few are rather fast at it and learn to avoid repeating mistakes after once or twice. Others are slower and repeat mistakes hundreds of times. Most programmers’ behavior falls somewhere in between: They reliably learn from their mistakes, but the process is slow and tedious. The probability of making a structurally similar mistake again decreases slightly during each of some dozen repetitions. Because of this a programmer often

A Defect logging and defect data analysis aims to decrease programmers’ repetitive errors. DLDA is inspired by the Personal Software Process but has a much lower learning cost. A controlled experiment validated the technique. 56

IEEE SOFTWARE

takes years to learn a certain rule—positive or negative—about his or her behavior. As a result, programmers might turn to the Personal Software Process to help decrease mistakes. We show how to accelerate this process of learning from mistakes for an individual programmer, no matter whether learning is currently fast, slow, or very slow, through Defect Logging and Defect data Analysis techniques. The Personal Software Process One successful approach to improving individual learning is the PSP, developed by Watts Humphrey.1 PSP provides effort estimation and time-planning techniques and a complete framework for process definition, process measurement and control, and continuous process improvement. IEEE Software published a report on the PSP course’s success.2

November/December 2001

PSP techniques for improving learning from mistakes are based on defect logs. Programmers record each defect they find during design, design inspection, coding, code inspection, testing, or operation. They also record the defect’s description, type classification and origin, the cause of the human error leading to the defect (if known), and the time required for locating and repairing it. When programmers collect enough defect data, they analyze the data for recurring mistakes; they then convert these insights into entries on inspection checklists and changes to the development process. However, learning the PSP methodology requires an immense effort. The standard form (aimed at undergraduate or graduate software engineering courses) requires 15 complete working days (spread over 15 weeks). Many project managers are inter0740-7459/01/$10.00 © 2001 IEEE

1999-04-13 1999-04-13 1999-04-13 1999-04-13 1999-04-13

13:09:37 13:12:23 13:22:24 14:39:34 14:46:35

bts be ee cd we ty # multiplied instead of adding be ee cd ma om # forgot to reset stdDeviation after first calc

be, “begin error”) was detected at 13:12 hours, three Figure 1. Two defect log entries. The first defect (b bts, “begin test”), and was located and removed (e ee, “end minutes after the start of the testing phase (b cd), is of the structural error”) 10 minutes later at 13:22. It had been introduced in the coding phase (c we), and the reason for making it was a typo (t ty). The second defect was of type “wrong expression” (w ma) and was produced owing to omission (o om—that is, in principle, the type “missing assignment” (m programmer knew that it had to be done). A typical DLDA describes defects between two to 10 phases, 10 to 50 defect types, and seven defect reasons (omission, ignorance, commission, typo, missing education, missing information, or external).

ested in PSP—until they hear about the learning effort. Most industrialists consider even a reduced PSP variant (such as in Humphrey’s Introduction to the Personal Software Process)3 far too expensive to learn in terms of time. Furthermore, the volume of bookkeeping the PSP proposes is so large, the data quality might become dubious.4 The problem with PSP is that programmers cannot learn it just by listening to a few presentations and then applying these techniques to their daily work. Under normal work conditions, few programmers can continuously follow all the cumbersome PSP techniques, especially before they experience their advantages. Programmers only experience the benefits if they practice the techniques for some time. That is why the course is required—for providing a pressure-free playground in which to learn about the effectiveness of PSP techniques. DLDA technique It is possible to learn and apply defect logging and defect data analysis (DLDA) in isolation, without attending a PSP course. The instruction time consists of only a half day, and the self-discipline requirements are moderate. DLDA techniques apply not only to coding but to most software process phases and activities. Our controlled experiment indicated that a programmer’s first DLDA application results in significant improvement. This suggests that it is possible to achieve benefits similar to PSP techniques without substantial investment in a PSP course. If a programmer commits a mistake, it is an error; a possible result of that error is a

defect in the software document. Once a defect is turned into code, its execution can result in failure of the software. Thus, errors (by human beings) and failures (by machines) are events, but defects are structural deficiencies. Much of the software process is concerned either with avoiding or detecting and removing such deficiencies. An explicit focus on avoidance and early detection is a characteristic of a mature process. DLDA aims to improve these activities, but it can be applied in otherwise immature software processes. As its name suggests, DLDA consists of two separate phases. The first, defect logging, is performed during all software construction activities where programmers might find defects in the product, such as during requirements definition, requirements review, design, design review, implementation, code review, testing, and maintenance. The second, defect data analysis, is a process improvement activity that programmers perform only rarely, when they have collected sufficient defect data (after several weeks). Defect logging First, programmers create a protocol entry whenever they detect a new defect in a software document. The entry starts with a time stamp recording. When the programmer localizes, understands, and repairs the defect, an additional time stamp and descriptive information completes the entry. The description can include the defect’s exact location, its type according to a fixed taxonomy, the phase when it was presumNovember/December 2001

IEEE SOFTWARE

57

We validated the DLDA technique in a controlled experiment.

ably created, a hypothesis as to why it was created (such as lack of information, other defects, trivial omission, or trivial mistake), and possibly a verbal description (see Figure 1). After programmers have practiced this technique, and when using a compact format, recording becomes simpler and faster if the programmer performs it with a tool that records the time stamps and does some simple consistency checking. Defect data analysis In the defect data analysis phase, programmers cluster the defects into related groups according to appropriate criteria. Programmers group defects by the categorizations used in the defect log. They then analyze these groups to pinpoint the most frequent and costliest mistakes to try to determine why they make those mistakes (root cause analysis). There is no prescribed method for doing this. The analysis is entirely data-driven and relates to whatever understanding programmers have of their own software process. Defect data tabulations, which are categorized by work phase, defect type, repair cost, and so on, are created automatically and are used to aid analysis. Unfortunately, defect logging requires quite a bit of discipline—most programmers cannot keep up with it along with their normal work. Because of this, coaching is the third important ingredient of DLDA. When programmers start applying DLDA, they need to be instructed how the technique works and should be motivated by at least one convincing example of an insight gained by DLDA—ideally an insight of someone they know well. During the first few days of learning the defect-logging technique, a coach must stop by from time to time, ask for the most recent defects, remind the programmer about logging, and discuss any how-to questions. Coaching is effective only with programmers who want to apply DLDA—there is no point in trying to force the technique on programmers who aren’t willing to give it a chance. Obviously, the coach should be somebody who not only knows how to do DLDA but also is convinced of its benefits. When programmers have collected enough defect data (about one hundred defects is a good start), their coach should counsel them during defect

58

IEEE SOFTWARE

November/December 2001

data analysis. The coach should explain how to summarize the data and help with the actual conclusions if the programmer does not find them. The experiment We validated the DLDA technique in a controlled experiment with participants divided into two groups, with one using DLDA. The programming tasks We prepared two algorithmic programming tasks. Task 1 consisted of computing the “skyline” for a set of “buildings,” which were represented as rectangles described by 2D coordinates. Task 2 consisted of computing the convex hull for a set of 2D points, described by their coordinates. In both cases, the algorithms that programmers used were given in coarse pseudocode (four statements for Task 1, seven statements for Task 2). Both tasks involve computations with 2D coordinates. The resulting programs had 80 to 206 lines of code for Task 1 and 93 to 160 lines of code for Task 2. Each experiment participant worked alone and solved both tasks, starting with Task 1. There were two groups: the experiment group, which used DLDA, and the control group, which did not. We asked both groups to record the time required for each work phase (design, coding, compilation, and test) and to track and subtract the interruption time. Participants finished Task 1 in 3.6 to 10.7 hours and Task 2 in 1.8 to 6 hours. We gave the experiment group a onepage defect-logging instruction sheet before they started Task 1. After reading it, we gauged their understanding of the method through a one-page scenario on which they filled in the defect log entries. The coach explained mistakes made in that test. The person who carried out the experiment also acted as the coach for all participants. We gave the participants a defect data record template, a two-page template for the defect data analysis, and one page of concrete instructions. In the experiment, we used a scaled-down DLDA variant because of time constraints. Besides a short description, programmers recorded the defect’s repair time, the inject and remove phase of the defect, and the error reason class (not the defect type class). Consequently, defect data analy-

Improvements from Task 1 to Task 2 in percent 100 X

50

X

X

X

X

X

X

X X X

0 -50

-150 De

f.d

De

en

f.d

sit

TD

y(L

en

)

sit

ef.

y(-

)

de

LO Tim Tim De De TD LO C(L f. f.la ef. C(e(de ) ) ten laten e(L) ) nsi nsi c c y y ty( ty( ( ( L ) ) -) L)

sis was simplified accordingly. However, because the tasks were small, we asked participants to log even simple defects such as syntactical defects and missing declarations. During the experiment, we observed the participants unobtrusively. If a participant failed to fill in the time log or filled out the defect log incorrectly, the coach reminded him. If necessary, the coach reiterated how to apply the method. The coach painstakingly watched participants to help them with DLDA techniques, but he did not offer help with the programming task. We accepted a finished program only when it passed a fixed set of tests. The participants The participants were 18 male graduate or senior undergraduate students of computer science, electrical engineering, and computer engineering from the University of Massachusetts at Dartmouth. We paired participants with similar experience levels, thus ensuring a reasonable balance between the groups. We randomly assigned members of each pair to the groups. That left 10 participants, with five in each group. Seven of these 10 had one or several years of experience as professional programmers. Several participants dropped out during the experiment; three were too inexperienced and could not solve the first task, and five were unwilling to invest the time for the second task after they finished the first.

We saved the participants’ source code of each program version compiled along with a time stamp. We used this data to determine the number of defects introduced into a program and the time required for eliminating each of them.

Figure 2. Performance changes of the defect-logging group (L) versus the nonlogging group (–) for several metrics: defect density, test defect density, defect latency (average defect lifetime), total work time, and program length. In each box plot, the whiskers indicate the value of the best and worst participant, the box indicates those of the second best and second worst, and the middle line those of the median.

Results With only five participants in each group, you might expect that obtaining meaningful results was impossible. But, as we discovered, this was not the case. Basic metrics From the data collected during the experiment, we computed these basic metrics for each participant’s task: ■ ■ ■ ■ ■

the total required time for the task, the resulting program size in lines of code (LOC), the number of compilations, the number of defects inserted in any program version, and the number of defects removed during testing.

From these, we then computed the defect density—that is, the number of defects inserted divided by the program’s size. The test defect density accordingly considers the defects removed during test only. The defect latency expresses a defect’s average lifetime from its insertion to its removal as visible in the compiled program versions. November/December 2001

IEEE SOFTWARE

59

Statistical Hypothesis Tests A statistical hypothesis test is a mathematical procedure for comparing two data samples (sets of related values). For instance, the Wilcoxon Rank Sum test compares the medians of two samples, the t-Test compares the means, and a Bootstrap-based means-differences test compares the means but does not require assuming a normal distribution. On the basis of the variation in the observed data, such a procedure will compute the probability (called the p-value) that the apparent differences between the two samples are not real but rather merely due to chance. In our case, if that probability is small, we are willing to believe that the observed additional improvements in the Defect Logging and Defect data Analysis group are not accidental and will consider the experiment good evidence of DLDA’s usefulness. Comparing the medians indicates whether we should expect an improvement for a single individual. In contrast, comparing the means indicates whether we should expect an improvement when averaging across a team (a group of individuals).

Because of our small group sizes, it would be dangerous for us to assume the exact equivalence of the two groups and to compare their performance on one task directly. Because the two tasks are obviously different, it is also not meaningful to compare Task 1 to Task 2 within one group. So, the most sensible approach to evaluating our data is to compare one group’s performance changes from Task 1 to Task 2 to that of the other group. Therefore we computed the improvements (in percent) for the above metrics by comparing Task 1 results to Task 2 results for each group. Figure 2 shows the most interesting results. Each box and the median line indicate improvements of the middle three subjects of one group for one metric. The whiskers indicate the other two subjects. With at most one exception per group, there was always an improvement from the first task to the second. Only Subject 1 (from the logging group) and Subject 8 (from the control group) performed worse in Task 2. Improvement analysis The improvement differences between the experiment group and the control group are not huge, but nevertheless are substantial. Are they real or accidental? To answer this question, we used a statistical hypothesis test (see the sidebar). A one-sided Wilcoxon Rank Sum test informs us that the reductions in median de60

IEEE SOFTWARE

November/December 2001

fect density and test defect density each have a probability of only 11 percent of being accidental—reasonable evidence that DLDA improves defect prevention. This is corroborated by the corresponding comparison of the mean reductions using a bootstrap-based test, which indicates a 9 and 10 percent probability for defect density and test defect density, respectively, that the observed differences are random, nonsystematic events. For the defect latency, the same tests indicate a 35 and 54 percent chance, respectively, that there is no real improvement, which suggests that DLDA did not cause faster defect removal, at least in this experiment. There is a 5 percent chance that the median work time reduction is accidental (3 percent for the mean). With a probability of 0.8, the improvement percentage is larger by at least 16 in the experiment group. We should be aware that part of this improvement is because the defect logging is getting faster itself—in the first task, the DLDA participants were undoubtedly slowed by their unfamiliarity with the technique. However, on average, they logged only 25 defects in six hours, which is a minor effort in any case. The difference in program length is quite interesting. During the defect data analysis, several participants recognized that they should have spent more effort on properly designing their program. The more careful design in Task 2 then resulted in more compact programs. There was a probability of only 5 percent that the median difference is accidental (2 percent for the mean). We determined that despite the small group sizes, we can be reasonably sure of the evidence found in the experiment. DLDA results in better defect prevention and thus increases productivity. Some skeptics might feel these results do not transfer to professional programmers. It is plausible that our inexperienced participants had so much more room for improvement than an experienced software engineer that DLDA might be worthless in practice, in spite of our results. However, we found evidence to the contrary in our data. For the test defect density, our data shows a clear trend that the more experienced participants actually obtained a larger improvement than the others. This is true no matter

whether we measure experience in years of professional programming experience or by the length of the largest program the participants ever wrote.

specific trainee? Should DLDA be combined with pair programming? Answering these questions will lead to faster learning in software programming.

D

Acknowledgments

LDA appears to be a viable technique for accelerating learning from experience: programmers learn to prevent mistakes faster than usual. For most programmers, learning DLDA will require an experienced coach, but apart from that, the learning cost is low. In a controlled experiment we found that a group using DLDA for only one small programming task (a half day) solved a second task faster and with smaller defect density than a control group—even though the experiment used only a simplified version of DLDA owing to time constraints. We believe in DLDA’s validity; we urge practitioners to try it with a few colleagues. It is a low-cost, low-risk technique with considerable potential benefits; you can usually find a champion to be the coach. Our research raises a number of questions. Our experience with teaching PSP indicates that more than half of all programmers are unable to keep up the self-discipline required for defect logging. This seems to be a personality issue. How can we quickly and safely determine whether training a programmer in DLDA will lead to actual usage? Merely asking the candidate solves only about half of the problem. What tools provide the best support for defect logging? The Web page www.ipd.ira. uka.de/PSP provides tools for defect logging and defect data summarization, as well as a defect classification standard. What defect classification categories are most useful, and in which contexts? What criteria are best for defect analysis? How far can defect analysis be standardized and simplified before it loses value? Our experiment indicates that for most (but not all) people, more value comes directly from the logging rather than from the analysis. However, a useful defect classification scheme might play an important role even then. What minimum intervention by the coach is sufficient? How can the coach best adapt the interventions to the needs of the

Georg Grütter performed this experiment. We thank Michael Philippsen for commenting on a draft of this article and our experimental subjects for their participation.

References 1. W.S. Humphrey, A Discipline for Software Engineering, Addison-Wesley, Reading, Mass., 1995. 2. W.S. Humphrey, “Using a Defined and Measured Personal Software Process,” IEEE Software, vol. 13, no. 1, May 1996, pp. 77–88. 3. W.S. Humphrey, Introduction to the Personal Software Process, Addison-Wesley, Reading, Mass., 1997. 4. P.M. Johnson and A.M. Disney, “The Personal Software Process: A Cautionary Case Study,” IEEE Software, vol. 15, no. 6, Nov./Dec. 1998, pp. 85–88.

For more information on this or any other computing topic, please visit our Digital Library at http://computer.org/publications/dlib.

About the Author Lutz Prechelt is the head of process man-

agement, training, and technical documentation at abaXX Technology, Stuttgart. He previously was a senior researcher at the School of Informatics at the University of Karlsruhe, Stuttgart. His research interests include software engineering (particularly using an empirical research approach), compiler construction for parallel machines, measurement and benchmarking, and research methodology. He has performed several controlled experiments on design patterns, inheritance hierarchies, and the Personal Software Process. He received his diploma and his PhD in informatics from the University of Karlsruhe. He is a member of the IEEE Computer Society, the ACM, and the German Informatics Society, and is the editor of the Forum for Negative Results, part of the Journal of Universal Computer Science. Contact him at [email protected].

Abstract—Defect

November/December 2001

IEEE SOFTWARE

61

requirements Editor: Suzanne Robertson

■

The Atlantic Systems Guild

■

[email protected]

The Top Risks of Requirements Engineering Brian Lawrence, Karl Wiegers, and Christof Ebert

T

he worst thing that can happen in requirements engineering is that your set of requirements, however expressed, doesn’t accurately represent your users’ needs and consequently leads your team down the wrong development path. The whole point of requirements engineering is to steer your development toward producing the right software. If you don’t get the requirements right, how well you execute the rest of the project doesn’t matter because it will fail. So how are we led astray? The risk is greatest at several points. Overlooking a crucial requirement Perhaps the greatest risk in RE is missing a critical functional or attribute requirement. If you overlook an important user class, you’ll probably have to do a big job—usually larger than stakeholders care to tolerate—to add in what that user needs. Missing a critical quality or performance attribute is typically even worse. Often, the only way adapt a software-based system to accommodate an important attribute is to re-architect. An example that many software developers have encountered recently is scalability in ecommerce. If designers don’t keep scalability in mind when choosing their architectures (and many don’t), they find themselves in a tough position when the usage load on their software leaps to a thousand or more times what they were expecting. System performance is inextricably tied to system architecture. In most instances, the only way to improve performance is to choose another one—that is, to start over from scratch. This is not popular when your senior managers 62

IEEE SOFTWARE

November/December 2001

promised you’d deliver next week. It’s even worse if you make the discovery when your software is already in the field. Inadequate customer representation One of the central activities in RE is negotiating agreement on requirements. To achieve this agreement, you must find out what your customers really need. Not much of a negotiation will take place if you never actually interact with them. “Take it or leave it” happens all too frequently when we assume our design ideas suit our customers and don’t bother to check if this assumption is really true. Customers only discover if we had the right idea when they attempt to use our software. If our confidence was misplaced, that’s a late time to discover it. For example, a corporate IT development team told Karl that they recently rolled out a new application for internal use, but that they developed the system with virtually no customer input. The first time the users saw it was on delivery, and they immediately rejected the system as completely unacceptable. On the day you proudly unveil your new baby to the world, you don’t want to hear, “Your baby is ugly!” Modeling only functional requirements Both the requirements literature and our practices have historically focused on functional requirements—the things our software systems are supposed to do. Functional requirements are the most obvious ones to the user, so most elicitation discussions focus on them. Perhaps more important, though, is 0740-7459/01/$10.00 © 2001 IEEE

REQUIREMENTS

gaining agreement on quality attribute requirements—the characteristics you intend your software to exhibit. Old standbys include reliability, performance, security, robustness, ease of use; others are scalability, innovation, coolness, or fun. Functional models, such as use cases, frequently gloss over the attribute requirements altogether. The attribute requirements are the heart and soul of why your customers will value your software. They determine why using your software is better than whatever they did before to achieve the same end. For example, a system that fails to handle exceptions effectively will not be robust and will crash when unexpected conditions occur. It does you no good to simply record the requirement that “the system shall be robust.” A skillful requirements analyst knows to ask the prompting questions that will elicit the user’s implicit expectations about various attributes, explore what the user has in mind when he or she says “robust,” and negotiate the inevitable tradeoffs among conflicting attributes. Not inspecting requirements The evidence is overwhelming and long known that the cost to remove defects in requirements increases geometrically with time. Once your software hits the field, removing a requirements defect costs at least a hundred times as much, assuming you can fix it at all. Inspecting your requirements models is the most effective way to identify ambiguities, unstated assumptions, conflicting requirements, and other defects at the earliest possible point. Of course, to hold an inspection, you must have something inspectable. And you have to believe that your set of requirements has defects that you need to identify. Personally, in all the years I’ve (Brian) seen requirements for countless software projects, I’ve never seen a defect-free set. If there’s an ironclad rule in software development, it’s “Always inspect your requirements.” And choose inspection teams with a broad constituency, including testers. One company we

know measured a 10-to-1 return on investment from performing inspections on requirements specifications, sustained over five years. For more on inspecting requirements, check out Karl Wiegers’ article “Inspecting Requirements” at StickyMinds.com. Attempting to perfect requirements before beginning construction The time when we could know everything we needed to know before starting software construction is long past. Today, we live in an emergent world—some information simply isn’t available early in our projects—and only emerges later. We can’t possibly know everything we’d like to know before we start development. It’s safer to assume that our requirements are going to change than that they won’t. Yet for some projects, participants feel as though they must completely understand the requirements before any other work begins. For most projects, this is a mistake. You need to do some design and construction before you can tell how hard the job will be and how much each part will cost. As this kind of information becomes evident, it could well affect your views about your requirements, further changing them. Do the best job you can early to get a good set of requirements, but don’t be discouraged if everything isn’t absolutely certain. Identify those areas of uncertainty and move on, ensuring that someone is responsible for closing those gaps in your knowledge before construction is complete. Track the uncertain requirements carefully as your project proceeds. Representing requirements in the form of designs Possibly the subtlest risk in requirements engineering is letting designs creep into, and then remain in, your requirements specifications. When you view designs as requirements, several things happen. You run the risk of choosing a particular solution that might not be the best one to implement. Also, you undermine your ability to validate your system, because you are specifying

the problem you hope to solve in terms of how you intend to solve it. All you can really do is verify that you built what you said you would. One surefire indicator that you’re falling into this trap is highlighted by references to technology. Any time a technology is specified, you’re using a design to represent the underlying requirement. Using designs as requirements is a subtle risk because although you might have specific information about what you want, it doesn’t represent the underlying need and is consequently vulnerable to mistaken assumptions.

W

hereas these risks are pretty serious, the greatest threat to project success is not performing requirements engineering at all. Requirements are your project’s foundation. They define the level of quality you need, facilitate decision making, provide a basis for tracking progress, and serve as the basis for testing. If you let them remain unstated, you have no opportunity to examine and negotiate them with your customer and no way to tell when your project has met its objectives. Without clear requirements, how will you know when you’re ready to release your product?

Brian Lawrence is a principal at Coyote Valley Soft-

ware, a software consulting firm in Silicon Valley, California. He helps software organizations model and manage requirements, plan projects, and conduct peer reviews. Contact him at brian@ coyotevalley.com. Karl Wiegers is the principal consultant at Process Impact, a software process education and consulting company in Portland, Oregon. He has written books about peer review in software and software requirements. Contact him at kwiegers@ acm.org. Christof Ebert is director of software coordination and

process improvement at Alcatel in Paris. He is also the IEEE Software associate editor for requirements. Contact him at christof. [email protected]. November/December 2001

IEEE SOFTWARE

63

quality time E d i t o r : J e f f r e y Vo a s

■

Cigital

■

[email protected]

Software Architecture Correctness K. Suzanne Barber, University of Texas at Austin Jim Holt, Motorola

S

oftware quality stems from many factors, including implementation decisions, software architecture, and requirements. In this column, we focus on software architecture, which can enable or inhibit many of a software system’s qualities. Because the cost of addressing quality concerns is a function of how late you address them (the later, the

more costly), addressing them in the architectural or requirements phase makes sense. Architectural quality Researchers have begun to recognize that we can view software architecture as a formal model for requirements specification. This recognition expands the possibilities for evaluating quality attributes early during development. In this context, when evaluating architecture properties, our goals include ■ ■

64

IEEE SOFTWARE

providing an early opportunity to correct requirements defects and ensuring that the software architecture

November/December 2001

provides an accurate blueprint for system implementers. In the first case, you could use a software architecture evaluation to check qualities such as safety, liveness, and completeness. In the second case, you might predict qualities including maintainability, performance, and reliability. First things first You must establish an architecture’s correctness before using it as a system blueprint. An incorrect architecture will not produce quality evaluations of merit. So, the blueprint must be correct before you can build a system targeting quality goals. Correctness evaluations also mitigate the effort and cost of performing simulations or other evaluations to predict qualities such as performance and reliability. Many techniques evaluate correctness properties of software architectures; however, no existing single technique can evaluate all correctness aspects. For example, model checking works well to evaluate safety and liveness but is not useful for evaluating completeness. Conversely, simulation helps detect completeness errors, but you can’t use it to prove the absence of safety or liveness errors. Automated translation of a software architecture specification into a form suitable for a model checker or simulation tool removes the necessity of having the software architect be an expert in model checking or simulation techniques. For the 0740-7459/01/$10.00 © 2001 IEEE

QUALITY TIME

same reason, it’s desirable to automatically collect and present the results of model checking or simulation. Automation is reasonably straightforward, provided the specification contains declarative elements, integration details, and behavioral information. Declarative elements typically take the form of sets of services collected into class interfaces, integration details define what data and events are exchanged between interfaces, and behavioral information outlines pre- and postconditions for state changes. You can use this set of information to translate the specification into a set of communicating processes defined in the model checker or simulation tool semantics. Architectural completeness A model checker can unambiguously detect safety and liveness errors. Completeness errors, however, are associated with unexpected system behavior, so a software architecture could exhibit an absence of safety and liveness errors but still lack a certain required functionality (a form of completeness error). Completeness errors typically manifest in sequences of service executions (scenarios) that ■ ■ ■ ■

are missing expected service executions, contain unexpected service executions, contain unexpected paths, or are missing paths.

At times, completeness errors have gone undetected during requirements elicitation and have propagated into the software architecture. Such omissions or inaccuracies in the requirements are not unlikely, given the challenges of the knowledge acquisition and modeling process used to gather requirements. In fact, the knowledge contained in the requirements is a function of many variables, including the ■

spectrum of expertise held by domain experts,

■ ■ ■ ■

time spent with each expert, knowledge acquisition approach, ability of knowledge engineers and experts to conceptualize, and degree to which experts can express knowledge and offer necessary detail.

So how does the software architect or domain expert use simulation results to evaluate the architecture’s completeness? Simulations can produce visualizations of the architecture’s execution. These visualizations are particularly useful for identifying completeness errors by inspection— you can visually spot errors in threads of execution the simulation captures. Unfortunately, simulations have two limitations: they do not provide an exhaustive means of evaluating an architecture model, and they require a human to interpret their output. So, simulations are best for showing the presence of completeness errors, but cannot prove their absence. A scenario space can help architects and domain experts visualize completeness errors. A scenario space is a directed graph that represents possible threads of execution composed of services in the software architecture. A vertex in the graph represents a service execution state, and an edge represents a path showing reachability from one service execution state to another. This visualization technique provides a high-level view of software

Simulations can produce visualizations of the architecture’s execution. These visualizations are particularly useful for identifying completeness errors by inspection.

architecture execution through partial-order reduction of simulation data to merge similar threads of execution produced over many simulation runs. Creating such a visualization can help evaluate whether executing the architecture will result in threads of execution that support the anticipated scenarios for the application domain. A human approach Regardless of method, the state of the art in evaluating correctness qualities of software architectures still requires human involvement. This invariably means human experts familiar with the application domain inspecting an artifact. Inspecting requirements specifications or static software architecture diagrams of large systems is daunting. However, cost savings provides clear motivation to perform these evaluations early. A formal software architecture representation allows for accurate, repeatable model checking and simulation for correctness evaluations, thus mitigating human effort and potential inaccuracies.

K. Suzanne Barber is an associate professor in the

Electrical and Computer Engineering Department at the University of Texas at Austin. She is also the director of the Laboratory for Intelligent Processes and Systems, where ongoing research projects address distributed, autonomous agent-based systems and formal software engineering methods and tools. Contact her at [email protected]. Jim Holt is a senior verification engineer for the Motorola Semiconductor Products Sector and a PhD candidate in the Department of Electrical and Computer Engineering at the University of Texas at Austin under the supervision of K. Suzanne Barber. His work at Motorola involves creation of advanced tools for automated code generation and verification of Motorola microprocessors and systems-on-a-chip. Contact him at jim.holt@ motorola.com. November/December 2001

IEEE SOFTWARE

65

manager Editor: Donald J. Reifer

■

Reifer Consultants

■

[email protected]

Knowledge Management: Insights from the Trenches Linda Larson Kemp, Kenneth E. Nidiffer, Louis C. Rose, Robert Small, and Michael Stankosky

T

he mandate came from our member companies: “Help us manage knowledge better.” So, in 1999, the Software Productivity Consortium—a not-forprofit alliance of industry, government, and academia—asked our members to indicate which technological advances they need most urgently. Most respondents stressed the need to better leverage an increasingly vast and complex array of intellectual assets. Such assets represent today’s new capital, marking a profound shift from more traditional types of capital. To address this urgent need, the Consortium (see www.software.org) launched a knowledge management program to develop our competency in KM so that we can better serve our members and to provide products and services that will help members develop their own KM competencies. We focused first on making access to Consortium assets easier through an enterprise portal. Then, to address the larger KM issues, we also partnered with George Washington University and its new Institute for Knowledge Management (http://km.gwu.edu/km/index.cfm), which seeks to establish a sound theoretical foundation for KM. Here, we recap the lessons we have learned in pursuing our KM mandate and set forth what we believe are the keys to KM’s future success. The promise and challenge of knowledge management Knowledge is to the Information Age as oil was to the Industrial Age. Today, the main asset of production is intellectual capi66

IEEE SOFTWARE

November/December 2001

tal as opposed to the tangible assets that previously drove manufacturing-based markets. “In the [new] age, a company’s value is largely determined by its ability to convert…knowledge into net worth,” according to a briefing by the National Knowledge & Intellectual Property Management Taskforce. To convert intellectual capital into market value, however, an organization must foster knowledge creation, utilization, and sharing. KM is about leveraging relevant intellectual capital to achieve organizational missions. While this premise might appear selfevident, successful outcomes are elusive. What is critical is how we leverage knowledge to measurably enhance effectiveness, efficiency, and innovation. Knowledge management at the Consortium Like most organizations, the Consortium faces daunting change. Member diversification has grown dramatically over the past four years, increasing demand for a wider range of products and services. Providing innovative support for information access and member collaboration has become an increasingly important aspect of our value proposition. KM offers a partial solution to these challenges. Given this situation, and in response to our board of directors, we held a workshop for 33 member representatives. Attendees exchanged ideas, declared needs, and came to consensus: to help them better manage knowledge, our KM program should focus first on technology transfer. To improve technology transfer, we could, for example, 0740-7459/01/$10.00 © 2001 IEEE

MANAGER ■

■

■

■

■

update our business model, putting KM at the core; reengineer our productservice lines, emphasizing intellectual capital; hold frequent member forums to boost outreach; invest in distance-learning capabilities to share more information more quickly to more people; or implement a portal with such features as intelligent search and support for communities of practice.

We quickly concluded that a portal was the key to our short-term investment strategy. We selected among candidate features and tools based on need, cost, and feasibility. Last fall, we piloted a portal prototype internally. Its functionality includes concept search, content spidering, and topic maps. In the fall of 2001, we are rolling out the portal to members as part of the Consortium’s Web site update. The update will include features such as concept-based searching using advanced search mechanisms, a tailorable user interface for a more effective engagement, and capabilities to facilitate communities of practice. Now we look forward to the next iteration of the portal’s evolving functionality, support for communities of practice. Lessons learned During nearly two years of effort, we have learned many things. For example, as with any other major institutional change initiative, several familiar factors proved critical to our KM program’s success: ■

Clear goals. Investment in KM must have specific ends defined in measurable terms and be aligned with business aims. Knowledge needs should be prioritized, and policies, systems, and infrastruc-

tute the essential competencies in KM. For example, we discovered that few, including ourselves, know enough about how to convert knowledge into net worth. Such competencies are not yet well developed within the theory and practice of KM. The KM community at large needs to address these essentials immediately.

■

■

■

■

tures should be designed to satisfy them. Strong sponsorship. A KM program requires constant executive commitment and support. Everyone should grasp KM’s importance to the organization and see their role in it. Realistic expectations. KM is no silver bullet. The complex challenge to identify and leverage relevant intellectual capital requires more than simplistic solutions. Balance. “Too much emphasis on technology without incorporating the other critical elements (i.e., leadership, organizational structure, and learning) can easily result in a failed system.”1 Iterative approach. Despite potential impatience with incremental gains, prototyping is important to clarify requirements, test capabilities, and gain buy-in. Moreover, continual system evolution is essential to keep pace with a dynamic business environment.

We also have learned about another set of critical factors, those that consti-

■

■

■

■

■ A systems approach. Typically, KM programs focus on narrow solutions (such as technology or learning). The roots of an organization’s overall mission receive little attention. KM must address the organization as a whole; we need to use system techniques tied to business goals. ■ A flexible framework. KM must be custom fit to each organization. While certain fundamental principles should always apply, no unified theory exists to explain these principles and enable diverse organizations to craft adept solutions to individual situations. An evolutionary process. Currently, no accepted process model supports the continual evolution of KM capabilities into the organization. We need to define and test a process model to meet these needs. Integrated measurement. We need guidance on how to identify those measures that are most vital for leveraging intellectual capital, which is the primary objective of any KM program. A capability model. Many KM approaches and tools exist, many claims are made, but what constitutes the essential core set of capabilities that truly represent KM proficiency? No industry-accepted model exists to assess the maturity of an organization’s KM program. Technical maturity. KM tool features need to converge; for exam-

November/December 2001

IEEE SOFTWARE

67

MANAGER

ple, support for moderated chat rooms and single sign-on is difficult to find in a set of compatible tools. Tool quality also must improve; for example, browser compatibility and support for integrated third-party technologies can be problematic.

How to Reach Us Writers For detailed information on submitting articles, write for our Editorial Guidelines (software@ computer.org) or access http://computer.org/ software/author.htm. Letters to the Editor Send letters to Letters Editor, IEEE Software 10662 Los Vaqueros Circle Los Alamitos, CA 90720 [email protected] Please provide an email address or daytime phone number with your letter. On the Web Access http://computer.org/software for information about IEEE Software. Subscribe Visit http://computer.org/subscribe. Subscription Change of Address Send change-of-address requests for magazine subscriptions to [email protected]. Be sure to specify IEEE Software. Membership Change of Address Send change-of-address requests for IEEE and Computer Society membership to [email protected]. Missing or Damaged Copies If you are missing an issue or you received a damaged copy, contact [email protected]. Reprints of Articles For price information or to order reprints, send email to [email protected] or fax +1 714 821 4010. Reprint Permission To obtain permission to reprint an article, contact William Hagen, IEEE Copyrights and Trademarks Manager, at [email protected].

68

IEEE SOFTWARE

November/ December 2001

In addition, there is growing frustration within industry and government over the KM community’s fragmentation, along with growing awareness of the need for a common, holistic approach to leveraging intellectual capital. A sound framework, containing all of the essential elements of a successful KM program and based on a strong theoretical foundation, would improve consistency within the KM community and advance the practice of KM.

T

o develop KM’s theoretical foundation and provide consistent practical guidance, the Institute and the Consortium recently partnered to develop and deploy the required interdisciplinary theory, for which they are well suited. The Institute contributes expertise across many fields; the Consortium is already in the business of transforming theory into practice. Together, we will review the KM landscape and facilitate industry consensus on the essential elements. We will scan the breadth of disciplines contributing to KM—from cultural anthropology to Web technologies— trying to consolidate the best and latest thinking in each area. From this baseline, we plan to develop a unified framework for KM with associated definitions, criteria, and practices. We want to enable organizations to better envision and deal with the realities of the new economy and follow through with the hard choices involved in adapting to change and aligning resources.

The proof will be in the pudding. The framework will need to be validated, tested, and evolved. You can help! We welcome your comments, inquiries, and participation.

Acknowledgments The authors thank the Consortium’s Information Support Group for their support of the portal, and Susan Polen and Deanna West for their immense dedication, creativity, and patience in rolling it out.

Reference 1. M. Stankosky and C. Baldanza, “A Systems Approach to Engineering a Knowledge Management System,” Knowledge Management: A Catalyst for Electronic Government, R.C. Barquin, A. Bennet, and S.G. Remez, eds., Management Concepts, Vienna, Va, 2001.

Linda Larson Kemp is a senior member of the Con-

sortium’s technical staff. Contact her at 2214 Rock Hill Rd., Herndon, VA 20170-4227; [email protected]; www.software.org.

Kenneth E. Nidiffer is vice president of new business at the Consortium. Contact him at 2214 Rock Hill Rd., Herndon, VA 20170-4227; [email protected]; www.software.org.

Louis C. Rose is manager of strategic initiatives at the Consortium. Contact him at 2214 Rock Hill Rd., Herndon, VA 20170-4227; [email protected]; www.software.org.

Robert Small is a principal member of the Consortium’s technical staff and former co-chairman of the Consortium’s Technical Advisory Board. Contact him at 2214 Rock Hill Rd., Herndon, VA 20170-4227; [email protected]; www.software.org.

Michael Stankosky is an associate professor of engineering management and systems engineering at George Washington University, founding director of the Institute for Knowledge Management, and creator of GWU’s KM program. Contact him at Melvin Gelman Library, 2130 H St. NW Suite 704, Washington, DC 20052; [email protected]; http://km.gwu.edu/km/ stankosky.cfm.

feature interface design

Web User Interface Design: Forgotten Lessons Uttara Nerurkar, Infosys Technologies

number of prescriptions are in vogue for Web user interface deThe methods used to design Web sites and graphical user interfaces differ

A

sign (let’s call it WUI, in line with GUI). However, Web site usability continues to be a serious issue.1 WUI design methods are quite different from their GUI precursors. The latter have matured considerably over some time and have been proven to yield highly usable interfaces.2 While the Web paradigm is apparently quite different from the GUI one, are there similarities that we can exploit to improve WUI

design techniques? I try to show here how I believe this can be achieved. Let us take a quick overview of one representative methodology each for WUI and GUI design.

The methods used to design Web sites and graphical user interfaces differ considerably, but the author argues that it is possible to improve Web design methods by applying well-established GUI design principles. 0740-7459/01/$10.00 © 2001 IEEE

WUI design methodology A popular Web design technique3 is based on building an information architecture, following these basic steps: 1. Define the site’s mission and vision. Also, envision your intended audience. 2. Determine the site’s content and functionality. 3. Define the organization of information on the site, including navigation, labeling, and search systems. Specify these in terms of Web page hierarchy diagrams, or information architecture blueprints. 4. Build and test. Preferably, test the site’s usability by observing members of your intended audience perform specific tasks on the site. Analyzing the Web design technique a bit more closely, we can see a few gaps in the scheme. At the second step, the link between

audience and design breaks. While the design is guided by the expected audience’s characteristics, there is no direct way of defining user requirements, especially in terms of content. This invariably leads to an almost imperceptible shift in the focus from what the user would like to find on the site to what the owner would like to display. Similarly, at the third step, the organization of content has no explicit reference to user requirements or expectations. It usually involves a logical partitioning of content and a drill-down architecture for details. For example, on an entertainment site, music and movies typically appear on separate pages because they are different types of items. However, would it be better if the movie’s soundtrack appeared on the movie page? We need to understand the user to answer this question. Indeed, an expert site designer recognizes this and often flouts the “logical partitioning” rule. But this leaves us to grapple with the question, “If not logical, then what?” GUI design gives us some pointers, as I shall reveal later. At the fourth step, the user focus explicitly returns to the design cycle—a bit late to avoid major design flaws in the earlier steps. November/December 2001

IEEE SOFTWARE

69

The GUI methodology’s main strength lies in the mapping between users and tasks and that between tasks and presentation elements.

GUI design methodology For GUIs, a proven method for improved interface usability is user-centered design.3 It broadly involves the following steps: 1. Set up user types. 2. Define tasks, or operational user scenarios, for each user type.4 3. Design the user interface by specifying presentation elements to complete these tasks. The GUI methodology’s main strength lies in the mapping between users and tasks and that between tasks and presentation elements. These two mappings provide not only a direct aid in focusing the designer’s attention on the user, but also help in discovering the application’s usability, such as by revealing the number of steps required to complete a task. How different are these paradigms? Let’s step back a bit from the interface design techniques and analyze the differences between the WUI and GUI application paradigms themselves. Major differences between them include the following: ■

■

■

■

User characterization is more difficult on the Web because it typically addresses an unknown audience. For applications, however, the user community is usually well defined. Applications are typically task-centric, whereas the Web is still largely information-centric. Customers are not stuck with a Web site the way they might be with an application or product they have purchased. At the slightest difficulty, the surfer tends to move on to another site, maybe never to return again. Navigation between Web pages is much more flexible than that between forms (or screens) of an application. Thus, you can reach a particular Web page by any number of navigation paths provided on the site. In contrast, in the GUI application, the routes to a particular screen are extremely limited. Indeed, most screens have only one path to them.

The first distinction reveals a hurdle in defining the Web paradigm’s user types. Web design experts have suggested the creation of personas or detailed characteriza70

IEEE SOFTWARE

November/December 2001

tions of users belonging to different demographic profiles to predict the surfer’s behavior on the site. Other design experts define relevant user characteristics and predict user behavior based on various combinations of these characteristics, such as expert versus novice. The second issue pertains to a problem in defining tasks for a Web site. In the information-centric paradigm, search is the key operation. It does not seem to fit the definition of a task because it is too flexible and vague. For this reason, the Web techniques assume that search will be made simpler if the content is organized logically and proper labeling and navigation schemes are provided. So why do so many content organizations fail on this account? I contend that organizations focus on the content they want to share with the world and not on what the world would like to know about them. I suggest that we take each of the user types defined earlier and set up tasks for them such as “Find product information” and “Find company profile” based on our guesstimate of user interest. Not only will such a scheme create a mapping between users and their tasks, but it will also provide direct guidance to how to organize content on the site. If users cannot accomplish the “Find product information” task within three page visits, the site’s design must be improved. Evidence that we can enumerate the search operation in this way comes from the WUI design method’s fourth step. We can define the same tasks given to the tester to perform on the site before we design the site. Difference three (customer loyalty) has two aspects. One, aesthetic appeal, is outside this discussion’s scope. The other aspect is usability, which is the subject of this article. Navigation flexibility is undoubtedly a nonissue in the traditional approach. Form navigation in older applications was normally so simple that it was unnecessary to model it explicitly using navigation maps. On the other hand, a way to model page navigation is critical for any Web design. I suggest bringing together the best of both worlds to yield a new and improved Web design methodology, thus: 1. Define the site’s mission and vision. 2. Envision the intended audience and model it as a finite number of user types.

3. Specify the tasks you expect each user type to perform. Include here the information they would be interested in viewing as “Find ” tasks. 4. Design the Web pages, including content and presentation elements, by building a mapping between the page and the task. The search or “Find ” tasks will then define the content required on the pages. Some tasks will span across a number of pages, thus automatically creating a page hierarchy and navigation path. Complete the page hierarchy and navigation paths for all remaining pages. 5. Design the labeling and search system so that it is optimized for the tasks. 6. Test the design.

T

he essentials of a user-centered interface design approach are definitely applicable for the Web. However, WUI gurus advocate a design path that

About the Author Uttara Nerurkar is a senior research associate at Infosys Technologies Limited, a CMM

Level 5 software services and consultancy company headquartered in Bangalore, India. She has worked in the software development area for the last nine years. Her research interests include software architecture, security, and user interfaces. She has a BTech in chemical engineering from Indian Institute of Technology. She is also a professional member of the ACM. Contact her at Software Concept Laboratory, Infosys Technologies Ltd., Electronics City, Hosur Rd., Bangalore 561 229, India; [email protected].

is totally divorced from the GUI approach. Applying some “ancient wisdom” from the GUI era could lead to more usable Web sites. References 1. K. Scoresby, “Win Consumers with Better Usability,” e-Business Advisor, vol. 18, no. 6, June 2000. 2. A.R. Puerta et al., “MOBILE: User-Centered Interface Building,” Proc. Conf. Human Factors and Computing Systems, ACM Press, New York, 1999. 3. L. Rosenfeld and P. Morville, Information Architecture for the World Wide Web, O’Reilly, New York, 1998. 4. B. Schneiderman, Designing the User Interface, Addison Wesley Longman, New York, 1998.

CALL FOR ARTICLES Initiating Software Product Lines

• What types of planning are necessary prior to initiating the product line? • How can existing products be incorporated into the product line? • What levels of resources are required to support the early phases of a product line project? • Are there lightweight approaches that can streamline the migration to product lines? • How much of the requirements and the architecture must be determined initially? • What must be done to existing development methods and processes to adapt them to the product line environment? • How long does it take to realize the benefits of a product line approach? We seek original articles on methods, tools, experiences, and techniques related to initiating software product lines. A full list of suggested topics is at computer.org/software/edcal.htm. Work should be presented in terms that are useful to the software community at large, emphasizing approaches taken and lessons learned. Please submit two electronic copies, one in RTF or Microsoft Word and one in PDF or Postscript, by 1 March 2002 to IEEE Software at [email protected]. Articles must not exceed 5,400 words including figures and tables, which count for 200 words each. The papers we deem within the theme’s scope will be peer-reviewed and are subject to editing for magazine style, clarity, and space. Please contact any of the special issue guest editors for more information about the focus or to discuss a potential submission; please contact IEEE Software ([email protected] or computer.org/software/author.htm) for author guidelines and submission details. Guest Editors: John D. McGregor Linda Northrop Clemson University Software Engineering Institute [email protected] [email protected]

Salah Jarrad Panasonic Wireless Design Center [email protected]

Klaus Pohl University of Essen [email protected]

November/December 2001

Submission Deadline: 1 March 2002 Publication Date: September/October 2002

The product line approach to software development is emerging as a technique for organizing the large-scale production of a number of related software products. Adopting the product line perspective on software development affects the usual way of doing business at the organizational and technical management levels as well as at the software engineering level. Many companies are investigating the potential benefits, establishing pilot projects, and baselining these initial efforts. They face a number of questions:

IEEE SOFTWARE

71

feature metrics

Implementing a Software Metrics Program at Nokia Tapani Kilpi, Nokia

Metrics play an important role in many software organizations’ continuous process improvement activities. In an organization with mature software processes such as Nokia, carefully adjusting the measurement program to the environmentspecific needs can save considerable effort. 72

IEEE SOFTWARE

ll software organizations experience chaos. In theory, the solution is easy: define and implement all key processes clearly enough, and the chaos is gone. Software process improvement is an approach for defining, organizing, and implementing software processes. Well-known SPI methodologies include ISO/IEC 9000 standards,1 the Capability Maturity Model,2 Bootstrap,3 and ISO/IEC 15504.4 Measuring plays a large role in these methodologies; to address this, an organization

A

will normally implement measurement programs,5 such as the Goal-Question-Metric method.6 However, organizations at varying maturity levels do not always need to follow the same steps. Focusing on effort usage—the working hours spent on implementation—this article will introduce and outline the Nokiaway metrics program, show how it differs from GQM, and explain the differences. Nokiaway Nokiaway is a real-world method used in Nokia’s Fixed Switching Research & Design Department. FSG R&D, named hereafter FSG, produces software for the DX200 product line of telephone switches. The DX200 program library has over 15 million source lines of code, and Nokia has committed more than 2,000 engineers to carry out DX200 development work. DX200 is

November/December 2001

possibly Finland’s largest software project. The Nokiaway metrics program comprises these continuous metrics activities: metrics definition, data collection, metrics analysis, and metrics reporting. Metrics definition In FSG, metrics definition is a part of continuous process improvement—that is, defining the metrics follows the same organizational structure as process improvement does. At the highest organizational level, the FSG management team steers FSG’s process improvement (see Figure 1). FSG’s processes comprise five separate process areas: release, design, testing, maintenance, and quality assurance. The process teams, each of which is responsible for a specific process area, manage these areas. A process development manager—a full-time process developer—heads each process team. Other team members are 0740-7459/01/$10.00 © 2001 IEEE

Other units in Nokia

• Master Improvement Plan accepted (monthly) • Milestone decisions • Resource commitment

Road maps Action plans

R&D management team

FSG R&D

Process teams Release

Design

Quality Testing Maintenance assurance

Headed by process development managers

Improvement project initiatives • Improve coordination • Improve portfolio management

FSG R&D departments

Figure 1. The FSG process improvement steering structure.

part-time process developers whose primary responsibilities include software development, testing, and so on. The FSG software quality assurance (SQA)7 process team has the overall responsibility of coordinating and running the measurement program. Other process teams are responsible for planning detailed improvement projects and defining the metrics belonging to their process area. The management team defines the long-term process improvement road maps and action plans and accepts and gives resource commitments to process improvement projects that the process teams propose. The FSG R&D departments implement the improvement projects. Typically, a process team defines and continuously updates a set of 10 to 20 metrics. Each metric addresses one of FSG’s approximately 20 metric profiles (including Release Program, Sw Project, Testing Project, Department, and Section). For instance, each process team has added one to three metrics to the Sw Project metrics profile. Many metrics are common to several different metrics profiles, but no profile resembles another. Typically, a metrics profile contains approximately ten metrics that most of the process teams (not necessarily all of them in all cases) define. The different process teams collectively define the sets of metrics for the FSG metrics library.

Data collection Ideally, data collection should be set so that the metrics save (that is, collect) data at its inception. This requirement demands a relatively high level of process maturity. The Nokiaway metrics program uses four different management tools and their data storage for collecting and creating metrics: Resource and Project, Fault, Test Case, and Inspection and Review. FSG project managers, software designers, test engineers, fault assistants, and people in other corresponding roles routinely use these tools. From the metrics point of view, this arrangement provides excellent circumstances for data collection. No one must collect the data after inception, because it already exists in data storage for each of the four tools. For example, project managers use the Resource and Project tool to control projects, so they are eager to feed in the required data. Other people in the project must also feed in their data, but it’s such a simple task that no one really complains. We similarly apply the other tools: they are a part of daily processes, and we feed data to them. Quality engineers then have the relatively easy task of producing the metrics values by using the management tools. This minimizes the traditional manual hard work, and we can direct attention to analyzing the metrics values. November/December 2001

IEEE SOFTWARE

73

600 Data analysis and interpretation total Data collection Develop GQM plan Conduct GQM interviews Identify and define goals GQM program planning

Person-hours

500 400 300 200 100 0

GQM team

Manager

Single engineer

10 engineers

Total

Figure 2. An example effort model for a routine application of GQM.8

Metrics analysis In many real-life metrics programs, metrics analysis gets little attention while other metrics activities are handled reasonably well. However, people in such organizations probably seldom ask themselves, “Who should react to the metrics values, and who should ensure that the necessary corrections are carried out?” If no one uses a metric, there’s no point in producing it. Defining analysis guidelines and analyzing responsibilities for metrics play a vital role in FSG metrics definition. This ensures that someone, with some instructions, analyzes a metric’s results. In FSG, SQA people and the quality engineers generate the metrics reports. They have been trained to analyze the metrics values based on each metric’s analysis guidelines. Typically, a quality engineer produces a set of approximately 10 metrics and analyzes them within the project’s scope. This quality engineer also has the expertise to carry out the local analysis of a metric’s value. In FSG, we could say that a quality engineer’s metrics duties focus primarily on analysis instead of data collection. This is not the traditional way, is it? Metrics reporting The metrics and analysis results are often all that most people in an organization see from a metrics program. These reports are typically saved to a database or published on the Web. From both of these places, everyone in the organization can access the 74

IEEE SOFTWARE

November/December 2001

reports—in theory. What’s missing is the periodical review carried out by predefined people. Also missing are the decisions these people make on the basis of the metrics results. In a situation such as this, there is no hope that changes are planned and carried out on the basis of the metrics. The FSG metrics profiles are defined to measure different types of targets. If the measured target is, for example, a project, the project’s quality engineer produces the Project profile metrics. Then, the quality engineer presents the metrics report at a project meeting. This works well when the quality engineer knows the project and people involved. This makes it easier to translate the metrics values to an understandable language and communicate the results to those who should be most concerned with the metrics values: the people in the project. A comparison of GQM and Nokiaway GQM is a well-known and widespread method for planning and implementing a metrics program.7 It represents a systematic approach for tailoring and integrating goals to the software process models, software products, and quality perspectives of interest—on the basis of the project’s and organization’s specific needs. (FSG defines the corresponding goals as part of the Master Improvement Plan, a yearly process improvement plan.) I compare Nokiaway with the GQM as exemplified by Rini van Solingen and Egon Berghout’s practical interpretation and example effort calculations of implementing a GQM metrics program.8 General characteristics Nokiaway is a free interpretation of GQM. With respect to a standard GQM approach, it has these essential enhancements: ■

■ ■ ■

■

It uses a quality metrics library instead of defining a new set of metrics for each project. It automatically, instead of manually, collects raw data for the metrics. It semiautomatically (instead of manually) produces the reports’ metrics charts. Most people running operative tasks in the measurement program are part-time instead of full-time employees. It uses the Quality Plan and FSG Met-

rics Guidelines instead of a GQM plan for defining a measurement program’s exact background and scope.

routine application of a typical GQM measurement program with these characteristics:8 ■

A fundamental reason for these extensions is that FSG is a multiproject environment; that is, several projects of the same type run in FSG instead of just one project. It is important that the experiences of all FSG’s measurement programs are collected and analyzed to further improve the metrics. FSG saves this data in the FSG Metrics Guidelines; that is, the document’s contents form a metrics library. Some GQM purists might criticize Nokiaway for depending too much on predefined metrics and losing sight of the GQM idea of defining goal-driven metrics separately for each specific case. I have this response:

■ ■ ■ ■ ■ ■

Measurement program target: one software project. Project team size: 11 persons (one project manager and 10 software engineers). Project duration: approximately one year. One major goal. An existing tool infrastructure. No need for special training; the participants are familiar with the program. Feedback sessions during the project: five.

Figure 2 also shows the following cost structure, which is typical for a routine GQM application:

In many reallife metrics programs, metrics analysis gets little attention while other metrics activities are handled reasonably well.

Roughly 30 percent of the effort goes toward defining the measurement program, and 70 percent goes toward continuing the measurement program, which is almost completely spent on feedback sessions. ■ On the measurement program, the GQM team spends 70 percent of the effort; the project team spends only 30 percent of the total effort. ■ The project team’s effort is less than 1 percent of their total working time. ■ A typical GQM measurement program over one calendar year in a project of 11 persons requires three person-months of effort.8 ■

■

■

■

■

■

Is there really no point in defining common metrics that can be applied across several similar projects in an organization? Do independently defined metrics for different but similar projects really vary that much from each other? Doesn’t management, after all, have the main responsibility in deciding an organization’s process improvement strategy—including goals? Could an organization provide individuality in different measurement programs by selecting a subset of metrics from a predefined, actively updated metrics library? Aren’t collecting raw metrics data and producing the metrics reports laborious tasks, which require exhaustive resources when performed manually?

While posing these questions with certain answers in mind, we must remember that FSG is a large organization with specific characteristics. Environment and circumstances greatly influence planning an organization’s measurement program. My hope is that FSG’s experiences can offer other large organizations something to think about when applying GQM or another measurement program method. Effort models Solingen and Berghout presented an example effort model (see Figure 2) for the

Table 1 compares GQM and Nokiaway activities, with Solingen and Berghout’s description of their effort model in the left column.8 In GQM, the GQM plan contains the detailed description of the measurement program’s activities, their schedule, and their associated responsibilities. In FSG, the quality plan contains the corresponding information for implementing a measurement program. Figure 3 presents the total resource use in routine applications of standard GQM and Nokiaway. (Analyzing the realized hours of 10 FSG projects has produced the estimations for the realized working hours spent on running a measurement program in FSG.) The total effort in Nokiaway appears to be less than that of the GQM measurement program’s. This is because November/December 2001

IEEE SOFTWARE

75

Table 1 Correspondences between the Goal-Question-Metric and Nokiaway activities Goal-Question-Metric8

Nokiaway

■ Plan GQM program. This includes identifying available input,

preconditions, and constraints; setting up an infrastructure; selecting an improvement area and a project; initially planning the measurement program; and preparing and training the project team.

■ Identify and define GQM goals. This includes characterizing projects

and organizations, identifying and selecting improvement goals, defining measurement and GQM goals, modeling relevant software processes, and identifying artifacts for reuse. ■ Conduct GQM interviews. This includes studying documentation; identifying, scheduling, and inviting interviewees; briefing project teams; and conducting and reporting GQM interviews. ■ Develop GQM deliverables. This includes defining, reviewing, and

refining a GQM plan; defining a measurement plan; identifying and defining data collection procedures; reviewing and refining a measurement plan; and developing an analysis plan. ■ Collect data. This includes a trial to test data collection procedures

and forms; briefing the project team; launching the measurement program; and collecting, validating, coding, and storing measurement data. ■ Analyze and interpret data. This includes analyzing the measurement

■ Define FSG metrics guidelines. This document defines all quality metrics

that FSG uses, their intended use, directions and criteria for analyzing them, and so on. ■ Prepare a quality plan. This includes definitions for quality organization and responsibilities, a schedule for quality feedback sessions, training activities, most common tools and instructions, internal and external verification activities, quality criteria and metrics (a subset selected from the FSG metrics guidelines), a schedule and target group for quality reports, and a status control table for the quality plan follow-up. ■ Define measurement key areas. FSG process teams define the process areas’ process improvement goals on the basis of their observations and on the quality metrics used to evaluate these areas. ■ Apply metrics feedback. All quality metrics that FSG uses have been

■ ■

■ ■

defined in close cooperation with FSG practitioners. The process teams and the SQA people collect the feedback between projects to improve the metrics for the next use. Update the FSG metrics guidelines. Prepare the quality plan’s measurement parts. This part of the quality plan defines a subset of the metrics included in the FSG metrics guidelines according to the project-specific process improvement goals and possibly defines project-specific directions for applying the metrics. Collect data. Process-oriented tools automatically collect all data needed for producing the FSG quality metrics. Analyze and interpret data. The FSG SQA personnel carry this out; most of them are software or test engineers working in SQA part-time.

data; preparing the presentation material; and planning, conducting, and reporting the feedback sessions.

Nokiaway uses a high degree of automation in data collection while GQM does not. However, because we can apply automated data collection to GQM, the overall Nokiaway effort profile differs insignificantly from an average GQM effort profile.

600 GQM FSG

Person-hours

500 400

A

300 200 100 0

GQM/SQA team

Figure 3. Nokiaway. 76

Manager

Effort

IEEE SOFTWARE

usage

Single engineer

10 engineers

comparison

November/December 2001

between

Total

GQM

and

well-planned and carefully focused measurement program is a useful aid for an organization that highly values continuous process improvement. Some capable methods, such as GQM, for planning and implementing a measurement program tend to offer an independent and complete set of steps and means for process improvement. There is nothing wrong in this, but many organizations already have a successful history in applying different approaches, such as ISO/IEC, CMM, and Bootstrap. As a result of this evolution,

these organizations have already developed their processes to a relatively high maturity level and do not need to start over. The enhancements to GQM I’ve described can save approximately 50 percent of the cost of implementing and running a measurement program that relies on manual work. These improvements also help minimize bureaucracy and prevent people from losing their motivation for measuring. References 1. ISO 90001, Quality Systems—Model for Quality Assurance in Design/Development, Production, Installation and Servicing, Int’l Organization for Standardization, Geneva, 1994. 2. B. Curtis, W.E. Hefley, and S. Miller, People Capability Maturity Model (P-CMM), tech. report CMU/SEI-95MM-02, Software Eng. Inst., Carnegie Mellon Univ., Pittsburgh, 1995. 3. P. Kuvaja et al., Software Process Assessment and Improvement: The Bootstrap Approach, Blackwell, Oxford, UK, 1994. 4. Information Technology: Software Process Assessment, ISO/IEC tech. report 15504 Type 2, Int’l Organization for Standardization and Int’l Electrotechnical Commission, Geneva, 1998.

About the Author Tapani Kilpi is the chief researcher at the Solid Applied Research Center, Oulu, Finland.

His current responsibilities focus on network management and 3G–4G applications. He received his MS and PhD in information processing science at the University of Oulu. Contact him at Solid, Elektroniikkatie 6, 90570 Oulu, Finland; [email protected].

5. N.E. Fenton and S.L. Pfleeger, Software Metrics: A Rigorous and Practical Approach, Int’l Thomson Computer Press, London, 1996. 6. V.R. Basili, C. Caldiera, and H.D. Rombach, “Goal Question Metric Paradigm,” Encyclopedia of Software Engineering, vol. 1, J.J. Marciniak, ed., John Wiley & Sons, New York, 1994. 7. ISO 9000-3, Guidelines for the Application of ISO 9001 to the Development, Supply and Maintenance of Software: Quality Management and Quality Assurance Standards—Part 3, Int’l Organization for Standardization, Geneva, 1991. 8. R.van Solingen and E. Berghout, The Goal/Question/ Metric Method: A Practical Guide for Quality Improvement of Software Improvement, McGraw-Hill, Cambridge, UK, 1999.

For more information on this or any other computing topic, please visit our Digital Library at http://computer.org/publications/dlib. COMPUTER

PURPOSE The IEEE Computer Society is the world’s

MEMBERSHIP Members receive the monthly magazine COM PUTER , discounts, and opportunities to serve (all activities are led by volunteer members). Membership is open to all IEEE members, affiliate society members, and others interested in the computer field.

BOARD OF GOVERNORS Term Expiring 2001: Kenneth R.Anderson, Wolfgang K. Giloi, Haruhisa Ichikawa, Lowell G. Johnson, Ming T. Liu, David G. McKendry, Anneliese Amschler Andrews Term Expiring 2002: Mark Grant, James D. Isaak, Gene F. Hoffnagle, Karl Reed, Kathleen M. Swigger, Ronald Waxman, Akihiko Yamada

EXECUTIVE COMMITTEE

President: BENJAMIN W. WAH* University of Illinois Coordinated Sci Lab 1308 W. Main St Urbana, IL 61801-2307 Phone: +1 217 333 3516 Fax: +1 217 244 7175

[email protected]

President: JOEL B. SNYDER

President-Elect: WILLIS K. KING* Past President: GUYLAINE M. POLLOCK* VP, Educational Activities: CARL K. CHANG (1ST VP)* VP, Conferences and Tutorials: GERALD L. ENGEL* VP, Chapters Activities: JAMES H. CROSS VP, Publications: RANGACHAR KASTURI VP, Standards Activities: LOWELL G. JOHNSON* VP, Technical Activities: DEBORAH K. SCHERRER

President-Elect: RAYMOND D. FINDLAY

(2ND VP)*

Term Expiring 2003: Fiorenza C.AlbertHoward, Manfred Broy, Alan Clements, Richard A. Kemmerer, Susan A. Mengel, James W. Moore, Christina M. Schober Next Board Meeting: 8 Feb 2001, Orlando, FL

†

†

IEEE

OFFICERS

Executive Director: DANIEL J. SENESE Secretary: HUGO M. FERNANDEZ VERSTAGEN Treasurer: DALE C. CASTON

Secretary: WOLFGANG K. GILOI* Treasurer: STEPHEN L. DIAMOND* 2000–2001 IEEE Division V Director: 2001–2002 IEEE Division VIII Director:

VP, Publications Activities:JAMES M. TIEN

THOMAS W. WILLIAMS†

VP, Standards Association: MARCO W. MIGLIARO VP, Technical Activities: LEWIS M. TERMAN President, IEEE-USA: NED R. SAUTHOFF

WEB

SITE

Acting Executive Director:

COMPUTER SOCIETY O F F I C E S Headquarters Office 730 Massachusetts Ave. NW Washington, DC 20036-1992 Phone: +1 202 371 0101 • Fax: +1 202 728 9614 E-mail: [email protected] Publications Office 10662 Los Vaqueros Cir., PO Box 3014 Los Alamitos, CA 90720-1314 Phone:+1 714 821 8380 E-mail: [email protected] Membership and Publication Orders: Phone:+1 800 272 6657 Fax:+1 714 821 4641 E-mail: [email protected] European Office 13, Ave. de L’Aquilon B-1200 Brussels, Belgium Phone: +32 2 770 21 98 • Fax: +32 2 770 85 05 E-mail: [email protected] Asia/Pacific Office Watanabe Building 1-4-2 Minami-Aoyama, Minato-ku, Tokyo 107-0062, Japan Phone: +81 3 3408 3118 • Fax: +81 3 3408 3553 E-mail: [email protected]

EXECUTIVE

DORIS L. CARVER†

VP, Educational Activities: LYLE D. FEISEL VP, Regional Activities: ANTONIO BASTOS

SOCIETY

The IEEE Computer Society’s Web site, at http://computer.org, offers information and samples from the society’s publications and conferences, as well as a broad range of information about technical committees, standards, student activities, and more.

largest association of computing professionals, and is the leading provider of technical information in the field.

ANNE MARIE KELLY†

* voting member of the Board of Governors

STAFF

Acting Executive Director : ANNE MARIE KELLY Publisher: ANGELA BURGESS Acting Director, Volunteer Services: MARY-KATE RADA Chief Financial Officer: VIOLET S. DOAN Director, Information Technology & Services: ROBERT CARE Manager, Research & Planning: JOHN C. KEATON 26-OCT-2001

feature

programming languages

Cracking the 500-Language Problem Ralf Lämmel and Chris Verhoef, Free University of Amsterdam

Parser implementation effort dominates the construction of software renovation tools for any of the 500+ languages in use today. The authors propose a way to rapidly develop suitable parsers: by stealing the grammars. They apply this approach to two nontrivial, representative languages, PLEX and VS Cobol II. 78

IEEE SOFTWARE

t least 500 programming languages and dialects are available in commercial form or in the public domain, according to Capers Jones.1 He also estimates that corporations have developed some 200 proprietary languages for their own use. In his 1998 book on estimating Year 2000 costs, he indicated that systems written in all 700 languages would be affected.2 His findings inspired many Y2K whistle-blowers to characterize this situation as a major impediment to solving the Y2K

A

problem; this impediment became known as the 500-Language Problem. In 1998, we realized that we had discovered a breakthrough in solving the 500LP— so we had something to offer regarding the Y2K problem. We immediately informed all the relevant Y2K solution providers and people concerned with the Y2K awareness campaign. In answer to our emails, we received a boilerplate email from Ed Yourdon explaining that the 500LP was a major impediment to solving the Y2K problem (which we knew, of course). Ed was apparently so good at creating awareness that this had backfired on him: he got 200 to 300 messages a day with Y2K questions and was no longer able to read, interpret, and answer his email other than in “write-only” mode. Although he presumably missed our input, his response regarding the 500LP is worth quoting:

November/December 2001

I recognize that there is always a chance that someone will come up with a brilliant solution that everyone else has overlooked, but at this late date, I think it’s highly unlikely. In particular, I think the chances of a “silver bullet” solution that will solve ALL y2k problems is virtually zero. If you think you have such a solution, I have two words for you: embedded systems. If that’s not enough, I have three words for you: 500 programming languages. The immense variety of programming languages (yes, there really are 500!), hardware platforms, operating systems, and environmental conditions virtually eliminates any chance of a single tool, method, or technique being universally applicable.

The number 500 should be taken poetically, like the 1,000 in the preserving process for so-called 1,000-year-old eggs, which last only 100 days. For a start, we 0740-7459/01/$10.00 © 2001 IEEE

should add the 200 proprietary languages. Moreover, other estimates indicate that 700 is rather conservative: in 1971, Gerald Weinberg estimated that by the following year, programming languages would be invented at the rate of one per week—or more, if we consider the ones that never make it to the literature, and enormously more if we consider dialects.3 Peter de Jager also helped raise awareness of the 500LP. He writes this about the availability of Y2K tools:4 There are close to 500 programming languages used to develop applications. Most of these conversion or inventory tools are directed toward a very small subset of those 500 languages. A majority of the tools are focused on Cobol, the most popular business programming language in the world. Very few tools, if any, have been designed to help in the area of APL or JOVIAL for example.

If everyone were using Cobol and only a few systems were written in uncommon languages, the 500-Language Problem would not be important. So, knowing the actual language distribution of installed software is useful. First, there are about 300 Cobol dialects, and each compiler product has a few versions—with many patch levels. Also, Cobol often contains embedded languages such as DMS, DML, CICS, and SQL. So there is no such thing as “the Cobol language.” It is a polyglot, a confusing mixture of dialects and embedded languages—a 500-Language Problem of its own. Second, according to Jones, the world’s installed software is distributed by language as follows: ■ ■ ■ ■

Cobol: 30 percent (225 billion LOC) C/C++: 20 percent (180 billion LOC) Assembler: 10 percent (140 to 220 billion LOC) less common languages: 40 percent (280 billion LOC)

In contrast, there were Y2K search engines for only about 50 languages and automated Y2K repair engines for about 10 languages.2 Thus, most languages had no automated modification support, clarifying the concerns of Jones, Yourdon, McCabe, de Jager, and others. These alarming figures underscored the 500LP’s importance.

What is the 500-Language Problem? We entered the new millennium without much trouble, so you might conclude that whatever the 500LP was, it is not relevant now. Of course, the problem existed before the Y2K gurus popularized it, and it has not gone away. Why is the problem still relevant? If you want tools to accurately probe and manipulate source code, you must first convert the code from text format to tree format. To do this, you need a so-called syntactic analyzer, or parser. But constructing a parser is a major effort, and the large up-front investment hampers initiatives for many commercial tool builders. Indeed, a few years ago Tom McCabe told us that his company, McCabe & Associates, had made a huge investment in developing parsers for 23 languages. Noting that 500 would be insurmountable, he dubbed this problem “the number one problem in software renovation.” Thus, the 500LP is the most prominent impediment to constructing tools to analyze and modify existing software assets. Because there are about a trillion lines of installed software written in myriad languages, its solution is a step forward in managing those assets. Solutions that don’t work A solution sometimes suggested for the 500LP is just to convert from uncommon languages to mainstream ones for which tool support is available. However, you need a full-blown tool suite—including a serious parser—to do this. And obtaining a parser is part of the 500LP. So language conversion will not eliminate the problem— on the contrary, you need a solution for the 500LP to solve conversion problems. A Usenet discussion on comp.compilers offered a second suggestion to solve the 500LP: generating grammars from the source code only, in the same way linguists try to generate a grammar from a piece of natural language. In search of solutions, we studied this idea and consulted the relevant literature. We did not find any successful effort where the linguistic approach helped to create a grammar for a parser in a costeffective way. We concluded that the linguistic approach does not lead to useful grammar inferences from which you can build parsers.5 November/December 2001

IEEE SOFTWARE

79

Tool 1 Tool 2 Pretty printer Parser

Grammar

Grammar

...

Tree walker Generic core Analyzer

...

Editor Tool n

Figure 1. Effort shift for renovation tool development. The longer the arrow, the more effort is needed. The dashed line represents the greater effort needed if the traditional approach is used.

Another, more reasonable suggestion is to reuse the parser from compilers: just tap a compiler’s parser output and feed it to a modification tool. Prem Devanbu’s programmable GENOA/GENII tool can turn a parser’s idiosyncratic output format into a more suitable format for code analysis.6 There is, however, one major drawback to this approach: as Devanbu points out, the GENOA system does not allow code modification. This is not a surprise: a compiler’s parser removes comments, expands macros, includes files, minimizes syntax, and thus irreversibly deforms the original source code. The intermediate format is good enough for analysis in some cases, but the code can never be turned into acceptable text format again. Hence, the approach does not help regarding mass modifications, for which the Gartner Group recommends tool support to handle a larger code volume.7,8 Obviously, this concerns Y2K and Euro conversions, code restructuring, language migrations, and so on. Another real limitation of Devanbu’s approach is that, even if you only want to do code analysis, you often cannot get access to a compiler company’s proprietary source. How we are cracking the 500LP Recall that Yourdon claimed that the large number of programming languages would virtually eliminate any chance of a single tool, method, or technique being universally applicable. Nevertheless, there is a single, feasible solution for the 500LP. It is 80

IEEE SOFTWARE

November/December 2001

cracked when there is a cheap, rapid, and reliable method for producing grammars for the myriad languages in use so that existing code can be analyzed and modified. Cheap is in the US$25,000 ± $5,000 range, rapid is in the two-week range (for one person), and reliable means the parser based on the produced grammar can parse millions of LOC. Why is this a solution? A grammar is hardly a Euro conversion tool or a Y2K analyzer. It is because the most dominant factor in building renovation tools is constructing the underlying parser. From grammar to renovation tool Renovation tools routinely comprise the following main components: preprocessors, parsers, analyzers, transformers, visualizers, pretty printers, and postprocessors. In many cases, language-parameterized (or generic) tools are available to construct these components. Think of parser generators, prettyprinter generators, graph visualization packages, rewrite engines, generic dataflow analyzers, and the like. Workbenches providing this functionality include Elegant, Refine, and ASF+SDF, for instance, but there are many more. Figure 1 depicts a grammar-centric approach to enabling rapid development of renovation tools. Arrow length indicates the degree of effort involved (longer arrows imply more effort). As you can see, if you have a generic core and a grammar, it does not take much effort to construct parsers, tree walkers, pretty printers, and so on. Although these components depend on a particular language, their implementation uses generic language technology: a parser is produced using a parser generator, a pretty printer is created using a formatter generator,9 and tree walkers for analysis or modification are generated similarly.10 All these generators rely heavily on the grammar. Once you have the grammar and the relevant generators, you can rapidly set up this core for developing software renovation tools. Leading Y2K companies indeed constructed generic Y2K analyzers, so that dealing with a new language would ideally reduce to constructing a parser. The bottleneck is in obtaining complete and correct grammar specifications. The longest arrow in Figure 1 expresses the current situation: it takes a lot of effort to create those grammars.

Implementing a high-quality Cobol parser can take two to three years, as Vadim Maslov of Siber Systems posted on the Usenet newsgroup comp.compilers (he has constructed Cobol parsers for about 16 dialects). Adapting an existing Cobol parser to cope with new dialects easily takes three to five months. Moreover, patching existing grammars using mainstream parser technology leads to unmaintainable grammars,11,12 significantly increasing the time it takes to adapt parsers. In contrast, Table 1 lists the effort expended on various phases of a typical Cobol renovation project that used our grammar-centric solution. Notice that the grammar part of this project took only two weeks of effort, so the team could start developing actual renovation tools much more quickly. This Cobol renovation project concerned one of the world’s largest financial enterprises, which needed an automatic converter from Cobol 85 back to Cobol 74 (the 8574 Project).13 The Cobol 85 code was machine-generated from a fourth-generationlanguage tool, so the problem to convert back was fortunately restricted due to the code generator’s limited vocabulary. It took some time to solve intricate problems, such as how to simulate Cobol 85 features like explicit scope terminators (END-IF, ENDADD) and how to express the INITIALIZE statement in the less-rich Cobol 74 dialect. The developers discussed solutions with the customer and tested them for equivalence. Once they solved these problems, implementing the components was not difficult because they had the generic core assets generated from a recovered Cobol 85 grammar. They cut the problem into six separate tools and then implemented all of them in only five days. The programming by hand was limited (fewer than 500 LOC), but compiled into about 100,000 lines of C code and 5,000 lines of makefile code (linking all

Table 1 Effort for the 8574 Project Phase

Effort

Extract the grammar Generate the parser Build six tools Assemble all the components Total

Two weeks One day Five days One hour Three weeks

the generated generic renovation functionality). After compilation to six executables (2.6 Mbytes each), it took 25 lines of code to coordinate them into a distributed, component-based software renovation factory, which then converted Cobol 85 code to Cobol 74 at a rate of 500,000 LOC per hour using 11 Sun workstations. Measuring this and other projects, it became clear to us that the total effort of writing a grammar by hand is orders of magnitude larger than constructing the renovation tools themselves. So the dominant factor in producing a renovation tool is constructing the parser. Building parsers using our approach reduces the effort to the same order of magnitude as constructing the tools. Building parsers in turn is not hard: use a parser generator. But the input for the generator is a grammar description, so complete and correct grammars are the most important artifacts we need to enable tool support. When we find an effective solution for producing grammars quickly for many languages, we have solved the 500LP. But how do we produce grammars quickly? For years, we and many others have been recapturing an existing language’s syntax by hand: we took a huge amount of sources, manuals, books, and a parser generator and started working. But then we realized that this hand work is not necessary. Because we are dealing with existing languages, we just steal and massage the underlying grammars according to our needs.

When we find an effective solution for producing grammars quickly for many languages, we have solved the 500LP.

Grammar stealing covers almost all languages The following exhaustive case distinction shows that our approach covers virtually all languages. Let’s look at the coverage diagram for grammar stealing shown in Figure 2. Because the software we want to convert already exists, it can be compiled or interpreted. We first enter the Compiler Sources diamond. There are two possibilities: the source code is or is not available to you. If it is, you just have to find the part that turns the text into an intermediate form. That part now contains the grammar in some form. You do this by lexically searching the compiler source code for the language’s keywords. Compiler constructors implement a parser in one of three ways: they hard-code it, use a parser generator, or do both (in a complex multilanguage compiler, for inNovember/December 2001

IEEE SOFTWARE

81

No

Start

Compiler sources?

Hard-coded parser

Quality?

One case: perl

Yes

Yes

Recover the grammar

BNF

No

No

Language reference manual?

General rules Yes

Recover the grammar

Quality?

Constructions by example

No cases known

Yes

Recover the grammar One case: RPG

No No cases known

Figure 2. Coverage diagram for grammar stealing.

stance). Figure 2 shows the first two cases— the third is just a combination. If you start with a hard-coded grammar, you must reverse-engineer it from the handwritten code. Fortunately, the comments of such code often include BNF rules (Backus Naur Forms) indicating what the grammar comprises. Moreover, because compiler construction is well-understood (there is a known reference architecture), compilers are often implemented with well-known implementation algorithms, such as a recursive descent algorithm. So, the quality of a hard-coded parser implementation is usually good, in which case you can easily recover the grammar from the code, the comments, or both. Except in one case, the Perl language,14 the quality of the code we worked with was always sufficient to recover the grammar. If the parser is not hard-coded, it is generated (the BNF branch in Figure 2), and some BNF description of it must be in the compiler source code. So, with a simple tool that parses the BNF itself, we can parse the BNF of the language that resides in the compiler in BNF notation, and then extract it. When the compiler source code is not accessible (we enter the Language Reference Manual diamond in Figure 2), either a reference manual exists or not. If it is available, it could be either a compiler vendor manual or an official language standard. The language is explained either by exam82

IEEE SOFTWARE

November/December 2001

ple, through general rules, or by both approaches. If a manual uses general rules, its quality is generally not good: reference manuals and language standards are full of errors. It is our experience that the myriad errors are repairable. As an aside, we once failed to recover a grammar from the manual of a proprietary language for which the compiler source code was also available (so this case is covered in the upper half of Figure 2). As you can see in the coverage diagram, we have not found low-quality language reference manuals containing general rules for cases where we did not have access to the source code. That is, to be successful, compiler vendors must provide accurate and complete documentation, even though they do not give away their compilers’ source code for economic reasons. We discovered that the quality of those manuals is good enough to recover the grammar. This applies not only to compiler-vendor manuals but also to all kinds of de facto and official language standards. Unusual languages rarely have high-quality manuals: either none exists (for example, if the language is proprietary) or the company has only a few customers. In the proprietary case, a company is using its in-house language and so has access to the source code; in the other case, outsiders can buy the code because its business value is not too high. For instance, when Wang went bankrupt, its

key customers bought the source code for its operating system and compilers to create their own platform and dialect migration tools. This explains why we do not know of low-quality manuals containing general rules. In one case, that of RPG, the manual explains the language through code examples, and general rules are absent. We can examine this case in more detail if we are asked for an RPG renovation project involving a large amount of RPG code. We think we can systematically extract RPG’s general rules from the code examples. In addition, because the manual contains code examples, there is a good chance that the compiler has tested these examples. This means that the manual’s formal content could be of a much higher quality than you would expect from such documents. Finally, we must deal with the case in which we have no access to the compiler sources or a reference manual. Capers Jones mailed us that “for a significant number of applications with Y2K problems, the compilers may no longer be available either because the companies that wrote them have gone out of business or for other reasons.” He did not come up with actual examples. Recall Wang’s bankruptcy: key customers just bought the source code and hence could solve their problems using the upper half of Figure 2. Theoretically, we cannot exclude Jones’s case—for instance, responding emotionally, Wang’s core developers could have thrown away the sources. You can learn an important lesson from this: Contracts between you and the vendor of a business-critical language should include a solution for source access in case of bankruptcy or terminated support (for example, giving the sealed source code to key customers). Summarizing, our coverage diagram shows that you can recover virtually any grammar, whether you have the compiler sources or not. But what about semantics? Some people think you need up-front, indepth knowledge of a language’s semantics to change code. If you recover the BNF, you can generate a syntax analyzer that produces trees, but the trees are not decorated with extra knowledge such as control flow, data flow, type annotation, name resolution, and so on. Some people also think you need a lot of semantical knowledge to analyze and

modify existing software, but this is not true. You can try to capture a language’s semantical knowledge on three levels: ■ ■ ■

for all the compilers of a language (different dialects), for one compiler product, or on a project-by-project basis.

Because we are trying to facilitate the construction of tools that work on existing software, there is already a compiler. This has implications for dealing with the semantical knowledge. Consider the following Cobol excerpt:

Some people think you need up-front, in-depth knowledge of a language’s semantics to change code.

PIC A X(5) RIGHT JUSTIFIED VALUE ’IEEE’. DISPLAY A.

The OS/VS Cobol-compiled code prints the expected result—namely, “ IEEE”—which is right justified. However, because of a change in the 1974 standard, the same code compiled with a Cobol/370 compiler displays the output “IEEE ” with a trailing space, which is left justified. This is because the RIGHT JUSTIFIED phrase does not affect VALUE clauses in the case of the Cobol/370 compiler. There are many more such cases, so trying to deal with the semantics of all compilers in advance is not feasible. Even when you restrict yourself to one compiler, this problem does not go away. Consider this Cobol fragment: 01 A PIC 9999999. MOVE ALL ’123’ to A. DISPLAY A.

Depending on the compiler flags used to compile this code, the resulting executables display either 3123123 or 1231231. There are hundreds of such problems, so it is also infeasible to capture the semantics in advance for a single compiler. No single semantics is available, and gathering all variants is prohibitively expensive and error prone given the semantical differences between compilers, compiler versions, and even compiler flags used. The good news is that you only need specific ad hoc elements of the semantics on a per-project basis. We call this demand-driven semantics. For instance, the NEXT SENTENCE November/December 2001

IEEE SOFTWARE

83

We recovered the PLEX grammar in two weeks, including tool construction, parser generation, and testing, at a cost of US$25,000.

84

IEEE SOFTWARE

(a)

IF X=1 THEN IF Y=1 THEN NEXT SENTENCE END-IF DISPLAY ’Nested IF passed’ END-IF. DISPLAY ’SENTENCE passed’.

(b)

IF X=1 THEN IF Y=1 THEN CONTINUE END-IF DISPLAY ’Nested IF passed’ END-IF. DISPLAY ’SENTENCE passed’.

Figure 3. A code segment (a), transformed inappropriately into the code segment in (b). The change in line 3 results in wrong output.

phrase in Cobol directs control to the statement after the next separation period (denoted with a dot). So, depending on where people put a dot, the code jumps directly behind the dot. Omitting a dot can lead to different behavior. One of our customers wanted tools to get rid of this potentially hazardous implicit jump instruction. Luckily, it turned out that for this project, we could replace the implicit jump instruction NEXT SENTENCE with the innocent no-op CONTINUE. So, after our semantical investigation, we knew we could use a simple transformation tool to make this change. However, in another project, this transformation might break down—for example, if the NEXT SENTENCE phrase is used in problematic code patterns. The transformation of the code in Figure 3a into the code in Figure 3b has changed the program’s meaning: you cannot turn NEXT SENTENCE into CONTINUE in the context of Figure 3. Specifically, assuming both X and Y are equal to 1, the code in Figure 3a prints “SENTENCE passed” while the code in Figure 3b prints first “Nested IF passed” and then “SENTENCE passed”. As you can see, you must be utterly aware of the semantics to find out whether it is necessary to implement any of it. In most cases we have seen, implementing this type of intricate semantical issue was not necessary—but knowing about the potential problems was necessary, if only to check whether they were present. To give you an idea how far you can go with demand-driven semantics, consider this: we have developed relatively dumb tools for some Cobol systems that can wipe out complex GO TO logic.15 You do need to know the semantics for many different tasks, but it is not necessary in advance to encode the compiler semantics in a parse tree or otherwise. So, a tool developer can construct (mostly) syntactic tools taking semantical knowledge into account on a per-project basis.

November/December 2001

Grammar stealing in practice We—and others from industry and academia—have applied grammar stealing successfully to a number of languages, including Java, PL/I, Ericsson PLEX, C++, Ada 95, VS Cobol II, AT&T SDL, Swift messages, and more. Here, we focus on PLEX (Programming Language for Exchanges), a proprietary, nontrivial, realtime embedded-system language (for which the compiler source code was accessible to us), and, at the other end of the gamut, VS Cobol II, a well-known business language (for which no compiler code was available). Both languages are used in businesscritical systems: the AXE 10 public branch exchange uses PLEX, and numerous IBM mainframe systems run VS Cobol II. These two languages represent the two main branches in Figure 2. Our approach uses a unique combination of powerful techniques: ■ ■ ■ ■

automated grammar extraction, sophisticated parsing, automated testing, and automated grammar transformation.

If one of these ingredients is missing, the synergy is gone. Extraction by hand is error prone, and basic parsing technology limits you to work with grammars in severely limited formats. With powerful parsing technology, you can work with arbitrary context-free grammars and test them regardless of their format. Without automated testing, you cannot find many errors quickly. Without tool support to transform grammar specifications, analyses are inaccurate and corrections are inconsistent; without transformations, you cannot repeat what you have done or change initial decisions easily. So, to steal grammars, you need to know about grammars, powerful parsing techniques, how to set up testing, and automated transformations.

= <program-header> <statement-row> ’END’ ’PROGRAM’ ’;’ %% xnsmtopg(1) ; %% -<= sect -Compound( -Reverse(STATEMENT-ROW.stat_list) => -PROGRAM-HEADER.sect.as_prog_stat : ix_stat_list_p -PROGRAM-HEADER.sect : ix_sect_node_p) ;

Figure 4. Raw compiler source code for the PLEX language.

Stealing from compiler source code Ericsson uses the extremely complex proprietary language PLEX to program public telephone switches. PLEX consists of about 20 sublanguages, called sectors, including high-level programming sectors, assembly sectors, finite-state-machine sectors, marshaling sectors, and others. We applied our grammar-stealing approach to PLEX as follows:16 1. Reverse-engineer the PLEX compiler (63 Mbytes of source code) on site to look for grammar-related files. We learned that there were BNF files and a hard-coded parser. 2. Find the majority of the grammars in some BNF dialect. 3. Find a hand-written proprietary assembly parser with BNF in the comments. 4. Write six BNF parsers (one for each BNF dialect used). 5. Extract the plain BNF from the compiler sources and convert it to another syntax definition formalism (SDF) for technical reasons. 6. Find the files containing the lexical analyzer and convert the lexical definitions to SDF. 7. Combine all the converted grammars into one overall grammar. 8. Generate an overall parser with a sophisticated parser generator. 9. Parse the code. We recovered the PLEX grammar in two weeks, including tool construction, parser generation, and testing with 8-MLOC PLEX code, at a cost of US$25,000. Ericsson told us that a cuttingedge reengineering company had estimated this task earlier at a few million dollars. When we contacted this company, they told us that US$25,000 was nothing for such a grammar. To illustrate the limited complexity of the work, consider the fragment of raw com-

piler source code in Figure 4. A PLEX program consists of a header, a list of statements, the phrase END PROGRAM, and a closing semicolon. The other code in the figure deals with semantic actions relevant to the compiler. Our tools converted this to a common BNF while removing the idiosyncratic semantic actions:

The tools we built automatically recovered most of the 3,000+ production rules in an afternoon.

plex-program ::= program-header statement-row ’END’ ’PROGRAM’ ’;’

Then our tools converted this into SDF, which was subsequently fed to a sophisticated parser generator accepting arbitrary context-free grammars. The output was Program-header Statement-row ”END” ”PROGRAM” ”;” -> Plex-program

The tools we built automatically recovered most of the 3,000+ production rules in an afternoon. Then we tested each sector grammar separately. We used a duplicate detector to weed out production rules that were used in more than one sector grammar, so that we could construct an overall grammar able to parse complete PLEX programs. One assembly sector parser was hard-coded (see Figure 2), so we had to recover its grammar through reverse engineering. The comments accompanying the code contained reasonably good BNF, so we had no problem with this task. With all the sector grammars combined, we generated a parser to test it with an 8-MLOC PLEX test suite. The only files that did not parse were compiler test files that were not supposed to parse—the rest passed the test. In addition, we generated a Web-enabled version of the BNF description as a basis for a complete and correct manual. November/December 2001

IEEE SOFTWARE

85

3.30 SEARCH Statement ___ Format 1--Serial Search _____________________________________ | | | >>__SEARCH__identifier-1______________________________________> | | |_VARYING____identifier-2___| | | |_index-name-1_| | | | | >_____________________________________________________________> | | |_________END__imperative-statement-1_| | | |_AT_| | | | | <_______________________________________________ | | >____WHEN__condition-1____imperative-statement-2___|__________> | | |_NEXT-SENTENCE__________| | | | | >____________________________________________________________>< | | | (1)| | | |_END-SEARCH____| | | | | Note: | x | (1) END-SEARCH with NEXT SENTENCE is an IBM extension. | |_________________________________________________________________| (a) search-statement = ”SEARCH” identifier [”VARYING” (identifier | index-name)] [[”AT”] ”END” statement-list] {”WHEN” condition (statement-list | ”NEXT” ”SENTENCE”)}+ [”END-SEARCH”] (b)

Figure 5. (a) The original syntax diagram for the Search statement; (b) the same diagram after conversion to BNF and correction.

Stealing from reference manuals Some of our colleagues felt a little fooled by the PLEX result: “You are not really constructing a parser; you only converted an existing one. We can do that, too. Now try it without the compiler.’’ Indeed, at first sight, not having this valuable knowledge source available seemed to make the work more difficult. After all, an earlier effort to recover the PLEX grammar from various online manuals had failed: they were not good enough for reconstructing the language.17 Later, we discovered that the manuals lacked over half of the language definition, so that the recovery process had to be incomplete by definition. We also found that our failure was due not to our tools but to the nature of proprietary manuals: if the language’s audience is limited, major omissions can go unnoticed for a long time. When there is a large customer base, the language vendor has to deliver better quality. 86

IEEE SOFTWARE

November/December 2001

In another two-week effort,5 we recovered the VS Cobol II grammar from IBM’s manual VS COBOL II Reference Summary, version 1.2. (For the fully recovered VS Cobol II grammar, see www.cs.vu.nl/grammars/vs-cobol-ii.) Again, the process was straightforward: 1. Retrieve the online VS Cobol II manual from www.ibm.com. 2. Extract its syntax diagrams. 3. Write a parser for the syntax diagrams. 4. Extract the BNF from the diagrams. 5. Add 17 lexical rules by hand. 6. Correct the BNF using grammar transformations. 7. Generate an error-detection parser. 8. Incrementally parse 2 million lines of VS Cobol II code. 9. Reiterate steps 6 through 8 until all errors vanish. 10. Convert the BNF to SDF. 11. Generate a production parser.

12. Incrementally parse VS Cobol II code to detect ambiguities. 13. Resolve ambiguities using grammar transformations. 14. Reiterate steps 11 through 13 until you find no more ambiguities. So, apart from some cycles to correct errors and remove ambiguities, the process is the same as in the earlier case, where we had access to the compiler source. An errordetection parser detects errors in the grammar from which it is generated. In this case, we used an inefficient top-down parser with infinite lookahead. It accepts practically all context-free grammars and does not bother with ambiguities at all. We use this kind of parser to test the grammar, not to produce parse trees. Because we only used compilable code, all the errors that this parser detects raise potential grammar problems. In this way, we found all the omissions, given our Cobol testbed. When all our test code passed the top-down parser, we converted the grammar to SDF, generated a parser that detects ambiguities, and corrected them. This project also took two weeks of effort, including tool construction and testing. We did this for free, so that we could freely publish the grammar on the Internet as a gift for Cobol’s 40th birthday.18 To get an idea of the limited complexity of this technique, consider the syntax diagram shown in Figure 5a, taken from the manual. After conversion to BNF and correction, the diagram looks like the one in Figure 5b. We used grammar transformations to remove the dash between NEXT and SENTENCE and to replace the two occurrences of imperative-statement with statementlist. The diagram was overly restrictive, allowing only one statement. However, in the manual’s informal text we learned, “A series of imperative statements can be specified whenever an imperative statement is allowed.” Our error-detection parser found these errors: first, the tool parsed code when it found NEXT SENTENCE, that is, without a dash. After inspecting the manual and grammar, we wrote a grammar transformation repairing this error. The error-detection parser also found that, according to the compiler, more than one statement was correct whereas the manual insisted on exactly one statement. We repaired this error with a grammar transformation.

Next, in a separate phase, we removed ambiguities. For example, the following fragment of a syntax diagram is present in the Cobol CALL statement: _____identifier__________________ |__ADDRESS__OF__identifier__| |__file-name________________|

This stack of three alternatives can lead to an ambiguity. Namely, both identifier and file-name eventually reduce to the same lexical category. So, when we parsed a CALL statement without an occurrence of ADDRESS OF, the parser reported an ambiguity because the other alternatives were both valid. Without using type information, we cannot separate identifier from file-name. This is the ambiguous extracted BNF fragment:

The more mainstream a language is, the more likely that you will have direct access to a reasonably good, debugged language reference.

(identifier | ”ADDRESS” ”OF” identifier | file-name)

With a grammar transformation, we eliminated the file-name alternative, resulting in (identifier | ”ADDRESS” ”OF” identifier)

The adapted grammar accepts the same language as before, but an ambiguity is gone. Note that this approach is much simpler than tweaking the parser and scanner to deal with types of names. In this way, we recovered the entire VS Cobol II grammar and tested it with all our Cobol code from earlier software renovation projects and code from colleagues who were curious about the project’s outcome. For the final test, we used about two million lines of pure VS Cobol II code. As in the PLEX case, we generated a fully Web-enabled version of both the corrected BNF and the syntax diagrams that could serve as the core for a complete and correct language reference manual.

A

part from PLEX and Cobol, we have recovered several other grammars, as have others. From our efforts in solving the 500LP, we learned two interesting lessons. First, the more uncomNovember/December 2001

IEEE SOFTWARE

87

mon a language is, the more likely that you will have direct access to the compiler’s source code, an excellent starting place for grammar recovery. Second, the more mainstream a language is, the more likely that you will have direct access to a reasonably good, debugged language reference, also an excellent source for grammar recovery.

Acknowledgments Thanks to Terry Bollinger, Prem Devanbu, Capers Jones, Tom McCabe, Harry Sneed, Ed Yourdon, and the reviewers for their substantial contributions.

For more information on this or any other computing topic, please visit our Digital Library at http://computer.org/publications/dlib.

References 1. C. Jones, Estimating Software Costs, McGraw-Hill, New York, 1998. 2. C. Jones, The Year 2000 Software Problem: Quantifying the Costs and Assessing the Consequences, Addison-Wesley, Reading, Mass., 1998. 3. G.M. Weinberg, The Psychology of Computer Programming, Van Nostrand Reinhold, New York, 1971. 4. P. de Jager, “You’ve Got To Be Kidding!” www.year2000. com/archive/NFkidding.html (current 20 Sept. 2001). 5. R. Lämmel and C. Verhoef, “Semi-automatic Grammar Recovery,” Software: Practice and Experience, vol. 31, no. 15, Dec. 2001, pp. 1395–1438; www.cs.vu.nl/~x/ ge/ge.pdf (current 20 Sept. 2001). 6. P.T. Devanbu, “GENOA—A Customizable, Front-End Retargetable Source Code Analysis Framework,” ACM Trans. Software Eng. and Methodology, vol. 8, no. 2, Apr. 1999, pp. 177–212. 7. B. Hall, “Year 2000 Tools and Services,” Symp./ITxpo 96, The IT Revolution Continues: Managing Diversity in the 21st Century, Gartner Group, Stamford, Conn., 1996. 8. N. Jones, Year 2000 Market Overview, tech. report, Gartner Group, Stamford, Conn., 1998. 9. M.G.J. van den Brand and E. Visser, “Generation of Formatters for Context-Free Languages,” ACM Trans. Software Eng. and Methodology, vol. 5, no. 1, Jan. 1996, pp. 1–41. 10. M.G.J. van den Brand, M.P.A. Sellink, and C. Verhoef, “Generation of Components for Software Renovation Factories from Context-Free Grammars,” Science of Computer Programming, vol. 36, nos. 2–3, Mar. 2000, pp. 209–266; www.cs.vu.nl/~x/scp/scp.html (current 20 Sept. 2001). 11. M.G.J. van den Brand, M.P.A. Sellink, and C. Verhoef, “Current Parsing Techniques in Software Renovation Considered Harmful,” Proc. 6th Int’l Workshop Program Comprehension, IEEE CS Press, Los Alamitos, Calif., 1998, pp. 108–117; www.cs.vu.nl/~x/ref/ref.html (current 20 Sept. 2001). 12. D. Blasband, “Parsing in a Hostile World,” Proc. 8th Working Conf. Reverse Eng., IEEE CS Press, Los Alamitos, Calif., 2001, pp. 291–300. 13. J. Brunekreef and B. Diertens, “Towards a UserControlled Software Renovation Factory,” Proc. 3rd European Conf. Maintenance and Reengineering, IEEE CS Press, Los Alamitos, Calif., 1999, pp. 83–90.

About the Authors Ralf Lämmel is a lecturer at the Free University of Amsterdam and is affiliated with the Dutch Center for Mathematics and Computer Science (CWI). His research interests include program transformation and programming languages. As a freelancer and consultant, he has designed, implemented, and deployed developer tools, migration tools, and software development application generators based on Cobol and relational databases. He received his PhD in computer science from the University of Rostock, Germany. Contact him at the Free Univ. of Amsterdam, De Boelelaan 1081-A, 1081 HV Amsterdam, Netherlands; [email protected]; www.cs.vu.nl/~ralf. Chris Verhoef is a computer science professor at the Free University of Amsterdam and principal external scientific advisor of the Deutsche Bank AG, New York. He is also affiliated with Carnegie Mellon University’s Software Engineering Institute and has consulted for hardware companies, telecommunications companies, financial enterprises, software renovation companies, and large service providers. He is an elected Executive Board member and vice chair of conferences of the IEEE Computer Society Technical Council on Software Engineering and a distinguished speaker of the IEEE Computer Society. Contact him at the Free Univ. of Amsterdam, Dept. of Mathematics and Computer Science, De Boelelaan 1081-A, 1081 HV Amsterdam, Netherlands; [email protected]; www.cs.vu.nl/~x.

88

IEEE SOFTWARE

November/December 2001

14. L. Wall, T. Christiansen, and R.L. Schwartz, Programming Perl, 2nd ed., O’Reilly & Associates, Cambridge, Mass., 1996. 15. M.P.A. Sellink, H.M. Sneed, and C. Verhoef, “Restructuring of Cobol/CICS Legacy Systems,” to be published in Science of Computer Programming; www.cs.vu.nl/~x/ res/res.html (current 20 Sept. 2001). 16. M.P.A. Sellink and C. Verhoef, “Generation of Software Renovation Factories from Compilers,” Proc. Int’l Conf. Software Maintenance, IEEE CS Press, Los Alamitos, Calif., 1999, pp. 245–255; www.cs.vu.nl/~x/ com/com.html (current 20 Sept. 2001). 17. M.P.A. Sellink and C. Verhoef, “Development, Assessment, and Reengineering of Language Descriptions,” Proc. 4th European Conf. Software Maintenance and Reengineering, IEEE CS Press, Los Alamitos, Calif., 2000, pp. 151–160; www.cs.vu.nl/~x/cale/cale.html (current 20 Sept. 2001). 18.

R. Lämmel and C. Verhoef, VS COBOL II Grammar Version 1.0.3, 1999; www.cs.vu.nl/grammars/vs-cobol-ii (current 20 Sept. 2001).