Evaluation and Performance Measurement of Research and Development: Techniques and Perspectives for Multi-level Analysis

Evaluation and Performance Measurement of Research and Development

Evaluation and Performance Measurement of Research and Development Techniques and Perspectives for Multi-Level Analysis

Vittorio Chiesa Full Professor of R&D Strategy and Organization, Politecnico di Milano, Italy

Federico Frattini Assistant Professor, Politecnico di Milano, Italy With contributions by: M. Calderini, G. Catalano, D. Chiaroni, A. Di Minin, A. Grandi, B.H. Hall, P. Landoni, V. Lazzarotti, R. Manzini, D. Moncalvo, S. Morricone, R. Oriani, A. Piccaluga, G. Scellato

Edward Elgar Cheltenham, UK • Northampton, MA, USA

© Vittorio Chiesa and Federico Frattini 2009 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical or photocopying, recording, or otherwise without the prior permission of the publisher. Published by Edward Elgar Publishing Limited The Lypiatts 15 Lansdown Road Cheltenham Glos GL50 2JA UK Edward Elgar Publishing, Inc. William Pratt House 9 Dewey Court Northampton Massachusetts 01060 USA

A catalogue record for this book is available from the British Library Library of Congress Control Number: 2009936222

ISBN 978 1 84720 948 1 Printed and bound by MPG Books Group, UK

Contents Abbreviations

vii

Introduction

1

PART I 1 2 3

FIRM

R&D function F. Frattini, V. Lazzarotti and R. Manzini R&D projects F. Frattini and D. Chiaroni R&D people A. Di Minin and A. Piccaluga

PART II 4 5

6 7

51 108

FINANCIAL MARKETS

R&D and financial investors A. Grandi, B.H. Hall and R. Oriani R&D information S. Morricone and R. Oriani

PART III

15

143 166

INNOVATION SYSTEM

Technology platform M. Calderini, D. Moncalvo and G. Scellato R&D policy G. Catalano and P. Landoni

References Index

189 218

255 293

v

Abbreviations ACARE

Advisory Council for Aeronautics Research in Europe

AHP

Analytical hierarchy process

ANP

Analytic network process

C&D

Connect and develop

CAPM

Capital asset pricing model

CAR

Cumulative abnormal returns

CRO

Contract research organization

DEA

Data envelopment analysis

DMADV Define, measure, analyse, design, verify DTA

Decision tree analysis

ECV

Expected commercial value

ETP

European technology platform

GAAP

Generally Accepted Accounting Principles

GSK

GlaxoSmithKline

HR

Human resources

HRM

Human resources management

IAS

International Accounting Standards

IFRS

International Financial Reporting Standards

IPR

Intellectual property rights

ITD

Integrated Technology Demonstrators

JTI

Joint Technology Initiative

JV

Joint venture

MBO

Management by objectives

NPD

New product development vii

viii

Evaluation and performance measurement of R&D

OI

Open innovation

PMS

Performance measurement system

R&D

Research and development

ROE

Return on equity

ROV

Real options valuation

S&T

Science and technology

SRA

Strategic Research Agenda

TOPSIS

Technique for order preference by similarity to ideal solution

WACC

Weighted average cost of capital

Introduction The subject of this book is research and development (R&D) evaluation and performance measurement. For the largest part of the twentieth century, establishing effective forms of control for R&D undertaken at either industrial or governmental level was considered a very challenging task. In order to understand the reasons underlying this unfavourable attitude toward R&D performance management, it is necessary to clarify first the scope and purposes of R&D. An established taxonomy of the activities included in the R&D concept distinguishes between basic research, applied research and new product development (NPD). Basic research has the purpose to produce new knowledge about the principles underlying natural and social phenomena, without any direct relationships with industrial applications (for example, new products, services or industrial processes). Applied research is aimed at the generation of new knowledge required to fulfil explicit needs and to enable industrial applications. Finally, NPD includes a number of heterogeneous tasks (that is, design, prototyping, testing, engineering, installation, after-sales services) that are necessary to apply existing bodies of knowledge to the development of new products or services.

CHALLENGES IN R&D PERFORMANCE MEASUREMENT AND EVALUATION From this definition it clearly follows that the most important outcome of basic and applied research is represented by new knowledge, new technologies, new bodies of competencies, that have intrinsically an intangible nature. An objective system for the measurement and evaluation of these intangible results is therefore very hard to establish, because no appropriate quantitative indicators can be designed and applied. On the contrary, NPD does produce tangible results, in the form of new products and services that are sold on the market, generate revenues and hence can be associated with quantitative measures of performance. However, the outcome of NPD becomes manifest a long time after design and engineering activities are carried out. The NPD process for a new drug might take for instance 10 years to be completed. As a result, evaluating design and engineering 1

2

Evaluation and performance measurement of R&D

performance measuring their tangible outcomes is practically unfeasible. Input measures for both basic and applied research and NPD, which use as proxy of R&D performance the resources available to undertake them (in terms of R&D expenses or number of R&D employees), could mitigate this problem. However, this form of control is largely inappropriate, as R&D activities are often serendipitous and their performance radically influenced by the experience and creativity of R&D professionals, which is something that is not mirrored in input indicators. Differently put, a larger availability of higher level resources does not necessarily lead to superior performance in R&D. Last but not least, R&D is fundamentally dependent on the creative and innovative behaviour of researchers, scientists and engineers. Any form of control, be it input-, output-, or process-based, can potentially thwart this behaviour and hence undermine a critical success factor in R&D. These are the reasons behind the limited attention devoted by both academics and practitioners to R&D evaluation and performance measurement for most of the twentieth century.

A RENEWED INTEREST IN R&D PERFORMANCE MEASUREMENT AND EVALUATION However, during the 1980s and 1990s some changes have occurred in the economic and social environment that have stimulated managers’, policy makers’ and researchers’ attention to the issue of R&D performance measurement and have encouraged the development of new approaches able to overcome, or at least minimize, the above mentioned barriers to an effective evaluation. First, technology has been advancing more and more rapidly, as new knowledge has been developed and applied to products and services faster and faster (Bayus, 1994; Wind and Mahajan, 1997). As a consequence, life-cycles have reduced in several product categories (Nevens et al., 1990), a higher number of new products and services have been introduced over time, and the distance between subsequent innovations has decreased (Bayus, 1998). In parallel, markets have become more and more turbulent and dynamic: customer needs, competitors, business models and the set of competencies necessary to compete in a definite industry, are nowadays changing over time with an unprecedented frequency (Mohr et al., 2005). Moreover, globalization, liberalization and convergence of markets and technologies have increased competition in several industries, both at a domestic and at a global level (Gupta and Wilemon, 1996). As a result of these changes, firms and countries have increasingly

Introduction

3

350 300 250

Europe United States Japan

200 150 100 50

1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006

0

Source: Adapted from National Science Foundation (2008). Data on European research and development include data related to Germany, France, United Kingdom and Italy.

Figure I.1

Research and development investments (US$ billions)

looked at the capability to develop continuously both incremental and radical new products, services and technologies, and therefore at excellence in R&D activities, as critical determinants of their competitiveness. A knowledge-based society, where wealth is mainly dependent on the ownership or control of critical competencies and technologies, has emerged and replaced an industrial paradigm fundamentally associated with a machine-based model of productivity. At the same time, R&D activities have become increasingly costly. The cost of developing a new drug, for example, is nowadays well over $800 million, more than ten times the level it reached in the mid 1990s. This is partially the result of the so-called technology fusion phenomenon (Kodama, 1995), whereby radically new products and services are increasingly the result of the merging and integration of competencies traditionally belonging to distinct disciplines, that therefore need to be mastered or anyhow acquired to achieve excellence in R&D. The growing magnitude of R&D expenses in the last two decades is corroborated by the data reported in Figure I.1, which shows the uninterrupted growth of total R&D investments in Europe, the United States and Japan, measured in absolute terms. Within the context described above, the last two decades have witnessed an exponential growth in practitioners’ attention toward R&D performance and, as a result, toward its measurement and control. In particular, scholars in management and economics have started addressing the

4

Evaluation and performance measurement of R&D

problem of R&D evaluation under different perspectives and employing different units of analysis: 1. Firm A first line of research has adopted the point of view of a firm’s executives and R&D managers and has looked into the problem of measuring R&D’s contribution to economic value creation and competitive advantage. Some examples of the issues addressed at this level of analysis are: the design and implementation of a performance measurement system (PMS) to be applied to the firm’s R&D units and laboratories (Kerssensvan Drongelen and Cook, 1997); the development and use of appropriate techniques for the evaluation of R&D projects; (Cooper et al., 2001; Poh et al., 2001); and the introduction of performance assessment and rewarding systems for R&D professionals (Farris and Cordero, 2002). 2. Financial markets A second stream of research has instead adopted the standpoint of financial markets and has studied the relationships between traded firms’ R&D investments and their market value. (Chan et al., 1990; Munari et al., 2005). An important topic that has received special attention within this body of research encompasses the anatomy and effects of the flow of information about R&D investments and the information asymmetries between companies and financial investors, which are mediated by the information disclosing rules that accounting standards establish (Hand, 2001, 2003b). 3. Innovation system A last line of research has adopted a broader perspective, studying the problem of measuring R&D at the level of national or regional innovation systems, whereby a national innovation system can be defined as ‘the network of institutions in the public and private sectors whose activities and interactions initiate, import, modify and diffuse new technologies’ (Freeman, 1987, p. 1). Within this vast stream of research, scholars have investigated some important issues such as the use of bibliometric, technometric and patent-based indicators to evaluate R&D carried out within a national innovation system and the assessment of R&D policies’ effectiveness (Feller, 2002; Foray, 2004).

GAPS IN THE EXISTING LITERATURE Despite the attention devoted to the measurement of R&D performance in the last two decades, a comprehensive picture of the techniques and approaches developed and applied by researchers and practitioners at the different levels of analysis (firm, financial markets and innovation system)

Introduction

5

has not been provided yet, and a number of issues still need to be fully unearthed. As a result, extant literature is highly fragmentary, and further research is needed to study how the different methods and perspectives could be integrated into a unified framework of analysis. What is more, a number of changes have recently occurred that have a deep impact on the approaches that managers and policy makers use to assess R&D performance. In particular, the management and organization of R&D activities have undergone a significant evolution as a result of (see also Chiesa and Frattini, 2008): 1. Increased reliance upon external sources of technology Firms often lack the technical and financial resources needed to develop the whole set of competencies required for effective R&D and innovation. As a result, they increasingly concentrate and specialize their internal R&D efforts on those core activities where they are more likely to excel. At the same time, they learn to leverage external sources of technology (for example, universities, competitors, firms from other industries, individuals) to acquire the missing competencies and to continuously feed their innovation pipeline (Chatterji, 1996; Howells, 1999; Roberts, 2001). 2. Leverage multiple channels for technology exploitation In the past, companies have generally incorporated the results of their R&D activities into new or improved products and services to be internally developed and launched in the final market. However, the high costs required for technology development and the soaring rate at which new knowledge is produced, make long term competitive advantage increasingly dependent on a firm’s capability to continuously and fully leverage its technology basis. As a result, firms are contemporarily exploiting multiple paths for converting their technologies into revenues, among which external exploitation approaches (for example, licensing out, new venture spin-off, joint ventures, contract research) are increasingly diffused (Haour, 2004; Mohr et al., 2005). 3. Entrepreneurial nature of R&D R&D has been traditionally conceived as a part of the firm’s overhead costs (Ortt and Smits, 2006) and administered as a technology-focused function where the firm’s body of knowledge was generated, protected and transformed into new or improved products and services. Nowadays, internal R&D is still the repository of the firm’s core technological competencies but, at the same time, it has become the engine of the innovation process and undertakes a number of critical business-related, brokering activities, for example the scouting of the external environment for the identification of valuable sources of knowledge, the

6

Evaluation and performance measurement of R&D

integration of internally generated with externally acquired technologies, and the pursuit of external exploitation opportunities for internal technologies. A value-based approach, where R&D is responsible for the business results it delivers, has therefore developed and diffused (Ortt and Smits, 2006). 4. Birth and growth of markets for technology The search for multiple channels for exploiting a firm’s competencies and technologies, the specialization in knowledge development and the associated division of labour in innovative activities, have contributed to the development of the so-called markets for technology (Arora et al., 2001). The capability to interact with the players on these markets, where knowledge disembodied from physical artefacts is exchanged, has become a critical capability for R&D organizations (Jones et al., 2000; Muller and Zenker, 2001). 5. Management of R&D on an international scale Finally, the management of R&D has assumed an international dimension. A number of pieces of research on the internationalization of innovation activities clearly show that foreign R&D is becoming a significant component of many countries’ R&D base (Jones and Teegen, 2002; United Nations, 2005). These changes in the management of R&D have been captured and systematized in the well-known paradigm of industrial innovation called ‘open innovation’ (Chesbrough, 2003a). Under this new approach, the role of R&D and the approaches used to administer it have become radically different from those employed in traditional innovations models. This has relevant implications also for the evaluation and performance measurement of R&D, at the firm-, financial markets-, and innovation system-level from which the issue can be addressed and studied.

CONTENT AND STRUCTURE OF THE BOOK That said, this book has two main purposes: 1.

2.

Provide a systematic and updated overview of the existing literature on R&D evaluation and performance measurement organized around the three levels of analysis (firm, financial markets, innovation system) presented above in this introduction; Describe the results of the research project carried out by the authors and the other contributors of the book in the last three years to address

Introduction

7

some important gaps in the existing research on R&D evaluation and performance measurement and, more importantly, to investigate the impact that the recent changes in the management and organization of R&D activities is having on the methods and approaches used to assess its performance. In doing so, the book develops and illustrates a comprehensive, integrated and multi-level framework that represents an original synthesis of the most recent research on the techniques and perspectives that can be used to evaluate R&D and to measure its performance at the firm-, financial markets-, and innovation system-level. The book is targeted mainly at an academic audience, made up of scholars and Ph.D. students in management and economics, whose research interests are related to R&D, technological innovation, performance measurement and evaluation. Therefore, particular attention has been devoted to reviewing and commenting on the most important literature contributions for each level of analysis, and to describing the methodological details of the empirical studies carried out by the authors and the other contributors to fill specific gaps in the existing literature. These details have been presented in the belief that they will improve the value of the book for its intended audience, who will hopefully find its framework a valuable starting point for future research into the complex and challenging topic of R&D performance measurement and evaluation. The book is organized into three parts that correspond to the three perspectives mentioned above in this introduction, under which the problem of R&D performance measurement and evaluation can be analysed and studied. Part I (Firm) addresses R&D performance measurement from the point of view of a firm’s executives and R&D managers who are interested in assessing the contribution of R&D to economic value creation, in monitoring R&D activities’ efficiency and effectiveness and in evaluating R&D professionals’ performance to better motivate, reward and retain them. Part I has three chapters, each adopting a specific unit of analysis: 1.

Chapter 1 (R&D function) focuses on the R&D function of an industrial firm and investigates the techniques that can be used to measure its performance. The first part of the chapter provides a detailed literature overview of the methods that can be used with this purpose. The second part focuses on a topic that has been relatively overlooked by extant research, namely the design of an integrated performance measurement system (or PMS) for the firm’s R&D function. Although a number of contributions have looked into specific aspects of this problem (for example, which indicators or metrics are best suited to

8

2.

3.

Evaluation and performance measurement of R&D

the characteristics of R&D activities, which is the optimal measurement frequency in R&D settings), very few of them have addressed it in a comprehensive manner. Relying on the results of a three-year empirical study, a reference framework for the design of a PMS for the R&D function is developed and illustrated. The chapter has two appendixes. The first one describes the application of the framework to a highly innovative biotech firm. The second one reports a detailed description of the empirical analysis that has been undertaken to develop the above mentioned reference framework. Chapter 2 (R&D projects) adopts the single R&D project and the firm’s portfolio of R&D projects as units of analysis. The chapter first focuses on the techniques that can be employed for the evaluation of single R&D projects. A taxonomy of the main classes of techniques that can be used with this purpose is illustrated and developed through a systematic review of the relevant academic and practitioners’ literature, and through a focus group where a number of R&D managers, from some of the largest and most innovative Italian firms, were invited to comment on the subject. In particular, the chapter discusses the contexts (defined, for example, in terms of type of R&D project being undertaken) in which each evaluation technique might be more appropriate. This represents an important contribution to the existing research on the topic, which has dedicated limited attention so far to looking into the fields of application for which different classes of R&D project evaluation approaches are most suitable. The second part of the chapter focuses on the firm’s portfolio of R&D projects to investigate the techniques and the approaches that can be employed to evaluate its adequacy and the extent to which it is consistent with the firm’s R&D and innovation strategy. An appendix to the chapter reports and discusses the case of a multinational firm that uses a very formalized process for the evaluation of its R&D projects, representing an original approach through which the techniques presented above can be combined to provide a more accurate evaluation. The case study allows reflection and comment on the organizational and managerial problems that may arise in the implementation and practical use of an evaluation technique. Chapter 3 (R&D people) focuses on the human resources management (HRM) practices adopted by industrial firms in their R&D laboratories. The chapter first reviews the existing literature about HRM practices used in R&D settings. Particular emphasis is given here to understanding how performance measurement and evaluation systems might contribute to improving the different areas around which HRM for R&D people can be organized and studied.

Introduction

9

The chapter then develops a framework that can be used to analyse and evaluate the degree to which the HRM system for R&D professionals adopted by an industrial firm is aligned with its competitive strategy and environment. This model is then used to support an extensive empirical analysis, which has involved a number of leading Italian firms and investigates the micro-organizational effect of Open Innovation on the management and evaluation of R&D professionals. An appendix to the chapter provides some methodological details about the empirical analysis that will hopefully inform future research into the matter. Part II (Financial Markets) adopts instead the whole firm as the unit of analysis and studies R&D evaluation from the point of view of financial investors, who are interested in estimating the impact of firm-level R&D investments on the value of the traded company. Part II has two chapters: 1.

2.

Chapter 4 (R&D and financial investors) focuses on the relationship between R&D investments and the market value of traded firms, explaining why and how R&D investments should be reflected into financial investors’ valuations and stock market prices. The chapter first reviews the theoretical and empirical foundations of the relationship between R&D investments and market value. Afterwards, it illustrates and discusses the two main classes of empirical models that can be used to analyse this relationship: (a) models that relate the flow or the stock of R&D investments to the market value of the firm (often measured relative to tangible assets, that is, as Tobin’s Q) at a given moment in time; (b) models that relate the arrival of new information on R&D investments (R&D announcements) with changes in the stock price (stock returns). Finally, the empirical results obtained from the application of the models are described answering to three main questions: whether R&D investments create value, how investors deal with uncertainty, and how different financial markets and ownership structures affect the market valuation of firms’ R&D investments. Chapter 5 (R&D information) deals with the possible R&D information problems arising between traded firms’ managers and outside investors, analysing in detail how information flows from companies to investors and the implications for their market value. Three important aspects are addressed in this chapter. First, it explains why and how R&D investments generate information asymmetries between

10

Evaluation and performance measurement of R&D

insiders (managers) and outsiders (investors). Second, it analyses the value relevance of R&D information for investors and stock prices. Third, it focuses on the role of financial analysts, as they normally complement public R&D information with private information supporting investors’ decisions. These questions are also examined in light of the changing international accounting standards and in particular the adoption of the International Accounting Standards (IAS/ IFRS) by European traded firms in 2005. In this respect, the chapter reports some descriptive evidence on the effect that the application of IAS/IFRS standards has on R&D reporting for a sample of companies publicly traded in Italy. A first appendix to the chapter illustrates the models that can be used in contemporaneous and intertemporal studies to examine the relationship between R&D information disclosure and stock price. A second appendix provides methodological details about the empirical investigation. Finally, Part III (Innovation System) broadens the scope of the analysis to focus on the evaluation of R&D activities undertaken by networks of institutions in the public or private sector and the initiatives promoted by governmental bodies to stimulate R&D activities and streamline the production and diffusion of new technologies in an innovation system. It focuses in particular on two topics: 1.

Chapter 6 (Technology platform) deals with the analysis and evaluation of technology platforms. Technology platforms are initiatives recently promoted by the European Commission that bring together a number of important stakeholders with the aim of defining long-term R&D projects and technological development objectives in areas that are critical for Europe’s future growth. They represent therefore an important means for vertical and horizontal coordination of R&D activities at the European level, although they have been dedicated scant attention so far in terms of assessment and evaluation of their performance. The chapter first reviews the theoretical rationale underlying the need for a systemic, inter-organizational view of knowledge production and dissemination to properly interpret differentials in firms’ innovation capabilities. This explains the system-level implications of the Open Innovation paradigm. Next, the chapter illustrates the concept of technology platforms, explaining how they are created and administered at the European level, making explicit reference to a real-word case, the ACARE (Advisory Council for Aeronautics Research in Europe) European Technology Platform. Finally, the chapter develops and illustrates a framework for the assessment of a

Introduction

11

technology platform that uses a process-based evaluation approach and distinguishes between a macro- and a micro-level at which the platform’s proficiency can be diagnosed. Chapter 7 (R&D policy) adopts as unit of analysis R&D policies, to provide a systematic picture of the methods that can be employed to evaluate their performance. The chapter reports first a detailed analysis of the theoretical reasons underlying the need for R&D policy intervention in modern economies. Next, it systematically reviews the literature contributions that have addressed so far the problem of why and how the effectiveness and efficiency of R&D policies should be measured and assessed. This analysis is used to develop an original taxonomy of the approaches that can be employed in practice to undertake this kind of evaluation. The taxonomy is based on two dimensions: evaluation methodologies and evaluation typologies. The first dimension distinguishes between qualitative (benchmarking and case study) and quantitative (evaluation of measurable effects and evaluation of detectable effects) methods. The latter classifies policies on the basis of their target (that is, firms, knowledge generating institutions or networking) and instrument (financial or legislative). In the second part of the chapter, this original taxonomy is used to support a thorough analysis of the empirical studies available in the literature that have applied the above mentioned methodologies to tackle real-world evaluation problems.

2.

The research project whose results are summarized in this book has been funded through a grant (FIRB-RBNE037AWA) awarded to the authors by MIUR, the Italian Ministry for University and Research, and has involved a number of Italian universities. The authors are grateful to all the researchers who have participated in the project and contributed to this book: ● ● ● ● ● ● ● ●

Mario Calderini from Politecnico di Torino (Chapter 6) Giuseppe Catalano from Politecnico di Milano (Chapter 7) Davide Chiaroni from Politecnico di Milano (Chapter 2) Alberto Di Minin from Scuola Superiore Sant’Anna (Chapter 3) Alessandro Grandi from Università degli Studi di Bologna (Chapter 4) Bronwyn H. Hall from University of California – Berkeley (Chapter 4) Paolo Landoni from Politecnico di Milano (Chapter 7) Valentina Lazzarotti from Università Carlo Cattaneo – LIUC (Chapter 1)

12

Evaluation and performance measurement of R&D ● ● ● ● ● ●

Raffaella Manzini from Università Carlo Cattaneo – LIUC (Chapter 1) Dario Moncalvo from Politecnico di Torino (Chapter 6) Serena Morricone from Università degli Studi di Bologna (Chapter 5) Raffaele Oriani from Luiss Guido Carli and Università degli Studi di Bologna (Chapters 4 and 5) Andrea Piccaluga from Scuola Superiore Sant’Anna (Chapter 3) Giuseppe Scellato from Politecnico di Torino (Chapter 6)

The assistance of Alberto Cavaliere with copy editing and the development of the case study discussed in Chapter 2 is gratefully acknowledged. The authors are also grateful to Nicola Dotti for his support with the analysis of the literature reported in Chapter 7 and for his help with the third and fourth sections of the same chapter.

PART I

Firm

1. 1.1

R&D function INTRODUCTION

This chapter examines a first dimension along which measurement of R&D performance can be undertaken and studied, that is, the R&D function of an industrial firm. This issue deserves particular attention because assessing performance and contribution to economic value of R&D has become an important concern for R&D managers in particular in the last decades (Kerssens-van Drongelen and Bilderbeek, 1999), as a result of the evolutionary trend discussed in the introduction of this book. Despite the fact that measuring R&D performance is a very challenging task, with potential negative impacts on the creativeness of scientists and engineers (Brown and Svenson, 1988), the topic has currently received large attention in R&D and innovation management literature, and it is raising the interest of managers and practitioners as well (Pappas and Remer, 1985; Brown and Svenson, 1988; Sivathanu and Srinivasa, 1996; Werner and Souder, 1997; Hauser, 1998; Driva and Pawar, 1999; Driva et al., 2000; Poh et al., 2001; Loch and Tapper, 2002; Godener and Soderquist, 2004; Ojanen and Vuola, 2006). The growing interest in the measurement of a firm’s R&D function performance has stimulated reflections about the tools that can be effectively employed by R&D and senior managers to this end and, as a result, a huge bulk of research has flourished around the issue. The first purpose of this chapter is to provide an updated overview of this vast literature that will be useful for R&D and innovation management scholars interested in the topic and will hopefully inform their future research. This overview is presented and discussed in the second section of the chapter. Next, the chapter focuses on a topic that has been relatively overlooked by extant research on R&D performance measurement, namely the design of a performance measurement system (or PMS) for the firm’s R&D function. Although a number of contributions have looked into specific aspects of this problem (for example, which indicators or metrics are best suited to the characteristics of R&D activities, which is the optimal measurement frequency in R&D settings), very few of them have addressed it in a comprehensive manner. Relying on the results of a three-year empirical investigation (described in Appendix A1.2) and an extensive literature analysis, the chapter develops 15

16

Evaluation and performance measurement of R&D

and discusses, in its third section, a reference framework for the design of a PMS for the R&D function. This model is grounded in both R&D and innovation management literature and management accounting research, and is especially useful for scholars who are interested in contributing to extant research on the approaches firms employ to measure the performance of their R&D function, for example through large-scale empirical analyses. Nevertheless, the framework may be of interest to R&D managers as well, who are provided with a template that can serve as a guide for their decisions about R&D performance measurement. For the purpose of this chapter, a distinction should be made between the concepts of R&D function and R&D unit (Chiesa, 2001). A firm’s R&D function is the set of organizational units devoted to R&D activities (that is, R&D units). In small or low-tech firms, it often happens that the R&D function is made of a single organizational unit, typically engaged in new product development (NPD). In large and technology-intensive firms, on the other hand, the R&D function generally includes a number of units that can be devoted to different phases of the R&D process (for example, basic, applied research and NPD), positioned at different levels of the organizational structure (for example, corporate and divisional), and operating in different countries. Therefore, although addressing the problem of designing the PMS for the firm’s R&D function, in this chapter we adopt as unit of analysis the single R&D unit, because it might entail specific design choices that are radically different from those required in the other units. The overall PMS for the firm’s R&D function will be made therefore of the set of PMSs employed in the different R&D units. As already mentioned, the chapter has two main sections. The first one provides an overview of the literature about the measurement of a firm’s R&D function performance, organized around the different levels of analysis it has adopted. The second section develops and discusses the reference framework for the design of a PMS for the firm’s R&D function, representing the main outcome of the chapter. After a concluding section, Appendix A1.1 illustrates the case of Company A (the real name has been blinded for confidentiality reasons). Appendix A1.2 reports a detailed description of the empirical analysis that has been undertaken to develop the above mentioned reference framework.

1.2

LITERATURE REVIEW

Research has addressed the problem of measuring the performance of an industrial firm’s R&D function at four different levels, as shown in Figure 1.1.

R&D function

17

Performance measurement systems for R&D within the firm’s internal and external context

Performance measurement systems for R&D

Dimensions for R&D performance measurement

Indicators and metrics for R&D performance measurement

Figure 1.1

Synthesis of research on performance measurement of a firm’s R&D function

At a first level, scholars have studied how indicators (or metrics) to measure R&D performance should be selected. Brown and Svenson (1988) suggest that firms should use a limited number of objective and external indicators to measure R&D performance, focused on results and outcomes rather than behaviour. Nixon (1998) underlines the importance of ensuring a strategic orientation in the selection of the indicators for R&D performance measurement. In the author’s opinion, these metrics should mirror the firm’s critical success factor, they should be easy to understand and use and capable of encouraging change in behaviour. Several authors (Werner and Souder, 1997; Driva and Pawar, 1999; Bremser and Barsky, 2004) state that the most effective measurement approaches for R&D are those that balance quantitative with qualitative metrics. Similarly, Presley and Liles (2000) distinguish the quantitative indicators into financial and non financial, whereas the qualitative ones measure something that is not possible to express with a number (for example, a firm’s market reputation,

18

Evaluation and performance measurement of R&D

which can be judged as ‘improved’, ‘stable’ or ‘worsen’). Beside qualitative and quantitative indicators, Pappas and Remer (1985) had already introduced a further intermediate category, labelled ‘semi-quantitative’. It comprises those numeric metrics based on the personal judgment of an expert, whose subjective evaluation is however translated into a numeric score through alternative techniques (for example, Gee, 1972; Collier, 1977). Similarly, Werner and Souder (1997) synthesize the different types of indicators used in R&D into three categories: 1.

2.

3.

Quantitative objective indicators (numeric metrics obtained from the application of a definite algorithm that produces the evaluation independently from the person responsible for the measurement); Quantitative subjective indicators (numeric metrics based on the personal judgment of an expert, whose subjective evaluation is however translated into a numeric score. This corresponds to the ‘semi-quantitative’ category advanced by Pappas and Remer); Qualitative subjective metrics (not expressed numerically, but through the personal judgment of the evaluator).

Among the quantitative indicators, Kerssens-van Drongelen and Cook (1997) underline the importance of bibliometric approaches, which have been analysed by a vast stream of research in the science and technology literature. This line of research has developed indeed a broad array of indicators for measuring scientific and technological research activity over the last decades (European Commission, 1997). Two major groups of output measures have been put forward: 1.

2.

Bibliometric (Verbeek et al., 2002), which rely on the information contained in research publications in order to evaluate an actor’s scientific output; Technometric (Debackere et al., 2002), which are based on patent data.

The most essential bibliometric indicators are based on the number of articles published by the unit of analysis being studied; however, appropriate measures have been developed in order to evaluate the ‘quality’, the ‘impact’ and the ‘importance’ of the publications (Martin, 1996), to distinguish between ‘short-term’ and ‘long-term’ impact (Moed et al., 1985), and to compare ‘actual’ and ‘expected’ impact of the publication (Noyons et al., 1998). The bibliometric methods were initially developed with the overarching objective of evaluating public and industrial research at a high level of aggregation, that is in policy-level applications aimed at investigating and comparing research capabilities under a national or

R&D function

19

regional standpoint. The growing interest in the management of industrial R&D and, in particular, its measurement and evaluation, has led to them being used for measuring the performance of a firm’s R&D function as well. When applied with this purpose, they show important shortcomings that have significantly hampered their diffusion among practitioners. First of all, they are difficult and expensive to calculate, since they need a large amount of information that is costly to gather and elaborate. Moreover, they are not useful for measuring short-term performance; in the case of a small firm operating in the field of pharmaceutical research, for instance, it would be a great result to obtain one or two patents in ten years. Furthermore, they do not allow for an adequate measurement of two R&D performance dimensions, namely costs and time, which are critical in today’s competitive environments. Another well known taxonomy of R&D performance metrics conceives R&D as a business process (Brown and Svenson, 1988; Cooper, 1993), and suggests its performance should be measured on the basis of four classes of indicators: 1.

2.

3.

4.

Input. This means using the quantity and quality of the inputs dedicated to the operation as a proxy of its performance. Examples of input indicators are the quantity and quality of current expenses, investments, human resources, technologies; Process. This means analysing the activities in which the R&D function is involved. Typical processes are concept generation, product development, project selection, and technology acquisition, whereas characteristic indicators are the average product life cycle length, the average time of redesign, and the percentage of innovation projects that do not meet established schedules, target costs or ‘standard of professionalism’, which refer to the capability to create professional documentations and establish successful cooperation with partners; Output. This means monitoring R&D in terms of the results that it achieves, for example patents, scientific publications, completed projects, new products developed; Outcomes. This measures the accomplishments of R&D that have value for the organization. Examples are average cost reductions, percentage of sales from new products, or degree of product improvements.

Overall, it emerges that extant research has put forward a number of different taxonomies and categories of indicators to be used in R&D performance measurement (see Table 1.1). Moreover, it has claimed the need to use a mix of qualitative and quantitative metrics in R&D, the former best suited to capture unmeasurable aspects, and the latter capable of reducing

20

Table 1.1

Evaluation and performance measurement of R&D

Classification of R&D performance indicators

Taxonomy/categories of indicators

References

Qualitative and quantitative

Werner and Souder (1997); Driva and Pawar (1999); Bremser and Barsky (2004)

Input, process, output and outcomes; focusing on results and focusing on behaviour; objective–external and subjective–internal

Brown and Svenson (1988)

Input, output and financial; bibliometric and technometric

Coccia (2001)

Technometric

Debackere et al. (2002)

Financial and non-financial; ex post (lagging) and ex ante (leading)

Nixon (1998); Bourne et al. (2000); Bremser and Barsky (2004); Anderson and McAdam (2005)

Qualitative, quantitative and semiquantitative

Pappas and Remer (1985)

Qualitative, quantitative and financial

Presley and Liles (2000)

Bibliometric

Verbeek et al. (2002)

Quantitative objective, quantitative subjective, and qualitative subjective

Werner and Souder (1997)

the subjectivity of the evaluation (Pappas and Remer, 1985; Werner and Souder, 1997; Driva et al., 2000). Furthermore, given that economicfinancial indicators are often questionable since it is very difficult to give a monetary evaluation of intangible and distant-in-time elements, as typically happens in R&D (Frattini et al., 2006), they are often integrated by non-financial indicators, which can be more easily estimated. At a second level, researchers have investigated the dimensions, or perspectives, along which R&D performance measurement should be carried out. Performance dimensions are evaluation criteria that identify those factors whose accomplishment is critical for success in R&D. They stand therefore at a higher conceptual level than indicators: the latter are operative metrics needed to translate the former into practice. Literature indicates that firms measure R&D functions’ performance along a very heterogeneous array of performance dimensions. Driva and Pawar (1999) show that R&D performance measurement should be organised around

R&D function

21

the following dimensions: time, costs, quality and flexibility. Similarly, Kim and Oh (2002) describe the following classes of perspectives for R&D performance measurement: market-oriented, R&D project-specific, R&D researcher-specific. Twiss (1986) suggests a checklist for project evaluation that is applied by Chiesa and Masella (1996) to R&D by translating Twiss’s evaluation criteria into three dimensions: technical success, efficiency and integration of R&D with the company’s other functions. Similarly, Griffin and Page (1996) focus their attention to the project level and categorize performance into customer-based, financial, and technical. Davila (2000) analyses the use of cost, time and customer (or market) information in performance measurement of NPD activities. Within this stream of research, a number of scholars suggest applying the Balanced Scorecard (BSC) approach (Kaplan and Norton, 1992, 1993, 1996) to R&D. Kerssens-van Drongelen and Cook (1997) illustrate the way to develop a PMS for R&D that allows to implement the firm’s R&D and competitive strategy through the integration of financial, client, internal business, innovation and learning perspectives. Other researchers analyse how the BSC perspectives in R&D could be used to pursue specific objectives. For instance, Bremser and Barsky (2004) describe how it could be possible to integrate the BSC approach with the stage-gate system (Cooper, 1993) for organizing innovation activities. Similarly, Sandstrom and Toivanen (2002) show that the BSC could be effectively used to align the incentive system and the behaviour of engineers working in an NPD department with the firm’s overall strategy. Lee and Lai (2007) analyse the relative importance that firms attach to the different BSC perspectives in the measurement of knowledge management (KM) performance in high technology industries. A synthetic view of the different classes of performance dimensions used in R&D performance measurement and put forth by extant research is reported in Table 1.2. The third level in the study of R&D performance measurement (see Figure 1.1) is characterized by the adoption of a ‘systemic’ perspective. The contributions belonging to this stream of research suggest that measuring R&D performance is not simply a matter of selecting appropriate dimensions and indicators. These choices need to be consistent, for example, with the frequency with which measurement is undertaken, or with the control objects whose performance is evaluated (individuals, project teams, R&D departments). They represent the constitutive elements of a performance measurement system (or PMS) for the R&D function, that should be designed taking into proper account both the characteristics of each single constitutive element, and the relationships that link them. Kerssens-van Drongelen and colleagues (1997, 1999, 2000) conceive, for example, the PMS for R&D as comprising the following integrated

22

Table 1.2

Evaluation and performance measurement of R&D

Classification of R&D performance dimensions

Taxonomy/categories of performance dimensions

References

Financial, customer, internal business and innovation and learning

Kerssens-van Drongelen and Cook (1997); Kerssens-van Drongelen and Bilderbeek (1999); Kerssens-van Drongelen et al. (2000); Sandstrom and Toivanen (2002); Bremser and Barsky (2004); Lee and Lai (2007);

Cost, quality and volumes

Brown and Svenson (1988)

Effectiveness, efficiency, value creation and time

Chiesa and Frattini (2007)

Technical success, efficiency and integration of R&D with other functions

Chiesa and Masella (1996)

Goal achievement and project effect

Cho and Lee (2005)

Financial, technological and scientific

Coccia (2004)

Customer-based, financial and technical

Griffin and Page (1996)

Financial and customer

Hultink and Robben (1995)

Market-oriented, R&D projectspecific, R&D researcher-specific

Kim and Oh (2002)

R&D effectiveness, recognition, user-oriented effectiveness and administrative effectiveness

Mohapatra et al. (2003)

Time, cost, quality and flexibility

Driva and Pawar (1999)

elements: metrics organized into a consistent structure (that is, metrics assigned to the different organizational levels whose performance should be measured), standards to measure performance against, frequency and timing of measurement, and format for information reporting. Similarly, Ojanen and Vuola (2006) suggest that internal consistency should be guaranteed among measurement perspectives, objectives, control objects and measurement process in an effective PMS for R&D. As far as the objectives for R&D performance measurement are concerned, Kerssens-van Drongelen and Cook (1997) suggest that motivating

R&D function

23

scientists and researchers and diagnosing activities and processes are the two main underlying reasons for R&D performance measurement. Loch and Tapper (2002) identify the following foremost objectives for which firms control their R&D performance: align behaviour and set up priorities, evaluate and reward researchers, establish an operative control and stimulate learning and improvement. Godener and Soderquist (2004) add that performance measurement, especially in complex NPD projects, can serve to favour communication and coordination among top managers, middle managers and researchers. Other objectives firms aim to pursue through performance measurement can be: diagnosing activity for supporting resource allocation (Loch et al., 1996; Kerssens-van Drongelen and Bilderbeek, 1999; Pearson et al., 2000; Bremser and Barsky, 2004); motivating personnel (Bowon and Heungshik, 2002; Kerssens van-Drongelen and Cook, 1997); enhancing communication and coordination (Szakonyi, 1995; Loch et al., 1996; Driva et al., 2000; Bremser and Barsky, 2004; Godener and Soderquist, 2004); stimulating learning (Driva et al., 2000; Loch and Tapper, 2002); and reducing uncertainty (Chiesa and Masella, 1996; Kerssens-van Drongelen and Cook, 1997; Davila, 2000). Research has also looked into the organizational levels at which performance is separately measured, labelled as ‘control objects’ or ‘levels of R&D performance’ (see Ojanen and Vuola, 2006, for a detailed analysis of this topic). Basically, R&D performance measurement can be conducted at the level of: the R&D function as a whole or the different units it is made of (Kerssens-van Drongelen and Bilderbeek, 1999; Cho and Lee, 2005); the functional departments within each R&D unit (Kerssens-van Drongelen et al., 2000; Bremser and Barsky, 2004); the project teams that are active in the R&D unit (Griffin and Page, 1993 and 1996; Chiesa and Masella, 1996; Kim and Oh, 2002; Loch and Tapper, 2002); or the individual scientists and engineers working in the R&D unit (Kerssens-van Drongelen et al., 2000; Kim and Oh, 2002; Cho and Lee, 2005). Another important constituency of a PMS for R&D functions is the measurement process. This concept captures the aspects that need to be defined to put performance measurement into practice. A first critical aspect is represented by the reference standards against which R&D performance should be compared (Burch, 1994; Merchant, 1998; Kerssensvan Drongelen and Cook, 1997; Driva and Pawar, 1999). Defining appropriate standards is very challenging because of the uniqueness of each R&D project, its degree of uncertainty, and the thick curtain of uncertainty that surrounds other firms’ R&D activities. The second fundamental process element is measurement frequency. Performance in R&D functions can be measured with different frequencies (for example, weekly, monthly, annually, or by milestones), but an optimal value can hardly be

24

Evaluation and performance measurement of R&D

defined (Driva and Pawar, 1999; Presley and Liles, 2000; Driva et al., 2000; Suomala, 2003). The choice of the most appropriate frequency is influenced in fact by the objectives for which a firm measures its performance, the dimensions and the indicators it establishes to employ, and the control objects that are selected. In the end, the fourth level of research identified in Figure 1.1 adopts a contextual perspective to emphasize that a PMS for R&D functions should be studied within the context in which it is used, which is both internal and external to the firm. This body of research is consistent with the largest part of management accounting and control research (for example, Gordon and Miller, 1976; Merchant, 1981; Gordon and Narayanan, 1984) and basically reminds us that the PMS is employed in a specific R&D setting, being it basic and applied research or NPD (Pappas and Remer, 1985; Hauser, 1998; Chiesa and Frattini, 2007), with a given amount and quality of available resources (Godener and Soderquist, 2004), within the scope of a firm’s specific business strategy, mission, values and management style (Griffin and Page, 1996; Nixon, 1998; Kim and Oh, 2002; Loch and Tapper, 2002), and, finally, in a broader competitive, economic, social, cultural and political context (Nayak, 1987; Loch et al., 1996). For instance, Pappas and Remer (1985) study which indicators are more appropriate to the different stages of the R&D process, suggesting that qualitative indicators are best suited to basic research, semi-quantitative to applied research, and quantitative for NPD. On the same topic, Hauser and Zettelmeyer (1997) classify R&D activities into: ‘Tier 1’, which corresponds to basic research, ‘Tier 2’, which is related to the development of the core technological competencies of the firm, and ‘Tier 3’, which includes R&D projects focused on the satisfaction of customer needs through incremental improvements, and discuss which metrics are best suited to the different ‘tiers’. Loch and Tapper (2002) comment on the impact that the type of R&D strategy pursued by the firm has on the design of a PMS used in an applied research group. Mohapatra et al. (2003) show that the choice of the indicators for R&D performance measurement depends also on the availability of resources to be devoted to the implementation and use of the PMS. Nixon (1998) claims that performance measurement in R&D should be consistent with the management systems and organizational structures used in R&D, as well as the culture and values that permeate the R&D unit. As far as industry belonging is concerned, Loch et al. (1996) advance that a distinction exists between the performance dimensions that are monitored by firms operating in a high number of product markets and those that have a more focused portfolio. Finally, Nayak (1992) compares the types of R&D performance indicators used by European and American companies with those employed by Japanese firms. It emerges

R&D function

25

that process indicators are more widespread among firms from Japan, which suggests that they pay higher attention to planning activities than their western counterparts. Table 1.3 provides a comprehensive view of the variables that might affect the design of the different constitutive elements of the PMS for R&D, together with the literature contributions that recognize the importance of the specific relationship.

1.3

DESIGNING A PERFORMANCE MEASUREMENT SYSTEM FOR THE R&D FUNCTION: A REFERENCE FRAMEWORK

As is clear from the last section, literature on performance measurement of industrial firms’ R&D is vast. It has addressed several aspects of the phenomenon, albeit in a very fragmented manner, and it has put forward a number of different taxonomies of the solutions that firms might employ to measure their R&D functions performance. In this section, a comprehensive reference model is introduced, that describes how firms design a PMS for their R&D function. This model has been developed through a multiple case study analysis of 15 Italian firms operating in different industries (see Appendix A1.2). It synthesizes and integrates the main literature contributions reviewed in the previous section, and can be conceptually positioned at the fourth level of the literature categorization described above. The framework represents a useful instrument for researchers interested in interpreting and understanding the methods companies use to measure their R&D performance, and in future empirical research into the topic. It is also a valuable support tool for R&D managers, who might employ it as a guide for designing a measurement system that is appropriate to their specific context and, therefore, more likely to be effective. 1.3.1

The Constitutive Elements of the PMS

A first part of the reference framework identifies the constitutive elements of a PMS for R&D (see Figure 1.2). The model shows that firms measure the performance of their R&D activities with multiple purposes. In particular, it is possible to identify the following list of major objectives a company might aim at when it comes to measure R&D performance: 1. 2.

Motivate scientists and engineers to improve their performance; Monitor the advancement of R&D activities as far as cost, time and quality targets are concerned;

26

Table 1.3

Contextual factors Type of R&D

Evaluation and performance measurement of R&D

The influence of the contextual factors over the constitutive elements of a PMS for R&D Objectives Perform- Indicators Control ance objects dimensions X

Stage in the R&D process

X

X

X

X

Competitive and R&D strategy

X

X

X

Availability of human and financial resources

X

X

X

X

X

R&D organization and management styles

Type of industry

Firms’ culture

X

X

X

X

X

Measure- References ment process X

Pappas and Remer (1985); Brown and Svenson (1988); Brown and Gobeli (1992); Hauser and Zettelmeyer (1997); Chiesa and Frattini (2007) Loch and Tapper (2002); Bremser and Barsky (2004); Cho and Lee (2005)

X

Griffin and Page (1996); Davila (2000); Loch and Tapper (2002) Emmanuel et al. (1990); Chiesa and Masella (1996); Mohapatra et al. (2003); Bremser and Barsky (2004); Cho and Lee (2005); Chiesa et al., (2006);

X

X

Nixon (1998); Davila (2000); Kerssensvan Drongelen et al. (2000); Kim and Oh (2002); Bremser and Barsky (2004); Cho and Lee (2005); Chiesa et al. (2008);

X

Loch et al. (1996); Kerssens-van Drongelen and Bilderbeek (1999); Davila (2000); Nayak (1987)

R&D function

27

OBJECTIVES • Motivate researchers and engineers; • Monitor the progress of activities • Evaluate the profitability of R&D activities • Support the selection of projects • Favor coordination and communication • Reduce the level of uncertainty • Stimulate organizational learning

DIMENSIONS OF PERFORMANCE

CONTROL OBJECTS

• Financial perspective • Customer perspective • Innovation and learning perspective • Business process perspective

• R&D unit • Functional departments • Project teams • Individuals

INDICATORS • Quantitative • Qualitative

PROCESS FREQUENCY STANDARDS • Regular • Milestones

Figure 1.2

3. 4. 5. 6. 7.

• Internal • External

Performance measurement system for R&D units: a reference framework

Assess the profitability of R&D activities and their contribution to economic value; Support the selection and prioritization of the projects to be started or discontinued; Streamline coordination and communication among the individuals and organizational units involved in R&D; Reduce the level of uncertainty in R&D activities; Stimulate and nurture individual- and organizational-level learning.

Some companies try to pursue several different objectives through R&D performance measurement, whereas others focus on a single or more limited number of purposes. In the case of Company A, a biotech firm that was studied and whose PMS for R&D is described in Appendix A1.1, performance measurement is undertaken with the main purpose of motivating

28

Evaluation and performance measurement of R&D

chemists and improving their performance in the optimization of novel molecules, which is a task requiring a very high level of innovativeness and creativity. In all cases, the choice of the objectives to be pursued represents a first critical moment in the design of the PMS for R&D, from which the characteristics of the other constitutive elements descend. A first key aspect in the design of the PMS for R&D is the definition of its structure, that is, the set of organizational objects (or control objects) whose performance is kept under control. As far as this issue is concerned, it emerges that firms generally measure the performance of one or more of the following objects: 1. 2. 3. 4.

The R&D unit as a whole; The functional departments the R&D unit is made of; The project teams that are activated in the R&D unit; The individual scientists and engineers working in the R&D unit.

These levels are not mutually exclusive; a PMS for an R&D unit has typically a complex structure, comprising a set of interrelated control objects. In the case of Company A, the PMS monitors, for instance, the performance of applied research projects, scientific departments (that is, medicinal chemistry, analytical chemistry, computational chemistry and screening) and individual researchers. Another biotech company that was studied during our analysis, on the other hand, adopts specific indicators and measurement practices for its Drug Discovery and Drug Development units (Chiesa et al., 2006). It should be noted that a firm might measure the performance of different control objects with different purposes. Whereas individuals’ performance could be measured with the main objective of motivating them and stimulating coordination and learning, at the project team level monitoring the progress of activities and evaluating their profitability is more likely to be the main concern. The choice of the objectives to be pursued and the structure of the PMS influence the dimensions along which performance measurement is undertaken. The analysis indicates that these dimensions can be brought back to the Balanced Scorecard perspectives, as suggested by a number of scholars (Kerssens-van Drongelen and Cook, 1997; Sandstrom and Toivanen, 2002; Bremser and Barsky, 2004). Companies measure R&D performance taking into account: 1. 2.

The economic and financial aspects associated with R&D (financial perspective); The extent to which R&D identifies and satisfies the needs of its internal and external customers (customer perspective);

R&D function

3. 4.

29

The efficiency with which specific tasks and processes are carried out (business process perspective); The extent to which R&D contributes to generating new knowledge and innovation opportunities (innovation and learning perspective).

There are firms that combine several different perspectives in the measurement of R&D performance, whereas others focus on a single most important one. The key point here is that different dimensions are used to pursue different objectives for performance measurement. For instance, in order to motivate researchers, the extent to which R&D contributes to the firm’s innovation capability is typically measured, as clearly emerges from the case of Company A (see Appendix A1.1). On the other hand, the financial perspective seems to be the privileged one when a firm uses performance measurement mainly to evaluate the profitability of R&D activities and their contribution to the firm’s economic value. The choice of the performance dimensions employed in the PMS for R&D should be consistent not only with the objectives the firm aims to pursue through the PMS, but also with the control objects whose performance is kept under control. For instance, it is generally useless and counterproductive to apply a financial perspective to measure individual performance, whereas it might be appropriate for the R&D unit as a whole. As far as the metrics used in the PMS are concerned, a critical distinction made by the studied firms is between qualitative and quantitative indicators. Consistently with extant research, firms tend to use a mix of both qualitative and quantitative metrics in R&D, the former best suited to capture unmeasurable aspects, and the latter capable of reducing the subjectivity of the evaluation (Pappas and Remer, 1985; Werner and Souder, 1997; Driva et al., 2000). Furthermore, given that economicfinancial indicators are often questionable since it is very difficult to give a monetary evaluation of uncertain and intangible elements (Frattini et al., 2006), they are often integrated by non-financial indicators, whose measurement is usually easier. This aspect is particularly evident in the case of Company A, reported in Appendix A1.1, whose PMS for R&D purposefully mixed quantitative indicators such as ‘Market potential of novel identified molecules’ expressed in euros, and qualitative ones such as the ‘Quality of communication networks among researchers’, expressed through peers’ evaluation. Again, the choice of the indicators to be employed should be consistent with the objectives for which R&D performance is measured (for example, quantitative indicators might not be necessary for stimulating coordination and communication, but they are key when it comes to evaluating the profitability of R&D projects), with the selected performance dimensions (for example, quantitative indicators

30

Evaluation and performance measurement of R&D

are required to implement measurement along a financial perspective) and control objects. As far as the measurement process is concerned, the complexity inherent in defining appropriate standards in R&D, which is particularly underlined by extant literature, is well documented in the empirical analysis as well. Sometimes it happens that a new project is similar to previous ones, for which a performance track record is available. In these cases, standards can be defined considering the values of past performance indicators, which may be slightly corrected to take into account risk or similar aspects. Nevertheless, it often happens that standards have to be defined on an ad hoc basis, for a specific R&D project or effort. It is really uncommon that information about competitors’ performance is available, and this severely limits the potential to define standards through external benchmarking. The analysis shows that a few companies operating in the aerospace sector were able to compare their development performance with industry benchmarks, whereas using internal past project performance was a relatively more diffused alternative among firms in chemical, cosmetic and pharmaceutical industries (Chiesa and Frattini, 2007). Obviously, the choice to use internal or external standards is influenced by the characteristics of the other PMS’s elements. For instance, it is very unlikely that external standards are available for performance measured at the individual level, whereas this is more likely when the relevant control object is the R&D project. With respect to the other process element, that is, measurement frequency, the empirical analysis confirms the possibility for companies to gather information about R&D performance on a regular basis or by milestones. The choice of the exact frequency should be undertaken in light of the PMS objectives, the dimensions of performance (and indicators) a firm has established to employ, and the different control objects. For example, it appears that the measurement of technical performance is linked to project milestones, whereas a regular measurement, typically on an annual or monthly basis, is more common for the other dimensions of performance. A strong relationship also exists between frequency and measurement objectives: in order to monitor the progress of activities, for example, frequency is generally higher than when motivation of researchers is the primary objective a firm wishes to pursue. 1.3.2

The Role of the Measurement Context

The last section showed that designing an effective PMS for a firm’s R&D unit requires above all that ‘internal’ consistency among its constitutive elements is ensured. Nevertheless, the empirical analysis suggests that companies take into account, in the design of the PMS, a number of

R&D function

PMS FOR R&D UNITS

MEASUREMENT CONTEXT • The firm’s competitive and technology strategy • The organizational structure of the R&D unit • The type of R&D activity subject to measurement • The availability of resources • The industry in which the firm operates

31

OBJECTIVES • Motivate researchers and engineers; • Monitor the progress of activities • Evaluate the profitability of R&D activities • Support the selection of projects • Favour coordination and communication • Reduce the level of uncertainty • Stimulate organizational learning

DIMENSIONS OF PERFORMANCE • Financial perspective • Customer perspective • Innovation and learning perspective • Business process perspective

CONTROL OBJECTS • R&D unit • Functional departments • Project teams • Individuals

INDICATORS • Quantitative • Qualitative PROCESS FREQUENCY STANDARDS • Regular • Internal • Milestones • External

Figure 1.3

Consistency between measurement context and PMS for R&D units

variables related to the characteristics of the firm in which measurement takes place, as well as its external environment. This is consistent with the largest part of extant management accounting and control research (for example, Gordon and Miller, 1976; Merchant, 1981; Gordon and Narayanan, 1984), and is a reminder that a PMS for R&D should be studied within the context in which it is going to be used. In other words, a second level of consistency should be taken into account when designing the PMS for a firm’s R&D unit, that can be labelled as ‘external’ and concerns the relationship between the characteristics of the PMS’s constitutive elements and the context in which measurement takes place. This further level of consistency is captured in Figure 1.3. In particular, it emerges that a number of contextual variables have a key role in the design of the PMS for R&D, that is: 1. 2. 3. 4.

The firm’s competitive and technology strategy; The organizational structure of the R&D units to which the PMS is applied (functional or project-oriented); The type of R&D activity which is subject to measurement (basic and applied research or NPD); The availability of resources (for example, time, money, personnel, technology, know-how) for designing, implementing and using the PMS, which often depends on the size of the firm or its R&D unit;

32

5.

Evaluation and performance measurement of R&D

The industry in which the firm operates, and especially the level of dynamism and turbulence that characterizes it and affects the failure rates and the level of technical/commercial risk of R&D projects.

Comparing a number of cases investigated in the empirical analysis, it emerged for instance that PMSs used for basic and applied research have very different characteristics from those employed in NPD (Chiesa and Frattini, 2007). In particular, the measurement of NPD projects is mainly focused on efficiency and time, while effectiveness and contribution to value become the critical performance dimensions in research. Moreover, quantitative indicators are predominant in NPD, whereas in research qualitative ones are relatively more diffused. Furthermore, the level of uncertainty surrounding R&D activities, which is influenced by the characteristics of the industry to which the firm belongs, affects as well the design of the PMS, and especially the performance dimensions it is based upon. The case of Company A, discussed in Appendix A1.1, provides a number of real world examples about how contextual factors might affect the design of the PMS for R&D units. 1.3.3

The Reference Framework in Practice: Archetypal Models for R&D Performance Measurement

The number of relationships that link the PMS’s constitutive elements to each other and with the dimensions of the measurement context, along which consistency should be ensured in the design of the PMS, is clearly almost infinite. Therefore, as a last step in the empirical research, it was decided to apply the framework reported in Figure 1.3 to a large-scale empirical analysis that involved a sample of Italian R&D-intensive firms (see Appendix A1.2), to find out whether some archetypal models of the PMS for R&D units exist, each made of an internally consistent set of constitutive elements and measurement context dimensions. The first interesting evidence is about the objectives that firms wish to achieve with performance measurement of R&D units. In particular, it seems that companies pursue the objectives listed in Figure 1.3 in clusters. A first set of companies use performance measurement in R&D with the foremost objective to improve the control they apply on R&D activities, and to have a support for taking more timely and successful decisions. Firms in this cluster share the following main objectives for R&D performance measurement: monitor the progress of activities, select R&D projects and evaluate the profitability of R&D activities. These can be labelled as ‘hard’ objectives. A second group of companies use performance measurement chiefly as a tool for motivating researchers

R&D function

33

and engineers, addressing their attention toward the long-term innovation goals of the firm, and overcoming the lack of commitment due to R&D’s mainly intangible results. These firms believe that motivating R&D professionals is critical for improving their performance, and often decide to reward them according to the performance collected through the PMS (as happens, for example, in Company A (see Appendix A1.1), where performance measurement elaborates the data used to run a Management By Objectives MBO system). These firms are mainly interested in pursuing ‘motivational’ objectives through R&D performance measurement. Finally, the analysis unravels the existence of a number of companies for which performance measurement serves also a second purpose, besides the ‘hard’ or ‘motivational’ objectives mentioned above. In particular, they think it is a critical means to improve the execution of R&D activities, overcoming their main organizational barriers. In particular, performance measurement is used also to enhance coordination and communication, inspire organizational learning, and lessen the level of uncertainty in R&D (which are labelled as ‘soft’ objectives). It should be underlined that these results are somehow consistent with the anecdotal empirical evidence reported in the literature (Kerssens-van Drongelen and Cook, 1997; Kerssens-van Drongelen et al., 2000; Loch and Tapper, 2002). The empirical investigation indicates that the importance of each cluster of objectives is affected by some characteristics of the context in which measurement occurs. In particular, ‘hard’ objectives are more diffused in those companies that measure performance of New Product Development activities. This is due to the fact that the need for control is stronger in NPD than basic and applied research, because in the former the market is closer and consequently deadlines, quality requirements and target costs become critical for the success of the new product or service and the firm’s competitiveness. Furthermore, the financial and human resources needed to carry out NPD activities are very large (in comparison with basic and applied research). This makes a careful assessment of the profitability of R&D a critical activity for R&D managers. It also emerges that the need for control in R&D is particularly strong in large R&D units. The larger a firm’s R&D unit, the higher the number of projects under development, the higher the number of people (often organised in separate departments or functional areas) working on these projects, and the larger the resources involved in R&D. These conditions make the need for a ‘hard’ form of control particularly evident. However, a close control of R&D requires that these activities are somehow predictable and that the different steps and phases an R&D project goes through can be easily identified. As far as this aspect is concerned, New Product Development activities can be

34

Evaluation and performance measurement of R&D

more easily predicted than basic and applied research, but the capability to anticipate the direction of a firm’s R&D efforts is heavily affected by industry belonging. Kodama (1995) proposes a taxonomy of industrial sectors based on the likelihood that an R&D project is discontinued, which is used as a proxy of the uncertainty, or, conversely, the predictability of R&D activities.1 High Tech industries have a mortality rate that decreases along the R&D process whereas science based industries are those where the freezing rate stays constantly high. Our analysis suggests that firms use performance measurement in NPD mainly to control activities in those industries (High Tech, in the definition of Kodama) where the failure rates of projects and activities, and hence their unpredictability, is smaller. On the other hand, companies use performance measurement with ‘motivational’ purposes in basic and applied research. Improving motivation becomes critical here because the activity is very uncertain, with unpredictable and distant outcomes, which impede an unambiguous alignment of individual activity with the firm’s strategic objectives. In order to improve individual performance it is necessary therefore to stimulate creativity and contemporarily to align and address behaviour to the critical aspects for the company. The importance attached to ‘motivational’ purposes in R&D performance measurement is affected not only by the type of R&D activity. Stimulating motivation can be felt indeed as a particularly critical goal of performance measurement also in New Product Development. This was found in those companies operating in science based industries (Kodama, 1995), where unpredictability of development activities is particularly high. Finally, the empirical investigation indicates that individuals’ motivation is a critical purpose for performance measurement in small-sized organizations. This is due to the fact that, in these environments, a personal (or clan) form of control prevails (Ouchi, 1979) and therefore it basically requires the capability to align employees’ efforts to the firm’s priorities. The analysis reveals that ‘soft’ objectives are pursued along with ‘hard’ or ‘motivational’ ones, whereas the last two classes of goals appear to be mutually exclusive. It also emerges that it is typically companies with very large R&D units that pursue ‘soft’ objectives (for instance, to improve communication and coordination, to stimulate organizational learning and to overcome decision making inertia) through their PMS in R&D. This is obvious if we consider that a larger size is generally correlated with higher degrees of organizational complexity, hierarchy and fragmentation, which raise the need for better coordination and more effective organizational learning processes. Figure 1.4 provides a synthetic representation of this evidence.

R&D function

35

• Large firms and R&D units

Soft

• New Product Development • Large firms and R&D units • High Tech Industries

• Basic and Applied Research • Small firms and R&D units • Science Based Industries

Hard

Motivational

Classes of objectives Measurement context

Figure 1.4

Relationship between measurement context and classes of objectives

A second group of interesting insights is about the characteristics of the PMS and, in particular, the dimensions along which R&D performance is assessed. It seems that firms employ different balanced scorecard perspectives (see Figure 1.3) to pursue different classes of objectives. It noticeably emerges that financial and customer perspectives are privileged by firms pursuing ‘hard’ objectives. This is due to the fact that selection of R&D projects is based on the their ability to contribute to the firm’s competitiveness, which is in turn affected by their economic performance and the appealing of their outcomes for prospective customers. On the other hand, the fact that firms pursuing ‘motivational’ objectives make extensive use of the innovation and learning perspective clearly indicates that individuals have to be motivated on the basis of the extent to which they contribute to the firm’s innovative capability. The internal business process perspective is instead used by firms pursuing both ‘hard’ and ‘motivational’ objectives. Those companies that are mainly interested in the ‘motivational’ use of the PMS employ the internal business perspective besides the innovation and learning one to introduce some indicators that can be directly and fully controlled by individuals. This aspect is critical indeed for motivational purposes, as shown by theories of action, design and expectation (for example,

36

Evaluation and performance measurement of R&D • Large firms and R&D units Soft • Business process perspective

•New Product Development •Large firms and R&D units •High Tech Industries

•Basic and Applied Research •Small firms and R&D units •Science Based Industries

Hard

Motivational

• Financial perspective • Customer perspective • Business process perspective

• Innovation and learning perspective • Business process perspective Classes of objectives Measurement context BSC perspectives

Figure 1.5

Three archetypes for R&D performance measurement

McClellan et al., 1953; Pritchard, 1990; Moizer, 1991). On the other hand, the use of the internal business dimension in firms that are mainly interested in the ‘hard’, diagnostic role of the PMS is due to the need to establish an operative type of in-progress control that financial and customer-related indicators do not allow to carry out. Firms pursuing ‘hard’ objectives use instead the business process perspective mainly to introduce an operative form of control that financial and consumeroriented measures do not allow. As far as ‘soft’ objectives are concerned, it emerges that they are associated mainly with the business process perspective. What is interesting to remark in this case is that efficiency in undertaking business processes is evaluated in an interactive manner. This result is consistent with Simons’s theory on the use of interactive management control systems. (Simons, 2000) Figure 1.5 indicates that three paradigmatic models for R&D performance measurement can be identified. Each of them is made of an internally consistent set of objectives, dimensions of performance and contextual variables. Figure 1.5 also suggests that the choice of the performance dimensions depends on the objectives a firm wants to pursue through the PMS, whereas it does not appear to be significantly affected by the characteristics of the measurement context.

R&D function

1.4

37

CONCLUSIONS

The chapter addresses the problem of measuring performance in the R&D function of an industrial firm. In the first part, a synthesis of the relevant literature is presented, to illustrate to scholars and researchers the perspectives from which this topic has been studied so far, and to serve as a framework to better position future pieces of empirical and theoretical research. In the second part of the chapter, a systemic reference model is introduced and discussed, which describes how firms design a PMS to be used in their R&D units. This model has been developed through a detailed analysis of 15 Italian firms operating in a number of heterogeneous industries. It explains that a crucial element for designing an effective PMS in R&D is the consistency among the elements it is made of, that is, objectives, dimensions of performance, control objects, indicators and measurement process (internal consistency), and between the PMS and the context in which measurement takes place, defined in terms of the firm’s competitive and technology strategy, the organizational structure of the R&D unit, the type of R&D activity, the availability of resources, and the industry in which the firm operates (external consistency). The existence of three archetypal models for R&D performance measurement was also unearthed through a large scale analysis on a sample of R&D-intensive firms. This indicates that firms facing a similar measurement context tend to measure the performance of their R&D functions pursuing the same objectives, which in turn bring them to use similar performance dimensions. This clearly indicates the importance that internal and external consistency play in the design of the PMS for R&D. The framework contributes to research into R&D performance measurement, because it provides an integrative perspective to analyse and interpret the characteristics of a PMS for R&D, which can serve as a starting point for future theoretical and empirical analysis addressing a number of research questions that are still unanswered: 1.

2.

3.

What are the organizational implications from the use of a PMS for R&D? How does the process of implementation of the PMS deploy? How do the characteristics of the PMS evolve over time, in response to changing environmental conditions and experience in using the PMS? How could the PMS be integrated into the firm’s overall management control system?

38

Evaluation and performance measurement of R&D

APPENDIX A1.1

DESIGNING A PMS FOR R&D IN COMPANY A

Company A is an Italian firm founded in 2001, which operates in the field of pharmaceutical research, offering technology-intensive services that support the new drug R&D process of client firms. The birth of the firm can be dated back to the merger between Glaxo-Wellcome and SmithKlineBeecham, two big pharmaceutical companies with subsidiaries in Italy (in Verona and Milan). The merger took place in 2000 and resulted in the birth of GSK (GlaxoSmithKline). In the wake of the company’s reorganization, it was decided that the drug discovery activities of SmithKlineBeechan had to be disbanded. Part of the SmithKlineBeecham’s drug discovery team decided to leverage on the excellent competences in medicinal chemistry they had developed in more than 20 years of research, and submitted a spin-off proposal to GSK. The idea was to create a small and highly innovative firm, whose core business was the provision of services able to support pharmaceutical firms’ R&D process. The spin-off proposal was approved by GSK, and the firm was born in 2001. One of the first problems Company A’s top management faced was the design and implementation of a performance measurement system for the firm’s activities. The company was mainly involved in undertaking R&D activities for third parties (on a contract research basis) and for internally generating new molecules to be licensed out to big players of the pharmaceutical market. The case of Company A is useful because it exemplifies the problems firms are confronted with when they have to design a PMS for their R&D units. The context in which the performance of Company A’s R&D activities was measured can be described along the following dimensions: 1. Industry and competitive environment At the time when Company A was born, the characteristics of the pharmaceutical industry were very advantageous for the establishment and for the success of firms providing high-technology services supporting the R&D process. The increasing complexity, costs, risks and investments of new drugs’ discovery and development processes, were forcing big pharmaceutical companies to reorganize their R&D structures and to outsource R&D activities to smaller and more flexible firms. (Muffatto and Giardina, 2003). In particular, with the completion of the human genome project, all the 25,000 human genes were fully identified and understood, and about 3000 were acknowledged to be ‘drugable’ or ‘physiologically relevant’ molecular targets. They represented, together with the diffusion of high throughput screening (HTS) techniques, a very strong stimulus for pharmaceutical firms to strengthen

R&D function

39

their efforts in screening activity. This led to the identification of a huge number of potentially effective molecules, which needed to be generated and optimized to be turned into leads that could enter the development phase. (For an overview of the drug discovery and development process in the pharmaceutical industry see, for example, Muffatto and Giardina, 2003; Chiesa, 2003.) It was especially the large scale to be achieved in the tasks of lead generation and optimization in order to remain competitive, and their high level of risk and costs, that encouraged most of the largest pharmaceutical companies to outsource these activities to specialized and flexible organizations, thus externally acquiring the missing skills and critical mass. 2. Competitive and technology strategy In this competitive scenario, GSK approved the spin-off proposal and transferred, through a management buy-out process, laboratories, personnel, and patents to the new-born Company A. The firm started working with a staff of about 25 chemists and researchers, as a service supplier specialized in the provision of ‘lead generation’ and ‘lead optimization’ services to large pharmaceutical companies. Company A represented therefore a typical example of contract research organization (CRO), where ‘contract research’ can be defined as the activity through which a client firm employs the services provided by an external organization to undertake a specific R&D task (Haour, 1992). In 2003, the top management started thinking about the strength of the firm’s business model, which was based entirely on the provision of technical and scientific services. In particular, they believed it could be exposed to the fluctuation of the pharmaceutical market and to the strong competition from service providers working in countries like India, Russia and China. Some analysts estimated that these would be able, in four to five years, to offer high quality services, comparable to European or American firms, but at a significantly lower price (from one-third to one-fifth). This brought Company A to revise its competitive strategy, and in particular to dedicate part of its resources to internal research, with the aim of identifying and generating novel leads to be internally optimized and developed until ready for the clinical trials. At that point, they should have been licensed out or partnered with large pharmaceutical firms that were able to complete the development phase and to bring the new drug into the end market. Engaging in this activity also gave Company A the opportunity to have a buffer of highly specialized employees that could be used to cope with unforeseeable and transitory increases in the demand for lead generation and optimization services. Although service supply and internal research call for a similar competence basis and do not differ in terms of the nature of the activities to be carried out, they are radically

40

Evaluation and performance measurement of R&D

dissimilar if we look at the critical success factors needed to achieve a competitive advantage in the two areas. In drug discovery services, key for competition are the quality level of the service, respect for the promised delivery time (punctuality) and confidentiality in the administration of the information about clients’ business strategies and molecules. These aspects are not so critical for internal research, where it is essential to gain the leadership in the identification of novel candidates for clinical development and the time horizon is very broad. Although tangible results are necessary also in internal research, time constraints are less severe due to the fact that manifest clients’ pressures are missing. Moreover, Company A’s top management believed that a further critical success factor existed, which was common to the two business areas, that is, the capability to build and nurture a reputation for scientific and technological leadership, at least in the areas that were critical for the firm’s portfolio of activities. This aspect was considered as very important because it provided contacts with new potential clients in the business of discovery services. In addition, it allowed the firm to build and nurture positive relationships with large pharmaceutical and biotech companies, that were a prerequisite for improving the internal research pipeline. 3. R&D organization and management The organization of the firm’s activities is an example of the well-known ‘matrix’ structure. Researchers are grouped first of all into departments (that is, medicinal chemistry, analytical chemistry, computational chemistry, screening), according to the discipline in which they are specialized. Each department is made of a number of laboratories and is headed by a director. People belonging to the different departments and laboratories take part in project teams, where interdisciplinary and complementary competencies are put together to perform specific tasks and to pursue specific R&D objectives (for example, supply the lead optimization service required by a big pharmaceutical firm). For each project that is launched, a project manager is appointed, generally the researcher with the longest experience or specialized in the scientific or technical discipline that is the most crucial for the success of the project. In this matrix structure, the stronger organizational power lies with the head of the scientific departments (light matrix). As far as the management of human resources is concerned, Company A’s top management realized that a participative leadership style is of foremost importance to stimulate collaboration and motivation from scientists and chemists, in a context where uncertainty, delays and unforeseeable events are very common. 4. Type of R&D activity According to the well known taxonomy of R&D recalled in the introduction of this book, the activities undertaken

R&D function

41

by Company A can be classified as basic and applied research. As opposed to new product development (NPD), they are characterized by a higher level of uncertainty, almost unforeseeable developments and outcomes, intangible results, longer length and lower demand for human and financial resources. 5. Available resources Company A was a very small firm at the time when the PMS was designed. Therefore, the availability of financial, technological and human resources to be involved in the design and operation of the PMS was very limited. The firm’s top management realized that this was one of the main factors that brought them to design and implement a simple and not very sophisticated system (which included, for instance, a limited number of performance dimensions, few metrics and a simple structure). Within this measurement context, the PMS designed and implemented by Company A, schematically represented in Figure A1.1.1, has the following major characteristics: 1. Objectives The peculiarities of the industry in which the firm operates and the type of R&D activity it undertakes (in particular, its high level of complexity, uncertainty, and creativity) has suggested to Company A’s top management that the PMS should be used mainly to motivate researchers and chemists, and to direct their efforts toward the achievement of corporate-wide objectives. As will be explained below, this had relevant implications on the design of the other elements of the PMS, and especially the dimensions of performance to be employed and the associated indicators. Top management acknowledged the importance of collectively defining the performance objectives to be achieved, in order to favour motivation. Moreover, these targets need to be measurable, although the activities researchers are engaged with do not produce tangible and highly visible results. The motivational purpose of the PMS was reinforced through the introduction of a management-by-objectives (MBO) rewarding system, according to which researchers were given financial incentives on the basis of the performance they had achieved. It is interesting to note that, in order to improve the measurability of the performance targets, these were defined not only at the individual level, but also at the organization level. In other words, individuals were rewarded on the basis of the financial and economic results of the firm as a whole, under the assumption that each researcher adds to the firm’s performance, although individual results and contributions are often hard to identify and evaluate.

42

Evaluation and performance measurement of R&D

2. Dimensions of performance The choice of the dimensions of performance to be monitored was driven mainly by the objectives the firm wished to pursue, as well as its innovation and R&D strategic priorities. Two main perspectives were used to measure R&D performance: the innovation and learning and the internal business processes dimensions. Top managers realized that motivation of researchers could be achieved by measuring and rewarding their contribution to the firm’s innovation potential and creative capabilities. Nevertheless, they also realized that the performance that is achieved along innovation and learning indicators is not completely controllable by the single researcher, making it difficult to use to support an MBO rewarding system. As a result, they decided also to use measures related to the capability of researchers to undertake specific tasks, such as acquiring a particular competence, establishing relationships with external organizations, and efficiently carrying out collaborative activities (internal business perspective). Obviously, the choice of the performance dimensions is also influenced by the firm’s strategic technology priorities. For instance, in Business Area 1 (provision of lead optimization services), indicators related to the internal business perspective are predominant and coupled with customer-related measures, because satisfying clients’ quality requirements and respecting temporal milestones and budgets in service provisions become especially critical strategic priorities. 3. Indicators Measuring in practice the selected dimensions of performance requires the identification of appropriate indicators or metrics. The top management in Company A was aware of the importance of employing indicators that allowed for a simple and direct measurement and that could be collectively discussed. These are the typical characteristics of quantitative indicators and are paramount in supporting the operation of an MBO rewarding system. Nevertheless, the top management realized that using only quantitative indicators was unfeasible because they were unable to capture the un-measurable and intangible aspects related to basic and applied research. Moreover, high levels of uncertainty and dynamism make it almost impossible to define appropriate standards for quantitative indicators. Therefore, it was decided that the PMS used in Company A should employ a balanced mix of both qualitative and quantitative indicators. For instance, in order to measure the capability of the firm to acquire new technologies and competencies, the numeric indicator based on the ‘time needed to acquire new technologies’ was integrated with the qualitative assessment of the international relevance of the technologies/competencies acquired. The set of indicators used by Company A is reported in Figure A1.1.1.

R&D function

43

4. Control objects When designing the PMS for the firm’s R&D function, top managers acknowledged the need that it closely mirrored the organizational structure of the R&D unit in which it was going to be employed. As a result, the critical control objects it is built around are the ‘R&D project’ and the ‘department’. Moreover, the need to use the PMS mainly as a means to motivate researchers, made it necessary to measure the performance of individuals as well. A critical aspect in the design of the PMS was the assignment of the selected indicators to the different control objects around which it was organized, which was driven by the above mentioned critical objective of the PMS. In particular, attention was paid to make each organizational object responsible only for those performance indicators it could directly and completely influence. It emerged that the controllable factors tend to vary across the organizational levels. For instance, an individual researcher could not be held responsible for the success of a specific molecular screening activity (that is, for the quality of novel identified leads), because levels of uncertainty are so high and hence difficult to control (the chances of a molecule becoming a lead are very low). Nevertheless, this can be considered a responsibility of a whole team engaged in a drug discovery and development project, because the project manager can influence its output, for example by improving processes and by allocating resources to a good portfolio of research activities. The alignment between the indicators and the control objects of the PMS, reported in Figure A1.1.1, is a very critical aspect because of its motivational implications. 5. Measurement process The definition of the performance measurement standards and the measurement frequency had to be undertaken in light of the culture and values that permeate a firm’s R&D activities as well as the characteristics of the human resources it employs. In the case of Company A, it was immediately clear to top management that the chemists and scientists were very autonomous in carrying out their work, as well as characterized by a very high educational level. This made it crucial for the effectiveness of the measurement system that it did not constrain the autonomy of researchers. The risk that they perceived the PMS as an impediment to their activity was in fact particularly high. As far as standards are concerned, for instance, internal and subjectively evaluated targets were the preferred ones. They could in fact encourage people’s initiative in those cases where results were different from those that had been agreed on. For instance, a number of filed patents that is lower than the established target, can still be considered a valuable achievement if the evaluation is subjectively undertaken by peers or superiors who are better able to understand the unforeseeable external conditions that might have

44

Evaluation and performance measurement of R&D

MEASUREMENT CONTEXT R&D AND TECHNOLOGY STRATEGY COMPETITIVE STRATEGY

R&D ORGANIZATION AND MANAGEMENT

Which are the long-term objectives pursued through the firm’s R&D? Which are the technical and scientific competencies to be developed? Does the company collaborate with external organizations in order to acquire the necessary competencies?

In which business area does the company compete? • BUSINESS AREA 1: Discovery Services • BUSINESS AREA 2: Internal Research INDUSTRY AND COMPETITIVE ENVIRONMENT Which are the peculiarities of the competitive environment that influence the firm’s competitive strategy? • Increasing costs, complexity, risks and investments of new drugs’ discovery and development processes • Outsourcing by big pharmas of R&D activities to smaller and more flexible firms

What is the organization of the firm’s R&D? What is the predominant management style in R&D? • Matrix structure • Participative leadership style

• BUSINESS AREA 1: • Service quality level • Service cost level • Respect of the promised delivery time (punctuality) • Confidentiality in the management of clients’ information • Building an external technology reputation • BUSINESS AREA 2: • Gaining the leadership in the identification of novel candidates to clinical development • Building an external technology reputation • The competencies to be developed concern pharmacokinetic, preliminary toxicology, MTS, additional ADME profiling • Partnerships and collaborations with other organizations, especially universities and other pharmaceutical firms

TYPE OF R&D ACTIVITIES Which types of R&D activities does the firm undertake? • Basic and applied research AVAILABILITY OF RESOURCES Are there any limitations in the resources available for the design, implementation and use of the PMS? • Severe resource limitations

PMS CONSTITUTIVE ELEMENTS OBJECTIVES Which are the main objectives to be pursued through the PMS? • Motivating researchers DIMENSIONS OF PERFORMANCE

INDICATORS

CONTROL OBJECTS

Which are the dimensions of performance to be monitored by the PMS?

Which are the indicators to be used to monitor each performance dimension?

Which organizational levels are monitored and the indicator associated to each of them?

• BUSINESS AREA 1: • Service effectiveness (CUSTOMER) • Capability of acquiring new technologies (INNOVATION AND LEARNING)

• Average customer satisfaction • International relevance of technologies/ competencies acquired during one year • Average service cost variance

• Respect of the planned service costs (BUSINESS PROCESS) • Service delivery on time (BUSINESS PROCESS) • Quality of communication with clients (CUSTOMER) • Level of external reputation (INNOVATION AND LEARNING) • BUSINESS AREA 2: • Quality of novel identified leads (INNOVATION AND LEARNING) • Quality of novel optimized leads (INNOVATION AND LEARNING) • Capability of acquiring new technologies and competencies (INNOVATION AND LEARNING) • Capability of scouting licensing and partnering opportunities (BUSINESS PROCESS) • Quality of collaborations with external organizations (BUSINESS PROCESS) • Level of external reputation (INNOVATION AND LEARNING)

PROJECT

DEPARTMENT INDIVIDUAL

X X

X

X

X

X

• Percentage of projects concluded on time • Frequency of meetings with clients

X X

• Number of citations of the publications of company’s researchers • Percentage of the novel leads with the required degree of target binding • Percentage of the optimized leads with the required pharmacokinetic properties • International relevance of technologies/ competencies acquired during one year • Number of collaborations stipulated/ number of collaborations identified • Percentage of collaborations’objectives fully satisfied • Number of citations of the publications of company’s researchers

X

X

X

X X

X

X X X

X

PROCESS How are the standards to measure performance against selected? Which is the timing and frequency of the measurement • Internal subjective standards • Performance measured every four months with a final evaluation at the end of the year

Figure A1.1.1

A synthetic representation of the PMS used in Company A

R&D function

45

impeded the achievement of the established target. Similarly, in the definition of the frequency of the measurement system, it was decided to assign individual objectives at the beginning of each year. The evaluation of the objectives’ progressive achievement occurs about every four months, with a final evaluation at the end of the year. This timing was considered as compatible with the typical development of researchers’ activity.

APPENDIX A1.2

METHODOLOGICAL DETAILS ABOUT THE EMPIRICAL ANALYSES

The empirical research we have undertaken to gain a better understanding of how firms design a PMS for their R&D function comprised two main steps, each employing different methodologies. In the first phase, we designed and carried out a multiple case study, to build the interpretative framework that describes how firms design a PMS for their R&D function, and integrates as well existing literature on the topic. In the second phase of the research, we put our framework into practice, carrying out a survey on a sample of Italian firms, with the aim of finding out whether some archetypal approaches to R&D performance exist. Further details about the two phases of the research are described. The Multiple Case Study We decided to employ case study research as a methodological approach for the first phase of our empirical analysis. Case study is particularly useful indeed to address ‘why’ and ‘how’ questions, (Yin, 2003) and to delve into complex phenomena that can hardly be isolated from the context in which they occur (Eisenhardt and Graebner, 2007). Particularly, a multiple case study design was chosen, at it allows improvement of the external validity of the results through cross-case comparisons (Eisenhardt, 1989) and to arrive at more robust and testable interpretations of the phenomenon under study (Eisenhardt and Graebner, 2007). We built a sample of 15 Italian firms from very heterogeneous industries (for example, chemicals, aerospace, machining centres, biotech, pharmaceuticals), that were investigated in the last two years. Table A1.2.1 provides some preliminary information about these firms, whose real names have been blinded for confidentiality reasons. We adopted as unit of analysis the PMS used in the firm’s R&D function. It often happened that the companies we investigated had more than a single organizational unit where R&D activities were undertaken. Nevertheless, we had the

46

Evaluation and performance measurement of R&D

Table A1.2.1

The sample of firms investigated in the multiple case study

Firm Sector of activity

No. of employees Role of people interviewed

1

Semiconductors

50 000

2

Electronics for industrial applications

500

Director of R&D and Quality Manager

3

Machining centres

160

Director of the technical department

4

Aerospace

1800

Planning and cost control manager

5

Pharmaceuticals

500

6

Pharma-biotech

60

Chief Operating Officer

7

Chemicals

19 300

Director of Innovation & Technology in Plastic Additives

8

Aerospace

9000

9

Pharma-biotech

700

Director of Oncology Division

10

Pharmaceuticals

70

General Director of Research Laboratories

11

Household electrical appliances and home automation

12

Power generation technologies

2200

Technology and business development manager

13

Medical imaging diagnostic

1000

Vice President for Research and Development

14

Pharmaceuticals

2800

Vice President for Corporate Drug Development

15

Energy conversion

2600

R&D director

60 000

R&D projects manager

Director of the development department

Program manager

R&D platform manager

chance to study the PMS used in only one of them. We employed the widely accepted dichotomy between research and development, comprising in the former basic and applied research activities, and in the latter the development of both incremental and radical new products. All the

R&D function

47

organizational units we looked into could be clearly classified as engaged in either research or development activities, suggesting a widespread separation between research and development that was already observed by some scholars (Chiesa, 1996a). Data were collected mainly through personal direct interviews. Specifically, we went through the following steps: 1.

2.

3.

4.

At the beginning of each case a senior manager from the selected companies was contacted. He or she was briefed about the research project through a telephone meeting. Our informants were also asked to introduce us to the director of the firm’s R&D function or to another manager with responsibilities for the performance of the R&D unit; Next, we interviewed these informants face-to-face. We carried out two semi-structured interviews for each of them (each interview lasted on average one and a half hours), relying on a semi-structured replicable protocol, which was made of a number of open questions for each of the relevant constructs in our research (for example, characteristics of the measurement context, objectives for R&D performance measurement, measurement perspectives used in the PMS); We tape-recorded all the interviews and transcribed them afterwards. A telephone follow-up was carried out here with the aim to gather some missing but relevant data; We gathered secondary information using company reports and project documentation. Specifically, all the reporting documents produced during the operation of the PMSs were collected and analysed. These provided the researchers with general information on the selected firms, the nature of the R&D activity they carried out, the characteristics of the industry, and their general approach to R&D performance measurement. Furthermore, this information was triangulated with direct interviews in order to avoid personal interpretation biases and enhance construct validity (Yin, 2003).

The collected data were manipulated before being analysed. Specifically, we used the following approaches (Miles and Huberman, 1984): 1.

2.

Data categorization, which calls for the decomposition and aggregation of data in order to unearth relevant characteristics (for example, objectives for R&D performance measurement or context in which measurement takes place) and streamline cross-case comparisons; Data contextualization, which requires a systematic analysis of the contextual variables, not included in the conceptual model, that may bring to light unforeseen links between events and circumstances.

48

Evaluation and performance measurement of R&D

A within-case analysis was then conducted; the aim was to consider each case study as a separate one and to identify the characteristics of the relevant constructs for our research. Then, explanation-building was conducted, to disclose the relationships between the constitutive elements of the PMS and the measurement context. Finally, a cross-case comparison was performed, to contrast the evidence unearthed in each case study and reach a general interpretation of the phenomenon. These procedures for data collection and analysis, together with the employment of the semistructured interview protocol, also improved the reliability of the research design (Yin, 2003). The Survey Study To test the existence of some archetypal approaches to R&D performance measurement, we focused our survey on R&D-intensive firms, which are defined as those companies that heavily invest in and are strongly committed to R&D activities (Deeds, 2001). It was decided to limit the scope of the analysis to R&D-intensive firms under the assumption that R&D performance is especially critical for their competitiveness and, therefore, they represent an ideal setting to investigate our research problem. As a population frame, we used the list of the companies that are members of the AIRI (Italian Association for Industrial Research),2 an association for Italian R&D-intensive firms, whose institutional mission is to stimulate the development of private R&D initiatives, providing a fertile ground for its members to exchange experiences and best practices in the field of R&D management. We verified that the R&D-intensity of the 130 firms included in our sample was actually ‘above average’, as their belonging to the AIRI would suggest. In particular, we measured the ratio between annual R&D expenses and revenues (average value for the years 2004–6) for each firm, acquiring balance sheet data from AIDA, a professional database which contains information for more than 700,000 Italian companies. As a result, we verified that the ratio for each firm in the sample is higher than the average R&D/sales value for Italian companies measured by ISTAT. After a preliminary telephone contact, we sent to the R&D directors of these companies the questionnaire by mail, together with a letter describing the research project and providing a set of detailed instructions for filling in the questionnaire. We asked respondents to specify the importance they attach to the objectives for R&D performance measurement that emerged in the first, exploratory step of our empirical research, using a five-point scale, where 1 means no relevance and 5 very high importance. In the same way, the importance of the balanced scorecard performance dimensions

R&D function

Table A1.2.2

49

Profile of the sample and the respondents (%)

Firm

Sample

Respondents

Nonrespondents

Type of activity

Research Development

47 53

44 56

49 51

Size

Small Medium Large

20 17 63

13 28 59

24 11 65

Industry

Dominant design High-tech Science-based

30 46 24

25 42 33

32 48 19

was evaluated. We asked each R&D director to specify the type of activity (‘basic and applied research’ or ‘new product development’) undertaken in the unit he or she was responsible for (therefore, the ‘type of activity’ represented our unit of analysis); moreover, in case the firm had both types of units, we asked our contact person to send the questionnaire to the director of the other relevant unit. As far as the type of R&D activity is concerned, we applied the traditional distinction between research and (experimental) development (OECD, 2002). The companies in the sample were classified into Dominant Design, High Tech and Science Based according to their industry type and consistently with the classification suggested by Kodama (1995) discussed above (see note 1). The contextual factors were therefore modelled as nominal variables. Taking size into account, we distinguished the firms in the panel into three categories (that is, small, medium, large), according to the criteria suggested by the European Commission (European Commission Recommendation, 2002). Table A1.2.2 reports the distribution of the type of activity, size and industry for the sampled firms. A test for non-response bias was executed using chi-square statistics, which revealed no significant differences between respondents and non-respondents. In addition, the contextual factors did not appear to be significantly correlated (chi-square, p.0.1). This allowed us to separately analyse the relationships between each contextual factor and R&D measurement objectives. We tried to ensure content validity by submitting the first draft of our questionnaire to a combined pool of university and industry experts. The experts’ comments basically concerned the definition and description of the measurement objectives as well as the way in which questions were formulated, and they were incorporated in the final version of the questionnaire. After

50

Evaluation and performance measurement of R&D

questionnaires were returned, in order to better interpret some ambiguous answers and to ensure alternate-form reliability, we recalled about 20 firms by phone. These interviews were concise but conducted using different words to represent the same concepts. The stability of the answers given by the whole set of respondents who were recalled in the follow up interviews indicates a good understanding of the questionnaire’s items, which ensures high levels of reliability (Fink, 1995, Morgan et al., 2001). A total of 48 companies returned a properly filled-in questionnaire, which resulted in 61 usable cases: 27 for research and 34 for development, giving a response rate of about 32 per cent. Due to the nominal and ordinal nature of most of the variables included in the questionnaire, median and mode were the most appropriate measures of central tendency, as well as Spearman correlations and non-parametric statistics, to test the hypothesized relationships (Morgan et al., 2001).

NOTES 1. Kodama (1995) classifies industrial sectors into Dominant Design, High Tech and Science-Based industries. These three clusters differ in respect to their level of risk, defined as ‘the probability that an R&D program expenditure is frozen’ in an intermediate phase of the R&D process (freezing rate). Dominant design industries are characterized by a freezing rate that dramatically decreases from basic through applied research, coming to zero in the development phase. Dominant design industries are food, textile, pulp and paper, printing and publishing, oil and paints, petroleum and coal, rubber, ceramics, iron and steel, transportation, energy. High tech industries are characterized by a freezing rate that decreases throughout the R&D process, but remains greater than zero even in the late development. High Tech industries are ordinary machinery, electrical machinery, communications and electronics, precision equipment, aerospace. Science Based industries are those in which the freezing rate remains always high throughout the whole R&D process. Science Based industries are basically pharmaceuticals and industrial chemicals. 2. We eliminated from the list (which initially comprised about 170 members) those organizations that are publicly held, and obtained a panel of 130 companies.

2. 2.1

R&D projects INTRODUCTION

This chapter examines a second dimension along which measurement and evaluation of R&D in industrial firms can be undertaken and studied, that is, the R&D project. A project is defined as ‘a temporary endeavour undertaken to create a unique product, service or result’ (PMBOK, 2000, p. 4) and it has a number of distinctive characteristics that can be summarized as follows: 1. 2. 3. 4. 5.

it has a definite beginning, a definite end and is not an ongoing process; the product, service or result produced by a project is unique, different from anything else on the market; it has a defined set of desired and measurable end results; it has a defined sequence, that is, it progresses from an idea through planning and execution, until it is complete; it develops in steps and proceeds by increments.

Industrial R&D, because of its inherent characteristics, is typically carried out in the form of projects. Obviously, R&D projects can encompass a range of very heterogeneous activities, as already noted in the introduction to this book. There might be projects mainly focused on basic research, that represent the domain of universities’ and governmental research centres’ R&D activities, and are hardly ever within the scope of privately founded R&D. Industrial R&D projects more likely focus on applied research or New Product Development (NPD). Moreover, it is not uncommon that an industrial R&D project encompasses a larger portion of R&D activities, as happens when projects are launched to identify the opportunity to apply an established body of knowledge to solve a technical problem (apply research) and then proceed to translate this opportunity into a new product or service that reaches the market (NPD). Obviously, R&D projects comprising different types of activities have very dissimilar characteristics in terms of failure rates, resource requirements and technical complexity. These characteristics are also deeply affected by the industry in which the R&D project is undertaken (Kodama, 1995). 51

52

Evaluation and performance measurement of R&D

Because of the project’s pervasiveness as an organizational object in R&D activities, scholars and practitioners have investigated by and large which systems and approaches should be employed to manage R&D projects effectively. This chapter focuses in particular on the techniques that can be used for the evaluation of R&D projects. Evaluating a project means identifying the degree to which it contributes to the firm’s capability to create value for its shareholders in the long run and, consequently, the extent to which it is consistent with the firm’s competitive and innovation strategy. The evaluation of an R&D project represents also a prerequisite for an accurate selection of the projects that should be initiated and those that should be withdrawn or, at least, frozen and postponed. Moreover, it is also necessary to provide a reference framework for an ongoing evaluation of project activities, which represents a key aspect in project management. The chapter is organized in two main parts. The first one focuses specifically on the techniques that can be employed for the evaluation of single R&D projects. A taxonomy of the main classes of techniques that can be used to this purpose is proposed, and developed through a systematic review of the relevant academic and practitioner’s literature and a focus group where a number of R&D managers, from some of the largest and most innovative Italian firms, were invited to discuss and comment on the subject. For each evaluation technique encompassed by the taxonomy a description of its working principles is first provided. Afterwards, the contexts (defined, for example, in terms of type of R&D project being undertaken or industry belonging) in which each evaluation technique seems to be more appropriate are identified and discussed. This represents an important contribution to the existing research on the topic, which has dedicated limited attention so far to identifying the fields of application to which different classes of R&D project evaluation approaches are best suited and hence should be more extensively employed. The second part of the chapter adopts a different unit of analysis. In particular, it focuses on the firm’s portfolio of R&D projects (or R&D program) to investigate the techniques and the approaches that can be employed to evaluate its adequacy and the extent to which it is consistent with the firm’s R&D and innovation strategy. This part of the chapter discusses the perspectives along which the R&D projects portfolio can be evaluated, as well as the approaches that can be employed to tackle the different types of unfeasibility that are likely to be unearthed. Finally, Appendix A2.1 reports and discusses the case of a multinational firm, a manufacturer of flow equipment products and systems for the oil and gas industry, that was studied in more detail in the scope of

R&D projects

53

the research. This firm uses a very formalized process for the evaluation of its R&D projects, representing an original approach through which the techniques presented in the first section of the chapter can be combined to provide a more accurate evaluation. Furthermore, the case study allows us to reflect and comment on the organizational and managerial problems that may arise in the implementation and practical use of an evaluation technique. Notwithstanding its relevance, this aspect has been given limited attention so far by R&D management scholars, and therefore the discussion of the case study is believed to hold interesting implications and represents an important starting point for future research.

2.2

R&D PROJECT EVALUATION TECHNIQUES: LITERATURE REVIEW AND TAXONOMY

The evaluation of R&D projects has potentially a strong impact on the company’s long-term business positioning, on its sustainability and, ultimately, on its ability to generate value. As a consequence, over the last few decades, a number of techniques have been developed in the literature and diffused within companies and organizations to support this critical managerial task. When confronted with the problem of evaluating an R&D project, managers and practitioners face the challenge of selecting the most appropriate technique for supporting their decision. Indeed, even if the analytical expertise of the company and/or organization (that is, its ability to manage the ‘mathematical’ and ‘operational’ complexity of each technique and to collect data) clearly affects the quality of the evaluation results, it is widely acknowledged that the choice of the evaluation methods is equally important for avoiding misleading or even wrong conclusions. However, only a few studies (for example, Poh et al., 2001) have been undertaken to identify and analyse the settings in which each of the different available techniques performs at its best, thus posing a challenging goal for this chapter. In order to accomplish this objective, it is necessary first to distinguish the huge number of alternative techniques that have been developed in the last decades into two broad categories (Chiesa, 2001; Poh et al., 2001), according to their basic working principles (see Figure 2.1): 1.

weighting and ranking techniques, which seek to weight the characteristics of different R&D projects and to develop a rank of preference for these projects, on the basis of their relative weights. The most common techniques included in this category are the scoring methods (Dean and Nishry, 1965; Bradbury et al., 1973; Balachandra et al.,

54

Evaluation and performance measurement of R&D R&D project evaluation techniques

Weighting and ranking techniques

Scoring methods

Figure 2.1

2.

Comparative methods

Benefit contribution techniques

Methods for economic analysis

Methods for decision analysis

Categories of R&D project evaluation methods

1996) where the overall ‘merit’ of each project in the evaluation set is mathematically calculated ‘per se’, and the comparative methods (Pessemier and Baker, 1971; Easton, 1973; Ormala, 1986; Calantone et al., 1999; Bozbura et al., 2007) where the ‘merit’ of a project is computed by comparison with either another project or a set of alternative projects. In both methods, the ‘merit’ of the project is (usually) a multi-dimensional concept, including both quantitative (that is, economical) and qualitative criteria; benefit-contribution techniques, which calculate the contribution of each project to the overall value creation objectives (expressed in economical terms) of a company/organization. This category comprises methods for economic analysis (Freeman, 1982; Ormala, 1986; Irvine, 1988; Locke, 1990; Brealey and Myers, 1996), which follow a capital budgeting approach for calculating the economic return of a project, and methods for decision analysis (Morris et al., 1991; Carlsson and Fullér, 1996; Clemen, 1996; Trigeorgis, 1996; Armstrong et al., 2004), which treat a project as a sequence of multiple (at least two) selfinfluencing decisions and calculate its economic return by considering the potential occurrence of different sequences.

As the main purpose of the chapter is to investigate where different techniques perform at their best, a further step is needed to define the key dimensions along which the capability of each technique to support R&D project evaluation can be analysed and assessed. Appraising the ‘merit’ of each technique against each of these dimensions would allow us to identify the ideal conditions for its application. The key variables are:

R&D projects

1.

2.

3.

4.

55

ability to deal with multiple objectives. R&D projects are by nature characterized by multiple objectives, most of which are also difficult to measure with a quantitative variable, ranging from increasing firm profitability, to improving technological competences and strengthening innovation performance. In this respect evaluation techniques that are structured to solve a multi-objectives problem have higher ‘merit’ than those that are not; ability to deal with risk and uncertainty. All R&D projects are characterized by a relevant degree of risk that their established targets are not achieved, both from a technological and commercial point of view, as well as by a high degree of uncertainty associated with the potential evolution of the context. As a consequence, the ability of an evaluation technique to explicitly measure risk and/or to take into account uncertainty increases its overall ‘merit’; simplicity. An evaluation technique has a higher ‘merit’ when its results are relatively easy to interpret by the company’s managers and the data required to run the technique are easily available, thus maximizing the efficacy of the evaluation technique as a decision support system; cost of implementation. Other things being equal, evaluation techniques that are less expensive to design and introduce are preferred to those that entail high costs of implementation. It is worth highlighting that the cost of implementation is not necessarily related to the simplicity of the technique. A technique requiring a huge amount of data, for example concerning the details of costs and revenues associated with the R&D projects under evaluation and providing, after a complex mathematical treatment of data, a scaled ranking of projects, may have the characteristic of simplicity, particularly if data are ‘easily’ available as they are recorded in the firm’s management control system. However, the design of such an evaluation system and its implementation, including for example an ad hoc extraction of data from existing databases, may require high costs.

The above list of features has been identified as described in the following. A first set of criteria has been drawn from a thorough analysis of the available literature on the topic (Hall and Nauda, 1990; Georghiou and Roessner, 2000; Poh et al., 2001). These criteria have been subsequently discussed in a focus group with the aim of identifying only the most critical variables. The focus group comprised ten R&D senior managers (including six R&D directors) and eight business development managers from eight Italian firms ranked in the top 15 R&D spenders1 in 2006. Two subsequent meetings of the focus group were devoted to the discussion of the above criteria. In the first meeting, the comprehensive list of criteria

56

Evaluation and performance measurement of R&D

derived from the literature (and including nearly 13 criteria) was reduced to a short list of six variables based on the experience of the managers involved in the evaluation of R&D projects. In the second meeting, after a further literature investigation on the remaining criteria, four of them were paired, leading to the final version of the list presented above. The next section focuses on each of the evaluation techniques identified in Figure 2.1. In particular, after briefly presenting their working principles and usual application fields, their ‘merit’ is discussed against the key features for R&D project evaluation.

2.3 2.3.1

WEIGHTING AND RANKING TECHNIQUES Scoring Methods

Working principle and application fields The principle at the basis of scoring methods is rather simple (Dean and Nishry, 1965). Assume that there is a known set of n factors (for example, availability of key technologies required, availability of manufacturing facilities, expected sales life of product, additional service requirements) that are considered to be relevant for determining the chance of success of a project or, in other words, its ‘merit’ against the company’s objectives. Each factor is provided with statements that define a scale of evaluation, which ranges from the lowest value (usually 1), corresponding to a marginal ‘merit’ of the project against the specific factor, to the highest value (usually 5), corresponding to the fulfillment of the requirements needed to ensure a greater chance of success. For example, as far as the factor ‘availability of key technologies’ is concerned, the corresponding scale may range from ‘key technologies protected by patents and controlled by our competitors’ to ‘key technologies fully available and already applied within the company’. Assume that the relative importance of these factors is also known and measured by a set of n weights (usually expressed as to make their sum equal to 1). The values of the weights can be determined by means of a statistical analysis on historical data of completed projects or alternatively through a subjective definition undertaken by the members of the committee in charge of project evaluation. Projects are then evaluated by assigning a score on the defined scales for each of the above n factors. The weighted average score (see Equation 2.1) achieved by each project represents its ‘merit’ in the evaluation and allows for defining a rank of the projects under analysis. The higher the ‘merit’, the better the project.

R&D projects

57

n

Si 5 a wj # sji

(2.1)

j51

where Si is the total score of the i project, wj is the weight of the j factor over the total n factors representing the evaluation criteria, and sji is the score of i project against the j factor.2 A simple sensitivity analysis, considering the effect on the overall ranking of a slight variation in factor score achieved by each project, can be performed for assessing the identified rank order and support the results of the evaluation. Similarly, and particularly when the number of projects evaluated is rather large, it is useful to run a statistical test to verify rank-order correlations of unweighted and weighted project scores, and thus assess the sensitivity of results to variations in weights. Hwang and Yoon (1981) propose an interesting variation of a traditional scoring method. The technique, named TOPSIS (Technique for Order Preference by Similarity to Ideal Solution), views the problem of evaluating projects as a geometric space with n dimensions, one for each factor defining the ‘merit’ of the project. The space has upper and lower boundaries determined by the scale range defined for each factor. Factors are weighted as in the classical scoring method. Projects are points in the n-dimensional space and their coordinates along the dimensions are determined by the score achieved against each factor. The technique is based on the concept that the best project is the one with the shortest distance from the positive-ideal solution (that is, top scoring in all factors) and the longest distance from the negative-ideal solution (that is, worst scoring in all factors). Both these distances are reflected in an index, called similarity, that measures the relative closeness to the positive-ideal solution. Projects are ranked on the basis of their similarity. A representation of TOPSIS is provided in Figure 2.2, where, for the sake of simplicity, a bi-dimensional space is considered. Appendix A2.1 discusses the case of a multinational firm that applies a complex scoring method to evaluate and prioritize its R&D projects. The case study is useful because it explores the organizational implications that using this type of technique entails, which represents a further unexplored topic in the existing literature. In particular, it shows how a scoring method can be applied along the different phases of the R&D project management process (screening of the ideas, approval of the projects, development of the projects), with different levels of detail and with different purposes. The case also suggests that the scoring method might represent an effective means of coordination and communication between different hierarchical levels and organizational units, justifying the high costs that running this type of evaluation system in a complex organizational setting might entail.

58

Evaluation and performance measurement of R&D

Dimension X2

Upper boundaries

Project 1

Lower boundaries

Dimension X1

Figure 2.2

An example of application of the TOPSIS technique

Literature documents the use of these techniques to evaluate both industrial and government R&D projects. As far as industrial projects are concerned, scoring methods are applied for instance in the corporate R&D division of an Indian company involved in the manufacture of heavy electrical equipment (Rengarajan and Jagannathan, 1997), in the British Military Aircraft and Aerostructures (Farrukh et al., 2000), and in the Japanese Sumitomo Electric Industries, or SEI (Osawa and Murakami, 2002). These techniques are also employed in the evaluation of R&D projects undertaken by government agencies, like the Rehabilitation Services Administration and the National Institute of Handicapped Research (Cardus et al., 1982), a US federal research laboratory (Henriksen and Traynor, 1999), or the Kuwait Institute for Scientific Research (Al-Mazidi and Ghosn, 1997). Assessment of key features Scoring methods are by definition able to deal with multiple objectives as they include a number of factors that can be heterogeneous (qualitative and quantitative) in nature. Therefore, they are able to evaluate R&D projects under as many different perspectives as the company considers relevant for taking its decision. The use of fixed scales to evaluate the ‘merit’ of projects against each

R&D projects

59

factor allows management of qualitative information, but it results in a rather poor assessment of economic (quantitative) data, where it is difficult to spot differences in the profitability profile of the R&D projects under evaluation. Moreover, scoring methods usually do not explicitly consider the amount of financial resources to be invested to start the project,3 therefore it is rather difficult to translate a slight difference in ranking (for example, project A totalling a score of 3 and project B a score of 4) into an investment decision when projects require investments with significantly different scales (for example, project A requiring €1 million and project B requiring €10 million). As far as the ability to deal with risk and uncertainty is concerned, scoring methods show their major weaknesses. Indeed, even if they may eventually treat risk as one of the n factors judged relevant for the evaluation, they are not able to take into account the mediating effect of risk on all the other n − 1 factors, that is, to catch the real nature of risk in R&D projects. The same is true for what concerns uncertainty about potential context evolution. Scoring methods are usually relatively simple as the scores of different projects are easy to manage and also results are clearly understandable. Moreover, they usually entail a low cost of implementation and, assuming that evaluators are available within the firm, they also require a very short time for elaborating results. Simplicity and costs, however, highly depend on the procedures required for data collection and analysis. In other words, if the evaluation is highly centralized (for example it is the responsibility only of the R&D unit), the collection of the required data is no more than a ‘low cost’ detailed description of the projects. In other cases, as for example the one discussed in Appendix A2.1, the evaluation might involve a high number of different functions, and data for assessing the score of the project have to be collected from a number of sources in different organizational units, this implying also a relevant effort of homogenization. The evaluation procedure is here more complex and more costly, even if it does not imply the calculation of cash flows. 2.3.2

Comparative Methods

Working principle and application fields Comparative methods evaluate the ‘merit’ of a project by comparison with another project or with a set of alternative projects. Using a relative evaluation, these methods do not identify an absolute rank against a set of criteria (as scoring methods do), but simply order projects under evaluation from the best to the worst. Among comparative methods, AHP – analytical hierarchy process – is certainly the most widely studied and applied (Saaty,

60

Evaluation and performance measurement of R&D Process R&D

Manufacturing

Capability

Facilities/ Capital Profitab. equipm. investm.

PROJECT A

Source:

Financial

Unit cost

Success probab.

Technical

Cost

Marketing/ Distribution

Time Resources Potential Capability Trends

PROJECT B

Adapted from Liberatore and Titus (1983).

Figure 2.3

Example of a hierarchy for the evaluation of R&D projects

1980; Rahaman and Frair, 1984; Liu and Xu, 1987; Donegan et al., 1991). Developed at the beginning of the 1980s, AHP structures the evaluation problem into a hierarchy. The overall goal of the evaluation (for example, the creation of value for the firm) is at the top of the hierarchy, followed by the main criteria (corresponding to the factors of the scoring methods) that influence the achievement of the goal, and perhaps by a number of subcriteria further detailing the evaluation variables. Figure 2.3 shows an example of a hierarchy for the evaluation of R&D projects. The method runs as follows. Each criterion (and subcriterion) is subject to a pairwise comparison with all the other criteria (and subcriteria) at the same hierarchical level. Comparisons are usually based on the evaluation scale developed by Saaty (1980) including odd numbers ranging from 1 to 9, where aij is equal to 1 if criterion i is equally important with criterion j (that is, they provide the same contribution to the overall goal of the evaluation) and aij is equal to 9 if criterion i should be absolutely preferred to criterion j. The results of the pairwise comparisons are collected in a squared matrix in which all the elements on the main diagonal are equal to 1 (being the result of the comparison of each criterion with itself) and all the elements below the main diagonal are the inverse of the corresponding elements above the main diagonal.4 The normalized eigenvalue of the squared matrix represents the vector of the weights of each criterion. In other words, differently from scoring methods, AHP provides weights as a result of the first iteration of the method. Once criteria (and eventually subcriteria) have been weighted, alternative projects are then evaluated in a pairwise comparison against each (sub) criterion using the same Saaty’s scale, where the score measures the degree to which each project meets the requirements in comparison to the other project. Again, the normalized

R&D projects

61

eigenvalue of the resulting squared matrix represents the relative score of projects in each (sub) criterion. Finally, alternatives are ranked by calculating their overall weighted score according to Equation 2.2. n

1

j51

k5m

Si 5 a asij # q wjk b

(2.2)

where Si is the overall score of the i project, sij is the normalized eigenvalue representing the score of the i project against the j criterion, and wjk is the normalized eigenvalue representing the weight of the j criterion at the k level of the hierarchy. As the hierarchy is usually made of more than one level, the total weight of the j criterion is calculated by backward induction of the weights of all the criteria that link the j criterion to the top of the hierarchy. As in scoring methods, sensitivity analysis is also usually performed in AHP. Triantaphyllou and Sanchez (1997), for example, developed a methodology for performing sensitivity analysis on the weights of criteria, in order to determine the most critical variables in the evaluation, that is, those whose small changes cause a switch in the rank of one or more alternatives. An interesting evolution of AHP proposed by Saaty (1996) and further developed by Lee and Kim (2000) and Meade and Presley (2002), is the ANP – analytic network process. ANP is claimed to be more appropriate for the characteristics of R&D projects. Indeed, by construction, AHP creates a unidirectional hierarchical relationship between the top element of the model (that is, the overall goal of the evaluation) and the other criteria and subcriteria. The hierarchy, therefore, decomposes the general into more specific attributes, until a level of manageable evaluation is reached. However, more than in other fields of application of comparative methods, in R&D projects evaluation criteria are highly interrelated among each other: for example, ‘distance from the state-of-the art of technology’, which could be used as a subcriterion for decomposing the evaluation of potential technical success, is also highly intertwined with the ‘expected timing of market entry’, which is rather a subcriterion for evaluating the market potential of the project. If the two subcriteria (as in the example) do not belong to the same hierarchical line, they are never compared against each other in the evaluation process, and this introduces a potentially relevant distortion in the resulting rank order of the projects. To overcome this problem, ANP allows two-way relationships between criteria (and subcriteria), even those belonging to different levels of the hierarchy. Pairwise comparisons are then performed within the same hierarchical level but also among interrelated criteria (and

62

Evaluation and performance measurement of R&D

subcriteria), despite their position in the overall hierarchy. ANP finally manages comparisons creating matrixes similar to those used in AHP and ends by providing a rank of project alternatives. Although it fits the evaluation of R&D projects better, the complexity of ANP, which increases significantly the number of comparisons and requires more sophisticated mathematical transformations on data, hinders its diffusion in real industrial applications. The use of comparative methods, and particularly AHP, in the evaluation of R&D projects, has been largely documented in the literature. For instance, Calantone et al. (1999) describe the application of AHP in a large division of a major US firm heavily involved in industrial NPD. Meade and Presley (2002) illustrate the use of AHP for the evaluation of the R&D projects carried out by a small high-tech company in the southwest of the United States, whereas Liang (2003) investigates the case of YMC, a Taiwanese automobile manufacturing company, Kamoda and Sugawa (2008) report the use of AHP in the R&D division of OMRON Corporation, and Shin et al. (2007) in the Korean national nuclear R&D projects. Similarly, Forman and Gass (2001) describe several successful applications of the AHP, for example in Xerox Corporation, NASA’s Lyndon B. Johnson Space Center, Northeast Fisheries Science Center (NEFSC), and Air Products and Chemicals, Inc. Assessment of key features Comparative methods are able to deal with multiple objectives as they include a number of factors that can be heterogeneous (qualitative and quantitative) in nature. Pairwise comparisons using a fixed scale, however, introduce the problem of the violation of the principle of consistent modelling (Marshall and Oliver, 1995) by combining related measures (for example, if criterion A is absolutely to be preferred to B, meaning that aab is equal to 9, and the same is true for B in comparison with C, the comparison of A and C should lead to a more than absolute prevalence, that is, more than 9 as pairwise value, which is not allowed by the method). Moreover, comparative methods suffer from the ‘rank reversal problem’ (Belton and Gear, 1982): where a new project alternative and/or criterion is added, a number of new pairwise comparisons are performed on the existing alternatives/criteria against the new one, eventually resulting in a change in the rank order of the original alternatives (even if the evaluators do not change the original results of the pairwise comparisons on existing alternatives/criteria).5 It is therefore critical to identify in advance the complete hierarchy of criteria and the alternatives to be evaluated, as adding additional options after the initial run of the method could lead to non consistent results.

R&D projects

63

As far as the ability to deal with risk and uncertainty is concerned, comparative methods have the same weaknesses of scoring methods when it comes to assess the mediating effect of risk (and uncertainty) on all the other criteria used for the evaluation. A potential solution is provided by ANP, where interactions among criteria can be explicitly taken into account. It has been already noted, however, that there is a trade-off between the ability to deal with interactions and the increase in the overall complexity of the evaluation system. Comparative methods are rather simple and straightforward to use (the ‘mathematical’ transformations being obviously managed by software). Data and results are relatively easy to manage and understand. In particular, as far as qualitative issues are concerned, comparative methods have an advantage in comparison with scoring methods. In fact, they do not evaluate projects using an absolute reference value that may be difficult to assess, but rather compare the relative expected performance of each pair of projects (Saaty, 1980). The cost of implementation is rather low. The elaboration of results, however, is more complex than for scoring methods as the number of pairwise comparisons that have to be performed by the evaluators is significantly higher (for example, a model with five criteria at the same level and five project alternatives requires 90 pairwise comparisons, whereas the same model would require only 25 single evaluations in a scoring method). 2.3.3

Applying the fuzzy logic to weighting and ranking techniques

Weighting and ranking techniques heavily rely on the knowledge of experts performing evaluations, that is, assigning a score to a single project or identifying a relative preference over two alternatives, which is a typical qualitative judgment. This brings into the evaluation a potential error due to the vagueness of human thoughts and perceptions (Beskese et al., 2004) and to the difficulty of translating them into a fixed evaluation scale. To overcome this problem, in the mid 1960s, Zadeh (1965) developed the concept of ‘fuzzy logic’, that is, a theory set to deal with reasoning that is ‘approximate rather than precise’. Accordingly to fuzzy logic a statement can have a degree of truth that ranges from 0 to 1, and it is not constrained into the classic Boolean logic of the true/false opposition. In the evaluation of R&D projects, ‘fuzzy logic’ allows the user to introduce and manage linguistic scales. For example, the statement ‘the project fits with the firm’s technology strategy’ has an associated answer ranging from ‘very low’ to ‘very high’ and usually comprising five potential ‘values’ (very low, low, medium, high, very high) expressed in a linguistic form. One might argue from this example that ‘fuzzy logic’ simply translates a

64

Evaluation and performance measurement of R&D y

1

x 0 ␮(x) =

Figure 2.4

a (x–a) / (b–a) (x–c) / (b–c) 0

b

c

1

a≤x≤b b≤x≤c otherwise

An example of triangular membership function

Likert-like scale using linguistic expressions rather than numbers. On the contrary, ‘fuzzy logic’ transforms each linguistic expression into mathematical functions μ(x) in the [0, 1] space, defining whether or not (and at what degree) the linguistic expression belongs to a given ‘fuzzy’ set. These functions (Kerre and Mordeson, 2005), as human thoughts and perceptions, partially overlap, thus re-creating the continuum of value associated with a given evaluation. For the sake of computational efficiency and ease of data acquisition, triangular membership functions are often used, even if alternative forms (for example, trapezoidal) are available. An example of triangular membership function is presented in the figure below (Figure 2.4). The triplet (a; b; c) is normally used to define triangular membership functions. Evaluation scales based on the translation of linguistic expressions can be therefore represented by a sequence of triplets as for example (Wang and Liang, 1993): ‘not important’ equals the triplet (0; 0; 0.3), ‘little important’ (0; 0.3; 0.5), ‘moderately important’ (0.2; 0.5; 0.8), ‘important’ (0.5; 0.7; 1), ‘very important’ (0.7; 1; 1), where the overlap is evident. Zimmermann (1978, 1991) and Hajek (1998) gave a boost to the application of ‘fuzzy logic’ in evaluation processes by developing basic arithmetic operations on the above functions. This allows for linguistic expressions to be treated (with sum and multiplications) in the same way as discussed in the previous subsections for traditional weighting and ranking techniques. The result of the evaluation is again a ‘fuzzy number’, a mathematical function covering part of the [0, 1] space. Projects are ranked according to their centre of gravity (Toletti and Noci, 2000; Azzone and Rangone, 1996), that is, the projection on the x-axis of their geometric centre (or barycenter). The lower the distance between the ‘centre of gravity’ of an R&D project and the top value (1), the higher is its position in the overall ranking.

R&D projects

65

‘Fuzzy’ versions of weighting and ranking techniques have been developed so far and widely discussed in the literature (for example, Stam and Duarte Silva, 1997; Bozbura et al., 2007).

2.4 2.4.1

BENEFIT-CONTRIBUTION TECHNIQUES Methods for economic analysis

Methods for economic analysis are currently the most diffused methods for the evaluation of R&D projects (Schall et al., 1978; Irvine, 1988; Ryan and Ryan, 2002). Although the existing methods largely differ in their implementation, they all share a common principle, that is, the capital budgeting approach for calculating the economic return of a project as a sequence of discounted cash flows. The basic procedure of capital budgeting, widely known as NPV (Net Present Value) calculation, consists in discounting all future cash flows (both in- and out-flow) resulting from the project with a given discount rate and then summing them together (see Equation 2.3). The ‘merit’ of the project is measured considering its contribution to the creation of economic value out of the investment needed. In its basic application the discount rate is calculated looking at the ‘real’ cost of capital employed in the project, that is, by calculating the weighted average cost of equity and debt used to finance the project. For small projects,6 where it is rather difficult to identify the quotas of equity and debt used for financing a single project, the cost of capital – also named WACC (Weighted Average Cost of Capital) – is usually assumed to equal the cost of capital of the whole company, that is, it is calculated using annual report data that take into account the firm’s overall equity and liabilities. NCT (t) NPV 5 a t (1 1 r) t

(2.3)

where NCF(t) is the Net Cash Flow generated by the project in year t7 and r is the discount rate, usually equal to WACC. Many scholars have criticized the use of NPV for the evaluation of R&D projects (for example, Kester, 1984; Hodder and Riggs, 1985; Chapman and Cooper, 1987; Brealey and Myers, 1996). Their criticism has focused on: 1. The calculation of the discount rate The discount rate in the evaluation of R&D projects should be made of two elements (Doctor et al., 2001):

66

Evaluation and performance measurement of R&D

(a)

a risk-free rate accounting for general risk and usually considered to be the interest rate offered by short-term government bonds; (b) a risk-premium rate to take into account the perceived risks (financial, technical and commercial) associated with the specific project. The use of WACC, on the contrary, allows the evaluator to account only for the financial risk and does not differentiate among projects. In practice, several companies use, for discounting future cash flows, ‘hurdle rates’ (15–20 per cent) over the actual WACC, in the attempt to account also for technical and commercial risks. The highest the discount rate, however, the highest the probability of discarding both R&D projects with relevant potential returns, although realized in the long term, and small projects with relatively low but rather secure returns in the short term.8 In these cases the distortion introduced in the evaluation by the use of a ‘general purpose’ hurdle rate, instead of a project-specific discount, is very clear; 2. The definition of cash flows in the long term horizon The use of NPV requires definition of the exact value of the cash flows to be discounted for each time period considered in the evaluation. This value could be rather difficult, if not impossible, to determine for those R&D projects having great potential in the long term, but for which managers are unable to make a proper evaluation. Again, NPV appears to discriminate unreasonably against longer term and more risky projects. In order to address these issues and to develop methods for economic analysis able to deal also with longer term and higher risk projects, a number of adjustments have been proposed to traditional NPV. Among them, in the remainder of this section, three methods will be discussed in detail: 1. 2. 3.

risk-adjusted NPV, also known as RAR; certainty equivalent (CEQ) NPV; stochastic NPV.

Working principle and application fields Risk-adjusted NPV Risk-adjusted NPV (Locke, 1990) is based on the assumption that the discount rate should be adjusted to consider explicitly the aforementioned two basic elements: the risk-free rate and the projectspecific risk premium. The former rate is widely available, as it is taken directly from short-term government (usually US) bonds, while the latter is typically calculated through the capital asset pricing model (CAPM). The CAPM (Sharpe, 1964; Lintner, 1965; Mossin, 1966) is widely accepted

R&D projects

67

in financial markets and establishes a relation between the performance of a given stock traded on the capital market and the average performance of the whole market. The difference between the performance of a given stock and that of the whole market is usually represented by a ratio index b. An index value greater than 1 means that the stock is highly ‘volatile’, amplifying the performance (both positive and negative) of the market and therefore being characterized by an above average risk, whereas an index value lower than 1 means that the stock is less risky than the average of the stocks traded on the capital market. Following the analogy with the financial market it is possible to treat an R&D project like a portfolio of stocks that replicates the returns estimated for the project. The risk associated with the R&D project can therefore be calculated as the weighted average of the risks (in terms of b) of the stocks included in the portfolio. The higher the risk, the higher the discount rate. Indeed, the project-specific risk premium is calculated by multiplying the average market risk for the resulting b and therefore the whole discount rate is equal to: r 5 i 1 b # rm

(2.4)

where i is the risk-free rate and rm is the average market risk rate. The NPV calculated with the above discount rate better accounts for project-specific risk, avoiding misapplication and misinterpretation of the results for R&D projects. The IEA (International Energy Agency) went further in 2003 in the application of risk-adjusted NPV for the calculation of the discounted value of energy production from different sources (hydrocarbon, wind, solar, hydro), suggesting that different discount rates should be applied to different categories of costs and revenues related to a given project. For example, power generation systems based on hydrocarbon have a higher cost ‘volatility’ for their basic inputs (mostly related to oil prices) than systems based on solar or hydro energy. Therefore, the discount rate for input costs should be higher in the former than in the latter cases. This approach, also known as component NPV (O’Brien, 2005), even if it can further reduce error in applying NPV valuation and represent an interesting advance in risk-adjusted NPV theory, has a limited ‘real-word’ applicability due to its extremely high complexity. Literature documents the use of risk-adjusted NPV applications for R&D project evaluation in different industries, and in particular in biotech and pharmaceutical companies. Among others, Projan (2003) illustrates the use of risk-adjusted NPVs to compare the value of development projects for different antibacterial agents, whereas Liu and Wen

68

Table 2.1

Evaluation and performance measurement of R&D

Example of application of certainty equivalent in an assetintensive firm

Activities

Perceived risk (mean s2 of NCFs)

a1

a2

a3

a4-onward

Research

(0–0.2) (0.2–1)

0.80 0.70

0.75 0.60

0.70 0.60

0.60 0.30

Development

(0–0.2) (0.2–1)

0.95 0.90

0.90 0.80

0.85 0.80

0.80 0.75

Product renewal

(0–0.2) (0.2–1)

0.95 0.90

0.95 0.90

0.90 0.80

0.85 0.80

Source: Adapted from Chiesa (2001).

(2006) describe the use of the same technique to estimate the value of Antibody-168 (discovered by AbGenomics), which is a promising candidate for the development of drugs against psoriasis. Certainty Equivalent NPV The certainty equivalent, developed by Robichek and Myers (1965, 1966) and underlying a well known model for evaluating financial options (Merton, 1973; Black and Scholes, 1973), is the certain cash flow that a risk-averse investor would be willing to exchange for a risky cash flow. In other words, instead of adjusting the discount rate, certainty equivalent NPV adjusts future cash flows generated by the project taking into account their risk through introducing a coefficient a, ranging from 0 to 1 as in Equation 2.5. It is worth mentioning that, the risk of each cash flow being neutralized by the use of the coefficient a, the discount rate to be applied for calculating the NPV is the risk-free rate. NPVCEQ 5 a t

at # E (NCF (t)) (1 1 i) t

(2.5)

where E(NCF(t)) is the expected value of cash flows in the year t and i is the risk free rate. It follows immediately that the higher the risk associated with a given cash flow (either because it is expected in the long term or it is related to a high ‘volatile’ input), the lower the value of the coefficient a. Tables (for an example see Table 2.1) can be created for use as a reference framework in the evaluation of R&D projects in a given company. Although certainty equivalent NPV has been claimed as one of the most appropriate methods for R&D project evaluation (Azzone and Bertelè,

R&D projects

69

1998; Chiesa, 2001), its applications documented in the literature are rare. Stochastic NPV Stochastic NPV goes further in considering each component of cash flow as a stochastic variable, with a given distribution of probability (usually a normal distribution), a mean value and a variance. For example, the revenues of a project aimed at substituting an existing product in a consolidated market might be characterized by a relative low variance and a mean value that is closely linked to current revenues. However, if the same project requires a technology the company has never managed before, development costs might be characterized by a huge variance and have as mean value the result of a reverse engineering effort made by the company on existing applications of this technology. NPV is therefore, in turn, a stochastic variable with an associated probability distribution (Chapman and Cooper, 1987; Ho and Pike, 1992; Raftery, 1994). The expected value of the NPV is obtained as: E (NPV) 5 a t

E (NCF (t)) (1 1 i) t

(2.6)

where E (NCF (t)) is the expected value of the net cash flow in each year t and i represents the risk-free rate. Also in this case, indeed, risk is neutralized in the estimation of cash flows. The overall risk of the project can also be calculated by the dispersion of the values of NPV, that is, through its variance, standard deviation or dispersion coefficient. The use of stochastic NPV in real-world R&D project evaluation has not been systematically documented in the literature so far. A relevant exception is the case, discussed in Azzone and Bertelè (1998), of the use of this technique to assess the development project for a new image processing software. A number of software and simulation tools are however commercially available to manage evaluations using stochastic NPV, which suggests a rather widespread use of this method in practice. Assessment of key features The main shortcoming of methods for economic analysis is their inability to deal with multiple objectives. Indeed, even if in the most complex formulation of these methods each component of a project can be isolated and treated as a stochastic variable, it must always be ‘transformed’ into a monetary value of revenue or cost. As a consequence, soft and intangible factors (for example, the impact of the project on the company’s competence basis or on its reputation) are usually excluded from the evaluation, or have to be considered separately (Grant-Muller et al., 2001).

70

Evaluation and performance measurement of R&D

On the contrary, the ability to deal with risk and uncertainty is one of the major advantages of the methods for economic analysis. All the three methods discussed above, indeed, allow consideration of technical, financial and commercial risk associated with the project. It is obvious that the more precise the analysis of risk, the more complex the implementation of the method, requiring an ad hoc analysis of each component of cash flows or the creation of far more difficult to manage financial portfolios. Methods for economic analysis are rather complex in their usage (even if calculations are obviously managed by software) as they require a relevant effort in defining different components of cash flows and, in the case of stochastic NPV, their distribution of probability. At the same time, however, results are more straightforward to cope with and to understand as they are expressed in monetary terms and assess the ‘merit’ of a project through its contribution to the creation of economic value for the company. The cost of implementation highly depends on the accuracy of the method, that is, on how many variables are individually considered in the evaluation, but is in general greater than that required for weighting and ranking techniques. 2.4.2

Methods for Decision Analysis

The techniques discussed so far share a common view of the project as a ‘black box’. They allow evaluators to distinguish the project’s basic components and, in some cases, to manage them in a different way (for example, risk-adjusted NPV allows the use of different discount rates for different typologies of costs or revenues), but for all of them starting the project is a pure Boolean (0, 1) decision: the project starts if its overall ‘merit’ is higher than a defined threshold, whereas it is stopped if its overall ‘merit’ falls below this threshold. However, in real-world evaluations, several R&D projects are started under a ‘try-and-see’ assumption. Research activities are often undertaken, even if the overall ‘merit’ of the project is below the threshold (or it can not be calculated with a reasonable level of confidence), under the assumption that some of its intermediate results will provide new insights on the real potential of the project and offer the chance for a more detailed future evaluation. In other words, there is a valuable characteristic of most R&D projects that is not considered in ‘black box’ evaluation methods, that is, their intrinsic operating flexibility. The vast majority of R&D projects, indeed, can be modelled like a sequence of interconnected steps, where each step is characterized by the achievement of a given (and measurable) intermediate result. Once an intermediate result is achieved and compared with the expected target, managers have the chance to stop or continue, or even to modify the project. The same flexibility applies to

R&D projects

71

project costs. Indeed, each step is interconnected but independent from the others, so that only after the decision to continue the project are the majority of the costs needed for the next step actually sustained. For example, in the pharmaceutical industry, the development of a new drug can be divided into five steps: pre-clinical tests, three steps of clinical tests, and the review of the New Drug Application (NDA). The results of each step offer new information about the real potential of the new drug and allow for better evaluating the ‘merit’ of the project and hence for taking a more appropriate decision to sustain the costs associated with the next step. Starting from these premises, some evaluation methods have been more recently developed that explicitly account for the value of R&D projects’ flexibility. Among them, in the remainder of this subsection, two methods will be discussed in depth: 1. 2.

decision tree analysis, also known as DTA; real options valuation, also known as ROV.

Working principle and application fields DTA Decision tree analysis is an established method having its roots in normative decision theory (Savage, 1954) and in decision analysis theory (Magee, 1964; Raiffa, 1968). Only in the last decade, however, has it been applied to the evaluation of R&D projects (Clemen, 1996; Doctor et al., 2001). Its working principle is relatively simple. The method starts by defining the basic outline of the project path from inception to completion and distinguishing its basic steps: each step is characterized by a well defined intermediate result representing a project’s milestone (for example, in pharmaceutical research, the R&D project passes clinical tests on healthy individuals showing a degree of tolerance at normal dosage higher than 90 per cent).9 Management decisions only occur at defined moments (represented as nodes in the tree) in correspondence to (and based on) the achievement of planned milestones. Potential outcomes of these decisions are usually two (binomial tree): 1. 2.

success, and in this case the project proceeds to the next development stage; failure, and in this case a liquidation value might be obtained by using the project for other purposes or selling it off.

Once the tree has been constructed, probabilities of success are defined for each node, usually starting from the root towards the ending nodes.

72

Evaluation and performance measurement of R&D pII success pI success

Phase II

Phase I

failure 1 – pII failure 1 – pI

Figure 2.5

V′C – C I′ – C II′

C I′I – C – C’ C’II′II

– C I′

The Decision Tree Analysis (DTA)

Standard protocols have been developed for the definition of probabilities of success relying on domain experts (Morgan and Henrion, 1990). Alternatively, probabilities of success can be obtained from a statistical analysis of past projects (the mean and the standard deviation of the probability distribution are required) or even from industry averages (for example, Di Masi et al., 2003, for the pharmaceutical industry). The construction of the tree is completed by assigning a time-scale to each step, that is, the expected time needed to achieve the milestone, and by calculating pay-offs at the end branches of the tree (see Figure 2.5). In a binomial tree, pay-offs are rather simple to calculate: 1.

2.

at the upper right end of the tree, corresponding to the final success of the project, the pay-off is the discounted10 value of cash in- and out-flows related to commercialization less discounted cash out-flows (costs) of all the development phases; at the other end-branches, corresponding to failures, pay-offs are the discounted value of cash out-flows (costs) of the development phases already implemented until the failure occurred, plus the discounted value of cash in-flows from liquidation.

The ‘merit’ (expected value) of the project is calculated at the inner node of the tree using backward induction: n

m

EV 5 a aPay 2 offi q pjb i51

(2.7)

J (i) 51

where i are the end-branches of the tree, Pay-off i are the discounted payoffs associated to each end-branch, j(i) are the branches of the tree that comprise the path from the inner node to the i end-branch, and pj is the probability of occurrence related to each j branch.

R&D projects

73

The expected value for the tree shown in Figure 2.5 can be calculated, therefore, as follows: EV 5 (2C9I ) * (1 − pI ) 1 pI * [(2C9I 2 C9II) * (1 2 pII) 1 (V9C 2 C9I 2 C9II) * ( pII)]

(2.8)

Different projects are ranked according to their expected value (under the assumption that projects with a negative expected value are discarded) and then selected starting from the highest value. Decision trees explicitly consider the value of flexibility, allowing for a better evaluation particularly of complex, multi-step R&D projects. Further evolutions of decision trees are multiple-outcomes decision trees (for example, Briggs et al., 2003) and influence diagrams (Howard and Metheson, 1981). Multiple-outcomes decision trees allow for explicitly considering more than two outcomes for a given step. A project showing an intermediate result below expectations, that can however be considered ‘acceptable’, might be for instance modified or re-scaled to achieve a less ambitious objective, that is, an alternative path to completion might be built for the project. The method’s working principles are the same as for binomial trees. However, multiple-outcomes decision trees are far more complex as they require calculation of pay-offs for a number of end-branches growing exponentially with the number of alternative paths created, and assessment of the probability of occurrence for each potential outcome. Influence diagrams give the opportunity to consider the probabilistic dependence of different steps. For example, if a project for a new drug passes the first phase of clinical tests (that is, the one concerning the tolerance of assumption in healthy individuals), then the probability increases that the same drug passes also the third phase of clinical tests (that is, the one concerning the efficacy of the new drug and the study of clinical interactions), because some of the potential interactions with other drugs and side-effects could be already excluded. Influence diagrams are graphical representations of decision problems where the probabilistic dependences among different steps are explicitly represented by either the presence or absence of arcs between the nodes. The expected value can be calculated only through simulation (see also this chapter’s subsection about Monte Carlo simulation), that is, by running a number of occurrences and calculating the mean of the expected value resulting for each occurrence. Literature documents a widespread use of decision tree analysis in the pharmaceutical industry, where a project for a new drug is subject to a

74

Evaluation and performance measurement of R&D

series of pre-clinical and clinical tests and the product launch will only occur once the project has been successfully completed across all its steps. For example, Kellogg et al. (1999) illustrate how the value of a biotechnology firm, Agouron Pharmaceuticals, Inc., can be estimated as the sum of the values of its current drug projects, whereas Viscovich and Krogsgaard (2006) use decision tree analysis to evaluate the R&D project for a new drug candidate for the treatment of osteoporosis. ROV The Real Options Valuations for R&D projects is the real investment counterpart of financial option theory used in capital markets. Kester (1984) was among the first to point out that traditional methods for valuing R&D projects basing on discounted cash-flows clearly prefer riskaverse behaviours, whereas the highest profits are usually gained through higher risk investments. At the same time, he argued that an investment in an R&D project is in fact rather similar to buying a financial option resulting in the possibility, not the obligation, of commercializing a new product. Modelling, as with DTA, an R&D project as a sequence of stages, the ROV considers that at each stage the firm has at least two choices: going on with the development of the project or abandoning it. The owner of the option (that is, the firm carrying out the R&D project) at the end of each stage therefore has the possibility, and not the obligation, to exercise the option (that is, to continue with the project development). Only in the case that the firm decides to continue the project, does it sustain the additional investments needed, and gains the profit deriving from the commercialization of the project’s results (when the project reaches its end). In a financial option, the owner of the option retains the right, but not the obligation, of buying a certain quantity of shares or a commodity such as oil or gold on capital markets at a specified price at a specified future date (European options), or before a specified future date (American options). When the future date arrives (in the case of European options) the owner of the option can decide whether or not to exercise the option. The owner will exercise the option if the market price of the share or the commodity is higher than the price specified in the option contract, resulting in a profit proportional to the difference between the market price and the option price. If the market price is lower than the option contract price, the owner will allow the option to expire and the loss will be limited to the amount of money originally invested in the option. The above discussion has been focused on the ‘option to abandon’ a project at a given stage of its development. This kind of option, which is rather similar to the binomial modelling of the tree commonly used in DTA, is actually the most diffused in ROV. In reality, however, a number

R&D projects

Table 2.2

75

Different typologies of real options

Typology of real option

Brief description

Key references

Option to abandon

Projects can be abandoned and some of the project specific assets and/or intermediate results (if any) can be sold to the market.

Myers and Majad (1990)

Option to defer

Projects can be temporarily halted waiting for additional information and/or investigation on the actual potential outcomes.

Tourinho (1979); McDonald and Siegel (1986); Luehrman (1998a); Paddock et al. (1988); Dixit and Pindyck (1994)

Option to stage

Projects can be articulated in ‘stand-alone’ stages and stopped also in the meantime if new information is unfavourable

Majad and Pindyck (1987); Carr (1988); Trigeorgis (1993)

Option to scale

Projects can be expanded, if conditions are favourable, or vice versa scaled down to better fit incoming information about the potential of outcomes.

Brennen and Schwartz (1985); McDonald and Siegel (1985); Andreou (1990); Dixit and Pindyck (1994); Bell (1995)

Option to switch

Projects allow changes in their input mix or outputs in response to changes in related market conditions.

Kensinger (1987); Kulatilaka and Trigeorgis (1994); Kamrad and Ricardo (1995)

Strategic growth options

Projects that are at the root or represent a key link in a chain of interrelated projects, whose success is highly dependent on the ones under scrutiny.

Myers (1977); Kester (1984, 1993); Trigeorgis (1988); Kogut and Kulatilaka (1994); Willner (1995); Lint and Pennings (1998)

Compound options

The options that result from the interaction of two or more of the above options related to a single project.

Brennen and Schwartz (1985); Kulatilaka (1993); Trigeorgis (1993, 1996); Luehrman (1998b)

Source:

Adapted from Benaroch (2001).

of alternative choices (for example, the choice of switching the mix of inputs or outputs for the project, or the choice of halting the project and waiting for future information) are available. An overview of the different typologies of real options identified in the literature is provided in Table 2.2.

76

Evaluation and performance measurement of R&D

In the evaluation process, the first step is to model what kind of option (among the ones listed in the table) best represents the choice the firm will face in the development of the project. Then, the ‘merit’ of the project is calculated as the value of the option, that is, the real counterpart of the price at which a financial option with the same characteristics will be traded on the capital market. Scholars have made an effort to consider how different factors in real world R&D project evaluation influence the value of real options: 1.

2.

3.

the value of the option increases if managerial flexibility is higher, that is, the larger the range of choices available for the firm regarding the project (Lint and Pennings, 2001); the value of the option increases as the variance of the expected value of net revenues minus commercialization costs increases (McGrath, 1997; Boer, 2000). As the potential loss is limited to the initial investment (that is, to the cost of buying the option), an R&D project which shows high impact opportunities with a modest or low probability of success does not imply high risk; the value of the option increases the longer the time to expiry. In other words, an investment opportunity is a more valuable option if the investment is in the early stages of the product development cycle (Bollen, 1999).

The most common method to determine the value of an ‘option to abandon’ is the Black and Scholes Model (Black and Scholes, 1973), directly derived from the application in financial markets. Without entering into the details of the calculation,11 it is useful to list the input variables used for the evaluation and their financial counterpart (see Table 2.3). A number of alternative models have been developed to evaluate different kinds of real option. The Geske model (Geske, 1979) is based on the Black and Scholes formula, but is extended with one or more extra decision moments, that is, it allows for considering a multi-stage decision process. The Margrabe model (Margrabe, 1978) allows for explicitly considering the uncertainty in the investment cost. The Carr model (Carr, 1988) combines both the characteristics of the Geske and the Margrabe models. Finally, the Stulz model (Stulz, 1982) allows for considering the switch options concerning inputs or outputs in the project development. Even if these models provide interesting theoretical insights on the effects of key variables in the evaluation process, their mathematical complexity has hindered their widespread application. Once the evaluation is made, different projects are ranked according to the value of the options they underlay and then selected starting from the highest value.

R&D projects

Table 2.3

Real option and financial option value drivers

Variable

Real option value drivers

Financial option value drivers

S

Present value of expected cash flows (estimated looking at the market potential of the project output)

Current value of the stock

s

Project value uncertainty (volatility)

Volatility of share price

K

Present value of investment costs

Exercise price of the option

T

Time until investment opportunity disappears

Time to expiry (fixed date)

i

Risk-free interest rate

Risk-free interest rate

Source:

77

Adapted from Cox and Rubinstein (1985).

The literature documents the use of ROV for R&D project evaluation in a number of different industries. For example, Pennings and Lint (1997) report the application of this technique to evaluate an R&D project undertaken in Philips Corporate Research, whereas Boer (2002) identifies three groups of firms making use of real option valuation techniques: pharmaceutical and biotech companies (in particular, Merck, Eli Lilly, Baxter International, Amgen, Genentech, Genzyme, and Smith & Nephew), petroleum companies (for example Mobil, Chevron, Petrobras, Texaco, Conoco, and Anadarko Petroleum) and energy firms (for example Dynegy, Amerada Hess, Duke Energy, and Aquila Energy). Furthermore, Boer indicates that at least half a dozen consulting companies offer real option valuation services for R&D projects, among them PriceWaterhouseCoopers, Navigant, and KPMG. Other scholars, however, acknowledge that the practical use of real options is still limited by their underlying complicated mathematics and the strict assumptions they are subject to when applied to real-world problems (Fredberg, 2007). Assessment of key features Methods for decision analysis, like methods for economic analysis, focus their assessment on the economic value of the project and are therefore less complete in comparison with weighting and ranking techniques. However, they have some advantages over methods for economic analysis (Lint and Pennings, 2001; Brandao, 2004). Indeed, they:

78

1. 2.

3.

Evaluation and performance measurement of R&D

consider explicitly the value of managerial flexibility; define a number of stages and milestones, with their mutual relationships, resulting in an explicit evaluation of the operational (other than economical) characteristics of the project; allow, at least from a theoretical point of view (the actual application being in most cases prevented by the high mathematical complexity), for considering a larger number of choices than a simple go–no go decision.

Methods for decision analysis perform at their best in the ability to deal with risk and uncertainty. Indeed, each component of risk (technical, financial and commercial where applicable) is explicitly considered in each stage of the R&D project development. Uncertainty is treated (particularly in ROV) as a stochastic variable and its effect on the ‘merit’ of the project is properly calculated. The construction of the decision tree is not only time consuming but can also be extremely messy when the problem is large and complex (Raiffa, 1968) and the same is true for ROV when the firm models the project with different typologies of options for different stages (see for example Miller and Clarke, 2005). This is particularly true for those development processes with an integrated design, manufacturing and rollout, which is for example a characteristic of the software industry (Lint and Pennings, 2001). Results are easy to manage as they are expressed in monetary values, but their understandability is rather low unless evaluators are experts in financial mathematical estimations. The cost of implementation is relatively high, even if software solutions are widely available for basic applications, as these methods require a relevant effort in designing the evaluation model before applying it to the firm’s projects. 2.4.3

Monte Carlo Simulation

In high risk environments the definition of the potential outcomes from a given stage of the project development and their implication on the next stages can be very complex. A valuable alternative is to use simulation modelling techniques to find the alternatives with the highest probability of occurrence. The Monte Carlo technique (Rubinstein, 1981) draws its name from the famous gambling centre and is based on the following idea. If it is possible to identify the different parameters that affect the value of a project during its lifetime and to estimate their mean value and random variation, the interactions of these parameters and their combined effect on the value of the project can be ‘calculated’ through simulating a large

R&D projects

Table 2.4

79

Key features of different classes of R&D projects evaluation techniques

Weighting and ranking techniques ● scoring methods ● comparative methods Benefit contribution techniques ● methods for economic analysis ● RAR ● CEQ ● Stochastic NPV ● methods for decision analysis ● DTA ● ROV

Ability to deal with multiple objectives

Ability to deal with risk and uncertainty

Simplicity

Cost of implementation

+++ ++

+ +

+++ ++

+++ +++

+ + +

++ ++ ++

++ ++ +

++ ++ ++

++ ++

++ +++

+ +

+ +

Notes: + = the technique does not perform well along this dimension; ++ = the technique performs well along this dimension; +++ = the technique performs very well along this dimension.

number of random outcomes rather than through complex mathematical formulation. The results form a large number of computer simulations produce an ‘empirical’ diagram of probabilities across a range of potential outcomes. For example, in a profitability model made of revenue, costs and profit, both the revenues and costs can be varied within parameters (also related to operational variables) defined by the firm, allowing the profit to be derived from the potential combinations. Monte Carlo simulation has become established as a tool to support complex evaluation models, like DTA and ROV, reducing time and complexity of implementation (Cowles and Carlin, 1996; Doctor et al., 2001; Spiegelhalter et al., 2002; Robert and Casella, 2004).

2.5

R&D PROJECT EVALUATION TECHNIQUES: A COMPARATIVE ASSESSMENT

Table 2.4 summarizes the discussion reported in the last section on the key features of the different classes of R&D project evaluation techniques. The

80

Evaluation and performance measurement of R&D

profile of each class of techniques, moreover, has been discussed in depth in the focus group mentioned at the beginning of the chapter. It immediately follows, from the analysis of the table, that the techniques are listed according to their increasing degree of complexity and implementation costs, whereas there is a clear trade-off between weighting and ranking techniques, which have as a key characteristic the ability to deal with multiple objectives, and methods for decision analysis, which on the contrary have a pronounced ability to cope with risk and uncertainty. Methods for economic analysis are able to deal with risk and uncertainty and are characterized as well by lower complexity and implementation costs than for DTA and ROV. In the previous sections, for each technique some cases of real-world applications reported in the literature have been reviewed and presented. The analysis synthesized in Table 2.4, however, allows for a further generalization of the application fields for the different classes of evaluation methods, which brings us to identify their ‘ideal’ setting, that is, the context where they can be more satisfactorily applied to evaluate R&D projects. Several taxonomies of R&D projects have been proposed in the literature so far. It is beyond the scope of this chapter to provide a complete overview of this body of research. In the remainder of this section two widely accepted classifications will be used to illustrate the most appropriate application context for the different classes of evaluation methodologies. The first taxonomy, already mentioned in the introduction of the book, distinguishes between basic research, applied research and new product development projects. Basic research projects are usually characterized by a high degree of risk and uncertainty. This suggests that using techniques able to deal with these variables, that is, methods for decision analysis, should be the preferred alternative. However, the absence of a direct relationship with industrial applications and the typical explorative nature of basic research projects, make it very challenging to estimate with an acceptable degree of confidence the input variables (mean value and variation as well as project stages/milestones and probability of success) required for applying DTA or ROV. As a consequence, those techniques (for example, scoring methods) that allow for including in the evaluation non-economic aspects and intangible factors, such as the value of advancing the knowledge in a given scientific field, should be preferred. In applied research projects, on the contrary, the definition of input variables for the complex methods of decision analysis is usually feasible. DTA and ROV techniques become therefore the preferred methods in those instances (for example, in large projects) where minimizing the

R&D projects

81

impact of risk and uncertainty is a key issue in the evaluation process. Small projects, where the impact of risk and uncertainty is not critical, can be alternatively evaluated using scoring and comparative methods, where the benefit associated with including intangible aspects in the evaluation is reinforced by the lower cost and complexity of the evaluation. Finally, in new product development projects, methods for economic analysis should be preferred. Indeed, in these projects the closeness to the market allows a more reasonable estimation of future cash flows and the ability of methods for economic analysis to deal particularly with technical and commercial risk ensures the best conditions for the evaluation. Moreover, in new product development projects, where the final outcome is already very well defined, managerial flexibility has only a marginal ‘value’ and does not justify the adoption of more complex (and costly) techniques. The second taxonomy of R&D projects takes into account their degree of innovativeness. In other words, an R&D project might entail a minor modification of an existing entity, usually as the result of a continuous improvement process. This is called an incremental R&D project. Alternatively, it might involve the discovery of entirely new entities, hence representing a discontinuity with existing conditions. In this case it is called a radical R&D project. The distinction between incremental and radical is usually associated with the degree of newness of the result of the R&D effort, and is not necessarily associated with its economic impact (Henderson, 1993; Chiesa, 2001). Radical projects have typically the capability to open up new windows of opportunity for the firm, like accessing new markets or entering new businesses. The most appropriate evaluation methods should be able therefore to explicitly take into account the value of proactive flexibility that is intrinsic in the nature of such projects. As a consequence, techniques for decision analysis, and particularly ROV, are best suited to these evaluation contexts as they allow the building of complex alternative paths toward R&D project completion and take into account the intrinsic multi-phase nature of radical innovation. Incremental R&D projects are, on the contrary, destined to reinforce the firm’s position in the businesses in which it is used to operating. The maximization of the generated profits, in a context characterized by a relatively lower risk and uncertainty, is the key issue in the evaluation of such projects, suggesting that methods for economic analysis should be preferred in these instances. It is worth mentioning that a track record of market and technical outcomes of previous projects would allow using stochastic techniques without a disproportionate increase in cost. These arguments are summarized in Table 2.5; it is obvious that the taxonomies, depicted in Tables 2.4 and 2.5, are not mutually exclusive.

82

Evaluation and performance measurement of R&D

Table 2.5

R&D project evaluation techniques and their field of application

R&D project typology by scope ● Basic research ● Applied research

●

New product development

by degree of innovativeness ● Incremental innovation ● Radical innovation

Most suitable evaluation technique Scoring methods Methods for decision analysis (for large projects) / Scoring – Comparative methods (for small projects) Methods for economic analysis Methods for economic analysis Real options valuation

Therefore, evaluators have to check through both taxonomies for determining the suggested evaluation methods. In case there is a complete discordance on what kind of technique to use, the choice should be informed by a deeper investigation of the characteristics of the project (for example, degree of risk, availability of information about inputs and outputs). This section of the chapter has adopted as unit of analysis the single R&D project. However, evaluating the ‘merit’ of R&D projects represents just a first step of a more complex ‘evaluation’ process that ends with the definition of a bundle of projects that will be initiated and undertaken in a given period. In other words, once evaluated on an individual basis, R&D projects have to be integrated into a comprehensive R&D program (or projects portfolio). However, the optimal project portfolio does not necessarily descend from the combination of single, optimal projects (Keeney, 1987). When analysed from an integrated perspective, a number of potential conflicts (for example, on the use of physical resources) between optimal R&D projects are likely to be unearthed, that could not be identified in individual evaluations. Therefore, two further critical questions should be addressed when evaluating R&D projects at the portfolio level: 1.

2.

what are the types of conflicts among R&D projects belonging to the firm’s portfolio and which approaches can be employed to address them? which criteria can be employed to evaluate the adequacy of a firm’s R&D project portfolio and its consistency with the firm’s overall R&D and innovation strategy?

R&D projects

83

The second section of this chapter will address these further critical issues in R&D project evaluation, therefore adopting the firm’s R&D project portfolio as unit of analysis.

2.6

R&D PROJECT PORTFOLIO ANALYSIS

Cooper, Edgett and Kleinschmidt (2001, p. 362), defining the concept of portfolio management as ‘a dynamic decision process, whereby a business’s list of active new product (and R&D) projects is constantly up-dated and revised’ stress the need for ‘periodic reviews of the total portfolio of all projects (looking at the entire set of projects, and comparing all projects against each other)’. The combination of projects that have the greatest ‘merit’ when considered individually does not necessarily produce the optimal project portfolio. The main reasons underlying a potentially sub-optimal combination can be identified as follows: 1.

2.

The complexity of the interdependences and the emergence of potential conflicts among different projects in the portfolio. Indeed, some of the characteristics of individual projects are additive (for example, cost of external inputs), in the sense that the total portfolio retains the same characteristic as the linearly added sum of the related projects; some other characteristics (for example, the overall expected return, the overall usage of internal resources) might be super-additive or minus-additive, in the sense that non-linear effects (for example, cannibalization of returns among different projects) affect the overall portfolio. In some cases, as for example where an over-allocation of internal resources is in place, these non-linear effects might also lead to the unfeasibility of the project portfolio; Not all the key factors determining the ‘merit’ of a project portfolio are necessarily included in the evaluation of individual R&D projects. For example, as happens with financial portfolios, one of the key indicators of the adequacy of a project portfolio is the achievement of a certain degree of risk diversification, obtained through balancing high risk and low risk projects (Cooper et al., 2001). This aspect cannot be considered when individual evaluations are performed. Other key factors, for example the consistency of the project portfolio with the firm’s product and market strategy (Cooper et al., 1998) or with the desired geographical coverage of markets (Hall et al., 1992), even if they can be assessed in the evaluation of individual projects (for example, strategic consistency or localization could be used as criteria

84

Evaluation and performance measurement of R&D

in a comparative method), can be evaluated only when the overall picture is clear, that is, when the portfolio is built. A survey has rated portfolio management as the weakest area in new product management (Cooper and Kleinschmidt, 1995), but a number of scholars and practitioners have addressed the above issues. The following subsections provide an overview of these scholarly contributions. 2.6.1

Analysing Interdependences among Projects

Interdependences among different projects in an R&D portfolio have been widely debated in the literature. A first stream of research is focused on distinguishing different typologies of interdependences that may take place in real world portfolios. In the attempt to provide a unifying taxonomy, Chien (2002) summarizes the existing contributions by highlighting the following four typologies of interrelation: 1.

2.

3.

4.

outcome or technical interdependences, which occur when the outcome or some of the stages of a project are related to the outcomes of one or more other projects, even beyond those constituting the project portfolio (Weingartner, 1966; Stummer and Heidenberger, 2003); serial interdependences, which occur when there are time relationships among different projects that have not been considered in previous, individual evaluation (Baker and Freeland, 1975; Martino, 1995). For example, a project (or a stage of development of a project) may start only when another project in the portfolio has ended; cost or resource-utilization interdependences, which occur when different projects share the same resources or costs (Aaker and Tyebjee, 1978). The evaluation and selection of single projects is done under the implicit assumption that all the physical resources (space, equipment, workforce) required for carrying out the project would be available when needed during its actual development. However, when the total portfolio of projects is considered it is often the case that several of the physical resources (for example, project managers, designers, laboratory machines) are over-allocated, that is, the linear sum of the requirements of the selected projects exceeds the available capacity. It might also happen that the selected projects overcome for example the quota of R&D budget required for the acquisition of new assets; impact or benefit interdependences, which occur when the payoffs of different projects are not additive (Aaker and Tyebjee, 1978; Zeleny, 1982; Fox et al., 1984), for example when there is a cannibalization effect between two or more projects.

R&D projects

85

Interdependences should be addressed as they: 1.

2.

3.

may require a rescheduling or more generally an intervention on one or more project’s plans (for example, when serial interdependences emerge and changes in the time-plan for the interrelated projects are required to provide time consistency among different projects); may lead to the unfeasibility of one or more of the selected projects (for example, when some of the inputs considered in the individual project evaluation become unavailable); may require a full reconsideration of the project portfolio (for example, when negative benefit interdependences are in place that reduce the overall ‘merit’ of the selected portfolio below the firm’s expectations).

A number of methods for addressing interdependences among projects have been developed so far in the literature. They can be distinguished, according to their nature, into two main categories: 1. mathematical optimization models These models are usually based on mathematical programming that explicitly considers project interdependences and are, for the most part, derived from finance portfolio theories addressing correlation problems among securities in an investment portfolio (Markowitz, 1959; Ingersoll, 1987). For example, Weingartner (1966) developed a binary (0-1) integer model for addressing outcome and cost interdependences. Nemhauser and Ullmann (1969) developed a dynamic programming model, based on optimal capital allocation theories, for addressing cost interdependences. Bell and Read (1970) developed a model for maximizing the portfolio’s economic value by assessing resource-utilization and serial interdependences. Finally, Fox et al. (1984) use binary integer programming to model benefit interdependences, called present-value interactions, in order to optimize the overall project portfolio profitability profile. Alternative and more complex mathematical models, involving the use of polynomial programming (Gear et al., 1971; Glover and Woolsey, 1974), of quadratic integer programming (McBride and Yorkmark, 1980) and sensitivity analysis (Canada and White, 1980), have also been discussed in the literature, as well as ‘hybrid’ models like the one proposed by Ghasemzadeh et al. (1996), which integrates integer linear programming with AHP to handle qualitative and quantitative measures of projects’ interdependences. Mathematical optimization models provide a solution to projects’ interdependences by modifying (by exclusion or inclusion of previously discarded projects) the composition of the total portfolio. Although these methods have been largely debated in the literature, their application in practice is rare (Archer and Ghasemzadeh, 1999)

86

Evaluation and performance measurement of R&D

because they have a number of important shortcomings – the need to collect a large amount of data (as interdependences have to be quantified and inputted into the model for each project), the inability to include all typologies of interdependence and their inherent complexity (Souder, 1978). 2. Heuristics models. These models usually provide practical procedures or ‘greedy’ algorithms for addressing infeasibilities due to projects’ interdependences in a given portfolio. For example, Peerenboom et al. (1989) developed an algorithm based on the calculation of the cost-effectiveness of different selected projects. Available resource units are then progressively assigned to the projects that present the highest effectiveness. The final portfolio will result from the exclusion of less cost-effective projects. Other authors (Golabi et al., 1981; Husseiny, 1981; Merkhofer and Keeney, 1987; French, 1988) have suggested applying a multi-attribute selection technique (similar to the comparative methods discussed for individual project evaluation) to evaluate the ‘merit’ of alternative portfolios created by the firm (through the inclusion or exclusion of other projects) to address infeasibilities. Finally, other contributions focus on portfolio adjustments that include changes to individual project characteristics (for example, duration, performance goal, time-plan). Archer and Ghasemzadeh (1999) developed a five-stage framework for project portfolio management, suggesting the use of visualization tools (matrixtype displays) for analysing interdependences and valuing the effects of changes. Chiesa (2001) discusses at length how to solve interdependences by combining one or more of the following three actions: (a)

discard the interrelated project. This is the simplest solution and is particularly suitable when there are only a few cases of interrelation, especially when they concern projects that were not highly ranked in the evaluation system; (b) increase the resources allocated to R&D activities so as to include interrelated projects. This solution is relatively straightforward to implement and is particularly suitable when the number of interrelated projects is relatively low and the problem involves top scoring projects; (c) modifying project characteristics (for example, valuing an alternative input source or re-scale the project’s objective) to reduce interdependences. 2.6.2

Evaluating the ‘Merit’ of an R&D Projects Portfolio

As far as the issue of identifying the key attributes or performance criteria of a portfolio of R&D projects is concerned, Table 2.6 summarizes the

R&D projects

Table 2.6

87

Selected criteria for evaluation of an R&D project portfolio

Authors

Key attributes of R&D portfolio

Notes

Cooper et al. (1997)

The authors identify three broad objectives that usually dominate the projects’ portfolio decision process: ● effectiveness: alignment of the mix of projects in the portfolio with the strategic goals of the firm; ● efficiency: value of the portfolio in terms of: ● long-term profitability, ● return-on-investment, ● likelihood of success. ● balance: diversification of the projects in the portfolio in terms of various trade-offs such as high risk versus sure bets, internal versus outsourced work, even distribution across industries.

The paper represents the first of a series of pieces of research done by the authors on the topic (see also Cooper et al., 1998, 2001, 2002, 2004).

Cooper et al. (2001)

The authors identify six metrics to capture the performance of the R&D project portfolio: ● projects are aligned with business’s objectives; ● portfolio contains very high value projects; ● spending reflects the business’s strategy; ● projects are done on time (no gridlock); ● portfolio has good balance of projects; ● portfolio has the right number of projects.

The paper reports the result of a recent survey of project portfolio management practices and performance of 205 companies operating in North America, undertaken in collaboration with the Industrial Research Institute. The paper shows that best performers, that is, those companies resulting in the top 20% rank of each metric, really excel (that is, present the highest distance from worst performance) in two metrics: ● achieving an overall balance of the portfolio; ● having the right number of projects for the available resources.

De Reyck et al. (2005)

The authors, discussing project portfolio management, identify among others the following key issues:

The authors conducted an online survey of 125 companies operating in the IT business, with the aim of

88

Evaluation and performance measurement of R&D

Table 2.6

(continued)

Authors

Key attributes of R&D portfolio portfolio has a good project diversification. Categorization of projects at the portfolio level is used to balance the mix of projects; portfolio has a good balance in the risk versus reward profile of the selected projects; portfolio has a good alignment with firm’s strategy.

assessing the impact of project portfolio management on information technology projects. Risk-reward analysis resulted as the key issue in maximizing results of project portfolio.

The authors propose a balanced score card-like (Kaplan and Norton, 1992) approach aimed at producing a balanced representation of the firm’s portfolio performance. The key dimensions are therefore: ● financial: assessing the overall economic contribution of the portfolio; ● market: assessing the market coverage and overall acceptance of selected projects; ● internal growth: assessing the consistency between the project portfolio and firm’s development strategies; ● innovation: assessing the consistency between the project portfolio and firm’s innovation strategies.

The paper introduces a methodology for the construction of portfolios of R&D projects, based on an extended data envelopment analysis (DEA) model that quantifies some of the qualitative concepts embedded in the balanced scorecard approach.

●

●

●

Eilat et al. (2006)

Notes

most relevant literature contributions in this area and provides an overview of the proposed attributes.

2.7

CONCLUSIONS

This chapter addresses the problem of evaluating and prioritizing industrial R&D projects undertaken by private firms. In the first part, relying on a systematic review of the literature and a focus group with a number of experienced managers, it illustrates a taxonomy of the main methods

R&D projects

89

that can be applied to evaluate R&D projects. For each class of evaluation techniques, special attention is given to describe application fields documented in the literature and to assess its characteristics mapped along four critical dimensions: ability to deal with multiple objectives, ability to deal with risk and uncertainty, simplicity and cost of implementation. This analysis allows us, in the second part of the chapter, to develop and illustrate a framework that identifies the types of R&D projects (basic research vs. applied research vs. new product development, radical vs. incremental) to which each technique appears to be more appropriate. This model has been developed relying on the opinion of the managers involved in the focus group mentioned above and a critical analysis of the available literature. At the end, the chapter addresses the problem of how to build a balanced portfolio of R&D projects, taking into account a number of relevant variables such as: project interdependence, conflict of resources, level of risk and temporal distribution. Finally, Appendix A2.1 focuses on the organizational implications associated with the use of a scoring method, which represents one of the most widely adopted techniques for R&D project evaluation. The chapter is believed to contribute to the existing research into R&D project evaluation because it addresses two important gaps in this body of research. First of all, it is one of the few contributions that compare different methods for R&D project evaluation with the aim of identifying their most appropriate field of application. Second, it documents, although with an exploratory intent and through the case study reported in Appendix A2.1, the organizational complexities and costs that putting into practice a scoring method for R&D projects might require. This suggests a number of insights for future investigation. First, scholars are encouraged to assess on a statistical basis the efficiency and effectiveness of different methods applied in different evaluation settings (for example, different types of R&D projects, different industries and hence different levels of risk and uncertainty). This would require the development of appropriate measures of proficiency for an evaluation technique applied to R&D projects (for example, managers’ satisfaction with it, implementation costs, organization-wide impacts), the selection of a relevant sample of firms undertaking different types of R&D projects and operating in different industries, and the statistical comparision of the efficiency of different techniques. A second avenue for future research, which is suggested by the case study reported in Appendix A2.1, is the analysis of how R&D evaluation techniques can be integrated with a stage-gate system for R&D project management and used to monitor the in-progress advancement of the project, perhaps including different elements in the evaluation when different stages of development are reached.

90

Evaluation and performance measurement of R&D

APPENDIX A2.1:

ORGANIZATIONAL IMPLICATIONS OF A SCORING METHOD

This appendix describes in detail the use of a very structured and formalized scoring method for the evaluation of R&D projects by the Italian subsidiary of a multinational firm headquartered in the USA. The case study is interesting for a number of reasons. First of all, it shows how the use of a methodology for the evaluation of R&D projects has a number of significant organizational implications. The case study indicates that, in order to put the scoring method into practice, it is necessary to involve people from different functions and hierarchical levels, and to introduce a formalized evaluation process. These aspects entail significant organizational complexity and costs, which should be carefully considered when the characteristics of the evaluation technique to be employed are decided, and especially its degree of formalization. Because the evaluation technique is deeply embedded in the organization in which it is used, however, it might represent an important coordination mechanism, able to stimulate communication and organizational learning. It is interesting to note that these organizational implications of the use of R&D project evaluation techniques have been rather neglected by extant research. The case study reported in this appendix can represent therefore a starting point for future research into the topic. The case study also suggests that an evaluation methodology can be employed along different phases of the innovation management process (screening of the ideas, approval of the projects, management of the projects), with different levels of detail and with different purposes. This further demonstrates the usefulness of this managerial tool as a means for sharing the results and the ‘in progress’ status of R&D projects across different levels and units of the firm’s organization, and not only for selecting and prioritizing them. The evaluation technique employed by the firm in our case study represents ‘per se’ an interesting example of how a scoring method can be designed to take into account a number of heterogeneous aspects (financial performance, time, technical and commercial risks, benefits for the corporation as a whole), most of which have an intangible nature and hence would be difficult to capture through the use of other evaluation approaches. The case study that is presented was built through a number of direct personal interviews with key informants, and especially with the Six Sigma Business Leader, who is responsible for the coordination of the different R&D projects that are activated each year, with an engineer working in

R&D projects

91

the R&D department, with two different project managers, who have had the responsibility for the evaluation process of several R&D projects in the last two years, and, finally, with the director of human resources. Introduction Company B (the real name has been blinded for confidentiality reasons) is a multinational enterprise headquartered in the USA and a worldwide leader in the supply of flow equipment products and systems for the oil and gas industry. It has an annual turnover of €4 bln (2007), it employs 12,500 people and has about 200 plants located all over the world and more than 55 brands. In 2006, Company B acquired an Italian firm (Company C) specialized in the development and manufacturing of actuators used in oil and gas pipelines, whose main purpose is to command the opening and closure of valves. The Italian firm, which is the subject of this case study, was integrated in the Drilling and Production Systems division of Company B, and had to conform to the procedures for the generation, evaluation and selection of innovation and R&D projects employed by Company B. Company C has a turnover of about €45 mln and it employs 180 people organized into eight main divisions (see Figure A2.1.1). The R&D department belongs to the engineering division, it has a staff of four engineers and is responsible for the management of new product development projects and for the non-routine testing procedures necessary for obtaining quality certifications. The R&D activities of Company B are managed in a very decentralized manner: the enterprise has 20 R&D departments located in the different subsidiaries all around the world. About 3 per cent of the annual turnover is invested in R&D. The Innovation Process The innovation process in Company B is organized around three main phases: 1. 2. 3.

screening of ideas; approval of the projects; management of the projects.

Screening of ideas The screening of the ideas for new products or processes is undertaken at the level of the local subsidiary (Company C). This phase opens with a set of brainstorming sessions to identify several ideas for potential

92

Evaluation and performance measurement of R&D Director of the subsidiary ERP & IT

Six sigma

Human resources

Product & Customer service management

Inside sales

Contract administration

Figure A2.1.1

Marketing

Engineering

After sales

Manufacturing

Quality assurance

Finance

R&D department

The organizational structure of Company C

innovation projects. Employees working in the Product & Customer Service Management, Engineering and Manufacturing functions (see Figure A2.1.1) take part in these sessions, which are generally held in the last months of the year. The cross-functionality of this team improves its capability to generate novel ideas, significantly enhances their quality level, and reduces the need for corrections and loops that is typical when idea generation is left in the hands of R&D only. The brainstorming sessions identify about ten to twelve innovative ideas, which are prioritized using a scoring method (explained in detail in the subsection ‘The scoring method’). This identifies on average four to five projects per year which move forward in the innovation process. The scoring method used at this stage of the process is the same as the one that is employed later on, in the phase of approval. Nevertheless, the identification of the scores for each project, along the different evaluation criteria, is based here on a subjective assessment undertaken by the brainstorming interfunctional team. In the phase of approval, the implementation of the scoring method is instead much more formalized and based on structured input data. Approval of the projects The approval of the project is undertaken at the divisional level (Drilling and Production Systems division of Company B). In this phase, for each of the four to five most promising projects previously selected, an interfunctional evaluation team of people from Company C is established. Key persons in this team are: 1.

the project manager, who is responsible for the coordination of the team;

R&D projects

2.

93

the product manager, who has responsibility over the evaluation of the project; at least one representative from the Product & Customer Service Management, Engineering and Manufacturing functions.

3.

The evaluation team, on average 12 people, undertakes all the detailed analysis required for the evaluation of the project and prepares a number of standardized documents that are presented to the director of the division for formal approval. The documents to be presented are six: ● ●

●

●

●

●

Project Charter (see Figure A2.1.2). It provides a synthetic description of the main characteristics of the project; Project Scope (see Figure A2.1.3). It illustrates in a qualitative way the impact of the new product on the resources of the subsidiary in all the phases of development, manufacturing and commercialization; Revenue & Margin Calculations (see Figure A2.1.4). This document is prepared by the Product & Customer Service Management function, and it illustrates the expected revenues and margins (calculated considering only industrial costs) associated to the new product; Risk Management Plan (see Figure A2.1.5). In the left section of the document, an assessment of the commercial and technical risk factors associated with the new product is reported, based on a scoring evaluation method. The right section of the document is used, in case the project is approved, to identify the actions that can be undertaken to reduce the level of risk (risk management section); Development Costs (see Figure A2.1.6). It estimates the costs required for the development of the new product. It takes into account both the resources needed to identify (through interviews and focus groups) the needs of the potential customers, and those involved in the development (for example, engineering, testing, prototyping) of the new product; R&D Project Submittal Form (see Figure A2.1.7). This document reports the result of the application of the scoring system (described in detail in the subsection ‘The Scoring Method’), which uses as inputs the data collected in the five documents mentioned above and comes out with a figure describing the contribution of the project to the firm’s economic value.

On the basis of these six documents, the different project proposals are discussed during a number of meetings or conference calls in which the head of the division (Drilling and Product System), the project manager, the product manager and the other members of the evaluation team participate. After introducing the modifications that might emerge during these

94

Evaluation and performance measurement of R&D Project Charter Project Title Project Category Revision

Dated:

Submitted by:

Identify target customers Identify target market Define Opportunity Define customer benefits/needs

Internal External

Identify project catalyst Create statement of opportunity

Define financial costs, benefits and uncertainty Identify business risks Assess Opportunity Link project to strategic and operating plan Link project to product/service roadmap Define competitive threats

Product manager Execution team members & roles Project Team and Schedule

Tollgate reviewers Identify out of scope items Create project goal statement Define Project metrics Project timeline (tollgate reviews)

Define

Measure Analyse

Design

Verify

Source: Adapted from Company C’s internal documents.

Figure A2.1.2

Project charter

meetings, the different projects are prioritized on the basis of the outcome of the scoring method reported in the R&D Project Submittal Form, and the ones with the highest rank are approved, on the basis of the available annual budget (on average two to three projects are approved per year). This evaluation and selection process might take months to complete. During this time, new ideas for R&D projects might come to light, that are included in the evaluation process. The formalization of the evaluation process, at

R&D projects

95

Project Scope

What is the basic product concept ? How many variations (sizes and so on) are likely to be required ? What testing is required ? What support products are required ? Facility resources Required Human resources requirements Health, safety and environmental considerations … Source:

Adapted from Company C’s internal documents.

Figure A2.1.3

Project scope Increased Sales Revenue & Margin

Description

Qty

Type

Sell Price

Revenue

Cost

Margin % Margin

Current Sales 2007 Current Sales 2007 Total Additional Sales Additional Sales Totals

Source:

Adapted from Company C’s internal documents.

Figure A2.1.4

Revenue and Margin calculations

least in the second step where the corporate division is involved, has a very important role in favouring coordination between different functions in the local subsidiary and between the local subsidiary and the parent company, improving communication and information sharing. Management of the projects After approval, the selected R&D projects are officially launched. Each of them goes through a number of different steps during its development (see Figure A2.1.8):

96

Evaluation and performance measurement of R&D Project Risk Management Plan To Be Completed For Every Project

Risk items Likelihood Impact Priority (potential of risk to (likelihood future item project x impact) problems occurring if risk derived from (1–10) item does brainstorm session) occur (1–10)

To Be Completed If Project Approved Actions to reduce the likelihood of risk occurring

When should actions be complete?

Status Likelihood Is new of of risk likelihood of risk actions item occurring less than after original? actions (1–10)

BRMP (Business Risk Management Plan) Market stability Not establishing our market position

TRMP (Technical Risk Management Plan) Not covering all customer specific requirements

Source:

Adapted from Company C’s internal documents.

Figure A2.1.5 1.

2.

3.

4.

Project risk management plan

Define. At this stage the characteristics of the project, in terms of purposes, temporal milestones and resource consumption, are defined. Obviously, these are delineated considering that other R&D projects have been activated, that compete for the same resources; Measure. The purpose of this step is to analyse the so-called Voice of the Customer (VOC). The needs of both internal and external customers are investigated in order to produce a list of prioritized requirements that serve as inputs for the subsequent phase of the process; Analyse. Different design and engineering alternatives for the new product are evaluated, to identify the one that is best able to properly satisfy the requirements defined in the last step of the process; Design. On the basis of the technical and functional specifics defined in the last step of the process, the detailed design and engineering of the product is undertaken.

R&D projects

97

VOC INTERVIEW STRATEGY Customer Types: Previous VOC performed By…….. Interviewer: Local Salesman primary, Product Manager alternate Positions to Interview: Instrumentation Leads, Buyers, VAC Sales & Technicians Focus: Requirements, Added Features, Common Failures, and so on DEVELOPMENT COST ESTIMATES: VOC Expenses (USD) Location User Travel

Customer 1 2 3 4

yes yes yes no

A B C D Sub-Total

Engineering Hours

Hotel

n/a

n/a

n/a

n/a

Rate / Hr

Qty

Total $

$ Marketing Expenses (USD) (English, Russian, Chinese, Spanish)

Sales Brochures Translations Technical Data Sheets IOMs 3D Models Animations Sub-Total TOTAL DEVELOPMENT COST ESTIMATE (USD):

$

SALES PROJECTIONS (USD thousands):

Source:

Actual for Customer: A B C D

Qty

$

2nd year after launch for Customer: A B C F

Qty

$

Adapted from Company C’s internal documents.

Figure A2.1.6

Development costs

Interviewer

98

Evaluation and performance measurement of R&D R&D Project Submittal Form Submitted by:

PRODUCT MANAGER

Date:

Project Name: Team Members: Project Classification Description of the Project: Aspect Increased Revenue

Increased Margin

Estimated Margin %

Development Duration

Importance of ‘Time to Market’

Ratings 1 = <$1.500M 2 = $1.5M–$4.5M 3 = >$4.5M 1 = <$0.75M 2 = $0.75M–$2.25M 3 = >$2.25M

% Margin / 20

1 = >18 months 2 = 6–18 months 3 = <6 months 1 Miss the Market 2 Possible loss of Mkt Share 3 Not important 1 = 2 or more priorities >40

Design, Supply Chain & MFG Risk

2 = 1 or more priorities >40 3 = No priorities >40 1 = 2 or more priorities >40

Market Risk

Benefit to Corporation

Customers Un-met Market Need

Development Cost (incl. engineering cost, prototype and testing)

Engineering Hours Required

2 = 1 or more priorities >40 3 = No priorities >40 1 = Low 2 = medium 3 = Very important 1 = Customers needs Met 2 = Potential Latent Needs 3 =Customer Needs Unmet 1 = >$350k 2 = $100–$350k 3 = <$100k 1 = >2000 hours 2 = 500–2000 hours 3 = <500 hours

Source: Adapted from Company C’s internal documents.

Figure A2.1.7

R&D project submittal form

Rating

Comments

R&D projects

Define

Figure A2.1.8 5.

Measure

Analyse

99

Design

Verify

The DMADV roadmap

Verify. The performance of the new product and the associated processes is evaluated and tested.

The same inter-functional team that evaluated the project has the responsibility to manage it along this process. In particular, the project manager maintains its coordination functions, whereas the product manager takes the responsibility for the first three steps represented in Figure A2.1.8. An engineering manager is further appointed, who has responsibilities over the Design and Verify phases of the process. A number of milestones along the process are defined. At each milestone, a meeting (called Tollgate Review) is organized, in which the head of the division, the project manager and the product manager (for the first three phases), or the engineering manager (for the last two phases), participate. The chairman of Company B, as well as the head of the other corporate divisions, might take part in the Tollgate Reviews of the projects they are interested in. At the end of each meeting, a document is prepared where the improvement actions that were decided are listed, together with the name of the person who is responsible for them (see Figure A2.1.9). The Define phase is particularly important for understanding the complex evaluation system used by Company B. In particular, at this first stage of the development process, the different documents already compiled during the Approval phase are updated, with the information that might have been disclosed in the meanwhile. This allows continuous monitoring of the scorings of the project and the risk management system. Furthermore, another indicator is introduced for the evaluation of the project, the expected commercial value (ECV), which is calculated as: ECV ⫽ [(NPV • Pcs ⫺ C) • Pts ⫺ D]

(A2.1)

where: NPV 5 Net Present Value calculated as the sum of the discounted net cash flows associated to the manufacturing and sale of the product Pcs 5 probability of commercial success C 5 costs for the commercialization and launch of the product

100

Evaluation and performance measurement of R&D

Group Tollgate Review Projects:

A) B) C) Attendees:

Date: Project

Actions / Point of Note

Responsibility

Current and future quotes need to be tracked to see trend in basic, moderate and

A

complex controls. This trend to be presented at each subsequent Tollgate

A

Update VOC costs and NPV / ECV

B

Update VOC costs and NPV / ECV Look at establishing additional sales based on developing a package offering to

C

customers–Package to be combined

C

Update VOC costs and NPV / ECV

Gen

Create a general VOC budget which each project ‘pays’ into. This central budget

Gen

Source:

can be identified in the R&D annual budget Create a resource plan based on the template to identify schedules for each project

Adapted from Company C’s internal documents.

Figure A2.1.9 Pts D

Group tollgate review

5 probability of technical success 5 development costs.

This indicator takes into proper account the following aspects: 1. 2.

3.

The firm will realize the expected NPV associated to the project only if the product will experience commercial success; In case of technical failure, the firm will not realize the expected NPV, but it will not sustain the costs for the commercialization of the product as well; The firm will bear the development costs of the project also in cases of technical or commercial failure.

Calculating this indicator is very straightforward because the cash flows and the development costs have already been estimated in the

R&D projects

Table A2.1.1

101

Matrix of probabilities of commercial success

Product newness

Organization’s competitive advantage Low: CBR < 1

Current product Significant enhancement to a current product New to organization New to market

Medium: High Very high CBR = 1 1 < CBR < 3 CBR > 3

0.5 0.35

0.65 0.5

0.8 0.65

0.95 0.8

0.2 0.1

0.35 0.2

0.5 0.35

0.65 0.5

Source: Adapted from Mader (2004). CBR = Cost Benefit Ratio, calculated as CBR = NPV / (D + C).

Table A2.1.2

Matrix of probabilities of technical success

Ability to Deliver

Very high: solution already exists and is viable High: solution has been tested and appears viable Medium: solution has not been tested but is believed to be feasible Low: solution is not yet known but is believed to be feasible Very low: solution is not yet known and its feasibility is unknown Source:

Ability to develop product Very low

Low Medium High

Very high

0.5

0.65

0.8

0.9

0.95

0.35

0.5

0.65

0.8

0.9

0.2

0.35

0.5

0.65

0.8

0.1

0.2

0.35

0.5

0.65

0.05

0.1

0.2

0.35

0.5

Adapted from Mader (2004).

documents called Revenue and Margin Calculations and Development Costs. Therefore, it is only necessary to assess the size of the investment in fixed capital (to be included in the Net Present Value), the commercialization and launch costs, and the probabilities of technical and commercial success. As far as these last two data are concerned, Company B uses the two matrixes proposed by Mader (2004), reported in Table A2.1.1 and Table A2.1.2. The ECV and the scoring method are calculated not only at the beginning of the development process, in the Define phase, but they are constantly

102

Evaluation and performance measurement of R&D

updated as long as the project moves into the stages of Measure, Analyse, Design and Verify, in order to continuously monitor the contribution of the project to the firm’s economic value. It is interesting to note that Company B uses the ECV only during the development process, but not for screening purposes. This is due to the fact that calculating the ECV in the early phase of the selection process is not easy because of the uncertainty that surrounds cash flows at that stage of development. Moreover, at that stage it is crucial to streamline communication and coordination among the different functions in Company C, as well as between them and the corporate division, and to capture also the intangible and qualitative aspects that might be crucial for the success of the R&D project and should be considered in the evaluation process. These requirements might be difficult to satisfy through the use of a synthetic indicator like the ECV. The latter becomes important during the development process, because it is much easier to calculate, and hence more appropriate for an in-progress form of monitoring. The Scoring Method The method used for the selection and prioritization of R&D projects in Company C is a typical non-financial scoring method. It comprises 11 attributes or evaluation criteria, to which a weight is associated that represents the importance of the specific attribute for the firm’s strategy. The overall score associated to each project is calculated as the weighed sum of the scores assigned to the project along the different criteria. It is interesting to note that both the criteria and the weights to be used are defined by Company B, and they might be different from one division to the other. The application of the technique is realized filling in the R&D Project Submittal Form (see Figure A2.1.7). It reports, besides the score assigned to each attribute, the name of the project, the name of the members of the evaluation team and the product manager, together with a description of the main characteristics of the project. A space for comments about the qualitative intangible factors that suggested a specific rating is included in the submittal form. On the following, a description of the attribute is provided: 1.

2.

Increased revenue: evaluates the improvement in the annual turnover that is likely to be gained, from the second year after launch, through the sales of the new product. This value is provided in the Revenue and Margin Calculations document (see Figure A2.1.4). The weight of this attribute is 15 per cent. Increased margin: considers the improvement in the annual margins that is likely to be gained, from the second year after launch, through

R&D projects

3.

the sales of the new product. This margin takes into account only the industrial costs and is provided in the Revenue and Margin Calculations document (see Figure A2.1.4). The weight of this attribute is 15 per cent. Estimated margin: contrasts the margin and the revenues from the sales of the new product. It is calculated as: Margin % 5

4.

5.

(Revenues 2 Industrial costs) Revenues

(A2.2)

The score to be assigned along this criterion is calculated as Margin % / 20. This is the only score that can have a value higher than 3. The weight of this attribute is 15 per cent. Development duration: Takes into account the length of the development process for the new product, which is critical because it impacts its overall time-to-market. The weight of this attribute is 7.5 per cent. Importance of ‘time-to-market’: it estimates the importance that time-to-market has for the success of the new product. As far as the rating is concerned: (a) (b)

(c)

6.

103

1 is assigned if a time-to-market longer than foreseen might determine a commercial failure of the product; 2 is assigned if a time-to-market longer than foreseen might determine a loss in the expected market share of the new product; 3 is assigned if a time-to-market longer than foreseen is not likely to lead to relevant negative consequences.

The weight of this attribute is 7.5 per cent. Design, supply chain & MFG risk: quantifies the importance of the technical risk for the project under consideration. In order to determine the appropriate rating, it is necessary to consider the Risk Management Plan (see Figure A2.1.5), where the relevance of the different risk factors has been calculated. Afterwards: (a) (b) (c)

1 is assigned if two or more risk factors have a relevance higher than 40; 2 is assigned if only one risk factor has a relevance higher than 40; 3 is assigned if no risk factors with relevance higher than 40 have been identified.

The weight of this attribute is 10 per cent.

104

7.

8.

9.

10.

11.

Evaluation and performance measurement of R&D

Market risk: takes into account the importance of the commercial risk for the project under consideration. Ratings are assigned in a similar way to the last attribute. The weight of this attribute is 10 per cent. Benefit to corporation: considers the degree to which the R&D project contributes to the economic value of the corporation as a whole. This criterion is introduced to encourage the evaluation team to assess the project in a comprehensive manner, and not merely from the perspective of the local subsidiary. The weight of this attribute is 5 per cent. Customers un-met market need: quantifies the degree to which the needs addressed by the new product are new and un-met. If the new product addresses un-met needs, it will be able to command a higher premium price and/or market share. The weight of this attribute is 5 per cent. Development costs: considers the magnitude of the development costs that have been estimated in the Development Costs document (see Figure A2.1.6). The weight of this attribute is 5 per cent. Engineering hours required: evaluates the overall number of hours dedicated to the development of the project by the employees from the engineering function. This element is particularly important because the maximum number of R&D projects that can be approved in one year depends on the availability of human resources. Despite the fact that the costs of the engineering hours have already been considered in the Development Costs, it was decided that this attribute should be separately considered. The weight of this attribute is 5 per cent.

It is interesting to note that the scoring method is used in all the three phases around which the firm’s innovation process is built. Nevertheless, the purpose for which it is used, as well as the level of detail with which it is calculated, significantly differs along the phases of the process (see Figure A2.1.10). The heterogeneity of the attributes considered in the scoring method, which captures issues related to financial performance, time, technical and commercial risks, and benefits for the whole corporation, allows Company B to undertake a comprehensive evaluation of the potential of each R&D project to contribute to its economic value. It should be noted however that attributes related to financial performance have an overall weight of more than 50 per cent, this indicating the importance Company B attaches to these aspects. The measurement system employed by Company B is interesting because it overcomes some of the limitations typical of scoring methods.

R&D projects

Screening of ideas

• Prioritization of potential projects • Low level of detail • High number of alternatives analysed

Figure A2.1.10

Approval of the projects

• Prioritization of screened projects • High level of detail • Low number of alternatives analysed

105

Management of the projects

• Updating of the ratings to monitor the relevant performance of the projects

The use of the scoring method along the phases of the innovation process

First of all, the ranges used to assign the ratings along the different criteria allow the modelling of the non linear relationships that might be present between specific performance and economic value creation: beyond some levels of performance, the improvement of an attribute might have a very limited impact over value creation capability because of saturation effects. The use of the ECV indicator in the project management phases allows an absolute evaluation besides the relative one provided by the scoring method. The ECV can also take into account the size of the investment, which is something that is not captured by scoring techniques. The inter-functional evaluation team and the shared definition of the ratings improve the capability of the technique to contribute to the firm’s organizational learning. The R&D project evaluation system used by Company B, however, has some limitations. First of all, it does not consider pay back as a criterion for the appraisal of the projects, nor does it allow for the valorization of different types of flexibilities, as real option methods would. Finally, it is a rather complicated and costly method to employ. Only its usefulness as an organizational coordination mechanism and communication means between the headquarters and the subsidiaries of Company B justify the time and resources needed to introduce and to run such an evaluation approach.

NOTES 1.

The ranking of top Italian R&D spenders was taken from the report of the European Commission ‘The 2007 EU industrial R&D investment scoreboard’ providing data on

106

2.

Evaluation and performance measurement of R&D top 500 EU companies by R&D investment. Please refer to: http://iri.jrc.ec.europa.eu/ research/scoreboard_2007.htm Alternative methods have been developed (Henriksen and Traynor, 1999) based particularly on weighting multiplicative approach where n

Si 5 q (sji) wj. j51

3.

4.

5.

6.

7.

Multiplicative scoring methods tend by nature to avoid the risk of selecting a project with a very low score in at least one of the n factors considered for the evaluation. For a comprehensive discussion of these methods please refer to Moore and Baker (1969). In reality, it is possible to consider the amount of initial investment by including it in the n factors used for the evaluation. However, in this case too, scoring methods suffer from a poor assessment of economic measure and also from the fact that even huge differences in the amount of initial investments can be dampened in the overall weighted score. In other words, if a12 (that is, the comparison between criteria 1 and 2) is equal to 5, meaning that the contribution of criterion 1 to the overall goal is greater than the contribution of criterion 2, a21 (that is, the comparison between criteria 2 and 1) has to be equal to 1/5. In other words, if the original evaluation made on the three project alternatives A, B, C resulted in the ranking C, B, A, adding the project alternative D and performing only the new needed pairwise comparisons (A/D, B/D, C/D) may result in an overall ranking D, B, C, A, where the relative position of B and C is changed. The size of the project is obviously relative to the size of the company. A large project is a project whose capital needs are as relevant as to require the company to negotiate a new debt and/or to define an ad hoc use of equity reserves, whereas a small project is one whose financing does not require an ad hoc retrieval of capital. The net cash flow of year t is obtained as: [(Revenues − Cash costs) (1 − t*) 1 (Depr. 1 Amort.) t*] − Inv. where: Revenues are the incremental revenues due to the innovation (new product or use of new production process) generated by the project; Cash costs are the incremental cash costs sustained for producing and selling the innovation (material, labour, rents); Depr. is the depreciation of the investments in tangible (fixed assets) acquired for the project; Amort. is the amortization of the investments in intangible assets (R&D expenses, brand names, patents) sustained for the project; t* is the fiscal rate; Inv. are the investments in fixed assets and working capital.

8.

9. 10. 11.

Returns in the long term horizon are indeed penalized by the high discount rate. For example, a 10 per cent discount rate already halves the value of cash inflows after only seven years. At the same time, applying such a discount rate would require small and rather conservative R&D projects to achieve a 10 per cent IRR (internal rate of return) in order to be funded. In other words, the use of the new drug causes negative side-effects in no more than 1 out of 10 healthy people, assuming the dosage usually required to treat the target disease. Cash flows are usually discounted using the company’s weighted average cost of capital (WACC), even if ad hoc discount rates can be also used. The Black and Scholes model calculates the value of an option as the result of a partial differential equation, assuming that the exercise price follows a geometric Brownian

R&D projects

107

motion. The solution to this equation exists in closed form for European options (where the exercise date is fixed) and runs as follows: value 5 S # N (d1) 2 K # e 2iT # N (d2) where: d1 5

ln (S/K) 1 (i 2 0,5 # s2) # T s # "T

;

d2 5 d1 2 s # "T; N(x) is the cumulative normal distribution. Please refer to Table 2.3 for a definition of the other variables. Further details on the mathematics of this equation can be found for example in Armstrong et al. (2004).

3.

R&D people

3.1

INTRODUCTION

This chapter introduces another dimension of Research & Development performance management and evaluation: people. Human resources (HR) in R&D are managed in most organizations as a separate body from the main organizational structure of the firm, and in most cases they receive special attention, given the peculiarities of the knowledge-intensive activities performed by researchers and engineers. The topic has been debated by practitioners and academics for years, but nonetheless the existing literature does not completely satisfy the current needs and does not respond fully to a series of issues which have emerged very recently. Szakonyi (1994a) observes that although almost everyone involved in managing industrial companies believes R&D should play a vital role in sustaining and growing a company’s business, surprisingly only a small percentage of companies have world-class R&D conscious management. People operating in R&D laboratories represent crucial assets in modern industrial organizations, especially in the case of knowledge-based firms. Each researcher brings to the company his/her tacit knowledge, technical background, vocational qualifications, professional certification, workrelated know-how, work-related competence, creativity and social links with other people and organizations. These elements represent at the same time the single greatest potential asset of the firm and also one of the most troubling sources of uncertainty and management complexity. In particular, it can be argued that human resources are among the most valuable assets in innovative organizations since they play a fundamental role in the creation of competitive advantage through technological leadership. However, due to its elusive and intangible nature, human capital is extremely difficult to manage and evaluate, especially with regard to creative activities. Scientists, engineers, and other technical personnel involved in R&D have to be attracted, motivated and retained in order to reach high performance in the generation of new ideas and products/processes, but the same tools commonly used by generalist HR managers to match these issues may not provide comparably good results. A need then emerges for specific HR practices designed to manage and improve R&D performance. In fact, R&D professionals usually work in environments characterized 108

R&D people

109

by lower levels of formalization and a large number of interactions with other people and organizations, both inside and outside the firm. Some new human resources management (HRM) practices can produce positive effects: among them, for example, the promotion of social networks dramatically expands HR potential, leveraging cross-functional synergies with colleagues and supervisors internally, and those with researchers belonging to other laboratories, externally. Also, some other common HRM practices have to be revisited and adapted, as they do not fit the R&D optimal configuration. In many R&D firms, for example, conventional career programs, based on power and status, within pyramidal organizations, are often substituted by dual ladder systems, considered to be useful ways for maintaining prestige and status by adding other forms of reward and recognition to the reduced advancement opportunities within the labs and the company as a whole. This chapter focuses on the analysis and evaluation of HRM practices adopted by industrial firms in their R&D labs. The next section reviews and discusses the main suggestions that R&D and HRM literature provide for practitioners in this area. A management student might find this account a useful review of the work of scholars in the field. Managers can also hopefully find useful ideas that have been established and empirically tested in a variety of contexts. Particular emphasis will be given to showing how performance measurement and evaluation might contribute to improving the different areas around which HRM of R&D professionals can be organized. The third section introduces the two main drivers of change that both practitioners and scholars need to consider when it comes to assessing their management and performance evaluation routines related to HR in R&D. These critical dimensions are: 1. 2.

Project intensification (or ‘projectification’); Open innovation.

Management studies have so far focused on the macro/company broad effects of such changes, but little has been written on the micro/organizational impacts of such drivers of change. In these circumstances, however, each R&D worker receives different pressures and responsibilities and must respond to challenges that are radically dissimilar to those a traditional human resource management system has been providing in the past. A different perspective will be taken in this section. It will be argued that the viewpoint of R&D employees, which can be collected through surveys and interviews, can provide management with a privileged perspective in order to analyse how scientists and technicians are experiencing these changes and adapt the performance evaluation system to better face these

110

Evaluation and performance measurement of R&D

challenges. The third section also introduces a theoretical model that comprises the most relevant areas for HRM in R&D and has been used to collect data from researchers working in firms conforming to the ‘open innovation’ principles (Chesbrough, 2003a, 2003b) already mentioned in the introduction of the book. The last section draws some conclusions and outlines avenues for future research. We believe that one of the main contributions of this chapter is that it discusses R&D management from R&D workers’ perspective and provides empirical evidence about the micro-organizational implications of the open innovation paradigm. We hope it will also make a valuable contribution to further investigation of advanced HRM practices in R&D.

3.2

HRM IN R&D: THE MAIN THEMES OF THE LITERATURE

HRM practices in R&D significantly differ from common rules applied to general HR operating in other functions. In fact, managers have to identify and apply all those adjustments of the HRM bulk of knowledge that meet the specific and complex needs of R&D. People who have to be attracted and retained in any R&D organization are usually highly motivated and skilled individuals. The sustained competitiveness of a firm might heavily depend upon their outstanding ideas and brilliant solutions (Cesaroni et al., 2005; Jordan, 2005), as well as their networking capacity. The relevance of R&D for the innovativeness of a firm and for its profitability makes it fundamental to understand the emerging trends in this specific field of management. This brief review of the literature draws extensively from the excellent work by George Farris and Rene Cordero (Farris and Cordero, 2002), which is highly recommended to students looking for a comprehensive introduction to the main lines of reasoning. In the last twenty years an increasing number of authors have started investigating how to cope with many specificities of R&D human resources. The literature on HRM in R&D is however not very developed if compared with contributions dedicated to the other business functions. Only a few authors have specifically explored efficient and effective organization models of industrial research. Interesting intuitions about the psychology and the motivation of workers were discussed by authors such as White (1959), Porter and Lawler (1968) and Costa (1992). Porter and Lawler, in particular, elaborate the so called ‘expectancy model’, extending an earlier model developed by Victor Vroom in 1964 (Vroom, 1964) and investigate relationships between personal satisfaction and productivity, but their

R&D people

111

perspective is general and not distinctively referred to R&D personnel. Recently, other authors have preferred to focus on conceptual organizational frameworks rather than on micro behavioural aspects. Current studies are however closing this gap. The prototype of an ideal R&D worker emerging from the literature is that of a dynamic and easily adaptive individual, able to define problems (rather than merely solve predefined issues) and work in a constantly changing environment. The traditional static role of the technician, whose main task was to find information, solve problems and eventually explore relationships among phenomena that may be useful to develop new products, has become in some ways obsolete. The complexity of the business environment requires new people to operate in R&D and new methods to manage them. Project cycle time reduction requested by global competition and the need to perform innovation faster is causing a shift in focus from research to development, and strategic opportunities such as international alliances and collaborations with public research institutions are increasingly taken into account. New emphasis is given to outsourcing and offshoring of R&D labour-intensive phases to low-cost locations worldwide. Such practices have become more common than in the past due to development of ICTs and the improvement of remote team management. Companies are able to concentrate in their main labs core competencies that bring higher levels of value to the output, as well as the most strategic phases of their R&D work, and at the same time keep ‘windows’ open on emerging scientific and technological trends. Great emphasis in the literature is also now given to teamwork, whose role is seen as fundamental for effective new product development (Barczak and Wilemon, 2003). Actions facilitating knowledge absorption and integration such as knowledge sharing (Grabher, 2001), reciprocal trust and social learning (Creed and Miles, 1996), are generally acknowledged by scholars as important success factors, together with the adoption of organizational schemes based on cross-functional teams (Cooper and Kleinschmidt, 1995) and communities of practices (Frost and Holzwarth, 2002). The emphasis given to these emerging aspects, however, does not reduce the importance of the traditional basics of HRM such as role definition, reward systems, career paths and performance evaluation methods, whose most interesting peculiarities, when applied in R&D, will be discussed in the following subsections. 3.2.1

The Hiring Process

Technical specialization and competence have always been the most relevant characteristics of R&D workers. Nowadays, however, these skills are

112

Evaluation and performance measurement of R&D

not enough since R&D laboratories need multiple skilled and multitasking workers. The hiring process of R&D personnel has to evolve in order to identify skilled people able to work in cross-functional teams, often geographically distributed and usually made of demographically diverse members. A combination of expertise and culture is needed to allow the collaboration of multiple international teams. The selection process is therefore aiming to hire not only specialists and excellent scientists, but also open and collaborative people, ready to work in an interactive environment and flexible enough to adapt to changing routines and rhythms. Traditional methods used by firms to hire scientists and engineers are now integrated with new tools identified by Perry (2002) as: 1. 2.

online recruiting techniques (performed through the Intranet first and then extended to public audiences through internet); people techniques (university openings, networking activity, internship programs, word of mouth).

Also, applicants and R&D workers need to be segmented according to a matrix of variables in order to capture relevant aspects of their profiles. The taxonomy developed by Roberts and Fusfeld (1981) still serves for this categorization, and it seems useful here to single out profiles which fit the changing needs of the R&D organization. In such a taxonomy, the ‘champion role’ is personified by a person who takes a particular interest in seeing that a particular process or product is fully developed and marketed (Markham and Griffin, 1998; Markham and Aiman-Smith, 2001). Champions provide informal project leadership, and this becomes extremely relevant when formal leadership or senior management support is lacking or not adequate to guide daily lab activities (McDonough, 2000). The ‘gatekeeper role’ is played by people integrating different functions and spanning boundaries within the organization or bringing information on external technological advancements into the organization. Some authors debate about the possibility to formalize these tasks (Nochur and Allen, 1992; Quinn, 2000). In the traditional taxonomy, ‘idea generators’ are considered the most fundamental category of R&D people. The hiring process for these people is similar to talent scouting; it can be very expensive, in particular for the special requests that need to be accommodated to fit them in the organization. The growth of these individuals within the organization is indeed a much more inexpensive alternative. From these ‘star scientists’ the lab is expecting the single most relevant contribution for the R&D work. Great emphasis in the literature is given to how the outputs of

R&D people

113

these scientists and engineers can be somehow stimulated and supported, given the fact that in general they respond to a different set of incentives. Some authors suggest that the entire HRM organization should pay great attention to the internal development and fit of these individuals. Not only adequate compensation, but also an appropriate organizational setting should complement and stimulate his/her innovative contributions (James, 2002). Finally, team leader and project leader act as catalysts and positively influence project outcomes by allowing team members to participate, facilitating communications, leading rather than commanding and controlling, selecting the right mix of team members, and providing team members with the right training (Jassawalla and Sashittal, 2000). Their role is very important, especially when the workforce is geographically dispersed or R&D workers have to collaborate in cross-functional teams (Kayworth and Leidner, 2002). Nonetheless, HR is not always able to select and employ the personnel who perfectly fit with certain tasks and roles. It is therefore quite important for HRM practices and tools to signal failures back to the recruiting teams, in order to align hiring procedures with the changing needs of the organization. 3.2.2

The Reward System

An intense debate is going on about the best typology of rewarding techniques. Literature distinguishes two basic kinds of rewards: 1. 2.

intrinsic; extrinsic.

Chen et al. (1999) argue that intrinsic rewards are more effective motivators than extrinsic compensations. The technical challenge (Alpert, 1992) and the opportunity to pursue personal research interests (James, 2002) lead engineers and scientists to be highly motivated and to achieve good results. Career advancements are not seen as intrinsic rewards: it is rather the opportunity to start a new scientific venture without investing and risking personal resources (Gomez-Mejia et al., 1990) and the challenge to contribute to a larger project that drive the motivations of R&D workers (McKinnon, 1987). Career advancement becomes a form of reward only indirectly: when through a change of job title the scientist has the perception of being challenged in new ways that he/she feels stimulating (Katz, 1988).

114

Evaluation and performance measurement of R&D

Even if it is not possible to raise scientists’ enthusiasm only through monetary compensations, they are nonetheless useful to improve productivity and reduce the risk of losing the best talents, who may be attracted by a higher salary offered by competitors, if expected working conditions and intrinsic rewards are similar. Extrinsic rewards are in fact helpful to motivate people but have to be inherent to the work itself and directly linked to the contribution of workers through the exploitation of their findings (Despres and Hiltrop, 1996). Moreover, since cross-functional teams are widely diffusing, they have to include the general performance of the team as well as the individual dimension (O’Dell, 1989). Salaries become flexible, varying within broad ranges depending on results. Outstanding performers are paid more than other workers; cash bonuses, stock options, awards and recognition by colleagues and managers have become more common forms of compensation for the most talented contributors. Patent revenues can be partially attributed to inventors, and patenting has become a priority for some companies beyond the life sciences sectors where intellectual property has been traditionally relevant. The static career-linked salary is more frequently replaced by a dynamic wage system based on a market aligned sum plus additional bonuses associated to performances (Gomez-Mejia et al., 1990; Geraci, 1994; Depres and Hiltrop, 1996; Triendl, 1998). This is very important to manage turnover: without any differentiation in extrinsic rewards, high performers would feel dissatisfied and leave if treated in the same way as low performers (Kochanski and Ledford, 2001). Brain drain determines high costs for the organization, especially in R&D intensive companies. Quite often R&D labs invest in junior talented scientists, through personalized training and informal coaching. Retaining the best resources within the firm is then one of the most important challenges for HRM. When an R&D worker leaves the company there can be a tremendous sunk cost that the company will suffer, and he/she might be taking out of the company precious tacit knowledge accumulated over time. Of course, in some cases the departure of R&D workers is aligned with the company’s strategies since those workers can then be employed by industrial partners and because new and younger researchers can be hired to substitute them, bringing into the company state-of-the-art knowledge, often coming directly from universities. Last but not least, culture and strategy can be designed to ease work– family tensions. Firms where work does not have a negative impact on family life due to quality-of-life arrangements (for example flexitime, telecommuting, on-site health clubs, on site gyms, child care services) are more attractive for talented engineers and scientists (Hammonds et al., 1997; Reimers, 2001).

R&D people

115

Performance evaluation, and in particular the interpretation of weak signals coming from R&D people, are instrumental to preventing unwanted turnover and ‘brain drain’, given that turnover is indeed easy to measure. However, climate analysis and other bottom-up perception indicators are currently being implemented in order to measure the ‘brain attraction’ of a particular firm, given the fact that it is through the hiring of talented individuals that companies are able to pursue their most ambitious R&D goals and improve the performance of their teams. 3.2.3

Performance Evaluation

In order to improve performance it is necessary to monitor and evaluate it. Evaluation of industrial R&D activities, from different perspectives and at different levels, is the central topic of this book. The guiding idea of this chapter is that an accurate analysis and evaluation of the HRM practices for R&D employees can provide managers with better knowledge and useful tools to control and understand motivations, incentives and climate within the lab. According to Cordero (1999), managers need to approach evaluation in HRM in a formal and systematic way. A wide range of people may intervene in the appraisal process of R&D workers: senior scientists, functional managers, project managers, team leaders and even team members. New knowledge is indeed difficult to measure and a direct link is also not always easy to establish between the results produced by an individual scientist or technician and the firm’s overall results. Nevertheless, performance evaluation is also necessary in R&D. Tools such as 360-degree feedbacks are often used since it is necessary to be as inclusive as possible when working with R&D scientists who are familiar with peer-review evaluation systems. A very large number of metrics can be defined to evaluate workers’ contributions to corporate wealth creation. These indicators can be based on qualitative parameters or quantitative elements also in R&D labs. In the first group are self-evaluations, assessments performed by specialists, consultants and managers and also peer reputation systems and customer feedbacks. The second group is composed of indicators such as the number of publications, the number of citations, the number of patents, and so on. Individuals are ranked taking into account a set of significant parameters whose weights measure the importance of the elements to be considered and whose values show the degree of accomplishment of each goal. Multiple measures are usually preferred to single ones since the latter produce evaluations that are often inaccurate and – in any case – partial. Indicators and metrics used to measure the performance of single R&D scientists and professionals are the subject of Chapter 1 of this book. In

116

Evaluation and performance measurement of R&D

the remainder of this chapter the focus will be on the approaches that can be employed to evaluate and diagnose the HRM practices adopted by industrial firms in their R&D labs. 3.2.4

Career Paths

A number of authors have explored the fields of career paths and dual ladder systems, which have been extensively supported (Shepard, 1958; Roth, 1982; Allen and Katz, 1986, 1992, 1995; Costa, 1992; Cha and Kim, 2000) and criticized (Goldner and Ritti, 1967; Kaufman, 1974; Dalton et al., 1977). However career issues cannot be fully conceptualized in a single static way since the context continuously evolves and many career decisions depend on contingencies. In recent times workers’ career perspectives have experienced some changes: the traditional covenants between the firm and the employees have gradually weakened and the interest has shifted from developing a career within the same organization to gaining competitive skills that might enable future employment opportunities in other organizations. As a consequence, career advancement is considered less relevant than the availability of free time, education opportunities, workers’ mobility, and flexibility to cope with personal life (Hammonds et al., 1997). As a consequence, standard career paths, even if widely and successfully implemented in other corporate functions, do not match current R&D workers’ needs. Thompson and Dalton (1976) suggest a simple linear model which describes four steps through which professional careers traditionally develop: 1. 2. 3. 4.

‘apprenticeship stage’, in which learning under supervision prevails; ‘independent contributor stage’, where technical skills are mastered and people become independent technical contributors; ‘mentor stage’, in which professionals become technical managers and have the tutorship of junior researchers; ‘sponsor stage’, in which professionals become general managers and are asked to provide development opportunities to other workers and manage the external relationships of the organization.

Although this model may be consistent with the general trends occurring de facto in the majority of firms, the assumption that every worker desires to be promoted with the final goal of managing others is to some extent misleading. The special attitudes and skills of scientists and engineers often make them prefer to avoid team responsibilities, which they do not fully understand or believe themselves able to control. Merton (1957)

R&D people

117

distinguishes between two kinds of R&D workers: ‘locals’ and ‘cosmopolitans’. Locals do have managerial attitudes but present less specialized technical skills; they identify themselves more with their organization than with the content of their work, appreciate the value of organizational performance and consider commercial success as a goal to be pursued. By contrast, cosmopolitans are more technically oriented and usually have a Ph.D., consider freedom of research to be highly important, value technical performance and identify themselves with their profession (Aryee and Leong, 1991; Allen and Katz, 1992; Womack and Jones, 1996). The dual career system (or ‘dual ladder’) meets the ambitions of both: locals are promoted in the managerial path, cosmopolitans in the technical one and their promotions are – at least in theory – comparable in terms of status, salary and managerial control. Advancements in the managerial path imply increasing responsibility and a larger number of people to coordinate, whereas promotions in the technical path mean more complex technical challenges to be tackled. Nonetheless Allen and Katz (1992) emphasize two serious problems that R&D organizations often incur by applying the dual ladder as the main career system: 1. 2.

the technical path becomes a sort of ‘dumping ground’ for bad managers; technicians risk being isolated from the rest of the organization.

Another relevant aspect that has to be considered is the fact that R&D workers may have multiple career orientations that are not mutually exclusive (Baugh and Roberts, 1994) and follow different paths simultaneously. Some new trends in the behaviour of R&D workers identified by the literature are: 1. 2. 3.

‘project orientation’, the desire to work in challenging projects; ‘technical transfer orientation’, the wish to move, together with the technology they have developed, to other organizational units; ‘entrepreneurial orientation’, the desire to develop new ventures based on the technologies they have worked on (Allen and Katz, 1986; McKinnon, 1987; Bailyn, 1991; Cha and Kim, 2000; Petroni, 2000).

Many of the restrictions related to the vertical integrated dual ladder disappear when a flatter multi-ladder career system is implemented (Womack and Jones, 1996), especially when it is associated with job rotation, since it provides more cross-functional and more mixed career development opportunities (Levi and Slem, 1995; Cordero, 1999).

118

Evaluation and performance measurement of R&D

As usual, performance evaluation represents the main input for career development. The emergence of different opportunities and requests from the corporate level, the permeability of a lab with respect to forces that can drive the best and the brightest people away from the company, lead to greater emphasis on the weak signals that can be collected through a bottom-up evaluation of colleagues’ performance, leadership, working environment, motivation, and satisfaction. 3.2.5

Cross-Functional Teams

Results in the new product development process heavily depend on the joint efforts of different functions (for example R&D, marketing, manufacturing). This is the reason why more and more R&D laboratories are now relying on cross-functional teams in order to improve cooperation among different specialists focused on the same objectives and concentrated within the same unit (McDonough, 2000). Cross-functional teams facilitate the accomplishment of product quality standards and reduce the costs of development, in terms of both money and time (Anderson, 1993; Cooper and Kleinschmidt, 1995; Zirger and Hartley, 1996; Phillips et al., 1999; Keller, 2001). Larger autonomy is given to team members and the collective perspective should prevail on the individual one, even if single members’ needs are understood and taken into account by others as much as possible (Jassawalla and Sashittal, 2000). Trust building and explicit information are key factors for successful team management, in particular when the group is geographically distributed (Bartlett and Goshal, 1989) and multiethnic (Baba et al., 2001). People coming from different cultures, in fact, may not share the same values, the same methods, the same approaches to teamwork (Gibson and Zellmer-Bruhn, 2001), or the same language. Then, formalized relationships, standardized procedures and frequent contacts among members are strongly needed. As a consequence, new important roles concerning boundary and information management have to be played by scientists and engineers (Ancona and Caldwell, 1992): 1. 2. 3. 4.

the ‘ambassador role’, addressed to link insiders with outsiders; the ‘task coordinator role’, devoted to manage the cross-functional activity; the ‘scouting role’, aimed at looking at the environment for fresh ideas; the ‘guarding role’, whose goal is to control the information flow towards outsiders.

R&D people

119

An obvious task in HRM in R&D is to analyse how each individual researcher can better contribute to a cross-functional team in one of these roles. This can serve both to identify consistent career development options and to better allocate resources for future projects. Moreover, cross-functional teams might serve different objectives and respond to a variety of control centres. As a result, there is a strong need to integrate different sources of information to better understand the relevance of the tasks performed. 3.2.6

Leadership

In the past close monitoring and top-down command were the tools most commonly used to manage R&D workers. Scientists’ and engineers’ tasks were precisely defined by rules, plans and procedures that had to be followed under the supervision and control of technical managers. Nowadays, many R&D technical managers have changed their role, shifting from a command and control approach to a leadership based approach (Hammer and Champy, 1993; Jassawalla and Sashittal, 2000; James, 2002), with increasing responsibilities for teams and individuals. R&D workers are empowered to achieve technical goals but also business and financial objectives. The integration is sponsored by technical managers, who do not act as ‘captains’ but as ‘catalysts’. Moreover, the stimulating work environment the latter provide – which includes clear objectives, challenging tasks, collaboration in teams, full communication opportunities, opportunities to grow and develop new skills, and a fair reward system linked to performance – substitutes for the need to direct the work of scientists and engineers (Cordero et al., 2002). In addition to leadership and technical skills, some administrative and financial competencies should also be developed by researchers in order to make their work more efficient. A bottom-up performance evaluation is here necessary to understand the impact of various team and lab leadership styles. Also, the previously discussed emphasis on the evaluation of single researchers’ characteristics and motivations should contribute to a better understanding of how each individual can contribute to a more participatory and distributed leadership environment. 3.2.7

Knowledge Management

Knowledge generation is the core element that an R&D lab expects from its human capital and new knowledge is a crucial competitive asset for companies in the knowledge economy. However knowledge, like every important source of value, must be managed properly. It resides within the individuals and the organization in two forms: explicit and tacit.

120

Evaluation and performance measurement of R&D

Explicit knowledge can be transmitted verbally or written in reports and manuals, while tacit knowledge comes from experience, is personal and may be hard to replicate; it is learnt and shared only during joint activities. Culture and structure enable knowledge management practices which promote knowledge creation and knowledge sharing (Armbrecht et al., 2001). ICT is obviously an important enabler of knowledge accumulation and codification. However, an effective information system is a necessary, but not a sufficient, condition to enable new knowledge generation. Support from the top management and vision, specific incentives, flat organizational structures, cross-functional teams, meeting areas and social events, are all elements that complement and need to be considered for the smooth operation of this critical process. Quite a few leading companies have decided they need a full-time knowledge manager, whose main task is to facilitate and oversee new knowledge creation and dissemination. In many cases, however, one of the best ways to enhance knowledge diffusion is the promotion of communities of practice on special topics, which are granted a high level of autonomy to determine the norms of interaction, value recognition and areas of interest (McDermott, 1999; Sakkab, 2002). While performance evaluation can only partially grasp the daily relevance and impact of these communities, an advanced monitoring system should at least oversee the resources these groups tend to absorb. Furthermore, researchers should be encouraged to acknowledge the impact that time spent in these interactions has on their job and competencies. This should allow some partial understanding of how informal interaction and tacit knowledge exchange impact on the activities of an R&D lab. In the medium to long term, the most relevant outcomes of such informal interactions should be codified and structured into proper centres of competence, so that performance evaluation and strategic management of such assets can be undertaken. 3.2.8

Demographic Diversity

The increasing demographic diversity in R&D laboratories is a source of both opportunities and threats. On the one hand, it can facilitate performance and satisfaction by bringing different perspectives to the task, stimulating new ideas, gaining market access and legitimacy and being morally and legally responsible through discrimination avoidance (Pelled and Adler, 1994; Cordero et al., 1996; Ely and Thomas, 2001). On the other hand complexities are multiplied in a cosmopolitan and diverse team. Sources of conflicts and misunderstandings need to be managed. Different forms of communication and styles can hurt or frustrate individual creativity, cause lower performance and lead to dissatisfaction and stress (Tsui

R&D people

121

et al., 1992; Cordero et al., 1996). In these instances, formal actions such as diversity training programs, mentoring programs, equal opportunity career development programs, conflict management and leadership programs may be very useful. The creation of a more inclusive, diverse and cosmopolitan working environment can be a source of opportunities, but it also represents a fundamental change for the organization and the daily activities of companies. Typically, these changes do not happen overnight, and performance evaluation should include this as a possible variable to interpret fluctuations in many indicators. Comparative analysis (of different teams, functions, labs) can very much serve the need of HR and R&D managers to understand how better to capitalize on results and minimize organizational conflicts.

3.3

EVALUATING A FIRM’S R&D HUMAN RESOURCES MANAGEMENT SYSTEM THROUGH THE EYES OF THE SCIENTISTS

This section is structured into three parts. The first one introduces two drivers – projectification and open innovation – which are determining change in management of R&D workers. In fact, projectification and open innovation are among the most important elements that need to be considered to understand how the work of scientists and technicians is being managed and evaluated. Nonetheless, despite the recognized relevance of these changes, there is no comprehensive coverage in the literature of the extent and impact of this new scenario on the micro dimensions of R&D management and on organization and HR performance evaluation. The second part of this section introduces a model which includes some of the most relevant dimensions of HRM to consider when analysing the impact of the two abovementioned factors on R&D people. The model is thus meant as a tool through which companies might eventually evaluate whether their HRM practices in R&D are consistent with the evolution of the competitive context and the innovation strategy adopted at firm level. In the third part, the model is used to analyse data from Italian industrial firms in order to describe the impact of open innovation and projectification on HRM practices in R&D. 3.3.1

Emerging Trends in R&D Strategy and Organization

Projectification The organizational changes that have occurred in the last decades have radically transformed the way firms perform their R&D. Many of them

122

Evaluation and performance measurement of R&D

have shifted from functional structures to matrix structures mainly dominated by the project dimension, which is considered more suitable for R&D since it better integrates the knowledge of specialists and sustains organizational flexibility (Katz and Allen, 1985). Projects have become the standard mode of organizing R&D activities (Bredin and Söderlund, 2006), and this phenomenon has progressively given more importance to project management (PM) in R&D (Midler, 1995; Söderlund, 2005), with visible effects on HRM practices (Hobday, 2000; Packendorff, 2002). Effective HRM and PM are now more than ever important, even if projects have played a key role in modern industries since the 1950s (Mintzberg, 1983). Since then, the increasing emphasis on projects – the so called ‘projectification’ (Midler, 1995) – and the shortening of lead times has generated what some authors call ‘project intensification’, which occurs when more activities are organized in projects at the same time as work in projects is intensified and compressed (Bredin and Söderlund, 2006). This phenomenon has led HR managers to reconsider their role and some of the tools they have been using for a long time. Together with some form of organizational change, other new needs have emerged: among them, the coordination and prioritization of projects (Engwall and Jerbrant, 2003; Lindkvist, 2004), and those related to adjusting supporting structures to the new organizational framework (Knight, 1977; Galbraith and Nathanson, 1978). The latter includes the management of information and control systems and affects many aspects of HRM, such as hiring, personal development, appraisal, job evaluation, manpower planning, performance measurement, career structures and so on. For example, as less time is dedicated to formal training, a lack of incentives for staff development emerges. As a result, common HRM issues such as motivation, commitment, empowerment, job satisfaction, time pressure and stress need to be revisited and considered under a new perspective. Some authors take into account the consequences of ‘project intensification’ on R&D management and its implications for R&D HRM. In their study of European firms, for example, Larsen and Brewster (2003) point out that the increasing use of matrix and project-based structures in knowledge-intensive organizations reduces the opportunities for a proper management of the long term development of individuals. According to McMeekin and Coombs (1999) the fact that employees spend more time with the project leader creates distance between the line managers and the individuals they are responsible for. Brewster and Larsen (2000) redefine both the HRM focus, driving it to the interplay between people, tasks and organization, and HRM itself, which is seen as an institutionalized way of handling the central issues of selecting, appraising, rewarding and

R&D people

123

developing people. Guest (1990) argues that HRM is not only the responsibility of the HR department and suggests some sort of decentralization process with managers at all levels integrated into HRM. Finally, in a recent work on R&D based companies, Bredin and Söderlund (2006) identify three main types of line managers: 1.

2. 3.

‘HR-oriented’ managers, who are responsible only for the staff, their competence and development, evaluation, compensation, and for balancing and planning their project participation; ‘task-oriented’ managers, focused on the technological or scientific development in the line unit; ‘balanced’ managers, focused on both HRM and tasks.

Moreover, they provide an overview which summarizes the main implications of project intensification, for which they define the following HRM practices: 1.

2.

3. 4. 5.

‘line managers’ competencies’: the new role of line managers is to understand, profile and develop human resources in the labs. It implies the need for HR-oriented line managers; ‘career paths’: development of separate career paths for line managers, project managers, and specialists. Introduction of ‘dual ladder’ systems (Allen and Katz, 1995); ‘project management competencies’: increasing importance of team, knowledge and technology management; ‘competence development’: the project environment leaves little space for formal training and development; ‘evaluation/compensation’: separation of performance and evaluation/compensation, putting greater pressure on collaborative efforts between managers.

Open innovation If ‘projectification’ has led academics and practitioners to find new ways of managing HR operating in R&D, an intense debate is now going on about more ‘open’ strategies that are supposedly diffusing in industrial R&D. The relationship between such phenomena and HRM in industrial R&D labs, however, has not been deeply investigated yet. The empirical study reported in this chapter represents one of the first attempts to look at the linkage between R&D strategy and HRM. It identifies and explains a set of variables related to the management of HR in R&D that fit with the reconceptualization of innovation proposed by Chesbrough (2003b) and is inspired by the example of P&G’s model of Connect and Develop (C&D).1

124

Evaluation and performance measurement of R&D

Generally speaking, globalization, competition and technological progress, in particular the diffusion of ICTs, has opened up new markets and affected the way in which firms innovate. According to Chesbrough, while in the 20th century large firms were dominated by a vertically integrated R&D model, they now seem to be switching to a system based on creating value from knowledge through networks, rather than from knowledge generated within the single firm. Many companies embrace globalization by outsourcing not only the manufacturing process, in order to produce at lower costs, but sometimes also core activities like R&D, in order to enjoy the benefits of a widespread range of resources. Value is created through the integration of internal and external flows of ideas and the core activity of R&D managers is evolving towards networking and knowledge sharing. An open innovation (OI) oriented company is then a company which seeks technical resources and ideas beyond its corporate borders, and which is able to exploit the results of its R&D investments not only through new products and services but also through other forms of commercialization. Chesbrough defines this phenomenon as a paradigm that assumes that firms can and should use external ideas as well as internal ideas, and internal and external paths to market, as they look to advance their technology. Open Innovation processes combine internal and external ideas into architecture and systems. Open Innovation processes utilize business models to define the requirements for these architectures and systems. The business model utilizes both internal and external ideas to create value, while defining internal mechanisms to claim some portion of that value. Open Innovation assumes that internal ideas can also be taken to market through external channels, outside the current business of the firm, to generate additional value. (Chesbrough, 2003a, p. xxiv)

The concept is graphically synthesized in Figure 3.1. Even if the OI theory has been widely and enthusiastically discussed by both practitioners and researchers (Gassmann, 2006), some authors are dubious about the real novelty of the model and argue that it may not represent a new paradigm in the organization of innovation (Dahlander and Gann, 2007).2 However, despite these doubts, it is widely acknowledged that managers are using new forms of organizational structure that better fit with open business models, and OI generally requires a differentiated approach to knowledge sourcing and development. As the success of firms is related to their ability to make profits from technological advances, there is a renewed interest in understanding the incentives for generating discoveries and inventions under the conditions brought by the OI system. Relationships among individuals, communication flows and aspects like

R&D people

125 New markets

Internal technology base Current market

External technology base

Boundaries of the firm External technology insourcing

Source:

Adapted from Chesbrough (2003a).

Figure 3.1

The open innovation concept

mobility, attraction, motivation and retention of talents have to be analysed in a new perspective in order to give HR managers operating in R&D labs the right tools to cope with change and define winning strategies. In any case, even if several contributions have already demonstrated the importance of openness, it is nonetheless still necessary to recognize its effects on HRM. For example, it has been shown that interaction with external actors is important in order to remain innovative (Freeman, 1991) as it determines absorption of technology (Ahujia, 2000), improved survival rates (Baum and Oliver, 1991), increased innovativeness (Baum et al., 2000; Stuart, 2000), improved performances (Hagedoorn and Schakenraad, 1994), and faster growth (Powell et al., 1996; Stuart, 2000), but no particular attention has been paid to its effects on HRM. The idea that innovations are brought to the market through a sequential, linear process has long been superseded by more realistic accounts of the innovation process as iterative and involving feedback loops among actors (Kline and Rosenberg, 1986). More precisely, according to Von Hippel (1988) innovation comes both from a firm’s internal investments in R&D and from a variety of external sources including competitors, consumers, public research institutions, universities and other types of organizations. A need for new R&D roles therefore emerges. They have to deal with the latest tools that firms use in formal relationships, such as licensing agreements, alliances, and joint ventures, as well as in informal relationships (non-contractual personal relationships) to source expertise outside their boundaries (Powell et al., 1996).

126

Evaluation and performance measurement of R&D

Nelson and Winter (1982) model the firm’s decision to search for new technology outside its own organization. Cohen and Levinthal (1990) write about the ‘two faces’ of R&D (inside and outside the firm) and remark on the importance of investing in internal research in order to be able to utilize external technology. If firms cannot develop this ‘absorptive capacity’, they may utilize strategic alliances in order to acquire such knowledge, or complementary resources to exploit that knowledge (Gulati, 1998; Noteboom, 1999). In OI external knowledge plays a role equal to the one previously attributed to internal knowledge, and therefore many new models are being developed to explain how firms can better exploit external knowledge: imitation, consumer consulting, partnership with public institutions like universities, networking. A comprehensive understanding of how these supposed changes influence business dynamics and daily operations of R&D labs has not been developed yet. We do not know exactly to what extent the shift towards an extended definition of R&D needs consistent changes in HR organization and management. At the micro/individual level, only a few HRM studies adequately take into account these emerging concepts and the complexity of the impact on personal incentives and productivity. Love and Roper (2002) and Cassiman and Veugelers (2006), for example, explore complementarities between internal processes of innovation and accumulation of external knowledge. In general, it is fair to say that there is still an insufficient understanding of the specific micro-level changes brought about by the adoption of open models of innovation on researchers’ behaviours and the effects of these on the management and evaluation of HR in industrial R&D. We can argue that evaluating and managing R&D professionals are activities that are increasingly difficult to plan and implement due to the complexity and novelty of the multiple roles that researchers have to understand and play, doing research in more open environments. HR managers are in fact facing an increasing volume of issues related to openness: from the attraction and retention of the talents who are more suitable for ‘open research’, to the management of communication flows among researchers, among subsidiaries and so on. If we reinterpret the OI paradigm, R&D workers are asked to act like ‘porous sponges’, absorbing external inputs of innovation to be transformed and improved inside the company’s labs; to do this they have to interact with a large number of external actors and adapt frequently to the changing conditions of the business environment. This way of working is very different from the ‘closed innovation’ paradigm, where most of the tacit knowledge and the technical competencies to perform R&D tasks were supposed to be available within the laboratory, if not within one single team.

R&D people

127

In the new framework, the role and relevant characteristics of R&D workers are being redefined, both at a formal level, through a new organizational chart, and in the development of daily operations. In an open innovation company managers need to be aware that the smartest person for the project is probably not working for his or her company (Chesbrough, 2003b), and this means that a new set of skills is requested for an ‘ideal’ scientist and technician, that greater emphasis needs to be given to HR management and performance evaluation, that much attention has to be paid to the bottom-up ‘weak signals’ coming from one of the most precious resources a company has. 3.3.2

A Model for the Evaluation of Industrial Firms’ R&D HRM Practices

One of the main purposes of this chapter is to explore the impact that projectification and open innovation are having on the HRM practices employed in industrial R&D laboratories. With this aim, an empirical analysis focused on the R&D labs of a sample of Italian firms operating in high-technology industries was undertaken. In order to support this analysis, a simple descriptive model that identifies the main dimensions around which the HRM system for R&D professionals can be designed and studied was developed. This framework is also useful for an accurate evaluation and diagnosis of a firm’s R&D HRM system with the purpose of understanding whether and to what extent it is consistent with its competitive strategy and environment. The starting point is that R&D people can be described through three groups of variable. The first one comprises their ‘skills’. In other words, researchers have an education background and a number of professional experiences, and have accumulated scientific, technical and managerial knowledge throughout their careers. A number of questions were put to R&D workers about these aspects in order to implement the model. Specific questions about their skills which are not directly related to science and technology but, rather, to ‘business and management skills’ were introduced as well. These skills comprise the capacity to read and write business and exploitation plans, to estimate the value of inventions, to understand clients’ and suppliers’ roles in a competitive environment and so on. Second, ‘training’ activities have a double effect on R&D people: they increase the workers’ knowledge base and they contribute to their satisfaction (or dissatisfaction), since they represent something that R&D people generally appreciate to increase their attractiveness on the job market. Researchers were asked to describe the characteristics of their current

128

Evaluation and performance measurement of R&D

Training

Skills

R&D workers

Conflict & stress

Performance

Incentives

Planning

Figure 3.2

Dimensions considered in the analysis

training activities as well as give information about their motivations and expectations. A third set of factors has to do with planning practices in the lab, including the organization of work, and in particular the incentives that are offered to R&D people to sustain and encourage their research efforts. R&D workers are a special type of knowledge worker, and they are particularly sensitive to incentives, both monetary and non monetary, which they might consider as improvements in their professional and/or personal life. In this case researchers were asked to provide their perceptions about planning and incentives in their companies. The three mentioned groups of variables influence R&D workers, who might in turn cause or suffer from conflicts and stress in the lab. With regard to this, researchers have been asked for their opinion on several types of conflict and stress situations with different categories of people in the lab. According to this model (Figure 3.2), R&D workers’ performance, that is, their capacity to generate inventions which can become valuable assets for their companies, is influenced by their skills as well as HRM practices, whose evaluation and discussion is a core issue of this chapter. 3.3.3

Empirical Analysis

Issues and methodology: HRM through the eyes of scientists A first attempt to empirically investigate the trends occurring in R&D HRM in light of the emerging projectification and open innovation trends is here presented and has been previously discussed in Di Minin, Piccaluga and Rizzone (2008).

R&D people

129

A bottom-up perspective has been used, based on a set of 53 questions put directly to a large group of individuals currently working in industrial R&D laboratories of Italian high-tech firms.3 The sample includes small, medium and large firms, and the purpose is to benchmark how different organizations are experiencing and living the changes discussed in this chapter. The information obtained from the survey was used to describe three main dimensions of the HR organization and HRM practices: 1. 2.

3.

‘Descriptive parameters’ (for example demographics, productivity, time horizons, satisfaction, expectations, mobility, education); ‘Job organization aspects’ (for example teamwork vs. individual research, flexibility, decisional centres, work time allocation, type of relationships, communication flows); ‘HRM tools’ (for example talent attraction, training, evaluation methods, goal definition, roles, leadership, responsibility, incentives, career systems, problem sources).4

The empirical investigation is not a comprehensive climate analysis of an R&D lab. Rather, it was aimed at investigating the critical impacts of projectification and open innovation on HRM practices and evaluation. The research design is therefore biased towards the analysis of these dimensions. For such purposes, much emphasis is placed on the role of team (size and composition), individual skills and motivation, business consciousness and interaction abilities. Training methods and objectives are discussed, and particular attention is also paid to the incentive system and the planning activities. It is believed that putting directly a number of questions to a large number of individuals from a few companies represents a useful perspective of analysis that is complementary to other studies based only on interviews with R&D directors. In fact, comprehensive evaluations of HRM practices in R&D surely require the direct involvement of R&D people themselves. Appendixes A3.1 and A3.2 provide a number of methodological details about the empirical analysis on which the remainder of the chapter is based. Findings: the impact of projectification and open innovation on HRM practices Some general trends emerging from the analysis are here described. These findings are based on early observations of the population, and on meetings with HR managers in some of the companies that agreed to take part in the study.

130

Evaluation and performance measurement of R&D

Conflicts and stress It emerges that ‘traditional sources’ of stress still remain very relevant for R&D scientists. Time constraints are the most serious stress factor for R&D workers. Dissatisfaction is also an important source of internal conflicts, especially when researchers are not enjoying their work and they are unhappy about their salaries or career path. These sources of stress, in turn, are also correlated with conflicts within the team and the organization, and eventually with a poor self-evaluation in terms of performance. However these problems can be exacerbated in an OI environment. It was found that: 1. 2.

3.

individuals more exposed to OI-based projects are also the ones more likely to generate conflict with R&D managers; researchers with some management background, more cosmopolitan and better informed about the ‘dual ladder’ career system tend to be more likely to criticize managers and to be in an uneasy situation with their working environments; the star scientists who perceive that their job is not fully appreciated and valued are more likely to leave the company.

As an organization opens up to various fields and enters different technical domains, it is necessary to organize work around multi-disciplinary projects. HR managers should however make sure to manage specialists properly as they are likely to get into conflict with other team members. On the other hand, it was found that technicians fully immersed in R&D activity ‘strictu sensu’ are less likely to conflict with colleagues. As far as the project-level is concerned, OI projects, characterized by more frequent contacts with clients and technology suppliers, and by international and cross-company teams are more at risk of conflict than more traditional in-house projects. Good leadership, clear planning and a definition of roles was confirmed to be the most effective cure to stress and conflicts in time of tumultuous change, especially when resources are meagre and it is difficult to provide everybody with what they expect. Providing feedbacks to criticism is also a crucial step to take in order to avoid ‘star scientists’ leaving the organization. The greater emphasis on projects meant that tutorship and advice from senior team members worked much better than other mechanisms to build reputation, improve relationships within teams and create a pleasing working environment. Greater emphasis on projects and increasing stress factors lead to a more relevant role for teamwork. Teams are often organizational units

R&D people

131

whose members respond to various cost centres, while the evaluation of their work depends on the decisions and performance of the group they are working with. In open environments teams form and dissolve constantly, creating greater flexibility but also greater uncertainty and more complex management. The analysis revealed that in firms characterized by significant interaction with suppliers and customers, team leaders are granted greater responsibilities and decision autonomy. In these situations face-to-face meetings occupy a large share of the researcher’s work. Good teamwork is key for a conflict-free corporate environment. Companies with a long experience in teamwork develop an inclusive team leadership style, and team members feel part of the decision making process. Team leaders who are recognized as capable and talented by researchers are given on average more autonomy and responsibilities. The sources of stress and conflicts discussed earlier can be mitigated: 1. 2.

if the team is characterized by good and inclusive leadership; if R&D people within the team are confident that the team as a whole is in control of the key competences needed to solve a problem.

In turn, decision making becomes harder when researchers perceive that the company is moving away from its ‘comfort zone’, and when competences need to be acquired from other teams or companies. The NIH (Not Invented Here) syndrome is therefore clearly at work in the studied sample of researchers. The role of the decision maker is traditionally played by managers. However, firms that embrace open innovation are more likely to give higher responsibility to researchers working in teams, and to those who have been identified as leaders by informal communities of practice and networks. This leads to a more intense flow of information within the firm, but also to an organization which is more complex and difficult to control. Skill R&D workers in an open innovation environment might be asked to cope with ‘research exploitation’ issues. In fact, they are more and more often asked to think about the commercial implications and the value added for the firm’s profitability of their work. Nevertheless, they do not normally receive a formal training in management, and have difficulties in speaking the language of their colleagues in the finance or sales departments. Therefore scientists and technicians in the sample were asked about their confidence with these topics and it emerges that, on average, the industrial R&D worker has become quite familiar with the need to find

132

Evaluation and performance measurement of R&D

possible applications for his/her ideas, but not adequately skilled to concretize them through the support of business plans and market analysis. About these skills, the data confirm the importance of learning by doing, since researchers report that informal coaching has a greater impact than formal training on their performance, in particular when it comes to R&D commercialization activities. Business consciousness is not evenly distributed across individual researchers or teams. R&D people that show a better attitude towards these topics are on average also more satisfied with their performance when business success is obtained, and they are also conscious about the potential benefits coming from linkages with academia, with multidisciplinary teams and with customers. They seem to be readier than others to face the speed of change in finding successful solutions to specific problems, eventually playing the role of ‘connectors’ by asking for contributions from external entities, such as universities or customers, or taking advantage from the multidisciplinary composition of internal teams. These are indeed the scientists and engineers who are more eager to accept to work in an OI environment. Considering these findings, it is possible to suggest that in the labs where respondents work there is evidence of a self-reinforcing mechanism: when OI and projectification are at work, teams and individual researchers get exposed directly to the commercial results of their activities. This means that when commercial results turn in, they become more motivated and more prepared to consider the business implications of their research. This in turn makes them more likely to accept an OI framework. Obviously in the case of commercial failures, the fact that the lab is less insulated from the market can have the same self-reinforcing, albeit negative, effect. Because of their familiarity with business issues and their precious ‘linking capabilities’, that appear fundamental in open systems of innovation, HR managers should be aware of the importance of such dynamics. Training Continuous education programs seem to have some positive impacts on climate. When recognized as useful and well performed, they can act as an incentive and increase the sense of belonging to a firm, as workers appreciate them as part of their career development. It is however through peer tutoring that researchers, especially younger ones, obtain more concrete results for their daily activities. Tutoring is also perceived as particularly beneficial in those contexts where teamwork is a relevant part of researchers’ work, and for teaching R&D personnel to deal with customers and suppliers. Besides formal training, therefore, encouraging senior researchers to train and motivate younger colleagues has positive effects on corporate

R&D people

133

climate, and it is a way to reinforce interpersonal relationships. It promotes trust among colleagues, acting as an invisible ‘internal glue’. HR managers should not miss this point, but consider it as a sort of long term investment to accumulate knowledge, which is useful for coaching purposes too. While the value of informal training is widely acknowledged, researchers are less able to identify the benefits of more formal training on their performance, in particular for non-technical topics. For example, it was found that on average it is significantly more difficult to convince researchers about the relevance of some basic management training, whereas researchers are keener to sit in technical/scientific classes, even if these topics are related to areas quite distant from their domain of expertise. Researchers are interested in management courses only when willing to pursue managerial or entrepreneurial careers. They see informal training as a stepping stone for acquiring the managerial skills they need. The bottom-up perspective suggests that R&D managers should pay attention to coaching, stimulating strong relationships among colleagues and creating collaborative environments where professional growth is stimulated and talents are trusted and empowered. Moreover, they should let information flow freely and ensure adequate levels of transparency and clearness about single tasks’ attribution and goals’ definition. Incentives Traditional incentives like monetary benefits and career advancement remain the most powerful signals management can give to researchers about their performance. Uncertainty about career or the perception of an unfair distribution of benefits is a natural killer for morale in any type of organization. In general, across the sample performance evaluation systems were well accepted by researchers. Nevertheless, it was possible to isolate large groups of people more frustrated by performance evaluation mechanisms, and these were not only the younger and less experienced, but also some of the most capable and senior R&D people. The bottom-up analysis reveals that each researcher has an individual understanding of what should represent the output of his/her job, and therefore the reference point for his/her evaluation. For some of the respondents this was the bottom line for the company, the value created for the customers. Nevertheless a good share of respondents identified their scientific achievements as the real bottom line for their lab. A necessary condition for incentives to work is that they have to be perceived as fair by R&D workers. The amount of resources available and the types of rewards do not matter at all: the analysis shows that there cannot be an effective incentive system unless a proper evaluation mechanism is

134

Evaluation and performance measurement of R&D

in place. However, the lack of consensus about the perception of the relevance of commercial versus scientific results makes the condition of fairness extremely hard to achieve. Companies living the transition towards the OI paradigm should indeed consider this problem, and try to mitigate situations of potential conflict. Planning Open innovation is not a synonym for last minute or unstructured planning. However, given the uncertain nature of R&D work, resource allocation and long term planning for the R&D department might be a difficult (yet necessary) activity. In an open innovation environment, in particular, planning has to be dynamic and ready to adapt to emerging opportunities and obstacles. This, in contrast, does not help to provide the certainty, transparency and objectivity researchers long for. The analysis tried to explore how R&D workers feel about the implementation of planning practices, and how this is influencing their work. It is fair to say that scientists and technicians do recognize the importance of planning, even if they are not likely to acknowledge that explicit planning is a key success factor for the firm they work for. In particular, in environments characterized by OI, researchers are more likely to suggest that companies should be more permeable, and performance objectives should be adapted to changing scenarios. Task planning and explicit goal definition are considered a necessary condition for a firm to achieve its goals, but far from a sufficient one. Rather, it emerged that planning is considered the single most visible signal of good management and is strongly correlated with top management reputation, competence and reliability. Scientists perceive sound and well organized planning as a signal of the solidity of the company, an indicator of how capable their managers are. However, they do not expect that planning can solve their daily problems, and they are confident that plans are likely to change. Nevertheless researchers who perceive planning as poor are more likely to suffer from stressful situations, do not recognize the fairness of their rewards, and are ultimately more likely to search for professional growth outside the company.

3.4

CONCLUSIONS

This chapter has focused on the analysis and evaluation of HRM practices adopted by industrial firms in their R&D labs. It is believed that its novelty is related mainly to the ‘bottom-up’ nature of the empirical analysis it reports, with questions directly put to R&D workers, whose opinions have not been ‘interpreted’ or ‘translated’ by HR or R&D directors. The survey

R&D people

135

draws upon a theoretical model which includes some key variables that have to be considered when assessing HRM practices in R&D. These variables have been analysed with specific attention to two increasingly important concepts for R&D processes: open innovation and projectification. This chapter tried therefore to ‘look through the eyes’ of a group of industrial R&D workers in order to obtain some early findings about the changes that are occurring in industrial R&D labs. Perceptions, feelings and behaviours of 330 researchers were investigated in order to gain evidence about new trends and practices related to the emerging open innovation paradigm. The rationale behind this analysis is that important managerial implications may derive from a better understanding of individual perceptions and behaviours of R&D personnel. In fact, the changing pattern of innovation processes implies parallel changes in the organization of R&D labs, along the lines of the main drivers identified in this chapter. Nonetheless, the role and perceptions of a fundamental component of research centres, that is, researchers themselves, are not always adequately considered by managers, either because they do not have this kind of information or because they do not adequately use feedback coming from climate analysis and other types of survey. In order to increase satisfaction, motivation and ultimately productivity, managers should also consider the opportunity to make research findings available outside the boundaries of the firm through the diffusion of clear rules on the patenting of research results, support researchers who pay attention to market dynamics from the earlier stages of the innovative processes, give them chances to grow and, obviously, constantly monitor personnel satisfaction. It also seems necessary to arrange mechanisms to further ‘open’ R&D practices, by encouraging internal and external communication, team work and external collaboration, as well as to promote stronger relationships with the scientific community. It is also important to create an adequate environment to allow researchers to cultivate interpersonal relationships. Nevertheless, it still remains fundamental to guarantee highly selective standards in the hiring process and to schedule continuous training for researchers to maintain very high levels in their qualification standard. Finally, R&D managers should fix goals as precisely and clearly as possible, giving researchers the opportunity to choose how to achieve the result and asking qualified project leaders (focused on specific purposes) to supervise and control R&D processes in order to appreciate researchers’ commitment and liability. Open innovation and projectification are definitely perceived by researchers as the new frontiers of their activities and are likely to play

136

Evaluation and performance measurement of R&D

an increasingly important role in the evolution of HRM practices in R&D. However, this chapter has important implications for research and also provides a number of interesting clues for future investigation. First, it is one of the few contributions that explore the micro-organizational implications of open innovation. Analysing the impact that open approaches to innovation management are likely to have on the skills, motivation, rewarding, recruiting and performance measurement of R&D professionals represents a very promising avenue for future research. In this respect, the model introduced in this chapter could be used to gather and interpret further empirical data with the aim of developing a richer understanding of the phenomenon. Second, the chapter proposes and uses a method for the analysis of HRM practices in R&D that relies on data collected ‘through the eyes of the scientists’. It represents a methodological approach that can be successfully applied to study other issues related to the motivation and satisfaction of human resources and, especially, highly professional workers, as it overcomes some of the main limitations that affect surveys relying on the responses of R&D managers and department heads.

APPENDIX A3.1

DESCRIPTIVE STATISTICS OF THE SAMPLE USED FOR THE EMPIRICAL ANALYSIS

The analysis previously discussed is based on an empirical investigation of Italian R&D intensive companies. Data collection started in 2007 and is still in progress. Table A3.1.1 provides some descriptive statistics about the sample.

APPENDIX A3.2

FACTOR ANALYSIS USED IN THE EMPIRICAL INVESTIGATION

The results from the survey are difficult to interpret only through a correlation analysis. The survey led in fact to the creation of more than 180 variables and each single variable contributes to the puzzle, helping define the characteristics of the individual researcher and his/her team and company. Factor analysis was therefore used to combine and better isolate key characteristics and common traits of the researchers in the sample, observing common dynamics in answers across various questions. To perform factor analysis variables referred to the same argument were first

R&D people

Table A3.1.1

137

Descriptive statistics of the sample

Number of individual respondents

330

Number of companies represented

28

Level of Education (%) Ph.D. University Degree High School Diploma No High School N/A

6.5 56.5 8.6 28.5 6.5

Average age of respondents

38.4

Average working experience (years)

12.1

Average time spent at the company (years)

11.4

Average % of working time spent on: R&D projects Training Interaction with clients Other activities

64.2 7.1 10.9 17.6

manually grouped. Then, three different analyses on these clusters were performed, in order to single out relevant factors. R&D Professionals’ Motivation The first group of variables relates to dimensions that can be somehow associated with motivation. The following five factors were retained and interpreted: 1. 2. 3.

4. 5.

Team spirit 5 f (Collaboration with colleagues, Skilled leaders, Lab and team members cohesion); Defined objectives and roles 5 f (Clear goals, Tasks and roles definition, Responsibilization); Trust in acknowledged valid people 5 f (Involvement in decision making, Engagement acknowledgement, Efficient responsibility system); Focus on professional growth of the researcher 5 f (Training programs, Information circulation, Engagement acknowledgement); Collaborative environment 5 f (Information circulation, Collaboration among colleagues, Skilled leaders).

138

Evaluation and performance measurement of R&D

These aggregations indicated that: 1. 2. 3. 4.

5. 6.

team spirit is fundamental for motivating researchers and increasing their productivity; it needs reciprocal trust among prepared people and sharing of relevant information; collaborative work environments motivate people; the most motivated researchers are those who are given trust through responsibility for some business and involvement in decisions, and those whose firms take care of personal growth, offering training programs; engagement, promotion and acknowledgement are important motivation tools; clear roles and defined goals are fundamental for motivating researchers.

R&D Professionals’ Satisfaction Factors A second factor analysis was performed to identify the most relevant sources of satisfaction as suggested by R&D workers. Five relevant factors were retained and interpreted: 1.

2.

3. 4. 5.

Good work environment populated with reliable people 5 f (Collaborative team, Pleasant firm atmosphere, Good team leader, Top management quality); Institutional external acknowledgements of the work done (popularity) 5 f (Opportunity to patent and publishing, Scarce relevance of work environment); Lone-wolf ‘arrogance’ 5 f (Bad external relationships, Poor top management skills, Sense of personal growth, Business successful results); Market appreciation of research results 5 f (Good external relationships, Commercial success, Scarce relevance of work environment); Professional growth opportunity (unique most relevant variable).

It emerges that: 1.

2.

in companies with a pleasant work environment researchers’ satisfaction is higher, greater if leaders and managers are cordial and competent; researchers are not so much looking for internal but external esteem through ‘institutional’ (such as patents and publications) or ‘market’ acknowledgements (result of commercialization);

R&D people

3.

4.

139

commercial success increases self-confidence in R&D workers and may also be a sort of personal revenge for their commitment, which is not always adequately valued by top managers; professional growth opportunities are important factors of personal satisfaction.

Perceived R&D Success Factors Finally, the sources of R&D success as perceived by researchers, were investigated. Nine different factors were defined: 1.

2.

3. 4.

5.

6. 7. 8. 9.

Adaptability and permeability (Open Model) 5 f (Job rotation, Links with universities, Publications, Multidisciplinary teams, Growth sense, Continuous training, Cooperation, Interaction with customers); Presence of stimuli to react 5 f (Multidisciplinary groups, Links with universities, Autonomy, Interesting and stimulating job, Lack of explicit goals, Conflicts, Insufficient confidence); Team play 5 f (Cooperation, Reciprocal support, Efficient communications); Focus on specific goals, with constant control and supervision performed by project leader 5 f (Efficient evaluation system, Engagement acknowledgement, Qualified project leader, Specific projects); Clear mission 1 Free methods (only results matter) 5 f (Explicit goals, Autonomy, Trust, Lack of relationships with customers, Lack of publications); R&D workers’ empowerment and burdening 5 f (Interesting and stimulating job for technicians, Autonomy, Lack of valuation); Qualified personnel 5 f (Training opportunity, Skilled technicians, No job rotation); Reciprocal trust 5 f (Confidence, Respect, Reliability among colleagues); Freedom of execution (unique most relevant variable).

On the basis of these observations it turns out that researchers in successful firms: 1.

2. 3.

react to endogenous or exogenous stimuli adopting an R&D model open to external inputs and ready to adapt easily to change; this model seems to lead back to the open innovation paradigm; stimulate reciprocal trust among colleagues in order to favour team play, which is useful to face innovation challenges; are highly qualified and burdened;

140

4.

5.

Evaluation and performance measurement of R&D

know that autonomy related to technicians’ empowerment needs a mission clearly defined by explicit goals; it is important to achieve them but not to define how to; put the task of focusing on single objectives into the hands of a project leader, who is capable of coordinating and assessing colleagues, and is trusted and appreciated by them.

NOTES 1. For more details about the C&D Model see also Huston and Sakkab (2006). 2. In 1919 Marshall, speaking about the division of labour and growth of knowledge, had already talked about the growth of external economies associated with the wider publication of research results, providing the opportunity for small firms to innovate if leadership is available: external organization becomes more important as the innovation system opens up due to wider dissemination of results. Later, Freeman (1974) defines corporate R&D laboratories as vehicles for absorbing science and methods in internal firm processes to make them more efficient. In a study on the iron production industry in England, Allen (1983) discovers the ‘collective invention’, observing that firms regularly shared designs and worked in a distributed and open manner. In the twentieth century, however, the benefits of scale and scope for internal R&D (relative to external market) gave rise to a ‘proprietary’ innovation model where large enterprises internalized their firm-specific R&D activities and commercialized them through internal development, manufacturing and distribution processes. The rise of the corporate R&D laboratory in American manufacturing was due to the costs of organizing innovation inside the firm, relative to organizing innovation relying on the market (Mowery, 1983). From the technology base created by internal R&D, firms naturally moved to exploit their accumulated knowledge to develop new products, thereby enhancing their economies of scope; in many industries large scale dedicated R&D functions emerged, providing a barrier to entry through economies of scale (Chandler, 1990). 3. Data collection started in late 2007 and is still in progress. Some of the findings of this study, emerging from an initial sampling of the collected answers, are briefly reviewed here. 4. Industrial R&D workers’ perceptions were collected through the use of an online survey platform, specifically configured for the purpose. The web-based collector system allowed us to monitor data entry errors, improve clearness and obtain a high completion rate. Moreover, the growing dataset of answers has been available for real-time comparisons and benchmark tuning.

PART II

Financial markets

141

4. 4.1

R&D and financial investors INTRODUCTION

This chapter aims to analyse the valuation of R&D investments at the firm level, taking the perspective of the investors operating in the financial markets. It will deal, in particular, with the relationship between R&D investments and the market value of traded firms, explaining why and how R&D investments should be reflected in financial investors’ valuations and stock market prices. It is well known in the business and economics literature, as well as in professional accounting practice, that R&D investments affect firm performance, expected profits, and cash flows. Since in efficient financial markets investors evaluate a firm based on its expected cash flows (that is, a firm’s market value should be equal to the present value of all the expected cash flows produced in the future), R&D investments should also be reflected in market values. Moreover, stock prices should embed all the information currently available on the firm’s R&D investments and should react to any new information about those investments (see Figure 4.1). Under these conditions, stock markets can provide useful information on the value and the expected performance of R&D investments. In particular, because the returns to R&D investments may be spread over a number of years, a forward-looking and market-based measure, such as stock price, which includes in principle all the expected effects of R&D investments over the firm’s entire life cycle, may be more suitable than short-term accounting indicators such as return on equity (ROE) or earnings before interest, taxes, depreciation and amortization (EBITDA). The stock market valuation of corporate R&D investments could then contain important signals. In particular, the market reaction to specific announcements about R&D-related decisions could supply different actors with information about the expected value creation of those decisions (see the feedback line in Figure 4.1). Recent theoretical work has proposed that managerial behaviour can be strongly influenced by stock market reactions to decisions related to R&D and technology (Benner, 2007). Empirically, Munari et al. (2005) have shown that different financial systems can affect, ceteris paribus, the level of firms’ R&D investments. For these reasons, a number of researchers have turned to stock market value as an indicator of the firm’s expected economic results from investing 143

144 R&D investment

Figure 4.1

Evaluation and performance measurement of R&D Firm’s expected cash flows

Firm’s market value

Stock prices

The relationship between R&D investment and stock prices

in R&D. Different approaches have been adopted in this respect. Some studies have analysed the relationship between R&D investments and a firm’s market value at a given time (Griliches, 1981; Jaffe, 1986; Hall, 1993a, 1993b; Hall and Oriani, 2006, among others), whereas others have focused on the short-term returns following corporate announcements about R&D activity (Chan et al., 1990; Woolridge and Snow, 1990, among others). Many of the results are relatively robust across different studies. From the current theoretical and empirical research on the market valuation of R&D investments, it is believed that three topics deserve particular attention. First, the most important question is whether R&D investments do create value for the firm. In general, the conclusion is that stock markets value firms’ R&D investments positively. The market value of traded firms is positively affected by R&D investments (see Hall, 2000 and Czarnitzki et al., 2006 for a review and Oriani and Sobrero, 2003 for a meta-analysis), and stock prices react positively to announcements of new R&D investments (for example, Woolridge and Snow, 1990; Szewczyck et al., 1996). However, several studies have shown that the market valuation of R&D investments is volatile over time (for example, Hall, 1993a, 1993b), across industries (Jaffe, 1986; Cockburn and Griliches, 1988), and across countries (Hall and Oriani, 2006). This variability draws attention to the potential factors that affect the market valuation of R&D investments and the criteria that investors in the stock market adopt to evaluate these investments. Second, expected results from R&D investments are subject to a very high degree of uncertainty (Mansfield et al., 1977). As a consequence, it is often hard to predict how they will impact on firm value and this fact is important to consider when analysing the market valuation of R&D investments (Oriani and Sobrero, 2008). An interesting question is which methods and criteria investors adopt in assessing the expected performance of a firm’s R&D investments under uncertainty. Clearly, this issue is relevant to critical decisions of managers seeking shareholder wealth maximization, such as resource allocation to corporate innovative activities or the recourse to capital markets for R&D financing. Third, recent empirical work suggests that corporate governance issues

R&D and financial investors

145

at the country- and firm-level can affect the market valuation of R&D investments (Hall and Oriani, 2006). Substantial differences are observed in the market value of R&D investments across countries. Moreover, within country, this valuation also seems to depend on the precise ownership structure of the firm. Finally, it has to be said that several authors have proposed that financial markets are not always efficient in evaluating R&D investments, mainly because of the information asymmetries that R&D investments create between insiders, typically the managers, and outsiders, the investors (Aboody and Lev, 2000; Hall, 2002). This chapter does not deal with the efficiency and information problems, as they will be analysed in depth in Chapter 5. It is important, however, to remark here that information problems with R&D investments can imply a higher cost of capital for these investments and a consequent underinvestment at the firm level (Lev, 2004). The questions reviewed in this section are certainly important for researchers interested in the problems of R&D valuation and financing, but they are also relevant for other subjects. Managers will be interested in understanding how financial investors evaluate their R&D decisions, as this information is critical for making those decisions. Financial investors may find the models and the results help them to better predict the values of traded firms. Policy makers should find these issues relevant as stock markets can represent one of the most important sources of R&D financing. In particular, as financing innovation often produces market failures (Hall, 2002), policy makers could be very interested in understanding how the investors evaluate firms’ R&D investments in their country. This chapter is organized as follows. In the next section the theoretical and empirical bases of the relationship between R&D investments and market value are presented, and in the section following the main empirical models are described. In the fourth section the main empirical results are reviewed. In this respect, three main questions will be discussed: whether R&D investments create value, how investors deal with uncertainty and how different financial markets and ownership structures affect the market valuation of firms’ R&D investments.

4.2

THE RELATIONSHIP BETWEEN R&D INVESTMENTS AND FIRM MARKET VALUE: THEORETICAL AND METHODOLOGICAL FOUNDATIONS

In this section the bases for the relationship between R&D investments and firm market value are examined. First, the reasons why we should expect

146

Evaluation and performance measurement of R&D

such a relationship are explained, which requires reviewing the assumption of market efficiency and its implications. Second, the methodology for estimating the relationship is described. 4.2.1

R&D Investments and Market Efficiency

The use of market-based measures to assess R&D performance clearly requires some assumptions on the way financial markets work. In particular, it builds on the statement of stock market informational efficiency, which implies that security prices fully reflect all available information (Fama, 1970, 1991). Fama (1970) makes a well known distinction between three degrees of market efficiency corresponding to different information subsets: the weak form, in which the information set consists only of historical prices; the semi-strong form, in which prices adjust to other information that is obviously publicly available, such as public announcements; the strong form, in which given investors have access to any information relevant for price determination.1 Even though the debate is still open, there exists robust empirical evidence supporting the efficiency hypothesis, above all in the first two forms (see Fama, 1991 for a discussion). There is much more scepticism about the strong form of efficiency, above all because of the existence of information asymmetries between insiders and outsiders (that is, managers have private information that investors on the market do not have). While the existence and the consequences of R&D-related market inefficiencies will be discussed in detail in Chapter 5, it is worth highlighting in this section the fact that the most commonly used empirical models rely to some degree on the assumption of market efficiency. The assumption of market efficiency has several important implications for the relationship between R&D investments and market value. First, it implies that the market capitalization of the firm can be considered a reasonable proxy of its underlying value. Moreover, this value will change if and only if the stock market receives new general or firm-specific information that modifies investors’ expectations about the expected cash flows of the firm (Pakes, 1985; Woolridge and Snow, 1990). Consequently, if R&D investments create or increase intangible capital which is able to generate future cash flows, these investments will affect the market valuation of the firm (Griliches, 1981). Second, the holders of shares of the firm will agree that all decisions, including decisions about investments with pay-offs in the long-run, should be evaluated according to their contribution to the market value of their residual claims on these cash flows (Fama and Jensen, 1985). Therefore, managers acting in the interests of shareholders are assumed to make investment choices

R&D and financial investors

147

aimed at the maximization of corporate value. Under these conditions, it is possible to show that R&D programs and other investment policies are maximizing the expected present value of the firm’s future cash flows (Pakes, 1985; Hall, 1993b). These assumptions on stock market efficiency and their implications for investment lie at the basis of the empirical relationships and models described below. 4.2.2

The Empirical Relationship between R&D Investments and Market Value

In order to observe an empirical relationship between R&D investments and firm market value, it is necessary to define the observable variables of interest and their relationships. Pakes and Griliches (1984) have presented the path diagram shown in Figure 4.2. Whereas the empirical models have evolved over time, as explained in the next section, the diagram in Figure 4.2 still presents very clearly the general framework on which these models build. In particular, it relates the unobservable DK, which is the net addition to knowledge capital K during a particular time period, to a set of observables (patents and R&D investments), random disturbances (n, w), and several indicators of performance (Z), which may include the stock market value of the firm. Firm performance is also assumed to be influenced by other observable variables, such as investment and labour input (X) and unobservable effects (e). The disturbance w reflects the effects of informal R&D activities and the inherent randomness of inventive success, whereas n represents noise in the relationship between the patents granted to the firm and the associated increment to total technological knowledge. Based on Figure 4.2, the empirical analysis of the relationship in which we are interested requires us to build measures for R&D and firm market value and then to define the functional form of the model linking these two variables. This will be discussed in detail in the next section of the chapter.

4.3

EMPIRICAL MODELS

In this section two general categories of empirical models that analyse the relationship between R&D and market value are presented. The first category is a group of models that relate the flow or the stock of R&D investments to the market value of the firm (often measured relative to tangible assets, that is, as Tobin’s Q) at a given moment in time. The

148

Evaluation and performance measurement of R&D

X ␧

Z

⌬K

␧

v

Pats

R&D

Source: Adapted from Pakes and Griliches (1984).

Figure 4.2

The empirical relationship between R&D investment and firm market value

second category is models that relate the arrival of new information on R&D investments (R&D announcements) with changes in the stock price (stock returns). Clearly, the two different classes of model examine slightly different questions, but they complement each other. The former tell us how the stock market is evaluating the resources that have already been invested in R&D at a given point of time, whereas the latter measures the incremental change in expected future cash flows due to an increase or decrease in R&D spending. Both are capable of yielding an estimate of the marginal value to the firm of an additional dollar (or other currency unit) spent on R&D at a given point in time. Each of these models is discussed in turn in the next two subsections of the chapter.

R&D and financial investors

4.3.1

149

Empirical Models of Tobin’s Q

The studies analysing the relationship between R&D investments and market value at a given time implicitly or explicitly assume that the stock market values the firm as a collection of tangible and intangible assets that are expected to yield cash flows in the future (Griliches, 1981). The dependent variable in these models (that is, the measure of market value relative to tangible assets, Tobin’s Q) is normally proxied by the ratio between the market value and the book value of the firm’s physical assets (plant and equipment, inventories, investments in other firms, and so on). These assets are those that appear on the balance sheet of the firm according to most accounting standards. Typically they exclude some intangible assets, such as those created by the firm’s own R&D investments.2 The market value measure is the sum of the current value of common stock, preferred stock, and debt ‘marked to market’. The model is outlined here, using a treatment that follows Hall (2000) and Hall and Oriani (2006). In equilibrium, the ‘shadow’ or marginal value of any asset results from the interaction between the capitalization of the firm’s expected rate of return from investment in that asset and the market supply of capital for that type of asset (Hall, 1993a). Assuming that we can enumerate all the cash-flow generating assets that compose the firm, it is possible to represent the market value V of firm i at time t as a function of these assets: Vit 5 V (Ait, Kit, I1it, . . ., I nit)

(4.1)

where Ait is the book value of tangible assets, Kit is the replacement value of the firm’s knowledge capital, in our case measured by the stock of R&D investments (R&D capital),3 and I jit is the replacement value of the jth other intangible asset. If assets enter value in a purely additive way, and ignoring the other intangible assets for the sake of simplicity, it is possible to express the market value of the firm as follows:4 Vit 5 b (Ait 1 gKit)s

(4.2)

where b is the market valuation coefficient of the firm’s total assets, reflecting its differential risk, overall costs of adjusting its capital, and its monopoly position, g is the shadow value of R&D capital relative to tangible assets, and the product bg is the absolute shadow value of the R&D capital. In practice, bg reflects the investors’ expectations about the overall effect of R&D capital K on the discounted value of present and future earnings of the corporation, while g expresses the differential valuation of the R&D capital relative to tangible assets. When g is unity, a currency

150

Evaluation and performance measurement of R&D

unit spent on R&D has the same impact on market value of a currency unit spent on tangible assets. Conversely, values of g higher (lower) than unity suggest that the stock market evaluates knowledge capital more (less) than tangible capital. Equation 4.2 can be interpreted as a version of the model that is known in the economic literature as the hedonic pricing model, where the good being priced is the firm and the characteristics of the good are its assets, both tangible and intangible. As in the case of the hedonic model, the coefficients of the right hand side variables are not ‘structural’ or ‘deep’ coefficients, but express the current equilibrium price of the particular asset. Because of this fact, they are not expected to be constant across time or sector. Taking the natural logs of both the sides in Equation 4.2, assuming constant returns to scale (s 5 1), and subtracting log Ait from both sides, the following expression is obtained:5 log(Vit/Ait) 5 logb 1 log(1 1 gKit/Ait)

(4.3)

The ratio V/A is a proxy for average Tobin’s Q, the ratio of the market value of tangible assets to their physical value. The estimation of Equation 4.3 allows one to assess the average impact of a euro, dollar, or other currency unit invested in knowledge on the market value of a firm at a particular point in time. Hall et al. (2005) estimate Equation 4.3 using non-linear least squares (NLLS). Other authors applying the same model have used the approximation log(1 1 x) ≈ x, obtaining the equation below, which can be estimated by ordinary least squares (Griliches, 1981; Jaffe, 1986; Cockburn and Griliches, 1988; Hall, 1993a, 1993b):6 log(Vit/Ait) 5 logb 1 gKit/Ait

(4.4)

The next problem in empirical implementation is the measurement of the R&D capital (K). A measure of R&D capital has often been computed as the capitalization of present and past R&D expenditures using a perpetual inventory formula like that used for tangible capital (Griliches and Mairesse, 1984; Hall, 1990): Kit 5 (1 − d) Ki,t−1 1 Rit

(4.5)

where Kit is the R&D capital at time t, Rit is annual R&D expenditures at time t and d is the depreciation rate of the R&D capital from year t − 1 to year t. The use of Equation 4.5 to capitalize R&D investments is needed because, as will be explained in Chapter 5, the Generally

R&D and financial investors

151

Accepted Accounting Principles (GAAP) in the USA and the IAS accounting standards in Europe require R&D costs to be expensed as incurred (with a few exceptions) because of the lack of a clear link between these expenses and subsequent earnings. Therefore, the balance sheet of the firm does not contain a measure of the R&D capital created by its own investments. The use of a depreciation rate is justified by the fact that knowledge tends to decay or become obsolete over time, losing economic value due to advances in technology and the investments of the firm’s competitors. Most of the studies that have estimated the model in Equation 4.4 have used a constant annual 15 per cent depreciation rate (Jaffe, 1986; Cockburn and Griliches, 1988; Hall, 1993a, 1993b; Blundell et al., 1999; Hall and Oriani, 2006). Other studies have used an estimation procedure that allows one to determine industry- and time-specific economic depreciation rates (for example, Lev and Sougiannis, 1996).7 There also exist analyses using annual R&D expenditures as an alternative measure of R&D capital (Cockburn and Griliches, 1988; Hall 1993a, 1993b; Munari and Oriani, 2005). Because R&D spending is usually fairly persistent over time at the firm level (Hall et al., 1986), results from specifications using the flow of R&D tend to be quite similar to those using the stock after they are adjusted by the appropriate capitalization rate (the inverse of the growth plus depreciation rates). For the same reason (persistence in R&D), it has proved difficult to estimate detailed depreciation schedules for R&D. Using the Tobin’s Q equation for a large sample of US manufacturing firms, Hall (2009) shows that R&D depreciation rates in the ICT sector are likely to be much higher than those in the chemicals sector (25–30 per cent as opposed to 15 per cent), reflecting the fast pace of technological change in that sector in the recent past. 4.3.2

Empirical Models Based on Event Studies

The models that analyse the relationship between R&D announcements and the response of share prices are mainly based on the event study methodology, which has been widely applied to investigate the effect of other strategic decisions as well as R&D investment decisions on firm market value (among others, McConnell and Muscarella, 1985; Woolridge and Snow, 1990; Bajo et al., 1998; Das et al., 1998). Event study methodology relates unexpected announcements of changes to existing investment programs to the excess market returns in the trading days immediately before and after the announcement. The implicit assumption is that the investors consider these announcements to be unexpected ‘news’ about investment strategies (McConnell and Muscarella, 1985). They therefore generate

152

Evaluation and performance measurement of R&D

a revision in the investors’ expectations about expected returns and, in efficient financial markets, a change in the stock price (McConnell and Muscarella, 1985; Chan et al., 1990; Woolridge and Snow, 1990). Conceptually, excess returns are that part of stock returns not explained by the returns of the market portfolio (in practice, a broad-based stock index). According to the capital asset pricing model (CAPM), they are generally calculated estimating the following equation over a time period preceding the event by a month or more to avoid contamination from the event itself.8 For example, using a window that runs from three months to one month before an event (that is assumed to take place at t 5 0), we have the following: rit 5 ai 1 btrmt 1 eit, t 5 (−90,−30)

(4.6)

where rit is the daily return of stock i in day t, rmt is the daily return of the market portfolio (for example, S&P500) in day t and eit is an independent error term. The excess return of stock i is the predicted residual of Equation 4.6, that is, the difference between the predicted and the realized stock return, computed over a period that brackets the event (for example, the period (t 5 −1 to t 5 1 3)). After estimating the excess returns, two different measures are normally selected for analysis: cumulative abnormal returns (CAR), equal to the sum of the daily excess returns over the observation period, or average excess returns, equal to the daily average excess return in the observation period.

4.4

THE EMPIRICAL RESULTS

Application of the models reviewed above to data has led to several interesting results. The hypothesis that R&D investments are positively valued by the stock market has been generally confirmed, although the magnitude of its impact is highly variable. In particular, the existing literature raises some questions of particular interest that can be summarized as follows: 1. 2. 3.

How does the market value R&D investments? How does uncertainty affect the market valuation of R&D investments? Does the type of corporate governance have any impact on the market valuation of R&D investments?

These questions will be addressed in this section by referring to the results provided in the existing literature on these topics.

R&D and financial investors

4.4.1

153

How Does the Market Value R&D Investments?

The question of how the stock market evaluates firms’ R&D investments has attracted the interest of many scholars and is still being studied, as is demonstrated by publications of very recent date. Following the seminal contribution of Griliches (1981), a large number of studies have used variations of a model similar to that in Equation 4.2 to analyse the relationship between R&D (measured by either R&D capital or R&D expenditures) and market value. The main results are summarized in Table 4.1, which reports the value of the estimated coefficients for either R&D capital (R&D cap) or annual R&D expenditures (R&D exp) and information about the sample and data sources. Clearly, the coefficients of R&D expenditures are on average greater than those of R&D capital, because annual expenditures are lower than capitalized R&D. Two main results seem in general confirmed, as also pointed out by previous surveys (for example, Hall, 2000). First, stock markets generally value R&D positively (that is, g . 0). Second, market valuation of R&D has progressively decreased over time from the 1970s to the present time, as appears, for example, in a comparison of the results of Hall and Oriani (2006) with those of earlier studies such as Jaffe (1986) and Cockburn and Griliches (1988). The meta-analysis conducted by Oriani and Sobrero (2003) on a subsample of these studies provides support for this finding. One of the main explanations for this result is a speeded up depreciation of R&D expenditures due to the shortening of technology cycles, which makes past R&D expenditures less valuable to investors. If R&D capital is constructed using the usual 15 per cent depreciation rate when the true depreciation rate is higher, then the resulting coefficient will be lower than it would be if the R&D capital were correctly measured. A second possible reason is the increased number of firms in many sectors that are pursuing R&D strategies, which will tend to drive the returns to this activity down. The analysis of Table 4.1 also poses other interesting questions. A first observation is that the coefficients, although positive in general, vary a lot across the different studies. Apart from the time dimension already discussed, there could be several important factors at the country-, industryand firm-level that affect the market valuation of R&D. Those related to uncertainty and corporate governance will be discussed later in this section. Other findings are very clear and rather straightforward. The strength of the appropriability regime enhances the market value of R&D (Cockburn and Griliches, 1988). Moreover, market share positively impacts on the valuation of R&D (Blundell et al., 1999, for UK data, confirmed by Hall and Vopel (1997) for US data), suggesting that size and market power may matter when appropriating the results from R&D investments.

154

Table 4.1

Evaluation and performance measurement of R&D

Overview of the main empirical findings of the models based on Tobin’s Q

Study

R&D coefficient

Sample characteristics (country, no. of firms, years, data source)

Griliches (1981)

Predicted R&D exp: 1.23 Surprise R&D exp: 1.58

USA, 157 firms, 1968–74, Compustat

Ben-Zion (1984)

R&D exp: 3.376

USA, 93 firms, 1969–77, Compustat

Jaffe (1986)

R&D cap: 2.95

USA, 432 firms, 1973 and 1979, Compustat

Cockburn and Griliches (1988)

R&D exp: 11.96 R&D exp * Appropr.: 2.788 R&D cap: 1.442 R&D cap * Appropr.: 0.303

USA, 722 firms, 1980, Compustat

Hall (1993a)

R&D exp: 3.10 R&D cap: 0.48 By year (1971–90): R&D exp: from 2.0 to 10.0 R&D cap. from 0.5 to 2.0 R&D cap: ~2.3

USA, 2400 firms, 1973–91, Compustat USA, 3000 firms, 1959–91, Compustat

Hall (1993b)

Haneda and Odagiri (1998) Blundell et al. (1999)

Bosworth and Rogers (2001)

R&D cap: 1.582 R&D cap * Market share: 1.745 R&D exp: 2.268

Rogers (2001)

R&D exp: 3.405

Toivanen et al. (2002)

By year: R&D exp: from 2.6 to 4.2

Japan, 90 firms, 1981–91, NEEDS database UK, 340 firms, 1972–82, LBS Share Price Database and Datastream Australia, 60 firms, 1994–96, Australian Stock Exchange and IBIS database Australia, 721 firms, 1995–98, Australian Stock Exchange and IBIS database UK, 877 firms, 1989–95, Extel financial company analysis

R&D and financial investors

Table 4.1

155

(continued)

Study

R&D coefficient

Sample characteristics (country, no. of firms, years, data source)

Munari and Oriani (2005)

Privatized R&D exp: -1.41 Private R&D exp: 3.059

Hall et al. (2005)

R&D cap: 1.736

Hall and Oriani (2006)

France – R&D cap: 0.28 Germany – R&D cap: 0.33 Italy – R&D cap: 0.01 UK – R&D cap: 0.88 USA – R&D cap: 0.33 R&D exp: 3.509

Finland, France, Germany, UK, Italy, Netherlands, 1982– 99, 38 privatized firms and 38 control firms, Datastream and Centrale dei bilanci USA, 4800 firms, 1965–95, Compustat France (51 firms), Germany (80 firms), UK (284 firms), Italy (49 firms) 1989–98; Datastream, Global Vantage, Worldscope, Centrale dei bilanci

Greenhalgh and Rogers (2006) Bloch (2008)

R&D cap: 2.28

Oriani and Sobrero (2008)

R&D cap : 1.19

UK, 347 firms, 1989–99, Extel financial company analysis and Thomson Denmark, 61 firms, 1989–2001, Danish Centre for Studies in Research and Research Policy’s R&D statistics, Account Database of Copenhagen Business School, Danish Stock Database UK, 290 firms, 1989–98, Datastream

Note that most of the earlier work summarized in Table 4.1 used US data from the Compustat Database, whereas more recently there has been a significant amount of work using data on other countries: Australia, Japan, and several European countries. Table 4.2 summarizes the studies that have used an event study methodology to assess the reaction of the stock markets to announcements concerning a firm’s R&D investments. As explained in the previous section, stock market reaction is measured by the short-term cumulative (CAR) or average excess returns within a time window of a few days around the date of the announcement. The results, although obtained from a slightly different perspective, provide several interesting insights

156

Table 4.2

Evaluation and performance measurement of R&D

Overview of the main empirical findings of the models based on event study

Study

Excess returns (%)

Sample characteristics (country, announcements, years, data source)

McConnell and Muscarella (1985)

Average excess return: (days –1, 0) 0.21

USA, 8 announcements of R&D increase from Wall Street Journal Index and Predicasts F&S Index, 1975– 81, Investment Statistics Laboratory (ISL) Database

Chan et al. (1990)

CAR (days 230, 21) 20.21 CAR (days 0, +1) 1.38 CAR (days +2, +12) 0.39 CAR (days 0, +1): high-tech 2.1 low-tech 20.9

USA, 167 announcements of R&D increase from the Dow Jones News Wire, 1979–85, CRSP

Woolridge and Snow (1990)

CAR (day 2 1) 0.80 CAR (day 0) 1.13 CAR (day 5) 0.81

USA, 52 announcements of new R&D projects from Wall Street Journal, 1972– 87, CRSP

Doukas and Switzer (1992)

CAR (days 22, 0) 0.56 CAR (days –2, 0) for firms in high concentration markets 1.44 Low concentration markets -0.01

USA, 87 announcements of unexpected variations of R&D expenditures from Wall Street Journal Index and Predicasts F&S Index, 1965–84, CRSP

Zantout and Tsetsekos (1994)

CAR (days 0,+1): announcing firms 0.742 competing firms 20.563

USA, 114 announcements of R&D increase from Dow Jones News Wire, 1979–90, COMPUSTAT and CRSP

Kelm et al. (1995)

Average excess return (days 21,0): innovation 0.88 commercialization 1.02

USA, 501 announcemens on progresses in R&D projects (innovation) or new product introduction (commercialization) from Wall Street Journal, 1977–89, COMPUSTAT and CRSP

R&D and financial investors

Table 4.2

157

(continued)

Study

Excess returns (%)

Sample characteristics (country, announcements, years, data source)

Szewczyck et al. (1996)

CAR (days 0,+1): – Firms with Tobin’s Q high 0.929 low -0.160 – Firms with CF/A high 0.499 low 0.227 CAR (days 0, +1) Loose competition +0.8 Aggressive competition 20.6

USA, 252 announcements of R&D increase from the Dow Jones News Wire, 1979–92, COMPUSTAT, CRSP and Standard & Poor’s Stock Guide

Sundaram et al. (1996)

Zantout (1997)

CAR (days –10,21) 0.059 CAR (days 0,+1) 0.474 CAR (days +2,+10) 20.369

USA, 125 announcements of unexpected variations of R&D expenditures from the Dow Jones News Wire, 1985–91, CRSP and Business Week R&D Scoreboard USA, announcements of R&D increase from the Dow Jones News Wire, 1979–92, CRSP and COMPUSTAT

that complement and integrate those provided by the studies on Tobin’s Q. First, Table 4.2 shows that the stock returns following an announcement of an unexpected increase in R&D investments or a new R&D project are generally positive. This result is consistent with the positive coefficients for the R&D capital found by the studies reviewed in Table 4.1 and reinforces the idea that the stock market places a positive value on the money spent in R&D. Second, Table 4.2 shows variability of the excess returns depending on the study and its characteristics. With respect to the factors affecting the excess returns, it is worth noting that the stock market reacts more positively in high-tech than low tech industries (Chan et al., 1990). Moreover, the level of competition seems to have an effect. The excess returns are higher in more concentrated industries (Doukas and Switzer, 1992) and in industries where competitors do not aggressively react to a firm’s R&D announcements (Sundaram et al., 1996). These two results suggest that investors expect that firms are more able to appropriate the returns to their R&D investments when competition is softer.

158

4.4.2

Evaluation and performance measurement of R&D

How Does Uncertainty Impact on the Market Valuation of R&D Investments?

An important problem related to the market valuation of R&D investments is the high level of uncertainty that characterizes their expected returns (Mansfield et al., 1977). Several authors have claimed that under these conditions, traditional valuation methods (that is, net present value, NPV) can fail to capture the full value of R&D investments (for example, Kogut and Kulatilaka, 1994). The question, therefore, is how financial investors evaluate R&D investments of traded firms. An increasing number of studies have suggested that real options (RO) theory can complement existing theories in understanding the value created by R&D investments. Some authors have developed formal models for R&D valuation at the project level (for example, Schwartz and Moon, 2000 and other contributions reviewed in Chapter 2 of this book), whereas others have simply applied an RO logic to analyse technological choices at the firm level (for example, McGrath and Nerkar, 2004). According to the latter approach, decision makers, such as managers or external financial investors, implicitly or explicitly use an ‘option lens’ (Bowman and Hurry, 1993) to analyse the value of different forms of flexibility inherent to R&D investments. Firms’ R&D investments create a portfolio of options, whose underlying asset is the present value of the cash flows that can be acquired through discretionary subsequent investments (McGrath and Nerkar, 2004). Since there is no obligation to exercise these options, their value (and the value of the whole portfolio) increases with the variance of the returns on the underlying assets. Accordingly, the volatility of the expected returns from R&D investments is relevant for market valuation. In a recent contribution, Oriani and Sobrero (2008) have recognized that this volatility can be ascribed to different sources of uncertainty (Huchzermeier and Loch, 2001). In particular, they focus on the distinction between market and technological uncertainty. Faced with market uncertainty, managers have two choices (Folta and O’Brien, 2004). They may delay the investment of additional resources in R&D, thus holding an option to wait or acquire a growth option by committing to incremental preemptive R&D investments. Similarly, faced with technological uncertainty, managers may decide not to invest additional resources in R&D, waiting for the evolution of the technology. Alternatively, as a form of hedging, they can devote incremental R&D investments to the creation of an option to switch to alternative technologies. Financial investors in the marketplace evaluate the firm conditional on its R&D decisions. Based on this reasoning, market and technological uncertainty, real options,

R&D and financial investors FINANCIAL MARKETS

Stock market investors

159

FIRM

Invest in R&D Create growth options

Corporate debt market investors Invest in R&D Create switch options Firm value in the financial marketplace Not invest in R&D Hold options to wait

Market uncertainty Technological uncertainty

Observed market uncertainty • Observed technological uncertainty

Managerial R&D decision making

Source:

Adapted from Oriani and Sobrero (2008).

Figure 4.3

Uncertainty, R&D investments, real options and firm market value

and firm value can be linked within a comprehensive framework, which is presented in Figure 4.3. When all the real options embedded in R&D investments are considered jointly, the two forms of uncertainty have a non-linear impact on the market valuation of R&D investments. In particular, the empirical results of Oriani and Sobrero (2008) on a sample of British traded firms show that the effect of market uncertainty on the market valuation of R&D is U-shaped, whereas the effect of technological uncertainty is inversely U-shaped.9 Their empirical results are summarized in Figure 4.4 and in Figure 4.5. Figure 4.4 shows the market valuation coefficient of R&D at different levels of the measure of market uncertainty. It can be seen that the market valuation of R&D investments decreases up to a certain level of market uncertainty and increases after that. The results show that NPV might not be the only component of R&D value considered by financial investors. NPV alone, in fact, does not explain the fact that the market valuation of R&D investments starts to increase after a certain level of market

160

Evaluation and performance measurement of R&D

1.8

R&D market valuation coefficient

1.6 1.4 1.2 1.0 0.8 0.6 0.4 0.2

0.13

0.12

0.11

0.10

0.09

0.08

0.07

0.06

0.05

0.04

0.03

0.02

0.01

0.0

–0.01

–0.02

–0.03

–0.04

–0.05

0.0

Market uncertainty indicator

Source: Adapted from Oriani and Sobrero (2008).

Figure 4.4

Market uncertainty and the R&D market valuation coefficient

uncertainty (NPV would predict a monotonically decreasing relationship between market uncertainty and the valuation of R&D investments). Figure 4.5 shows that the market valuation of R&D investments first increases and then decreases with technological uncertainty. In this case too, the real options reasoning and the results indicate the existence of a non-linear relationship between market value of R&D investments and technological uncertainty that is not consistent with NPV. 4.4.3

Does Corporate Governance Have an Effect on the Market Valuation of R&D Investments?

A third issue that the recent literature has studied is the possible effect of corporate governance on the market valuation of R&D investments. This effect can be observed at two different levels. At the country level, features of the financial system and resulting investors’ behaviour can be relevant

R&D and financial investors

161

1.4

R&D market valuation coefficient

1.2

1.0

0.8

0.6

0.4

0.2

0.0 –0.04 –0.03 –0.02 –0.01

0.0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

Technological uncertainty indicator Source:

Adapted from Oriani and Sobrero (2008).

Figure 4.5

Technological uncertainty and the R&D market valuation coefficient

for R&D (for example, Tylecote and Ramirez, 2006). At the firm level, we might observe an effect of ownership structures on the value that the stock market places on R&D investments. Clearly, the effects of the financial system and the ownership structure could interact, as will be discussed later in this subsection. With respect to country level differences, Hall and Oriani (2006) analyse the market valuation of R&D investments in several continental European countries, and compare it with two Anglo-Saxon countries (United Kingdom and United States). This paper is the first in-depth empirical analysis of the valuation of firms’ R&D by the stock market in European countries other than the United Kingdom. Extending the analysis to these countries is important for several reasons: the importance of their economies, the different nature of their corporate governance systems as compared to Anglo-Saxon countries, and the variations in the public incentive schemes for private R&D. With respect to capital markets specifically, it is generally recognized that publicly traded firms in continental European

162

Evaluation and performance measurement of R&D

countries are subject to less shareholder pressure on their investment decisions (see, for example, Franks and Mayer, 1990). This could be for the good, in the case of profitable long-term investments that might not be undertaken by firms with short horizons, or the bad, if it implies that rate of return tests might not be imposed on these investments, or that projects might be continued too long when they have been demonstrated to be unsuccessful. Under the admittedly strong assumption of efficient capital markets, these differences should also imply market valuations of capital and R&D investments that may be either higher or lower on average than those in the USA. The results obtained by Hall and Oriani exhibit several interesting features. German and French samples show a statistically significant and robust positive valuation of R&D capital by the stock market, although the estimated coefficients of R&D capital are considerably less than unity in all countries, suggesting that R&D investments are less valued by the stock markets than investments in tangible assets. The coefficients are also significantly smaller than the coefficients reported by previous studies using data on US and UK firms. Nevertheless, when permanent unobserved differences across firms were controlled for, the results for the Anglo-Saxon countries were consistent with those for the continental European countries, which confirms that the market valuation of R&D expenditures has decreased in all the countries over time, in line with the previous discussion. In addition, the very narrow gap observed between the R&D coefficients across countries is consistent with the anecdotal evidence of a progressive alignment of the European financial markets to the Anglo-Saxon ones within the last two decades (see Rajan and Zingales, 2003). An interesting finding is that the UK sample shows a substantially greater valuation of R&D capital in the cross section. From the perspective of financial investors, this means that a currency unit spent on R&D by a company in the United Kingdom has on average an impact whose magnitude is nearly three times larger than in France and Germany. The fact that Bond et al. (1998) find much higher marginal productivity of R&D in the UK than in Germany confirms that this result is probably real. It suggests that UK firms face a somewhat higher cost of R&D capital than US firms or firms in continental Europe. With respect to the ownership structures of corporations, the study of Munari and Oriani (2005) on the effect of privatization on R&D performance has shown that R&D investments of privatized firms are evaluated by the stock market significantly and consistently less than those of a control sample of private firms. This result suggests the presence of some form of inefficiency due to the former state ownership. A second interesting finding on ownership structures is provided by Hall and Oriani (2006),

R&D and financial investors

163

who show that in France and Italy, the market places a significantly positive value on R&D spending only for firms without large controlling shareholders. In some cases, especially in France, this may be because the large shareholder is the government. In other cases, it may simply be that majority holders do not respond to market pressures that signal low values for their investment strategies. One avenue for future research could be further exploration of the relationship between the types of large shareholders (governments, families, or other firms) and the valuation of firm-level R&D strategy. Another issue worthy of exploration is the interaction between corporate governance systems at the country level and ownership structures at the firm level. In fact, agency problems specifically related to R&D investments are likely to arise between the controlling and the minority shareholders because the former are relatively protected from takeover threats and monitoring activities and have the opportunity to divert firms’ profits from outside investors to their own benefit. This problem can be exacerbated by a weaker legal protection of minority shareholders. In a series of studies adopting a legal approach to financial markets, La Porta et al. (among others, La Porta et al., 1998, 2002) have reported that civil law systems, such as those of France, Germany and Italy, grant fewer rights to minority shareholders than common law systems of Anglo-Saxon countries.

4.5

CONCLUSIONS

This chapter has analysed how investors in financial markets value firms’ R&D investments. This issue is important because stock prices can provide managers with useful information on the value of the firm’s R&D activities. The first part of the chapter has reviewed the theoretical and methodological foundations of the studies on the market valuation of R&D investments and the models adopted. This was necessary in order to better understand their results and implications. Based on this review, three main open questions about the market valuation of R&D investments have been identified. First, how does the stock market evaluate R&D investments? In this respect, it has been observed that in general the investors place a positive value on the money that firms invest in R&D activities. However, the valuation is rather erratic over time and across countries and industries. This raises questions about whether all the factors affecting the market valuation of R&D investments have been controlled for. Second, how does uncertainty impact R&D valuation? It is known that the returns to R&D investments are subject to a high degree of both market

164

Evaluation and performance measurement of R&D

and technological uncertainty. This chapter has reported the results of the recent work of Oriani and Sobrero (2008), where a real options logic has been applied to better understand how financial investors evaluate the R&D investments of traded firms. The study shows that the valuation is consistent with real options theory and suggests that the traditional valuation methods based on NPV capture only a part of the market value of R&D investments. Third, how do corporate governance and ownership structures affect the market valuation of R&D investments? Based on recent studies dealing with these issues, it has been proposed that corporate governance provisions at the level of both the firm and the country can have a significant effect on how R&D investments are evaluated by the stock market. The issues reviewed and the results reported and discussed in this section have some potentially important implications for different actors, and highlight opportunities for future steps in the research on the market valuation of R&D investments. The fact that stock markets in general positively evaluate R&D investments of traded firms is a signal for managers, who are reassured about the possible market myopia on R&D investments. Policy makers receive confirmation about the importance of stock markets for the financing of innovation and of the measures to favour the listing of younger and more innovative firms. However, the high variability of the results also shows that several aspects of the market valuation of R&D investment have still to be clarified. In particular, it seems that firm-specific factors affecting R&D market value have not been fully investigated yet. Given the recent attention to the role of corporate governance for firm performance (for example, Gompers et al., 2003) and innovation development (for example, Tylecote and Ramirez, 2006), the relationship between corporate governance arrangements (including ownership structures), management of R&D and firm value seems a particularly promising avenue for future research.

NOTES 1. These definitions implicitly require the precondition that information has no cost. However, it is possible to give a definition of market efficiency even in the presence of costly information. In this case, security prices reflect all the available information to the point where the marginal benefits of acting on information do not exceed the marginal costs (Jensen, 1978). 2. Some intangible assets, such as purchased patents, and the goodwill acquired via the acquisition of other firms, will appear on the balance sheet and be included in the denominator of Q. 3. Other researchers have sometimes used a patent-based measure of knowledge capital K (for example, Cockburn and Griliches, 1988; Hall et al., 2005). These studies will

R&D and financial investors

4.

5. 6. 7. 8.

9.

165

not be reviewed here because this chapter is focused on the market valuation of R&D investments. The additive functional form is the most commonly used form in the literature, and can be thought of as a first-order approximation to the value function. It is possible that other functional forms, such as additive in logarithms, might also be useful for estimation. Theory is to some extent silent on the exact relationship. The assumption of constant returns to scale (homogeneity of degree one) in the value function has been confirmed repeatedly in the literature, at least for cross sections of firms. As the knowledge intensity of firms has grown over time, the approximation has become more and more inaccurate, so later authors are more likely to use the nonlinear form of the equation (that is, Equation 4.3 rather than Equation 4.4). More precisely, the authors estimate a regression model in which the dependent variable is the annual operating income and the independent variables are the lagged values of total assets and advertising expenditures and a vector of the past R&D investments. See Brealey, Myers and Allen (2005), Chapters 7–9, for details on CAPM. Some studies adopt the more general three-factor models of Fama and French where stock returns depend not only on the return of the market portfolio, but also on the returns of two other portfolios (see Fama and French, 1992). Please refer to Oriani and Sobrero (2008) for more complete results and the definition of the indicators of market and technological uncertainty.

5. 5.1

R&D information INTRODUCTION

As shown in Chapter 4, the market valuation of R&D investments is a very important issue for managers, investors and researchers. Chapter 4 has presented the main methods and results of the literature linking R&D investments and firm market value, but it has not dealt with the possible R&D information problems arising between managers and outside investors. Indeed, despite the continuous growth of R&D spending, the information sources and level of disclosure on R&D have not significantly changed for the last two decades. Several studies have advanced that the relative scarcity of R&D disclosure and the inability of accounting rules to recognize fully the economic value of R&D investment may give rise to information deficiencies (among others, Lev, 2001; Hand and Lev, 2003). Accordingly, the main contribution of this chapter is to analyse how R&D information flows from firms to investors and the implications for firm market value. The uniqueness of R&D economic attributes makes it difficult for outsiders to learn about its value. Accounting rules exacerbate the information asymmetries, as R&D investments are in general fully expensed in financial reports or not even disclosed. This chapter will deal with three interrelated aspects related to the R&D information chain. First, it explains why and how R&D investments generate information asymmetries between insiders (managers) and outsiders (investors). Second, it analyses the value relevance of R&D information for investors and stock prices. Third, it focuses on the role of financial analysts, as they normally complement public R&D information with private information supporting investors’ decisions. These questions are also examined in the light of the changing international accounting standards. In 2005, the adoption of International Accounting Standards (IAS/IFRS) became compulsory for all companies traded in the European stock markets. The European normative context is analysed to highlight the main differences in R&D reporting that have followed the mandatory application of IAS/IFRS standards. The chapter is organized as follows. The next section deals with the problem of information asymmetries generated by firms’ R&D investments. Section 5.3 analyses if and to what degree information about R&D 166

R&D information

167

investments is relevant for investors and stock prices, while Section 5.4 analyses two issues related to the reduction of information asymmetries: voluntary R&D disclosure and the role of financial analysts within the R&D information chain. Section 5.5 presents descriptive evidence on the effect of the application of IAS/IFRS standards on R&D reporting for a sample of companies publicly traded in Italy. In the last section some conclusions are drawn from the review of the literature and the evidence presented in the chapter.

5.2

R&D AND INFORMATION ASYMMETRIES

The inability of information systems to adapt to the economic surge of R&D raises the risk of widespread adverse private and social consequences for managers, investors and policy makers. There is plenty of empirical evidence of information deficiency effects, such as abnormal returns to investors with private information, increasing bid-ask security spreads, higher cost of capital, systematic undervaluation of R&D-intensive firms, and decreasing value relevance of key accounting information (Lev, 2001). The information deficiencies related to R&D are intrinsic to the nature of these investments. The economic attributes of R&D that explain its importance as a source of competitive advantage also lead to information asymmetries (Hand, 2001, 2003b). This problem is magnified by the relative scarcity of public information on R&D. Financial reports remain the main information source on R&D activities, so that valuation criteria and accounting discretion play a meaningful role in shaping the features of R&D information. Economic attributes of R&D and the related accounting problems are analysed in this section. 5.2.1

Economic Attributes of R&D Investments

Economic attributes of R&D are the primary sources of information asymmetry. Information asymmetry exists when corporate insiders possess private information about the firm’s value while outsiders have access only to public information. A thorough analysis of the economic properties of R&D allows for a better understanding of how accounting criteria make it more difficult for investors to learn about value of R&D. As noted by Grossman and Helpman (1991), when firms invest in R&D, the output that they produce is a technology that, as a form of knowledge, has some peculiar properties. Lev (2001), in particular, has classified these economic attributes as value drivers and value detractors, in order to explain how different features of R&D simultaneously justify their

168

Evaluation and performance measurement of R&D

economic relevance and limit their value. Specifically, attributes such as non-rivalry, increasing return and network effects explain why R&D could be a source of competitive advantage sustaining superior firm performance (Hand, 2001, 2003b; Lev, 2001). Yet, high risk, partial excludability and the absence of transparent markets are overall the main factors that foster information asymmetries. The major value driver of innovation activities is the non-rivalry attribute, which is the ability to use innovations in simultaneous and repetitive applications without diminishing their value (Romer, 1990; Grossman and Helpman, 1991; Lev, 2001). Consequently, while physical and financial assets are rival assets and can be leveraged to a limited degree, R&D activities are characterized by large fixed (sunk) cost and negligible marginal cost (Lev, 2001).1 A direct consequence of the non-rivalry attribute is the increasing return to scale of innovation, so that investments in R&D create cumulative knowledge (Romer, 1990; Grossman and Helpman, 1991; Teece, 2000; Hand, 2003b). Similarly, network effects are prominent in technology and science-based sectors, so that the benefits of innovation increase with the number of adopters (Lev, 2001). These value driver attributes are however mainly restricted by partial excludability of innovation. That is, it is difficult for the creator or owner of an innovation to prevent others from making unauthorized use of it (Grossman and Helpman, 1991). In particular, the difficulty of fully appropriating the benefits of innovation is a function of both the technology and the legal system. As noted by Arrow (1962, p. 615), ‘no amount of legal protection can make a thoroughly appropriable commodity of something so intangible as information.’ Consequently, some investments in innovative activities may not yield property that confers an exclusive use for the innovator (Teece, 2000). Nevertheless, even when property rights are legally well defined, they are often imperfectly enforced, generating technological spillovers. Specifically, spillovers refer to the feasibility ‘to acquire information created by others without paying for that information in a market transaction’ (Grossman and Helpman, 1991, p. 16). Partial excludability and spillovers deeply affect firms’ public disclosure of innovation activities and enhance information asymmetries between insiders and outsiders. Several studies have documented that patents are often a channel of strategic information on innovation activities and that under certain conditions firms regard secrecy as a more effective way to protect innovation (Levin et al., 1987; Cohen et al., 2000). The inherent high risk of R&D is a further attribute that distinguishes this investment from other corporate activities. Several studies have documented how the likelihood of success in innovation efforts is very low (for example, Scherer et al., 1998), and the uncertainty of R&D is

R&D information

169

substantially higher than that of tangible assets (Kothari et al., 2002; Amir et al., 2007). In particular, Kothari et al. (2002) have found that R&D investments increase future earnings volatility three times more than other capital expenditures. Given the inherent risk of R&D and the difficulty of providing reliable estimates of its future benefits, accounting criteria generally require the full expense of R&D in the profit and loss account, differently from other long-term investments that are capitalized in the balance sheet. The absence of organized markets for R&D further contributes to information asymmetries. As observed by Aboody and Lev (2000, p.2750): ‘whereas investors can derive considerable information from prices of traded tangible and financial assets concerning their value at the firm level, there is no direct price-based information on firm-specific change on the value and productivity of R&D.’ Actually, most of the results of R&D investments cannot be sold, making their valuation extremely difficult (Griliches, 1995). 5.2.2

Accounting Standards as a Source of R&D Information Asymmetries

Financial reporting represents the foremost channel of information between the firm and outsiders, whereas other forms of voluntary disclosure of firms’ innovation activities are more rare. Consequently, financial reports should reduce the information gap between managers and investors. Nevertheless, in the case of R&D, accounting standards do not seem to improve the quality of the information available to investors. In most countries, R&D activities are not reported in the balance sheet, but fully expensed, in the profit and loss account. There are two main reasons why R&D investments are expensed rather than being capitalized in the balance sheet. First, their inherent risk and the consequent uncertainty over their future outcomes. For example, US Generally Accepted Accounting Principles (GAAP) have mandated since 1974 the full expensing of R&D (SFAS No. 2), with the exclusion of software development costs,2 due to the presumed absence of a relationship between R&D expenditures and subsequent economic benefits.3 Second, in general, asset recognition criteria require that in the absence of legal rights to protect them, most intangible investments, including R&D, must be excluded from a firm’s assets. Therefore, the uncertainty over the control of future economic benefits is another main argument against the recognition of these resources on the balance sheet (Powell, 2003). Although R&D investments are characterized by higher uncertainty, they significantly contribute to the firm’s growth, productivity and value

170

Evaluation and performance measurement of R&D

(see also Chapter 4). In particular, Sougiannis (1994), analysing the long-run impact of R&D on productivity, has found that, on average, a one-dollar increase in R&D leads to a two-dollar increase in profit over a seven-year period. Consequently, we have a mismatch between the economic relevance of R&D and reporting conservatism. This inconsistency undermines the fundamental accounting measurement process matching costs with revenues, and in the end it adversely affects the informativeness of accounting information (Lev and Zarowin, 1999; Lev et al., 2007). As noted by Oswald and Zarowin (2007), capitalization enables managers to communicate information about the success of the projects and their probable future benefits; consequently, expensing R&D investments considerably impairs the transparency concerning these investments. 5.3

Value Relevance of R&D Information for Stock Market Investors

The last section has shown that R&D investments generate information asymmetries for reasons related to both their economic nature and accounting standards. This is an important issue since the value of accounting information, for example on R&D, lies mainly in meeting outsiders’ demand for information (Dye, 1988; Ronen and Yaari, 2008). In particular, the concept of value relevance refers to a broad financial and accounting literature that empirically investigates the relationship between market value and accounting information (Beaver, 2002). Value relevance studies use stock market values as benchmarks in order to understand whether information is reflected into firm value in an effective and timely manner (Barth, 2000).4 If accounting information provides inputs used by market traders to decide on their demand for and supply of securities, and if markets are semi-strongly efficient under the maintained hypothesis, then the prices impound by definition the information content of the accounting numbers (Ronen, 2001). Accordingly, an accounting item such as R&D is value relevant, that is, it has a predicted significant relation with share prices, only if this value conveys information relevant to investors in evaluating the firm and is measured reliably enough to be reflected in share prices (Barth, 2000; Barth et al., 2001a; Holthausen and Watts, 2001). The information deficiency of R&D investments has motivated the research questions of a large number of scholars, who, since the mid 1990s, have analysed whether firms’ R&D investments represent useful information for investors. Following Lev and Sougiannis (1996), the studies that have analysed the association between R&D accounting information and stock market measures can be distinguished with respect to the period of analysis and the methodology. Value relevance or contemporaneous

R&D information

171

studies analyse the association between accounting information and current stock prices or returns and indicate the extent of current recognition of R&D relevance by investors (Lev and Sougiannis, 1996). Intertemporal studies evaluate the relationship between R&D information and future stock returns in order to investigate whether the expected benefits from R&D benefits are recognized at a future moment. The empirical models adopted in contemporaneous and intertemporal studies are described in Appendix A5.1. As concerns the value relevance studies, the empirical findings document if and to what degree R&D investments represent useful information for investors (see Holthausen and Watts, 2001, and Wyatt, 2008, for a review). In general, scholars recognize that accounting information on R&D, although scant, is on average impounded in current stock prices. As concerns the US GAAP accounting regime, under which R&D is fully expensed, a large body of studies prove the value relevance of R&D investments. Sougiannis (1994) among others, indicates that from 1975 to 1985 investors have placed a high value on R&D investments so that, on average, a one-dollar increase in R&D expenditure has produced a five-dollar increase in stock price. However, these findings also suggest that current R&D investments are particularly informative only in a short time period, whereas past R&D is unrelated to stock price. Similar conclusions have been reached by Hand (2003a). Analysing US internet firms from 1997 to 2000, he has shown that the higher the level of R&D investments, the more the investors consider R&D expenditures relevant. In particular, since US GAAP mandate the full expensing of intangible investments, high R&D intensive firms usually report negative income. Consequently, R&D expenditures in loss firms are priced as assets, whereas this does not happen when income is positive. The benefit of R&D capitalization has also been widely investigated. In order to examine the concerns about the reliability and relevance of R&D information under US GAAP, several scholars have documented that reporting R&D as a firm’s asset leads to value relevant information for investors.5 Lev and Sougiannis (1996), computing the R&D capital that would have been reported if capitalization had been allowed, have shown that such estimates are statistically reliable and economically meaningful.6 Healy, Myers and Howe (2002) have confirmed that capitalizing R&D investment is more value relevant than expensing it. In particular, using a simulation model for pharmaceutical companies, they have compared the value relevance of R&D under expensing and capitalizing regimes. Their findings document how stock return is highly and more associated with R&D investments if a firm reports successful R&D investments among assets and writes down R&D investments found to be unsuccessful. Aboody and Lev (1998), examining software R&D, have found that the

172

Evaluation and performance measurement of R&D

cumulative software assets reported on the balance sheet are associated with stock price. In summary, as Oswald and Zarowin (2007) have pointed out, R&D capitalization enables management to communicate information about the success of R&D projects and their probable future benefits. This information is not generally recognized under expensing regimes; thus, capitalization leads to more and better information for investors. However, the information shortfall led by accounting regimes is not always pervasive. As Deng and Lev (2006) note, capitalizing instead of full expensing R&D does not significantly affect financial information, such as earnings and assets value, when the growth rate of R&D investments is low. Actually, if R&D investments are constant over time, annual R&D expenditures match the hypothetical amortization rate of the R&D capital and therefore earnings are equivalent under expensing and capitalization. On the contrary, when the R&D growth rate is high, expensing R&D leads to systematic undervaluation of earnings and corporate assets. This effect is accounted as the main determinant of decreasing value relevance over time of key accounting variables, such as earnings and book values (among others, Ely and Waymire, 1999; Lev and Zarowin, 1999). Recently, Lev, Nissim and Thomas (2007) have demonstrated that if R&D expenditures were capitalized and amortized over industry-specific useful lives, there would be a potential improvement in the informational usefulness of earnings and book value. The studies on the intertemporal relationship between R&D and stock returns analyse the association between firm-level R&D investments and subsequent stock returns. The empirical models start in general from the return model of Fama and French (1992) and they add R&D as an explanatory variable (for example Lev and Sougiannis, 1996; see Appendix A5.1 for more details). A significant coefficient for the R&D investments signals in this case that more R&D-intensive stock will earn higher returns in the future. Other authors have adopted a different methodology to analyse the same problem. They have built portfolios of R&D-intensive stocks and compared their returns with those of control portfolios, including stocks with similar market-to-book ratio and size (for example, Chan et al., 2001). Eventual higher returns of the former portfolios would suggest again that R&D-intensive stocks earn higher returns than other stocks, after controlling for other factors such as the market-to-book ratio or market capitalization. Although they adopt different methodologies, all these studies are based on a similar logic. If the stock market evaluates the expected benefits from R&D investments correctly and immediately, R&D-intensive firms should

R&D information

173

not exhibit systematic abnormal returns in the future. On the contrary, if R&D investments are under valued by the stock market (that is, the investors do not fully include expected returns from R&D investments in stock prices), R&D-intensive stocks will have abnormal positive returns in the future because the investors will recognize the benefits from current R&D investments only at a later moment (that is, when they become manifest). The different studies provide convergent results. Lev and Sougiannis (1996, 1999) show a significant positive relationship between R&D investments and subsequent monthly stock returns. This implies that there is a systematic undervaluation of R&D investments by the stock market, which is corrected only at a later moment through higher stock returns. In particular, Lev and Sougiannis (1999) and Chan et al. (2001) have shown that the excess return over control portfolios is significantly higher for portfolios with high R&D-intensive stocks than for portfolios with low R&D-intensive stocks. Again, this evidence suggests that more R&Dintensive stocks are systematically under valued and that the prices only slowly return to equilibrium levels. Not all the authors, however, fully agree on the undervaluation and its causes. Some studies have concluded that mispricing is mainly due to accounting biases, as investors do not understand conservative accounting (for example, Lev et al., 2005). Consistently with this explanation, Eberhart et al. (2004) have shown how investors underreact to new information contained in R&D expenditure increases. Conversely, other scholars have advanced that mispricing is primarily caused by risk premium and related to the high uncertainty of R&D investments (for example, Chambers et al., 2002).

5.4

REDUCING INFORMATION ASYMMETRIES: VOLUNTARY R&D DISCLOSURE AND FINANCIAL ANALYSTS

Economic attributes and accounting rules are important determinants of R&D information asymmetries. This section will examine two mechanisms to reduce information asymmetries. First, the discretional disclosure of additional R&D information by managers. Second, the role played by financial analysts in providing new information to the investors. 5.4.1

Voluntary Disclosure of R&D

If additional information about firms’ R&D investments were disclosed, concerns about capitalizing or expensing R&D would be less important.

174

Evaluation and performance measurement of R&D

Requiring descriptive information on R&D behind accounting numbers would potentially convey relevant information to investors about R&D value creation and expected benefits. Even with a full R&D expensing regime, if firms provided detailed information on R&D activities, information asymmetries could be greatly reduced. However, accounting standards such as US GAAP or IAS/IFRS do not require the disclosure of detailed descriptive information. Therefore, firms voluntarily choose whether or not to provide additional information on R&D in financial reports. Notwithstanding its potential usefulness, most firms do not provide additional information on R&D. This in turn affects the transparency of R&D activities, impairing investors’ ability to obtain useful information from financial reports. In the specific case of high-technology firms involved in different and multiple research projects, R&D accounting data are too general to allow outsiders to understand their benefits (Lev, 2001; Wyatt, 2008).7 The reasons behind firms’ decision to withhold R&D information still remain an open research question that empirical studies have only partially addressed. Theoretical studies offer compelling explanations of non-disclosure rationales. Even if most of these studies aim at explaining earnings management, their theoretical assumptions could be generalized to any reporting information that is relevant for investors. In particular, several studies have applied game theory to model the relation between investors in the capital markets and the firms’ disclosure, in order to understand the incentives for the management’s strategic use of accounting information (see Ronen and Yaari, 2008 for a review).8 Game theory is able to explain the conditions under which stock market reactions to accounting information create an incentive to a full and truthful disclosure or to a selective disclosure (Grossman and Hart, 1980; Grossman, 1981; Milgrom, 1981; Verrecchia, 1983; Dye, 1985, 1986). The R&D information flow between management and investors can be modelled as a reporting game in which management (the sender), possessing private information on R&D, can use its discretion either opportunistically or non-opportunistically, and respectively reveal or not reveal information (Ronen and Yaari, 2008). The investors (the receivers) respond to a message from the sender by making decisions that determine the payoff of both actors. The management therefore will choose a truthful message disclosing R&D information only if this strategy maximizes its own payoff given its expectation regarding the investors’ response. The reporting game between management and investors is mainly guided by the disclosure principle that management has a payoff in revealing private information. With specific respect to R&D investments, the private

R&D information

175

information to be disclosed is possessed by management and not available to outsiders. The disclosure principle applies to situations wherein disclosure is certifiable ex-post, so that the alternative to full revelation is omission (Grossman, 1981; Milgrom, 1981; Verrecchia, 1983; Dye, 1985). Therefore, in a sender–receiver game in which the sender is the management and the receiver is the investor in the stock market, the disclosure principle implies that the stock market interprets non-disclosure as a signal of bad news (Verrecchia, 1983; Dye, 1985). As stated by Dye (1985), since valuemaximizing shareholders prefer managers who adopt a policy designed to increase the value of their shares, they will design (in principle) contracts that encourage managers to suppress information that negatively affects the firm’s value and release information positively valued by investors. However, if information is verifiable ex-post by investors so that they know that management has information that has not been released, then investors will revise downward their demand for the firm’s shares and the price will decrease until managers release the information. The negative stock market reaction represents the management’s incentive to full disclosure of R&D and implies that management is encouraged to disclose the information to distinguish it from the worst information it could possibly have (Grossman, 1981; Dye, 1985).9 As Verrecchia (1983) clarifies, management’s decision to withhold information depends upon how investors interpret its absence, and investors’ conjecture about the content of the withheld information depends upon the management’s motivation for withholding it. The disclosure principle regulates the reporting game when the private information that has to be disclosed is non-proprietary, which is information whose disclosure reduces the present value of cash flows of the firm endowed with the information (Dye, 1986).10 Consequently, only when disclosure entails costs (proprietary information), is non-disclosure rational, as the benefit of non-disclosure is higher than the cost of disclosure (Verrecchia, 1983). The disclosure principle also does not apply when management possesses both non-proprietary and proprietary information and if there is a known statistical interdependence between the two types of private information. In such a case, disclosing non-proprietary information may partially reveal proprietary information. Then, even non-proprietary information will be not disclosed unless the effect of the disclosure on the firm’s value is dramatic (Dye, 1986).11 In summary, according to the theoretical models reviewed, voluntary disclosure does not occur if the information content is not verifiable ex-post by firm outsiders and when the disclosure of private information entails costs. Nevertheless, these conditions frequently characterize R&D information. Actually, most firms’ information on R&D can hardly be verified by investors, since economic benefits from R&D generally span long time periods.

176

Evaluation and performance measurement of R&D

Similarly, economic attributes of R&D, such as partial excludability and spillover effects, frequently characterize R&D information as proprietary, so that the release of details on R&D investments may be useful to competitors in a harmful way. In this case, the release of good news by one firm makes its rivals more optimistic about their own opportunities in the R&D race, which gives them a greater incentive to invest (Jansen, 2004). 5.4.2

The Role of Financial Analysts

In the information chain between firms and investors, financial analysts, acting as information providers, play a determinant role in reducing information asymmetries (for example, Brennan and Hughes, 1991). By interpreting firms’ disclosure and providing additional information through reports and forecasts, sell-side analysts enrich the information available to investors (for example Lang and Lundholm, 1996). In this sense, information embodied in analysts’ research might encompass both new analystspecific information and interpretation of corporate disclosure (Frankel et al., 2006). The importance of the analysts’ role in capital markets has motivated a large and growing number of studies that analyse the informativeness of analysts’ research (among others, Givoly and Lakonishok, 1979; Lys and Sohn, 1990; Francis and Soffer, 1997; Frankel et al., 2006; Kelly and Ljungqvist, 2007). Overall, scholars concur in documenting how analysts’ activity provides valuable information and, in particular, affects investment decisions of unsophisticated investors (see Ramnath et al., 2008, for a comprehensive review). Although the demand for analysts’ research is always motivated by information asymmetry, information deficiency on R&D and the complexity of R&D information make the value of analysts’ research even more important (for example, Barth et al., 2001b; Gu and Wang, 2005). The work of Amir et al. (1999) is the first study that relates R&D information and analysts’ efforts, investigating whether analysts’ research complements the information of financial statements. In particular, the authors have found that analysts’ contribution is larger in sectors where the informativeness of financial reports is low, such as in high-tech industries, followed by low-tech industries, and regulated firms. Other scholars have pointed out that the uncertainty on R&D outcomes and scant information about the contribution of R&D to earnings imply strong market incentives for analysts to provide value-added information (for example, Barth et al., 2001b; Gu and Wang, 2005). Financial analysts have greater incentives to cover R&D-intensive firms, since their value is measured less effectively by financial reports. Accordingly, Barth et al.

R&D information

177

(2006) have found that analyst coverage is significantly wider for firms with larger R&D investments relative to their industry, and for firms in industries with higher R&D intensity. Amir et al. (2003) have estimated the incremental information contribution of earnings forecasts over the information contained in financial reports, and compared analysts’ contribution for firms with and without R&D. In particular, they have proved that analysts’ incremental contribution to investors’ decisions is larger in R&D-intensive companies than in companies with low levels of (or no) R&D, indicating that the intangiblesrelated financial report deficiencies are compensated to some extent by analysts’ activities. However, this compensation is modest and far from complete, as indicated by the documented association between R&D intensity and the bias and accuracy of analysts’ forecasts. The information deficiency of R&D implies not only a need for a greater effort in analysts’ research, but also a higher content of private information relative to public information in the earnings forecasts. According to Barron et al. (2002), since in R&D-intensive firms financial analysts place greater reliance on their own idiosyncratic knowledge and skills, the disagreement among analysts’ forecasts is greater. Thus, the consensus in analysts’ forecasts, measured as the correlation in analysts’ forecast errors, is negatively associated with a firm’s level of R&D investments. Moreover, the full expensing of R&D generates more variance in analysts’ forecasts. Other studies have confirmed the existence of a relationship between R&D and analysts’ forecast errors. Gu and Wang (2005) have analysed whether higher R&D investments affect analysts’ forecast errors. Their empirical findings show that the more a firm’s R&D investments deviate from industry average, the larger the analysts’ forecast errors will be. They have also found other specific results. In particular, the diversity of a firm’s technology investment portfolio and the innovativeness of technology investments are positively associated with analysts’ forecast errors. Conversely, stricter rules on the research process (that is, FDA approval of new drugs and new medical equipment), increasing the transparency on R&D outcomes, positively affect the quality of analysts’ research. Thus, in the biotech, pharmaceutical and medical equipment industries, analysts’ abilities to assimilate the information are better.

5.5

THE EFFECT OF IAS/IFRS ADOPTION ON R&D REPORTING: THE ITALIAN CASE

Apart from voluntary disclosure and analysts’ reports, a potentially important role in reducing information asymmetries is played by accounting

178

Evaluation and performance measurement of R&D

rules imposing stricter norms for disclosure. In European countries all public companies reporting consolidated financial statements have had to adopt since 2005 IAS/IFRS, instead of national GAAP. The adoption of IAS/IFRS, aimed at increasing the international comparability of financial statements, represents one of the main accounting changes and poses both a theoretical and a practical need to understand its implications. Above all, the importance of IAS adoption arises from the widespread engagement of most European countries in passing from a stakeholder-oriented accounting system into a shareholder-oriented reporting model (Bartov et al., 2005; Hung and Subramanyam, 2007). Moreover, IAS adoption deeply affects the quality of accounting information on intangible assets and on R&D in particular. Considering the accounting treatment of R&D in France, Italy and Spain, previous local GAAP uniformly allowed the recognition, among intangible assets, of development costs and applied research investments, whereas investments in basic research were fully expensed. Conversely, German GAAP were more conservative, denying the capitalization of all research and development costs. Consequently, the European IAS/ IFRS adoption limits (except in Germany) the amount of R&D investment reported in financial statements. Specifically, IAS 38 requires that development costs can be capitalized under certain criteria,12 whereas all research activities (applied and basic research) are recognized as expenditures. Similarly to US GAAP, IAS/IFRS state that firms cannot demonstrate that research investments will generate probable future economic benefits given their peculiar high uncertainty. Conversely, since the development phase of innovation is characterized by decreased uncertainty, future benefits can be determined with higher reliability. Moreover, the IAS/IFRS exclusion of research investments from the firm’s assets is mainly attributable to a conservative definition of asset. Like US GAAP, IAS/IFRS require that an asset can be recognized as intangible if: it is identifiable, that is, it arises from contractual or other legal rights; it is probable that the future economic benefits that are attributable to the asset will flow to the enterprise; and the cost of the asset can be measured reliably. For these reasons, this section reports descriptive evidence on how the compulsory adoption of IAS/IFRS standards has changed R&D reporting in the case of Italian publicly traded firms (see Morricone et al., 2008, for a complete description of the empirical findings). The empirical analysis, based on a comprehensive sample of 267 Italian public firms (see Appendix A5.2 for details on sample selection and variable definition), reveals that the disclosure behaviour of firms does not change after the adoption of IAS and, conversely, the number

R&D information

179

of R&D reporting firms remains on average constant in the time period 1996–2006 (see Panel A of Table A5.2.2 in Appendix A5.2). Looking at firms’ attributes such as performance and size (Panel B of Table A5.2.2 in Appendix A5.2), loss firms are on average more likely to disclose R&D investments than profit firms, and both small and big firms are more likely to report R&D than medium size firms. The results of the descriptive statistics (Table A5.2.3 in Appendix A5.2) indicate that book value of equity and net income are on average greater under IAS, but for the former the difference is not statistically significant. Considering the intangible effect on the key accounting variables, book value of equity less intangible capitalized is significantly reduced under IAS, whereas net income plus R&D expensed is still significantly higher after the accounting switch. These results foresee, respectively, an increased and unchanged importance of intangible assets and R&D after IAS adoption. Actually, the total amount of intangible assets and development costs are on average significantly greater under IAS, with the exception of R&D investments that do not significantly change under the two accounting regimes.

5.6

CONCLUSIONS

This chapter has examined the issue of how R&D information flows from firms to investors. Whereas Chapter 4 has focused on the valuation problem, without dealing with information issues, this chapter has addressed the question of how the characteristics of the R&D information chain can affect the market value relevance of R&D investments. It has been explained why R&D investments are likely to generate information asymmetries between insiders (managers) and outsiders (investors). The information problems are connected not only with the very nature of the R&D investments, but also with the well known deficiencies of the accounting data in financial reports. The empirical results reviewed in the chapter show that although R&D information is relevant for investors’ valuation and stock prices, it seems to be undervalued by the stock market. Two different mechanisms to reduce information asymmetries have been considered: voluntary R&D disclosure and financial analysts’ research. Finally, the role of accounting standards, with a specific focus on IAS/IFRS standards in Europe, has been examined. Overall, this chapter has shown that the information problem is a significant one when dealing with the valuation of R&D investments. Whereas some results are relatively well established (low quality of R&D accounting information, existence of R&D information asymmetries, potential

180

Evaluation and performance measurement of R&D

undervaluation of R&D investments), several questions are still open and currently at the centre of the academic and policy debate. In particular, the question of whether and how accounting standards can improve the provision of better information on firms’ R&D activities seems very promising. The literature review and the preliminary descriptive evidence on the application of IAS/IFRS also highlight some interesting research opportunities. First, it is possible to investigate whether the application of the new accounting standards has an impact on the relevance of R&D information for firm market valuation. Morricone et al. (2008) have analysed this question for Italian traded companies, for which IAS/IFRS adoption became compulsory in 2005, finding that the new accounting standards have not improved the relevance of the accounting information on intangible assets, including R&D investments. Related to this finding, a second question is whether and to what extent financial investors rely on non-accounting and qualitative information to derive their valuation of a firm’s R&D investments. Amir and Lev (1996) have already reported that investors increasingly use qualitative information in their valuations. Therefore, it would be interesting to examine whether and how the qualitative information about R&D projects and financing eventually disclosed in financial reports matters for investors’ expectations and valuations.

APPENDIX A5.1

EMPIRICAL MODELS OF VALUE RELEVANCE AND INTERTEMPORAL STUDIES

Value relevance or contemporaneous studies analysing the association between current stock price or return and accounting information, indicate the extent of current recognition of R&D relevance by investors (Lev and Sougiannis, 1996). As Barth et al. (2001a) have pointed out, value relevance valuation models are based on the assumption that share prices reflect investors’ consensus beliefs, so that value relevance research does not require assuming market efficiency.13 Consequently, inferences relate to the extent to which the accounting amounts reflect the amounts implicitly assessed by investors as reflected in equity prices (Barth et al., 2001a). In particular, valuation models used in value relevance literature are the price and return models, which aim at determining what is reflected respectively in firm value and in changes in value over a specific period of time.

R&D information

181

In the price model (see Equation 5.1), the dependent variable is the market value generally three months after fiscal year-end t, whereas independent variables include the key accounting variables, such as net income and book value of equity, and other accounting information whose value relevance has to be tested. In particular, value relevance studies, in order to include R&D in the valuation model, generally adjust net income (for R&D expensed) and book value of equity (for R&D capitalized). Aboody and Lev (1998) estimate the following price model with per share variables:14 Pit 5 a b0YRit 1 b1EPSit 1 b2BVPSit 1 b3CAPSOFit 1 eit (5.1) where Pit is the share price of the firm i three months after fiscal year-end t, EPSit is the reported annual earnings per share, BVPSit is book value of equity per share minus the capitalized software asset per share at year end t, CAPSOFit is the net balance of the software asset per share; YR is a year dummy. Differently from price model, in the return model (see Equation 5.2) the dependent variable is the annual stock return, measured generally from nine months before fiscal year-end t through three months after it, whereas independent variables are expressed both as level and as change. For instance, Aboody and Lev (1998) estimate the following return model: Rjt 5 a b0YRit 1 b1DCAPit 1 b2DEXPit 1 b3DEXPCAPit 1 b4DAMRTit 1 b5Xait 1 b6DXait 1 b7CAPREit 1 eit

(5.2)

where: DCAPit is the annual change in the capitalized amount of software development costs, DEXPit the annual change of software development expenses of ‘expenser’ companies, DEXPCAPit the annual change of software development expenses of ‘capitalizer’ companies, DAMRTit the annual change in the amortization of software assets for capitalizer firms, Xait is the adjusted (pre-software items) annual net income of firm i in the year t whereas DXait is the annual change in the adjusted (pre-software items) annual net income. YRit and CAPREit are the year dummy and capitalization intensity. Conversely, intertemporal studies (see Equation 5.3) focus on future stock returns in order to investigate whether investors fully recognize the value relevance of R&D information today. In particular, Lev and Sougiannis (1996) estimate the following model in which they add firm’s estimated R&D capital (scaled by market value) to the Fama and French (1992) model:

182

Evaluation and performance measurement of R&D

Table A5.2.1

Sample selection criteria

Sample Initial sample Less: non-ordinary shares eliminated Less: observations without fully consolidated financial statement or price data, total assets Less: firms that do not adopt IAS in 2005–2006 Less: firms without intangible investments Less: extreme observations (negative book value, top or bottom 1% of distribution of book value and net income) Final sample

Firms

Obs.

363 68 2

3993 748 1280

24 0 2

176 8 38

267

1743

Ri,t1n 5 c0,j 1 c1,jbi,t 1 c2,j ln (M) i,t 1 c3,j ln (B/M) i,t 1 c4,j ln (A/B) i,t 1 c5,j ln (E( 1 ) /M) i,t 1 c6,j ln (E/Mdummy) i,t 1 c7,j ln (RDC/M) i,t 1 ei,t1n

(5.3)

where: Ri,t+n is the monthly stock returns of firm i, starting with the 7th month after fiscal t year-end ( j 5 1,…12); bit is the CAPM-based beta of firm i; Mit is firm size (market value of firm i); B/Mit is the book to market ratio of firm i at fiscal year-end; A/Bit is financial leverage (ratio of book value of total assets to book value of common equity) of firm i; E(+)/Mit is the earnings to price ratio (positive earnings before extraordinary items scaled by the market value of equity) of firm i; E/Mit dummy is set equal to 1 when earnings are negative; R&D/Mit is the R&D capital estimated by cumulating for each year the unamortized portion of the annual R&D expenditures.

APPENDIX A5.2

SAMPLE, DATA SOURCES AND VARIABLES

The sample covers the period from 1996 to 2006. In particular, the years from 1996 to 2004 are considered Italian GAAP years, while 2005 and 2006 are IAS/IFRS adoption years. The sample has been selected, as explained in Table A5.2.1, retrieving data from Worldscope, Extel database, Osiris database and company financial statements. The initial sample is composed of 363 distinct firms listed in the

R&D information

Table A5.2.2 Panel A:

Sample composition: R&D reporting firms

time distribution

Year

non-R&D

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 Total

81 89 97 115 135 131 129 123 127 121 136 1284

Panel B:

183

R&D

Total

41 37 33 30 42 49 47 45 46 48 41 459

122 126 130 145 177 180 176 168 173 169 177 1743

Distribution by firm size and performance R&D reporting

Firm size Small

non-R&D

R&D

355 81.4% 653 75% 276 63.3%

81 18.6% 218 25% 160 36.7%

a

Medium Large c2 (2, 1743) Cramér’s V Firm Performanceb Loss Profit c2 (1, 1743) Cramér’s V

(38.42)*** 0.15 344 79.5% 940 71.8%

89 20.6% 370 28.2%

(9.92)** 0.075

Notes: a. Total assets is the proxy used for the firm size. b. Loss(Profit) firms are those that report negative (positive) net income. *** significant at the 1% level; ** significant at the 5% level; * significant at the 10% level in a two-tailed test.

184

Evaluation and performance measurement of R&D

Table A5.2.3

Descriptive statistics by accounting standards (sample period 1996–2006) Meana Italian GAAP

Book Value of equity

0.393 21.02 Net Income 0.006 22.72*** Book Value of equityLI 0.289 3.09*** Net IncomePI 0.013 22.61*** Intangible Assets 0.104 26.24*** Development costs 0.001 22.87*** R&D 0.007 20.33

Medianb IAS

Italian GAAP

0.407

0.371 20.43 0.022 21.57 0.287 3.36*** 0.026 21.08 0.056 26.15*** 0 27.22*** 0 0.52

0.020 0.240 0.027 0.166 0.003 0.008

IAS

Standard Deviationc Italian GAAP

0.374 0.194 0.777*** 0.024 0.110 2.07*** 0.236 0.237 0.77*** 0.026 0.113 1.75*** 0.103 0.128 0.55*** 0 0.011 1.23** 0 0.015 1.40***

IAS 0.220 0.076 0.269 0.085 0.173 0.010 0.013

Notes: a. The difference in means is based on pairwise t-tests with unequal variance. t-values in italic. b. The difference in medians is based on signed rank test. z-values in italic. c The difference in standard deviations is based on F-test. f-values in italic. *** significant at the 1% level; ** significant at the 5% level; * significant at the 10% level in a two- tailed test.

Italian stock market between 1996 and 2006, which do not belong to the financial, banking, insurance sectors, real estate or soccer. The first sample selection criterion excluded preferred shares as well as dual class shares yielding 3245 firm-years (295 distinct firms). Observations without available data on price and key accounting variables (total assets, book value of equity, net income, and intangible assets) have been excluded. This data requirement decreases the sample to yield 1965 firm-years, covering 293 distinct firms. The sample is then restricted to firms that adopt IAS in 2005. We retained only the firms with intangible assets and we excluded observations characterized by negative book value of equity. Finally, we excluded those observations in the top or bottom 1 per cent of the distribution of net income and market-to-book ratio. These requirements yielded a final sample of 267 distinct firms and 1743 observation-years. Table A5.2.2 describes the sample composition,

R&D information

185

whereas Table A5.2.3 outlines the descriptive statistics by accounting standards. The variables used in the analysis have been calculated as follows. Book value of equity represents the book value of the common equity of the company, while Book value of equityLI indicates the book value of equity less intangible capitalized for year t. Net income is net income before extraordinary items and Net incomePI is net income before extraordinary items plus R&D expensed. Intangible assets (INT) are the total net intangible assets capitalized and represent the sum of the following net variables: goodwill, brand and patents, development costs, licenses, computer software, other intangible. Development costs (DEV) are the development costs capitalized, R&D represents the research and development costs expensed.

NOTES 1. 2. 3. 4.

5.

6. 7. 8.

9.

Doubling the volume of production generally requires heavy investment in plant and machinery, but quadrupling the volume of drugs sold does not require any change in the underlying patents or R&D (Lev, 2001, p. 23). In particular, SFAS No. 86 allows the capitalization of software development costs if technological feasibility of the product under development has been established. See Section 5.5 of this chapter for a detailed discussion of R&D accounting treatment under IAS/IFRS. Whereas the studies analysed in Chapter 4 investigate the relationship between R&D investments and market value, the empirical literature reviewed in this section is substantially different in terms of methods, variables and objectives. In fact, the studies on R&D and market value reviewed in Chapter 4 are interested in the factors affecting the market valuation of innovation, as measured by R&D investments. On the contrary, the studies on value relevance seek to understand the relationship between accounting and market variables. It is easy to verify by reading the two chapters that the empirical models are consistently different. See Loudder and Behn, 1995; Lev and Sougiannis, 1996; Abrahams and Sidhu, 1998; for software R&D and Aboody and Lev, 1998; Ely and Waymire, 1999; Chan et al., 2001; Chambers et al., 2002; Healy et al., 2002; Cazavan-Jeny and Jeanjean, 2003; Gu and Lev, 2003; Lev et al., 2005; Ahmed and Falk, 2006; Oswald and Zarowin, 2007; Oswald, 2008. Specifically, they restate the earnings and book value of shareholders’ equity using these estimates of R&D assets and amortization and find the R&D assets are value relevant. Gu and Wang (2005) demonstrate that the diversity of technology investment portfolio renders more complex the information environment of R&D firms. Some of the theoretical models on the reporting game focus primarily on the agency relation between management and shareholders conceived as board of directors. In this case the game is mainly justified by contracting motivation (Holmström, 1979; Demski, 1994; Christensen and Feltham, 2005). Other studies analyse the reporting game between management and investors. In this case the semi-strong market efficiency represents the premise of the reporting game (Verrecchia, 1983, 1990; Dye, 1985, 1986, 1990). The statement of disclosure principle states: ‘If investors know a manager is endowed with one particular bit of nonproprietary, relevant, effable information, the release of which does not alter the manager’s compensation, and investors can take positions on

186

10.

11. 12.

13.

14.

Evaluation and performance measurement of R&D the market prior to information’s release, then this information will be disclosed’ (Dye, 1985, p.127). Private information has a proprietary cost when its release is unfavourable to a firm and therefore is potentially damaging (Verrecchia, 1983; Dye 1985, 1986). As pointed out by Verrecchia (1983), proprietary information includes not only bad news about the firm but also favourable information that however may be useful to competitors in a way which is harmful to the firm’s prospects. In this situation, disclosure of good news may assuage investors’ concerns regarding the firm’s future earnings prospects while, at the same time, worsening these prospects by divulgence of proprietary data (Dye, 1986). In particular, capitalization criteria required by IAS 38 are: technical feasibility of completing the intangible asset so that it will be available for use and sale; firm’s intention to complete the intangible asset to use or sale it; firm’s ability to use or sale the intangible asset; the existence of a market for the output of the intangible asset or the usefulness of the intangible assets; the availability of adequate technical, financial and other resources to complete the development and to use or sell the intangible asset; firm’s ability to measure reliably the expenditure attributable to intangible asset during its development (IAS 38, par. 57–59). As Barth et al. (2001a, p. 95) specify, ‘the research needs not assume that equity market values are “true” or unbiased measures of the unobservable “true value’’ of equity, or that they reflect unbiased measures of unobservable “true” economic values of firms’ assets and liabilities or income generating ability’. In the following examples reference is made primarily to valuation models used by Aboody and Lev (1998) in order to ease the description of different empirical models to address the relation between market value and R&D information. Moreover, Aboody and Lev (1998) is one of the few studies that compare the value relevance of R&D capitalized and expensed, which require different adjustment to key accounting variables.

PART III

Innovation system

6. 6.1

Technology platform INTRODUCTION

This chapter deals with the issue of assessing and evaluating R&D and innovation by adopting a broader unit of analysis, which moves from the single firm to systems of firms. The need to address the characteristics of the innovation process according to a perspective that goes beyond individual firms’ boundaries is supported by two distinct insights: on the one hand, a large body of empirical literature since 1990 has clearly highlighted how a number of systemic interactions taking place within both structured and informal networks of firms significantly affects the rates of production and diffusion of new knowledge; on the other hand, new paradigms of innovation management have emerged that explicitly stress the importance of different types of external sources of innovation in defining companies’ technological strategies. The fundamental results of the first stream of research are mainly represented by the new geography of innovation (Audretsch and Feldman, 2004). The specific features of innovative knowledge as an economic good imply that the system into which a company is embedded, also geographically defined and including institutions, is relevant to its ability to develop and introduce new technologies. Following Von Hippel (1994), it is possible to state that while information is codified and can be formalized, knowledge is tacit, embedded in human capital, difficult to codify and often only serendipitously recognized. Hence, co-localization is expected to mitigate the inherent uncertainty of innovative activity because proximity enhances the ability of firms to exchange ideas and be cognizant of important incipient knowledge (Feldman, 1999). Innovation clusters spatially where knowledge externalities reduce the costs of scientific discovery and commercialization. In this perspective, localized knowledge spillovers across a system of firms play a significant role in providing access to new economic knowledge and increasing the productivity of economic actors. The seminal contributions of Romer (1990) on the endogenous growth theory reinforce this view by identifying the separation of economically useful scientific-technological knowledge into two parts. The total set of 189

190

Evaluation and performance measurement of R&D

knowledge consists of the subsets of non-rival, partially excludable knowledge elements, which can be practically considered public goods, and rival, excludable elements of knowledge. Codified knowledge is non-rival since it can be used by several actors at the same time and for many times historically. Codified knowledge is only partially excludable because other companies can benefit from it, even in the case of patenting. Rival, excludable knowledge elements are primarily the tacit knowledge of individuals whose exchange also calls for non-market interactions. The Hayekian notion of distributed knowledge, dispersed and fragmented in a myriad economic agents, provides the basis for understanding knowledge complementarities. Only when a complementary set of knowledge fragments is brought together within a context of consistent interactions, can successful innovations be introduced and adopted: technological knowledge becomes the product of a collective activity. Furthermore, the contributions of Lundvall (1992) and Von Hippel (2008) on the key role of user–producer interactions, both upstream and downstream, as basic engines for accumulating new technological knowledge and introduction of new technologies, confirm that vertical interactions among heterogeneous agents are important in generating knowledge. Hence, the overall economic literature on knowledge spillovers stresses how the innovation potential of a firm is subject to a sort of mis-measurement whenever we do not take into account its industrial environment consisting of co-localized competitors and suppliers. A major criticism of this stream of literature is that there is a lack of evidence about the actual channels that the flows of knowledge go through across the system of firms. With respect to this point, the managerial literature on the so-called open innovation paradigm (Chesbrough, 2003a) stresses how the positive effects exerted by the external environment in the form of knowledge externalities and exploitation opportunities are not spontaneous, but rather require a proactive approach by companies. Firms located in the proximity can take advantage of knowledge generated by other co-localized firms. However, they can do so only if, and after, dedicated resources have been invested in such activities as research, networking, communication and various forms of agreements to implement non-market interactions, eventually leading to acquisition and exploitation of external technological capabilities (Mokyr, 2006). In this perspective, knowledge externalities are pecuniary rather than involuntary: external knowledge has a cost and the formation of a system of firms requires substantial investments. The two theoretical approaches to the systemic view of innovation, namely the one proposed by the economic theory of knowledge spillovers and the managerial one, are to some extent complementary. Moreover, their joint analysis can provide useful

Technology platform

191

tools for interpreting and evaluating differentials among firms’ innovation capabilities. This chapter first seeks to identify the most relevant contributions of the two streams of literature in order to answer the following questions. Why do firms need to access external knowledge? Through which channels can external knowledge be conveyed into the company and exploited? Which environmental conditions favour such exploitation? How does the growing complexity of technology affect the rate of knowledge flows within the system? As explained below, answering these questions calls for a preliminary identification of the boundaries of the systems of firms and the criteria for identifying them. The bulk of economic literature has long assumed that the geographical variable is the only criterion for identifying systems of firms whose joint R&D and innovation dynamics were worth studying. However, other streams of literature have analysed innovation dynamics in systems of firms defined by other non-geographically based criteria, such as the presence of joint research projects or explicit research and development alliances. In this chapter, the focus will be on a rather new approach: the technological platform. This concept can be adopted as a useful unit of analysis for addressing the innovation potential of a set of firms actually linked more by knowledge complementarities than by simple geographical proximity or sectoral specialization. What is a technology platform? According to the official definition of the European Commission (2005, p. 3): European Technology Platforms focus on strategic issues where achieving Europe’s future growth, competitiveness and sustainability depends on major technological advances. They bring together stakeholders, led by industry, to define medium to long-term research and technological development objectives and lay down markers for achieving them . . . They cover the whole economic value chain, ensuring that knowledge generated through research is transformed into technologies and processes, and ultimately into marketable products and services.

Although this definition might appear to be strictly policy-oriented, it is interesting to note how it integrates a series of relevant concepts. For one thing, the idea of horizontal coordination of R&D activities among heterogeneous subjects (stakeholders and firms) to some extent reflects the so-called triple-helix paradigm (Leydesdorff and Etzkowitz, 1996), which stresses the importance of coordination among companies, public research and the government. At the same time, the focus on vertical coordination along the innovation value chain goes beyond a traditional model of technology cluster and

192

Evaluation and performance measurement of R&D

emphasizes the role of the demand side in stimulating the emergence of new technological trajectories (Kleinknecht and Verspagen, 1990; Adner and Levinthal, 2001). While the previous theoretical definition is certainly interesting, actually translating the technology platform concept into an operational model is much less straightforward. The objectives of a technology platform consist mainly in the development of common research agendas among partners and in the establishment of joint technology initiatives on long run R&D projects. Hence, the technology platform aims at building an innovation environment which is expected to ease problems of coordination in investments and to favour the emergence of shared technological paradigms, particularly in sectors characterized by high technological complexity. Therefore, the last part of this chapter presents a real case of technology platform implementation in the aerospace industry in Europe. This allows us to assess the relevant impact, for both managers and policy makers, of sources of innovative knowledge that are beyond the firm’s boundaries. The innovation potential of a company cannot be fully explained by its internal scientific and technological competences – the analysis should also capture the availability of complementary external knowledge that the company is able to leverage. In this line of reasoning the technology platform is an ideal unit of analysis for seizing innovation linkages that otherwise would be systematically underestimated. The adoption of the technology platform perspective also requires the definition of specific evaluation criteria, indicators and metrics, for consistent further comparison and analysis. In this respect, it is critical to understand how to assess the effectiveness and efficiency of a technological platform, that is, which indicators should be used to evaluate whether the organization of the platform is adequate for achieving the objectives for which it has been established.

6.2

DETERMINANTS FOR GOING BEYOND THE FIRM’S BOUNDARIES

This section reports a brief survey of the motivations pushing a firm to define its R&D and innovation strategy involving a variety of different players located beyond its boundaries. A basic determinant could be the lack of critical mass required to set up R&D infrastructures or to launch R&D projects. While this is widely diffused, especially among SMEs, many other issues should be brought into play to grasp the overall phenomenon of R&D deployment beyond the single firm boundaries. The nature of knowledge as an economic good entails specific paradigms

Technology platform

193

for both its conceptualization and its management; these paradigms, especially in technology intensive industries, can have dramatic impacts on the reconfiguration of the overall value chain and call for theoretical considerations that are not fully captured by the traditional transaction costs approach. In particular, the theoretical contributions that have addressed the reasons why a company needs to interact with external entities can be classified according to a twofold scheme that divides the innovation process into the quest for innovative knowledge (knowledge sourcing) and development processes (development interdependence). In the following subsections, these two phases are discussed in more detail. 6.2.1

Knowledge Sourcing

Under a managerial perspective, innovation is based on the ability to integrate, build and reconfigure internal and external competencies to address rapidly changing environments (Teece et al., 1997). In this line of reasoning, the linkage between the firm and the sources of knowledge from which effective improvement inputs could be supplied is crucial for business success in both the short and the long term. In their seminal work, Cohen et al. (2002) propose a systematic assessment of the relative importance of a broad range of knowledge sources through a large survey of US R&D managers. Such sources, besides internal R&D departments, include external sources such as competitors, customers, buyers, consultants, research Joint Ventures and academic and public R&D centres. Moreover, they asked R&D managers to evaluate the importance of such sources both for the selection of new innovation projects and the completion of existing innovation projects. Their results provide a picture that clearly stresses how innovation strategy is shaped by the external environment in both phases. Clear evidence of the importance of buyers’ feedbacks among the predominant source of suggestions about new project proposals (in keeping with the non-linear view of the innovation process proposed by Von Hippel, 2008) comes from every kind of industry. More than 40 per cent of companies evaluated the knowledge source related to competitors as important, and a relevant value is attributed to public research, particularly for the completion phase. Concerning the second phase, which is closer to the final market, the data provided by the scholars clearly reflect the risk of expropriation of competitive advantage that companies fear. In other words, the importance of competitors as a source of knowledge strongly decreases, while we observe a substantial increase in the value of internal research departments. Two final relevant points raised by Cohen et al. (2002) concern the high sector heterogeneity

194

Evaluation and performance measurement of R&D

in the dynamics of external knowledge sourcing and the overall nonnegligible impact of public research on firms’ innovation capabilities. Such knowledge based on public R&D investments is conveyed to companies not only through codified channels, but mainly through informal communication between companies and researchers. This evidence helps to identify two key issues that will be discussed further below: the trade-off between the use of external sources and the risk of knowledge expropriation; and the need to adopt a non-linear innovation management approach where feedback from buyers is a fundamental driver. Building upon the evidence, this section articulates its discussion on knowledge sourcing along three dimensions: the theory of path dependence and the risk of technological lock-in; firms’ knowledge bases and the rationale for R&D alliances; and the users as a source of external knowledge. As mentioned, when analysing knowledge sourcing strategies, particular attention should be paid to the proper balance of both internal and external sources of knowledge, according to their characteristics and dynamics (Fey, 2005). The process of technological knowledge accumulation that characterizes firms operating in technology intensive industries embeds the well-known ‘path dependence’ phenomenon (Nelson and Winter, 1982). The use of sources of innovative knowledge located beyond a firm’s boundary can be a prime tool for avoiding the negative effects of path dependence. Following Coombs and Hull (1998), it can be stated that the notion of path dependence is strictly related to the concept of positive returns. Doing things in a particular way, whether it be designing, manufacturing or marketing a product, yields results that predispose an organization to perform such activities in the same way the next time around. Path dependence of corporate activities involves different dimensions of the company. The first domain is that of technological endowments, which comprises the existing machinery, equipment and production processes that may shape the future possibilities for developing other products. The second area of path dependence relates to the ‘knowledge base’ of the firm, which includes the set of scientific and technological capabilities embedded in the human capital, as well as the corporate culture and the cognitive framework shared among individuals operating within the firm. The interpretation of such a cognitive framework can be extended to include common understanding and foresight of market dynamics and technological evolution. Finally, the third area of path dependence concerns operational routines, which include the system of uncodified rules governing the development of new products and services, and more generally the problem-solving process. While path dependence is advocated as a relevant concept for

Technology platform

195

explaining differentials in firms’ performances leading to distinctive specialization and competitive advantage, it should be emphasized that it also tends to increase technological lock-in and may reduce the capacity to exploit new knowledge. This aspect clearly indicates the need for a company to go beyond its boundaries systematically, in order to scan the relevant technology frontiers and integrate the most interesting opportunities into its operations. In this sense, interactions beyond a firm’s boundaries represent a major channel for grasping technological opportunities and updating the internal knowledge base and routines. The concept of technological opportunities helps to shift the analysis from the single firm to a system of firms. The technological opportunity set is endogenously created by investments in new knowledge and creates opportunities for use by third party firms and new ventures (Shane, 2001). The creation of new knowledge gives rise to new opportunities through knowledge spillovers. Therefore, entrepreneurial activity involves not simply the arbitrage of opportunities, but also the exploitation of new opportunities created, but not appropriated, by competing organizations (Acs et al., 1994). Closely connected to the knowledge accumulation processes leading to technological path dependence and lock-in effects is the issue of exploring new technological trajectories in order to enhance or challenge technology paradigms (Dosi, 1982). The distinctive technological knowledge of a firm is called on to face both the intrinsic shock implied by the creation of new technological solutions and the competitors’ moves along the same development track. Therefore, the general research environment and the related technological trajectories in which a firm or an entire supply chain is committed to compete should be analysed to choose the proper internal R&D strategies. Similar conclusions can be reached about the need for entrepreneurs and managers to define the innovation framework exploiting external knowledge sources by following the perspective of the so-called absorptive capacity. Dating back to Cohen and Levinthal (1990), the term indicates a firm’s ability to value, assimilate, and apply new knowledge. It is directly linked with the approach and the routines adopted to renew corporate capabilities and to maintain a proper linkage with interesting sources of knowledge and innovation. In technology-based industries strategic alliances are often observed, and the main motivation for the allying firms is either to explore new knowledge (Hagedoorn, 1993; Khanna et al., 1998; Kale et al., 2000), or to access other firms’ knowledge assets in order to exploit complementarities (Grant and Baden-Fuller, 2004). Besides these theoretical insights, the empirical assessment of the phenomenon is still highly controversial given the actual difficulty in finding an appropriate measure of knowledge base complementarities. Zhang

196

Evaluation and performance measurement of R&D

et al. (2007) stress an additional theoretical cause that might explain the mixed empirical findings on R&D alliances performance and knowledge bases of partnering companies: most studies do not adequately consider the important influence of management, particularly the R&D organizational structure, on the connection between R&D inputs and absorptive capacity. For example, a centralized R&D structure may facilitate dense internal communication flows and thus increase a firm’s absorptive capacity (Jansen et al., 2005). This in turn may influence the willingness of firms to enter into strategic alliances. Nevertheless, the empirical literature on strategic alliances has come to one less ambiguous result, which seems to hold across different industries: the broader the knowledge base of a multitechnology firm, the higher its commitment to alliances and its capability to internalize and exploit knowledge generated beyond its boundaries (Mowery et al., 1996; Brusoni et al., 2001; Orsenigo et al., 2001). Such an effect is clearly mitigated by a coexisting trade-off due to the risk of internal knowledge being expropriated and hence of competitive advantage being lost as a consequence of R&D alliances (Ahuja, 2000). The most recent contribution to the debate about knowledge sourcing and innovation beyond firms’ boundaries comes from the ‘user driven development’ stream. In this contemporary field of research, stimulated by internet-related phenomena such as Creative Commons and Open Source, the final customers and their organized communities, rather than other firms, are involved in inspiring new product development (Jeppesen and Molin, 2003). While in Open Source software those user organizations directly affect the creation of technological innovation (for example, by producing new codes to enhance computer program features), generally they supply outstanding inputs to firms’ marketing departments. The user-driven innovation paradigm essentially builds on a more established concept of market-induced innovation and stresses the proactive role of the demand side (Von Hippel, 2008). 6.2.2

Development Interdependence and Open Innovation

In the previous subsection, a set of theoretical points were highlighted that stress the relevance of knowledge under its many different facets as the key driver leading firms to interact with external subjects while defining their R&D and innovation strategy. This subsection focuses on a second array of determinants for such behaviour that are likely to be relevant even under the hypothesis that a firm has full control of the technological and procedural competences required to develop a specific technological component. The new set of rationales for the adoption of a systemic focus for

Technology platform

197

assessing R&D and technological innovation is based on the realization of the increasing interdependence that occurs in new product and technology development (Hobday et al., 2000; De la Mothe and Link, 2002) As emphasized by Rycroft and Kash (2002), cooperation between firms could be the answer to absorb complexity and share risks, even when contrasting results from the variety of variables and events interfere in the overall process performance. In products that have modular characteristics and in those that experience relevant network economies, a fundamental role in fostering the innovation outlook beyond the single firm boundaries is constituted by issues related to technological standards (Van Wegberg, 2004). The standard setting processes oblige firms to reach agreement or entrust the fate of rival standards in products and services to market selection. In both cases, firms work in a situation of strategic interaction that could generate a hybrid form of coordination between competition and cooperation, implying a systemic vision to be properly dealt with. While recent history provides us with anecdotal evidence about strong technological battles for new technical standards in different industries, it is clear that interoperability and modularity and joint development are becoming key issues in the strategies of innovation leaders. The influential paper by Teece (1986) highlighted the role of complementary technological assets for a technology to be commercially exploited. Such complementarities need not be physically bound within a firm, but can be the result of a systemic interaction among firms. Many technologies are systemic today, and successful commercialization requires bringing together complementary technology as well as complementary patents (Teece, 2006; Somaya and Teece, 2006). As highlighted above, different streams of literature, although adopting different perspectives, have stressed in recent years the key role that the systemic dimension plays in shaping the innovation potential of firms, in terms of knowledge source availability, and capability to exchange and recombine innovative knowledge, so as to develop complex technologies and identify new market opportunities. The early Schumpeterian model of the lone entrepreneur bringing innovations to markets appears inadequate to some extent. Innovators rarely innovate alone. They tend to band together in teams and communities of practice embedded in formal or informal networks of interactions (Laursen and Salter, 2006). In this line of reasoning, the open innovation paradigm, proposed by Chesbrough (2003a) and still in development (Chesbrough and Appleyard, 2007), seems to offer a structured synthesis of most of the above-mentioned determinants. According to a traditional approach to innovation strategy, the ability to have full control of the outcomes

198

Evaluation and performance measurement of R&D

of research and development activities is considered a major incentive to R&D investment. An open innovation approach, on the other hand, stresses the centrality of the process of screening new ideas. In any R&D process, researchers and their managers must separate the bad proposals from the good ones so that they can discard the former while pursuing and commercializing the latter. A full set of interactions with the external environment can significantly enhance such screening capability and also sustain the actual development of the new technological solutions: ‘firms that can harness outside ideas to advance their own businesses while leveraging their internal ideas outside their current operations will likely thrive in this new era of open innovation’ (Chesbrough, 2003c, p. 41). Hence, at the core of the open innovation model and other similar conceptualizations of innovation is how firms use ideas and knowledge of external actors in their innovation processes. Chesbrough (2003c) suggests that firms that are ‘too focused internally’ are ‘prone to miss a number of opportunities because many will fall outside the organization’s current business or will need to be combined with external technologies to unlock their potential’ (Chesbrough, 2003c, p. 37). The literature provides some interesting analyses of the impact of openness on innovation activities. Ahuja (2000) finds that indirect and direct ties influence the ability of a firm to innovate, but that the effectiveness of indirect ties is moderated by the number of the firm’s direct ties. Rosenkopf and Nerkar (2001) explore the role of boundary-spanning searches for both organizational and technological boundaries and find that search processes that do not span organizational boundaries generate lower effects on subsequent technological evolution, indicating that the impact of explorative search is greatest when the search spans both organizational and technological boundaries. Laursen and Salter (2006) examine how different strategies for using external sources of knowledge influence innovative performance. Their results strongly suggest that searching widely and deeply across a variety of channels can provide ideas and resources that help firms gain and exploit innovative opportunities. However, the ability of a firm to pursue an efficient search strategy depends on past experiences and future expectations of managers, particularly for technological domains characterized by high uncertainty on dominant design (Levinthal and March, 1993). It has been shown that different streams of literature have stressed the absolute and increasing importance of inter-firm interactions for the definition of R&D strategies. Such interaction can be targeted to access specific pieces of knowledge or to set up research trajectories that take into account the role of complementary knowledge. Regardless of the various motives behind the need for such interactions, it is evident that

Technology platform

199

they require the existence of some form of either formal or informal network. Technological clusters represent a form of endogenously generated network, driven by different typologies of economies of agglomeration. Recent radical changes in the processes of production and diffusion of both goods and innovative knowledge suggest that knowledge networks cannot be geographically bounded. Hence, the technology platform appears to be a more appropriate unit of analysis with which to observe systems of firms and to assess their relative innovation potential.

6.3

TECHNOLOGY PLATFORMS: A NEW R&D ORGANIZATION FRAMEWORK

In technology-intensive industries, the rate of competence specialization of the players is continuously rising. The emergence of clear technological focus inside firms increases the relevance of complementarities and interdependences, thus requiring a broader interpretative framework to manage technological complexity. This is the fundamental rationale behind adopting new units of analysis that consider the specific knowledge and industrial interactions taking place among a heterogeneous variety of players sharing the core technological competences that distinguish a specific market. The origin of the definition of technology platform dates back to many seminal works about new product development (Abernathy and Utterback, 1975), where the term indicates a technical design used to optimize engineering and production costs by the sharing of common units and physical commonalities. The concept of technology platform rapidly evolved into a production organization paradigm (Meyer and Utterback, 1993), and it is still employed to describe an architecture and a set of subsystems from which a pool of different derivative products can be efficiently developed and manufactured. As a tool to manage complexity, the concept of technological platform has been extended to the R&D organizational level, in terms of both management and strategy. Taking into account the tight interdependences developed by many large hi-tech companies and their extended supply chains, Iansiti and Levien (2004) have introduced the idea of business ecosystems. In such a unit of industry and competition analysis, a keystone company and its networks of partners and suppliers share a set of solutions as a ‘club good’ confined in their business ecosystem. This resource, such as a common operative system, is explicitly labelled as a ‘technology platform’ and plays a fundamental role to ensure both competitiveness and growth to all the members of the business ecosystem. Strongly diffused in this more recent branch of strategic

200

Evaluation and performance measurement of R&D

literature is the idea of an innovation network, whereby the internal R&D of a corporation is considered as the hub of a complex innovation system, which has to balance the efforts dedicated to internal discovery and external innovation absorption. This definition of technology platform underlines the presence of a good, belonging to a flagship corporation, on which a system of firms relies. It is thus a coordination mechanism and a technology development scaffold. A different approach is adopted by considering the technology platform as a stakeholders’ organization, devoted to set up a coordinated R&D agenda between different players interested in jointly developing technological innovation. A technology platform is therefore an organizational tool to bring together all the interested parties in a particular technological domain, including subjects operating both upstream and downstream in the development process of a given bundle of products, as well as any other relevant institutions in the field. In this sense, technology platforms are also expected both to sustain the coordination among private companies and, eventually, to foster effective public–private partnership. The creation of the network is based on the involvement of all key stakeholders and is led by industry, around a shared vision for the development of the relevant technologies. The involvement of leading industrial companies, research centres and academia in defining and sharing a long term research agenda, basically the core activity of a typical technology platform, is beneficial for the coordination and the resource allocation process of a variety of R&D players. For industrial R&D managers, a technology platform framework implies the possibility of addressing their research efforts towards specific targets and developing proper actions to exploit their technical knowledge. Academic and private researchers can exploit an established research agenda to gain an industry-based view about what is going to be developed in their field of interest, thus leading to more adequate research lines and infrastructure investments selection. Finally, from a research policy perspective, it enables public decision makers to analyse, select and adopt broadly shared financing priorities, thus leading to effective investment in innovation activities. The technology platform can be seen as an explicit organizational form of the triple helix model. According to that theoretical paradigm, the interaction of academia, industry and government as heterogeneous agents playing overlapping roles in innovation, affects knowledge production (Etzkowitz, 2003) and could be positively shaped by organizational settings like a technology platform. The concept of technology platform, as a system perspective on R&D organization, is independent from R&D and innovation policies, but, for research purposes, publicly funded technology platforms are the most

Technology platform

201

accessible and observable experiences. It is worth stressing that the identification of technology platforms at the European level has followed a bottom-up process which started from industry. Hence, publicly funded technology platforms are likely to represent evidence of a pre-existing complex set of relationships among companies. The policies simply had the role of stimulating the emergence of such bundles of relationships. Therefore, in the following subsections, we provide a description of technology platforms in Europe and discuss the case of a technology platform in the aerospace sector in more detail. 6.3.1

European Technology Platforms: R&D Coordination and Policy Making

With the spring European Council held in Brussels in March 2003, a formal base for the development of European technology platforms (ETP) was set up to bring together technological know-how, industry, regulators and financial institutions, to develop a strategic agenda for leading technologies. Even though there were some differences in the stage of deployment and outcome delivery, 25 ETPs had been established by 2005, including the major technologies for European growth and development. In 2007, at the beginning of the VII Framework Programme for Research, Technological Development and Demonstration Activities (FP7), the stable pool of active ETPs comprised 31 elements. While aggregation and interaction of stakeholders involved in a particular platform is deeply influenced by specific knowledge and industrial features, a basic common lifecycle and structure as well as a typical set of outputs can be described to explain the overall functioning of ETPs. The development of ETPs was a bottom-up process that started in 2003 with the promotion and endorsement of the ETP concept by the European Commission. Since then, in many technology domains, under a clear leadership exercised by prominent industrial players, many initiatives of aggregations of stakeholders have emerged. An initial critical phase can be identified in the informal process of the communication and negotiation between technological stakeholders at the European level, to define the participation in the ETP and the selection of the governance form. Due to the policy requirement of transparency and openness of a technology platform to be held to account by the European Commission, this stage is generally facilitated by a public call for expressions of interest to start the technology platform. At this stage, the interaction among the subjects joining the ETP consists mainly in identifying shared long-term technology and market scenarios in which the ETP is expected to claim a coordination role subsequently. The key deliverable is

202

Evaluation and performance measurement of R&D

the strategic research agenda (SRA), which might be considered the main challenge in both the ETP qualification process and its usefulness outside the policy-making context. To support the definition of the SRA, a proper governance setting has to be defined. There are no official prescriptions from the EU Commission, but the general principle of taking into adequate account all the involved stakeholders’ perspectives, while giving the leadership to industrial players, is the basis of the governance scheme adopted by the individual ETPs to balance power and interests among their constituents. The most visible short term goal of the ETPs was to steer the definition of the research themes of FP7, thus contributing to the policy-making process of the allocation of €50 billion in the time span 2007–13. With respect to this aspect, even with some sector-specific differences, it could be observed that the ETPs have fulfilled their stakeholders’ expectations, since more than 70 indications coming from the SRA of the 31 official ETPs have been adopted in the FP7. These results, witness to the quality to the ETPs’ activities, reflect also the endorsement of the players subscribing to the SRA from a financial and operative point of view. The implementation of the joint efforts in technology development coming from SRA is not limited to FP7. In the vast majority of cases, it includes strategic international networking between public and private research performers, as well as an extensive and coordinated effort in regional, national and European fundraising to support the SRA implementation initiatives. In a limited number of cases, ETPs have been set up to develop special public–private partnerships to deal with those high-level technology development programmes that could not be achieved by standard types of public policy. In those cases, where market failure, strategic relevance and industry commitment are clearly demonstrable, joint technology initiatives (JTI) can be set up through special negotiation with the European Commission to develop technology program benefiting from a dedicated research budget. With their role of established technology stakeholder aggregations, the ETPs represent a way for the European Commission to deal with consensus building upon R&D choices and to shift outside its governance bodies those negotiation processes required to set efficiently the priorities for public R&D investment in the major European industrial sectors. Nevertheless, ETPs are expected to be more than a simple policy initiative to favour consensus, as they favour the consolidation of strategic relationships among players (innovative firms, private research centres, public research). More specifically, technology platforms are expected to contribute to a reduction in technological uncertainty and to significantly enhance coordination among firms involved in the development of

Technology platform

203

complex technologies, particularly when these are subject to standardization processes. Furthermore, R&D platforms are expected to facilitate a convergence between industry R&D trajectories and the lines of public investment by European governments. 6.3.2

European Technology Platforms and Aeronautics: A Case Study

This subsection analyses the technology platform approach by taking the aeronautics knowledge domain as a reference to illustrate both the specific technology features and the theoretical frameworks that justify the creation of an ETP, in this case devoted to aeronautics R&D and technological innovation. In order to clarify the potential benefits of this innovative type of coordination mechanism, we also briefly discuss the key characteristics of the R&D activities in the aerospace industry. R&D in the aerospace industry The aerospace sector deals with the development, manufacturing, testing and maintenance of a number of very complex systems, such as commercial aircraft and helicopters, launchers, spacecraft and satellites, and systems for command, control and communication. Apart from its central role in the defence and foreign policy strategies of all countries, the value-added provided by the aerospace sector to GDP is surprisingly low. According to the annual survey by the AeroSpace and Defence Industries Association of Europe (ASD) and surveys by the Aviation Industries Association (AIA), it proves to be 3.3 per cent in the USA; 3.2 per cent in Canada; 3 per cent in France; 1.5 per cent in Germany; 3.4 per cent in the UK; 1.3 per cent in Italy; and 1.6 per cent in the EU11. By contrast, the aerospace sector’s contribution to technological innovation is witnessed by a yearly patent production that accounts for more than 10 per cent of all patent filings at the European Patent Office (EPO) and United States Patent and Trademark Office (USPTO). Moreover companies in this industry typically spend about 14 per cent of their yearly turnover in R&D activities, which makes aerospace one of the most knowledge-intensive industrial activities. Looking at R&D spending in the aerospace sector, important differences can be highlighted in the share of the public/private investment mix between the USA and Europe, where the public intervention is particularly heavy (more than 50 per cent of total R&D spending), and other countries. This characteristic could be partly explained by structural differences in the composition of the aerospace sector, which in the USA presents a larger military share (about 50 per cent) than in Europe (about 30 per cent). In

204

Evaluation and performance measurement of R&D

the markets, a selection and concentration process has taken place in the USA and in Europe since the 1950s, which has resulted in the survival of a few leading companies capable of managing the manufacturing program of very complex systems. The industry is characterized by a hierarchical structure, shaped as a multi-level pyramid governing economic and legal relations, organization of work and knowledge and information flow (Esposito, 1996; Niosi and Zhegu, 2005). The large companies at the top of the pyramid (for example, Boeing, EADS, Lockheed Martin, Northrop Grumman) manage the overall production program, coordinating the information and material flows between the lower levels, and select the other companies participating in the program. The second level of the pyramid is occupied by companies dealing with major subassemblies (typically engines/propulsion systems, avionics, hydraulics, landing gears) and linked to the leading enterprises by various agreements, such as subcontracting, association and risk-sharing (Esposito, 1996). At the third level, there are companies with specific know-how for the production of components, while at the base of the production process there are many SMEs, which supply parts to other companies placed at the third level and in special positions across the supply chain. Figure 6.1 illustrates the aerospace productive structure, underlining both its pyramid and multiple tier structure. Inside the pyramid, an intense vertical information flow supports the production process. However, significant information flows also take place horizontally inside the sector and from other industries. In this way, the aerospace sector accelerates technology and methodology transfer to other sectors or fosters their technological evolution by setting high-level goals to subcontractors. The complexity of the set of interactions behind the simplified scheme illustrated in Figure 6.1 is reflected also in the R&D and innovation process in this industry, which requires a high level of coordination among subjects that, at the same time, are often competitors on the final markets. In the case of the aerospace sector, the demand in the final market is difficult to predict, since the time frame of a new development project can last for many years and there are likely to be changes in requirements because of unforeseen circumstances (for example increase in fuel costs) or variability in customer specifications (for example low-cost flights). Moreover, the R&D process is very costly and concentrated in the first years of the program, while the return on the investments takes place at a later date and only if a worldwide commercialization plan is successful (Hayward, 1986; Bonaccorsi, 1996). A further distinctive feature of the aerospace sector lies in the importance of economies of learning, scale and scope. Learning by doing and learning by using are at the basis of

Tie

r4

Tie

r3

Tie

r2

Tie r

1

Technology platform

Source:

System integrators

205

• Design and project management • Overall technical responsibility • Final selling

Sub-system producers

• Avionics • Propulsion • Structures

Components suppliers

• Electronics • Mechanical parts • Hydraulics

Small and medium-sized enterprises

• Pieces and parts required to build third level components

Adapted from Niosi and Zhegu (2005).

Figure 6.1

The aerospace productive structure

product improvement as long as the manufacturing experience is growing (Alchian, 1963; Rosenberg, 1976, 1982), or following informal activities of problem solving, also in cooperation with final clients. Scope and scale economies result from the possibility of extending a program to support the manufacturing of a family of products (for example several models of aircraft designed to meet the requests of adjacent market segments), sharing the costs of R&D, design and use of production facilities. Extensive quantitative data support this view, particularly with reference to jet and turbo propeller engines (Bonaccorsi and Giuri, 2000). Innovation in the aerospace sector is characterized by the concept of ‘dual use’, which identifies technologies, know-how or products that can be exploited (even with changes or in different time frames) in both civil and military applications. This concept emerged in the 1990s when the US government decided to reduce the barriers between commercial and military industries, in order to speed up the development of new products, yet reduce their procurement costs. Besides creating the conditions for important economies of scale and scope, the dual use practice facilitates the transfer of technologies between the two sides and brings the civilian lifecycle (with frequent incremental changes) closer to the military one (with less frequent but radical changes).

206

Evaluation and performance measurement of R&D

At the same time, this remarkable feature of the industry has a particular effect on R&D activities. While military requirements are mainly focused on performance, thus pushing technology to be exploited, the civil technical specification takes into account different kinds of variables, such as fuel consumption or comfort, that lead to complementary improvement of the final products. The characteristics of the aerospace sector directly linked to its knowledge content have been studied from different viewpoints. The special role of leading firms acting as system integrators has been recognized by Granstrand, Patel and Pavitt (1997) and by Piscitello (2000). Such firms build on wide and diversified know-how, accumulated over many years of activity. In order to solve unforeseen problems or to cope with changes in user demand (Hobday, 1998; Prencipe, 2000), they must be able to modify and redesign a part if the interaction with the remaining body of the system raises problems during final commissioning and testing. In other words, system integrators must maintain a diversified technological base (even if not a diversified productive base) and outstanding capabilities in organization and coordination. Another analysis (Dosi et al., 2002), based on patents as indicators of technological competences, shows that integrators are characterized by a technological diversification that is greater than their productive diversification, because they must maintain research capabilities to monitor and evaluate external know-how and productive inputs. The analysis of patents also shows that diversification is greater when the complexity of products increases (Patel and Pavitt, 1997). The aeronautical industry proves to be one of the most diversified sectors, as only 8.3 per cent of the patents belong to the macro class ‘Transports’, including the micro class ‘Aeronautics’, while a substantial number of patents (48.5 per cent) belong to the class of ‘Non-electrical machinery’ and ‘Electronics’ (31.2 per cent). In other sectors, such as chemistry or ICT, the main share of patents (70–80 per cent) belong to the same main class. Detailed studies of engine manufacturers (Prencipe, 2000) stress that they play the role of technological trajectory shapers because of the key function of the technology they are working on. A common characteristic of aerospace companies is that the technological competences of firms span a wide number of patent classes, and the technological profile is persistent in time since there is no trend to increase the technological specialization, for example by abandoning patents or leaving less attractive research areas. These specificities of the aerospace sector, particularly with regard to research and innovation, suggest that both horizontal and vertical technological coordination is absolutely vital among companies involved in developing and using new complex products. The sector shows the presence of

Technology platform

207

a high level of complementarities among pieces of knowledge dispersed across different companies. The average life of development projects is so long that the absence of some form of preliminary coordination would mean a strong decrease in innovation incentives. The ‘dual’ nature of the sector and its strategic value imply an above average involvement of the public sector. The research activities conducted in this technology domain are highly diversified and build on innovation outputs from a number of other sectors (material sciences, engineering, energy, telecommunications, semiconductors and so on). All these facets of the aerospace sector make it a natural candidate for a technology platform approach. The ACARE technology platform The Advisory Council for Aeronautics Research in Europe (ACARE) can be considered a flagship initiative in the ETP development process. In 2001, the main aerospace stakeholders across Europe were brought together by the European Commission, which started the discussion about future technology in the aeronautics sector by publishing ‘European Aeronautics: a Vision 2020’. The broad aeronautics sector has a long-term oriented and concentrated group of leading players, and easily converged into a single ETP – ACARE – with about 40 members, representing the complexity of the sector due to their very different knowledge backgrounds and roles. Table 6.1 reports the taxonomy of the statutory members elected by the ACARE stakeholders and appointed to represent them in its governance. The high-level involvement of all the relevant aerospace stakeholders is evident from the variety of the organizations from which they are selected and from the profile requested of the representative of each category. The policy-making level is represented at both the national and the European level in order to take into account the R&D investment plans of individual countries and align them with the European Community guidelines. Stakeholders include universities, manufacturers, final users, infrastructure managers and technology-related regulation authorities, and are in charge of defining and suggesting technology evolution and investment priorities. Moreover, permanent effort is devoted to making ACARE participation and outcomes available to SMEs and to a broader public by specific involvement actions as well as communication initiatives. ACARE has a light structure, basically relying on a consulting board and two co-chairmen. A Strategic Research Agenda was initially released in October 2002 (SRA-1) and fully revised and published in October 2004 (SRA-2). It is worth analysing the process of identification and synthesis that was crucial to reaching an agreement on future R&D investment lines.

208

Table 6.1

Evaluation and performance measurement of R&D

Composition of the ACARE technology platform

Stakeholder

Nomination

Profile of the representative

Governmental Institutions

Member States

Director in charge of national research programs in favour of aeronautics industry Director in charge of research programs related to aeronautics industry and air transport

European Commission

Authorities

Regulators EUROCONTROL

Director in charge of research program Director in charge of research program

Firms

Manufacturing industry

Director in charge of research and / or strategy (for smaller companies this can be CEO level)

Users

Airlines Airports

Director in charge of strategy Director in charge of strategy

Public research and education

Academia

Senior professor with wide international contacts

At the beginning of Vision 2020, the two fundamental objectives, namely ‘Meeting society’s needs’ and ‘Creating competitive leadership’, were stated to inspire the overall agenda setting. To meet those ambitious aims, five technical challenges and a cluster of goals for each were identified: quality and affordability, environment, safety, air transport system efficiency, and security. During the work towards the second edition of the SRA, the ‘high level target concept’ was introduced to take into account the trade-off between the goals of the previously identified technical challenges and to link specific technological trajectories to the aviation sector in which they could be fully developed, and applied. Moreover, a detailed taxonomy of technology areas and domains was developed, to break down the wide spectrum of relevant technologies involved in the expected future of the sector and to assess them individually in terms of impact intensity and timing. Table 6.2 reports the analysis of an aeronautical engine technology to be developed in order to achieve the environmental targets defined by ACARE, and shows an example of the details that could be found for any single technology in one of the ten areas covered by the SRA.

Technology platform

Table 6.2

Example of SRA technology analysis

Technology Top level objectives ●

Variable pitch for fan blades to achieve high thrust at low speeds

209

●

●

Meet society’s needs Creating competitive leadership

Technical challenges ●

●

●

To reduce fuel consumption and CO2 emissions by 50% To reduce perceived external noise by 50% To reduce NOx by 80%

High level target concept ●

Ultra green air transport system

Area and domain ● ●

Propulsion Performance

Kind and time of impact ●

●

●

Pace technology Mediumhigh impact level 2015–20 impact time

As a direct result of ACARE activities, a number of member state mirror groups, or ‘national ACARE’, have emerged to favour the participation of national players and the representation of their interests by developing joint positions among national stakeholders. These groups exert a direct influence on the national level programs by helping their integration in the European support strategy. The chief outcome of ACARE, by far, is the aeronautical Joint Technology Initiative (JTI) ‘Clean Sky’, projected to be one of the largest European research projects, with an estimated budget of €1.6 billion shared equally by the European Commission and the aerospace industrial players over the period 2008–13. This R&D program, adopted by the Commission on 13 June 2007, comes from the principles described in Vision 2020 in January 2001, first reported in the technical challenge ‘environment’ of the SRA-1, and then developed into the two ‘ultra green’ and ‘highly cost efficient’ high level target concepts of the SRA-2, published in October 2004. In the Clean Sky program the ACARE targets are conveyed in six Integrated Technology Demonstrators (ITD), as shown in Figure 6.2, led by chief industrial players and funded with specific shares of the total programme budget. The aim is to produce industrial prototypes, drawing on technology developed according to the Strategic Research Agenda. Organized as a public–private partnership, Clean Sky is expected to speed up technological breakthrough developments and shorten the time to market for new solutions tested on full scale demonstrators. This very detailed R&D initiative has defined a budget involving 50 per cent

210

Evaluation and performance measurement of R&D

SMART Fixed Wing (24%)

• Airbus (F, D, UK, E) + SAAB (SE)

Green Regional Aircraft (11%)

• Alenia Aeronautica (I) + EADS CASA (ES)

Green Rotorcraft (10%)

• Agusta Westland (I, UK) + Eurocopter (F, D)

Green Engines (27%)

• Rolls-Royce (UK) + Safran (F)

Systems for Green Operation (19%)

• Thales (F) + Liebherr (D)

Ecodesign (7%)

• Dassault Aviation (F) + Fraunhofer Gesellschaft (D)

Figure 6.2

Information about the Integrated Technology Demonstrators (ITD)

of private expenditure, a set of priorities ranked by proper share of the overall budget and directly linked to demonstrators’ production by ITD leaders, a medium-term research agenda, and a participation enlargement program to involve the SMEs operating in the aerospace supply chain. Including a starting base of 54 companies, 15 universities and 17 research centres, Clean Sky is the operative outcome of ACARE and represents an R&D and technological innovation program that cannot be ignored by any of the stakeholders in the European aerospace sector. Its role in shaping competition by addressing knowledge production is larger than any effort that could be set up by an individual firm and underlines the importance of taking into account system initiatives that structurally involve broad and heterogeneous players clustered around a technological hub. 6.3.3

Technology Platform Evaluation

Having illustrated, with the most notable examples, the operational application of the technology platform concept, we now turn to a further

Technology platform

211

critical issue in the analysis, namely the implication for R&D evaluation when technology platforms represent the key unit of analysis in the assessment process. In fact, both when technology platforms are used as policy tools and when they are emergent properties of spontaneous coordination efforts among different and heterogeneous players, we need to switch from more traditional units of analysis (such as companies, industries or geographical entities), to a new one, which embodies a new definition of both the perimeter and the inherent properties and objectives that need to be evaluated. There is one main reason why the technology platform is a very different unit of analysis from traditional ones. The latter are not deliberately created to perform R&D processes, their definition is conventional and independent from the specific problem of improving, shaping or directing R&D efforts. In contrast, technology platforms are defined and created with the explicit aim to improve the R&D performance of different stakeholders in a selected knowledge and technological domain. This implies that in the traditional case we have to evaluate the R&D performance of a set of players, however defined, whereas with the technology platform the problem lies in evaluating the effectiveness and efficiency of the coordination mechanism that is set up to improve the R&D performance of a group of stakeholders combining to address one or more shared R&D and innovation targets. The coordination mechanism is in practice defined by a group of focusing devices that help relevant stakeholders to keep their R&D agendas consistent and coordinated. The overall question is then: how can the coordination mechanism performance be evaluated? The evaluation must take into account the R&D performance of those that are within the perimeter of the unit of analysis and the performance of the unit of analysis as a whole. Furthermore, the fact that the platform is both a unit of analysis and a coordination mechanism entails that, when the relevant output of performance measurement is defined, it is necessary to take into account the primary objectives of the unit of analysis itself. These arguments imply that there are two relevant dimensions in the evaluation of the platform as a coordination mechanism (see Figure 6.3). Evaluation at these two levels can be carried out using measurement approaches that look at the proficiency with which core R&D processes are carried out (Szakonyi, 1994b; Chiesa, 1996b). As illustrated in Figure 6.3, there is a macro process whereby R&D performed by a set of players is focused and directed toward selected and specific objectives and is organized around such objectives in an open and cooperative way. Furthermore, the functioning of the platform itself, as a coordination mechanism, is to be evaluated on the basis of its efficiency in transforming efforts of the players into focusing devices that are in turn used to support the macro process. This is what we call the micro process.

212

Macro level

Micro level

Figure 6.3

Evaluation and performance measurement of R&D

Input

Process

Output

• R&D efforts • Propensity to open, out of boundaries R&D

• Technological platform

• Open, collaborative organization of research activities

Input

Process

Output

• Resources allocated to coordination efforts

• Coordination activities

• Focusing devices

The technology platform levels of analysis

At the macro level, it is necessary to evaluate the performance of the platform on the basis of its role as a coordination mechanism. This means that the platform itself is evaluated according to its capacity to achieve its primary objectives, that is, to allow a set of players and relevant stakeholders to organize and openly and cooperatively orchestrate their research activities. Figure 6.3, using the traditional input–process–output framework, illustrates that the platform is the process component that uses a set of inputs (internal organization of R&D, stakeholders’ agendas, knowledge base, capabilities, research efforts and so on) to obtain a new organization of R&D activities, based on external cooperation involving the complex system of stakeholders that are tied in by the focusing devices produced by the platform. Under this perspective, the platform is the key driver of a process that transforms R&D input into innovation output for a set of players that is itself defined by a coordination and involvement mechanism (the platform). Table 6.3 illustrates a framework that seizes the performance of the platform at a macro level, with reference to input, process and output.

Technology platform

Table 6.3

213

The evaluation framework of the platform as a focusing device – macro level

Input Internal R&D activities

Process The platform

Output Cooperative and open research

Resources allocated to R&D activities Propensity to open, out of boundaries R&D

Focusing devices

Shared proprietary knowledge Research joint ventures Joint R&D investments Contract research Products

The definition of the measures is inspired by the notion that the platform is expected to produce two relevant results, namely a strong concentration of effort toward specific objectives and a substantial reorganization of the R&D activities that players perform in order to achieve those objectives. This reorganization is characterized by the attempt to take relevant parts of R&D activities out of the boundaries of the firm and connect them with other relevant stakeholders. Therefore, the measures of input and output (reported in Table 6.4) are defined in order to assess, on a quantitative basis, the change in the level of connectedness of relevant players’ R&D activities, and the extent to which the effect of focused, application-driven cooperation is able to increase innovation performance. On a micro level, it is necessary to evaluate the platform strictly according to its performance in producing the focusing devices that are needed to coordinate the efforts of the players in the macro process. Always making reference to the input–process–output framework (Table 6.5), it is possible to argue that the platform is the process that uses coordination resources of the players and of the agent in charge of the governance of the platform, and produces focusing devices that generate coordinated and joint research efforts, together with a new model of external research, in the macro process. As far as the micro process is involved, the operational definition of the indicators that could be used to evaluate the performance of the platform in producing focusing devices can only be based on a mixed portfolio of qualitative and quantitative measures, the latter being often categorical and discrete rather than continuous. As illustrated in Table 6.6, it is possible to define one main input, four indicators for process and one metric for output. As far as this last is concerned, the evaluation has to be focused on the enumeration and qualitative evaluation of the focusing devices produced by the platform. Process indicators are instead a set of

214

Table 6.4

Input

Evaluation and performance measurement of R&D

Definition of indicators for the macro process Input/process/output

Definition

Resources allocated to R&D

R&D expenses related to the applications emerging from the platform as defined by focusing devices Amount of co-authorships, co-patenting, co-licensing, contract research, joint ventures, consortia

Propensity to open, out of boundaries R&D Process

Focusing devices

Qualitative evaluation of items listed in Table 6.5

Output

Shared knowledge base

Co-authorship of scientific papers, technical reports and documents Co-application for IPRs, cross-licensing, patent pools, disclosure agreements Number and value of new joint ventures and research consortia created and managed Co-funding of specific research projects Value of contracted research among the partners of the platforms New products brought to market related to focusing devices

Shared proprietary knowledge Research joint ventures and consortia Joint R&D investments Contract research Product

Table 6.5

The evaluation framework of the platform as a focusing device – micro level

Input Coordination resources

Process Coordination activities

Output Focusing devices

Resources allocated for coordination

Involvement of players Equilibrium Openness Engagement Timing

Strategic research agenda Joint technological initiative High level target concept Technological challenge Demonstrators

quantitative parameters to define, per se, if the coordination mechanism is properly designed and managed, in terms of representativeness of the players involved, openness to additional players, efficiency in the use of resources, and efficacy in terms of timing and respect of milestones.

Technology platform

Table 6.6

215

Definition of indicators for the micro process Input/process/output

Definition

Input

Resources allocated for coordination

Time and related value allocated by stakeholders. Tangible and intangible assets allocated by stakeholders (patents, prototype drawings, feasibility studies…).

Process

Involvement of players

Share and quality of relevant stakeholders actively involved in the process. Share of segments in the value chain that are significantly represented by stakeholders within the platform. Effective management of the addition of new relevant stakeholders throughout the definition of the focusing devices. Compliance to time schedule and targets.

Equilibrium

Openness

Timing Output

6.4

Focusing devices

Qualitative evaluation of the items listed in Table 6.5

CONCLUSIONS

This chapter has investigated the rationales underlying the need to adopt a broader perspective in R&D evaluation, which also includes subjects external to the boundaries of the firm. In particular, it has been claimed that in industries characterized by elevated product complexity, a systemic approach appears to be a key element in the definition of effective firm level R&D strategies. A number of theoretical and empirical contributions from both the economic and the managerial literature have highlighted the value of the external environment in supporting firm level R&D and innovation capabilities. However, until recently these studies have mainly focused on the geographical dimension and on the localized knowledge spillover paradigm as drivers of such external support. The chapter has shown that the new economic and technological context calls for a vision of systemic interactions that are only partially represented by the traditional clustering approaches. It highlights two classes of drivers – knowledge sourcing and inter-dependent development – that stimulate the interactions of the

216

Evaluation and performance measurement of R&D

companies with external subjects in their process of technological change and new product development. Specifically, the external acquisition of knowledge turns out to be a fundamental component of firms’ innovative conduct in response to problems linked to path dependence and rigidities in internal routines. From a managerial perspective, the literature increasingly stresses the relevance of an open approach to innovation, in which external subjects significantly contribute to the design of research trajectories within the company. Given these new challenges for companies involved in R&D activities, it is crucial to identify an appropriate framework to seize complex innovation systems that shape the technological landscapes by involving a variety of different and scattered players. This chapter has presented a specific phenomenon, the European technology platform, which explicitly tries to offer a virtual environment to coordinate R&D activities and favour the emergence of shared technological trajectories. However, the achievement of this task is difficult and depends on a platform setting that is appropriate in terms of composition, openness and transparency. As discussed above, these dimensions should be conceived as the key parameters for evaluating the platform. Moving to firm-level analysis, it has been shown how the technology platform can be a useful source of information in the construction of a portfolio of R&D investments for several reasons: first, it is expected to provide insights about future technology evolution in the sector, allowing a more informed assessment of the inherent level of risk of R&D projects; second, it offers the opportunity to improve the timing of research projects; third, it should guarantee a more efficient alignment of public research efforts and private investments; finally, it provides a vertical coordination mechanism along hi-tech supply chains. It is worth pointing out that technology platforms represent an innovative example of a systemic network of firms. The fact that the network is not built on the basis of simple sector affiliations, positioning in the value chain or geographical localization, but rather on a transversal link to a technology, significantly enhances its innovation potential. The technology platform should favour the emergence of latent R&D and innovation potential and new technological opportunities, thanks to the interactions among players confronted with the need to set up common research priorities. Under this perspective, it is notable that in most cases of technology platforms the public funding of the research initiatives, when provided, will not exceed 50 per cent, which suggests the potential sustainability of this type of network. From an academic research perspective, technology platforms are a completely new field of investigation. The lack of an established body

Technology platform

217

of knowledge about them, due to their very recent rise, offers a variety of interesting research questions. Among others, the role of institutions in technology platform governance and funding, the effects of strategic research agenda and joint research project disclosure on stock markets, and the development of quantitative evaluation metrics for projects including technology platform outcomes are promising lines of inquiry to be pursued in the future, when a broad empirical base in terms of historical series and number of cases is available.

7.

R&D policy

7.1

INTRODUCTION

The centrality of R&D activities for socio-economic development is widely acknowledged. Their contribution to the development of social welfare through new and improved knowledge, products and services is unquestionable; furthermore science and technology are regarded as important determinants of economic growth. Economic growth is studied analytically using a production function that represents the relationships between the economic output and the factors of production. Whilst in the past the factors of production were almost exclusively the stock of capital and the stock of labour, more recent economic theories have acknowledged the importance of other factors and among these the centrality of R&D activities. In particular Solow (1957) showed that almost 90 per cent of the growth in the US economy in the first half of the last century could not be explained by the growth in capital and labour, and speculated that the changes could reflect technology advance over time. Nowadays both economic theories based on more sophisticated versions of the classic production function (for example Nelson and Phelps, 1966) and theories that emphasize the influence on growth of other factors, that are not directly specified in an expanded version of the classic equation (for example the so-called ‘new growth’ theory of Romer, 1986, 1994), acknowledge the centrality of investments in R&D and technological change. As stated by Audretsch et al. (2002, p. 166) ‘regardless of whether one adheres to the more narrow old theories or the broader new theories, the evidence is overwhelming that technology drives economic growth’. Given the social and economic benefits associated to R&D activities, policies to foster and sustain these activities have always been present, albeit in different ways, from the patronage of the past to the comprehensive programs and subsidies of our times. On the other hand, even though these policies are ever more important in the new ‘knowledge based economy’, many questions remain about policy scope and rationale (necessity and objectives), typologies, tools and results (efficacy and efficiency). The evaluation of R&D policies is extremely challenging because of their multiple goals, the indirect and complex linkages between policy outputs and outcomes, and the serendipitous and long-term nature of 218

R&D policy

219

policy impacts (Feller, 2002). Furthermore, as explained in the following, the evaluation of policies is difficult due to the changed nature and understanding of R&D activities and processes. This chapter discusses the evaluation of R&D policies, presenting and reviewing the literature on the topic and introducing an original framework to organize the complexity of the theme and to give suggestions for future research. A theoretical approach to the evaluation of R&D policies is adopted, starting with definitions and concluding with quantitative figures of the different streams of literatures. The main research questions addressed in this chapter are: 1. 2.

Is there consensus on what is the value of R&D policies, and how to evaluate them? What are the main methodological and theoretical issues in evaluating the results of these R&D policies?

It is shown that, despite the importance of R&D activities, the evidence on the value of R&D policies is limited, and the theoretical discussion about the necessity of these policies and their scope is still open. The chapter is organized in four sections. In the next section the theoretical discussion about the scope and rationale of R&D policies is presented. In Section 7.3 the debate regarding the different typologies and methodologies is summarized, and an original framework to organize and interpret the relevant literature is introduced. Section 7.4 analyses the main contributions following the categories of the framework. In the last section conclusions are drawn on the results of the study and its implications for the future.

7.2 7.2.1

SCOPE AND RATIONALE OF R&D POLICIES R&D Definitions

The first step in the evaluation of policy interventions is the analysis of the objectives and scope of the policies themselves (policy theory). As a matter of fact, a policy can be evaluated only by confronting it with its objectives, that is, evaluating if and how its objectives have been met. This section reviews and analyses the literature regarding the opportunity and the importance of R&D policies. After this theoretical evaluation of the need for introducing this typology of policies, the evaluation of specific and different policy measures will be discussed. For the evaluation of R&D policies this step is especially important,

220

Evaluation and performance measurement of R&D

given the difficulties in defining these policies and in distinguishing them from other similar and in some cases complementary policies. First of all it is useful to note that R&D policies are different from more general policies like industrial policies, economic growth policies, innovation policies and in many respects technology policies. In particular technology policies sometimes include R&D policies, but are also aimed, for instance, at boosting the overall competitiveness of firms, their internationalization and the diffusion of technologies, or at regulating firms’ concentration, standards, and so on. Second, it is necessary to understand the concepts underlying the term R&D. The need for common definitions of research and development activities has been widely recognized, and from 1964 such definitions have been provided by the Organisation for Economic Co-operation and Development (OECD) through the ‘Proposed Standard Practice for Surveys of Research and Development’, better known as the ‘Frascati Manual’. In particular these definitions have been proposed for statistical reasons and they are periodically updated to give national statistical agencies a common framework for R&D activities data. The 2002 version of these international standards reports the following definitions: Research and experimental development (R&D) comprises creative work undertaken on a systematic basis in order to increase the stock of knowledge, including knowledge of man, culture and society, and the use of this stock of knowledge to devise new applications. (p.30) Basic research is experimental or theoretical work undertaken primarily to acquire new knowledge of the underlying foundations of phenomena and observable facts, without any particular application or use in view. (p.77) Applied research is also original investigation undertaken in order to acquire new knowledge. It is, however, directed primarily towards a specific practical aim or objective. (p.78) Experimental development is systematic work, drawing on knowledge gained from research and practical experience, that is directed to producing new materials, products and devices; to installing new processes, systems and services; or to improving substantially those already produced or installed. (p.79)

These definitions, as discussed below, have some limitations, but they constitute ‘the internationally accepted definitions of R&D’, and more generally the Frascati Manual ‘has become a standard for R&D surveys worldwide’ (OECD, 2002, p.3). Furthermore the distinction between research activities undertaken to ‘increase the stock of knowledge’ and research activities undertaken to use ‘this stock of knowledge to devise new applications’ is an important starting point for evaluating the need

R&D policy

221

for R&D policies. In the following, a review of the literature on these evaluations is provided, using this approach to organize contributions from economics of science and innovation, finance and R&D management literature. 7.2.2

Policies for Research ‘To Increase the Stock of Knowledge’

The need for policies to foster and protect research activities that generate new knowledge stems from the characteristics of knowledge itself, as pointed out in the seminal contributions of Nelson (1959) and Arrow (1962). In particular the two properties of knowledge that are pivotal in economics terms are its non-excludable and non-rival nature, as noted in Chapters 5 and 6 of this book. Knowledge is non-excludable because it is extremely difficult and costly to control. It can flow easily through different channels, such as people, communication systems and networks, and it can also be acquired by analysing products that contain or are based on it. The difficulty for firms of keeping research results secret is well known and the importance of this spillover effect has been empirically tested (for example, Mansfield, 1985). Knowledge is thus rarely fully appropriable by those who generated it and invested in its production: others that have not contributed to its production can use it and benefit from its use. ‘Positive externality’ is the economic term for this positive impact on others without an economic compensation. Positive externalities are common also for other goods but they are normally hindered by the limited availability of the resource. Knowledge, on the contrary, is also non-rival, that is, it can be utilized by multiple users at the same time and also at no additional cost (the marginal cost for using knowledge is zero because no additional copies are needed and knowledge is not consumed by use). These two properties define knowledge as a public good, a good for which there is a difference between the private and the social return: the subjects that create new knowledge generate social benefits that are not compensated by privately appropriable benefits. This difference can be very significant given the non-rival nature of knowledge: the recipients can extend it both geographically, in space, and historically, over time, and the initial knowledge can also multiply and generate new knowledge because in many cases it is also cumulative (Foray, 2004). There is in this case a ‘problem of public goods’ (Pigou, 1932), that is, a failure of the market because private incentive does not make it possible to achieve a social optimum: private underinvestment in research that generates new knowledge is likely to occur because the private sector is unable to appropriate and profit adequately from its production. To cope with this

222

Evaluation and performance measurement of R&D

problem a policy intervention is needed. First, governments can substitute the private actors with public actors (for example, universities): society pays for the costs of generating new knowledge and society is the recipient of this new knowledge (which therefore should not be appropriated by private agents). Second, governments can create a market for knowledge by means of intellectual property rights, that is, granting the exclusive use of knowledge for a limited period of time to those who have made a new discovery (Nordhaus, 1969). Finally, public financing can bridge the gap between private investment and the social optimum. All these policies have limitations. In general terms the presence of positive externalities and their relevance in the case of knowledge is, as the word suggests, positive. ‘Knowledge commons’ are not subject to the classic tragedy of the commons (Hardin, 1968): they are not exhaustible resources (for example, a pasture) subject to destruction by unregulated exploitation. Knowledge is continuously expanding and growing, it is not consumed by use and it benefits from diffusion. It is indeed this potential that constitutes one of the main reasons behind the importance of knowledge for economic growth. In order to fully exploit this potential, given the fact that its marginal cost is zero, knowledge should be a free good and its diffusion should be fostered in order to reduce duplications and sub-optimal agents, and in order to maximize the probability of new discoveries and inventions (David and Foray, 1995). This latter point introduces the ‘knowledge dilemma’: on the one hand the use of knowledge should be rapid and complete, and thus free, on the other hand its production can be extremely expensive (Foray, 2004). In particular, this dilemma is relevant for intellectual property rights policies, which introduce monopolies of knowledge. Similarly public financing policies can alter market efficiency and knowledge dynamics. Finally, if governments substitute private actors with public actors such as universities, problems of management and control of these organizations arise. It is important to consider the case in which governments become the main actors in the production of new knowledge through their own organizations, or the main funders of private initiatives such as private universities. Indeed, despite the difficulties and the problems implied in the creation of a publicly supported enterprise for the development of new knowledge, it is exactly in the case of basic research or, more generally, in the case of science, that the public nature of the knowledge good is more relevant. A publicly supported science system is thus in place in almost every country. In his seminal work, Nelson (1959) explicitly focused on basic research, and both he and Arrow (1962) considered knowledge as information. More recently it has been underlined that knowledge is not completely a

R&D policy

223

public good, in particular because it is to some degrees tacit. Information is codified knowledge but knowledge can also be tacit, that is, as conceptualized and summarized by Polanyi (1966, p. 30), ‘we can know more than we can tell’. In the context of innovation studies, Reed and DeFillippi (1990) define tacitness as the implicit and noncodifiable accumulation of skills that result from learning by doing. Nonaka (1994) argues that tacit knowledge cannot be easily communicated and shared and is highly personal; Mowery and Rosenberg (1989) underline that tacit knowledge is not easily replicable and transferable. Knowledge is costly to transfer from one site to another in useable form or, as Von Hippel put it, knowledge is sticky (Von Hippel, 1998). Whilst the result of basic research (scientific knowledge) is more easy to articulate and codify, technological knowledge always comprises a tacit component, because it is often hard to translate fully into exact norms or theories. Technological knowledge has an ‘essentially pragmatic nature’ and it ‘is also accumulated through experience in production and use on what has come to be known as learning-by-doing and learning-by-using’ (Pavitt, 1987, p. 9). The rationale for policies for basic research stems from the fact that, as previously discussed, scientific knowledge can be more easily seen as a public good, but also from other characteristics of this good. In particular, as highlighted again by Nelson (1959) in his influential article, first there is an extremely relevant uncertainty concerning the outcomes of the scientific activities and, as Arrow (1962) further emphasizes, it involves ‘nonprobabilisable’ risk, that is, complete Knightian uncertainty (there is no known probability distribution over the arrival of results). Second, there are quite often very long lags between the scientific results and their practical implications. Third, serendipity is very common in research activities: search might lead to results and applications far from expected ex ante. 7.2.3

Policies for Research ‘to Devise New Applications’

Policies for basic research and scientific activities, as previously seen, have a strong rationale. Some authors question the value of these policies (as discussed in the concluding section), but the majority agree that pure research should be funded by government as a public good that is unlikely to find alternative support. In contrast, as Stokes (1997, p. 237) notes, many believe that ‘research and development activities that are nearer to the market should be funded by private firms applying a market test’. However, as previously noted, a large share of knowledge and in particular technological knowledge is not easily translated in the form of codified instructions or information: ‘it is tacit and naturally excludable, which sharply reduces the dimension of externalities’ (Foray, 2004, p. 23).

224

Evaluation and performance measurement of R&D

The rationale for supporting these activities, besides the arguments presented in the previous section, can also be found in the financial (especially the entrepreneurial) literature, and in particular in the works that highlight the presence of market failure related to the financing of R&D activities. Even if a firm is convinced of the potential benefit of an R&D investment, and of the fact that it will be able to appropriate (almost) exclusively these benefits, it may face financing constraints that prevent it from undertaking the actual investment. The firm can try to access the financial market (including banks, venture capital, and so on) in order to find the required resources but, due to information asymmetries, may fail to obtain them. In these cases viable and useful projects are not undertaken, and a market failure arises. Information asymmetries between the firm and potential financiers are assumed to be particularly important in R&D activities (Alam and Walton, 1995; Hubbard, 1998) and have been analysed in depth in Chapter 5 of this book. First of all there is a general problem in valuing the information, summarized in the ‘Arrow paradox’ (or ‘disclosure paradox’): if potential buyers don’t know the content of the information, they cannot appreciate its value, but if they know it, they no longer need to buy it (Arrow, 1962). More precisely, information asymmetries can limit the access to resources for R&D projects through adverse selection and moral hazard problems. Adverse selection arises from the difficulties in the evaluation of R&D projects’ potential, in particular a ‘market for lemons’ (Akerlof, 1970) can develop in which only the worst projects are available for receiving financial support and the rate of return asked by the financiers is high. Indeed, the financiers are not able to assess the value of the R&D project correctly, so they assume an average value and impose a higher cost of capital, and at the same time the entrepreneurs with projects above average try to finance the projects themselves or decide not to undertake them. ‘Moral hazard problems essentially arise because it is impossible to write complete contracts’ and, with incomplete contracts, entrepreneurs may have incentives to expropriate wealth from the financiers (Huyghebaert and Van De Gucht, 2007, p. 109). Asymmetric information causes financial constraints in both equity and debt financial markets. Stiglitz and Weiss (1981) have shown the rise in interest rates in the debt market; furthermore the pecking order theory introduced by Donaldson (1961) and developed by Myers and Majluf (1984) has shown that these problems also affect the capital market. The difficulties in financing R&D projects are even greater for new firms and in particular new high-tech firms that lack tangible assets that may be pledged as collateral (Berger and Udell, 1998), or track records to signal their current and future capabilities to potential financiers (Ho and Wong, 2007). The solutions proposed to cope with these information

R&D policy

225

asymmetries are signalling and financial intermediation, but both may fail to eliminate financial constraints (Takalo and Tanayama, 2008). In particular, screening activities of financial intermediaries may be inefficient (Chan, Greenbaum and Thakor, 1986) and entrepreneurs may be reluctant to share the information regarding their promising R&D efforts to avoid expropriation risk. As far as these issues are concerned, see again Chapter 5. Given these considerations, the opportunity for R&D policies to support research activities ‘to devise new applications’ clearly emerges. Furthermore, Tassey (1997) underlines other characteristics of this research phase that suggest a policy intervention. First, he notes that technologies have complex life-cycles and that in the early phases of technology development firms often encounter considerable technical and market risks that are beyond their capabilities. Second, he notes that the efficient development of technology by industry requires a complex set of supporting technical infrastructures and that sophisticated infrastructure support is also needed for the equally sophisticated products and services involved in the transactions in the technology-based markets. Third, it has been suggested that the market structure can reduce expected rewards from technology investment. In particular, given the fact that an increasing number of technologies have to interact with others, achieving compatibility or interoperability can result in significant additional costs, especially when competing private interests attempt to provide this type of infrastructure independently. Finally, it is important to highlight that firms’ R&D efforts are useful for them even when they do not lead to significant results and new technologies, because they contribute to the capability to absorb technology from external sources and to catch up with new scientific results, new technologies and innovations (Cohen and Levinthal, 1989). 7.2.4

The New Understanding of Research Activities and Processes

So far, this evaluation of the scope and rationale of R&D policies has followed the standard definition of R&D. This approach has allowed us to highlight contributions from different sources and streams of the literature and in particular to go beyond the view based only on the identification of knowledge as a public good: adopting this perspective, it has been possible to underline the rationale for the support of applied research and development activities that is normally overlooked. As Tassey (1999, p. 2) notes, ‘most science and technology analyses gloss over the economics of R&D investment and the associated market failure mechanisms, jumping to a set of poorly defined and supported policy recommendations’.

226

Evaluation and performance measurement of R&D

On the other hand the dichotomy introduced between basic research and applied research by the OECD standard definitions and the subsequent linear model of the innovation process have been widely rejected and are increasingly proved as inadequate to explain the complex processes involving R&D and innovation today. The two extreme typologies of research activity are still acknowledged to be widely present, so the rationales previously introduced continue to hold their importance. However, new insights regarding the innovation process have emerged and a third typology of research activity has been described, allowing for a better understanding of R&D policy rationale. A new taxonomy of research activities The idea that understanding (basic research) and use (applied research) are conflicting goals, and that mixing them is a problem because it limits the fundamental activities of ‘pure’ research is not new and can be traced back in history. On the other hand, although a great deal of research is wholly guided by the goal of either understanding or use, many other research efforts are directed at these goals, as can be proved by looking at the history of science. For example, as Stokes (1997) has highlighted, in the 19th century Louis Pasteur had the applied goals of preventing rabies in animals and humans, spoilage in milk, wine and vinegar, anthrax in sheep and cattle, and so on, but at the same time he studied the underlying phenomena at the most fundamental level. In doing so, he developed a framework for understanding a whole new class of natural phenomena (microbiology) and demolished the medieval doctrine of the spontaneous generation of life. Other notable examples are the physics of William Thomson (Lord Kelvin), inspired by the needs of industry and his country, and the work of Giulio Natta in collaboration with Montecatini S.p.A., which led to the discovery of new polymers and earned the scientist a Nobel Prize. Today whole industries are deeply rooted in this kind of research activity, notably the biological, biotechnology, nanoscience and nanotechnology fields. In these fields ‘both the founders and the undertakers of research often have well in mind the possible social and economic payoffs from what they are doing’ (Nelson, 2004, p. 456). The limit of the basic vs. applied research dichotomy is acknowledged by many authors (for example, Brooks, 1967; Comroe and Dripps, 1976; Martin and Irvine, 1983). Stokes (1997) has proposed a new taxonomy that introduces the new category of ‘use-inspired basic research’ besides the two established categories. This new category has been debated in the various versions of the OECD manuals and, though never fully adopted in the standards, it is considered to some extent in the 2002 document: ‘Oriented basic research is carried

R&D policy

227

out with the expectation that it will produce a broad base of knowledge likely to form the basis of the solution to recognized or expected, current or future problems or possibilities’ (OECD, 2002, p. 72). The non-linear view of the innovation process The linear model of R&D and innovation activities has been clearly rejected as well, although it is still widely diffused and no generally accepted alternatives have emerged. Three main flaws are associated with the linear model. First, only in a few cases can basic research be seen as a curiosity-driven, separate generator of scientific discoveries that are then converted into new products and processes by applied research and development in subsequent stages. In many cases the scientific and practical efforts are simultaneous or parallel. Second, even though technology is increasingly science based, many technological innovations proceed today without the stimulus of advances in science, but because of stimuli from, for instance, the market. Third, science is not exogenous to technology, either in terms of instrumentation, or (increasingly) in terms of objects of investigation. If technology is increasingly science-based it can also be said that in many cases science is technology-based. As Nelson (2004, p. 459) notes ‘in many cases, advances in practice come first and lead to efforts to understand scientifically’, and Rosenberg (1996) has argued that many of the most challenging puzzles science has had to face have been made visible by, or have been created by, new technologies. As previously noted, despite the widespread criticism of the linear model, no generally accepted models have emerged to replace it. On the other hand many theories agree on the presence of dual upwards, interactive but only partially coupled, semiautonomous trajectories of fundamental scientific knowledge and technological know-how. As science can be seen as evolving in the context of ‘scientific paradigms’ (Kuhn, 1962) or research programs (Lakatos, 1970), technology can in parallel be seen as having discontinuities or continuous changes along ‘technological trajectories’, that is, in the context of a ‘technological paradigm’ (Dosi, 1982). Especially in the innovation and R&D management literature different models have emerged, adopting a firm’s perspective of the balance between technology (science) push factors and market (or demand) pull stimuli. Among these are the chain-linked model proposed by Kline and Rosenberg (1986), the coupling model proposed by Rothwell and Zegveld (1985) and the interactive model proposed by Roberts (1988). All these frameworks acknowledge the presence of a sequential process with feedback loops, and conceive the process of innovation as ‘the confluence of technological capabilities and market needs within the framework of the innovating firm’ (Rothwell and Zegveld, 1985, p.12).

228

Evaluation and performance measurement of R&D

Potential consequences in terms of R&D policy scope and rationale The presence and the spread of ‘use-inspired basic research’ constitute an opportunity and thus can be seen as a potential rationale for R&D policies. It is possible to support oriented research that could have both theoretical (advancements in knowledge) and practical (advancements in technologies and innovations) results. The increasing number of oriented or, as they are sometimes called, ‘strategic’ research programs, for instance on biotechnology and nanotechnology, is a clear example of the importance already given to this rationale. The proximity of the scientific research results to the application context, especially in some fields, has amplified the attention towards a subset of R&D policies generally referred to as knowledge transfer policies. On the one hand the rationale of these policies is to favour the diffusion of scientific knowledge to society (diffusion of the public good leveraging on its characteristics and externalities). On the other hand, the oriented nature of some pieces of research undertaken in the scientific context and their technological components, imply a more significant departure from the characteristics of a public good and in particular result in more tacit and ‘sticky’ knowledge. In these cases, as Eisenberg (1987) noted, publication of results is not the same as making the discovery a public good, and additional political initiatives to foster knowledge transfer may be needed. Thus one of the most convincing rationales for political support of public research spin-off firms is the difficulty associated with the diffusion of knowledge, scientific results and know-how that are deeply embedded in the scientists that have developed the research, and so are hard to transfer. Similarly, Rosenberg (1990) has underlined the incentives that firms have to invest in science, showing that in many cases they hire Ph.D.s and allow their researchers considerable leeway in the allocation of their time and publication of their results. These investments are useful for firms because they can lead (maybe in serendipitous, unexpected ways) to radical innovations, but also because they have a motivational impact on researchers and play a role in keeping the firm up to date (absorptive capacities). In particular, Rosenberg (1990) highlights the value of the firm’s scientists in explaining scientific advances relevant to the firm, and, we might add, in transferring knowledge from public research organizations. Issues related to the management and evaluation of human resources (HR) in R&D have been addressed in Chapter 3 of this book. Just as, in the scientific context, there are many cases in which the knowledge generated departs from the characteristics of a public good, so, in the industrial context, in many cases the knowledge generated has pronounced features of such goods. Firms experience significant externalities and spillovers that are useful for society; public

R&D policy

229

policies can thus be directed to sustain these activities and avoid underinvestment in them. As Tassey underlines, ‘generic technologies are like the results of basic research, they have an important enabling role for development of other technologies’ (Tassey, 1997, p. 33). Following this latter principle, many ‘strategic’ research programs favour or require the simultaneous presence of both scientific and industrial partners in the financed projects. Finally, the non-linearity of the innovation process and the complexity of the relationships between science, technology and market, further strengthen the need for policies to favour and support the synthesis and pooling of the complementary efforts of the different actors involved. In recent years, the concepts of national, regional and sectoral innovation systems (Freeman, 1987; OECD, 1997; Cooke et al., 1997; Breschi and Malerba, 1997) have been proposed, and some economists have adopted an evolutionary perspective (Nelson and Winter, 1982) that emphasizes the importance of these complementarities. Furthermore, as summarized by Chesbrough (2003c) with the concept of open innovation, firms are increasingly shifting their attention from R&D activities undertaken inside their organizations to the knowledge and the technologies developed outside their boundaries, and the need has clearly emerged for policies to favour networking activities and horizontal and vertical collaborations (not only with the scientific institutions). The impacts of this new awareness about R&D and innovation processes and the different typologies of research on the rationale for R&D policies are still debated. The difficulty of providing an organic systematization to the new evidence regarding the innovation system is believed to be one of the main causes of the limits of R&D policy evaluation literature and hence one of the most promising avenues for its future development.

7.3

TYPOLOGIES AND METHODOLOGIES OF EVALUATION FOR SPECIFIC R&D POLICIES

After the discussion of the rationale and scope of R&D policies, it is interesting to deepen the analysis of the literature on the evaluation of R&D policies and, in particular, those contributions analysing the efficacy and the efficiency of policy interventions and the best methodologies to assess them. A systematic analysis of the existing scientific literature has therefore been performed and brought to the development of an original framework of analysis. The next subsection provides methodological details about this investigation.

230

7.3.1

Evaluation and performance measurement of R&D

The Process of Data Collection and Analysis

The breadth of the literature regarding the evaluation of R&D policies required some choice of focus in order to obtain a consistent and significant sample of contributions and themes. First of all, the literature review is focused on R&D policies. Therefore, contributions regarding the evaluation of R&D activities performed inside the firm are not considered here. They are instead the subject of other chapters of the book. Similarly, studies concerning the evaluation of R&D activities performed inside universities and public research centres and, more generally, the evaluation of these organizations, are not included in the analysis. Furthermore, anticipating the many possible typologies of R&D policy evaluations to be discussed, the analysis has been limited to ‘ex post’ policy evaluations. R&D policy evaluations for some authors (for example, Meyer-Krahmer, 1995) can be performed before the policy is deployed (ex ante), during its implementation (on going) or afterwards (ex post). In this review the ‘on going’ and the ‘ex-ante’ evaluations are only marginally included because they can be regarded respectively as ‘monitoring activities’ and ‘decision making activities’. The analysis was primarily based on a sample of articles published on the most relevant journals, those included in the Web of Science (WOS) online database of Thomson Scientific. The search for relevant articles on this database was made through a careful analysis based on complex queries and on verifications on the basis of the citations contained in the contributions identified. The search has considered the period 1990–2007, and included only journal articles (not proceedings), and only articles written in English. It has led to the identification of 242 contributions. These come from 70 journals but many articles are concentrated in a limited number of journals that are pivotal for these themes. More precisely Research Policy with 47 articles accounts for 19 per cent of the total, and is followed by Scientometrics (12 per cent), Technovation (10 per cent), International Journal of Technology Management (7 per cent), Research Evaluation (6 per cent), R&D Management (5 per cent) and Technological Forecasting and Social Change (3 per cent). 7.3.2

Typologies and Methodologies: A Review of the Main Contributions

The literature regarding the evaluation of R&D policies has grown in parallel with this policy area. In particular the debate started to be more intense during the 1970s in the USA (Jacobs, 1998; Jang, 2000). Since the 1990s several contributions have introduced and discussed a

R&D policy

231

number of taxonomies of the methodologies that can be applied to evaluate different types of R&D policies. The main contributions belonging to this body of literature, which is mostly theoretical rather than empirical in intent, are briefly presented here. Meyer-Krahmer (1995), studying technology policies in Germany, describes different typologies distinguishing between ex ante (strategic and operational), real time/ongoing monitoring and ex post evaluations. The author underlines the importance of ex post evaluations, that, as previously noted, are the focus of this literature review, and highlights that the aims of these evaluations are strategic, operative and administrative efficiency, description of the results, analysis of causality and analysis of further need for support. Furthermore the article draws up a list of different analyses and data that can be part of R&D policy evaluations and underlines that different combinations of typologies, analysis and data can be made in specific R&D policy evaluations. Shapira, Youtie and Roessner (1996), reviewing the evaluation methodologies both at a national and at a federal level in the USA, underline that, while concern about the evaluation of programs has grown, ‘it has also become clear that developing robust evaluation approaches (as in most other fields of public policy) is not an easy task’ (p. 186). According to the authors, the first challenge for the evaluation is the presence of very diverse subjects in terms of objectives, nature, instruments and results. As a consequence, they recommend a strong interaction with the evaluated subjects and, at the same time, they suggest that the evaluators should try to make tangible the effects of the policies and that they should carefully design the evaluation level (industrial sectors, scientific fields, typologies of organizations, and so on). In terms of evaluation methodologies the article reflects on the opportunity of using simple data or indicators and introduces six typologies of evaluation: program activity monitoring; customer valuation; external reviews, audits and consultant studies; economic and regional impacts; comparative evaluation and control groups; and assessment of practice and tools. Youtie, Bozeman and Shapira (1999), analysing an important regional program in the USA, review different evaluation methodologies summarized in eleven categories: case studies; peer review; content analysis; surveys; bibliometric; cost/benefit (C/B) analysis and return on investment (ROI); benchmarking; input-output(I-O) analysis; systems and flow analysis; performance indicators; and diffusion and network studies. According to the authors, in order to have accurate and repeatable methodologies, the choice of the appropriate methods must be made on the basis of the parameters that can be measured and the resources available for the evaluation. The classification is based on the distinction between routine

232

Evaluation and performance measurement of R&D

methods (performance indicators, flows analysis, and so on) and comprehensive methods (C/B analysis, case studies, content analysis, and so on). Georghiou and Roessner (2000) review the different policies and the related evaluation methodologies and observe that broad social and political trends influence the practice of evaluation. In this article, the policy classification is based on four categories: evaluation of the socio-economic impacts of research in universities and public laboratories; evaluation of linkages; evaluation of collaborative R&D; and evaluation of diffusion and extension programs. The authors conclude by noting that ‘evaluation work has had less of an impact in the literature than it deserves’ (p. 674). Bobe (1991) reviews the main evaluation methodologies adopted in Europe by the European Commission starting from the 1976 Milan conference, in which the theme was discussed for the first time. The premises of this work are that evaluation is a necessary tool in policy design processes and that it must integrate the scientific (results in terms of research), managerial (monitoring of public programs) and economic (business performances) aspects. In the opinion of the author, the European Commission has focused its attention from the start on the integration of policy and evaluation, including indicators of the expected results before policy implementation in order to be able to verify these indicators ex-post. The article suggests the use of diachronic (pre/post) methods through not only the beneficiaries but also the opinion of experts in direct (peer review) or indirect (bibliometric) ways. Furthermore it is argued that the assessments must necessarily involve the stakeholders because there are too many important elements that might otherwise not be detected. In the context of the debate regarding the European programs, the contributions of Georghiou and Metcalfe (1993) and Luukkonen (1998) should be mentioned as well. The first notes that ‘the key elements of an evaluation in this area are firstly those concerning the movement of knowledge’ (p. 169), whereas the second underlines the difficulties in assessing the impact of R&D policies and states that ‘more qualitative and longitudinal studies ought to be carried out’ (p. 599). Both articles note the importance of being aware of the theoretical background in which the policies are developed and of the objectives of the evaluation. Cozzens (2000) analyses the US case with particular attention to the criticality of the institutional framework and the strong heterogeneity of the subjects involved in R&D policies. The paper underlines the difficulties of the evaluation and remarks that ‘various new audiences for assessment are asking for different, sometimes contradictory, measures of effectiveness’ (p. 5). Given this complexity, the author recommends the choice of methods that are simple, clear, objective and aware of their limitations, abandoning the search for extremely complicated mechanisms.

R&D policy

233

Roessner (2002) reviews the different evaluation methodologies in the USA at a federal level and underlines that the ‘most important and most difficult problem is deciding what to measure, not how to measure it’ (p. 89). He remarks the importance in the USA of peer reviewing methodologies, which are also explored in Cozzens (1995). Melkers and Roessner (1997) compare evaluation methodologies in the USA and in Canada. According to the authors’ analysis, Canada has a long tradition of centralized evaluation, and the USA, after many years of fragmented and decentralized evaluations, has chosen the same path, although differences still remain between the two. In this article the comparison is not only at an organizational level, but also focuses on legislative and institutional aspects of evaluation and of its implementation. The authors point out the procedural aspects of the evaluation, focusing on peer reviewing, customer satisfaction and, in part, case studies. Smith (1995) reconstructs the way the Canadian federal government has chosen its methods of evaluation on the basis of the methodologies already used to evaluate firms’ projects. The methodologies used, according to the classification of the author, are peer reviewing, benchmarking (with a focus on managerial aspects), network analysis and bibliometrics. Smith states that, in order to choose a methodology, it is important to know whether the results will be used internally or externally, and he highlights the difficulties in evaluating the results of companies participating in R&D projects that are outside their core business activities. Langfeldt (2004) performs an in-depth analysis of six international evaluations of Norwegian programs with a case study methodology. In this article the focus is on expert panel evaluations and the author argues that these qualitative evaluations are not in contrast with more quantitative ones. In his opinion, for instance, panels have problems in terms of composition and organization and can benefit from the use of quantitative bibliometric indicators. Kostoff (1993) similarly analyses semiquantitative methodologies highlighting pros and cons of retrospective methods (historiographic analyses) and ‘accomplishment books’ methods. Brown, Curlee and Elliott (1995), considering a USA-specific program, analyse the problem of the proper identification of control groups and performance differentials between the beneficiaries of the policies and those without access to the financing schemes. Similarly focused on specific programs, respectively in the UK and in Japan, are the contributions of Cunion (1995) and Gonda and Kakizaki (1995), who focus their attention on the evaluation process and the required phases. Benzler and Wink (2003) identify some trends in the scientific debate and in particular the fact that the practice has evolved from a linear approach (before/after) to more complex assessments including interaction

234

Evaluation and performance measurement of R&D

of several factors (for example, interdependence between different actors, differentiation of objectives and rationality among the players, spatial dimension of the processes). In particular, Benzler and Wink classify the scientific literature into four major streams: process and elements of evaluations; evaluation and explanation of public and private funding schemes; evaluation in its territorial context; evaluation and consequences for policy design. As far as quantitative methodologies are concerned, the contribution of Cozzarin (2006) identifies a classification of statistical-econometric analyses: net present value; internal rate of return; benefit–cost analysis; microeconomic analysis; bibliometrics; econometric studies. Molas-Gallart and Davies (2006) reconstruct the different approaches to evaluation in the light of various theoretical concepts of innovation process, an approach that is in part followed also by Pilorget (1995). Kauko (1996) introduces the theoretical distinction between econometric studies and interview studies and underlines how the first should be accurately designed in order not to present wrong conclusions, for instance because of problems of endogeneity of the variables. Finally, Cozzens, Bobb and Bortagaray (2002) analyse the distributional consequences of science and technology programs that, in their opinion, have been neglected. In particular, the authors discuss how to design and evaluate S&T policies in order to reduce inequalities. 7.3.3

Typologies and Methodologies: An Original Framework

The debate on the types and methods of evaluation, as seen in the previous section, has many elements of interest and shows the importance of the issue for scholars and practitioners. In particular, it clearly shows that a widely acknowledged taxonomy of the methodologies that can be employed to evaluate different typologies of R&D policies has not emerged yet. The classifications used in the contributions reviewed are mainly specific to the individual authors or institutions involved in the policy evaluations and often were developed ad hoc for the policies evaluated, rather than being taxonomies subsequently adopted by a broader scientific community. The lack of a common framework and the lack of consensus on the classifications proposed is due, in particular, to the limits of the classifications themselves and to the fact that the authors often focused their attention on the implementation and description of policy evaluations and on the impact of the policies, rather than on the reconstruction of the scientific debate on these themes. In order to cope with the lack of acknowledged typologies and methodologies for R&D policy evaluation, this section develops an original

R&D policy

Table 7.1

235

R&D policy evaluation methodologies

Quantitative

Evaluation of measurable impacts (standard data) Evaluation of detectable effects (ad hoc data)

Qualitative

Benchmarking evaluation Case study evaluation

framework of classification that could be useful for policy makers and is able to support a comprehensive review of the empirical literature on the issue. This framework has been designed to satisfy three requirements that are often missing in other classifications but that are fundamental in this kind of effort. First, the framework has to be ‘theoretical’, that is, general enough to include all the different R&D policy evaluations without ambiguities and without losing some contributions. In other words, it should not be too specific or focused only on a particular set of typologies or methodologies of evaluation. Second, the framework has to be ‘functional’ to the needs of scholars, evaluators and policy makers. The classifications should thus be useful for organizing the subject and for identifying different typologies and methodological alternatives. Following this consideration the framework should also be simple enough to understand and to manage the complexity of the topic without adding another layer of difficulty. The framework developed on the basis of these requirements is composed of two parts, one related to the evaluation methodologies and the other concerning different typologies of evaluation. These two parts are briefly presented in the two following subsections. Evaluation methodologies The classification of the evaluation methodologies is based on the typology of information/data collected for the evaluations, starting with the classic distinction between quantitative and qualitative data. The resulting four evaluation methodologies are presented in Table 7.1 and described in the following. Their relevance and use is further discussed in the concluding section after the analysis of the main empirical contributions and examples. Evaluation of measurable impacts (standard data) These quantitative evaluations are based on statistical and/or standard data such as turnover, patents, publications, and so on, on indicators built around these data and on statistical and econometric analysis. As a result, these methodologies take into account only the tangible and measurable outputs (impacts) of R&D policies.

236

Evaluation and performance measurement of R&D

Evaluation of detectable effects (ad hoc data) These quantitative evaluations are based on surveys that collect ad hoc information regarding the policy effects involving the subjects targeted by the policy. The data collected can be results, performances and so on, or subjective information and opinions (for example usefulness of the policy, impact of the policy on processes and organization, and so on). The collected information is normally used in statistical and econometric analysis. Benchmarking evaluation These evaluations are qualitative and comparative, that is, they compare different policies or similar policies in different contexts such as regions, nations and so on, collecting for each of them qualitative and quantitative information following a common evaluating scheme. Information is collected both directly from the subjects targeted by the policies or the promoter of the policies, and indirectly using already available data. Information is related to both the results and the performances of the policies and to their objectives and characteristics. The analyses are mainly qualitative, often based on expert reviews and opinions, and aimed at highlighting the pros and cons of the different policies in terms of results, efficiency, organizational processes, and so on. Case study evaluation These evaluations are similar to benchmarking methods in terms of data collected and typology of analysis, but are focused on the evaluation of a single policy. The policy is not evaluated in comparison with other policies but its characteristics and results are deeply explored with both quantitative and qualitative data collected ad hoc through the involvement of multiple subjects. In some cases the evaluators can also be involved in the policy process (action research). Evaluation typologies To organize the literature on R&D policy evaluation, besides the classification of the different methodologies, it is also useful to identify different typologies of evaluation in which these methodologies can be applied. As previously highlighted, different typologies have been proposed, but none of them has been widely accepted. In order to complete the framework to review the literature, two main variables of classification have been identified that, as explained in the following, can be used to distinguish between R&D policy evaluations (Table 7.2). The classification of the evaluation typologies is based on the different policies that can be evaluated, distinguishing between the ‘objectives’ and the ‘instruments’ of these policies. The first variable considers the objective of the policies, that is, the organizations or the processes targeted. R&D policies can be aimed at

R&D policy

Table 7.2

237

R&D policy evaluation typologies Policy instrument Financial

Policy objective

Legislative

Firms Knowledge generating institutions Networking

improving the R&D activities of different subjects. They can be directed towards firms, acknowledging the centrality of innovation for these organizations, or towards the knowledge generating institutions (universities and public research centres), considering their role for the advancement of science and technology. Finally, policies can be implemented to foster collaborations between different organizations (networking), in particular between firms and knowledge generating institutions (knowledge or technology transfer), and indirectly through the promotion of new intermediary organizations such as joint laboratories, consortia and science and technology parks. The second variable takes into account the instruments used by the policies under evaluation. R&D policies can be based on financial instruments or legislative instruments. Financial instruments are aimed at sustaining R&D investments through fiscal reductions (automatic or related to specific actions), grants, loans, call for projects, and so on. Legislative instruments are mainly aimed at fostering R&D investments, networking and organizational and managerial changes, modifying the context and the norms the firms should conform to. The instruments variable in the taxonomy is simple and general enough to include all the different policies but, at the same time, it provides an interesting distinction between the policies that have to be evaluated. The first variable (objectives) is consistent with the discussion in the first section and in particular it avoids basing the identification of the policy typologies on the ambiguous distinction between basic research and applied research. At the same time, the distinction between knowledge generating institutions and firms is consistent with the different aims and values of these organizations, as underlined by many authors (for example, Dasgupta and David, 1994). The two selected variables lead to six typologies of R&D policy evaluation that are used in the following sections to organize the comprehensive review of the empirical literature. In particular, the next section provides an overview of the main articles that have applied a specific methodology

238

Evaluation and performance measurement of R&D

(Table 7.1) to evaluate actual R&D policies with a certain objective and employing a given instrument (Table 7.2). This review of the empirical literature on R&D policy evaluation should be helpful to scholars. It also shows the usefulness of the taxonomy developed in the chapter and its capability to support future research.

7.4

SPECIFIC R&D POLICY EVALUATIONS

The main contributions regarding the evaluation of specific R&D policies are presented here on the basis of the framework introduced in the previous section. Articles on the value of R&D policies or on the different typologies and methodologies of R&D policy evaluations have been already discussed and are not re-examined. 7.4.1

R&D Policies for Firms

Financial R&D policies Financial policies for firms are the most common form of R&D policies, due to their simple implementation. On the other hand, their evaluation is not straightforward. As summarized by Hall and Van Reenen (2000) in their review of fiscal incentives, two main approaches can be identified. The first analyses ‘whether the level of the good supplied after the implementation of the policy is such that the social return is equal to the social cost’ (p. 456), whilst the second compares the amount of incremental R&D with the loss in tax revenue. Evaluation of measurable impacts (standard data) The majority of the contributions compare the financial aid received by the firms with selected outcomes. Two emblematic cases are Darby, Zucker and Wang (2004), who correlate the funds obtained through the Advanced Technology Program (ATP) in the USA with the number of patents, and Zhu, Xu and Lundin (2006) who evaluate the relationship between the funds from the Chinese government and the investments in R&D. In some cases, for instance in the work of Justman and Zuscovitch (2002) in Israel, the evaluations compare the effects on different industries. In other cases, attention has been given to the recipients’ profiles. For example Blanes and Busom (2004) analyse the structural characteristics of the companies subsidized by Spanish policies, checking in particular the social capital and the human resources of the firms involved. Similarly, Lerner (1999) and Wallsten (2000) consider the characteristics of the firms that participated in the SBIR (Small Business Innovation Research) program in the

R&D policy

239

USA and their results in terms of turnovers, patents and employment. In these cases, the authors also compare the results of firms that received the financial aid with firms that did not. Some evaluations have been explicitly designed to identify, besides the policy effects, the most promising areas of intervention for R&D financial incentives, distinguishing between different research areas and industries. The ‘mapping measurement impact’ methodology of the UK Department of Trade and Industry presented by Bowns et al. (2003), is one example. An issue frequently highlighted for these policies is freeriding behaviour, that is, the problem of a substitute effect. Some empirical works (for example Heijs, 2003) show that firms are using public funds to substitute their own private investments, while other studies (for example Hall and Van Reenen, 2000) find that public aid generates additional private investments in R&D. In this stream of literature focused on analysing the additionality of the public subsidies, Hujer and Radic (2005) and Czarnitzki and Licht (2006) look at R&D policies in West and East Germany. The first is based on participating and non-participating firms, whilst the second compares the firms of the two different geographical areas and their patenting activities. Other examples are Gorg and Strobl (2007), who focus on R&D policies in Ireland, and the analysis of five international policies performed by Klette, Moen and Griliches (2000), who highlight the difficulty of identifying counterfactuals to compare the outcomes. A review study of the evidence on this topic is described in Bloom, Griffith and Van Reenen (2002), which focuses on OECD countries over a 19-year period (1979–97). The authors find evidence that tax incentives are effective in increasing R&D intensity and in particular that a 10 per cent fall in the cost of R&D stimulates approximately a 10 per cent rise in R&D in the long-run. Other works regarding the elasticity of R&D financial policies have been conducted considering different time frames and geographical areas and have obtained similar results (for example Russo, 2004; Dahlby, 2005; McKenzie, 2005 in Canada; Koga, 2003 in Japan, considering also the differences in terms of firms’ size). Besides the additionality issue, some authors, such as Jaffe (2002), show that reliable measurements of financial policies’ effectiveness are hampered by a selectivity problem, that is, the fact that funds are normally assigned to proposals judged in advance to be likely to succeed. Other authors, for example Mamuneas and Nadiri (1996) studying US manufacturing industries, underline the importance of combining both publicly financed R&D investments, which could crowd out private investment, and tax credits, which could have a significant impact on these privately financed R&D investments. Similarly, Wu (2005) verifies that, besides the additionality

240

Evaluation and performance measurement of R&D

of R&D credit programs, state services in higher education and R&Dtargeted programs also matter in private decisions on R&D investment. A synthesis of the older literature on substitution and complementarity effects can be found in the work of David, Hall and Toole (2000). Evaluation of detectable effects (ad hoc data) These evaluations, using ad hoc data, are able to analyse the managerial aspects of R&D policies and also to highlight indirect effects. A first example is given in the work of Bach et al. (1995) which, through direct interviews, evaluates three EU R&D programmes. The firms analysed in this work highlight the outcomes of the projects in terms of patents, technologies, and so on, but also through secondary effects like the establishment of new collaborations and financial benefits not directly related to the R&D results. Other examples are the survey of Japanese high-technology start-ups undertaken by Koga (2005), with a focus on the additionality issue, and the work by Lillis et al. (2002) on the technological capabilities of firms that increased thanks to New Zealand government funds. Some evaluations, for instance the one by Archibald and Finifter (2003) regarding the NASA small business innovation research program, analyse the effects of changes in the supporting schemes (in this case in terms of selection criteria for funding), comparing the strategies of the firms before and after these changes. Similarly, Falk (2007) surveys Austrian firms to understand their possible reactions to different funding schemes and to the denial of public aid. In this case the evaluation is on hypothetical rather than actual policies. This methodology can also be used to perform comparisons between different industrial sectors, geographical areas and so on. For instance Schoening et al. (1998) compare policies of the US, UK, Taiwanese and South Korean governments, surveying firms that received public support in these countries. The authors consider many S&T policies in each country and evaluate their effectiveness considering especially the role of the government in providing strategic information and access to new technologies, and in reducing bureaucratic and legislative barriers. Benchmarking evaluation These evaluations are used to compare financial R&D policies, detailing the different mechanisms implemented and considering a wide set of issues. For instance Grupp (1993) analyses the government – direct or indirect – involvement in the system of research and development in the telecommunications field in ten countries. The author examines R&D expenditure, R&D personnel, scientific publications, patents and turnovers and adds to these indicators expert assess-

R&D policy

241

ments in order to qualitatively verify the effectiveness of the different systems. Another example is the work by Kastrinos (1995), which considers the role of additionality in the impact of European Commission R&D programs on firms, comparing case studies of financed projects and opinions of project leaders. This evaluation methodology is also used to underline the historical developments of policies. For instance the article by Leyden and Link (1993) provides an overview of the history of R&D-related tax policies both in the USA and in 22 other industrial countries, and reports the existing empirical evidence on their effectiveness. Case study evaluation Case studies are performed to obtain in-depth evaluations of single policies or a set of policies in specific contexts. Tsipouri (1991) relates the aid from EU structural funds to the R&D dynamics in Greece. Kaufmann and Todtling (2002) and Kaufmann and Wagner (2005) perform similar analyses in Austrian regions, exploring the regional strategies and the different political instruments introduced. With an historical perspective, Hines (1999) reconstructs R&D taxation in the USA, focusing in particular on the related import–export dynamics and on fiscal pressure. Cooper (2003), on the other hand, studies the development over time of the US Small Business Innovation Research (SBIR) program, together with all its political–institutional and legislative history. An original approach is explored by Van Den Ende, Wijnberg and Meijer (2001), who discuss EU and Dutch policies directed toward the IT sector from the 1960s, analysing their impact on the innovative capabilities of a single large firm (Philips). Legislative R&D policies The literature regarding legislative R&D policies for firms is scarce and mainly focused on intellectual property rights as one of the main instruments to foster R&D investments. Mann (2005) considers for instance the pros and cons of patents in the software industry, with a case study methodology on juridical aspects and the impact on financing. Lichtenberg and Philipson (2002) estimate the relative effect of betweenpatent and within-patent competition on innovative returns of researchbased pharmaceutical companies. The authors underline that previous works have emphasized that intellectual property regulations stimulate R&D by protecting innovative returns from imitators of the same product, but the effects of these regulations on between-patent competition by new patents has been ignored.

242

Evaluation and performance measurement of R&D

Mutti and Yeung (1996) study the protection of intellectual property in the United States. They do not directly evaluate the legislation, but analyse the features of the firms that benefit from it. This method allows them to evaluate, not the norms per se, but the recipients with respect to the general population potentially interested. Aoki and Hu (1999) evaluate the effect of the legal system on incentives to innovate by comparing the US and UK systems, with a focus on patent licensing and litigation behaviour. Kiyota and Okazaki (2005) discuss foreign technology acquisition policy and firm performance in Japan using statistical data of firms. They find that government regulation effectively constrained technology acquisition, whilst after deregulation in 1968 the number of acquired technologies sharply increased. Similarly, Hubner (1996) analyses decisions on innovation and diffusion and the limits of deregulation and privatization in several states, exploring the socio-political aspects and the relationship to economic theory. Finally, even though the number of contributions focused on legislative R&D policies is limited, it is worth noting that some articles regarding financial mechanisms, and especially those based on case study and benchmarking evaluations, partially address this issue. 7.4.2

R&D Policies for Knowledge Generating Institutions

Financial R&D policies The literature regarding financial R&D policies for knowledge generating institutions does not consider tax incentives (given the not-for-profit nature of these organizations), and is mainly based on bibliometric and peer reviewing methodologies. Evaluation of measurable impacts (standard data) The most common standard data are publications and the related indicators, such as citations and journal impact factors. For instance Lewison and Cunningham (1991) use bibliometric indicators to evaluate two European Community research programs, in biotechnology and environmental chemicals. Dalpé and Anderson (1995) analyse the implementation of strategic research programs and contracts on the development of solar and bioenergy research in Canada. The authors consider patents (together with publication data) as an indicator of more applied and technology-related activities and compare the results with funds that the researchers received in order to assess whether or not a reorientation occurred. Other similar examples are the evaluations reported in Van Leeuwen, Van Der Wurff and Van Raan (2001), who explore, in the Netherlands,

R&D policy

243

whether or not the public resources funded the ‘best’ research in their fields, and in Jiménez-Contreras et al. (2003), who analyse the impact of the establishment of a National Commission for Research Evaluation on the Spanish system. Similarly, Krauskopf (1992) evaluates the policies for the development of Chilean universities and Anduckia et al. (2000) verify the effects of publicly financed projects. Evaluation of detectable effects (ad hoc data) These evaluations are mainly based on peer reviewing processes and, in a limited number of cases, on surveys directed to researchers and managers of knowledge generating institutions. The methodologies implemented are therefore standard across fields and countries. A survey-based evaluation is provided by Luukkonen (1995), comparing the effect of a research policy initiative in four Nordic European countries. Kostoff (1994a, 1994b) describes the use of peer review for federal research impact evaluation in the USA, highlighting the different alternatives and the deficiencies of this approach for policy evaluation (reviewers’ biases, costs, limited number of specialized reviewers, and so on) and the use of ad hoc semiquantitative methods (look-back method and accomplishment documents approach) in the evaluations of other federal research programs of the USA. Guimaraes and Humann (1995) evaluate a wide investment program that started in Brazil in the 1960s and involved several institutions. The aim of the program was to support education and training of qualified personnel in the R&D field; the achievement of this goal is measured through biannual reports, interaction with all the actors involved, external evaluators and analysis of employment in R&D. Langfeldt (2004) describes an in-depth analysis of six evaluations of Norwegian research activities based on expert panels. The author compares two evaluations of research fields, two of research institutes and two program evaluations, and underlines the importance of organizational constraints and in particular of the time and resources available for the evaluations. Benchmarking and case study evaluation Only a limited number of evaluations of financial policies for knowledge generating institutions have adopted a qualitative approach. Chetal and Raj (1998) describe and compare the results of different funding policies in India, considering support for the development of R&D facilities, creation of employment through project posts, development of new technologies, quantity and quality of research publications and generation of doctoral theses.

244

Evaluation and performance measurement of R&D

Similarly, Markusova et al. (2004) give a descriptive overview of R&D conducted in Russia and analyse the impact of changes in the policies for R&D funding of universities. Legislative R&D policies There are a limited number of articles that evaluate legislative R&D policies for knowledge generating institutions. The evaluations have focused on policies directed at increasing the valorization of research results and the possibility of appropriating value through intellectual property rights. In particular, one of the most studied cases is the Bayh-Dole Act approved in the 1980 in the USA. Evaluation of measurable impacts (standard data) and of detectable effects (ad hoc data) A typical example of these evaluations is the work by Sampat, Mowery and Ziedonis (2003), which verifies changes in university patent quality after the Bayh-Dole Act using long time series of patent citations data. Shane (2004) also studies this policy, focusing on its role in shifting the patenting behaviours of universities. The author, analysing licensing trends in different fields, observes that the Bayh-Dole Act provided incentives for universities to increase patenting in those fields in which licensing is an effective mechanism for acquiring new technical knowledge. Similarly, Iversen, Gulbrandsen and Klitkou (2007) evaluate the impact of academic patenting legislation in Norway, identifying and measuring the inventor activity of university researchers. A case of use of ad hoc data is provided by Chang et al. (2006), who survey Taiwanese higher education institutions (HEI) in order to assess the effects of the Science and Technology Basic Law, which, like the US act, allows HEIs to own the patents that arise from government research grants. Benchmarking and case study evaluation Legislative R&D policies have also been studied through single and comparative case studies. A typical example is Mowery et al. (2001), comparing the effects of the Bayh-Dole Act on three leading universities: California, Stanford, and Columbia. The evidence suggests that the Bayh-Dole was only one of several important factors behind the rise of university patenting and licensing activity, and that it had little effect on the content of academic research at these universities. Taking a longer historical perspective, Sampat (2006) analyses the debate on the Bayh-Dole Act with a review of changes in American universities’ patenting policies, procedures and practices throughout the 20th

R&D policy

245

century, an assessment of the rationale underlying the new legislation, and an overview of its effects on economic returns from university research. In the context of the European Union, articles by Conceiçao, Heitor and Oliveira (1998) and Verspagen (2006) analyse the specificities of the patenting legislations in this area, compare them with the choices of other non EU countries and discuss the possibility of introducing new legislative policies to foster public research and knowledge transfer. 7.4.3

R&D Policies for Networking

Financial R&D policies Financial R&D policies that aim at fostering collaborations between different organizations (networking), and in particular between firms and knowledge generating institutions (knowledge or technology transfer), have been widely diffused and evaluated. These policies can also operate indirectly through the promotion of new intermediary organizations such as joint laboratories, consortia and science and technology parks. Furthermore, they are very complex given the different objectives of the organizations involved. Evaluation of measurable impacts (standard data) These evaluations have been based especially on financial data (for example R&D spending), bibliometric data (publications and patents) and information regarding new products, innovations, and so on. For instance Irwin and Klenow (1996) evaluate US policies aimed at sustaining joint R&D consortia. In their analysis the authors verify the ‘commitment’ hypothesis that the consortia obligate the member firms to spend more on high-spillover R&D, and the ‘sharing’ hypothesis that the consortia reduce duplication of members’ R&D spending. Branstetter and Sakakibara (2002) examine the impact of Japanese government-sponsored research consortia on the research productivity of participating firms by measuring their patenting before, during, and after participation. They suggest that consortia are most effective when they focus on basic research and find consortium outcomes positively associated with the level of potential R&D spillovers within the consortium and (weakly) negatively associated with the degree of product market competition among consortium members. Similarly Watanabe, Kishioka and Nagamatsu (2004) examine the Japanese consortia in order to analyse the effects and limits of the role of the government in spurring technology spillover. European Union policy has been in favour of balanced regional growth and at the same time of strengthening the science and technology bases of

246

Evaluation and performance measurement of R&D

the member states. Some works have analysed the results of this policy and in particular of the EU framework programs in terms of collaborations. For instance Clarysse and Muldur (2001) use regional participation data to measure the direct impact of EU R&D policy on technology development, and collaboration data, by means of social network techniques, as an indicator of technology diffusion. Other studies of EU policy have been directed to evaluate more generally the impact on collaborations among the different organizations involved. Widhalm et al. (2001) focus the analysis on the patterns of cooperation and collaboration in research networks within the EU framework program, involving the industrial, research and education sectors of EU member states. The analysis of this paper is complemented by a graphical display of networks of partners based on bibliometric data. As underlined by the authors, ‘visualisation of networks indicates specific co-operation patterns beyond quantitative one-dimensional analysis and thus is a valuable method for characterisation of cooperational networks’ (p. 138). Granadino, Plaza and Vidal (2005) analyse the Spanish Joint Action Program aimed at promoting the mobility of scientists as well as cooperation with foreign research centres. The authors perform the evaluation by means of the scientific outputs resulting from the joint research projects and in particular co-authored articles. Evaluation of detectable effects (ad hoc data) These evaluations have been mainly directed at verifying the effects of networking R&D policies in terms of participating organizations, deepening the understanding of the managerial aspects of the collaborations promoted and financed. Peterson (1993) analyses the results of two surveys of firms and public research organizations in the context of the EU Eureka program. The author argues that firms are motivated to participate in the program more by its ability to encourage new collaborative links than by its provision of public funding; at the same time, it emerges that the program methodology causes problems in the management of projects. Bozeman (1994) evaluates the US policy aimed at improving technology transfer from government laboratories to industry on the basis of questionnaires mailed to laboratory directors. He underlines the difficulties in conceptualizing technology transfer effectiveness. In particular he adopts an out-the-door model, which asks whether the technology (or information) is transferred to another organization; but at the same time he highlights that, under this concept of effectiveness, ‘it is the transfer itself that is important, not the impact after the transfer’ (p. 323). Davenport, Davies and Grimes (1999) evaluate a program that supports

R&D policy

247

collaborative R&D projects between New Zealand firms and research institutions. The authors note that the mix of different organizational cultures can result in conflicting attitudes towards the management of the project and that this can provide a barrier to the establishment of trust between the partners. They thus evaluate the importance of collaborative policy instruments in establishing different levels of trust. Tripsas et al. (1995) evaluate policies aimed at R&D collaborations and, using data from an ad hoc survey, underline the need for government help in controlling opportunistic behaviour. Laredo (1998) evaluates the networks promoted by the European Union framework programme using mailed questionnaires sent to all the participants in France. He identifies three main network configurations and suggests changes in policy. Mauguin (1991) also evaluates European policies, and presents an analysis of financial and contractual data. Some works evaluate policies for the promotion of research-based spinoffs as a form of technology transfer. For example Heydebreck, Klofsten and Maier (2000) analyse how the Swedish Teknopol approach has satisfied new technology based firms’ need for innovation support services. A limited number of articles have also evaluated policies to foster the development of joint laboratories between public research organizations and firms. Gray and Steenhuis (2003) for instance evaluate a US program quantifying the benefits of participating in an industry–university research centre. The authors focus in particular on R&D cost avoidance, considering data collected from 18 industrial sponsors. Other evaluations have been aimed at analysing networking policies among public research centres. Rank and Williams (1999) evaluate the Canadian Networks of Centres of Excellence program, which links researchers at universities across the country to work on joint projects. The evaluation is based on a partial benefit–cost analysis using data obtained through interviews with the researchers. Benchmarking evaluation Qualitative methodologies of evaluation (benchmarking and case studies) are frequently used to evaluate networking policies; this is due mainly to the complexity of the policies under evaluation and to the different objectives they pursue. Kingsley, Bozeman and Coker (1996), comparing a set of case studies, evaluate technology transfer outcomes from government supported research, development and demonstration projects in the USA. The authors underline that this evaluation is complicated by the variety of paths through which technology can move from producer(s) to user(s). The paper by Storey and Tether (1998) provides a review of public policy measures implemented in EU countries to support new technology-based

248

Evaluation and performance measurement of R&D

firms (NTBFs). Among the policy areas examined are science parks, the supply of Ph.D.s in science and technology, and the relationships between NTBFs and universities/research institutions. Harding (2003) analyses technology transfer policies in nine advanced economies (including the USA, UK, Japan, France, Germany and Australia), comparing objectives and instruments implemented. Feller (1992) compares American state government policies for R&D, underlining that they reflect contrasting theories about the linkages among academic research, technological innovation, economic growth, and administrative practices. O’Gorman and Kautonen (2004) describe the evolution of two knowledge-based agglomerations in Ireland and in Finland, emphasizing the role of policy interventions in the process of agglomeration formation. Another example, in this case related to the policy for the development of science and technology parks, is provided by Chordà (1996), who compares a number of examples in France. Case study evaluation The evaluations performed through case studies have been able to investigate the institutional contexts in which the policies are implemented. Such evaluations have been applied also to the policies of emerging and developing countries. Kyriakou (1995) analyses the macroeconomic aspects of science and technology program evaluation with a focus on European Union policies and the relationships among the different actors. Kyrgiafini and Sefertzi (2003) evaluate the impact of European regional policies on the systems of innovation in Greece, and especially on technology transfer processes. Similarly, Grimes and Collins (2003) seek to uncover the role that Irish participation in the European Union Framework Programme has played in building Ireland’s knowledge economy through its promotion of research collaborations. Link and Finan (1997) evaluate the US National Cooperative Research Act with a detailed case study that documents the private returns to member companies involved in a specific collaborative research venture. Ahn (1995) presents and evaluates the Korean program aimed at fostering university research potentials and at building university–industry research relationships through the establishment of centres of excellence. Kim and Dahlman (1992) underline that technology policies on foreign technology transfer, technology diffusion, and R&D should change over time in response to the changing external environment. Furthermore, they claim the need for a balance among policies to promote the demand side of technology, the supply side of technology, and the provision of effective links between demand and supply sides.

R&D policy

249

Odagiri, Nakamura and Shibuya (1997) investigate the advantages and disadvantages of publicly supported government–industry research consortia as vehicles for conducting basic research, taking Japan’s Fifth Generation Computer Systems Project as a case. Ekboir (2003) examines the importance of networking policies in Brazil, highlighting the role of networks to develop and disseminate complex technologies. Legislative R&D policies There are just a few evaluations of legislative R&D policies fostering networking processes. This is mainly due to the fact that many policies that have included a change of the legislation have also considered financial incentives (and have been already presented in the previous sections). Given the complexity of these policies, the preferred methodology of evaluation is qualitative (benchmarking and case study evaluation) and normally focuses on the economic and historical contexts. For instance Hansen (1995) evaluates the Danish licensing scheme for the diffusion to firms of inventions developed by private inventors and public sector researchers. Tanaka (1995) discusses the role that advisory councils play in developing science and technology policies in Japan, exploring the technical nature of the policy-making process and reviewing the significance of the whole process. Jaffe and Lerner (2001) analyse patent policy and the collaboration of national technology laboratories in the USA. In particular they present the historical context that drove legislative changes and present the initiatives since 1980 to encourage patenting and technology transfer. The authors analyse empirical data that illustrate the success of these policies and the determinants of this success. Uzun (2006) evaluates science and technology policy in Turkey from 1983 to 2003; by using several indicators of science and technology, the author finds substantial developments and stronger cooperation between university and industry. Koh, Koh and Tschang (2005) analyse the policy role in promoting and supporting the development of science parks. In particular, they propose an analytical framework to examine them and explore the case of the Singapore science park as part of a national policy to promote R&D and collaborations. Similarly Appold (2004), using county-level data in the USA, analyses research parks and the location of industrial research laboratories, to assess the effectiveness of policy interventions. Quintas, Wield and Massey (1992) consider science parks as policy instruments and, drawing on empirical research in the UK, assess their

250

Evaluation and performance measurement of R&D

potential and actual role in linking academic research with industrial activity. Huang, Varum and Gouveia (2006) evaluate China’s reform of the science and technology sector inherited from the planned economy. The authors analyse R&D input and output data at the country aggregate and provincial level to disclose the impact of the drawn-out reform on the efficiency of the whole sector. The work of Hoekman, Maskus and Saggi (2005) is one of the few that analyse national and international policy options to encourage the international transfer of technology. The authors distinguish between four major channels of such transfer and develop a typology of countries and appropriate policy ‘rules of thumb’ for both national policy makers and multilateral rule making. They argue that the optimal policy mix varies across countries and recommend differentiation in the design and application of rules in trade agreements, as well as a more explicit focus on evaluation of the impacts of policies.

7.5

CONCLUSIONS

The review of the literature suggests many different conclusions and clues for further analysis. The contributions that were identified can be further re-organized and analysed in order to explore and understand the topic from alternative perspectives, for instance focusing on the specific data, indicators and information used for the evaluations, the different particular methodologies implemented, and so on. In this concluding section, attention is focused on a subset of these possible interpretations that seems to offer promising future developments for this topic. 7.5.1

Evaluation Typologies

First, from the synthesis of the evaluation typologies (Table 7.3) it emerges that the contributions are not equally distributed among the different evaluation typologies. In particular, firms have been the main target of articles on R&D policy evaluation, whilst only a limited number of papers evaluated R&D policies directed at knowledge generating institutions. The scarce attention to the latter policies is in part related to the limited number of financial and legislative initiatives directed towards these organizations and in part to the difficulties in evaluating policy interventions for them (as they have multiple aims and a public role). However the main reason is that many policies and evaluations for knowledge generating institutions

R&D policy

Table 7.3

251

Synthesis of the evaluation typologies – percentage of contributions for each typology Policy instrument

Policy objective

Firms Knowledge generating institutions Networking

Financial

Legislative

40 10

3 5

44 15

32 83

9 17

41

Note: Only articles with an empirical focus are considered in this table.

have been focused on networking and in particular knowledge and technology transfer. In light of the discussion developed in Section 7.2, this result is interesting because it shows that in the last years the focus has been on firms’ R&D and, for universities and public research organizations, on more applied research efforts or at least on research activities with potential interest for firms. The value of these policies should not be underestimated, but (again referring to the arguments in Section 7.2 and in particular to the role of the knowledge generating institutions in more basic and oriented research), we suggest that it would be useful to have more policies and evaluations regarding only these organizations and their primary role. Similarly, from the synthesis it emerges that the majority of the contributions have analysed financial policies whilst limited attention has been paid to the evaluation of legislative policies. More policies and evaluations of the latter would seem desirable, considering their importance in steering the innovation system towards more efficient managerial and organizational practices. It is also striking that the majority (60 per cent) of the articles evaluate policies that were developed at a national level. Few articles have looked at supranational policies (for example EU policies, policies coordinated between two or more nations, and so on) and even fewer have focused on policies at a regional level (including policies developed by countries in federal nations like the USA or Germany). In this case, although the difference in level has an impact (for example in terms of available resources, specialization, and so on), it seems that the methodologies and evaluation approaches are not significantly differentiated. Similarly the evaluations do not show relevant differences when they are focused on policies for specific industrial or technological sectors (for

252

Evaluation and performance measurement of R&D

example biotechnology, electronics, and so on) or on policies in different countries. Attention to these aspects has been very limited in the literature and few contributions have highlighted elements for comparison. It would be interesting to study further the differences between the policies and then the evaluations for specific sectors, especially as the strength of the links between science and technology varies across different fields (see Section 7.2). There is also a need for more attention to R&D policies in developing countries because literature has so far focused almost exclusively on OECD countries and partially on some economies in transition (for example India and Brazil). 7.5.2

Evaluation Methodologies

In terms of evaluation methodologies it emerges that the majority of the contributions have used quantitative data and, in particular, standard data. This is mainly because these evaluations are easier to implement due to availability of data, but also because of their objectivity and the possibility of repeating them over time. On the other hand, the analysis of the contributions highlights the pros and cons of the different methodologies (Table 7.4). In particular, ad hoc data collected through surveys can be useful for understanding the specificities of the policies, but costlier qualitative approaches can highlight the contextual factors in which policies were developed and the reasons behind the results of those policies, giving policy makers a more detailed picture and support for decisions regarding new interventions. In general it is clear that limited attention has been devoted to the decisional processes and objectives leading to specific policies, to their impact and to the influence of the different actors interested in the results of the policies. Ad hoc data and qualitative data have proved especially important in the discovery and study of secondary effects, that is, indirect effects of policies, such as the increase in the relational capital after a collaborative project, the impact of financial support on the strategies of the founded research group, and so on. Furthermore the analysis shows that many authors have used and agreed on the use of hybrid methodologies in order to overcome the shortcomings of single methods. In terms of methodologies, the analysis unearths a significant number of evaluations that consider the differences in the recipients of the policies before and after the interventions (longitudinal evaluations). These methodologies usefully highlight the differential impact of R&D policy, taking into account the initial state of the subjects addressed by the policy. On the other hand, they should be carefully designed to include control variables that could have influenced the results independently from the policies under evaluation.

R&D policy

Table 7.4

Synthesis of the evaluation methodologies Methodology

Interaction with the evaluated subjects

Quantitative Evaluation of No, external measurable evaluation impacts (standard data)

Qualitative

253

Costs Pros – time required

Cons

Low

Limited by the typology, quantity and quality of available data (proxies) Subjectivity of some information, not easy to repeat over time

Evaluation of detectable effects (ad hoc data)

Limited, questionnaires

Benchmarking evaluation

High, High interviews and expert opinions

Case study evaluation

Very high, detailed analysis

Objective analysis, allows analysis of wide policies and analysis on time series

Medium Almost objective analysis, allows user to obtain specific information and data

High

Allows user to study and compare the specificities of the policies and the organizational aspects Allows user to highlight the innovativeness of the polices and their direct and indirect effects, allows user to understand the role of contextual factors

Subjectivity of the information and of the analysis, very difficult to repeat over time Very high subjectivity, results are difficult to compare with other cases and over time

In contrast, only a limited number of contributions have considered control groups and ‘match and pair’ methodologies, that is, evaluations between subjects that have received the policy ‘treatment’ and subjects that have not. This method is among the most promising for the future development of R&D policy evaluation even if, in many cases, it is hard

254

Evaluation and performance measurement of R&D

to identify and involve subjects that are not targeted by the policy under evaluation. 7.5.3

Final Remarks

The analysis in this chapter shows that no widely acknowledged methodologies and processes have emerged in the field of R&D policy evaluation. Indeed, even for important and widely studied policies like the BayhDole Act in the USA, evaluations have been conducted with different strategies and have led in some cases to opposite conclusions. The field, despite the presence of a significant number of contributions, is still in its infancy, with the majority of the studies focused on policies that involved many subjects and resources, and that have been in place or have been replicated for many years. In the R&D policy evaluation debate, therefore, there is still the opportunity and the need for further research. In the time period examined, the mean number of yearly publications has remained more or less constant, and it has not been possible to identify a clear evolution of the debate, but policy makers and society in general are paying more attention to R&D policies and, in particular in a period of limited resources, there is a growing need for evaluation and control of the resource allocation process. Among the most interesting issues for future development, besides the ones discussed above, are: the rationale underlying the choice of the evaluators and their impact on the evaluations; the role of the objectives of the policy makers and of their expectations in terms of policy results; the measurements of the secondary or indirect effects of the policies; and, especially, the impact of the new understanding of the innovation process discussed in Section 7.2, on R&D policy evaluation at the level of the innovation system as a whole (on this last point see for instance the seminal article by Molas-Gallart and Davies, 2006). R&D policy evaluation is demanding, especially considering the difficulties in defining the different research activities, the limited usefulness of judging or formalizing the research processes, the long time lag between research activities and their tangible results, and so on. On the other hand it benefits now from the availability of large databases (for example in terms of publications and patents) and its instruments could significantly evolve in the near future. Enhancing the effectiveness of policy for R&D is pivotal given the presence of market failure and the importance of promoting technological innovation and the expansion of the endless frontier of science.

References Aaker, D.A. and T.T. Tyebjee (1978), ‘A model for the selection of interdependent R&D projects’, IEEE Transactions on Engineering Management, 25 (2), 30–6. Abernathy, W. and J. Utterback (1975), ‘A dynamic model of product and process innovation’, Omega, 3 (6), 639–56. Aboody, D. and B. Lev (1998), ‘The value-relevance of intangibles: the case of software capitalization’, Journal of Accounting Research, 36 (Supplement), 161–91. Aboody, D. and B. Lev (2000), ‘Information asymmetry, R&D, and insider gains’, Journal of Finance, 55 (6), 2747–66. Abrahams, T. and B.K. Sidhu (1998), ‘The role of R&D capitalisations in firm valuation and performance measurement’, Australian Journal of Management, 23 (2), 169–83. Acs, Z., D. Audretsch and M. Feldman (1994), ‘Research and development spillovers and recipient firm size’, Review of Economics and Statistics, 76 (2), 336–40. Adner, R. and D. Levinthal (2001), ‘Demand heterogeneity and technology evolution: implications for product and process innovation’, Management Science, 47 (5), 611–28. Ahmed, K. and H. Falk (2006), ‘The value relevance of management’s research and development reporting choice: evidence from Australia’, Journal of Accounting and Public Policy, 25 (3), 231–64. Ahn, S.I. (1995), ‘A new program in cooperative research between academia and industry in Korea, involving centers of excellence’, Technovation, 15 (4), 241–57. Ahuja, N. (2000), ‘Collaboration networks, structural holes, and innovation: a longitudinal study’, Administrative Science Quarterly, 45 (3), 425–55. Ahujia, G. (2000), ‘Collaboration networks, structural holes and innovation: a Longitudinal Study’, Administrative Science Quarterly, 45 (3), 425–55. Akerlof, G.A. (1970), ‘The market for lemons: quality uncertainty and the market mechanism’, Quarterly Journal of Economics, 85, 488–500. Al-Mazidi, S. and A.A. Ghosn (1997), ‘A management model for technology and R&D selection’, International Journal of Technology Management, 13 (5/6), 525–41. Alam, P. and K.S. Walton (1995), ‘Information asymmetry and valuation effects of debt financing’, The Financial Review, 30, 289–311. Alchian, A. (1963), ‘Reliability of progress curves in airfare production’, Econometrica, 31, 679–93. Allen, R.C. (1983), ‘Collective invention’, Journal of Economic Behaviour and Organization, 4 (1), 1–24. Allen, T.J. and R. Katz (1986), ‘The dual ladder: motivational solution or managerial delusion?’, R&D Management, 16 (2), 185. Allen, T.J. and R. Katz (1992), ‘Age, education, and the technical ladder’, IEEE Transactions on Engineering Management, 39 (3), 237–45. 255

256

Evaluation and performance measurement of R&D

Allen, T.J. and R. Katz (1995), ‘The Project-oriented engineer: a dilemma for human resource management’, R&D Management, 25 (2), 129–40. Alpert, M. (1992), ‘The care and feeding of engineers’, Fortune, September 21, 87–95. Amir, E. and B. Lev (1996), ‘Value relevance of non-financial information: the wireless communications industry’, Journal of Accounting and Economics, 22, 3–30. Amir, E., B. Lev and T. Sougiannis (1999), ‘What value analyst?’, Working paper, Columbia University. Amir, E., B. Lev and T. Sougiannis (2003), ‘Do financial analysts get intangibles?’, European Accounting Review, 12 (4), 635–59. Amir, E., Y. Guan and G. Livne (2007), ‘The association of R&D and capital expenditures with subsequent earnings variability’, Journal of Business Finance & Accounting, 34 (1–2), 222–46. Ancona, D. and D. Caldwell (1992), ‘Bridging the boundary: external activity and performance in organizational teams’, Administrative Science Quarterly, 37, 634–65. Anderson, R.E. (1993), ‘HRD’s role in concurrent engineering’, Training and Development, 47 (6), 49–54. Anderson, K. and R. McAdam (2005), ‘An empirical analysis of lead benchmarking and performance measurement: guidance for qualitative research’, International Journal of Quality & Reliability Management, 22 (4), 354–75. Andreou, S.A. (1990), ‘A capital budgeting model for product-mix flexibility’, Journal of Manufacturing and Operations Management, 3, 5–23. Anduckia, J.C., J. Gomez and Y.J. Gomez (2000), ‘Bibliometric output from Colombian researchers with approved projects by COLCIENCIAS between 1983 and 1994’, Scientometrics, 48 (1), 3–25. Aoki, R. and J.L. Hu (1999), ‘Licensing vs. the effect of the litigation: legal system on incentives to innovate’, Journal of Economics & Management Strategy, 8 (1), 133–60. Appold, S.J. (2004), ‘Research parks and the location of industrial research laboratories: an analysis of the effectiveness of a policy intervention’, Research Policy, 33 (2), 225–43. Archer, N. and F. Ghasemzadeh (1999), ‘An integrated framework for project portfolio selection’, International Journal of Project Management, 17 (4), 207–16. Archibald, R.B. and D.H. Finifter (2003), ‘Evaluating the NASA small business innovation research program: preliminary evidence of a trade-off between commercialization and basic research’, Research Policy, 32 (4), 605–19. Armbrecht, F.M.R. Jr., R.B. Chapas, C.C. Chappelow, G.F. Farris, P.N. Friga, C.A. Hartz, M.E. McIlvaine, S.R. Postle and G.E. Whitwell (2001), ‘Knowledge management in research and development’, Research-Technology Management, 44 (4), 28–48. Armstrong, M., A. Galli, W. Bailey and B. Couet (2004), ‘Incorporating technical uncertainty in real option valuation of oil projects’, Journal of Petroleum Science and Engineering, 44, 67–82. Arora, A., A. Fosfuri and A. Gambardella (2001), Markets for Technology, Cambridge, Massachusetts: The MIT Press. Arrow, K. (1962), ‘Economic welfare and the allocation of resources for invention’, in R.R. Nelson (ed.), The Rate and Direction of Inventive Activity, Princeton, NJ: Princeton University Press, pp. 609–25.

References

257

Aryee, S. and C.G. Leong (1991), ‘Career orientations and work outcomes among industrial R&D professionals’, Group and Organizational Studies, 16 (2), 193–205. Audretsch, D.B. and M.P. Feldman (2004), ‘Knowledge spillovers and the geography of innovation’, in J.V. Henderson and J.F. Thisse (eds), Handbook of Regional and Urban Economics, 4 (61), Amsterdam: Elsevier, pp. 2713–39. Audretsch, D.B., B. Bozeman, K.L. Combs, M. Feldman, A.N. Link, D.S. Siegel, P. Stephan, G. Tassey, C. Wessner (2002), ‘The economics of science and technology’, Journal of Technology Transfer, 27, 155–203. Azzone, G. and U. Bertelè (1998), Valutare l’innovazione, Milan: ETAS libri. Azzone, G. and A. Rangone (1996), ‘Measuring production competence: a fuzzy approach’, International Journal of Production Research, 34 (9), 2517–32. Baba, M., T. Alcordo, D. Britt, R. Champieux, J. Gluesing, K. Harris, W. McKether and H. Ratner (2001), ‘The development of globally distributed teams: an ecological approach’, Working Paper. Bach, L., N. CondeMolist, M.J. Ledoux, M. Matt and V. Schaeffer (1995), ‘Evaluation of the economic effects of Brite-Euram programmes on the European industry’, Scientometrics, 34 (3), 325–49. Bailyn, L. (1991), ‘The hybrid career: an exploratory study of career routes in R&D’, Journal of Engineering and Technology Management, 8, 1–14. Bajo, E., M. Bigelli and S. Sandri (1998), ‘The stock market reaction to investment decisions: evidence from Italy’, Journal of Management and Governance, 2 (1), 1–16 Baker, N. and J. Freeland (1975), ‘Recent advances in R&D benefit measurement and project selection methods’, Management Science, 21 (10), 1164–75. Balachandra, R., K.K. Brockhoff and A.W. Pearson (1996), ‘R&D project termination decision: processes, communication and personnel changes’, Journal of Product Innovation Management, 13 (3), 245–56. Barczak, G. and D. Wilemon (2003), ‘Team member experiences in new product development: views from trenches’, R&D Management, 33, 463–79. Barron, O., D. Byard, C. Kile and E. Riedl (2002), ‘High-technology intangibles and analysts’ forecasts’, Journal of Accounting Research, 40 (2), 289–312. Barth, M.E. (2000), ‘Valuation-based research implications for financial reporting and opportunities for future research’, Accounting and Finance, 40 (1), 7–31. Barth, M.E., W.H. Beaver and W.R. Landsman (2001a), ‘The relevance of the value relevance literature for financial accounting standard setting: another view’, Journal of Accounting and Economics, 31 (1–3), 77–104. Barth, M., R. Kasznik and M. McNichols (2001b), ‘Analyst coverage and intangible assets’, Journal of Accounting Research, 39 (1), 1–34. Bartlett, C.A. and S. Ghoshal (1989), Managing Across Borders, Boston, MA: Harvard Business School Press. Bartov, E., S. Goldberg and M. Kim (2005), ‘Comparative value relevance among German, U.S. and International Accounting Standards: A German stock market perspective’, Journal of Accounting, Auditing and Finance, 20 (2), 95–119. Baugh, S.G. and R.M. Roberts (1994), ‘Professional and organizational commitment among engineers: conflicting or complementing?’, IEEE Transactions on Engineering Management, 41(2), 108–14. Baum, J.A. and C. Oliver (1991), ‘Institutional linkages and organisational mortality’, Administrative Science Quarterly, 31, 187–218.

258

Evaluation and performance measurement of R&D

Baum, J.A., T. Calabrese and B.S. Silverman (2000), ‘Don’t go it alone: alliance network composition and startups’ performance in Canadian biotechnology’, Strategic Management Journal, 21, 267–94. Bayus, B.L. (1994), ‘Are product life cycles really getting shorter?’, Journal of Product Innovation Management, 11 (4), 300–08. Bayus, B.L. (1998), ‘An analysis of product lifetimes in a technologically dynamic industry’, Management Science, 44 (6), 763–75. Beaver, W.H. (2002), ‘Perspective on recent capital market research’, The Accounting Review, 77 (2), 453–74. Bell, G.K. (1995), ‘Volatile exchange rate and the multinational firm: entry, exit and capacity options’, in L. Trigerogis (ed.), Real Options in Capital Investment: Models, Strategies and Applications, Westport, Connecticut: Prager. Bell, D.C. and A.W. Read (1970), ‘The application of a research project selection model’, R&D Management, 1, 35–42. Belton, V. and T. Gear (1982), ‘On a short-coming of Saaty’s method of analytic hierarchies’, Omega, 11 (3), 228–30. Ben-Zion, U. (1984), ‘The R&D and investment decision and its relationship to the firm’s market value: some preliminary results’, in Zvi Griliches (ed.), R&D, Patents, and Productivity, Chicago: The University of Chicago Press. Benaroch, M. (2001), ‘Option-based management of technology investment risk’, IEEE Transactions on Engineering Management, 48 (4), 428–44. Benner, M.J. (2007), ‘The incumbent discount: stock market categories and response to radical technological change’, Academy of Management Review, 32 (3), 703–20. Benzler, G. and R. Wink (2003), ‘Evaluating innovation processes: the political dimension’, International Journal of Technology Management, 26 (2–4), 121–30. Berger, A.N. and G.F. Udell (1998), ‘The economics of small business finance: the roles of private equity and debt markets in the financial growth cycle’, Journal of Banking and Finance, 22, 613–73. Beskese, A., C. Kahraman and Z. Irani (2004), ‘Quantification of flexibility in advanced manufacturing systems using fuzzy concept’, International Journal of Production Economics, 89 (1), 45–56. Black, F. and M. Scholes (1973), ‘The pricing of options and corporate liabilities’, Journal of Political Economy, 81, 637–54. Blanes, J.V. and I. Busom (2004), ‘Who participates in R&D subsidy programs? The case of Spanish manufacturing firms’, Research Policy, 33 (10), 1459–76. Bloch, C. (2008), ‘The market valuation of knowledge assets’, Economics of Innovation and New Technology, 17 (3), 269–84. Bloom, N., R. Griffith and J. Van Reenen (2002), ‘Do R&D tax credits work? Evidence from a panel of countries 1979–1997’, Journal of Public Economics, 85 (1), 1–31. Blundell, R., R. Griffith and J. Van Reenen (1999), ‘Market share, market value and innovation in a panel of British manufacturing firms’, Review of Economic Studies, 66 (228), 529–54. Bobe, B. (1991), ‘Trends in the use of research-and-development output indicators in EC program-evaluation’, Scientometrics, 21 (3), 263–82. Boer, F.P. (2000), ‘Valuation of technology using “real options”‘, ResearchTechnology Management, 43 (4), 26–30. Boer, F.P. (2002), ‘Financial management of R&D’, Research-Technology Management, 45 (4), 23–35.

References

259

Bollen, N.P.B. (1999), ‘Real options and product life cycles’, Management Science, 45 (5), 670–84. Bonaccorsi, A. (1996), Cambiamento Tecnologico e Competizione nell’Industria Aeronautica Civile. Integrazione delle Conoscenze e Incertezza, Milan: Guerini e Associati. Bonaccorsi, A. and P. Giuri (2000), ‘When shakeout doesn’t occur: the evolution of the turboprop engine industry’, Research Policy, 29 (7–8), 847–70. Bond, S., D. Harhoff and J. Van Reenen (1998), R&D and Productivity in Germany and the United Kingdom, mimeo, Mannheim: ZEW. Bosworth, D. and M. Rogers (2001), ‘Market value, R&D and intellectual property: an empirical analysis of large Australian firms’, The Economic Record, 77 (239), 323–37. Bourne, M., J. Mills, M. Wilcox, A. Neely and K. Platts (2000), ‘Designing, implementing and updating performance measurements systems’, International Journal of Operations & Production Management, 20 (7), 754–71. Bowman, E.H. and D. Hurry (1993), ‘Strategy through the option lens: an integrated view of resource investments and the incremental-choice process’, Academy of Management Review, 18 (4), 760–82. Bowns, S., I. Bradley, P. Knee, F. Williams and G. Williams (2003), ‘Measuring the economic benefits from R&D: improvements in the MMI model of the United Kingdom National Measurement System’, Research Policy, 32 (6), 991–1002. Bowon, K. and O. Heungshik (2002), ‘An effective R&D performance measurement system: survey of Korean R&D researchers’, Management Science, 30, 19–31. Bozbura, F.T., A. Beskesea and C. Kahramanb (2007), ‘Prioritization of human capital measurement indicators using fuzzy AHP’, Expert Systems with Applications, 32 (4), 1100–12. Bozeman, B. (1994), ‘Evaluating government technology-transfer – early impacts of the cooperative technology paradigm’, Policy Studies Journal, 22 (2), 322–37. Bradbury, F.R., W.M. Gallagher and C.W. Suckling (1973), ‘Qualitative aspects of the evaluation and control of research and development projects’, R&D Management, 3 (2), 49–57. Brandao, L. (2004), Real Option Valuation, Austin: University of Texas. Branstetter, L.G. and M. Sakakibara (2002), ‘When do research consortia work well and why? Evidence from Japanese panel data’, American Economic Review, 92 (1), 143–59. Brealey, R.A. and S.C. Myers (1996), Principles of Corporate Finance, 5th Edition, New York: McGraw-Hill. Brealey, R.A., S.C. Myers and F. Allen (2005), Principles of Corporate Finance, New York: McGraw-Hill. Bredin, K. and J. Söderlund (2006), ‘HRM and project intensification in R&Dbased companies: a study of Volvo Car Corporation and Astra Zeneca’, R&D Management, 36 (5), 467–85. Bremser, W.G. and N.P. Barsky (2004), ‘Utilizing the balanced scorecard for R&D performance measurement’, R&D Management, 34 (3), 229–38. Brennan, M.J. and P.J. Hughes (1991), ‘Stock prices and the supply of information’, Journal of Finance, 46 (5), 1665–91. Brennen, M. and E. Schwartz (1985), ‘Evaluating natural resource investments’, Journal of Business, 58 (2), 135–57. Breschi, S. and F. Malerba (1997), ‘Sectoral innovation systems’, in C. Edquist

260

Evaluation and performance measurement of R&D

(ed.), Systems of Innovation: Technologies, Institutions and Organisations, London: Pinter. Brewster, C. and H.H. Larsen (2000), Human Resource Management in Northern Europe: Trends, Dilemmas and Strategy, Oxford: Blackwell Publishers Ltd. Briggs, A.H., A.E. Ades and M.J. Price (2003), ‘Probabilistic sensitivity analysis for decision trees with multiple branches: use of the dirichlet distribution in a Bayesian framework’, Medical Decision Making, 23 (4), 341–50. Brooks, H. (1967), ‘Applied Science and Technological Progress’, Science, 156, 1706–12. Brown, M.A., T.R. Curlee and S.R. Elliott (1995), ‘Evaluating technology innovation programs – the use of comparison groups to identify impacts’, Research Policy, 24 (5), 669–84. Brown, W. and D. Gobeli (1992), ‘Observations on the measurement of R&D productivity: a case study’, IEEE Transactions on Engineering Management, 39 (4), 325–31. Brown, M.G. and R.A. Svenson (1988), ‘Measuring R&D productivity’, Research Technology Management, 31 (4), 11–15. Brusoni, S., A. Prencipe and K. Pavitt (2001), ‘Knowledge specialization, organizational coupling, and the boundaries of the firm: why do firms know more than they make?’, Administrative Science Quarterly, 46 (4), 597–621. Burch, J. (1994), Cost and Management Accounting: A Modern Approach, Saint Paul: West Publishing Company. Calantone, R.J., C.J. Di Benedetto and J.B. Schmidt (1999), ‘Using the analytic hierarchy process in new product screening’, Journal of Product Innovation Management, 16, 65–76. Canada, J.R. and J.A. White (1980), Capital Investment Decision Analysis for Management and Engineering, Englewood Cliffs, NJ: Prentice-Hall. Cardus, D.M., J. Fuhrer, A.W. Maritin and R.M. Thrall (1982), ‘Use of benefitcost analysis in the peer review of proposed research’, Management Science, 28 (4), 439–45. Carlsson, C. and R. Fullér (1996), ‘Fuzzy multiple criteria decision making: recent developments’, Fuzzy sets and systems, 78 (2), 139–53. Carr, P. (1988), ‘The valuation of sequential exchange opportunities’, Journal of Finance, 43 (5), 1235–56. Cassiman, B. and R. Veugelers (2006), ‘Are external technology sourcing strategies substitutes or complements? The case of embodied versus disembodied technology acquisition’, IESE Research Papers D/672, IESE Business School. Cazavan-Jeny, A. and T. Jeanjean (2003), ‘Value relevance of R&D reporting: a signaling interpretation’, Working Paper, ESSEC. Cesaroni, F., A. Di Minin and A. Piccaluga (2005), ‘Exploration and exploitation strategies in industrial R&D’, Creativity and Innovation Management, 14 (3), 222–32. Cha, J. and Y. Kim (2000), ‘Career orientations of R&D professionals in Korea’, R&D Management, 30 (2), 121–37. Chambers, D.J., R. Jennings and R.B. Thompson (2002), ‘Excess returns to R&Dintensive firms’, Review of Accounting Studies, 7 (2–3), 133–58. Chan, Y.S., S.I. Greenbaum and A.V. Thakor (1986), ‘Information reusability, competition and bank asset quality’, Journal of Banking and Finance, 10, 243–53. Chan, S.H., J.D. Martin and J.W. Kensinger (1990), ‘Corporate research and

References

261

development expenditures and share value’, Journal of Financial Economics, 26 (2), 255–76. Chan, L.K.C., J. Lakonishok and T. Sougiannis (2001), ‘The stock market valuation of research and development expenditures’, Journal of Finance, 56 (6), 2431–56. Chandler, A.D. (1990), Scale and Scope: The Dynamics of Industrial Capitalism, Cambridge, Massachusetts: The Belknap Press of Harvard University Press. Chaney, P.K., T.M. Devinney and R.S. Winer (1991), ‘The impact of new product introductions on the market value of firms’, Journal of Business, 64 (4): 573–609. Chang, Y.C., M.H. Chen, M.S. Hua and P.Y. Yang (2006), ‘Managing academic innovation in Taiwan: towards a “scientific-economic” framework’, Technological Forecasting and Social Change, 73 (2), 199–213. Chapman, D. and C. Cooper (1987), Risk Analysis for Large Projects: Models, Methods & Cases, New York: John Wiley & Sons Ltd. Chatterji, D. (1996), ‘Accessing external sources of technology’, ResearchTechnology Management, 39 (2), 49–56. Chen, C.C., C.M. Ford and G.F. Farris (1999), ‘Do rewards benefit the organization? The effects of reward types and the perceptions of diverse R&D professionals’, IEEE Transactions on Engineering Management, 46 (1), 47–55. Chesbrough, H. (2003a), Open Innovation: The New Imperative for Creating and Profiting from Technology, Boston, Massachusetts: Harvard Business School Press. Chesbrough, H. (2003b), Open Innovation, New York: Free Press. Chesbrough, H. (2003c), ‘The era of open innovation’, MIT Sloan Management Review, 44 (3), 35–41. Chesbrough, H. and M. Appleyard (2007), ‘Open innovation and strategy’, California Management Review, 50 (1), 57. Chesbrough, H., W. Vanhaverbeke and J. West (2006), Open Innovation: Researching a New Paradigm, Oxford: Oxford University Press. Chetal, R. and A. Raj (1998), ‘Sponsored R & D in India: The project sponsoring pattern and main outcome of projects sponsored by major central departments/ agencies’, Scientometrics, 43 (3), 393–421. Chien, C.-F. (2002), ‘A portfolio-evaluation framework for selecting R&D projects’, R&D Management, 32 (4), 359–68. Chiesa, V. (1996a), ‘Separating research from development: evidence from the pharmaceutical industry’, European Management Journal, 14 (6), 638–47. Chiesa, V. (1996b), ‘Managing the internationalization of R&D activities’, IEEE Transactions on Engineering Management, 43 (1), 7–23. Chiesa, V. (2001), R&D Strategy and Organisation: Managing Technical Change in Dynamic Contexts, London: Imperial College Press. Chiesa, V. (2003), La Bioindustria. Strategie competitive e organizzazione industriale nel settore delle biotecnologie farmaceutiche, Milan: ETAS. Chiesa, V. (2005), Valuation of Technology, Milan: MBA lecture notes. Chiesa, V. and F. Frattini (2007), ‘Exploring the differences in performance measurement between Research and Development: evidence from a multiple case study’, R&D Management, 37 (4), 285–303. Chiesa, V. and F. Frattini (2008), ‘Designing the organisation for innovation’, in C. Van Beers, A. Kleinknecht, R. Ortt and R. Verburg (eds), Determinants of Innovative Behaviour: A Firm’s Internal Practices and its External Environment, Houndmills: Palgrave, pp. 79–125.

262

Evaluation and performance measurement of R&D

Chiesa, V. and C. Masella (1996), ‘Searching for an effective measure of R&D performance’, Management Decision, 34 (7), 49–57. Chiesa, V., F. Frattini, V. Lazzarotti and R. Manzini (2006), ‘How to measure R&D performance: a design framework and an empirical study’, in J.F. Manzoni and M. Epstein (eds), Performance Measurement and Management Control: Improving Organizations and Society, Amsterdam: Elsevier, pp. 187–207. Chiesa, V., F. Frattini, V. Lazzarotti and R. Manzini (2008), ‘Designing a performance measurement system for the research activities: a reference framework and an empirical study’, Journal of Engineering and Technology Management, 25, 213–226. Cho, E. and M. Lee (2005), ‘An exploratory study on contingency factors affecting R&D performance measurement’, International Journal of Manpower, 26 (6), 502–12. Chordà, I.M. (1996), ‘Towards the maturity stage: an insight into the performance of French technopoles’, Technovation, 16 (3), 143–52. Christensen, P.O. and G.A. Feltham (eds) (2005), Economics of Accounting Volume II: Performance Evaluation, New York, LLC: Springer-Verlag. Clarysse, B. and U. Muldur (2001), ‘Regional cohesion in Europe? An analysis of how EU public RTD support influences the techno-economic regional landscape’, Research Policy, 30 (2), 275–96. Clemen, R.T. (1996), ‘Making hard decisions. An introduction to decision analysis’, 2nd Edition, California: Duxbury Press. Coccia, M. (2001), ‘A basic model for evaluating R&D performance: theory and application in Italy’, R&D management, 31 (4), 453–64. Coccia, M. (2004), ‘New models for measuring the R&D performance and identifying the productivity of public research institutes’, R&D Management, 34 (3), 267–80. Cockburn, I. and Z. Griliches (1988), ‘Industry effects and appropriability measures in the stock market’s valuation of R&D and patents’, American Economic Review, 78 (2), 419–23. Cohen, W.M. and D. Levinthal (1989), ‘Innovation and learning: the two faces of R&D’, Economic Journal, 99, 569–96. Cohen, W.M. and D. Levinthal (1990), ‘Absorptive capacity: a new perspective on learning and innovation’, Administrative Sciences Quarterly, 35, 569–96. Cohen, W.M., R.R. Nelson and J.P. Walsh (2000), ‘Protecting their intellectual assets: appropriability conditions and why U.S. manufacturing firms patent (or not)’, Working Paper 7552, National Bureau of Economic Research. Cohen, W., R. Nelson and J. Walsh (2002), ‘Links and impacts: the influence of public research on industrial R&D’, Management Science, 48 (1), 1–23. Collier, D.W. (1977), ‘Measuring the performance of R&D departments’, Research Management, 20 (2), 30–4. Comroe, J.H. Jr. and R.P. Dripps (1976), ‘Scientific basis for the support of biomedical science’, Science, 192, (4235), 105–11. Conceiçao, P., M.V. Heitor and P. Oliveira (1998), ‘University-based technology licensing in the knowledge based economy’, Technovation, 18 (10), 615–25. Cooke, P., M.G. Uranga and G. Etxebarria (1997), ‘Regional innovation systems: institutional and organisational dimensions’, Research Policy, 26, 475–91. Coombs, R. and R. Hull (1998), ‘Knowledge management practices and pathdependency in innovation’, Research Policy, 27 (3), 239–55.

References

263

Cooper, R. (1993), Winning at New Products: Accelerating the Process from Idea to Launch, Reading: Addison-Wesley. Cooper, R.S. (2003), ‘Purpose and performance of the Small Business Innovation Research (SBIR) program’, Small Business Economics, 20 (2), 137–51. Cooper, R.G. and E.J. Kleinschmidt (1995), ‘Benchmarking the firm’s critical success factors in new product development’, Journal of Product Innovation Management, 12 (5), 374–391. Cooper, R.G., S.J. Edgett and E.J. Kleinschmidt (1997), ‘Portfolio management in new products development: lessons from the leaders – Part I’, Research Technology Management, 40 (5), 16–28. Cooper, R.G., S.J. Edgett and E.J. Kleinschmidt (1998), Portfolio Management for New Products, Reading, Massachusetts: Addison-Wesley. Cooper, R.G., S.J. Edgett and E.J. Kleinschmidt (2001), ‘Portfolio management for new product development: results of an industry practices study’, R&D Management, 31 (4), 361–80. Cooper, R.G., S.J. Edgett and E.J. Kleinschmidt (2002), ‘Portfolio management for new product development: results of an industry practices study’, R&D Management, 31 (4), 361–80. Cooper, R.G., S.J. Edgett and E.J. Kleinschmidt (2004), ‘New product portfolio management: practices and performance’, Journal of Product Innovation Management, 16 (4), 333–51. Cordero, R. (1999), ‘Developing the knowledge and skills of R&D professionals to achieve process outcomes in cross-functional teams’, Journal of High Technology Management Research, 10 (1), 61–78. Cordero, R., N. Di Tomaso and G. F. Farris (1996), ‘Gender and race/ethnic composition of technical work groups: relationship to creative productivity and morale’, Journal of Engineering and Technology Management, 13, 205–21. Cordero, R., G. Farris and N. Di Tomaso (2002), ‘The technical supervisor as captain and catalyst’, Proceedings of the 2002 International Engineering Management Conference, Cambridge, UK. Costa, G. (1992), Manuale di Gestione del Personale, vol. 1-2-3, Turin: UTET. Cowles, M.K. and B.P. Carlin (1996), ‘Markov chain Monte Carlo convergence diagnostics: a comparative review’, Journal of American Statistics Association, 91, 883–904. Cox, J.C. and M. Rubinstein (1985), Options Markets, New Jersey: Prentice Hall. Cozzarin, B.P. (2006), ‘Performance measures for the socio-economic impact of government spending on R&D’, Scientometrics, 68 (1), 41–71. Cozzens, S. (1995), ‘US research assessment: recent developments’, Scientometrics, 34 (3), 351–62. Cozzens, S.E. (2000), ‘Assessing federally-supported academic research in the United States’, Research Evaluation, 9 (1), 5–10. Cozzens, S.E., K. Bobb and I. Bortagaray (2002), ‘Evaluating the distributional consequences of science and technology policies and programs’, Research Evaluation, 11 (2), 101–07. Creed, W.E. and R.E. Miles (1996), ‘Trust in organisations: a conceptual framework linking organizational forms, managerial philosophies, and the opportunity costs of controls’, in R.M. Kramer and T.R. Tyler, Trust in Organisations: Frontiers of Theory and Research, Thousand Oaks, CA: Sage, pp. 16–38. Cunion, K.M. (1995), ‘UK government departments experience of RT&D programme evaluation and methodology’, Scientometrics, 34 (3), 363–74.

264

Evaluation and performance measurement of R&D

Czarnitzki, D. and G. Licht (2006), ‘Additionality of public R&D grants in a transition economy’, Economics of Transition, 14 (1), 101–31. Czarnitzki, D., B.H. Hall and R. Oriani (2006), ‘The market valuation of knowledge assets in US and European firms’, in Derek L. Bosworth and Elizabeth Webster (eds), The Management of Intellectual Property, Cheltenham, UK: Edward Elgar, pp. 111–31. Dahlander, L. and D. Gann (2007), ‘How open is innovation?’, DRUID Summer Conference 2007 on Appropriability, Proximity, Routines and Innovation, Copenhagen, DRUID. Dahlby, B. (2005), ‘A framework for evaluating provincial R&D tax subsidies’, Canadian Public Policy-Analyse De Politiques, 31 (1), 45–58. Dalpé, R. and F. Anderson (1995), ‘National priorities in academic research – strategic research and contracts in renewable energies’, Research Policy, 24 (4), 563–81. Dalton, G.W., P.H. Thompson and R.L. Price (1977), ‘The four stages of professional careers – a new look at performance by professionals’, Organizational Dynamics, 6 (1), 19–42. Darby, M.R., L.G. Zucker and A. Wang (2004), ‘Joint ventures, universities, and success in the Advanced Technology Program’, Contemporary Economic Policy, 22 (2), 145–61. Das, S., P.K. Sen and S. Sengupta (1998), ‘Impact of strategic alliances on firm valuation’, Academy of Management Journal, 41 (1), 27–41. Dasgupta, P. and P.A. David (1994), ‘Toward a new economics of science’, Research Policy, 23 (5), 487–521. Davenport, S., J. Davies and C. Grimes (1999), ‘Collaborative research programmes: building trust from difference’, Technovation, 19 (1), 31–40. David, P. and D. Foray (1995), ‘Accessing and expanding the science and technology knowledge base’, STI Review, 16. David, P.A., B.H. Hall and A.A. Toole (2000), ‘Is public R&D a complement or substitute for private R&D? A review of the econometric evidence’, Research Policy, 29 (4–5), 497–529. Davila, T. (2000), ‘An empirical study on the drivers of management control systems’ design in new product development’, Accounting, Organisations and Society, 25, 383–409. De la Mothe, J. and A. Link (2002), Networks, Alliances and Partnerships in the Innovation Process, Boston: Kluwer Publishing. De Reyck, B., Y. Grushka-Cockayne, M. Lockett, S. Calderini, M. Moura and A. Sloper (2005), ‘The impact of project portfolio management on information technology projects’, International Journal of Project Management, 23, 524–37. Dean, B.V. and M.J. Nishry (1965), ‘Scoring and profitability models for evaluating and selecting engineering projects’, Operations Research, 13 (4), 550–69. Debackere, K., A. Verbeek, M. Luwel, E. Zimmermann (2002), ‘Measuring progress and evolution in science and technology – II: The multiple uses of technometric indicators’, International Journal of Management Review, 4 (3), 213–31. Deeds, D.L. (2001), ‘The role of R&D intensity, technical development and absorptive capacity in creating entrepreneurial wealth in high-tech start-up’, Journal of Engineering and Technology Management, 18, 29–47. Demski, J.S. (ed.) (1994), Managerial Uses of Accounting Information, Norwell, Massachusetts: Kluwer Academic Publishers.

References

265

Deng, Z. and B. Lev (2006), ‘In-process R&D: to capitalize or expense?’, Journal of Engineering and Technology Management, 23 (1–2), 18–32. Despres, C. and J.M. Hiltrop (1996), ‘Compensation for technical professionals in the knowledge age’, Research-Technology Management, 39 (5), 48–56. Di Masi, J.A., R.W. Hansen and H.G. Grabowskic (2003), ‘The price of innovation: new estimates of drug development costs’, Journal of Health Economics, 22 (2), 151–85. Di Minin, A., A. Piccaluga and M. Rizzone (2008), ‘Through the eyes of industrial researchers: how new “Connect & Develop” practices change the role of human resources in the lab’, R&D Management Conference, Ottawa. Dixit, A.K. and R.S. Pindyck (1994), Investment Under Uncertainty, Princeton: Princeton University Press. Doctor, R.N., D.P. Newton and A. Pearson (2001), ‘Managing uncertainty in research and development’, Technovation, 21, 79–90. Donaldson, G. (1961), Corporate Debt Capacity: A Study of Corporate Debt Policy and the Determinants of Corporate Debt Capacity, Boston, MA: Harvard Business School, Harvard University. Donegan, H.A., F.J. Dodd and T.B. M. McMaster (1991), ‘New approach to AHP decision-making’, The Statistician, 41 (3), 295–302. Dosi, G. (1982), ‘Technological paradigms and technological trajectories: a suggested interpretation of the determinants and directions of technical change’, Research Policy, 11 (3), 147–62 Dosi, G., M. Hobday, L. Marengo, A. Prencipe (2002), ‘The economics of system integration: toward an evolutionary interpretation’, LEM working paper series. Doukas, J. and L. Switzer (1992), ‘The stock market’s valuation of R&D spending and market concentration’, Journal of Economics and Business, 44 (2), 95–114. Driva, H. and K.S. Pawar (1999), ‘Performance measurement for product design and development in a manufacturing environment’, International Journal of Production Economics, 60, 61–68. Driva, H., K.S. Pawar and U. Menon (2000), ‘Measuring product development performance in manufacturing organizations’, International Journal of Production Economics, 63, 147–59. Dye, R.A. (1985), ‘Disclosure of nonproprietary information’, Journal of Accounting Research, 23 (1), 123–45. Dye, R.A. (1986), ‘Proprietary and nonproprietary disclosures’, Journal of Business, 59 (2), 331–36. Dye, R.A. (1988), ‘Earnings management in an overlapping generations model’, Journal of Accounting Research, 26 (2), 195–235. Dye, R.A. (1990), ‘Mandatory versus voluntary disclosures: the case of financial and real externalities’, The Accounting Review, 65 (1), 1–24. Easton, A. (1973), Complex Managerial Decision Involving Multiple Objectives, New York: John Wiley & Sons. Eberhart, A.C., W.F. Maxwell and A.R. Siddique (2004), ‘An examination of long-term abnormal stock returns and operating performance following R&D increases’, Journal of Finance, 59 (2), 623–49. Eilat, H., B. Golany and A. Shtub (2006), ‘Constructing and evaluating balanced portfolios of R&D projects with interactions: a DEA based methodology’, European Journal of Operational Research, 172, 1018–39. Eisenberg, R.S. (1987), ‘Proprietary rights and the norms of science in biotechnology research’, Yale Law Journal, 97 (2), 177–231.

266

Evaluation and performance measurement of R&D

Eisenhardt, K.M. (1989), ‘Building theories from case study research’, Academy of Management Review, 14 (4), 532–50. Eisenhardt, K.M. and M.E. Graebner (2007), ‘Theory building from cases: opportunities and challenges’, Academy of Management Journal, 50, 25–32. Ekboir, J.M. (2003), ‘Research and technology policies in innovation systems: zero tillage in Brazil’, Research Policy, 32 (4), 573–86. Ely, R.J. and D.A. Thomas (2001), ‘Cultural diversity at work: the effects of diversity perspectives on work group processes and outcomes’, Administrative Science Quarterly, 46, 229–73. Ely, K. and G. Waymire (1999), ‘Intangible assets and stock prices in the pre-SEC era’, Journal of Accounting Research, 37 (Supplement), 17–44. Emmanuel, C., D. Otley and K. Merchant (1990), Accounting for Management Control, London: Chapman & Hall. Engwall, M. and A. Jerbrant (2003), ‘The resource allocation syndrome: the prime challenge of multiproject management?’, International Journal of Project Management, 21 (6), 403–9. Esposito, E. (1996), Economia delle imprese ad alta tecnologia, Naples: Edizioni Scientifiche Italiane. Etzkowitz, H. (2003), ‘Innovation in innovation: the Triple Helix of universityindustry-government relations’, Social Science Information, 42 (3), 293–337. European Commission (1997), Second European Report on S&T Indicators, Final Report, EUR 17 639 EN, Brussels/Luxembourg: Office for Official Publications of the European Communities. European Commission (2005), European Technology Platform, Brussels/ Luxembourg: Office for Official Publications of the European Communities. European Commission Recommendation (2002), Modifications to Recommendation 96/280/CE with reference to definition of small and medium firms, Bruxelles, June, 25. Falk, R. (2007), ‘Measuring the effects of public support schemes on firms’ innovation activities – Survey evidence from Austria’, Research Policy, 36 (5), 665–79. Fama, E.F. (1970), ‘Efficient capital markets: a review of theory and empirical work’, Journal of Finance, 25 (2), 383–417. Fama, E.F. (1991), ‘Efficient capital markets: II’, Journal of Finance, 46 (5), 1575–617. Fama, E.F. and K.R. French (1992), ‘The cross-section of expected stock returns’, Journal of Finance, 47 (2), 427–65. Fama, E.F. and M.C. Jensen (1985), ‘Organizational forms and investment decisions’, Journal of Financial Economics, 14 (1), 101–19 Farris, G. and R. Cordero (2002), ‘Leading your scientists and engineers 2002’, Research-Technology Management, 45 (6), 13–25. Farrukh, C., R. Phaal, D. Probert, M. Gregory and J. Wright (2000), ‘Developing a process for the relative valuation of R&D programmes’, R&D Management, 30, 43–53. Feldman, M. (1999), ‘The new economics of innovation, spillovers and agglomeration: a review of empirical studies’, Economics of Innovation and New Technology, 8, 5–25. Feller, I. (1992), ‘American state governments as models for national science policy’, Journal of Policy Analysis and Management, 11 (2), 288–309. Feller, I. (2002), ‘Performance measurement redux’, American Journal of Evaluation, 23 (4), 435–52.

References

267

Fey, C. (2005), ‘External sources of knowledge, governance mode, and R&D performance’, Journal of Management, 31 (4), 597–621. Fink, A. (1995), How to Analyze Survey Data, Thousand Oaks: Sage. Folta, T.B., J.P. O’Brien (2004) ‘Entry in the presence of dueling options’, Strategic Management Journal, 25, 121–38. Foray, D. (2004), Economics of Knowledge, Cambridge, MA and London, UK: MIT Press. Forman, E.H. and S.I. Gass (2001), ‘The analytic hierarchy process: an exposition’, Operations Research, 49 (4), 469–86. Fox, G.E., N.R. Baker and J.L. Bryant (1984), ‘Economic models for R&D projects selection in the presence of project interactions’, Management Science, 30, 890–902. Francis, J. and L. Soffer (1997), ‘The relative informativeness of analysts’ stock recommendations and earnings forecast revisions’, Journal of Accounting Research, 35 (2), 193–211. Frankel, R., S.P. Kothari and J. Weber (2006), ‘Determinants of the informativeness of analyst research’, Journal of Accounting and Economics, 41 (1–2), 29–54. Franks, J. and C. Mayer (1990), ‘Capital markets and corporate control: a study of France, Germany and the UK’, Economic Policy, 10 (1), 189–232. Frattini, F., V. Lazzarotti and R. Manzini (2006), ‘Towards a performance measurement system for the research activities: NiKem Research case study’, International Journal of Innovation Management, 10 (4), 425–54. Fredberg, T. (2007), ‘Real options for innovation management’, International Journal of Technology Management, 39 (1/2), 72–85. Freeman, C. (1974), The Economics of Industrial Innovation, London: Pinter. Freeman, C. (1982), The Economics of Industrial Innovation, London, UK: Frances Printer. Freeman, C. (1987), Technology and Economic Performance: Lessons from Japan, London: Pinter. Freeman, C. (1991), ‘Networks of innovators: a synthesis of research issues’, Research Policy, 20 (5), 499–514. French, S. (1988), Decision Theory: An Introduction to the Mathematics of Rationality, New York: John Wiley. Frost, B. and C. Holzwarth (2002), ‘La motivazione nelle Communities of Practice’, Sistemi & Impresa, 8, 21–26. Galbraith, J.R. and D.A. Nathanson (1978), Strategy Implementation: The Role of Structure and Process, St. Paul: West Publishing Co. Gassmann, O. (2006), ‘Opening up the innovation process: towards an agenda’, R&D Management, 36 (3), 223–28. Gear, A., G.A. Lockett and A.W. Pearson (1971), ‘Analysis of some portfolio selection methods for R and D’, IEEE Transactions On Engineering Management, 18, 66–76. Gee, R.E. (1972), ‘The opportunity criterion – a new approach to the evaluation of R&D’, Research Management, 15 (3), 64–71. Georghiou, L.G. and J.S. Metcalfe (1993), ‘Evaluation of the impact of European Community research programs upon industrial competitiveness’, R&D Management, 23 (2), 161–9. Georghiou, L.G. and D. Roessner (2000), ‘Evaluating technology programs: tools and methods’, Research Policy, 29 (4–5), 657–78.

268

Evaluation and performance measurement of R&D

Geraci, J. (1994), ‘Real Managers Don’t Boss’, Research-Technology Management, 37 (6), 12–13. Geske, R. (1979), ‘The valuation of compound options’, Journal of Financial Economics, 7 (1), 63–81. Ghasemzadeh, F., P. Iyogun and N. Archer (1996), ‘A zero-one ILP model for project portfolio selection’, Innovation Research Centre Working Paper, Michael G. De Groote School of Business, McMaster University, Hamilton, ON. Gibson, C.B. and M.E. Zellmer-Bruhn (2001), ‘Metaphors and meaning: an intercultural analysis of the concept of teamwork’, Administrative Science Quarterly, 46, 274–303. Givoly, D. and J. Lakonishok (1979), ‘The information content of financial analysts’ forecasts of earnings’, Journal of Accounting & Economics, 2 (1), 165–85. Glover, F. and E. Woolsey (1974), ‘Converting the zero-one polynomial programming problem to a zero-one linear programming’, Operational Research, 22, 180–82. Godener, A. and K.E. Soderquist (2004), ‘Use and impact of performance measurement results in R&D and NPD: an exploratory study’, R&D Management, 32, 191–220. Golabi, K., C.W. Kirkwood and A. Sicherman (1981), ‘Selecting a portfolio of solar energy projects using multi-attribute preference theory’, Management Science, 27, 174–89. Goldner, F. and R.R. Ritti (1967), ‘Professionalization as career immobility’, American Journal of Sociology, 72 (5), 489. Gomez-Mejia, L.R., D.B. Balkin, and G.T. Milkovich (1990), ‘Rethinking Rewards for Technical Employees’, Organizational Dynamics, 18 (4), 62–75. Gompers, P., J. Ishii and A. Metrick (2003), ‘Corporate governance and equity prices’, Quarterly Journal of Economics, 118 (1), 107–55. Gonda, K. and F. Kakizaki (1995), ‘Research, technology and development evaluation; Developments in Japan’, Scientometrics, 34 (3), 375–89. Gordon, L.A. and D.A. Miller (1976), ‘A contingency framework for the design of accounting information systems’, Accounting, Organizations and Society, 1, 59–69. Gordon, L.A. and V.K. Narayanan (1984), ‘Management accounting systems, perceived environmental uncertainty and organizational structure: an empirical investigation’, Accounting, Organizations and Society, 9, 33–47. Gorg, H. and E. Strobl (2007), ‘The effect of R&D subsidies on private R&D’, Economica, 74 (294), 215–34. Grabher, G. (2001), ‘Ecologies of creativity: the village, the group, and the heterarchic organisation of the British advertising industry’, Environment and Planning A, 33 (2), 351–74. Granadino, B., L.M. Plaza and C. Vidal (2005), ‘Analysis of Spanish scientific output following the joint action program (Acciones Integradas) of the Ministry Of Science and Technology (MCYT)’, Research Evaluation, 14 (2), 97–102. Granstrand O., P. Patel and K. Pavitt (1997), ‘Multi-technology corporations, why they have “distributed” rather than “distinctive core” competencies’, California Management Review, 39, 8–25. Grant R.M. and C. Baden-Fuller (2004), ‘A knowledge accessing theory of strategic alliances’, Journal of Management Studies, 1, 61–84. Grant-Muller, S.M., P. MacKie, J. Nellthorp and A. Pearman (2001), ‘Economic appraisal of European transport projects: the state-of-the-art revisited’, Transport Reviews, 21 (2), 237–61.

References

269

Gray, D.O. and H.J. Steenhuis (2003), ‘Quantifying the benefits of participating in an industry university research center: an examination of research cost avoidance’, Scientometrics, 58 (2), 281–300. Greenhalgh, C. and M. Rogers (2006), ‘The value of innovation: the interaction of competition, R&D and IP’, Research Policy, 35 (4), 562–80. Griffin, A. and A.L. Page (1993), ‘An interim report on measuring product development success and failure’, Journal of Product Innovation Management, 10, 291–308. Griffin, A. and A.L. Page (1996), ‘PDMA success measurement project: recommended measures for produce development success and failure’, Journal of Product Innovation Management, 13 (6), 478–96. Griliches, Z. (1981), ‘Market value, R&D and patents’, Economics Letters, 7 (2), 183–87. Griliches, Z. (1995), ‘R&D and productivity: econometric results and measurement issues’, in Paul Stoneman (eds) (1995), Handbook of Economics of Innovation and Technological Change, Oxford, UK: Blackwell Publishers Ltd, pp. 52–89. Griliches, Z. and J. Mairesse (1984), ‘Productivity and R&D at the firm level’, in Zvi Griliches (ed.), R&D, Patents and Productivity, Chicago: The University of Chicago Press. Grimes, S. and P. Collins (2003), ‘Building a knowledge economy in Ireland through European research networks’, European Planning Studies, 11 (4), 395–413. Grossman, S.J. (1981), ‘The informational role of warranties and private disclosure about product quality’, Journal of Law and Economics, 24 (3), 461–83. Grossman, S.J. and O.D. Hart (1980), ‘Disclosure laws and takeover bids’, Journal of Finance, 35 (2), 323–34. Grossman, G.M. and E. Helpman (eds) (1991), Innovation and Growth in the Global Economy, Cambridge, Massachusetts: MIT Press. Grupp, H. (1993), ‘Efficiency of government intervention in technical change in telecommunications – 10 national economies compared’, Technovation, 13 (4), 187–220. Gu, F. and B. Lev (2003), ‘Intangible assets: measurement, drivers, usefulness’, New York Stern University, Working paper 2003-05. Gu, F. and W. Wang (2005), ‘Intangible assets, information complexity, and analysts’ earnings forecasts’, Journal of Business Finance & Accounting, 32 (9–10), 1673–702. Guest, D.E. (1990), ‘Human resource management and the American dream’, Journal of Management Studies, 27 (4), 377–97. Guimaraes, J.A. and M.C. Humann (1995), ‘Training of human-resources in science and technology in Brazil – the importance of a vigorous postgraduate program and its impact on the development of the country’, Scientometrics, 34 (1), 101–19. Gulati, R. (1998), ‘Alliances and networks’, Strategic Management Journal, 19 (4), 293–317. Gupta, A.K. and D. Wilemon (1996), ‘Changing patterns in industrial R&D management’, Journal of Product Innovation Management, 13 (6), 497–511. Hagedoorn, J. (1993), ‘Understanding the rationale of strategic technology partnering’, Strategic Management Journal, 14 (5), 371–85. Hagedoorn, J. and J. Schakenraad (1994), ‘The effect of strategic technology alliances on company performance’, Strategic Management Journal, 15 (4), 291–311.

270

Evaluation and performance measurement of R&D

Hajek, P. (1998), ‘Basic fuzzy logic and BL-algebras’, Soft Computing, 2 (3), 124–28. Hall, B.H. (1990), ‘The manufacturing sector masterfile: 1959–1987’, Working Paper 3366, National Bureau of Economic Research. Hall, B.H. (1993a), ‘Industrial research during the 1980s: did the rate of return fall?’, Brookings Papers on Economic Activity, Microeconomics, (2), 289–343. Hall, B. H. (1993b), ‘The stock market’s valuation of R&D investment during the 1980s’, American Economic Review, 83 (2), 259–64. Hall, B.H. (2000), ‘Innovation and market value’, in R. Barrell, G. Mason and M. O’Mahoney (eds), Productivity, Innovation and Economic Performance, Cambridge: Cambridge University Press. Hall, B.H. (2002), ‘The financing of research and development’, Oxford Review of Economic Policy, 18 (1), 35–51. Hall, B.H. (2009), ‘Measuring the returns to R&D: the depreciation problem’, Annales d’Economie et de Statistique, forthcoming. Hall, B.H., Z. Griliches and J.A. Hausman (1986), ‘Patents and R&D: is there a lag?’, International Economic Review, 27, 265–83. Hall, D.L. and A. Nauda (1990), ‘An interactive approach for selecting IR&D projects’, IEEE Transactions on Engineering Management, 37 (2), 126–33. Hall, B.H. and R. Oriani (2006), ‘Does the market value R&D investment by European firms? Evidence from a panel of manufacturing firms in France, Germany and Italy’, International Journal of Industrial Organization, 24 (5), 971–93. Hall, B. and J. Van Reenen (2000), ‘How effective are fiscal incentives for R&D? A review of the evidence’, Research Policy, 29 (4–5), 449–69. Hall, B.H. and K. Vopel (1997), ‘Market value, market share, and innovation’, NBER, the University of California at Berkeley, and the University of Mannheim, available at www.econ.berkeley.edu/~bhhall/papers/HallVopel97.pdf, June. Hall, N.G., J.C. Hershey, L.G. Kessler and R.C. Scotts (1992), ‘A model for making project funding decisions at the National Cancer Institute’, Operations Research, 22, 180–82. Hall, B.H., A.B. Jaffe and M. Trajtenberg (2005), ‘Market value and patent citations’, Rand Journal of Economics, 36 (1), 16–38. Hammer, M. and J. Champy (1993), Reengineering The Corporation, New York: HarperCollins. Hammonds, K.H., R. Furchgott, S. Hamm and P.C. Judge (1997), ‘Work and family’, Business Week, September 15, 96–9. Hand, J.R.M. (2001), ‘The market valuation of biotechnology firms and biotechnology R&D’, Working paper, University of North Carolina at Chapel Hill. Hand, J.R.M. (2003a), ‘Profit, losses, and the nonlinear pricing of Internet stocks’ in J. Hand and B. Lev (eds), Intangible Assets: Values, Measures, and Risks, New York: Oxford University Press, pp. 248–68. Hand, J.R.M. (2003b), ‘The increasing returns-to-scale of intangibles’, in John Hand and Baruch Lev (eds), Intangible Assets: Values, Measures, and Risks, New York: Oxford University Press, pp. 303–34. Hand, J.R.M. and B. Lev (eds) (2003), Intangible Assets: Values, Measures, and Risks, New York: Oxford University Press. Haneda, S., and H. Odagiri (1998), ‘Appropriation of returns from technological assets and the values of patents and R&D in Japanese high-tech firms’, Economics of Innovation and New Technology, 7 (4), 303–22.

References

271

Hansen, P.A. (1995), ‘Publicly produced knowledge for business – when is it effective’, Technovation, 15 (6), 387–97. Haour, G. (1992), ‘Stretching the knowledge base of the enterprise through contract research’, R&D Management, 30 (2), 177–82. Haour, G. (2004), Resolving the Innovation Paradox. Enhancing Growth in Technology Companies, Houndmills: Palgrave. Hardin, G. (1968), ‘The tragedy of the commons’, Science, 162 (3859), 1243–8. Harding, R. (2003), ‘New challenges for innovation systems: a cross country comparison’, International Journal of Technology Management, 26 (2–4), 226–46. Hauser, J.R. (1998), ‘Research, development and engineering metrics’, Management Science, 44 (12), 1670–89. Hauser, J.R and F. Zettelmeyer (1997), ‘Metrics to evaluate R, D&E’, ResearchTechnology Management, 40 (4), 32–38. Hayward, K. (1986), International Collaboration in Civil Aerospace, London: Frances Pinter. Healy, P.M., S.C. Myers and C. Howe (2002), ‘R&D accounting and the tradeoff between relevance and objectivity’, Journal of Accounting Research, 40 (3), 677–710. Heijs, J. (2003), ‘Freerider behaviour and the public finance of R&D activities in enterprises: the case of the Spanish low interest credits for R&D’, Research Policy, 32 (3), 445–61. Henderson, R. (1993), ‘Underinvestment and incompetence as responses to radical innovation: evidence from the photolithographic alignment equipment industry’, Rand Journal of Economics, 24 (2), 248–70. Henriksen, A.D. and A.J. Traynor (1999), ‘A practical R&D project-selection scoring tool’, IEEE Transactions on Engineering Management, 46 (2), 158–70. Hershock, R., C.D. Cowman and D. Peters (1994), ‘From experience: action teams that work’, Journal of Product Innovation Management, 11, 95–104. Heydebreck, P., M. Klofsten and J.C. Maier (2000), ‘Innovation support for new technology-based firms: the Swedish Teknopol approach’, R&D Management, 30 (1), 89–100. Hines, J. R. (1999), ‘Lessons from behavioral responses to international taxation’, National Tax Journal, 52 (2), 305-22. Ho, S.S.M. and R.H. Pike (1992), ‘The use of risk analysis techniques in capital investment appraisal’, in J. Ansell and F. Wharton (eds), Risk: Analysis, Assessment and Management, New York: John Wiley & Sons, pp. 71–95. Ho, Y.P. and P.K. Wong (2007), ‘Financing, regulatory costs and entrepreneurial propensity’, Small Business Economics, 28 (2–3), 187–204. Hobday, M. (1998), ‘Product complexity, innovation and industrial organization’, Research Policy, 26 (6), 689–710. Hobday, M. (2000), ‘The project-based organization: an ideal form for managing complex products and systems?’, Research Policy, 29 (7–8), 871–94. Hobday, M., H. Rush and J. Tidd (2000), ‘Innovation in complex products and systems’, Research Policy, 29 (7), 793–804. Hodder, J.E. and H.E. Riggs (1985), ‘Pitfalls in evaluating risky projects’, Harvard Business Review, Jan/Feb, 128–35. Hoekman, B.M., K.E. Maskus and K. Saggi (2005), ‘Transfer of technology to developing countries: Unilateral and multilateral policy options’, World Development, 33 (10), 1587–602.

272

Evaluation and performance measurement of R&D

Holmström, B. (1979), ‘Moral hazard and observability’, The Bell Journal of Economics, 10 (1), 74–91. Holthausen, R.W. and R.L. Watts (2001), ‘The relevance of the value relevance literature for financial accounting standard setting’, Journal of Accounting and Economics, 31 (3), 3–75. Howard, R.A. and J.E. Metheson (1981), Readings on the Principles and Applications of Decision Analysis, California: SDG. Howells, J. (1999), ‘Research and technology outsourcing’, Technology Analysis & Strategic Management, 11 (1), 17–29. Huang, C., C.A. Varum and J.B. Gouveia (2006), ‘Scientific productivity paradox: the case of China’s S&T system’, Scientometrics, 69 (2), 449–73. Hubbard, R.G. (1998), ‘Capital-market imperfections and investment’, Journal of Economic Literature, XXXVI, 193–225. Hubner, H. (1996), ‘Decisions on innovation and diffusion and the limits of deregulation’, Technovation, 16 (7), 327–39. Huchzermeier, A. and C.H. Loch (2001), ‘Project management under risk: using the real option approach to evaluate flexibility in R&D’, Management Science, 47 (4), 85–101. Hujer, R. and D. Radic (2005), ‘Evaluating the impacts of subsidies on innovation activities in Germany’, Scottish Journal of Political Economy, 52 (4), 565–86. Hultink, E.J. and H.S.J. Robben (1995), ‘Measuring new product success: the difference that time perspective makes’, Journal of Product Innovation Management, 12, 392–405. Hung, M. and K. Subramanyam (2007), ‘Financial statement effects of the adoption of International Accounting Standards: the case of Germany’, Review of Accounting Studies, 12 (4), 623–57. Husseiny, A.A. (1981), ‘Prioritization of R&D programs on probabilistic reactor safety’, Proocedings of the International AND/ENS Topical Meeting, LaGrange: American Nuclear Society. Huston, L. and N. Sakkab (2006), ‘Connect and develop: inside Procter & Gamble’s new model for innovation’, Harvard Business Review, 84 (3), 58–66. Huyghebaert, N. and L.M. Van De Gucht (2007), ‘The determinants of financial structure: new insights from business start-ups’, European Financial Management, 13 (1), 101–33. Hwang, C.L. and K. Yoon (1981), Multiple Attribute Decision Making Methods and Applications, Berlin and Heidelberg: Springer. Iansiti, M. and R. Levien (2004), ‘Strategy as ecology’, Harvard Business Review, 82 (3), 68. Ingersoll, J.E. Jr. (1987), Theory of Financial Decision Making, Maryland: Rowmand and LittleField. Irvine, J. (1988), Evaluating Applied Research: Lessons from Japan, London, UK: Pinter Publishers. Irwin, D.A. and P.J. Klenow (1996), ‘High-tech R&D subsidies – estimating the effects of Sematech’, Journal of International Economics, 40 (3–4), 323–44. Iversen, E.J., M. Gulbrandsen and A. Klitkou (2007), ‘A baseline for the impact of academic patenting legislation in Norway’, Scientometrics, 70 (2), 393–414. Jacobs, D. (1998), ‘Innovation policies within the framework of internationalization’, Research Policy, 27 (7), 711–24. Jaffe, A.B. (1986), ‘Technological opportunity and spillovers of R&D: evidence

References

273

from firms’ patents, profits, and market value’, American Economic Review, 76 (5), 984–1001. Jaffe, A.B. (2002), ‘Building programme evaluation into the design of public research-support programmes’, Oxford Review of Economic Policy, 18, 22–34. Jaffe, A.B. and J. Lerner (2001), ‘Reinventing public R&D: patent policy and the commercialization of national laboratory technologies’, The RAND Journal of Economics, 32 (1), 167–98. James, W.M. (2002), ‘Best HR practices for today’s innovation management’, Research-Technology Management, 45 (1), 57–60. Jang, Y.S. (2000), ‘The worldwide founding of ministries of science and technology, 1950–1990’, Sociological Perspectives, 43, 247–70. Jansen, J. (2004), ‘Strategic information revelation in an R&D race with spillovers’, Working paper, WZB Berlin. Jansen, J., F. Van Den Bosch and H. Volberda (2005), ‘Managing potential and realized absorptive capacity: how do organizational antecedents matter?’, Academy of Management Journal, 48 (6), 999–1015. Jassawalla, A.R. and H.C. Sashittal (2000), ‘Strategies of effective new product team leaders’, California Management Review, 42 (2), 34–51. Jensen, M.C. (1978), ‘Some anomalous evidence regarding market efficiency’, Journal of Financial Economics, 6 (2/3), 95–101. Jeppesen, L. and M. Molin (2003), ‘Consumers as co-developers: learning and innovation outside the firm’, Technology Analysis and Strategic Management, 15 (3), 363–83. Jiménez-Contreras, E., F.D. Anegon and E.D. Lopez-Cozar (2003), ‘The evolution of research activity in Spain – The impact of the National Commission for the Evaluation of Research Activity (CNEAI)’, Research Policy, 32 (1), 123–42. Jones, G. and H. Teegen (2002), ‘Factors affecting foreign R&D location decisions: management and host policy implications’, International Journal of Technology Management, 25 (8), 791–813. Jones, G., A. Lanctot and H. Teegen (2000), ‘Determinants and performance impacts of external technology acquisition’, Journal of Business Venturing, 16 (3), 255–83. Jordan, G.B. (2005), ‘What matters to R&D workers’, Research-Technology Management, 48 (3), 23–32. Justman, M. and E. Zuscovitch (2002), ‘The economic impact of subsidized industrial R&D in Israel’, R&D Management, 32 (3), 191–99. Kale, P., H. Singh and H. Perlmutter (2000), ‘Learning and protection of proprietary assets in strategic alliances: building Relational capital’, Strategic Management Journal, 21 (3), 217–37. Kamoda, H. and S. Sugawa (2008), ‘Research and development evaluation at an early stage using the analytic hierarchy process (AHP)’, 4th IEEE International Conference on Management of Innovation and Technology, ICMIT 2008, 1444–1449. Kamrad, B. and E. Ricardo (1995), ‘Multiproduct manufacturing with stochastic input prices and output yield uncertainty’, in L. Trigeorgis (ed.), Real Options in Capital Investment: Models, Strategies, and Applications, Westport: Praeyer Publishers, pp. 281–302. Kaplan, R.S. and D.P. Norton (1992), ‘The balance scorecard – measures that drive performance’, Harvard Business Review, 70 (1), 71–9.

274

Evaluation and performance measurement of R&D

Kaplan, R.S. and D.P. Norton (1993), ‘Putting the balanced scorecard to work’, Harvard Business Review, 71 (5), 134–47. Kaplan, R.S. and D.P. Norton (1996), ‘Using the balance score card as a strategic management system’, Harvard Business Review, 74 (1), 75–85. Kastrinos, N. (1995), ‘The impact of EC R-and-D programs on corporate R-and-D practice in Europe – a knowledge-base approach’, R&D Management, 25 (3), 269–79. Katila, R. and G. Ahuja (2002), ‘Something old, something new: a longitudinal study of search behavior and new product introduction’, Academy of Management Journal, 45 (8), 1183–94. Katz, R. (1988), ‘Managing careers: the influence of job and group longevities’, in M.L. Tushman and W.L. Moore (eds), Readings in the Management of Innovation, 2nd edition, New York: Harper Business. Katz, R. and T.J. Allen (1985), ‘Project performance and the locus of influence in the R&D Matrix’, Academy of Management Journal, 28 (1), 67–87. Kaufman, H.G. (1974), Obsolescence and Professional Career Development, New York: AMACOM. Kaufmann, A. and F. Todtling (2002), ‘How effective is innovation support for SMEs? An analysis of the region of Upper Austria’, Technovation, 22 (3), 147–59. Kaufmann, A. and P. Wagner (2005), ‘EU regional policy and the stimulation of innovation: the role of the European Regional Development Fund in the objective 1 region Burgenland’, European Planning Studies, 13 (4), 581–99. Kauko, K. (1996), ‘Effectiveness of R&D subsidies – a sceptical note on the empirical literature’, Research Policy, 25 (3), 321–23. Kayworth, T.R. and D.E. Leidner (2002), ‘Leadership effectiveness in global virtual teams’, Journal of Management Information Systems, 18 (3), 7–40. Keeney, R.L. (1987), ‘An analysis of the portfolio of sites to characterize for selecting a nuclear repository’, Risk Analysis, 7, 195–216. Keeney, R.L. and H. Raiffa (1976), Decision with Multiple Objectives: Preferences and Value Tradeoffs, New York: John Wiley & Sons. Keller, R.T. (2001), ‘Cross-functional project groups in research and new product development: diversity, communications, job stress, and outcomes’, Academy of Management Journal, 44 (3), 549–55. Kellogg, D., J.M. Charnes and R. Demirer (1999), ‘Valuation of a biotechnology firm: an application of real-options methodologies’, 3rd Annual International Conference on Real Options. Kelly, T.B. and A. Ljungqvist (2007), ‘The value of research’, Working paper, New York University. Kelm, K.M., V.K. Narayanan and G.E. Pinches (1995), ‘Shareholder value creation during R&D innovation and commercialization stages’, Academy of Management Journal, 38 (3), 770–86. Kensinger, J.W. (1987), ‘Adding the value of active management into the capital budgeting equation’, Midland Corporate Finance Journal, 5 (1), 31–42. Kerre, E.E. and J.N. Mordeson (2005), ‘A historical overview of fuzzy mathematics’, New Mathematics and Natural Computation, 1, 1–26. Kerssens-van Drongelen, I.C. and J. Bilderbeek (1999), ‘R&D performance measurement: more than choosing a set of metrics’, R&D Management, 29 (1), 35–46. Kerssens-van Drongelen, I.C. and A. Cook (1997), ‘Design principles for the

References

275

development of measurement systems for research and development processes’, R&D Management, 27 (4) 345–57. Kerssens-van Drongelen, I.C., B. Nixon and A. Pearson (2000), ‘Performance measurement in industrial R&D’, International Journal of Management Reviews, 2 (2), 111–43. Kester, W.C. (1984), ‘Today’s options for tomorrow’s growth’, Harvard Business Review, 62 (2), 153–60. Kester, W.C. (1993), ‘Turning growth options in real assets’, in R. Aggarwal (ed.) Capital Budgeting Under Uncertainty, Englewood Cliffs, New Jersey, pp. 187–202. Khanna, T., R. Gulati and N. Nohria (1998), ‘The dynamics of learning alliances: competition, cooperation, and relative scope’, Strategic Management Journal, 19 (3), 193–210. Kim, L.S. and C.J. Dahlman (1992), ‘Technology policy for industrialization – an integrative framework and Korea experience’, Research Policy, 21 (5), 437–52. Kim, B. and H. Oh (2002), ‘An effective R&D PMS: survey of Korean R&D researchers’, International Journal of Management Science, 30, 19–31. Kingsley, G., B. Bozeman and K. Coker (1996), ‘Technology transfer and absorption: an “R&D value-mapping” approach to evaluation’, Research Policy, 25 (6), 967–95. Kiyota, K. and T. Okazaki (2005), ‘Foreign technology acquisition policy and firm performance in Japan, 1957–1970: Micro-aspects of industrial policy’, International Journal of Industrial Organization, 23 (7–8), 563–86. Kleinknecht, R. and B. Verspagen (1990), ‘Demand and innovation: Schmookler re-examined’, Research Policy, 19 (4), 387–94. Klette, T.J., J. Moen and Z. Griliches (2000), ‘Do subsidies to commercial R&D reduce market failures? Microeconometric evaluation studies’, Research Policy, 29 (4–5), 471–95. Kline, S.J. and N. Rosenberg (1986), ‘An overview of innovation’, in R. Landau and N. Rosenberg (eds), The Positive Sum Strategy: Harnessing Technology for Economic Growth, Washington DC: National Academy Press. Knight, K. (1977), Matrix Management: A Cross-functional Approach to Organization, Westmead: Gower Press, Teakfield. Kochanski, J. and G. Ledford (2001), ‘How to keep me – retaining technical professionals’, Research-Technology Management, 44 (3), 31–8. Kodama, F. (1995), Emerging Patterns of Innovation, Boston: Harvard Business School Press. Koga, T. (2003), ‘Firm size and R&D tax incentives’, Technovation, 23 (7), 643–8. Koga, T. (2005), ‘R&D subsidy and self-financed R&D: the case of Japanese hightechnology start-ups’, Small Business Economics, 24 (1), 53–62. Kogut, B. and N. Kulatilaka (1994), ‘Options thinking and platform investments: investing in opportunity’, California Management Review, 36 (2), 52–71. Koh, F.C.C., W.T.H. Koh and F.T. Tschang (2005), ‘An analytical framework for science parks and technology districts with an application to Singapore’, Journal of Business Venturing, 20 (2), 217–39. Kostoff, R.N. (1993), ‘Semiquantitative methods for research impact assessment’, Technological Forecasting and Social Change, 44 (3), 231–44. Kostoff, R.N. (1994a), ‘Assessing research impact – federal peer-review practices’, Evaluation Review, 18 (1), 31–40.

276

Evaluation and performance measurement of R&D

Kostoff, R.N. (1994b), ‘Assessing research impact – semiquantitative methods’, Evaluation Review, 18 (1), 11–19. Kothari, S.P., T.E. Laguerre and A.E. Leone (2002), ‘Capitalization versus expensing: evidence on the uncertainty of future earnings from capital expenditures versus R&D outlays’, Review of Accounting Studies, 7 (4), 355–82. Krauskopf, M. (1992), ‘Scientometric indicators as a means to assess the performance of state supported universities in developing-countries – the Chilean case’, Scientometrics, 23 (1), 105–21. Kuhn, T.S. (1962), The Structure of Scientific Revolutions, Chicago: University of Chicago Press. Kulatilaka, N. (1993), ‘The value of flexibility: the case of a dual-fuel industrial steam boiler’, Financial Management, 22 (3), 271–80. Kulatilaka, N. and L. Trigeorgis (1994), ‘The general flexibility to switch; real options revisited’, International Journal of Finance, 6 (2), 778–98. Kyrgiafini, L. and E. Sefertzi (2003), ‘Changing regional systems of innovation in Greece: the impact of regional innovation strategy initiatives in peripheral areas of Europe’, European Planning Studies, 11 (8), 885–910. Kyriakou, D. (1995), ‘Macroeconomic aspects of S/T programme evaluation’, Scientometrics, 34 (3), 451–59. La Porta, R., F. Lopez-De-Silanes, A. Shleifer and R. Vishny (1998), ‘Law and finance’, Journal of Political Economy, 106 (6), 1113–35. La Porta, R., F. Lopez-De-Silanes, A. Shleifer and R. Vishny (2002), ‘Investor protection and corporate valuation’, Journal of Finance, 57 (3), 1147–70. Lakatos, I. (1970), ‘Falsification and the methodology of scientific research programmes’, in I. Lakatos and A. Musgrave (eds) Criticism and the Growth of Knowledge, New York: Cambridge University Press, pp. 91–195. Lang, M. and R. Lundholm (1996), ‘Corporate disclosure policy and analyst behavior’, Accounting Review, 71 (4), 467–92. Langfeldt, L. (2004), ‘Expert panels evaluating research: decision-making and sources of bias’, Research Evaluation, 13 (1), 51–62. Laredo, P. (1998), ‘The networks promoted by the framework programme and the questions they raise about its formulation and implementation’, Research Policy, 27 (6), 589–98. Larsen, H. H. and C. Brewster (2003), ‘Line management responsibility for HRM: what is happening in Europe?’, Employee Relations, 25 (3), 228–44. Laursen, K. and A. Salter (2006), ‘Open for innovation: the role of openness in explaining innovation performance among UK manufacturing firms’, Strategic Management Journal, 27 (2), 131–50. Lee, C.-L. and S.-Q. Lai (2007), ‘Performance measurement systems for knowledge management in high-tech industries: a balanced scorecard framework’, International Journal of Technology Management, 39 (1–2), 158–76. Lee, J.W. and S.H. Kim (2000), ‘Using analytic network process and goal programming for interdependent information system project selection’, Computers and Operations Research, 27 (4), 367–82. Lerner, J. (1999), ‘The government as venture capitalist: the long-run impact of the SBIR program’, Journal of Business, 72 (3), 285–318. Lev, B. (2004), ‘Sharpening the intangibles edge’, Harvard Business Review, 82 (6), 109–16. Lev, B. (ed.) (2001), Intangibles: Management, Measuring, and Reporting, Washington: The Brookings Institution Press.

References

277

Lev, B. and T. Sougiannis (1996), ‘The capitalization, amortization, and valuerelevance of R&D’, Journal of Accounting & Economics, 21 (1), 107–38. Lev, B. and T. Sougiannis (1999), ‘Penetrating the book-to-market black box: the R&D effect’, Journal of Business Finance and Accounting, 23 (3–4), 419–49. Lev, B. and P. Zarowin (1999), ‘The boundaries of financial reporting and how to extend them’, Journal of Accounting Research, 37 (2), 353–85. Lev, B., B. Sarath and T. Sougiannis (2005), ‘R&D reporting biases and their consequences’, Contemporary Accounting Research, 22 (4), 977–1026. Lev, B., D. Nissim and J. Thomas (2007), ‘On the informational usefulness of R&D capitalization and amortization’, Working Paper, Columbia University. Levi, D. and C. Slem (1995), ‘Teamwork in research and development organizations: the characteristics of successful teams’, International Journal of Industrial Ergonomics, 16, 29–42. Levin, R.C., A.K. Klevorick, R.R. Nelson and S.G. Winter (1987), ‘Appropriating the returns from industrial research and development’, Brooking Papers on Economics Activity, 3, 783–832. Levinthal, D.A. and J.G. March (1993), ‘The myopia of learning’, Strategic Management Journal, Winter Special Issue, 14, 95–112. Lewison, G. and P. Cunningham (1991), ‘Bibliometric studies for the evaluation of trans-national research’, Scientometrics, 21 (2), 223–44. Leyden, D.P. and A.N. Link (1993), ‘Tax policies affecting research-anddevelopment – an international comparison’, Technovation, 13 (1), 17–25. Leydesdorff, L. and H. Etzkowitz (1996), ‘Emergence of a triple helix of universityindustry-government relations’, Science and Public Policy, 23 (5), 279–86. Liang, W.Y. (2003), ‘The analytic hierarchy process in project evaluation’, Benchmarking: An International Journal, 10 (5), 445–56. Liberatore, M. and G. Titus (1983), ‘The practice of management science in R&D project selection’, Management Science, 29 (8), 962–74. Lichtenberg, F.R. and T.J. Philipson (2002), ‘The dual effects of intellectual property regulations: within- and between-patent competition in the US pharmaceuticals industry’, Journal of Law and Economics, 45, 643–72. Lillis, D.A., A.C.M. Ho, H. Campbell, D. Bartle, C.J. Marks and A.C. Hinton (2002), ‘Evaluating support for technological research and innovation in some New Zealand businesses: a survey’, Research Evaluation, 11 (1), 37–48. Lindkvist, L. (2004), ‘Governing project-based firms: promoting market-line processes within hierarchies’, Journal of Management and Governance, 8, 3–25. Link, A.N. and W.F. Finan (1997), ‘Quantifying the private returns to collaborative research: The case of SEMATECH’, International Journal of Technology Management, 13 (5–6), 695–705. Lint, O. and E. Pennings (1998), ‘R and D as an option on market introduction’, R&D Management, 28, 279–87. Lint, O. and E. Pennings (2001), ‘An option approach to the new product development process: a case study at Philips Electronics’, R&D Management, 31 (2), 163–72. Lintner, J. (1965), ‘The valuation of risk assets and selection of risky investments in stock portfolios and capital budgets’, Review of Economics and Statistics, 47, 13–37. Liu, T.-L. and S.-Y. Wen (2006), ‘Evaluations of drug development status in biopharmaceutical industry’, XVI ACME International Conference on Pacific Rim Management.

278

Evaluation and performance measurement of R&D

Liu, B. and S. Xu (1987), ‘Development of the theory and methodology of the analytic hierarchy process and its application in China’, Mathematical Modelling, 9, 179–84. Loch, C.H. and S. Tapper (2002), ‘Implementing a strategy-driven performance measurement system for an applied research group’, Journal of Product Innovation Management, 19 (3), 185–98. Loch, C., L. Stein and C. Terwiesch (1996), ‘Measuring development performance in electronics industry’, Journal of Product Innovation Management, 13, 3–20. Locke, S. (1990), ‘Property investment analysis using Adjusted Present Values’, The Appraisal Journal, July, 373–8. Loudder, M. and M. Behn (1995), ‘Alternative income determination rules and earnings usefulness: the case of R&D costs’, Contemporary Accounting Research, 12 (1), 185–205. Love, H. and S. Roper (2002), ‘Internal versus external R&D: a study of R&D choice with sample selection’, International Journal of the Economics of Business, 9 (2), 239–55. Luehrman, T. (1998a), ‘Investment opportunities as real options: getting started with the numbers’, Harvard Business Review, 76 (4), 51–67. Luehrman, T. (1998b), ‘Strategy as a portfolio of real options’, Harvard Business Review, 76 (5), 89–99. Lundvall, B.A. (ed.) (1992), National Systems of Innovation: Towards a Theory of Innovation and Interactive Learning, London, UK: Frances Pinter. Luukkonen, T. (1995), ‘The impacts of research field evaluations on research practice’, Research Policy, 24 (3), 349–65. Luukkonen, T. (1998), ‘The difficulties in assessing the impact of EU framework programmes’, Research Policy, 27 (6), 599–610. Lys, T. and S. Sohn (1990), ‘The association between revisions of financial analysts’ earnings forecasts and security price changes’, Journal of Accounting & Economics, 13 (4), 341–63. Mader, D.P. (2004), ‘Selecting design for six sigma projects’, Quality Progress, 37 (7), 65–70. Magee, J.F. (1964), ‘Decision trees for decision making’, Harvard Business Review, 42 (4), 126–38. Majad, S. and R. Pindyck (1987), ‘Time to build, option value and investment decisions’, Journal of Financial Economics, 18 (1), 7–27. Mamuneas, T.P. and M.I. Nadiri (1996), ‘Public R&D policies and cost behavior of the US manufacturing industries’, Journal of Public Economics, 63 (1), 57–81. Mann, R.J. (2005), ‘Do patents facilitate financing in the software industry?’, Texas Law Review, 83 (4), 961–1030. Mansfield, E. (1985), ‘How rapidly does new industrial technology leak out’, Journal of Industrial Economics, 34 (2), 217–23. Mansfield, E., J. Rapoport, A. Romeo, S. Wagner and J. Beardsley (1977), ‘Social and private rates of return from industrial innovations’, Quarterly Journal of Economics, 91 (2), 221–40. Margrabe, W. (1978), ‘The value of an option to exchange one asset for another’, Journal of Finance, 33 (1), 177–86. Markham, S.K. and L. Aiman-Smith (2001), ‘Product champions: truths, myths, and management’, Research-Technology Management, 44 (3), 44–45. Markham, S.K. and A. Griffin (1998), ‘The breakfast of champions: associations

References

279

between champions and product development environments, practices, and performance’, Journal of Product Innovation Management, 15, 436–54. Markowitz, H. (1959), Portfolio Selection, New York: John Wiley. Markusova, V.A., V.A. Minin, A.N. Libkind, C.N.M. Jansz, M. Zitt and E. Bassecoulard-Zitt (2004), ‘Research in non-metropolitan universities as a new stage of science development in Russia’, Scientometrics, 60 (3), 365–83. Marshall, A. (1919), Industry and Trade, London: Macmillan. Marshall, K.T. and R.M. Oliver (1995), Decision Making and Forecasting, New York: McGraw Hill. Martin, B.R. (1996), ‘The use of multiple indicators in the assessment of basic research’, Scientometrics, 36, 343–62. Martin, B.R. and J. Irvine (1983), ‘Assessing basic research. Some partial indicators of scientific progress in radio astronomy’, Research Policy, 12, 61–90. Martino, J.P. (1995), R&D Project Selection, New York: John Wiley. Mauguin, P. (1991), ‘Using a contracts database for evaluating the dynamics of a technological program – the case of the European nonnuclear energy program’, Scientometrics, 22 (1), 207–28. McBride, R.D. and J.S. Yorkmark (1980), ‘An implicit enumeration algorithm for quadratic integer programming’, Management Science, 16, 282–96. McClellan, D.C., J. Atkinson, R.A. Clark and E.L. Lowell (1953), The Achievement Motive, New York: Appleton Century Crofts. McConnell, J.J. and C.J. Muscarella (1985), ‘Corporate capital expenditure decisions and the market value of the firm’, Journal of Financial Economics, 14 (3), 399–422. McDermott, R. (1999), ‘Why information technology inspired but cannot deliver knowledge management’, California Management Review, 41 (4), 103–17. McDonald, R. and D. Siegel (1985), ‘Investment and the valuation of firms when there is an option of shut down’, International Economic Review, 28 (2), 331–49. McDonald, R. and D. Siegel (1986), ‘The value of waiting to invest’, Quarterly Journal of Economics, 101 (4), 707–27. McDonough III, E.F. (2000), ‘Investigating the factors contributing to the success of crossfunctional teams’, Journal of Product Innovation Management, 17, 221–35. McGrath, R.G. (1997), ‘A real options logic for initiating technology positioning investments’, Academy of Management Review, 22 (4), 974–96. McGrath, R.G. and A. Nerkar (2004), ‘Real options reasoning and a new look at the R&D investment strategies of pharmaceutical firms’, Strategic Management Journal, 25 (1), 1–21. McKenzie, K.J. (2005), ‘Tax subsidies for R&D in Canadian provinces’, Canadian Public Policy-Analyse De Politiques, 31 (1), 29–44. McKinnon, P.D. (1987), ‘Steady-state people: a third career orientation’, Research Management, 30 (1), 26–32. McMeekin, A. and R. Coombs (1999), ‘Human resource management and the motivation of technical professionals’, International Journal of Innovation Management, 3 (1), 1–26. Meade, L.M. and A. Presley (2002), ‘R&D project selection using the analytic network process’, IEEE Transactions on Engineering Management, 49 (1), 59–66. Melkers, J. and D. Roessner (1997), ‘Politics and the political setting as an influence

280

Evaluation and performance measurement of R&D

on evaluation activities: national research and technology policy programs in the United States and Canada’, Evaluation and Program Planning, 20 (1), 57–75. Merchant, K.A. (1981), ‘The design of a corporate budgeting system: influences on managerial behaviour and performance’, Accounting Review, 56, 813–29. Merchant, K.A. (1998), Modern Management Control Systems: Text and Cases, Englewood Cliffs: Prentice Hall. Merkhofer, M.W. and R.L. Keeney (1987), ‘A multi-attribute utility analysis of alternative sites for the disposal of nuclear waste’, Risk Analysis, 7, 173–94. Merton, R.K. (1957), Social Theory and Social Structure, Glencoe: The Free Press. Merton, R.C. (1973), ‘Theory of rational option pricing’, Bell Journal of Economics and Management Science, 4, 141–83. Meyer, M. and J. Utterback (1993), ‘The product family and the dynamics of core capability’, Sloan Management Review, 34 (3), 29–47. Meyer-Krahmer, F. (1995), ‘Technology policy evaluation in Germany’, International Journal of Technology Management, 10 (4–6), 601–21. Midler, C. (1995), ‘Projectification of the firm: the Renault case’, Scandinavian Journal of Management, 11 (4), 363–75. Miles, M.B. and A.M. Huberman (1984), Qualitative Data Analysis, Newbury Park: Sage. Milgrom, P.R. (1981), ‘Good news and bad news: representation theorems and applications’, The Bell Journal of Economics, 12 (2), 380–91. Miller, B. and J.-P. Clarke (2005), ‘Investments under uncertainty in air transportation: a real options perspective’, Journal of the Transportation Research Forum, 44 (1), 61–74. Mintzberg, H. (1983), Structure in Fives, Englewood Cliffs, NJ: Prentice Hall. Moed, H.F., W.J.M. Burger, J.G. Frankfort and A.F.J. Van Raan (1985), ‘The use of bibliometrics data as tools for university research policy’, Research Policy, 14, 131–49. Mohapatra, P.K.J., S. Roy and P.S. Nagpaul (2003), ‘Developing a model to measure the effectiveness of research units’, International Journal of Operations & Production Management, 23 (12), 1514–31. Mohr, J., S. Sengupta and S. Slater (2005), Marketing of High-Technology Products and Innovations, Upper Saddle River, New Jersey: Pearson Education. Moizer, P. (1991), ‘Performance appraisal and rewards’, in Ashton, D., T. Hopper and R.W. Scapens (eds), Issues in Management Accounting, Englewood Cliffs: Prentice Hall, pp. 126–44. Mokyr, J. (2006), ‘Useful knowledge as an evolving system: the view from economic history’, in L.E. Blume and S.N. Durlauf (eds), The Economy as an Evolving Complex System, III. Current Perspectives and Future Directions, Oxford: Oxford University Press, pp. 309–36. Molas-Gallart, J. and A. Davies (2006), ‘Toward theory-led evaluation – The experience of European science, technology, and innovation policies’, American Journal of Evaluation, 27 (1), 64–82. Moore, J.R. and N.R. Baker (1969), ‘Computational analysis of scoring models for R and D project selection’, Management Science, Application Series, 16, 4, B212-B232. Morgan, M.G. and M. Henrion (1990), Uncertainty: A Guide to Dealing With Uncertainty in Quantitative Risk and Policy Analysis, New York: Cambridge University Press.

References

281

Morgan, G.A., O.V. Griego and G.W. Gloeckner (2001), SPSS For Windows: An Introduction to Use and Interpretation in Research, London: Lawrence Erlbaum Associates. Morricone, S., R. Oriani and M. Sobrero (2008), ‘The value relevance of intangible assets: the Italian adoption of International Accounting Standards (IAS/IFRS)’, Working paper, University of Bologna. Morris, P.A., E.O. Teisberg and A.L. Kolbe (1991), ‘When choosing R and D projects, go with the long shots’, Research-Technology Management, 34 (1), 35–40. Mossin, J. (1966), ‘Equilibrium in a capital market’, Econometrica, 34, 768–83. Mowery, D. (1983), ‘The relationship between intrafirm and contractual forms of industrial research in American manufacturing, 1900–1940’, Explorations in Economic History, 20, 351–74. Mowery, D.C. and N. Rosenberg (1989), Technology and the Pursuit of Economic Growth, Cambridge, New York, Port Chester: Cambridge University Press. Mowery, D., J. Oxley and B. Silverman (1996), ‘Strategic alliances and interfirm knowledge transfer’, Strategic Management Journal, 17, 77–91. Mowery, D.C., R.R. Nelson, B.N. Sampat and A.A. Ziedonis (2001), ‘The growth of patenting and licensing by US universities: an assessment of the effects of the Bayh-Dole act of 1980’, Research Policy, 30 (1), 99–119. Muffatto, M. and G. Giardina (2003), ‘Innovazioni nei processi di ricerca in campo farmaceutico’, Economia & Management, 6, 107–21. Muller, E. and A. Zenker (2001), ‘Business services as actors of knowledge transformation: the role of KIBS in regional and national innovation systems’, Research Policy, 30 (9), 1501–1516. Munari, F. and R. Oriani (2005), ‘Privatization and economic returns to R&D investments’, Industrial and Corporate Change, 14 (1), 61–91. Munari, F., R. Oriani, and M. Sobrero (2005), ‘Do owner identity and financial markets affect R&D investments? An analysis of European firms’ in: M.K. Weawer, 2005 Annual Meeting Proceedings, Barcliff Manor, Academy of Management. Mutti, J. and B. Yeung (1996), ‘Section 337 and the protection of intellectual property in the United States: the complainants and the impact’, Review of Economics and Statistics, 78 (3), 510–20. Myers, S.C. (1977), ‘Determinants of corporate borrowing’, Journal of Financial Economics, 5 (2), 147–75. Myers, S.C. and S. Majad (1990), ‘Abandonment value and project life’, Advances in Futures and Options Research, 4, 1–21. Myers, S.C. and N.S. Majluf (1984), ‘Corporate financing and investment decisions when firms have information that investors do not have’, Journal of Financial Economics, 13 (2), 187–221. National Science Foundation (2008), ‘National patterns of R&D resources: 2007 data update’, available at http://www.nsf.gov/statistics/nsf08318/. Nayak, P.R. (1987), ‘Measuring product creation effectiveness’, Journal of Business Strategy, 13, 48–52. Nayak, P.R. (1992), ‘Measuring Product Creation Effectiveness’, Journal of Business Strategy, 13 (6), 48–52. Nelson, R.R. (1959), ‘The simple economics of basic scientific research’, Journal of Political Economy, 67 (3), 297. Nelson, R.R. (2004), ‘The market economy, and the scientific commons’, Research Policy, 33, 455–71

282

Evaluation and performance measurement of R&D

Nelson, R.R. and E.S. Phelps (1966), ‘Investment in humans, technological diffusion, and economic growth’, American Economic Review, 56, 69–75. Nelson, R.R. and S.G. Winter (1982), An Evolutionary Theory of Economic Change, London, UK and Cambridge, MA: The Belknap Press of Harvard University Press. Nemhauser, G. and Z. Ullmann (1969), ‘Discrete dynamic programming and capital allocation’, Management Science, 15, 494–505. Nevens, T.M., G.L. Summe and B. Uttal (1990), ‘Commercializing technology: what the best companies do’, Harvard Business Review, 68 (4), 154–63. Niosi, J. and M. Zhegu (2005), ‘Aerospace clusters: local or global knowledge spillovers?’, Industry and Innovation, 12 (1), 1–25. Nixon, B. (1998), ‘Research and development performance measurement: a case study’, Management Accounting Research, 9, 329–55. Nochur, K.S. and T.J. Allen (1992), ‘Do nominated boundary spanners become effective technological gatekeepers?’, IEEE Transactions on Engineering Management, 39 (3), 265–9. Nonaka, I. (1994), ‘A dynamic theory of organizational knowledge creation’, Organizational Science, 5, 14–37. Nordhaus, W. (1969), Invention, Growth and Welfare, Cambridge: MIT Press. Noteboom, B. (1999), Inter-firm Alliances. Analysis and Design, London: Routledge. Noyons, E.C.M., M. Luwel and H.F. Moed (1998), ‘Assessment of Flemish R&D in the field of information technology; a bibliometric evaluation based on publication data and patent data, combined with OECD research input statistics’, Research Policy, 27, 285–300. O’Brien, T.J. (2005), ‘Foreging exchange and cross-border valuation’, Journal of Applied Corporate Finance, 16 (2–3), 147–54. O’Dell, C. (1989), ‘Team play, team pay: new ways of keeping scores’, Across the Board, 38–45. O’Gorman, C. and M. Kautonen (2004), ‘Policies to promote new knowledgeintensive industrial agglomerations’, Entrepreneurship and Regional Development, 16 (6), 459–79. Odagiri, H., Y. Nakamura and M. Shibuya (1997), ‘Research consortia as a vehicle for basic research: the case of a fifth generation computer project in Japan’, Research Policy, 26 (2), 191–207. OECD (1997), The Evaluation of Scientific Research: Selected Experiences, Paris: OECD Publications. OECD (2002), Proposed Standard Practise for Surveys of Research and Experimental Development, Frascati Manual, sixth edition. Ojanen, V. and O. Vuola (2006), ‘Coping with the multiple dimensions of R&D performance analysis’, International Journal of Technology Management, 33, 279–90. Oriani, R. and M. Sobrero (2003), ‘A meta-analytic study of the relationship between R&D investments and corporate value’, in Mario Calderini, Paola Garrone and Maurizio Sobrero (eds), Corporate Governance, Market Structure and Innovation, Cheltenham, UK: Edward Elgar, 177–99. Oriani, R. and M. Sobrero (2008), ‘Uncertainty and the market valuation of R&D within a real options logic’, Strategic Management Journal, 29 (4), 343–61. Ormala, E. (1986), Analysis and Supporting R&D Project Evaluation, Espoo: Technical Research Centre of Finland.

References

283

Orsenigo, L., F. Pammolli and M. Riccaboni (2001), ‘Technological change and network dynamics – lessons from the pharmaceutical industry’, Research Policy, 30 (3), 485–508. Ortt, J.R. and R. Smits (2006), ‘Innovation management: different approaches to cope with the same trend’, International Journal of Technology Management, 34 (3/4), 296–318. Osawa, Y. and M. Murakami (2002), ‘Development and application of a new methodology of evaluating industrial R&D projects’, R&D Management, 21 (1), 79–85. Oswald, D.R. (2008), ‘The determinants and value relevance of the choice of accounting for research and development expenditures in the United Kingdom’, Journal of Business Finance & Accounting, 35 (1–2), 1–24. Oswald, D.R. and P. Zarowin (2007), ‘Capitalization of R&D and the informativeness of stock prices’, European Accounting Review, 16 (4), 703–26. Ouchi, W. (1979), ‘A conceptual framework for the design of organizational control mechanisms’, Management Science, 25 (9), 833–48. Packendorff, J. (2002), ‘The temporary society and its enemies: projects from an individual perspective’, in K. Sahlin-Andersson and A. Söderholm (eds), Beyond Project Management: New Perspectives on the Temporary-Permanent Dilemma, Malmö: Liber Ekonomi; Copenhagen Business School Press, 39–58. Paddock, J.L., D.R. Siegel and J.L. Smith (1988), ‘Option valuation of claims on real assets: the case of offshore petroleum leases’, Quarterly Journal of Economics, 103, 479–508. Pakes, A. (1985), ‘On patents, R&D, and the stock market rate of return’, Journal of Political Economy, 93, 390–409. Pakes, A. and Z. Griliches (1984), ‘Patents and R&D at firm level: a first look’, in Zvi Griliches (ed.), R&D, Patents, and Productivity, Chicago: The University of Chicago Press. Pappas, R.A. and D.S. Remer (1985), ‘Measuring R&D productivity’, ResearchTechnology Management, 28, 15–22. Patel, P. and K. Pavitt (1997), ‘The technological competencies of the world’s largest firms: complex and path-dependent, but not much variety’, Research Policy, 26 (2), 141–56. Pavitt, K. (1987), ‘The objectives of technology policy’, Science and Public Policy, 14, 182–8, reprinted in K. Pavitt (ed.) (1999), ‘The nature of technology’ in Technology, Management and Systems of Innovation, Cheltenham: Edward Elgar, pp. 3–14. Pearson, A.W., W.A. Nixon and I.C. Kerssens-van Drongelen (2000), ‘R&D as a business – what are the implications for performance measurement?’, R&D Management, 30 (4), 355–66. Peerenboom, J.P., W.A. Buehring and T.W. Joseph (1989), ‘Selecting a portfolio of environmental programs for a synthetic fuel facility’, Operations Research, 37, 689–99. Pelled, L.H. and P.S. Adler (1994),’Antecedents of intergroup conflict in multifunctional product development teams: a conceptual approach’, IEEE Transactions on Engineering Management, 41 (1), 21–8. Pennings, H.P.G. and L.J.O. Lint (1997), ‘The option value of advanced R&D’, European Journal of Operational Research, 103 (1), 83–94. Perry, P.M. (2002), ‘Battle for the best’, Research-Technology Management, 45 (2), 17–21.

284

Evaluation and performance measurement of R&D

Pessemier, E.A. and N.R. Baker (1971), ‘Project and program decisions in research and development’, R&D Management, 2 (1), 3–14. Peterson, J. (1993), ‘Assessing the performance of European collaborative research-and-development policy – the case of Eureka’, Research Policy, 22 (3), 243–64. Petroni, A. (2000), ‘Myths and misconceptions in current engineers’ management practices’, Team Performance Management, 6, 15–24. Phillips, R., K. Nearley and T. Broughton (1999), ‘A comparative study of six stage-gate approaches to product development’, Integrated Manufacturing Systems, 10 (5), 289–97. Pigou, A.C. (1932), The Economics of Welfare, London: Macmillan. Pilorget, L. (1995), ‘Evaluation of 2 support programs promoting international technology-transfer between SMEs’, International Journal of Technology Management, 10 (7–8), 867–78. Piscitello, L. (2000), ‘Relatedness and coherence in technological and product diversification of the world’s largest firms’, Structural Change and Economic Dynamics, 11 (3), 295–315. PMBOK (2000), A Guide to the Project Management Body of Knowledge, Newtown Square, Pennsylvania, USA: Project Management Institute. Poh, K.L., B.W. Ang and F. Bai (2001), ‘A comparative analysis of R&D project evaluation methods’, R&D Management, 31, 63–76. Polanyi, M. (1966), The Tacit Dimension, London: Routledge & Kegan Paul. Porter, L. and E. Lawler (1968), Managerial Attitudes and Performance, Homewood, IL: Dorsey Press. Powell, S. (2003), ‘Accounting for intangible assets: current requirements, key players and future directions’, European Accounting Review, 12 (4), 797–811. Powell, W.W., K.W. Kline. Koput and L. Smith-Doerr (1996), ‘Interorganizational collaboration and the locus of innovation: networks of learning in biotechnology’, Administrative Science Quarterly, 41, 116–45. Prencipe, A. (2000), ‘Breadth and depth of technological capabilities in complex product systems: the case of the aircraft engine control system’, Research Policy, 29 (7–8), 895–911. Presley, A. and D. Liles (2000), ‘R&D validation planning: a methodology to link technical validations to benefits measurement’, R&D Management, 30 (1), 55–65. Pritchard, R.D. (1990), Measuring and Improving Organizational Productivity: a Practical Guide, New York: Praeger. Projan, S.J. (2003), ‘Why is Big Pharma getting out of antibacterial drug discovery?’, Current Opinion in Microbiology, 6, 427–30. Quinn, J.B. (2000), ‘Outsourcing innovation: the new engine of growth’, Sloan Management Review, 41 (4), 13–28. Quintas, P., D. Wield and D. Massey (1992), ‘Academic-industry links and innovation – questioning the science park model’, Technovation, 12 (3), 161–75. Raftery, J. (1994), Risk Analysis in Project Management, London: E&FN Spon. Rahaman, S. and L.C. Frair (1984), ‘A hierarchical approach to electric utility planning’, Energy Research, 8, 185–96. Raiffa, H. (1968), Decision Analysis, New York: Random House. Rajan, R.G. and L. Zingales (2003), ‘Banks and markets: the changing character of the European finance’, Working Paper 9595, National Bureau of Economic Research.

References

285

Ramnath, S., S. Rock and P. Shane (2008), ‘The financial analyst forecasting literature: a taxonomy with suggestions for further research’, International Journal of Forecasting, 24 (1), 34–75. Rank, D. and D. Williams (1999), ‘Partial benefit/cost in the evaluation of the Canadian Networks of Centres of Excellence’, Evaluation and Program Planning, 22 (1), 121–9. Reed, R. and R.J. DeFillippi (1990), ‘Causal ambiguity, barrier to imitation, and sustainable competitive advantage’, Academy of Management Review, 15, 88–102. Reimers, B.D. (2001), ‘Keep talent from taking flight’, Network Computing, August 6, 42–5. Rengarajan, S. and P. Jagannathan (1997), ‘Project selection by scoring for a large R&D organisation in a developing country’, R&D Management, 27 (2), 155–64. Robert, C.P. and G. Casella (2004), Monte Carlo Statistical Methods, New York: Springer Texts in Statistics. Roberts, E.B. (1988), ‘Managing invention and innovation’, Research-Technology Management, 31 (1), 13–29. Roberts, E.B. (2001), ‘Benchmarking global strategic management of technology’, Research-Technology Management, 44 (2), 25–36. Roberts, E. and A. Fusfeld (1981), ‘Staffing the innovative technology-based organization’, Sloan Management Review, Spring, 19–34. Robichek, A. and S.C. Myers (1965), Optimal Financing Decisions, Englewood Cliffs, NJ: Prentice-Hall. Robichek, A. and S.C. Myers (1966), ‘Conceptual problems in the use of riskadjusted discount rates’, Journal of Finance, 21, 727–30. Roessner, J.D. (2002), ‘Outcome measurement in the USA: state of the art’, Research Evaluation, 11 (2), 85–93. Rogers, M. (2001), ‘Firm performance and investment in R&D and IP’, Melbourne Institute Working Paper No. /01, University of Melbourne. Romer, P. (1986), ‘Increasing returns and long-run growth’, Journal of Political Economy, 94 (5), 1002–37. Romer, P. (1990), ‘Endogenous technical change’, Journal of Political Economy, 98 (5), 71–103. Romer, P. (1994), ‘The origins of endogenous growth’, Journal of Economic Perspectives, 8, 3–22. Ronen, J. (2001), ‘On R&D capitalization and value relevance: a commentary’, Journal of Accounting and Public Policy, 20 (3), 241–54 Ronen, J. and V. Yaari (eds) (2008), Earnings Management: Emerging Insights in Theory, Practice, and Research, New York, US: Springer Series in Accounting Scholarship. Rosenberg, N. (1976), Perspectives on Technology, Cambridge: Cambridge University Press. Rosenberg, N. (1982), Inside the Black Box: Technology and Economics, Cambridge: Cambridge University Press. Rosenberg, N. (1986), ‘The impact of technological innovation: a historical review’, in Ralph Landau and Nathan Rosenberg (eds), The Positive Sum Strategy: Harnessing Technology for Economic Growth, Washington: National Academy Press, pp. 17–32. Rosenberg, N. (1990), ‘Why do firms do basic research (with their own money)?’, Research Policy, 19 (2), 165–74.

286

Evaluation and performance measurement of R&D

Rosenberg, N. (1996), ‘Uncertainty and technological change’, in R. Landau, T. Taylor, G. Wright (eds), The Mosaic of Economic Growth, Stanford: Stanford University Press. Rosenkopf, L. and A. Nerkar (2001), ‘Beyond local research: boundary-spanning, exploration, and impact in the optical disk industry’, Strategic Management Journal, 22 (4), 287–306. Roth, L.M. (1982), A Critical Examination of the Dual Ladder Approach to Career Advancement, New York: Columbia University Press. Rothwell, R. and W. Zegveld (1985), Reindustrialization and Technology, Harlow, UK: Longman. Rubinstein, R.Y. (1981), Simulation and the Monte Carlo Method, New York: John Wiley & Sons. Russo, B. (2004), ‘A cost-benefit analysis of R&D tax incentives’, Canadian Journal of Economics – Revue Canadienne D’Economique, 37 (2), 313–35. Ryan, P.A. and G.P. Ryan (2002), ‘Capital budgeting practices of the Fortune 1000: how have things changed’, Journal of Business and Management, 8 (4), 389–419. Rycroft, R. and D. Kash (2002), ‘Path dependence in the innovation of complex technologies’, Technology Analysis & Strategic Management, 14 (1), 21–35. Saaty, T.L. (1980), The Analytic Hierarchy Process, New York: McGraw-Hill. Saaty, T.L. (1996), Decision Making with Dependence and Feedback: The Analytic Network Process, Pittsburgh: RWS Publications. Sakkab, N.Y. (2002), ‘Connect & Develop complements Research & Develop at P&G’, Research-Technology Management, 45 (2), 38–45. Sampat, B.N. (2006), ‘Patenting and US academic research in the 20th century: the world before and after Bayh-Dole’, Research Policy, 35 (6), 772–89. Sampat, B.N., D.C. Mowery and A.A. Ziedonis (2003), ‘Changes in university patent quality after the Bayh-Dole act: a re-examination’, International Journal of Industrial Organization, 21 (9), 1371–90. Sandstrom, J. and J. Toivanen (2002), ‘The problem of managing product development engineers: can the balanced scorecard be an answer?’, International Journal of Production Economics, 78, 79–90. Savage, L.J. (1954), The Foundations of Statistics, New York: John Wiley. Schall, L.D., G.L. Sundem and W.R. Jr. Geijsbeek (1978), ‘Survey and analysis of capital budgeting methods’, Journal of Finance, 33 (1), 281–88. Scherer, F., D. Harhoff and J. Kukies (1998), ‘Uncertainty and the size distribution of reward from technological innovation’, Journal of Evolutionary Economics, 10 (1), 175–200. Schoening, N.C., W.E. Souder, J. Lee and R. Cooper (1998), ‘The influence of government science and technology policies on new product development in the USA, UK, South Korea and Taiwan’, International Journal of Technology Management, 15 (8), 821–35. Schroder, H. (1975), ‘The quality of subjective probabilities of technical success in R and D’, R&D Management, 6, 15–21. Schwartz, E.S. and M. Moon (2000), ‘Evaluating research and development investment’, in Michael J. Brennan and Lenos Trigeorgis (eds) Project Flexibility, Agency, and Competition, Oxford: Oxford University Press, pp. 85–106. Shane, S. (2001), ‘Technological regimes and new firm formation’, Management Science, 47 (9), 1173–90. Shane, S. (2004), ‘Encouraging university entrepreneurship? The effect of the

References

287

Bayh-Dole Act on university patenting in the United States’, Journal of Business Venturing, 19 (1), 127–51. Shapira, P., J. Youtie and J.D. Roessner (1996), ‘Current practices in the evaluation of US industrial modernization programs’, Research Policy, 25 (2), 185–214. Sharpe, W.F. (1964), ‘Capital asset prices: a theory of market equilibrium under conditions of risk’, Journal of Finance, 19 (3), 425–42. Shepard, H.A. (1958), ‘The dual hierarchy in research’, Research Management, 1 (3), 177–87. Shin, C.O., S.H. Yoo, S.J. Kwak (2007), ‘Applying the analytic hierarchy process to evaluation of the national nuclear R&D projects: the case of Korea’, Progress in Nuclear Energy, 49 (5), 375–84. Simons, R. (2000), Performance Measurement and Control Systems for Implementing Strategy, Englewood Cliffs, New Jersey: Prentice Hall. Sivathanu, P.A. and R.K. Srinivasa (1996), ‘Performance monitoring in R&D projects’, R&D Management, 26, 57–65. Smith, W.A. (1995), ‘Evaluating research, technology and development in Canadian industry: meeting the challenges of industrial innovation’, Scientometrics, 34 (3), 527–39. Söderlund, J. (2005), ‘Developing project competence: empirical regularities in competitive project operations’, International Journal of Innovation Management, 9 (4), 451–80. Solow, R.M. (1957), ‘Technical change and the aggregate production function’, Review of Economics and Statistics, 39, 312–20. Somaya, D.J. and D. Teece (2006), ‘Patents, licensing and entrepreneurship: effectuating innovation in multi-invention contexts’, in E. Sheshinski, R.J. Strom and W.J. Baumol (eds), Entrepreneurship, Innovation and the Growth of FreeMarket Economies, Princeton: Princeton University Press. Souder, W.E. (1978), ‘A system for using R and D project evaluation models’, Research Management, 21, 29–37. Sougiannis, T. (1994), ‘The accounting based valuation of corporate R&D’, Accounting Review, 69 (1), 44–68. Spiegelhalter, D.J., N.G. Best, B.P. Carlin and A. Van Der Linde (2002), ‘Bayesian measures of model complexity and fit’, Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64 (4), 583–639. Stainer, A. and B. Nixon (1997), ‘Productivity and performance measurement in R&D’, International Journal of Technology Management, 13 (5/6), 486–96. Stam, A. and A.P. Duarte Silva (1997), ‘Stochastic judgments in the AHP: the measurement of rank reversal probabilities’, Decision Sciences, 28 (3), 655–88. Stiglitz, J.E. and A. Weiss (1981), ‘Credit rationing in markets with imperfect information’, American Economic Review, 71, 393–410. Stokes, D. (1997), Pasteur’s Quadrant: Basic Science and Technological Innovation, Washington DC: Brookings Institution. Storey, D.J. and B.S. Tether (1998), ‘Public policy measures to support new technology-based firms in the European Union’, Research Policy, 26 (9), 1037–57. Stuart, T. (2000), ‘Interorganizational alliances and the performance of firms: a study of growth and innovative rates in a high-technology industry’, Strategic Management Journal, 21, 791–811. Stulz, R. (1982), ‘Options on the minimum or the maximum of two risky assets’, Journal of Financial Economics, 10 (2), 161–85.

288

Evaluation and performance measurement of R&D

Stummer, C. and K. Heidenberger (2003), ‘Interactive R&D portfolio analysis with project interdependencies and time profiles of multiple objectives’, IEEE Transactions on Engineering Management, 50 (2), 175–83. Sundaram, A.K., T.A. John and K. John (1996), ‘An empirical analysis of strategic competition and firm values: the case of R&D competition’, Journal of Financial Economics, 40 (3), 459–86. Suomala, P. (2003), ‘Multifaceted new product development performance: survey on utilization of performance measures in Finnish industry’, Proceedings of the 2nd Workshop on Performance Measurement and Management Control, Nice, France, 18–19 September. Szakonyi, R. (1994a), ‘World-Class R&D management’, R&D Innovator, 3 (11). Szakonyi, R. (1994b), ‘Measuring R&D effectiveness – I’, Research-Technology Management, 37 (2), 27–32. Szakonyi, R. (1995), ‘Measuring R&D effectiveness – I’, Research-Technology Management, 37, 27–32. Szewczyck, S., G.P. Tsetsekos and Z. Zantout (1996), ‘The valuation of corporate R&D expenditures: evidence from investment opportunities and free cash flow’, Financial Management, 25 (1), 105–10. Takalo, T. and T. Tanayama (2008), ‘Adverse selection and innovation financing: is there need for R&D subsidies?’, PRIME Conference on the Dynamics of Science-based Entrepreneurship, Sestri Levante, Italy, 31 March–2 April 2008. Tanaka, Y. (1995), ‘Effect of Japan advisory councils on science and technology policy’, Technology in Society, 17 (2), 159–73. Tassey, G. (1997), The Economics of R&D Policy, Westport, Connecticut: Quorum Books. Tassey, G. (1999), ‘R&D policy models and data needs’, APPAM 1999 Research Conference, November 4, 1999, Washington DC, USA. Teece, D.J. (1986), ‘Profiting from technological innovation: implications for integration, collaboration, licensing and public policy’, Research Policy, 15 (6), 285–305. Teece, D.J. (2006), ‘Reflection on “Profiting from Innovation” ‘, Research Policy, 35 (8), 1131–46. Teece, D.J. (ed.) (2000), Managing Intellectual Capital: Organizational, Strategic and Policy Dimensions, Oxford, UK: Oxford University Press. Teece, D.J., G. Pisano and A. Shuen (1997), ‘Dynamic capabilities and strategic management’, Strategic Management Journal, 18 (7), 509–33. Thompson, P.H. and G.W. Dalton (1976), ‘Are R&D organizations obsolete?’, Harvard Business Review, 54 (6), 105–16. Toivanen, O., P. Stoneman and D. Bosworth (2002), ‘Innovation and market value of UK firms, 1989–1995’, Oxford Bulletin of Economics and Statistics, 64 (1), 39–61. Toletti, G. and G. Noci (2000), ‘Selecting quality-based programmes in small firms: a comparison between the fuzzy linguistic approach and the analytic hierarchy process’, International Journal of Production Economics, 67, 113–33. Tourinho, O.A.F. (1979), The Valuation of Natural Resources: An Option Pricing Approach, Ph.D. dissertation, University of California, Berkeley. Triantaphyllou, E. and A. Sanchez (1997), ‘A sensitivity analysis approach for some deterministic multi-criteria decision making methods’, Decision Sciences, 28 (1), 151–94. Triendl, R. (1998), ‘Rewarding innovators’, Research-Technology Management, 41 (6), 4.

References

289

Trigeorgis, L. (1988), ‘A conceptual options framework for capital budgeting’, Advances in Futures and Options Research, 2 (1), 145–67. Trigeorgis, L. (1993), ‘The nature of option interactions and the valuation of investments with multiple real options’, Journal of Financial and Quantitative Analysis, 28 (1), 1–20. Trigeorgis, L. (1996), Real Options; Managerial Flexibility and Strategy in Resource Allocation, Cambridge, MA and London, UK: MIT Press. Tripsas, M., S. Schrader and M. Sobrero (1995), ‘Discouraging opportunistic behavior in collaborative research-and-development – a new role for government’, Research Policy, 24 (3), 367–89. Tsipouri, L.J. (1991), ‘Effects of EC research-and-development policy on Greece – some thoughts in view of the Stride Program’, Scientometrics, 21 (3), 403–16. Tsui, A.S., T.D. Egan and C.A. O’Reilly III (1992), ‘Being different: relational demography and organizational attachment’, Administrative Science Quarterly, 37, 549–79. Twiss, B. (1986), Managing Technological Innovation, London: Pitman. Tylecote, A. and P. Ramirez (2006), ‘Corporate governance and innovation. The UK compared with the US and “insider” economies’, Research Policy, 35 (1), 160–80. United Nations (2005), ‘UNCTAD survey on the internationalisation of R&D. Current patterns and prospects on the internationalisation of R&D’, available at www.unctad.org/wir. Uzun, A. (2006), ‘Science and technology policy in Turkey. National strategies for innovation and change during the 1983–2003 period and beyond’, Scientometrics, 66 (3), 551–59. Van Den Ende, J., N. Wijnberg and A. Meijer (2001), ‘Public policy and innovative capabilities: the case of Philips’ IT activities’, Technology Analysis & Strategic Management, 13 (3), 389–405. Van Leeuwen, T.N., L.J. Van Der Wurff and A.F.J. Van Raan (2001), ‘The use of combined bibliometric methods in research funding policy’, Research Evaluation, 10 (3), 195–201. Van Wegberg, M. (2004), ‘Standardization process of systems technologies: creating a balance between competition and cooperation’, Technology Analysis & Strategic Management, 16 (4), 457–78. Verbeek, A., K. Debackere, M. Luwel and E. Zimmermann (2002), ‘Measuring progress and evolution in science and technology – I: The multiple uses of bibliometric indicators’, International Journal of Management Review, 4 (2), 179–211. Verrecchia, R. (1983), ‘Discretionary disclosure’, Journal of Accounting and Economics, 5 (12), 365–80. Verrecchia, R. (1990), ‘Information quality and discretionary disclosure’, Journal of Accounting and Economics, 12, 365–80. Verrecchia, R. (1999), ‘Disclosure and the cost of capital: a discussion’, Journal of Accounting and Economics, 26 (1–3), 271–83. Verspagen, B. (2006), ‘University research, intellectual property rights and European innovation systems’, Journal of Economic Surveys, 20 (4), 607–32. Viscovich, M. and C. Krogsgaard (2006), ‘Valuation of a new drug candidate using decision tree analysis exemplified by NB S101 of Osteologix A/S’, Biotech Business Working Paper No. 07-2006. Von Hippel, E. (1988), The Sources of Innovation, New York: Oxford University Press.

290

Evaluation and performance measurement of R&D

Von Hippel, E. (1994), ‘Sticky information and the locus problem solving: implications for innovation’, Management Science, 40 (4), 429–39. Von Hippel, E. (1998), ‘Economics of product development by users: the impact of “sticky” local information’, Management Science, 44 (5), 629–44. Von Hippel, E. (2008), ‘Users as sources of invention’, in B.H. Hall and N. Rosenberg (eds), Handbook of Economics of Technological Change, Amsterdam: Elsevier B.V. Press. Vroom, V. (1964), Work and Motivation, New York: John Wiley & Sons. Wallsten, S.J. (2000), ‘The effects of government-industry R&D programs on private R&D: the case of the Small Business Innovation Research program’, Rand Journal of Economics, 31 (1), 82–100. Wang, M.J. and G.S. Liang (1993), ‘A fuzzy multi-criteria decision-making approach for robot selection’, Robotics & Computer-Integrated Manufacturing, 10 (4), 267–74. Watanabe, C., M. Kishioka and A. Nagamatsu (2004), ‘Effect and limit of the government role in spurring technology spillover – a case of R&D consortia by the Japanese government’, Technovation, 24 (5), 403–20. Weingartner, H.M. (1966), ‘Capital budgeting of interrelated projects: survey and synthesis’, Management Science, 12, 485–516. Werner, B.M. and W.E. Souder (1997), ‘Measuring R&D performance: state of the art’, Research-Technology Management, 40 (2), 34–42. White, R.W. (1959), ‘Motivation reconsidered: the concept of competence’, Psychological Review, 66 (5), 297–331. Widhalm, C., M. Topolnik, A. Kopcsa, E. Schiebel and M. Weber (2001), ‘Evaluating patterns of co-operation: application of a bibliometric visualisation tool to the Fourth Framework Programme and the Transport Research Programme’, Research Evaluation, 10 (2), 129–40. Willner, R. (1995), ‘Valuing start-up venture growth options’, in L. Trigeorgis (ed.), Real Options in Capital Investment: Models, Strategies, and Applications, Westport: Praeger Publishers, pp. 221–40. Wind, J. and V. Mahajan (1997), ‘Issues and opportunities in new product development: an introduction to the special issue’, Journal of Marketing Research, 34 (1), 1–12. Womack, J.P. and D.T. Jones (1996), ‘From lean production to the lean enterprise’, IEEE Management Review, Winter, 38–48. Woolridge J.R. and C.C. Snow (1990), ‘Stock market reaction to strategic investment decisions’, Strategic Management Journal, 11 (5), 353–63. Wu, Y.H. (2005), ‘The effects of state R&D tax credits in stimulating private R&D expenditure: a cross-state empirical analysis’, Journal of Policy Analysis and Management, 24 (4), 785–802. Wyatt, A. (2008), ‘What financial and non-financial information on intangibles is value relevant? A review of the evidence’, Accounting and Business Research, 38 (3), 217–56. Yin, R.K. (2003), Case Study Research: Design and Methods, London: Sage. Youtie, J., B. Bozeman and P. Shapira (1999), ‘Using an evaluability assessment to select methods for evaluating state technology development programs: the case of the Georgia Research Alliance’, Evaluation and Program Planning, 22 (1), 55–64. Zadeh, L.A. (1965), ‘Fuzzy sets’, Information Control, 8, 338–53. Zantout, Z.Z. (1997), ‘A test of the debt-monitoring hypothesis: the case of corporate R&D expenditures’, Financial Review, 32 (1), 21–48.

References

291

Zantout, Z.Z. and G.P. Tsetsekos (1994), ‘The wealth effects of announcements of R&D expenditure increases’, The Journal of Financial Research, 17, 205–16. Zeleny, M. (1982), Multiple Criteria Decision Making, New York: McGraw-Hill. Zhang, J., C. Baden-Fuller and V. Mangematin (2007), ‘Technological knowledge base, R&D organization structure and alliance formation: evidence from the biopharmaceutical industry’, Research Policy, 36 (4), 515–28. Zhu, P.F., W.M. Xu and N. Lundin (2006), ‘The impact of government’s fundings and tax incentives on industrial R&D investments – empirical evidences from industrial sectors in Shanghai’, China Economic Review, 17 (1), 51–69. Zimmermann, H.J. (1978), ‘Fuzzy programming and linear programming with several objective functions’, Fuzzy Sets System, 1, 45–55. Zimmermann, H.J. (1991), Fuzzy Set Theory and Its Applications, London: Kluwer Academic Publishers. Zirger, B.J. and J.L. Hartley (1996), ‘The effects of acceleration techniques on product development time’, IEEE Transactions of Engineering Management, 43, 143–52.

Index 360-degree feedback 115 ACARE technology platform 207–10 context 203–7 accounting standards generating information asymmetries 166–7, 169–70 see also IAS/IFRS adoption case study aerospace sector see ACARE technology platform AHP (analytical hierarchy process) 59–61, 62 AIRI (Italian Association for Industrial Research) 48 Al-Mazidi, S. 58 ambassador role 118 analytic network process (ANP) 61–2, 63 analytical hierarchy process (AHP) 59–61, 62 Anderson, K. 20 ANP (analytic network process) 61–2, 63 applied research 1, 220 project evaluation techniques 80–81 R&D policy for 223–5 see also Company ‘A’ apprenticeship stage 116 Armstrong, M. 54 Arora, A. 6 Arrow paradox 224 Azzone, G. 64, 68, 69 Baker, N.R. 54 Balachandra, R. 53 balanced managers 123 Balanced Scorecard (BSC) approach 21, 28, 35 Barsky, N.P. 17, 20, 21, 22, 23, 26, 28

basic research 1, 220 project evaluation techniques 80 R&D policy for 221–3 see also Company ‘A’; oriented basic research/strategic research Bayh-Dole Act 244 Bayus, B.L. 2 Belton, V. 62 Bertelè, U. 68, 69 Beskese, A. 63 bibliometric indicators 18–19 Bilderbeek, J. 15, 22, 23, 26 Black and Scholes Model 76 Black, F. 68 Bourne, M. 20 Bowon, K. 23 Bozbura, F.T. 54, 65 Bradbury, F.R. 53 brain attraction 115 Brealey, R.A. 54, 65 Bremser, W.G. 17, 20, 21, 22, 23, 26, 28 Briggs, A.H. 73 Brown, M.G. 15, 17, 19, 20, 22 Brown, W. 26 BSC (Balanced Scorecard) approach 21, 28, 35 Burch, J. 23 business process perspective 29, 35–6 Calantone, R.J. 54, 62 CAPM (capital asset pricing model) 66–7 Cardus, D.M. 58 career paths 116–18 project intensification 123 Carlsson, C. 54 Carr model 76 certainty equivalent NPV 68–9 champion role 112 Chan, S.H. 4 Chapman, D. 65, 69

293

294

Evaluation and performance measurement of R&D

Chatterji, D. 5 Chesbrough, H. 6 Chiesa, V. 5, 16, 21, 22, 23, 24, 26, 28, 30, 32, 39, 47, 53, 68, 69 Cho, E. 22, 23, 26 ‘Clean Sky’ Joint Technology Initiative 209–10 Clemen, R.T. 54, 71 clusters 189–91 see also technology platforms Coccia, M. 20, 22 Collier, D.W. 18 Company ‘A’ 27–8 available resources 41 competitive and technology strategy 39–40 context 38–9 overview 38 performance measurement system 44 control objects 43 dimensions of performance 42 indicators 42 measurement process 43, 45 objectives 41 R&D organization and management 40 type of R&D activity 40–41 competence development 123 contextual perspective, firm level 24–5 contract research organizations (CRO) 39 control objects 23 Cook, A. 4, 18, 21, 22, 23, 28, 33 Cooper, C. 65, 69 Cooper, R.G. 4, 19, 21 Cordero, R. 4 corporate governance and, stock market valuation of R&D 160–63 cosmopolitans (R&D workers) 117 cost of capital/WACC (Weighted Average Cost of Capital) 65–6 CRO (contract research organizations) 39 cross-functional teams 118–19 customer perspective 28 Davila, T. 21, 23, 26 Dean, B.V. 53, 56 Debackere, K. 18, 20 decision tree analysis (DTA) 71–4

Deeds, D.L. 48 demographic diversity, in R&D laboratories 120–21 Development Costs document 93, 97, 101 Di Masi, J.A. 72 dimensions, firm level 20–21, 22 disclosure paradox 224 discount rate calculation, in NPV 65–6 DMADV (Define, Measure, Analyse, Design) roadmap 96, 99 Doctor, R.N. 65, 71 documents, project approval 93 Development Costs 93, 97, 101 Project Charter 93, 94 Project Scope 93, 95 Project Submittal Form 93, 98, 102 Revenue & Margin Calculations 93, 95, 101 Risk Management Plan 93, 96 Donegan, H.A. 59 Driva, H. 15, 17, 20, 22, 23, 24, 29 drug development 71, 73–4 DTA (decision tree analysis) 71–4 dual career system 117 dual ladder 117 dual use 205–6 Duarte Silva, A.P. 65 earnings before interest, taxes, depreciation and amortization (EBITDA), and R&D 143 Easton, A. 54 Eisenhardt, K.M. 45 Emmanuel, C. 26 empirical analyses methodological details multiple case study 45–8 survey study 48–50 entrepreneurial orientation 117 European Technology Platforms (ETPs) policy making 201–3 see also ACARE technology platform Expectancy Model 110 experimental development 220 explicit knowledge 120 external consistency 31

Index external sources of technology, increased reliance upon 5 extrinsic rewards 114 Farris, G. 4 Farrukh, C. 58 feedback see 360-degree feedback Feller, I. 4 Fifth Generation Computer Systems Project 249 financial analysts, role in R&D information flows 176–7 financial markets’ perspective 4, 143–5 financial perspective 28, 29 Fink, A. 50 firm perspective 4 literature 16–25 firms R&D policies for financial policies 238–41 legislative policies 241–2 Foray, D. 4 Forman, E.H. 62 Frair, L.C. 59 Frascati Manual 220 Frattini, F. 5, 20, 22, 24, 26, 29, 30, 32 Freeman, C. 4, 54 Fullér, R. 54 fuzzy logic 63 applied in R&D project evaluation 63–5 Gass, S.I. 62 gatekeeper role 112 Gear, T. 62 Gee, R.E. 18 Georghiou, L.G. 55 Geske model 76 Ghosn, A.A. 58 Giardina, G. 38, 39 GlaxoSmithKline (GSK) 38 Gobeli, D. 26 Godener, A. 15, 23, 24 Gordon, L.A. 24, 31 Graebner, M.E. 45 Grant-Muller, S.M. 69 Griffin, A. 21, 22, 23, 24, 26 growth, and technology 218 GSK (GlaxoSmithKline) 38

295

guarding role 118 Gupta, A.K. 2 Hajek, P. 64 Hall, D.L. 55 Hand, J.R.M. 4 Haour, G. 5, 39 hard objectives 32–6 Hauser, J.R. 15, 24, 26 Henriksen, A.D. 58 Henrion, M. 72 Heungshik, O. 23 heuristics models 86 hiring process 111–13 Ho, S.S.M. 69 Hodder, J.E. 65 Howard, R.A. 73 Howells, J. 5 HR-oriented managers 123 Huberman, A.M. 47 Hultink, E.J. 22 human genome project 38 human resources management empirical analysis of projectification and open innovation impact conflicts and stress 130–31 descriptive statistics of the sample 136–7 factor analysis 136–40 findings 129–34 incentives 133–4 issues and methodology 128–9 planning 134 skills demands 131–2 training 132–3 evaluation model for HRM practices 127–8 literature 110–21 overview 108–10, 134–6 scientists’ view open innovation 123–7 projectification 121–3 hurdle rates 66 Hwang, C.L. 57 IAS/IFRS adoption case study 178–9, 182–5 idea generators 112 incremental R&D projects, evaluation techniques 81

296

Evaluation and performance measurement of R&D

independent contributor stage 116 indicators/metrics firm level 17–20 taxonomy 20 influence diagrams 73 information flows see R&D information flows innovation and learning perspective 29 innovation process in case study company 91 non-linear view 227 innovation systems’ perspective 4 input indicators 19 Integrated Technology Demonstrators (ITD) 209–10 intellectual property rights (IPR) Mutti and Yeung (1996) study 242 rationale for 222 technological spillovers and 168 intrinsic rewards 113–14 IPR (intellectual property rights) Mutti and Yeung (1996) study 242 rationale for 222 technological spillovers and 168 Irvine, J. 54, 65 ITD (Integrated Technology Demonstrators) 209–10 Jagannathan, P. 58 Japanese firms, use of process indicators 24–5 joint technology initiatives (JTI) 202 Jones, G. 6 JTI (joint technology initiatives) 202 Kamoda, H. 62 Kaplan, R.S. 21 Kellogg, D. 74 Kerre, E.E. 64 Kerssens-van Drongelen, I.C. 4, 15, 18, 21, 22, 23, 26, 28, 33 Kester, W.C. 65, 74 Kim, B. 21, 22, 23, 24, 26 Kim, S.H. 61 knowledge, as a public good 221, 223 knowledge dilemma 222 knowledge management 119–20 knowledge sourcing, beyond firm’s boundaries 193–6

Kodama, F. 3, 34, 49, 50, 51 Krogsgaard, C. 74 Kuwait Institute for Scientific Research 57 Lai, S.-Q. 21, 22 leadership 119 Lee, C.-L. 21, 22 Lee, J.W. 61 Lee, M. 22, 23, 26 lemons, market for 224 Liang, G.S. 64 Liang, W.Y. 62 Liles, D. 17, 20, 24 line managers’ competencies 123 Lintner, J. 66 Liu, B. 59 Liu, T.-L. 67 locals (R&D workers) 117 Loch, C. 23, 24, 26 Loch, C.H. 15, 23, 24, 26, 33 Locke, S. 54, 66 Magee, J.F. 71 Mahajan, V. 2 management-by-objectives (MBO) reward system 41 management of R&D, change in 5–6 ‘mapping measurement impact’ methodology 239 Margrabe model 76 Marshall, K.T. 62 Martin, B.R. 18 Masella, C. 21, 22, 23, 26 mathematical optimization models 85–6 matrix structure 40 MBO (management-by-objectives) reward system 41 McAdam, R. 20 McClellan, D.C. 36 Meade, L.M. 61, 62 measurement frequency 23–4 measurement process 23, 30 mentor stage 116 Merchant, K.A. 23, 24, 31 Merton, R.C. 68 Metheson, J.E. 73 Miles, M.B. 47 Miller, D.A. 24, 31

Index MIUR (Italian Ministry for Education, University and Research) 11 Moed, H.F. 18 Mohapatra, P.K.J. 22, 24, 26 Mohr, J. 2, 5 Moizer, P. 36 Monte Carlo Simulation 78–9 moral hazard, writing contracts 224 Mordeson, J.N. 64 Morgan, G.A. 50 Morgan, M.G. 72 Morris, P.A. 54 Mossin, J. 66 motivational objectives 33, 34–5 Muffatto, M. 38, 39 Muller, E. 6 Munari, F. 4 Murakami, M. 58 Myers, S.C. 54, 65, 68 Narayanan, V.K. 24, 31 Natta, Giulio 226 Nauda, A. 55 Nayak, P.R. 24, 26 Net Present Value (NPV) 65 criticisms 65–6 see also certainty equivalent NPV; risk-adjusted NPV; stochastic NPV networking R&D policies for financial policies 245–9 legislative policies 249–50 Networks of Centres of Excellence program 247 Nevens, T.M. 2 new product development (NPD) 1 and hard objectives 33 and motivational objectives 34 project evaluation techniques 81 Nishry, M.J. 53, 56 Nixon, B. 17, 20, 24, 26 Noci, G. 64 Norton, D.P. 21 Noyons, E.C.M. 18 NPD (new product development) 1 and hard objectives 33 and motivational objectives 34 project evaluation techniques 81

297

NPV (Net Present Value) 65 criticisms 65–6 see also certainty equivalent NPV; risk-adjusted NPV; stochastic NPV O’Brien, T.J. 67 Oh, H. 21, 22, 23, 24, 26 Ojanen, V. 15, 22, 23 Oliver, R.M. 62 open innovation 6, 124–5 conflicts and stress 130 and human resources management 123–7 incentives 133–4 motivations 197–9 planning 134 skills demands 131–2 training 132–3 Open Source software 196 options (financial) 74 see also ROV oriented basic research/strategic research 226–7 see also use-inspired basic research Ormala, E. 54 Ortt, J.R. 5, 6 Osawa, Y. 58 Ouchi, W. 34 outcomes as indicators 19 output indicators 19 Page, A.L. 21, 22, 23, 24, 26 Pappas, R.A. 15, 18, 20, 24, 26, 29 Pasteur, Louis 226 patent revenues, attribution to inventors 114 path dependence, knowledge sources and 194–5 Pawar, K.S. 15, 17, 20, 22, 23, 24 Pearson, A.W. 23 performance evaluation, people 115–16 performance measurement system (PMS) archetypal models 32–6 assessment dimensions 35 objectives 32–5 Company ‘A’ study 44 control objects 43 dimensions of performance 42

298

Evaluation and performance measurement of R&D

indicators 42 measurement process 43, 45 objectives 41 constitutive elements 25–30 literature 15 measurement context 30–32 reference framework 27 see also R&D performance measurement Pessemier, E.A. 54 Pike, R.H. 69 planning open innovation and 134 see also Risk Management Plan PMS (performance measurement system) archetypal models 32–6 assessment dimensions 35 objectives 32–5 Company ‘A’ study 44 control objects 43 dimensions of performance 42 indicators 42 measurement process 43, 45 objectives 41 constitutive elements 25–30 literature 15 measurement context 30–32 reference framework 27 see also R&D performance measurement Poh, K.L. 4, 15, 53, 55 policies see R&D policies positive externalities 221 Presley, A. 17, 20, 24, 61, 62 Pritchard, R.D. 36 process indicators 19 Projan, S.J. 67 project approval documents 93 Development Costs 93, 97, 101 Project Charter 93, 94 Project Scope 93, 95 Project Submittal Form 93, 98, 102 Revenue & Margin Calculations 93, 95, 101 Risk Management Plan 93, 96 Project Charter document 93, 94 project evaluation see R&D project evaluation project intensification 122–3

project management competencies 123 project orientation 117 Project Scope document 93, 95 Project Submittal Form 93, 98, 102 projectification conflicts and stress 130–31 scientists’ view 121–3 projects definitions 51 see also R&D project evaluation Proposed Standard Practice for Surveys of Research and Development (OECD) 220 qualitative indicators 29 qualitative subjective metrics 18 quantitative indicators 18–19 quantitative objective indicators 18 quantitative subjective indicators 18 R&D definitions 219–21 new understanding of research activities and processes 225–6 non-linear view of the innovation process 227 taxonomy 1, 226–7 R&D information flows 166–7, 179–80 asymmetries 167 from accounting standards 166–7, 169–70 see also IAS/IFRS adoption case study from economic attributes of R&D 167–9 financial analysts role 176–7 value relevance for stock market investors 170–73 empirical models 180–82 voluntary disclosure 173–6 R&D-intensive stocks 172–3 R&D investment, growth in 3 R&D performance measurement challenges 1–2 interest in 2–4, 15 literature gaps 4–6 R&D policies for firms financial policies 238–41 legislative policies 241–2

Index future developments evaluation methodologies 252–4 evaluation typologies 250–52 for knowledge generating institutions financial policies 242–4 legislative policies 244–5 for networking financial policies 245–9 legislative policies 249–50 overview 218–19 scope and rationales 219–21 ‘to devise new applications’ 223–5 ‘to increase the stock of knowledge’ 221–3 potential consequences of new taxonomy 228–9 study of methodologies and typologies 229 framework for 234–8 method 230 review of main contributions to literature 230–34 R&D project evaluation benefit-contribution techniques 54 methods for decision analysis 70–78 methods for economic analysis 65–70 Monte Carlo Simulation 78–9 comparative assessment of techniques 79–83 organizational implications case study 90–91 approval of projects 92–5 innovation process 91 introduction 91 projects management 95–102 scoring method 102–5 screening of ideas 91–2 overview 51–2, 88–9 taxonomy 53–6 weighting and ranking techniques 53–4 applying fuzzy logic 63–5 comparative methods 59–63 scoring methods 56–9 R&D project portfolio analysis 83–4 evaluating portfolio ‘merit’ 86–8

299

interdependences among projects 84–6 radical R&D projects, evaluation techniques 81 Raftery, J. 69 Rahaman, S. 59 Raiffa, H. 71 Ramirez, P. 161 Rangone, A. 64 rank reversal problem 62 real options (RO) theory, and stock market valuation of R&D 158–9 Real Options Valuations (ROV) 74–7, 78 Remer, D.S. 15, 18, 20, 24, 26, 29 Rengarajan, S. 58 return on equity (ROE), and R&D 143 Revenue & Margin Calculations document 93, 95, 101 reward systems 113–15 Riggs, H.E. 65 risk-adjusted NPV (RAR) 66–8 risk-free rate 66 Risk Management Plan 93, 96 risk-premium rate 66 Robben, H.S.J. 22 Roberts, E.B. 5 Robichek, A. 68 ROE (return on equity), and R&D 143 Roessner, D. 55 ROV (Real Options Valuations) 74–7, 78 Ryan, G.P. 65 Ryan, P.A. 65 Saaty, T.L. 59, 60, 61, 63 Sanchez, A. 61 Sandstrom, J. 21, 22, 28 Savage, L.J. 71 Schall, L.D. 65 Scholes, M. 68 Science and Technology Basic Law (Taiwan) 244 scouting role 118 SEI (Sumitomo Electric Industries) 57 semi-quantitative metrics 18 ‘shadow’ value of an asset 149 Sharpe, W.F. 66 Shin, C.O. 62 Simons, R. 36

300

Evaluation and performance measurement of R&D

Sivathanu, P.A. 15 Smits, R. 5, 6 Soderquist, K.E. 15, 23, 24 soft objectives 33, 34, 36 Souder, W.E. 15, 17, 18, 20, 29 sponsor stage 116 SRA (strategic research agenda) 202 ACARE 207–8 Srinivasa, R.K. 15 Stam, A. 65 standards 23, 30 see also accounting standards; Frascati Manual star scientists 112, 130 stochastic NPV 69 stock market valuation of R&D 143–5, 147, 163–4 empirical models, Tobin’s Q 149–51 corporate governance and 160–63 decrease over time 153 empirical models 147–8 based on event studies 151–2 impact of uncertainty 158–60 literature 153–7 market efficiency 146–7 value relevance of R&D information 170–73 empirical models 180–82 strategic research see oriented basic research/strategic research strategic research agenda (SRA) 202 ACARE 207–8 Stulz model 76 Sugawa, S. 62 Sumitomo Electric Industries (SEI) 57 Suomala, P. 24 Svenson, R.A. 15, 17, 19, 20, 22 system integrators 206 ‘systemic’ perspective, firm level 21–4 Szakonyi, R. 23 tacit knowledge 120, 223 Tapper, S. 15, 23, 24, 26, 33 task coordinator role 118 task-oriented managers 123 technical transfer orientation 117 Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) 57–8

technological opportunities, knowledge sources and 195 technological spillovers and, IPR (intellectual property rights) 168 technology markets, growth of 6 technology platforms definitions 191, 199–201 determinants for R&D beyond firm’s boundaries 192–3 development interdependence 196–7 knowledge sourcing 193–6 open innovation 197–9 evaluation 210–15 overview 189–92, 215–17 see also European Technology Platforms technometric indicators 18 Teegen, H. 6 Teknopol approach 247 Thomson, William, Lord Kelvin 226 Tiers of activities 24 Tobin’s Q, empirical models 149–51 Toivanen, J. 21, 22, 28 Toletti, G. 64 Tollgate Reviews 99–100 TOPSIS (Technique for Order Preference by Similarity to Ideal Solution) 57–8 tragedy of the commons, nonapplicability to knowledge 222 Traynor, A.J. 58 triangular membership functions 64 Triantaphyllou, E. 61 Trigeorgis, L. 54 Twiss, B. 21 Tylecote, A. 161 US, Generally Accepted Accounting Principles (GAAP) 169, 171 use-inspired basic research 226, 228 see also oriented basic research/ strategic research value drivers 167–8 Verbeek, A. 18, 20 Viscovich, M. 74 VOC (Voice of the Customer) 96 Voice of the Customer (VOC) 96 Vuola, O. 15, 22, 23

Index WACC (Weighted Average Cost of Capital) 65–6 Wang, M.J. 64 Weighted Average Cost of Capital (WACC) 65–6 Wen, S.-Y. 67 Werner, B.M. 15, 17, 18, 20, 29 Wilemon, D. 2 Wind, J. 2

Xu, S. 59 Yin, R.K. 45, 47, 48 Yoon, K. 57 Zadeh, L.A. 63 Zenker, A. 6 Zettelmeyer, F. 24, 26 Zimmermann, H.J. 64

301