Human error and system design and management

Lecture Notes in Control and Information Sciences Editor: M. Thoma

253

Springer London Berlin Heidelberg New York Barcelona Hong Kong Milan Paris Santa Clara Singapore

Tokyo

P.F. Elzer, R.H. Kluwe and B. Boussoffara (Eds)

Human Error and SystemDesign and Management With 55 Figures

~ Springer

Series Advisory

Board

A. Bensoussan • M.L Grimble • P. Kokotovic • A.B. Kurzhanski • H. Kwakernaak • LL. Massey • M. Morari

Editors P.F.Elzer, MSc, P h D I n s t i t u t e for P r o c e s s a n d P r o d u c t i o n C o n t r o l T e c h n o l o g y , T e c h n i c a l U n i v e r s i t y o f Clausthal, J u l i u s - A l b e r t - S t r . 6, D-38678 Clausthal-ZeUerfeld, G e r m a n y R.H. Kluwe, D r phil, D r h a b i l

Institute for Cognitive Research, University of the Federal Armed Forces, Hamburg, Holstenhofweg 85, D-22043 Hamburg, Germany B. Boussoffara, PhD Institute for Process and Production Control Technology, Technical University of Clausthal, Jttlius-Albert-Str. 6, D-38678 Clausthal-Zellerfeld, Germany

ISBN 1-85233-234-4 Springer-Verlag London Berlin Heidelberg British Library Cataloguing in Publication Data Human error and system design and management. - (Lecture notes in control and information sciences; 253) 1. Human-computer interaction - Congresses 2.Human-machine systems - Congresses I.Elzer, P. (Peter) II.Kluwe, Rainer H. III.Bousoffara, B. 629.8 ISBN 1852332344 Library of Congress Cataloging-in-Publication Data A catalog record for this book is available from the Library of Congress Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms of licences issued by the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent to the publishers. © Springer-Verlag London Limited 2000 Printed in Great Britain The use of registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant laws and regulations and therefore free for general use. The publisher makes no representation, express or implied, with regard to the accuracy of the information contained in this book and cannot accept any legal responsibility or liability for any errors or omissions that may be made. Typesetting: Camera ready by contributors Printed and bound at the Athenmum Press Ltd., Gateshead, Tyne & Wear 69/3830-543210 Printed on acid-free paper SPIN 10746187

Preface "Human Error and System Design and Management" contains a collection of contributions presented at an international workshop with the same name held from March 24 - 26, 1999, at the Technical University of Clausthal, Germany. The workshop included participants from Germany, Great Britain, the Netherlands, Norway and the USA. It was supported by the "Volkswagen Stiftung". Its purpose was to discuss the results of a research project investigating the "Influences of Human-Machine-Interfaces on the Error-proneness of Operator Interaction with Technical Systems" in a broader context. Therefore experts from academia and from industry had been invited to discuss the theoretical as well as the practical aspects of the subject matter. The book contains the latest state of this discussion and can be regarded as a supplement to engineers and researchers who are active in the area of human machine interfaces. In particular it is important that interaction between humans and machines can not be discussed in isolation as the relation "operator(s) - machine". The correctness of identification and assessment of the state of a technical system and the (eventually) necessary operator actions also depends on the construction proper of the technical system and on the organisational structures ("management") surrounding it. Topics of the workshop also included more recent considerations concerning "Multimedia" and "ecological interfaces" as well as "situation awareness". Proper training of operators and modern principles in the design of control rooms were discussed from an industrial viewpoint. Finally, one of the original assumptions of the project was confirmed, i.e. that in the particular area less experimental results exist than is generally assumed. Therefore, it appeared appropriate to intensify experimental work at the border-line of cognitive science and technology.

Outline The book is organized into five parts and preceded by a State-of-the-Art lecture. The parts including full and short papers as well as summaries of discussions are: I

Research in Human Error, Operator Behaviour and Operator Models

II

Influence of Human-Machine-Interfaces in Technical Systems

III

Technical Aspects

IV

Management Issues and Operator Training

V

Final Discussion

Acknowledgements The editors are grateful to:

The Volkswagen Foundation, Hanover, Germany for supporting the project (Ref.# I/69 886) and sponsoring the workshop. The authors want to thank all the workshop contributors for contributing to this book project. Finally, Mrs. Alison Jackson at Springer-Verlag London should be thanked for editorial suggestions and for helping us with general publishing questions.

Clausthal, September 1999

Peter F. Elzer Badi Boussoffara

Hamburg, September 1999

Rainer H. Kluwe

Contributors Alty, J. L., IMPACT Research Group, Loughborough University of Technology, Great Britain. Borys, B-B., Systems Engineering and Human-Machine Systems, University of Kassel, Germany. Boussoffara, B., Institute for Process and Production Control Technology, Technical University of Clausthal, Germany. Bubb, H., Chair of Ergonomics, Technical University Munich, Garching, Germany. DeVries, J.W., Nuclear Power Plant Borssele, Netherlands. Dr~ivoldsmo, A., OECD Halden Reactor Project, Institute for Energy Technology, Halden, Norway. Elzer, P. F., Institute for Process and Production Control Technology, Technical University of Clausthal, Germany. Endsley, M. R., SA Technologies, Marietta, USA. Flach, J. M., Psychology Department, Wright State University, Ohio, USA. Geb, T., Seamans Center for the Engineering Arts and Sciences, University of Iowa, USA. Grams, T., University of Applied Science, Fulda, Germany. Harrison, M. D., Dept. of Computer Science, University of York, York, Great Britain. Heinbokel, T., Institute for Cognitive Research, University of Federal Armed Forces, Hamburg, Germany. Hollnagel, E., Graduate School for Human-machine Interaction,University of Link6ping,, Sweden. Kanse, L., Human Reability Associates HRA-NL, Breda, Netherlands. Khalil, Ch. J., IMPACT Research Group, Loughborough University of Technology, Great Britain. Kluwe, R. H., Institute for Cognitive Research, University of Federal Armed Forces, Hamburg, Germany. Lee, J. D., Seamans Center for the Engineering Arts and Sciences, University of Iowa, USA. Marrenbach, J., Department of Technical Computer Science, RWTH Aachen, Germany.

viii

Pollack, E., Seamans Center for the Engineering Arts and Sciences, University of Iowa, USA. R6se, K., Institute for Production and Automation, University Kaiserslautern, Germany Sch6bel, M., Research Center System Safety, University of Technology Berlin, Germany. Sheridan, T. B., Human-Machine System Laboratory, Massachusetts Institute of Technology, USA. Stolze, Peter, Institute for Safety Technology GmbH, Garching, Germany. Str~iter, O., Chair of Ergonomics, Technical University Munich, Garching, Germany. Sttirenburg, H-G., Kraftwerksschule e.V. Essen, Germany. Szameitat, S., Research Center System Safety, University of Technology Berlin, Germany. Trimpop, R., Friedrich-Schiller University of Jena, Germany. van der Schaaf, T. W., Safety Management Group, Eindhoven University of Technology, Netherlands. Zinser, K., Asea Brown Boveri, Utility Automation GmbH, Mannheim, Germany.

Contents H C I in Supervisory Control: Twelve Dilemmas .................................................. 1

T. B. Sheridan I

Research in Human Operator

Error, Operator

Behaviour

and

M o d e l s ....................................................................................... 13

1

Errors in Situation Assessment: Implications for System Design .............. 15 M. R. Endsley

2

Errors and E r r o r Recovery ........................................................................... 27 T. W. van der Schaaf, L, Kanse Analysis and Prediction of Failures in Complex Systems: Models & Methods .......................................................................................................... 39 E. Hollnagel

4

Scenarios, Function Allocation and H u m a n Reliability .............................. M. D. Harrison

43

Experience Feedback and Safety Culture as Contributors to System Safety .............................................................................................................. 47 M. Sch~bel, S. Szameitat 6

O p e r a t o r Modelling and Analysis of Behavioural Data in H u m a n Reliability Analysis ......................................................................................... 5 l O. Str~iter

D i s c u s s i o n Session I .............................................................................................. 55

II Influence of Human-Machine-Interfaces

in Technical

S y s t e m s ........................................................................................

57

A Project Overview: Influence of Human-Machine-Interfaces on the Error-Proneness of O p e r a t o r Interaction with Technical Systems ................. 59 P. F. Elzer, B. Boussoffara Attributes of the Interface Affect Fault Detection and Fault Diagnosis in Supervisory Control .................................................................................. 65 T. Heinbokel, R. H. Kluwe

Evaluation of Interfaces by Means of Experiments: what's behind Taxonomy? ......................................................................................................

79

B. Boussoffara, P. F. E l z e r

Human Performance and Interface-Design - Some Remaks based on Experiments ....................................................................................................

89

P. Stolze, O. Str~ter

4

Putting the Normative Decision Model into Practice .................................. 99 T. G r a m s

Discussion Session II ..........................................................................................109 III

Technical Aspects .................................................................

111

Multimedia Interfaces and Process Control: The Role of Expressiveness ..............................................................................................

113

J. L. Alty

2 Ecological Interface Design: Some Premises .............................................. 125 J. M. Flach 3

Ecological Interface Design (EID) and the Management of Large Numbers of Intelligent Agents ..................................................................... 137 J. D. Lee, T. Geb, E. P o l l a c k

4 Operator Support in Technical Systems ..................................................... 153 H. B u b b 5

Interfaces for Every Day Things ................................................................. 163 K. R 6 s e

6

Operator Process Interfaces - A R e t r o s p e c t i v e V i e w of the "90s ............. 167 K. Zinser

7 Acceptance of new Technology and Layout in Control Rooms ................ 173 J. W. de Vries

8 Advanced User Interface Design for Aircraft Cockpit Devices ................ 177 J. M a r r e n b a c h

AMEBICA- An Auto Adaptive Multimedia Environment Based on Intelligent Collaborating Agents .................................................................181 C. J. Khalil

xi

IV

Management

Issues and Operator

T r a i n i n g ..................... 187

1 Safety Culture ............................................................................................... 189 R. Trimpop 2

Study of Errors by Means of Simulation and Training ............................. 201 A. Dr~ivoldsmo

3 Operator Training and Implication for the Practice ................................. 207 H-G. Sttirenburg Function Distribution between Man and Machine: Experiments p e r f o r m e d in FANSTIC I I ........................................................................... 211 B-B. Borys

V

F i n a l D i s c u s s i o n ....................................................................

217

Discussion Session V .......................................................................................... 219

HCI in Supervisory Control: Twelve Dilemmas

Thomas. B. Sheridan Massachusetts Institute of Technology, 3-346 Cambridge, MA 02139, USA e-mail: [email protected]

Abstract: Increasingly in recent decades the computer has become a mediator

between the human operator and the physical system being controlled. This occurred first in aviation, then in process control, manufacturing and military systems. More recently it happened in hospitals, trains, automobiles and home appliances. This new form of control is often called supervisory control, where the role of the human operator becomes more like a manager: planning, setting goals and constraints, diagnosing failures, intervening in control as necessary, and learning from experience. The hope has been to relieve the operator of tedious work and make the system both more efficient and more reliable. Unfortunately human-computer interaction (HCI) and reliability of supervisory control have not been as positive as had been hoped. This paper reviews twelve dilemmas that have emerged from experience. They have implications for human error and system reliability.

1. The imperative of automation: How to moderate the rush? Technology development clearly drives the appetite for more technology. The fact that a system can be automated encourages developers to claim that the system should be automated. Why not automate whatever can be automated and thereby save the human operator or worker the tedium of doing the task? Does not automation not only relieve human effort but also prove more economical, more reliable and safer in the long term than having people perform those same tasks? There is now sufficient experience with automated systems to give an unequivocal answer: no. Of course automation is proving itself increasingly effective and trustworthy across a wide spectrum and heretofore human-controlled tasks. But we have also experienced a number of failures, where poorly designed automation has led to more load on the human, more human error, automatic actions which are not appropriate to the circumstances, and in the end greater cost and compromise with safety. In most complex systems where human safety or economic costs are at risk, the human is not removed from the system but is asked to play the role of supervisor: to

2

HCI in Supervisory Control: Twelve Dilemmas

monitor the automatic operations, to detect whether the automation is failing in any respect, and to fix whatever has gone wrong immediately. However, humans are known to be poor monitors, who become bored and inattentive if nothing much is happening and who are therefore likely to miss the signs that serious trouble is occurring. So supervisory control is not in itself a solution. It must be done properly. Sheridan [9] details some of the pitfalls and the methods of solution. In any case, in spite of many recent efforts, improving on the Fitts List by rational means to allocate functions between human and machine has not been easy. The proper level of automation must be selected, and this level is likely to be very different for different stages of a process: information acquisition; information analysis; action decision; and action implementation [11]. This is suggested in Figure 1.

Stages: acqAire information] people stored records sensors times places strategies

analyze & ] display

decide action

algorithms criteria past states actions present states future states confidences interpretations display formats

]

I

implement action

control methods speed precision resources used

Levels: 1. The computer offers no assistance: the human must do it all. 2. The computer suggests alternative ways to do the task. 3. The computer selects one way to do the task, and 4 ..... executes that suggestion if the human approves, or 5 ..... allows the human a restricted time to veto before automatic execution, or 6 ..... executes automatically, then necessarily informs the human, or 7 ...... executes automatically, then informs the human only if asked. 8. The computer selects, executes, and ignores the human.

Figure 1: Four stages of a process, and eight levels of automation.

HCI in Supervisory Control: Twelve Dilemmas

3

2. Increasing situation awareness AND having time to act: Do they conflict? Currently there is great interest in ways to improve situation awareness by the human operator. But the more we insist on improving situation awareness the more time and stress constraint we are likely impose on taking action. Automation is supposed to relieve the need for situation awareness and the preparedness for taking action based upon that awareness. Because of the time or effort resources it demands it may do neither. Particularly in situations where the automation is failing and the human is expected to know what is going on and to step in to fix it, the human may come up short. In such situations the human may have been relatively inattentive and be suddenly faced with a transient in mental workload to understand what is going on and to act. Workload transients are likely to be particularly difficult to cope with. One approach to analysis of this problem is as follows [10]. It is generally conceded that the human requires some time to "become aware of the situation", though the quantitative measure of this is not well established. The analysis works just as well if we are dealing with some resource other than time, such as mental workload capacity. Hypothetically, insofar as more of the available time (or other resource) is used for attending to the "situation" (we will call that fraction x), a smaller fraction of that resource capacity is available for attending to decision-making and execution of appropriate control action. So there is an obvious trade-off.

j

x

~ { A G - AL]= (G-L)

ideal operating point, d'= ~

x=~

A Total gain G

information acquisition effort, x 2a: marginal SA gain and control loss

0

Total IossL ~

max

x=0 2b: ROC plot

Figure 2: Compromise of situation awareness and time stress for decision/action. Assume that AG(x) is the probability density for whatever incremental gain accrues from spending some incremental bit Ax of time (resource) x in order to acquire

4

HCI in Supervisoo, Control: Twelve Dilemmas

more situation awareness, and that AL(x) is the probability density of the incremental cost of having to control with the residual time (resource), where both densities are distributed as a function of x as shown in Figure 2a. Then the expected net gain as a function of x is the integral of (AG - AL), the heavy line in 2a. Figure 2b shows a cross plot of the component integrals G and L, a plot that superficially resembles the ROC curve of signal detection theory. In this case the quantity (G-L) is the height of the curve above the diagonal, and the optimum operating point x* is the point where the maximum (G-L) is obtained. Obviously one can come closer to the ideal of combined greatest gain from situation awareness and least cost from unavailable residual attention (represented at the upper left corner of 2b) when the AG and AL density functions do not overlap, or are most spread apart. In that case there is a value of x* for which the integral G is maximum and the integral L is zero.

3. Decision aids: How much to trust them? While automation is often not trusted by the user, management or public, having a human operator available provides a sense of comfort. To assist the human operator a decision aid (expert system, advisory system, predictor display, etc.) is often proposed, but the sense of comfort remains because, after all, the human need not take the advice of the decision aid. However, if a decision aid is very trustworthy the operator is likely to take its advice uncritically. If the decision aid unexpectedly fails he is likely to be unprepared. If the decision aid is unreliable he will simply pay no attention to its advice. These are limiting circumstances, mostly unavoidable. Between these limits is a range of circumstances for which the operator may be unsure of how much to trust or distrust the decision aid. The subject of trust is a relatively new one for human factors and seemingly irrelevant for systems engineering, and in either case one that is not particularly tractable to scientific investigation. But in fact there have been several recent investigations in which trust has been measured experimentally as a function of failure rates of automation and other properties of systems [6],[8]. Validating a decision aid is in some sense an impossibility, since it requires that some optimum advice be known for all possible situations which might be encountered. First, all possible situations can never have occurred in any finite time, and second, even if they did it would be impossible to say what optimum advice might be for that circumstance, since that depends on the relatively unknown ability of the operator to understand the message and be able to act upon it. One might assert that if optimum advice were available for all possible circumstances there would be no need for the human operator, since the system could then be automated reliably without need for a human supervisor. Reality poses a very different situation, for most decision aids are quite good for those circumstances which are predictable and have been experienced, but are of little or no use otherwise.

HC! in Supervisory Control: Twelve Dilemmas

5

The challenge then is to provide decision aids which convey to the user some estimate of how confident they are in their advice, and how much confidence the user should place in them. Decision aids which make use of fuzzy rules, for example, carry along in the computation a membership function, which itself is a measure of self-confidence based on the membership of the degree of relevance of the fuzzy rules employed in relation to the input data provided.

4. Ultimate authority: Who should be on top when? It is comfortable to assert that the human should always have final authority. Realistically, however, there are many situations where engineers have given to the automation the authority to override human commands. The empirical fact is that humans have neither the computational speed nor the response speed sufficient for many situations, the latter largely due to distractions or diminished alertness. For example in a nuclear power plant when core neutron flux reaches a certain level it is imperative that moderating boron control rods be dropped into the reactor core instantaneously. There is no time to wait for human attention and deliberation to catch up. The same may be said for an aircraft which is about to stall at some combination of altitude, airspeed and attitude if a pilot command were to be followed; in this case the computer inhibits the command from being executed. In the nuclear power industry in the U.S. there is a "tenminute rule", which recommends that the human should never be the ultimate authority when action must be taken within ten minutes. In Europe there is a corresponding "20 minute rule"! Most of the time, when the human is available, and if the required response time is not too short (i.e., where there is time for the human to be called to attend to some situation by an alarm or other means, when there is time for the human to read and interpret the display, where there is additional time for the human to contemplate, decide upon and then execute some action) it is best that the human be in charge.

5. Naturalistic decision-making and traditional decision theory. Can they be compatible?

normative

Many cognitive engineers are asserting that traditional quantitative decision theory is inappropriate for characterizing real human decision-making. Qualitative theories for naturalistic, recognition-primed, ecological, and situated, decision-making are becoming popular, stemming from the ideas of philosopher Heidegger and the psychologist J. Gibson. Naturalistic decision-making has been espoused by Zsambok and Klein [ 13] and refers to the seemingly intuitive and non-rational ways that people arrive at decisions.

6

HCI in Supervisory Control: Twelve Dilemmas

Recognition-primed refers to the idea that decision and action are automatically triggered by recognition of appropriate action opportunities or necessities in the environment, a basis for much of Gibson's theories of action. Gibson called such opportunities or necessities affordances. Ecological is a term used by Vicente and Rasmussen and other to mean perceptions and actions which correlate naturally from understanding, i.e., from a good mental model. Situated is a term attributed to Suchman in describing how naive users of mechanical devices explore action alternatives in a context of what they assume and what they know. These researchers for the most part have asserted that normative models of decisionmaking simply do not fit the empirical facts and therefore should be rejected. This author [11] agrees that much of human decision making, especially when under time stress or in familiar situations, does not fit the available normative theory, in the sense that all the alternative alternatives are not initially and explicitly considered, together with their corresponding probabilities and utilities, and best expected value decision made therefrom in a rational way, the rejection of normative decision theory is misguided. However, quantitative decision theory is needed for engineering. The current naturalistic, recognition-primed, ecological and situated theories are purely qualitative and provide little that is useful for prediction. Current normative decision theory asserts what be decided based on well defined assumptions. It is definitely not a descriptive theory. Much research has been done to show how people deviate from these norms, and that work must continue. Much work must also be done to show how people take short-cuts based on what has been learned and what works in a satisficing way [7], a way that produces decisions which are satisfactory to the point that further effort to improve the decision is not worthwhile. The available normative decision theory is a solid base from which to build a theory of how humans actually decide. There are no alternatives in sight.

6. Error attribution: Can it be done in closed-loop systems? The operator and the machine being operated form a closed loop, Figure 3a. In closed- loop systems attribution of cause and effect are not possible since the effect of one of the two elements is the cause of behaviour in the other, and that in turn causes behaviour in the first, and so on. Attribution (isolation of cause) of system error (a change in some variable to an unacceptable value) cannot be made to A or B in this case.

3a: AB interaction

3b:

reference

input

Figure 3: Problem of error attribution for interacting systems.

HCI in Supervisory Control: Twelve Dilemmas

7

Only where a known transient signal is independently introduced into the loop (Figure 3b) can transfer functions be determined, and failure (or equivalently, parameter variations) be isolated. If A and B are linear differential equation transfer functions, then for a reference input r, a/r = A/(I+AB) and b/r = AB/(I+AB). Then by taking a ratio of the latter to the former one gets a/b=A. Since a and b were empirically determined, A is determined. If A fails, its transfer function changes. In this way, i.e., by injecting independent reference inputs any change in either A or B can be determined. With non-linear systems the determination is more difficult.

7. Designer predictability versus free will and creativity: How to make the trade-off? System designers prefer to have predictability in their systems. That is the basis of all engineering models. Yet human operators allegedly possess free will and creativity, which are not predictable behaviours. As tasks become more complex, the cognitive demands on the human operator are ever greater when the automation fails or is not up to the task, which poses ever greater demands on the operator's free will and creativity. How should the designer treat this dilemma? The answer must lie in compromise. With a maximum of human free will and creativity there is no predictability and the system designer can do nothing. The design is a random process. On the other hand, with no free will and creativity there is hardly any purpose served in having a human in the system. Or at least in this case the human is acting like a robot, an automaton, performing in completely predictable (and some would say inhuman) way. One purpose in having humans in systems is to allow for improvisation and learning when unexpected circumstances occur. Learning and adaptation require some variation in behaviour, plus some means to evaluate the benefits and costs that result from one or another behaviour. This principle is essentially the same as what Darwin called requisite variety in the species to allow for its own improvement or survival in a competitive and predatory world. A reasonable guideline is that there be sufficient invariance of response that no harm is done, but sufficient tolerance for variation in response that the human can make reasonable explorations and experiments on how to improve performance or adapt to new circumstances. This necessarily involves a compromise.

8

HCI in Supervisory Control: Twelve Dilemmas

8. Belief/trust/valuation: A plethora of inconsistent methods? Whatever the task, and however well designed the control system, much will depend upon the operator's beliefs, trusts, subjective probabilities, and relative valuations of the external circumstances and response options. The problem is that we have a plethora of mostly incommensurable methods for dealing with belief and valuation. Von Neumann utility theory and Bayesian probability estimation are the traditional quantitative methods. Fuzzy logic and Dempster-Shafer belief theory are more recent quantitative approaches. Of course, most elicitation of belief from groups of people is done in by simple questionnaires with qualitative categories of response form which respondents may choose. Appendix I provides an example of a simple questionnaire [11] which includes four belief elicitation techniques: conventional qualitative categories, probability estimates, fuzzy scale, and Dempster-Shafer scale. Different answers would necessarily result from these different techniques. It comes down to the fact that if you ask a different question (or ask the same question in different ways) you get a different answer.

9. Gap in modelling HCI and supervisory control: Is there any hope? While useful mathematical models have become available for psychophysics, signal detection, simple information processing, vision, hearing and simple motor skill, no generally acceptable model incorporating the essential elements of supervisory control has yet emerged. Perhaps this is because supervisory control necessarily includes the mix of cognition and behaviourai complexity as well as automation and system complexity. The GOMS model of Card et al [2] and the PROCRU model of Baron et al [1] useful, but are not sufficiently robust or comprehensive to incorporate all the elements of supervisory control planning, teaching, monitoring, intervening and learning).

10. Human-centred design: The emperor's new clothes? The phrase human-centred design is currently very fashionable. Generally the term means that considerations of the human operator or user are kept in the forefront of the design process, an appropriate principle which is frequently ignored. However,

H C I in Supervisory Control: Twelve Dilemmas

9

b e y o n d this g e n e r a l m e a n i n g there is m u c h a m b i g u i t y w h e n m o r e specific m e a n i n g is a t t e m p t e d . In fact m a n y d i f f e r e n t m e a n i n g s h a v e b e e n e m p l o y e d in the literature, m o s t o f w h i c h p o s e p r o b l e m s o f o n e kind or a n o t h e r . T a b l e 1 lists ten a l t e r n a t i v e m e a n i n g s . A f t e r e a c h t h e r e is a p h r a s e s u g g e s t i n g w h y this m e a n i n g is i n a p p r o p r i a t e in s o m e cases, that is, use o f the t e r m h u m a n - c e n t r e d d e s i g n m a y p r o v e i n c o n s i s t e n t or c o n f u s i n g u n d e r s o m e c i r c u m s t a n c e s . 1. Allocate to the human the tasks best suited to the human, allocate to the automation the tasks best suited to it. Unfortunately there is no agreement on how best to do this. 2. Make the operator a supervisor of subordinate automatic control system(s). For many tasks direct manual control may prove best. 3. Keep the human operator in the decision and control loop. Humans can handle only control tasks of bandwidth below one Hz, a demonstrated fact from much research in manual control. At the same time, if the signals to be observed are changing too slowly the human is not likely to maintain awareness of such changes, the well-known result from vigilance experiments. 4. Maintain the human operator as the final authority over the automation. This is not always the safest way. There are many systems where the human is not to be trusted. 5. Make the human operator's job easier, more enjoyable, or more satisfying through friendly automation. Operator ease and enjoyment are nice to have, provided system performance is not compromised. 6. Empower the human operator to the greatest extent possible through flexibility of interface or through automation. The operator may feel a false sense of empowerment by having many controls and displays, and great flexibility in how these may be configured. This is the wellknown peril, leading to complex and unreliable interaction and mode errors. 7. Support trust by the human operator With too much trust there may be a tendency to follow the advice of a decision aid without concern for whether the advice is appropriate, or to abandon responsibility. The challenge is to engender the right amount of trust, not too little or too much. 8. Give the operator information about everything he or she should want to know. The problem here is that too much information will overwhelm. 9. Engineer the automation to minimize human error and response variability. Error is a curious thing. Darwin taught us about requisite variety many years ago. A good system tolerates some reasonable level of "error" and response variability.

10. Achieve the best combination of human and automatic control, where best is defined by explicit system objectives. Don't we wish we always had explicit system objectives. Table 1: Ten alternative meanings of human-centred automation, and some reasons to question

11. Individual alienation to technology: How to deal with it? T h e r e are m a n y aspects o f c o m p u t e r m e d i a t i o n in s y s t e m s that are a l i e n a t i n g to o p e r a t o r s a n d s y s t e m users. O f t e n t h e s e are s u b l i m i n a l f e e l i n g s that are not readily a d m i t t e d to. T a b l e 2 lists a n u m b e r o f these.

I0

HCI in Supervisory Control." Twelve Dilemmas

Those who worry about these aspects of technology are often labeled "doomsayers" by the proponents of the new technology, and are seen as impeders of progress. But such worrying cannot and should not be ignored. While in the short term the worries often do not manifest themselves, over the long term these alienating effects can become very troublesome, in some cases resulting in total abandonment of the technology. For this reason it is important to appreciate that these feelings can be overcome by means of education, more enlightened management, or a more gentle pace of introduction of the technology. 1.

Threatened or actual unemployment

2.

Erratic mental workload and work dissatisfaction

3.

Centralization of management control and loss of worker control

4.

Desocialization

5.

Deskilling

6.

Intimidation of greater power and responsibility

7.

Technological illiteracy

8.

Mystification and misplaced trust

9.

Sense of not contributing

10. Diffusion and abandonment of responsibility 11. Blissful enslavement Table 2: Alienation factors of automation.

12. The world as a multi-HCl supervisory control system: An enlargement of our responsibility? As more sensors and computing elements are added to any one system the complexity necessarily increases. As communication technology allows systems to be interconnected at long distances the complexity increases still further. The internet has already connected information systems world-wide. The information systems themselves are beginning to be interfaced with sensors and effectors (e.g., video cameras, robot arms and vehicles, etc.). Much of this vast interconnectivity supports goals such as education and commerce. Others activities being supported are military systems and espionage, developed of course with the stated goal of security and peace, but capable at a moment's notice to kill and destroy. In any case, for a variety of reasons system complexity is continuing to increase. It is possible now for one person to perform some action anywhere on earth, including the deep ocean and space, from anywhere else. The person or group that has the greatest access to the system becomes very powerful. The person or group

HCI in Supervisory Control: Twelve Dilemmas

11

w h o has little access b e c o m e s less powerful, and this inequity naturally creates serious p r o b l e m s o f jealously, intimidation, and fear. W e see it already o c c u r r i n g in international relations. A d d to this the fact that c o m p l e x i t y leads to unpredictability, and c o n f u s i o n about what cause p r o d u c e d what effect. This makes it easy for an amateur hacker or a serious terrorist can produce trouble on a grand scale, g i v e n only that they have access to the net. The challenge, then, is to ensure security of c o m m u n i c a t i o n channels by m o n i t o r i n g and by design.

13. Conclusion C o m p u t e r mediation and associated human supervisory control offer great advantages in control o f c o m p l e x systems which are highly a u t o m a t e d at l o w e r levels. H o w e v e r , the h u m a n - c o m p u t e r interaction remains fraught with p r o b l e m s which m u s t be appreciated and dealt with. T w e l v e such d i l e m m a s are described, w h e r e h u m a n designers and operators must m a k e and c o m p r o m i s e s .

Appendix L Questionnaire on Degree of Belief among Given Set of Propositions, Comparing Four

Different Measurement Methods The purpose of the experiment is to explore the differences which emerge when asking people in alternative ways about their belief regarding a given proposition. Admittedly any one way of asking a question about a complex issue will pose problems -- hence the reason to compare people's answers for the four different ways of questioning belief about the same proposition. The proposition is as follows: Within the next five years, somewhere in the world, there will occur a nuclear power plant accident with some degree of core melt. Method 1. Standard Ouestignnairr

Please check the one of the two possible answers which most closely fits your belief: True

False

Method 2. Subiective Prob;tbllit u

Next to each answer write the probability (as a decimal number from zero to one) you would assign to that answer. The two decimal numbers must add to 1. True False Sum = 1? (check) Method 3. F~zz3, Logic This time there are five phrases each of which may characterize to some degree your belief in the above proposition. For each phrase write a decimal number between 0 and 1 which specifies the degree to which that set of words characterizes your belief (1 is maximum degree, 0 is none). In this case the numbers need not add to 1. Surely t r u e

Probably t r u e

Ambivalent__ Probably f a l s e

Surely f a l s e

12

HCI in Supervisory Control: Twelve Dilemmas

Method 4, Demp~ter-Sh~fer Thc0rV of Evidence In this case please use a decimal number between 0 and I to indicate your degree of belief for each of the three given answers. Again, for this method, the numbers must add to one. True

False

(No idea which is t r u e ) _ _ Sum = 1?

(check)

References [1]

[2] [3] [4] [5] [6] [7] [8]

[9] [10]

[11]

[12]

[13]

Baron, S., Zacharias, G., Muralhidaran, R., and Lancraft, R. (!980). PROCRU: a model for analyzing flight crew procedures in approach to landing. In Proc. 8'h IFAC Congress, Tokyo, 15: 71-76. Card, S., Moran, T. and Newell, A. (1983). The Psychology of Human-Computer Interaction, Mahwah, N J: Erlbaum. Gibson, J.J. (1979).The Ecological Approach to Visual Perception, Boston, MA: Houghton Mifflin. Heidigger, M. (1962). Being and Time. Macquarrie, J. and Robinson, E. (translators), Harper and Rowe. Landauer, T.K. (1995). The Trouble with Computers. Cambridge, MA: MIT Press, pp. 246, 307, 389. Lee, J.D. and Moray, N. (1994). Trust, self confidence, and operators' adaptation to automation. International Journal of Human-Computer Studies, 40, 153-184. March, J. G., and Simon, H. A. (1958). Organizations. Wiley. Muir, B.M. and Moray, N. (1994). Trust in automation, Part 11. Experimental studies of trust and human intervention in a proces control simulation. Ergonomics, 39 (3). 429-461. Sheridan, T.B. (1992). Telerobotics, Automation and Human Supervisory Control. Cambridge, MA: MIT Press. Sheridan, T.B. (1993). How far to commit to open loop action: a probabilistic decision approach with analogies to signal detection theory, IEEE Trans. on Systems, man and Cybernetics, Vol. 23, No. 3, May/June. Sheridan, T.B. (1998). Technical safety and social safety. Proc. Seminar on Humans and Technology, Institute of Nuclear Safety Systems, Mihama, Fukui, Japan, 21-22 September. Vicente, K.J. and J. Rasmussen (1992). Ecological interface design: theoretical foundations. IEEE Trans. on Systems, Man and Cybernetics, Vol. SMC-22, No. 4, July/August, pp. 589-606. Zsambok, C.F. and Klein, G. (1997). Naturalistic Decision-Making. Mahwah, NJ: Erlbaum.

Errors in Situation Assessment: Implications for System Design

Mica. R. Endsley SA Technologies, Inc., Marietta, Georgia, USA

1. Introduction In complex and dynamic environments, human decision making is highly dependent on situation awareness (SA) - - a constantly evolving picture of the state of the environment. SA is formally defined as a person's "perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future" [1]. It encompasses not only an awareness of specific key elements in the situation (Level 1 SA), but also a gestalt comprehension and integration of that information in light of operational goals (Level 2 SA), along with an ability to project future states of the system (Level 3 SA). These higher levels of SA (Levels 2 and 3) have been found to be particularly critical to effective functioning in complex environments, such as the cockpit, air traffic control, driving, medicine and control rooms. The failures of human decision making are frequently cited in investigations of error in a wide variety of systems. In aviation mishaps, failures in decision making are attributed as a causal factor in approximately 51.6% of all fatal accidents and 35.1% of non-fatal accidents, of the 80-85% of accidents which are attributed to human error [10]. While some of these incidents may represent failures in actual decision making (action selection), a high percentage are actually errors in situation awareness. That is, the aircrew makes the correct decision for their picture of the situation, but that picture is in error. This represents a fundamentally different category of problem than a decision error - - in which the correct situation is comprehended, but a poor decision is made as to the course of action to take - - and indicates different types of remediation attempts. Problems with SA were found to be the leading causal factor in a review of military aviation mishaps [9]. In a study of accidents among major airlines, 88% of those involving human error could be attributed to problems with situation awareness as opposed to problems with decision making or flight skills [4]. Based on a review of literature on human information processing and cognition, a taxonomy for

I6

Errors in Situation Assessment," Implications for System Design

classifying and describing errors in SA was developed [5]. The taxonomy, presented in Table 1, incorporates factors affecting SA at each of its three levels. Loss of Level 1 SA - Failure to correctly perceive the situation (80.2%)

Loss of Level 2 SA - Failure to correctly comprehend the situation

(16.9%) Information not available (11.6%) i, system & design failures 2. failure of communication 3. failure of crew to perform needed tasks Information difficult to detect (11.6%) 9 poor runway markings 9 poor lighting 9 noise in the cockpit. Information not observed (37.2%) 9 omission from scan 9 attentional narrowing 9 task related distractions 9 workload 9 over-reliance on automation 9 stressors 9 other distractions Misperception of information (8.7%) 12, negative interference from prior expectations 13. task distractions Memory error (11.1%) 9 disruptions in routine 9 high workload 9 distractions

Lack of/incomplete mental model (3.5%) 9 Automated systems 9 unfamiliar airspace Incorrect mental model (6.4%) 9 Mismatching information to expectations of model or model of usual system Over-reliance on defaults values in the mental model (4.7%) 9 General expectations of system behaviour Other (2,3%)

Loss of Level 3 SA - Failure to correctly project situation (2.9%) Lack of/incomplete mental model (Off_@) Qver-projection of current trends (0 6%) Other (2.3%)

Table 1: Causal Factors related to Errors in Situation Awareness in Aircraft Operations (Jones & Endsley, 1995).

Errors in Situation Assessment: Implications for System Design

17

2. A T a x o n o m y o f Errors in Situation Awareness The Author [4] applied the SA Error Taxonomy to an investigation of causal factors underlying aircraft accidents involving major air carriers in the United States based on National Transportation Safety Board (NTSB) accident investigation reports over a four year period. Of the 71% of the accidents that could be classified as having a substantial human error component, 88% involved problems with SA. Of 32 SA errors identified in these accident descriptions, twenty-three (72%) were attributed to problems with Level 1 SA, a failure to correctly perceive some pieces of information in the situation. Seven (22%) involved a Level 2 error in which the data was perceived but not integrated or comprehended correctly, and two (6%) involved a Level 3 error in which there was a failure to properly project the near future based on the aircrew's understanding of the situation.

2.1 Causal Factors Associated with SA Errors in Aircraft Operations Jones and Endsley [11] further applied this taxonomy to a more extensive study of SA errors based on voluntary reports in NASA's Aviation Safety Reporting System (ASRS) database. The analysis included 111 incidents, involving pilots. This analysis provides some indication of the types of problems and relative contribution of causal factors leading to SA errors in the cockpit, as shown in Table 1. Level 1 - Failure to correctly perceive the situation. At the most basic level, important information may not be correctly perceived. In some cases, the data may not be available to the person, due to a failure of the system design to present it or a failure in the communications process. This factor accounted for 11.6% of SA errors, most frequently occurring due to a failure of the crew to perform some necessary task (such as resetting the altimeter) to obtain the correct information. In other cases, the data is available, but is difficult to detect or perceive, accounting for another 11.6% of SA errors in this study. This included problems due to poor runway markings and lighting and problems due to noise in the cockpit.

Many times, the information is directly available, but for various reasons, is not observed or included in the scan pattern, forming the largest single causal factor for SA errors (37.2%). This is due to several factors, including simple omission - - not looking at a piece of information, attentional narrowing and external distractions that prevent them from attending to important information. High taskload, even momentary, is another a major factor that prevents information from being attended to. In other cases, information is attended to, but is misperceived (8.7% of SA errors), frequently due to negative interference from prior expectations. Finally, in some cases it appears that a person initially perceives some piece of information but then forgets about it (11.1% of SA errors) which negatively effects SA as it relies on keeping information about a large number of factors in memory. Forgetting was

18

Errors in Situation Assessment: Implications for System Design

found to be frequently associated with disruptions in normal routine, high workload and distractions. Level 2 SA - Failure to comprehend the situation. In other cases, information is correctly perceived, but its significance or meaning is not comprehended. This may be due to the lack of a good mental model for combining information in association with pertinent goals. 3.5% of SA errors were attributed to the lack of a good mental model, most frequently associated with an automated system.

In other cases, the wrong mental model may be used to interpret information, leading to 6.4% of the SA errors in this study. In this case, the mental model of a similar system may be used to interpret information, leading to an incorrect diagnosis or understanding of the situation in areas where that system is different. A frequent problem is where aircrew have a model of what is expected and then interpret all perceived cues into that model, leading to a completely incorrect interpretation of the situation. In addition, there may also be problems with over-reliance on defaults in the mental model used, as was found for 4.7% of the SA errors. These defaults can be thought of as general expectations about how parts of the system function that may be used in the absence of real-time data. In other cases, the significance of perceived information relative to operational goals is simply not comprehended or several pieces of information are not properly integrated. This may be due to working memory limitations or other unknown cognitive lapses. 2.3% of the SA errors were attributed to miscellaneous factors such as these. Level 3 SA - Failure to project situation into the future. Finally, in some cases, individuals may be fully aware of what is going on, but be unable to correctly project what that means for the future, accounting for 2.9% of the SA errors. In some cases this may be due to a poor mental model or due to over projecting current trends. In other cases, the reason for not correctly projecting the situation is less apparent. Mental projection is a very demanding task at which people are generally poor. General. In addition to these main categories, two general categories of causal factors are included in the taxonomy. First some people have been found to be poor at maintaining multiple goals in memory, which could impact SA across all three levels. Secondly, there is evidence that people can fall into a trap of executing habitual schema, doing tasks automatically, which render them less receptive to important environmental cues. Evidence for these causal factors was not apparent in the retrospective reports analyzed in the ASRS or NTSB databases. SummaoLOverall, this analysis indicates that the primary cause of human error in complex systems is not poor decision making, but poor situation awareness. The factors underlying those errors are myriad, and include solutions at both the system design and training levels. Particular problems with situation a~careness are

Errors in Situation Assessment: Implications for System Design

19

indicated at the level of attention and working memory, and can be tied to system designs that over-stress these human limits. 2.2 Causal Factors Associated with SA Errors in Air Traffic Control Although operational errors are very rare in traffic control, of those that occur, problems with situation awareness appear to be a critical factor. SA problems have also been well documented in air traffic control operations. In 1997, the three levels of situation awareness were added to the Federal Aviation Administration's (FAA) forms for investigations of operational errors in air traffic control operations. These forms are filled out by FAA quality assurance investigators (also air traffic controllers) following an operational error (separation of aircraft at less than the prescribed limits). Prior to this change, some 57% of operational errors investigated were attributed to problems involving the radar display, with 14% involving misidentification of information and 47% involving "inappropriate use of displayed data, other" [ 13], a catch-all category that indicated many errors were not well accounted for in the analysis form. Following the addition of the three levels of SA to the form, it was found that the quality assurance investigators in the field were able to greatly reduce the number of errors that were regarded as "other". Of the top ten causal factors for operational errors in Tracon air traffic control operations in the United States in 1997, 58.6% were associated with SA problems [12]. In order of frequency the causal factors involved: (lst) Failure to project future (Level 3 SA) - 29.0% (2nd) Failure to detect displayed data (Level 1 SA) - 17.8% (4th) Failure to comprehend displayed data (Level 2 SA) - I 1.8% The numbers were very similar for enroute ATC operations, where 69.1% of the operational errors in air traffic control involved SA problems: (l st) Failure to project future (Level 3 SA) - 32.8% (2nd) Failure to detect displayed data (Level 1 SA) - 19.6% (3rd) Failure to comprehend displayed data (Level 2 SA) - 16.7% Situation awareness problems can be seen to be pervasive in the air traffic control domain, accounting for the vast majority of errors attributed to the human controllers. The causal factors associated with these errors are also well accounted for by the SA error taxonomy. Endsley and Rodgers [8] conducted a study in which 14 operational errors made by air traffic controllers were examined. An analysis of the errors showed that ten (71%) involved Level 1 SA problems. In these cases there was a failure to monitor key situation parameters, most frequently due to distractions by other pertinent tasks (35.7%). There were also misperceptions (21.4%), and memory problems (14.3%) observed, leading to Level 1 SA errors. Level 2 SA problems were found to exist in 21.4% of the errors, and Level 3 SA problems in 28.6% of the errors. (Note: some operational errors involved more than one SA problem). Interestingly,

20

Errors in Situation Assessment: Implications for System Design

in two-thirds of the cases, the controllers were not even aware at the time that the operational error was occurring. In the study, twenty active duty controllers were asked to watch re-creations of the operational errors and to report on their situation awareness and workload at two stops during the re-creations. In these re-creations, for 10 of the 14 errors (71.4%) there were no subjects (out of 4 observing each operational error) who were aware of the problem at a stop which occurred 2 minutes before the operational error. In 8 of the 14 errors (57.1%), at least one of the four subjects was not aware the error was occurring at the time of the operational error. As the subjects in the study were all highly trained air traffic controllers observing traffic flows in sectors they were familiar with and qualified on, this finding highlights the great difficulty associated with the conditions under which these operations take place. There are significant demands on controllers situation awareness that makes effective SA and performance very difficult. The fact that so many people could experience SA problems when confronted with the same situational challenges highlights the need for more effective design to support the SA processes of these operators.

3. Addressing SA Errors Through System Design An examination of the underlying factors associated with situation awareness and human error provides a more integrated perspective for addressing the system design issue. This perspective focuses on the means by which the human operator maintains an on-going representation of the state of the environment. If designs can be created that enhance an operator's awareness of what is happening in a given situation, decision making and performance should be dramatically better. A situation awareness-oriented design process has been developed that seeks to improve human decision making and performance by optimizing situation awareness. The process includes analysis of SA requirements using a goal-directed cognitive task analysis, the application of design guidelines for enhancing SA, and the validation of resultant designs.

3.1. SA RequirementsAnalysis. Designing interfaces that provide SA depends on domain specifics that determine the critical features of the situation that are relevant to a given operator. A goaldirected task analysis methodology [2] has been used successfully for determining SA requirements in several different domains, including aircraft, air traffic control and remote maintenance control centers. This methodology focuses on the basic goals of operators (which may change dynamically), the major decisions they need to make relevant to these goals, and the SA requirements for each decision. SA requirements are established in terms of the basic data that is needed (Level 1 SA),

Errors in Situation Assessment: Implications for System Design

21

required integration of the data for a comprehension of system state in light of goals (Level 2 SA), and projection of future trends and events (Level 3 SA). Conducting such an analysis is usually carried out using a combination of cognitive engineering procedures. Expert elicitation, observation of operator performance of tasks, verbal protocols, analysis of written materials and documentation, and formal questionnaires have formed the basis for the analyses. In general, the analysis has been conducted with a number of operators, who are interviewed, observed and recorded individually, with the resultant analyses pooled and then validated overall by a larger number of operators. An example of the output of this process is shown in Table 2.

1.3 Maintain aircraft conformance 1.3.1 Assess aircraft conformance to assigned parameters 9 aircraft at~proceeding to assigned altitude ? 9 aircraft proceeding to assigned altitude fast enough? 9 9 9

9 9

aircraft at~proceeding to assigned airspeed? aircraft proceeding to assigned airspeed fast enough ? 9 9

9 9 9

time until aircraft reaches assigned altitude amount of altitude deviation climb/descent 9 altitude (current) 9 altitude (assigned) 9 altitude rate of change (ascending/ descending)

time until aircraft reaches assigned airspeed amount of airspeed deviation 9 airspeed (indicated) 9 airspeed (assigned) 9 groundspeed

aircraft on/proceeding to assigned route ? aircraft proceeding to assigned route fast enough ? aircraft turning? 9 time until aircraft reaches assigned route/heading 9 amount of route deviation 9 aircraft position (current) 9 aircraft heading (current) 9 route/heading (assigned) 9 aircraft turn rate (current) 9 aircraft heading (current) 9 aircraft heading (past) 9 aircraft turn capabilities 9 aircraft type 9 altitude

22

Errors in Situation Assessment: Implications for System Design

9 9 9

aircraft groundspeed weather winds (direction, magnitude)

1.3.2 Resolve non-conformance 9

Reason f o r non-conformance ?

9

Verify data 9

Is presented altitude correct?

9 9 9

Aircraft altimeter setting Aircraft altitude (indicated)

Is presented airspeed correct?

9 Aircraft airspeed (indicated) 9 groundspeed 9 winds (magnitude, direction) 9 Is presented position/heading correct?

9 9 9 9

Fix distance to Nav aid range/bearing to Fix track code

Will current behaviour cause a problem?

9 Assess a i r c r a f t s e p a r a t i o n (1.1.1) 9 Assess aircraft/airspace s e p a r a t i o n (1.2.1) 9 Assure m i n i m u m altitude r e q u i r e m e n t s (1.4) 9

Action to bring into conformance ?

9

Provide clearance (2.2)

Table 2: Example of Goal-Directed Task Analysis for En-route Air Traffic Control [7]. The method is significantly different from traditional task analyses in that: 1.) it is not pinned to a fixed timeline, a feature which is not compatible with the work flow in dynamic systems, 2.) it is technology independent, not tied to how tasks are done with a given system, but to what information is really, ideally needed, and 3.) the focus is not just on what data is needed, but on how that data needs to be combined and integrated to support decision making and goal attainment. This last feature, defining comprehension and projection needs, is critical for creating designs that support SA instead of overload the operator with data as many current systems do.

3.2. SA-Oriented Design. The development of a system design for successfully providing the multitude of SA requirements that exist in complex systems is a significant challenge. A set of design principles have been developed based on a theoretical model of the

Errors in Situation Assessment: Implications for System Design

23

mechanisms and processes involved in acquiring and maintaining SA in dynamic complex systems [5]. These guidelines are focused on a model of human cognition involving dynamic switching between goal-driven and data-driven processing and feature support for limited operator resources, including: 9 Direct presentation of higher level SA needs (comprehension and projection) instead of low level data, 9 Goal-oriented information display, 9 Support for global SA, providing an overview of the situation across the operator's goals at all times (with detailed information for goals of current interest), enabling efficient and timely goal switching and projection, 9 Use of salient features to trigger goal switching, 9 Reduction of extraneous information not related to SA needs, and 9 Support for parallel processing. An SA-oriented design is applicable to a wide variety of system designs, It has been successfully applied as a design philosophy for systems involving remote maintenance operations, medical systems and flexible manufacturing cells. 3.3. Evaluation

Many concepts and technologies are currently being developed and touted as enhancing SA. Prototyping and simulation of new technologies, new displays and new automation concepts is extremely important for evaluating the actual effects of proposed concepts within the context of the task domain and using domain knowledgeable subjects. If SA is to be a design objective, then it is critical that it be specifically evaluated during the design process. Without this it will be impossible to tell if a proposed concept actually helps SA, does not effect it, or inadvertently compromises it in some way. The Situation Awareness Global Assessment Technique (SAGAT) has been successfully used to provide this information by directly and objectively measuring operator SA in evaluating avionics concepts, display designs, and interface technologies [3]. A primary benefit of examining system design from the perspective of operator situation awareness is that the impact of design decisions on situation awareness can be objectively assessed as a measure of quality of the integrated system design when used within the actual challenges of the operational environment. An example of the use of SAGAT for evaluating the impact of new system concepts can be found in [6]. A totally new form of distributing roles and responsibilities between pilots and air traffic controllers was examined. Termed "free flight", this concept was originally described to incorporate major changes in the operation of the national airspace. It may include aircraft filing direct routes to destinations rather than along pre-defined fixed airways, and the authority for the pilot to deviate from that route, either with the air traffic controllers permission or perhaps even

Errors in Situation Assessment: Implicationsfor System Design

24

fully autonomously'[14]. As it was felt that such changes could have a marked effect on the ability of the controller to keep up as monitor in such a new system, a study was conducted to examine this possibility [6].

100

eo 8 0 " O

"~ 30 O

=~ 20 0

Baseline

D~,rect Deviation Deviation Rout(m with Intenl wilho~ Inter.

Baseline

Condition

I~ect Deviation Deviation Routes w~h Intent without Intent Condition

100

50

8O

E3 60

2O ,
0

Baseline

Dnct Deviation Devmtion Rot~es with Inlent without Intent Condition

0

Baselne

D~rect De~ation Dewat~on Routes v,lth Intent wilhod Inter4 Condition

Figure 1: Example of SAGAT Results: Free Flight Implementations [6]. Results showed a trend towards poorer controller performance in detecting and intervening in aircraft separation errors with these changes in the operational concept and poorer subjective ratings of performance. Finding statistically significant changes in separation errors during ATC simulation testing is quite rare however. More detailed analysis of the SAGAT results provided more diagnostic detail as well as backing up this finding. As shown in Figure 1, controllers were aware of significantly fewer of the aircraft in the simulation under free flight conditions. Attending to fewer aircraft under higher workload has also been found in other studies [8]. In addition to reduced Level 1 SA, however, controllers also had a significantly reduced understanding (Level 2 SA) of what was happening in the traffic situation, as evidenced by lower SA regarding which aircraft weather would impact on and a reduced awareness of those aircraft that were in a transitionary state. They were less aware of which aircraft had not yet completed a clearance, and for those aircraft

Errors in Situation Assessment: Implications for System Design

25

whether it was received correctly and whether they were conforming. Controllers also demonstrated lower Level 3 SA with free flight. Their knowledge of where the aircraft was going to (next sector) was significantly lower under free flight conditions. These findings were useful in pinpointing whether concerns over this new and very different concept were justified, or whether they merely represented resistance to change. The SAGAT results showed not only that the new concept did indeed induce problems for controller SA that would prevent them from performing effectively as monitors to back-up pilots with separation assistance, and also showed in what ways these problems were manifested. This information is very useful diagnostically in that it allows one to determine what sorts of aids might be needed for operators to assist them in overcoming these deficiencies. For instance, in this example, a display that provides enhanced information on flight paths for aircraft in transitionary states may be recommended as a way of compensating for the lower SA observed. Far from just providing a thumbs-up or thumbs-down input on a concept under evaluation, this rich source of data is very useful in developing iterative design modifications and making tradeoffs decisions.

4. Conclusions An analysis of human error in different domains shows that much of it is not due to poor decision making, but poor situation awareness. These situation awareness problems can be tied to a number of factors associated with the system design and operational conditions under which people must perform their tasks. Following a structured approach from analysis to design to testing, SA can be incorporated as a significant and attainable design goal that can significantly reduce this class of human error. It is important that such procedures are applied in the design process to address the very real problems leading to poor SA and human performance errors in complex settings. While not all human error is preventable, a significant portion of those problems that are currently labelled as human error can be addressed proactively and successfully by tackling the major design factors that lead to situation awareness problems.

References [ 1] [2] [31

Endsley, M. R. (1988). Design and evaluation for situation awareness enhancement. In Proceedings of the Human Factors Society 32nd Annual Meeting (pp. 97-101). Santa Monica, CA: Human Factors Society. Endsley, M. R. (1993). A survey of situation awareness requirements in air-to-air combat fighters. International Journal of Aviation Psychology, 3(2), 157-168. Endsley, M. R. (1995a). Measurement of situation awareness in dynamic systems. Human Factors, 37(1), 65-84.

26

Errors in Situation Assessment: Implications for System Design

[4]

Endsley, M. R. (1995b). A taxonomy of situation awareness errors. In R. Fuller, N. Johnston, & N. McDonald (Eds.), Human Factors in Aviation Operations (pp. 287292). Aldershot, England: Avebury Aviation, Ashgate Publishing Ltd. Endsley, M. R. (1995c). Toward a theory of situation awareness in dynamic systems. Human Factors, 37(1), 32-64. Endsley, M. R., Mogford, R., Allendoerfer, K., Snyder, M. D., & Stein, E. S. (1997). Effect of free fight conditions on controller performance, workload and situation awareness: A preliminary investigation of changes in locus of control using existing technology (DOT/FAA/CT-TN 97/12). Atlantic City, NJ: Federal Aviation Administration William J. Hughes Technical Center. Endsley, M. R., & Rodgers, M. D. (1994). Situation awareness information requirements for en route air traffic control. In Proceedings of the Human Factors and Ergonomics Society 38th Annual Meeting (pp. 71-75). Santa Monica, CA: Human Factors and Ergonomics Society. Endsley, M. R., & Rodgers, M. D. (1998). Distribution of attention, situation awareness, and workload in a passive air traffic control task: Implications for operational errors and automation. Air Traffic Control Quarterly, 6(1), 21-44. Hartel, C. E., Smith, K., & Prince, C. (1991, April). Defining aircrew coordination: Searching mishaps for meaning. Paper presented at the Sixth International Symposium on Aviation Psychology, Columbus, OH. Jensen, R. S. (1982). Pilot judgment training and evaluation. Human Factors, 24(1), 61-73. Jones, D. G., & Endsley, M. R. (1995). Investigation of situation awareness errors. In R. S. Jensen & L. A. Rakovan (Ed.), Proceedings of the 8th International Symposium on Aviation Psychology (pp. 746-751)~ Columbus, OH: The Ohio State University. Rodgers, M. D., Mogford, R. H., & Strauch, B. (in press). Post-hoc assessment of situation awareness in air traffic control incidents and major aircraft accidents. In M. R. Endsley & D. J. Garland (Eds.), Situation awareness analysis and measurement. Mahwah, NJ: Lawrence Erlbaum. Rodgers, M. D., & Nye, L. G. (1993). Factors associated with the severity of operational errors at Air Route Traffic Control Centers. In M. D. Rodgers (Eds~), An analysis of the operational error database for Air Route Traffic Control Centers (DOT/FAAJAM-93/22). Oklahoma City, OK: Human Factors Research Laboratory, Civil Aeromedical Institute, Federal Aviation Administration. RTCA (1995). Report of the RTCA Board of Directors select committee on free flight. Washington, D.C.: Author.

[5] [6]

[7]

[8]

[9]

[10] [11] [12]

[13]

[14]

Errors and Error Recovery

Tjerk. W. van der Schaaf~, L. Kanse 2 ~Safety Management Group, Eindhoven University of Technology, P.O. Box 513, Pav. U8, 5600 MB Eindhoven, The Netherlands e-mall: [email protected] 2Human Reliability Associates, HRA-NL:Arenberglaan 280, 4822 ZR Breda, The Netherlands e-mail: superbear @compuserve.com

Abstract: This paper highlights the positive role that human operators often play in preventing small failures and errors from developing into an actual system breakdown. The resulting 'near misses' may provide an insight into a powerful alternative to human error prevention, namely: human recovery promotion. Theoretical approaches to modelling error recovery are discussed and translated into empirical research questions. These are partly answered by a number of pilot studies. The main conclusions are that error recovery is much more than simple luck or coincidence, that its root causes can be identified, and that these should have design implications for the technical and organisational context of the human operator's task.

1. Introduction The basic focus of reliability and performance management so far has been on the prevention of errors and failures. When taking a closer look at what exactly we want to prevent, we find that it is rather the (negative) consequences of a failure than the occurrence of the failure itself [1, 2, and 3]. This idea leads to a relatively new, alternative or complimentary area of research, namely that of recovery promotion. Recovery factors are those factors that contribute to (complete or partial) recovery once an error or failure has occurred, thus preventing or reducing the negative consequences of that error or failure. This paper describes a research project focussing on the positive role that human operators often play in the recovery process. In the next chapter a brief overview is given of theoretical approaches and existing insights in the domain of error recovery. We will first describe a simple incident causation model, in which the presence or absence of successful human recovery plays a role in determining the effects of process deviations which result from

28

Errors and Error Recovery

(combinations of) technical, organisational and human failures. This forms the starting point of our reasoning. Then we will describe the human recovery process in more detail and from this we will propose four ways of classifying human recovery. In the third chapter, these theoretical approaches and existing insights are translated into specific empirical research questions on which the described research project focussed. A set of recent pilot studies of incidents in a steel plant and an energy production unit, as well as medical errors in a surgical ward, will then be presented in chapter 4, and their results will be used to formulate initial answers to the research questions. Data from a petro-chemical plant is included in the fifth chapter, which shows again the importance of human operators in the recovery process and the difference between factors contributing to failure and those contributing to recovery. Finally, in chapter 6, implications of the research findings for designing recovery into socio-technical systems will be discussed.

2. Theoretical Approaches & Existing Insights 2.1. Incident Causation Modal In [4] a simple incident causation model is used (see figure 1). This model shows that when incident development cannot be stopped by the system's predetermined barriers [5] and lines of defence, the only distinguishing factor between an accident and a near miss is the presence or absence of successful 'accidental' or unplanned recovery. Although actual accidents also may contain attempts at recovery, it is obvious that near misses as defined above are the optimal source of data to study the phenomenon of recovery as the positive counter 9art of failure.

r~:~i~ [ f~il~r, IX

I ,--1

,J.,.i_, i -

I

failu

e

Figure 1: The incident causation model.

i

Errors and Error Recovery

29

The Eindhoven Classification Model (ECM) of system failure [6, 7, and 4] has succeeded so far in modelling the failure factors (or root causes) which lead to process deviations. Figure 1 shows that three broad categories of root causes of incidents and accidents may be distinguished: technical, organisational and human (operator) failures. The ECM breaks each of these three categories down into more detail. The different types of human failure distinguished in the ECM reflect the three levels of a very well-known hierarchical model of human behaviour, from which the main concepts are widely accepted by both researchers and practitioners in the domain of human performance and reliability management: Rasmussen's SRK model [8]. The SRK model distinguishes three levels of behaviour, namely skill-based, rule-based and knowledge-based. Skill-based behaviour refers to routine tasks, requiring little or no conscious attention during task performance. During rule-based behaviour, familiar procedures have to be applied to frequent decision-making situations. Knowledge-based behaviour refers to task situations where more complex problem-solving activities have to be performed, for which no readily available standard solutions (e.g. rules or procedures) exist. At each of these three levels of behaviour, errors can be made, all three being quite different from each other in nature [9]. Slips and lapses occur most often during skill-based behaviour. Slips occur during the actual execution of a task: the action itself or outcome of the action doesn't correspond with the original intention of the individual performing the task. Lapses are the result of problems during the storage of an action plan, resulting in an error occurring when it is time to apply the plan. Mistakes can occur either during rule-based behaviour (when an inappropriate, but familiar, pre-established plan or procedure is selected and applied) or knowledgebased behaviour (when no appropriate routine plan is available and a plan has to be developed based on the knowledge possessed by the individual).

2.2. Human Recovery Process As shown in figure 1 and the preceding text, recovery is the distinguishing factor between an accident and a near miss. The following definition of human recovery is proposed by Van der Schaaf [10]: "the (unique?) feature of the human systemcomponent to detect, localize and correct earlier component failures. These component failures may be either own previous errors (or those of colleagues) or failing technical components (hardware and software)". This definition implies the following phases in the recovery process: 9

detection: of deviations, symptoms, etc.;

9

localisation: of their cause(s) (diagnosis in the strictest sense);

9 correction: of these deviations by timely, effective counter-actions, after which these deviations are nullified and the system returns to a stable status again. Other theorists also distinguish three recovery process phases that are more or less similar to the ones proposed by van der Schaaf. Zapf and Reason [11] distinguishes

30

Errors and Error Recovery

three main process phases: the error itself, error diagnosis and error recovery. The error diagnosis phase consists of error detection and error explanation (corresponding with Van der Schaaf's detection and localisation phases), and the error recovery phase (which corresponds with Van der Schaaf's correction phase) consists of a planning and an execution step. Van der Schaaf's three recovery process phases can also be found back in Kontogiannis' framework for the analysis of the error handling process [12], in which the error detection, error explanation and error correction aspects of the error handling process are distinguished. Most of the research on the recovery process has focussed on the detection phase. This focus makes sense, since the whole recovery process depends on errors or failures being detected. Without detecting that a failure has occurred, there is no change that a complete recovery process will be started. Based on her diary study of everyday, self-produced errors, Sellen [13] has identified action-based detection, outcome-based detection, and detection through limiting function as detection modes for the detection of slips and mistakes. Action-based detection involves catching an error on the basis of perception of some aspect of the erroneous action itself. Outcome-based detection involves detection based not on the perception of the action itself, but on some aspect of the consequences of the action. Errors detected via the limiting function mode are detected because constraints in the person's environment prevent further action. For each of the detection modes, Sellen has also produced some insights in factors influencing their success or failure. In an experiment involving subjects performing a computer-based task, Bagnara et ai. [14] use a distinction between different types of detection that corresponds closely with Sellen's detection modes. According to them, detection can be the result of three types of mismatch between operator's expectations and the available information about the performed or to be performed activities, respectively: inner feedback (where information concerning the failed expectation or mismatch is internal to the individual), external feedback (when the information concerning the mismatch is present in the environment) and forcing function (where properties of the environment force the detection of the error). Let us turn now to the phases of localisation and correction. As noted by Reason [9] in his Generic Error Modelling System (GEMS), people seldom go through the entire analytic process of fault diagnosis when confronted with a deviation. This was confirmed by Brinkman [15], who collected verbal protocols during a fault finding task. He observed the following three reactions after his subjects detected an error in their reasoning process: D Ignore the error and continue: rely on system redundancy and subsequent error recovery factors. 9

Simply repeat the most recent sequence of actions: try again, without any attempts at fault localisation.

Errors and Error Recovery

31

Attempt fault localisation and optimise corrective actions: either by forward analysis (repeat the most recent action sequence and check every step), or backward analysis (trace back from symptom detection to previous actions, until the error is found). Bagnara et al. [14] observed the same and distinguish six behavioural patterns after an error is detected, based on the amount and type of analysis that takes place. Immediate correction, automatic (fast) causal analysis, conscious causal analysis (a hypothesis is formulated), explorative causal analysis (more hypotheses are tested), and overcoming of the mismatch if all else fails. We described thus far the three phases of recovery and some of the empirical evidence and existing insights about these phases. What about the probability of occurrence of a successful recovery process? In their paper about the nature of recovery from error, Embrey and Lucas [16] discuss several factors affecting the probability of recovery from error. Their distinction of human error types is based on Rasmussen's SRK model [8] (see preceding text in this paragraph) and Reason's distinction between the three error types slips, lapses and mistakes (see preceding text, also explained in [9]): on the one hand there are the slips and lapses, or errors of execution, on the other hand mistakes, or errors of judgement and planning. Embrey and Lucas argue that the differences between recovery from slips and lapses and recovery from mistakes "may be described in terms of the amount, type and timing of feedback (regarding the error), and in the relationship between the factors which cause the error and those which influence its detection". Their main points can be summarized as follows: 9

Recovery from skill-based slips and lapses is relatively independent of the process causing the initial error, and the probability of detection and recovery is quite high, especially if more operators work on a team (to detect someone else's slip), appropriate feedback is provided, and independent checks are performed by supervisors. 9 For rule- and knowledge-based mistakes the opposite holds: their recovery depends largely on the information processing that caused the error in the first place; probability of recovery is small as a result of cognitive limitations during information processing, such as group-thinking and fixation on confirmation of one hypothesis for which some initial support is found. Factors that will increase the probability of recovery from mistakes include the introduction of a new team to the situation, replacing the team originally dealing with the situation, the provision of some form of decision support system, and rapid feedback regarding the changed state of the system involved. The main implication of these insights for our research project is that the nature of the preceding human error(s) should be highly predictive of any subsequent recovery.

32

Errors atut Error Recovery

Findings from the computer programming experiment by Bagnara et al. [14] also indicate that a dependency exists between type of error, the detection mechanism and subsequent recovery patterns: Slips were most often detected via external feedback, and recovered via immediate correction without causal analysis. Rulebased mistakes had no preferred detection mechanism, and most often conscious or (less often) automatic causal analysis was undertaken to recover. Knowledge-based mistakes were most often detected via forcing function, and the most used recovery strategy was to undertake an exploratory causal analysis or otherwise to try to get rid of the mismatch. In the preceding text we have shown how human recovery from errors and other failures can form the distinguishing factor between an accident and a near miss. We described the three phases a recovery process generally follows, with some existing insights and empirical evidence about each of these phases (detection mechanisms and the behavioural patterns for Iocalisation and correction followed by human operators in order to recover from an error or failure after detection of its symptoms). We also described how the probability of successful recovery and the preferred recovery strategy are linked to the type of preceding failure or error. As pointed out already in chapter 1, in the domain of human performance and system reliability, far less attention has been given so far to research on error recovery than to research on error prevention. While the insights described above are useful as initial steps towards a better understanding of the recovery process, a comprehensive model of the recovery process and contributing factors has yet to be developed and validated.

2.3. Modelling Human Recovery Aspects Based on the preceding overview of the recovery process, existing insights and empirical evidence, the following four ways of classifying (human) recovery aspects can be proposed as potential ways towards the development of a comprehensive model of the recovery process and contributing factors, as described above: a)

Classification according to Recovery Process Phase:

Recovery promoting factors can be categorised according to whether they contribute to error or failure detection, to error localisation or diagnosis of what went wrong, or to the actual correction of the problem. b)

Classification according to Operator Reaction After Symptom Detection:

As mentioned earlier, different recovery patterns or mechanisms used by human operators can be distinguished, based on the amount and type of causal analysis they undertake after error detection and the solution that is chosen to overcome the problem. Recovery can be classified according to which of these recovery patterns is followed.

Errors and Error Recovery

e)

33

Classification Based on Preceding Failure:

Both Van der Schaaf [7] and Van Vuuren [4], and Embrey and Lucas [16] provide the rationale of this taxonomy. The probability of recovery and the recovery mechanism that is followed may depend on the preceding type of failure or error, so that based on the type of failure predictions can be made with regard to recovery possibilities. d)

Classification according to Type of Recovery Promoting Factor:

Such a classification may be the most important one for the designers of sociotechnical systems. This type of classification, more so than the other three mentioned previously, focuses on exactly the factors that contribute to recovery per se, thus providing a taxonomy of actual recovery promoting factors, instead of a classification according to factors to which recovery factors are merely linked (c), or one that is not really discriminating with regard to the recovery factor itself (a) (since one recovery promoting factor could influence more than just one phase of the recovery process), or a classification that focuses on the recovery mechanism as opposed to the factors that make it possible to start following any recovery mechanism in the first place (b). Combined with the incident causation model from figure 1, the Eindhoven Classification Model (ECM, see beginning of this paragraph) for root causes of failure, with various subcategories for technical, organisational, and humanskill, rule- and knowledge-based factors, could serve as a basis for the development of a model of the recovery process and recovery promoting factors, too. Technical factors contributing to recovery are the barriers and defences (such as system feedback provided via process control systems, and safety valves in chemical industry) built into the technical system in which the human being operates, that prevent a dangerous situation/process deviation from developing into a potential incident. Organisational recovery promoting factors include the use of teams and supervisory checks. Human operator recovery promoting factors include training, self-checks, verbalisation of planning during task performance, etceteras. The following suggestions for adapting the ECM to a model of recovery promoting factors (instead of failure factors) can already be made: Technical design of the process: aim at maximum reversibility of process reactions [8], to allow for correction, and 'linear interactions' plus 'loose coupling' [17], of process components; these may be achieved by structural characteristics (e.g. buffers, parallel streams, equipment redundancy) and by dynamic characteristics (e.g. speed of process reactions, response delays). Technical design of the man-machine interface: aim at maximum observability [8], to allow early detection of deviations and their effects (e.g. transparency instead of alarm inflation).

34

Errors and Error Recovery

O__rganisational and management factors: especially an updated, clearly formulated and well-accepted set of operating procedures, and a positive safety culture must be mentioned here (see also [4]). H__uman operator factors: optimize the cognitive capabilities (e.g. accurate mental process model) of operators through selection and (simulator-) training, but also by supporting them with software tools to test hypotheses and avoid certain biases. Another promising model of the human recovery process and related recovery taxonomy, is proposed by Kontogiannis [12]. It shows the sequence of the error (slips and mistakes are distinguished) itself, followed by user strategies in error recovery (which are influenced by management and workplace factors), which effect the recovery- or error handling process phases of detection, explanation and correction, finally leading to either backward, forward or compensatory recovery as the ultimate corrective goal of the recovery process. Backward recovery means bringing the system back to its state from before the error; with forward recovery operators bring the system to an intermediate state to buy time in order to solve the problem later, and with compensatory recovery operators may activate redundant equipment to bring the system to the state they initially intended. Based on an elaboration of other researchers' findings, Kontogiannis has produced two additional models, one detailing each of the error handling process phases and another detailing the different user strategies possible for error recovery.

3. Empirical Research Questions Based on the proposals in the previous section, the human recovery research project of the Eindhoven Safety Management Group has focussed on the following empirical research questions: 1) Is recovery more than sheer luck or coincidence? If so, then recovery can be built into a socio-technical system and be managed. 2) Can recovery be classified with the same root causes as those used for failures? If so, what is the contribution of human recovery in relation to technical and organisational failure barriers? How large is the contribution of human recovery in a variety of task situations and over a variety of system effects? 3) Are recovery factors identical to failure factors in a given socio-technical system? If so, then preventing errors and promoting recovery would have to focus on the same socio-technical system aspects. 4) In which phase(s) of the recovery process do recovery factors mostly contribute to system performance: symptom detection, fault Iocalisation, or correction?

Errors and Error Recovery

35

4. Pilot Studies Pilot studies investigating the recovery process have been carried out in a steel production plant [18], a surgical ward [19] and an energy production unit of a chemical plant [20]. A variety of system effects have been investigated: safety, reliability, and environmental effects of system breakdown.

Method The critical incidenl technique, using confidential interviews [21], has been used in all pilot studies to collect data on a set of recent near misses in each of the pilot studies.

Results The results of the pilot studies are summarised in the following tables (see also [22]). .................................................................................... i;~;~;;~;;~.~;;~!~EZ~;t~!!~;~i~Z~.~i~i~iEL;i i...~te~!-~-!~t ....................................................~.~. ...................................................................................!.~4 ...........................................................3.4. ..................................................................i i.....~.r~i~.~...~.~ ......................................!..Z................................................................................... ~.5............................................................. ~. .................................................................. ! i.....~ne~g~..~.~.~..~J.L.~....~.~ ................................................................................... .8~ .........................................................~. ..................................................................

Table 1: Results of pilot studies. For the analysis of both the failure and the recovery factors, the subcategories of the ECM have not been used, only the main groups of technical (T), organisational and management (O) and human operator (H) factors, plus unclassifiable (X).

iSteelplant

21% 33% 44% 2% iSteelplant

47%]21% 121% [11% i

i Surgical ward 6% 37% 56% 11% i Surgicalward 16%i20% i55% i9% , ............................................................................................................................................ .i...................~................................................................................... ~...................~..................3...................... Energy production unit 40% 23% 33% i4% ienergy production unit 27%[5% 166% i! 2% J i i i Table 2: Analysis of failure and recovery factors. Based a range of only 2 to 11% unclassifiable (that is: luck or coincidence) causes of recovery, a positive answer to research question 1 can be given: around 90 % or more of all recovery factors are clearly technical, organisational or human in nature and therefore eventually manageable. Comparison of the percentages of human recovery root causes with those of human failure root causes shows that they both vary between the case studies, respectively 21 to 66% and 33 to 56%. The human component should therefore also be taken seriously in terms of recovery possibilities (question 2). Zuijderwijk [20] shows that the patterns of failure and recovery factors are clearly different. Rule- and skill-based factors dominate the operator failures, while human

36

Errors and Error Recovery

recovery also includes knowledge-based insights as very important. Similarly, 'material defects' are the most prominent technical failures, while 'design' covers all technical recovery factors (see question 3). Finally, data from the steel plant [18] showing that 26% of the recovery factors contribute to the detection phase, 9% to the localisation phase, and 53% to the correction phase, provides an answer to question 4: Hardly any recovery process goes through the more analytic localisation phase. Again, this could be interpreted as a confirmation of Rcason's GEMS model, but there is also the possibility of an explanation in terms of time stress. If recovery is present only in the verb, last phase of the accident production chain of events (as was the case in most of the steel plant near misses) there may simply not be time enough for a time-consuming diagnostic effort; detection and correction 'just-in-time' may be all one can do in such cases.

5. Data From A Petro-Chemical Process Plant Near Miss Management System ! ss.~i 3o~o

Contribution i Failure & Recovery

' -----~

I

25%

0%

-

IE TC "~1r O~ OM OK HK1 HK2 HR1 HR2 HR3 HR4 HR5 HR6 i-l$1 HS2 XXX

Figure 2: NMMS data from petro-chemicai plant. Figure 2 is derived from data from the Near Miss Management System (NMMS) database of a large, international petro-chemical plant in Rotterdam. This NMMS contains more than 3000 reports so far and has been operational since 1994. Now circa 800 reports per year are added to the system. For those incidents that are selected for a causal tree analysis and classification following the ECM (10-20%), the figure shows the distribution across the different categories of root causes both for failure factors and recovery factors. Once more the difference in distribution of the failure factors from that of the recovery factors is shown, and the obvious importance of the role of the human operator in recovery.

Errors and Error Recovery

37

6. Implications for Socio-Technical System Design Despite the immaturity of the proposed models and classifications, and the small amount of recovery data gathered so far, these ideas and results are promising enough to allow us to formulate the following tentative implications for designing a socio-technical system: 9 Consider recovery promotion as an alternative to failure prevention, especially when certain errors or failures are predictably unavoidable. 9 Do not simply design out failure factors without considering the possible reduction of recovery factors: raising the level of automation in process control, or installing too many decision support tools for your operators may leave them helpless under certain situations. 9 Support all recovery phases (detection (observability), localisation, correction (reversibility)) primarily by means of an optimal man-machine interface. 9 Invest in deep process knowledge of operators: reasoning beyond procedures appears to be essential for many recovery actions. Also, consider error management training [1]: learning to learn f r o m errors is perfectly in line with the concept of recovery promotion.

References [1]

[2] [3] [4] [5] [6] [7] [8] [9]

Frese, M. (1991). Error management or error prevention: Two strategies to deal with errors in software design. In H.J. Bullinger (Ed.), Human aspects in computing: Design and use of interacive systems and work with terminals (pp. 776-782). Amsterdam: Elsevier Science Publishers. Frese, M., van der Schaaf, T.W., & Heimbeck, D. (1996). Error Management and Recovery in Technical System Design. In: Proceedings of the '96 Multiconference, Symposium on Robotics and Cybernetics, July 1996, Lille, France, pp. 161-164. Kanse, L., & Embrey, D. (1998). Recovery from failure. In: Proceedings of 4 'h World Conference on Injury Prevention and Control, May 17-20, Amsterdam, the Netherlands, p 1020. Vuuren, W. van (1998). Organisational failure: An exploratory study in the steel industry and the medical domain. PhD thesis, Eindhoven University of Technology. Svenson, O. (1991). The Accident Evolution and Barrier function (AEB) model applied to incident analysis in the processing industries. Risk Analysis, 11,499-507. Schaaf, T.W. van der (1991). Development of a near miss management system at a chemical process plant. In: Near miss reporting as a safety tool (Schaaf, T.W. van der, D.A. Lucas & A.R Hale (Eds.)). Butterworth Heinemann, Oxford. Schaaf, T.W. van der (1992). Near miss reporting in the chemical process industry. PhD thesis, Eindhoven University of Technology. Rasmussen, J. (1986). Information processing and human-machine interaction. Amsterdam: Elsevier Science Publishing. Reason, J.T. (1990). Human Error. Cambridge: Cambridge University Press.

38

Errors and Error Recovery

[10] Schaaf, T.W. van der (1988). Critical incidents and human recovery. In L.H.J. Goossens (Ed.): Human recovery: Proceedings of the COST AI Seminar on Risk Analysis and Human Error. Delft: Delft University of Technology. [1 I] Zapf, D., & Reason, J.T. (1994). Introduction: Human errors and error handling. Applied Psychology: An International Review, 43(4), 427-432. [12] Kontogiannis, T. (1999). User strategies in recovering from errors in man-machine systems. Safety Science, 32, 49-68. [13] Sellen, A.J. (1994). Detection of Everyday Errors. Applied Psychology: An International Review, 43(4), 475-498. [14] Bagnara, S., Ferrante, D., Rizzo, A., & Stablum, F. (1988). Causal analysis in error detection and recovery: when does it occur? Paper presented at the international conference on joint design of technology, organization and people growth, Venice, October 12-14, 1988. [15] Brinkman, J.A. (1990). The analysis of fault diagnosis tusks: Do verbal reports speak for themselves ? PhD thesis, Eindhoven University of Technology. [16] Embrey, D.E., & Lucas, D.A. (1988). The nature of recovery from error. In L.H.J. Goossens (Ed.), Human recovery: Proceedings of the COST A1 Seminar on Risk Analysis and Human Error. Delft: Delft University of Technology. [17] Perrow, C. (1984). Normal accidents: living with high-risk technologies. New York: Basic Books. [18] Mulder, A.M. (1994). Progress report 1993 of the SAFER project. Report Hoogovens steel industry (in Dutch). IJmuiden: Hoogovens. [19] Hoeff, N.W.S. van der (1995). Risk management in a surgical ward (in Dutch). MSc thesis, Eindhoven University of Technology. [20] Zuijderwijk, M. (1995). Near miss reporting in reliability management. MSc thesis, Eindhoven University of Technology. [21] Flanagan, J.C. (1954). The critical incident technique. Psychological Bulletin, 51,327358. [22] Schaaf, T.W. van der (1996). Human recovery of errors in man-machine systems. In: CCPS '96, International conference and workshop on process safety management and inherently safer processes, October 8-1 I, Florida, USA. New York: American Institute of Chemical Engineers.

Analysis and Prediction of Failures in Complex Systems: Models & Methods

Erik Hollnagel Graduate School for Human-machine Interaction,University of Link0ping, Sweden

Disclaimer: The following text is a slightly revised version of the outline submitted for the Workshop on "Human Error and System Design and Management", that was held on 24-26 March 1999 in Clausthal, Germany. The text was never written with the intention of providing a comprehensive and detailed presentation of the models and methods that are applied in failure research, but rather as a short outline of the main issues. It is therefore in this vein that it should be read.

1. Introduction Accident analysis and performance prediction have traditionally been pursued as two separate activities, despite the obvious fact that they refer to the same reality namely unexpected events in complex systems, leading to unwanted outcomes. In both cases the common motivation has been the dramatic rise since the 1970s in the number of cases where human action failures have been seen as the cause of accidents. Accident analysis has had a strong psychological flavour, looking inward toward "human error mechanisms" and various deficiencies of information processing in the mind [6] Performance prediction has been dominated by the engineering quest for quantification, as epitomised by the PSA event tree, and models and methods have been aimed squarely at that [2]. In both cases there has been a strong predilection for considering "human error" as a category by itself, either referring to complex information processing models of how actions can go wrong, or estimates of single "human error probabilities".

2. Models And Methods For Accident Analysis An accident model is a generalised description of how an accident may have happened. Such models are invariably based on the principle of causality, which states that there must be a cause for any observed event. The first accident models

40

Analysis and Prediction of Failures in Complex Systems: Models & Methods

tended to see accidents as caused either by failures of the technology or incorrect human actions ("human errors"). This view was gradually extended to recognise both the contribution of latent system states, and the complexity of conditions that could end in an incorrectly performed human action - even leading to the extreme notion of "error" forcing conditions. In contemporary accident models, a distinction is made between actions at the "sharp" end, which often are the initiating events, and actions at the "blunt" end, which create the conditions that either make an action failure near inevitable or turn minor mishaps into major disasters. Despite these developments, specifically the increasing sophistication in accounting for the organisational determinants of accidents [7], there is an almost intransigent preference to refer to "human error" as a singular concept. The history of accident analysis clearly demonstrates that the notion of a cause itself is an oversimplification, since a cause is an attribution after the fact or a judgement in hindsight, rather than an unequivocal fact. This acknowledgement notwithstanding, accident models seem to be firmly entrenched both in the idea that a "true" or root cause can be found, and in the idea that "human errors" necessarily must be part of the explanations. This view should be contrasted with the so-called ecological view which points out that action failures are both an unavoidable and necessary element of efficient human performance [1]. Without making shortcuts or using heuristics we would be unable to work effectively, and without failing every now and then we would not be able to learn. The challenge for cognitive engineering and system design is, of course, to make sure that the systems are so robust that minor variations in performance do not lead to fatal consequences, yet so sensitive that operators have the freedom necessary to create and optimise control strategies.

3. Models And Methods For Performance Prediction As mentioned above, performance prediction has on the whole been separated from accident analysis. This becomes obvious if one tries to apply any of the established "human error" models for prediction. Neither the methods nor the categories used allow a reversal of the direction from going into the past to going into the future. Performance prediction has typically been carried out in the context of Human Reliability Assessment (HRA), which has established itself as the premier way of finding the human failure probabilities that are required by Probabilistic Risk Assessment (PSA). Performance prediction, as part of HRA, can in principle confine itself to an investigation of the ways in which actions can possibly fail, often referred to as action error modes - or just error modes. One contentious issue is whether "human error probabilities" (HEP) can be described prior to the effect of the performance conditions (Performance Shaping Factors), or whether it should not rather be the other way around, i.e., that performance characteristics depend on the working conditions. Although the quest is for the probabilities or numbers, it is generally

Analysis and Prediction of Failures in Complex Systems: Models & Methods

41

acknowledged that any quantitative prediction must be based on a qualitative analysis, which provides the basis for making sense out of the numbers. The qualitative analysis should also be of interest to system design in general, since it is important to be able to anticipate the ways in which a system can fail. The exchange of ideas and methods between system design, specifically HCI/HMI, and HRA has nevertheless been quite limited. (The same can be said about software engineering.) Performance prediction requires an underlying model of human performance. This should not be a model of human information processing in disguise, but a model of how human performance is influenced by and reflects the context. This type of model is exemplified by the notions of "cognition in the wild" or "embedded cognition", although neither have considered performance prediction specifically [5]. The general type of model has become known as contextual control models, which focus on how humans and machines function as joint (cognitive) systems, rather than on how humans interact with machines [3]. The emphasis is on how human-machine co-operation maintains an equilibrium rather than on how humancomputer interaction can be optimised. This de-emphasis of descriptions of "cognition in the mind" mirrors a distinction between performance phenotypes (error modes) and performance genotypes (putative causes). The Cognitive Reliability and Error Analysis Method (CREAM) is a proposal for a model and a classification scheme that can be used for both accident analysis and (qualitative) performance prediction [4]. It entails a common, bi-direclional method, which can serve as the foundation for more detailed approaches to analysis and prediction. CREAM is based on the notion that people (process operators and others) strive to maintain control of what they do, but that control both may he lost and regained. This has shown itself to be practically useful to analyse performance and accidents for both individuals and teams, and to predict the error modes that may be expected to occur under given conditions.

References [1] [2] [3] [4] [5] [6] [7]

Amalbeni, R., 1996, La conduite des syst#mes h risque. Paris: PUF. Dougheny, E. M. Jr., Fragola, J. R., 1988, Human reliability analysis. A systems engineering approach with nuclear power plant applications. New York: John Wiley & Sons. Hollnagel, E., 1993, Human reliability analysis: Context and control. London: Academic Press. Hollnagel, E., 1998, Cognitive reliability and error analysis method. Oxford: Elsevier. Hutchins, E., 1995, Cognition in the wild. Cambridge, MA: MIT Press. Reason, J. T., 1990, Human error. Cambridge, U.K.: Cambridge University Press. Reason, J. T., 1997, Managing the risks of organizational accidents. Aldershot, UK: Ashgate.

Scenarios, Function Allocation and Human Reliability

Michael. D. Harrison Department of Computer Science, University of York, Heslington, York, YOI0 5DD. UK. e-mail: michael.harrison@ cs.york.ac.uk

Abstract: This paper briefly reviews research on methods of human error assessment and allocation of function at the University of York. One connection between the two methods developed is the use of scenarios.

1. Introduction - Requirements for a Representation o f W o r k This position paper briefly reviews research within the Human Computer Interaction Group at the University of York. The research is concerned with ways in which scenarios can be used to represent work that human operators have to perform in safety critical contexts. The problem is to support systems engineers as they develop computer-based systems in safety critical settings. How can such systems be designed so that they are less prone to human error? How can better and more traceable decisions be made about what parts of the system should be automated? An important part of this system engineering problem is to understand the broader system and its context sufficiently early in the design process, not after implementation has been completed when changes would be extremely costly. Work at York, mainly sponsored by British Aerospace but also by the UK Engineering and Physical and Research Council and the UK Defence Evaluation Research Agency, has focussed on two particular aspects of design: 9

Analysing ways in which an artefact will encourage interaction failure early in design (Human Error Assessment);

9

Supporting rationale about which parts of a system should be automated to improve operator effectiveness (Allocation of Function).

It is widely perceived that there are problems with existing methods of Human Error Assessment [8] and Allocation of Function [7]. Human error assessment is seen to be extremely time consuming. Most methods are based on "exhaustive" task

44

Scenarios, Function Allocation and Human Reliability

analyses or expensive team based approaches. They tend to be used late in development and cannot be used to establish early requirements. It is also perceived that these methods fail to recognise effectively the particular demands of the context in which the artefact is being used. Function allocation methods have similar problems dealing with context, They are based on fixed lists of characteristics best supported by human or machine. A system is initially described in terms of a set of functions. Functions are taken as the basic activities that have to be performed in order to achieve the work. These functions are device independent in the sense that no decision has been made about whether they should be performed by either human operator or device. The decision about how the function should be implemented is based on what capabilities are required to perform the function. This activity is too simplistic. Functions are not carried out in isolation and depend on many factors and are highly sensitive to particular contexts. Scenarios (see [3] for a discussion) help to expose the complexity of the working context. They provide information about the organisational trade-offs that are under consideration. A scenario is a narrative about a situation in which the work is being carried out that is either typical of the work or represents a situation which is "extreme" at the limit of the performance of the system. It provides a context in which a newly designed specification can be explored. Scenarios have a number of characteristics, and in order to encourage the identification of these characteristics during elicitation we have produced a questionnaire that clarifies who is involved in the scenario, what the physical situation of the scenario is, as well as the task context. Scenarios are derived by a number of means. They are gathered through discussion with users of an earlier version of the system. Here elicitation may focus on typical examples of the work or situations which crew found particularly difficult to handle. They can be derived from accident reports. Scenarios provide a basis for the process of imagination about how a new system might work in practice. Because of their narrative form, they provide an ideal medium for communication. Their very richness makes it difficult however to see how to generate scenarios systematically in order to capture all the important attributes of the working environment. We shall now consider the way that the two methods we have developed make use of scenarios and in a final section briefly discuss ongoing work that is concerned with the generation of scenarios.

2. A Technique for Human Error Assessment (THEA) THEA [6] has been developed for the purpose of assessing a system artefact during the early stages of its design. The technique uses scenarios as a rich and selective form of work description. During the process of assessing the design, a task representation is used to structure the scenario's description of the actions that take place. This structuring enables a systematic understanding of possible human errors

Scenarios, Function Allocation and Human Reliability

45

that might arise as a result of the design of the artefact. Evaluation is carried out in the context of these scenarios through formalised questions based on Norman's [9] model of human cognition. From this, we can identify a number of cognitive failures or ways in which human information processing can fail, possibly resulting in "incorrect" behaviour. These failures are associated with goals, plans, actions, perceptual failures and failures of interpretation. This method has been used for a number of types of system. At the "sharp end" it has been used to analyse a number of subsystems in an aircraft flight deck and to analyse crew operation of a ship based command and control system. It has also been used to analyse procedures associated with producing software upgrades for a safety critical system in the National Air Traffic Control Service [4].

3. Allocation of Function Our allocation of function technique uses scenarios as a basis for deciding how alternative proposals for automation match defined roles for the human operators [5]. Here scenarios are required that capture the range of functions and contexts that are involved in the work. They are used to provide a contextual analysis of whether a function should be automated. A matrix is generated for each scenario in which the list of functions that are used in the scenario are placed in the matrix according to two scales: how feasible it is to automate the function and how separable the function is from the role(s) of the operator(s). Functions that appear in the centre of the matrix are candidates for partial automation and they are subjected to further analysis to give the operators tasks that are coherent with their roles. Here the work context offered by each scenario provides an integrative function so that a utility judgement can be made about whether a function is automated or not is not decided in isolation. The automation decision can then be made by considering each function and optimising over the set of scenario based scenarios.

4. A Step towards Generating Scenarios It is not possible to generate a complete and adequate set of scenarios. However, as a step towards aiding this process we have developed a method using mathematical specifications of interactive systems. Here the focus for scenario generation is the extent to which an artefact satisfies interface properties. A primary focus to date has been the complexity of interface modes, but other interface properties are feasible. Interface properties are checked using a model checker [1,2]. If the property fails, the model checker produces a sequence of actions that provides a counter example. This sequence of actions forms a basis upon which a domain expert or a human factors expert can construct a scenario. A story is told based on the failure.

46

Scenarios, Function Allocation and Human Reliability

References [1]

[2] [3] [4]

[5]

[6]

[7] [8] [9]

Campos, J.C. & Harrison, M.D. (1999) Detecting Interface Mode Complexity with lnteractor Specifications. Submitted to ACM Transactions On Computer Human Interaction. Campos, J.C. & Harrison, M.D. (1999) From lnteractors to SMV: a case study in the automated analysis of interactive systems. Technical Report YCS 317, Department of Computer Science, University of York. Carroll, J. M. (ed.) (1995) Scenario-Based Design: Envisioning Work and Technology in System Development. Wiley. Cartmale, K. & Forbes, S.A. (1999) Human error analysis of a safety related air traffic control engineering procedure. People in Control. lEE Conference Publication 463. pp. 346-351. Dearden, A.M., Harrison, M.D. & Wright, P.C. (1998) Allocation of function: scenarios, context and the economics of effort. International Journal of HumanComputer Studies Vol. 51. in press. Fields, B., Harrison, M. & Wright, P., 1997. THEA Human Error Analysis for Requirements Definition. Technical Report YCS 294, Department of Computer Science, University of York. Fuld, R. (1999) The myth of function allocation revisited. International Journal of Human Computer Systems. Vol. 51 in press. Hollnagel, E. (1993) Human reliability analysis: Context and control. London: Academic Press. Norman, D.A. (1988) The Psychology of Everyday Things. Basic Books.

Experience Feedback and Safety Culture as Contributors to System Safety

Markus Sch0belt, Steffen Szameitat~ ~University of Technology Berlin, Research Center System Safety FR 3-8, Franklinstr. 28, D-10587 Berlin e-mail: Markus.Schoebel @TU-Berlin.DE

Abstract: The present paper examines safety management and safety culture as psychological research topics and intervention, strategies in the specific context of process control industries. Structured methods and techniques as well as new information technologies are pointed out as key issues of Safety Management Systems. Safety Culture as an up-coming concept in addition to traditional Safety Management is illustrated in terms of procedural non-compliance and implicit norms.

I. Introduction Because of the growing complexity of technical installations and the increased fatalities after major accidents, work psychologists discovered a new field within the classical research area: system safety. System Safety is a quality of a system to function without major breakdowns, under predetermined conditions with an acceptable minimum of accidental loss and unintended harm to the organization and its environment [1]. With the perspective of high risk organization as technical and social subsystems, a multi-disciplinary human-centered research developed in the last decade. Among the different paradigms in system safety research, two concepts get growing importance among practitioners: Safety management [2] and safety culture [3]. Both employ a theoretical framework which holistically considers the interactions of Man-Technology-Organizations as salient factors for system safety, hence transcend received approaches of the immediate human-machine interface. The Safety Management paradigm models different management functions and their relation to safety. Safety Culture paradigms provide conceptual explanations for cultural aspects in high-reliability organizations.

48

Experience Feedback and Safety Culture as Contributors to System Safety

2. Safety management The aim of safety management is to optimize the reliability of men, machine and organization with traditional management systems. The main feature of Safety Management Systems (SMS) is the production of new knowledge concerning safe design and operation. Rasmussen [4] identifies three major strategies within safety management: The feedback strategy for controlling hazards using experience feedback from incident investigation and statistics, (2) the feedforward strategy focuses on proper design and operation using estimations of failure probability, and (3) a combined feedforward and feedback strategy. Mostly feedforward methods, like the Probabilistic Safety Analysis (PSA) for technical components or the Human Reliability Analysis (HRA) for operators, depend on safety data collected in the past. Thus, for every strategy for improving safety a management system is demanded regulating collection, processing, distribution, storage and use of relevant information as a basis for decisions and effective corrective measures. Explicit methods for collecting information and data bases for storing it are the key issues of a SMS. Methods for the causal analysis of incidents produce new knowledge about reliable design and operation of high hazard systems, which increases the knowledge from operation experience in an organization as a whole [51. Databases within a SMS are critical points as well. New information technologies provide advantages for information processing and storage [6]: they lower administrative efforts, secure the timeliness of information and provide user guidance during the data entry, analysis and recall. But in the same time, there is the risk of collecting and storing vast amount of information with little value of expressiveness, adjustment and reliability, because each cue is easy to process and store. Experiences in Norwegian off shore oil and gas industry operations indicate that conventional accident and near miss analysis systems flood the employees with useless information, instead of supplying them with the specific operational knowledge that they require [7]. Consequently, employees learn only short-range corrections and show lack of motivation in using accident and near miss analysis systems in a proper way, so that the reliability and efficiency of the SMS decreases. Hence, one topic of the psychological safety research is the evaluation of explicit methods and databases within SMSs with regard to learning from experience. The goal is to increase information's quality related to safety within SMS and to implement concepts of collaboration during the process of building shared knowledge about reliable design and operation.

3. Safety culture Another research strategy to improve the reliability of complex technical installations is the analysis and measurement of safety culture. In order to analyse

Experience Feedback and Safety Culture as Contributors to System Safety

49

safety culture as a sub-type of organizational culture, and not as a quality assurance standard, the relationship between values and norms on the one side and patterns of behaviour on the other is investigated [8]. Norms as elements of a culture are postulated to be closely related to organizational performance and member attitudes and perceptions. In the area of nuclear industry, two kinds of norms are differentiated: Explicit norms and Implicit norms. Explicit norms form the explicit normative basis of all activities in nuclear power plants (NPPs) in the form of written procedures or manuals. Implicit norms are informal unwritten rules developed for the guidance of the actions within work groups. Implicit norms might differ from the codified, explicit ones or even contradict them, and they might not necessarily hamper the efficiency of the co-operative effort in nuclear installations

[9]. The scientific aim is to develop a theoretically founded and empirically validated psychological technique for the elicitation and measurement of implicit norms in NPPs. Based on the theory of planned behaviour [10] a multivariate questionnaire study was developed and carried out in an Eastern European nuclear power plant. First empirical results show, that normative influences are better predictors for procedural non-compliance than safety-related attitudes. A second research topic is the development and dissemination of safety-relevant implicit norms in an organizational setting. According to Tomasello et al. [ 11] three types of cultural learning are postulated to mediate the formation of implicit norms: imitative, instructed and collaborative learning. Especially imitative processes are considered to be the most effective way in transmitting cultural contents within work groups and shaping the safety culture of NPPs.

References [1]

[2] [3] [4] [5] I6]

Fahlbruch, B. & Wilpert, B., 1999, System Safety - An Emerging Field for I/(9 Psychology. In International Review of Industrial and Organizational Psychology I999, 14, 55-94. DNV, 1998, Managing marine accident investigation and analysis course. Det Norske Veritas Hamburg. INSAG - International Nuclear Safety Advisory Group, 1991. Safety Culture, Safety Series, Report 4 (INSAG-4). International Atomic Energy Agency (IAEA) Vienna. Rassmussen, J., 1991, Safety Control: Some Basic Distinctions and Research Issues in High Hazard Low Risk Operation. Paper presented on 11'b Workshop "New Techologies and Work", Bad Homburg (G), May 1991. Wilpert, B. & Fahlbruch, B., 1997, Event Analysis as Problem Solving Process. In A. Hale, B. Wilpert & M. Freitag (eds.), After the Event: From Accident to Organisational Learning. Pergamon-Elsevier Oxford (UK). Baggen, R., Wilpert, B., Fahlbruch, B. & Miller, R., 1997, Computer Supported Event Analysis in Indus;ries with High Hazard Potential. Paper presented on ESREL'97 International Conference on Safety and Reliability, Lisbon, June 1997.

50

[7]

Experience Feedback and Safety Culture as Contributors to System Safety

Aase, K. & Ringstad, A. J., 1999, Experience transfer and corrective measures. Paper presented on 17~ Workshop "New Technologies and Work", Bad Homburg (G), June 1999. [8] Schein, E. H., 1995, Unternehmenskultur. Campus Frankfurt. [9] Ignatov, M., 1999, Implicit social norms in reactor control rooms. In Misumi, J., Wilpert, B. & Miller, R. (eds.), Nuclear Safety: A Human Factors Perspective. Taylor & Francis London. [10] Ajzen, I., 1991, The theory of planned behavior. Special Issue: Theories of cognitive self-regulation. Organizational Behavior and Human Decision Processes, 50, 179-21 I. [11] Tomasello, M., Kruger, A.C. &Ratner, H.H., 1993, Cultural learning. Behavioral and Brain Sciences, 16, 495-552.

Operator Modelling and Analysis of Behavioural Data in Human Reliability Analysis

Oliver Str~iter i, 2 IGesellschaft ftlr Anlagen- und Reaktorsicherheit, Forschungsgel~inde, D-86748 Garching, [email protected] 2Lehrstuhl fur Ergonomie, Technische Universit~t Miinchen, Boltzmannstr. 15, D-85747 Garching, straeter @lfe.mw.tum.de

Abstract: The importance of Human Reliability Analysis (HRA) is currently

growing. We observe an increasing portion of the "human factor" in nearly every technological area that can only be faced by a well founded and validated HRA. From the HRA point of view, the relevant aspect for this is that Humans are not only faced to react on demands of safety-systems but they actively change the systemstates based on their judgement about the situation. The paper will give an outline of a solution to predict this active involvement of humans in technical systems.

1. Introduction Actual methods for evaluating human errors are mainly modelling the passive role of operators. These methods are considering PSFs (Performance Shaping Factors) like stress factors, completeness of procedures and training, and time available for performing a required task as parameters for assessment. Common known approaches are THERP, ASEP and HCR for instance. However, for assessing Human Reliability in situations with active involvement of humans in system safety, their application becomes difficult. Main problem areas are: 9 9

The data situation in HRA The modelling of the active role of the operator

The data situation in HRA is more or less not solved since the beginning of HRA. No method ever designed for assessing human reliability has ever perceived a complete and sufficient validation of it's database. THERP seems to be in a slight superior position since here at least some data were confirmed. However, such

52

Operator Modelling and Analysis of BehaviouraI Data in Hlvnan Reliability Analysis

insufficient situation (one has to keep in mind that nuclear power plants may get problems due to bad numbers in the safety-assessment) raises the question whether it is possible to quantify human behaviour at all. If we cannot perform good statistics about human reliability, should we then leave it at all? On the other hand, there exist the possibility to derive at least data that are accepted as inter-subjective agreement and the question above should therefore be denied. One solution is the use of systematic evaluation of behavioural data. For instance, a database system for plant evaluation was used for evaluating nuclear power plant events in Str~iter [3]. The investigation revealed confirmation and improvement of the THERP data as well as additional information about the active involvement of operators in scenarios.

2. The modelling of the active Role of the Operator To understand the active role of the human in technical systems, one should remember that we all are permanently collecting information and are comparing this with our internal representations about the world. This happens sometimes conscious but most of the times unconscious. This process can be described as a "Cognitive Mill" [2]. Figure 1 illustrates how this mill is generating human behaviour.

i }

f=

-= | PTocessingof %,

t

,)

_

itegirdinglnformirlol

~hlea.ng*=i, ' ~ I

Pmcess~Qof

~

-] Infotmltmn

r-~

reg/r
] u

I'

-

~s

I

~'Useestal~shed~a~o~ Lendcogr~processk'ig

Figure 1: Coherency between dissonance and possible cognitive behaviour: the "Cognitive Mill". As the cognitive mill implicates, cognitive performance has to be linked to the situational conditions an operator works in. Depending on slight deviations of these

Operator Modelling and Analysis of Behavioural Data in Human Reliability Analysis

53

situational conditions, he will use completely different cognitive strategies. This link is often ignored in investigating and predicting human behaviour. Rather more, HRA-methods attempt a prediction by assuming a rather stable set of information processing stages that is used by an operator with more or less success under given situations. Table 1 reveals a new approach that is based on the work of the EARTH- group [ I ] and the evaluation of events from German Nuclear Power Plants [3], [4]. This approach suggest to systematically combine certain situational conditions and certain cognitive aspects by considering certain coping-strategies an operator may choose. We called these coping-strategies "Cognitive Tendencies". They are independent from the usual concept of assuming a conditional reliability of certain processing stages (like perception, decision and action). Cognitive dissonance Situational complexity apparently simple

no dissonance (Operator does situation)

9 Omission (e.g., no action) 9 marking, labelling 9 fixation Example: Driving by car to the office as usual day by day task making us ignoring the surrounding we are driving through

9

obviously complex

not care about

quantitative commissions (e.g., too much/less) 9 precision and design of procedures 9 eagemess to act Example: Driving by car to the office and the usual way is blocked. We then usually take a known altemate way quickly.

dissonance (Operator cares about situation)

9

omission (e.g., no action) or commissions (e.g., wrong action) 9 reliability and equivocation of equipment 9 frequency oriented reasoning Example: Car-Light failure is assumed as a broken lamp, no as fuse-failure 9 delay (e.g., too late) 9 arrangement of equipment, reliability 9 reluctance to act Example: Car is broken and one has to decide between the two alternatives that have pros and cons (buying new one, repair of oLd one)

Table 1: Relation of Cognitive dissonance and Situational complexity. Principal error type (O), samples for typical PSFs -Performance Shaping Factor-(O) and cognitive behaviour ( 9

3. Discussion There exists a lot of unsolved problems we know about in cognitive science that have to be included in the above mentioned prospect. Cognitive Models should, for instance, explain attention, explain how cues trigger the decision/diagnosis process,

54

Operator Modelling and Analysis of Behavioural Data in Human Reliability Analysis

explain how emotions stress and other unspecified effects impact cognition and attitudes. In the future, these aspects have to be included in order to achieve an enhanced model for predicting the active involvement of operators.

References [1] [2] [3] [4]

Mosneron-Dupin, F., Reer, B., Heslinga, G., Str/iter, O., Gerdes, V., Saliou, G. & Uilwer, W. (1998) Human-Centered Modeling in Human Reliability Analysis. Reliability Engineering and System Safety. Elsevier. Volume 58, 3. Pages 249 - 273. Neisser, U. (1976) Cognition and Reality. W. H. Freeman. San Francisco. S tr~iter, O. (1997a) Beurteilung der menschlichen Zuverliissigkeit auf der Basis von Betriebserfahrung. GRS- 138. GRS. KOln/Germany. (ISBN 3-923875-95-9) Strater, O. (1997b) Investigations on the Influence of Situational Conditions on Human Reliability in Technical Systems. In: Sepp~il~i, P., et al. (Eds.) 13th Triennial Conference of the IEA. June 1997. Tampere/Finland. Vol. 3. p. 76ff.

Discussion Session I

As could be expected, the paper from Hollnagel about the implications of a "circular error model" where it is no longer possible to identify one "true" (or unambiguous) "cause" of an unwanted event created some controversy. However, the idea was generally accepted as a valid contribution to the further refinement of the understanding of accidents and their causes. With respect to the discussion about "situation awareness" an interesting remark was that the old notion of "overview" may have been rather vague and badly defined in many cases, but may also have implied the aspects of "assessment" and "prediction" as it had expressed been more clearly and explicitly in Endsley's work on "situation awareness". The observation that humans often contribute in a very important way to system safety by being able to "recover" the system from unwanted states was accepted with interest, but questions arose as to what was planned recovery and what sheer luck. It was also asked whether humans "recovered" the system from their own previous errors or from externally induced disturbances. Questions concerning the preconditions for good "recovery performance" also touched upon the problems of reduced options for action - which can be beneficial by precluding "erroneous" actions, but as well detrimental by ruling out "unconventional" actions necessary for recovery. It was also stated that successful recovery required a lot of knowledge about the process from the side of the operating personnel. A quite extensive discussion ensued after the subject of "reporting systems" had been mentioned. There are application areas that have a long standing tradition with this method - like e.g. medicine or nuclear power plants or insurance companies. On the other side there are areas where they are looked upon with much scepticism from a legal point of view. Finally the role of "explicit" and "implicit" norms was discussed. It was nearly generally agreed upon that explicit rules are "bypassed" or avoided if they do not comply with the functional requirements or necessities at hand. "Implicit" norms work in a more subtle (and therefore more pervasive?) way. As an interesting facet it was mentioned that "uniforms" that discriminate (in a positive sense) "professionals" from "lay persons" work rather differently in different countries and technical areas. There the influence of the "surrounding culture" on the working habits becomes very apparent. Summarized by Peter F. Eizer

A Project Overview Influence of Human-Machine-Interfaces on the ErrorProneness of Operator Interaction with Technical Systems

Peter F. Elzer, Badi Boussoffara Institute for Process and Production Control Technology (IPP) of the Technical University of Clausthal (TUC) Julius-Albert-Strasse 6 D-38678 Clausthal-Zellerfeld, Germany

1. Introduction This paper gives an overview over the research project MeMoVis (Mental Models and Visualization) that has been the starting point of this workshop. Its aim has been to experimentally investigate the influence of the Human Computer Interface (HCI) on the error-proneness of operators during classification and diagnosis of various process states. Since evaluation of the complete chain "activation detection - recognition - diagnosis - action" would have caused some ambiguities with respect to the interpretation of the experimental data only detection and classification of unwanted process states have been evaluated. It has been jointly conducted by four research institutions in Germany since October 1995 with the financial support of the Volkswagen Foundation Germany (Ref.nr. 1/69886). The project partners have been: Institute of Process and Production Control Technology (IPP) (project leader) of the Technical University of Clausthal (TUC) Prof. Dr.-Ing. P. F. Elzer Institute for Cognitive Research (IfKog) of the University of the Federal Armed Forces, Hamburg

Prof. Dr. R. H. Kluwe

Department of Applied Computer Science and Mathematics University of Applied Science, Fulda Prof. Dr.-Ing. T. Grams Institute for Safety Technology (ISTec) Garching

Dr.-Ing. P. Stolze

60

A Project Overview

2. Structure of the Project According to the respective skills and traditions of the participating institutions each of the partners has performed specialized tasks. These are described in short form in the following sections. The four papers following this project overview present some of the results of the individual partners in detail. IFKOG Taking into account results of cognitive psychological research the main task of IfKOG was the investigation of interfaces" attributes which determine the operator's understanding of a technical system and affect control performance. Some empirical studies were conducted, which were directed at the goal of identifying interface attributes which may be crucial for fault detection and diagnosis during supervisory control. For this purpose different versions of an interface for a conventional power plant simulator have been developed and modified in order to support a more integrated, multilevel form of information search and analysis by the operator. All interfaces provided the same information; the difference is given with the representation of the integration of process variables on the interface [5], [61, [7].

IPP The main task of IPP was the design and the completion of series of experiments with a large number of interfaces and subjects. The aim of the investigations was the search for a suitable combination of displays or visualization forms that support shortening detection time and reducing error rates during the detection and classification of abnormal process states. The experiments were especially designed for the investigation of the influence of time related and pattern oriented process visualization forms on the error proneness of operators during supervisory tasks. Taking into account measured performance of interfaces the results were discussed in a taxonomy context under the aspects of mapping and integration of information from the different kinds of displays in order to detect and classify abnormal process states correctly [1], [2].

ISTee The main task of ISTec within the project was the identification of scenarios on the nuclear power field that can be used as a realistic basis for experiments. For this purpose the collection of national and international operating experience from nuclear facilities was used. A total number of 3000 events were evaluated. Out of these 100 events have been reviewed, finally leading to 6 experimental scenarios for MMI evaluation. It was decided to use a fast running simulation model of a German pressure water reactor (PWR). To simulate the defined scenarios with the simulator of GRS the model had to be adapted in some aspects. ISTec prepared data streams to be used by the other participants of the project and carried out additional

A Project Overview

61

investigations on the GRS Analysis Simulator (ATLAS) [9].The graphical interface of the GRS Analysis Simulator has been modified for the experiments in cooperation with IPP.

Fulda The main task of Fulda was the development of a normative model of operator behaviour, called decision event tree (DET). Basically this normative model is a model of multistage decision making under risk. Operator errors are defined to be deviations from the norm [4]. By this the methods of technical risk assessment, economic decision theory and psychology can be utilized for an assessment of operator errors. The normative model of operator behaviour was first used during the review of taxonomies known from the literature on human factors, secondly as a basis for the design of psychological experiments, and finally as a classification scheme for operator behaviour in the course of experiments conducted by the project partners.

3. The experiments Depending on the requirements of the various types of experiments the following experimental environments were set up: IPP, IfKog: a simulator of a coal-fired power station (courtesy of ABB'); ISTec: a simulator of a nuclear power station (courtesy of GRS2); several types of interfaces designed by IPP and IfKog, implemented at IPP interfaces designed and implemented at ISTec. The following experiments were conducted: No. Of Interfaces Scenarios Replications Groups Subjects Measurements Emphasis of observations

IPP

IfKog

6 4 6 7 5 1 3* 1"* 6 10 6480 560 detection time and error rate for classification of abnormal process states

2

1"* 12

ISTec

1"*

1"**

12

Evaluation of control behaviour and control performance regar-ding fault diagnosis

1"* 10 480 as at IPP

*) Beginners, Advanced, Experts (Students and Engineers); **) Students; ***) Professionals Table 1: Experiments conducted by the project partners.

62

A Project Overview

4. Summary and Prospects One of the original assumptions of the project partners was confirmed, i.e. that in the particular area less experimental results exist than is generally assumed. Therefore, it appears appropriate to broaden the basis for this interdisciplinary research. It also appeared important that interaction between humans and machines can not be discussed in isolation as the relation "operator(s) - machine" [6]. The correctness of identification and assessment of states of a technical system and the (eventually) necessary operator actions also depend on the construction proper of the technical system and on the organisational structures ("management") surrounding it. Fig. 1 shall illustrate this.

Management

I|

J

Inadequate

Goalsan~c Instruction,,

Design and Engineering I

~

9,/Design flaws 1

'

Technical System

2

T

T I

HCI

/

Inadequate

Training

T Errors in Detection and Understanding

Operator _ . . . . Wrong Actions

Figure 1: Some possible sources of errors in Human-Machine-Systems. As far as the design of Human-Machine-Interfaces proper is concerned it turned out that a major source of problems is the fact that operators can not (or only with difficulties) integrate the information presented to them into a consistent view of the current state of the technical system at hand. This also holds for the development of (unwanted) system states over time. This aspect should therefore be investigated in the future in appropriate depth. This should also include more recent considerations concerning "situation assessment" and "situation awareness" [3].

A Project Overview

63

References I1]

[2]

[3] [4]

[5]

[6]

[7] [8]

[9]

Boussoffara, B.; Behnke, R.; Elzer, P. (1997), Systematic Evaluation of HCI's by means of User Experiments. In B. Borys, G.Johannsen, C. Wittenberg and G. Str~tz (eds.). Proceedings of the XVI. European Annual Conference on Human Decision Making and Manual Control. Kassel: University Kassel. Boussoffara, B.; Elzer P. (1998), About Human Pattern Matching and Time Information in S&C of large Technical Systems. In Proceedings of the XVII. European Annual Conference on Human Decision Making and Manual Control. Valenciennes: University Valenciennes, France. Endsley, M. R. (1995), Toward a theory of situation awareness. Human Factors, 37 (1), 32-64. Grams, T. (1998): Operator Errors and their Causes. In: Computer Safety, Reliability and Security. Proceedings of the 17th International Conference, SAFECOMP '98, Heidelberg, Germany. In W. Ehrenberger (ed) Lecture Notes in Computer Science. Berlin Heidelberg: Springer-Verlag Heinbokel, T., Leimann, E., Willumeit, H. & Kluwe, R.H. (1997)., A cognitive psychological framework for the description and evaluation of interfaces. In D. Harris (ed.) Engineering Psychology and Cognitive Ergonomics, Vol. 2 (pp. 437r444). Aldershot: Avebury. Heinbokel, T.; Kluwe, R.H.; Willumeit, H. (1997), Effects of interface attributes on fault detection and fault diagnosis. In B. Borys, G. Johannsen, C. Wittenberg and G. Str~itz (eds.). Proceedings of the XVI. European Annual Conference on Human Decision Making and Manual Control. Kassel: University Kassel. Heinbokel, T.; Kluwe, R.H.; Willumeit, H. (1998), Support of fault diagnoses during supervisory control by means of interface design. In D. Harris (ed.) Engineering Psychology and Cognitive Ergonomics. Heinbokel, T.; Leimann, E.; Kluwe, R.H. (1997), A cognitive psychological framework for the description and evaluation of interfaces. In D. Harris (ed.) Engineering Psychology and Cognitive Ergonomics Volume Two. Aldershot: Ashgate. Stolze, P. (1999), Konzeptsammlung und systematische Auswertung theoretischer und empirische Befunde zu StOrfiillen in Kraftwerken. Internal Report MeMoVis-TISTec09.

ASEA B R O W N BOVERI, Research Center, Heidelberg, Germany. 2Gesellschaft for Reaktorsicherung, Garching, Germany.

Attributes of the Interface Affect Fault Detection and Fault Diagnosis in Supervisory Control

Torsten Heinbokel, Rainer H. Kluwe Institut for Kognitionsforschung, Universit~it der Bundeswehr Hamburg, Holstenhofweg 85, D-22043 Hamburg, E-mail: [email protected]

Abstract: In this paper, a cognitive psychological framework of human-machine

interaction, the methodological approach and the results of two empirical studies, are reported. The research was directed at the goal of identifying interface attributes which are assumed to be crucial for fault detection and diagnosis during supervisory control. In the first study, it was assumed that due to attributes of standard P&Iinterfaces it would likely that operators fail when they are confronted with unfamiliar scenarios. The results of 11 single-cases revealed that in most cases the operators registered information which was sufficient for fault diagnoses; in the case of more difficult scenarios, however, only few operators were able to integrate this information adequately. On the basis of these results and taking into account cognitive psychological aspects, two different variants of the P&I-interfaces were developed and modified for the second study. Results of an experimental evaluation of these interface modifications show specific improvements of control performance and more integration of information. A comparison between the results of both interface modifications leads to the conclusion that interface attributes and control demands may interact. As a consequence, this will require more flexible support for the operator.

1. Introduction Control of complex systems like a chemical or a power plant raise high cognitive demands for the human operator. This is mainly due to attributes which are shared by most of the systems of these types, as e.g. high amounts of information, dynamics and uncertainty with regard to the time course of system states. Although automation contributes remarkably to the safety and efficiency of plant operations, there always occur faults that cannot be anticipated by the system designers. It is assumed that the diagnosis and management of these unpredictable events for which no built-in solutions are provided, constitute the highest demands experienced by the human operator [1] [2]. Therefore, it is important to provide for interfaces that support the development of an adequate mental model by the human

66

Attributes of the Interface Affect Fault Detection and Fault Diagnosis

operator and present the relevant information for the diagnosis and management of unpredictable events. The widely used conventional P&I-interfaces (P&I = Piping and Instrumentation) have been criticized for not being able to offer optimal support to the operator. P&Iinterfaces present topological and elementary physical information about system parameters like valve positions or feedwater flow by using a set of special symbols to display pipes, pumps, valves and other technical devices. A major critique refers to the lack of a complete overview over the process because information is presented in a serial manner [3]. Moreover, P&I-interfaces require the operator to determine and collect relevant data from a variety of individual instruments without integrating these data; this demanding task is left to the operator [4]. The goal of this research project is to identify crucial attributes of an interface, and develop improved interface versions which go beyond the limits of traditional single-sensor-single-indicator interfaces like the P&I-interface. It is argued, however, that interfaces cannot be specified top-down at the beginning of a developmental process, since system design is an evolutionary process [5]. Instead, it would be a reasonable research strategy, to identify interfaces and interface attributes that effectively support task performance and enhance efficiency of use first; then interfaces could be modified on this basis; finally, these interfaces would have to be evaluated empirically [6]. Following this line of research strategy, the goal of the first phase of this research project was to develop a coherent framework for human-machine interaction which would allow for a description of task and interface attributes. Additionally, the knowledge requirements of typical fault scenarios were determined within this framework. In the next phase, an empirical examination of conventional P&Iinterfaces was performed under specific conditions. On the basis of the empirically obtained results in this study, as well as with the background of cognitive psychological considerations which were deduced from the framework, modifications for two interface variants were developed systematically. In a last step, these interface versions were experimentally evaluated.

2. A Cognitive Psychological Framework for Human-Machine Interaction There are many models and taxonomies on human-machine systems which refer to both properties of the technical system: to either the user's internal representations or to the interaction between user and technical system [7]. In the following, a framework for human-machine interaction will be proposed that comprehends three factors which are conceived of as being relevant for the design of user-interfaces: (1) cognitive processes, (2) task demands, and (3) interface attributes. It allows to relate particular task demands and cognitive processes to interface attributes. This

Attributes of the Interface Affect Fault Detection and Fault Diagnosis

67

provides a basis for the selection and specification of appropriate interfaces. Furthermore, the framework is intended to guide the evaluation of interfaces with regard to criteria which are relevant for accomplishing a task and for the required user behaviour. 2.1 Description of cognitive processes in the control of h u m a n action The framework for human-machine interaction proposed here [8] combines two dimensions: (1) The first dimensions refers to different, sequentially ordered steps in the action process; (2) the second dimension refers to different modes of processing which are hierarchically ordered. A combination of both dimensions provides a scheme for the description of cognitive processes which are assumed to be important in supervisory control. steps in the action process goal formation

modes of proce~ing automatic processing ~:,;'~;~;'~

:~.~ :,.A" . : '

;,,i~ "~

orientation

preattentional perception of environmental stimuli: pattern formation

action specification

activation of movement oriented schemata and cognitive routines

execution and monitoring

execution of stored motor programs (physical execution) and cognitive routines (mental execution) processing of proprioceptive (e.g. kinaesthetic) and exterioceptive (e.g. visual) feedback

feedback processing

skilled processing activation of goals and subgoals: triggered by signals from the environment

controlled processing development of a goal hierarchy: components and dependency analysis interpretation, genesignal detection, observation and classification: ration of hypothesis and prognosis: complex categorization, recognition, chunking, simple judgements, analogical reasoning and use of judgements metaphors action planning and selection of an action selection: reasoning, use schema associated with of heuristics (e.g. signals meansend analysis), decision-making execution and monitoring execution and monitoring of plans and of action schemata strategies: encoding and' retrieval from memory, elaboration of plan detection, observation and interpretationof feedclassification of external back: analysis and synthesis of inforenvironmental feedback mation, evaluation (signals) against the set goals

Table 1 : Dimensions of the framework for human-machine-interaction. The cells in the matrix specify cognitive processes. Table I provides a description of cognitive processes that are assigned to specific cells of the resulting matrix. According to this scheme, there is no goal formation in automatic processing, since automatic processing is assumed to be under control of higher order goals and subgoals.

68

Attributes of the Interface Affect Fault Detection and Fault Diagnosis

2.2 Description of task demands and human behaviour In complex systems the operator usually acts as a kind of supervisor, "setting initial conditions for, intermittently adjusting, and receiving information from a computer that itself closes a control loop (i.e. interconnects) through external sensors, effectors, and the task environment" [9, p. 1244]. This includes different tasks as (a) planning, (b) monitoring, (c) intervenmg and (d) programming. steps in the task

action process

planning

goal formation

m o d e s of processing automatic Pr0cessing 9 ~.:~'~ .... ~,.w~/: ..... ~.~:~:;... ".

monitoring

orientation

skilled controlled processing processing "................1 . . . . . . . . . .:.:.i, .:..:... * specification of :.:...::. :i.:.,:.: wocess and work 9. ~:,~.":~=.': results 9.;,,~!.~r 9 specification of :~,~ ,.,.,.:..!:i.~,!i:::.:~:~standards and rated

9 detection of deviations 9 reading parameter values 9 alarm detection ~

identification of system states and events and detection of processes 9 identification 9 identification of of system states functional and and events causal re-lations selection and planning of input sequences for: 1. producing a transition between .... :,i, ........ i'."'.;~ system states ~.~....,, 2. execution of functions or " i, :~.~,(~ '. !:::: : procedures. 9,~;,<~:'": . ,.~,~ ~ 3. information acquisition (data ...... .~I:~:~:~: gathering) "~.~::. " ...... r 9 selection o f ' . * planning of ~:::::. , .. :"i " . , input sequences mput sequences 9 setting para- execution of input sequences meters 9 control and 9 data entry 9 execution of well-known input elaboration of input sequences sequences 9 information storage evaluation of system states and work results

~i:i~ii

action specification

intervening and

programming

execution and monitoring

feedback processing

T a b l e 2: A p p l i c a t i o n o f the f r a m e w o r k requirements.

to tasks. T h e c e l l s o f t h e m a t r i x s p e c i f y t a s k

Applying the framework from table 1 to tasks in system control results in a specification of task requirements as illustrated in table 2. It is described, for example, that monitoring requires the operator to detect deviations and alarms. On higher levels of cognitive processing it is required to identify system states, events and processes. This instantiation of the framework is intended to provide a basis for two steps in the research process: (a) for task analysis, i.e. how a task is performed

Attributes o f the Interface Affect Fault Detection and Fault Diagnosis

69

typically, and (b) for the description of an operator's behaviour, i.e. how an operator actually performs a task. m o d e s o f processing

Steps in the action process

automatic processing ~-.~,:.

goal formation

controlled processing

skilled processing

?~

. . . . . . .

9 ..:..::~:.:::.:?~ ~ ., .:. . . . . . . . . . . .. 9 representation / I.. , . .~ 9. :... :,:?~,:.t::::~: , . : : 9 ~.~.~.>;~.~~ ~:. ;,~% information content: .:7 ::" :'"~.'::".7:":". st'; ,,~ ""i":::v;~.~,~"~!~;~.:~e.., .,~,.~.~:! . :~. t :::~::~:,0.~. :::~'~i , ::~. " levels of abstraction 9

...........

9 -.-.

orientation

,,~

.:::-.

~

"

:

,::

. : !~..::,

;-.-----'..'.'-.::~.~:;~..' : ~ .

/.!.z~:::~. >'

9 physical properties 9 semantic properties of of stimuli: modality, information (signs as form, position, color signals for procedure) 9 amount of infor9 grouping of data marion 9 context information 9 pre~processing of data

action specification ;~:;: :~:::~:.::.::? :.~:...::.::i:!~

9

:i~; :,: :

execution and monitoring

..,~'~:%~zr

S:~..7

~..

9 physical properties of input devices: modality

feedback processing

35 <

]

signs as signals (cues) for procedure selection instead of generation goal related procedures 9 default values 9 memory support: reminders feedback related to procedure type of feedback: process vs. result

conceptual model of the system and overview representation symbolic properties (sign as symbol): analo-gies and metaphors representation of a time frame representation of relations hierarchization of information representation (content): levels of abstraction wizards, histories and logs decision support / expert systems memory support: external storage of information content of feedback, e.g. related to higher levels of abstraction type of feedback: process vs, result

Table 3: Relating attributes of the interface to the action x processing-framework for manmachine-interaction. The cells of the matrix specify interface attributes that are relevant for particular processing demands. 2.3 Description

of interface

attributes

The interaction b e t w e e n the operator and the technical system takes place via the interface. Information about the technical system is presented on a display. The operator controls the technical system with m e a n s of input controls as a keyboard or a touchscreen. In the following the action x processing-framework from table 1 is applied to the description of interface attributes. The matrix in table 3 refers to attributes of the interface which are assumed to be relevant when there are specific tasks (table 2) and specific d e m a n d s for cognitive processing (table 1). Table 3 specifies interface attributes that are assumed to be relevant for supporting particular cognitive d e m a n d s raised by specific tasks. If for example, the operator's

70

Attributes of the Interface Affect Fault Detection and Fault Diagnosis

task requires to detect deviations of the system state quickly, then specific attentional and perceptual demands are raised for the cognitive system; in addition, particular aspects of the presentation of information on the display gain importance in this context.

3. Cognitive Task Analyses of Fault Scenarios Cognitive task analysis designates various methods that incorporate facts about human cognition into the decomposition of tasks [10]. In this research project, selected fault scenarios were analyzed systematically in order to determine the task demands from a cognitive psychological point of view. The first step of the task analysis was directed at the identification and classification of process parameters with regard to their role in fault diagnosis. Parameters which were assumed to be relevant for the diagnosis were classified (e.g. as fault symptom or as a global parameter indicating the stability of the system). The second step referred to the definition and classification of knowledge which was assumed to be necessary for a correct diagnosis. According to the classification of process parameters, the contents of these knowledge classes differed between scenarios. The classification resulted in a comprehensive description of each of the selected fault scenarios in terms of process parameters and knowledge requirements. This type of task analysis provided an important basis for the analysis of the empirically obtained data about the behaviour and the knowledge of an operator. It allowed for a more precise determination of the quality of information that was used in the process of fault diagnosis: (a) the appropriateness and completeness of information provided by the interface and registered by the operator, (b) the availability and use of knowledge necessary to make a correct diagnosis.

4. Empirical Evaluation of P&l-lnterfaces (Study 1) The first study was directed at the evaluation of a standard P&I-interface under the condition of fault detection and diagnosis. The general assumption, derived of from earlier criticism, was that - due to the features of P&I-interfaces - it would be likely that operators fail to diagnose and understand fault scenarios which require the interpretation and integration of different sources of information.

Attributes of the Interface Affect Fault Detection and Fault Diagnosis

71

4.1 Method

Participants: 12 male subjects, all students of technical engineering with at least two years of theoretical background, participated voluntarily and were paid. Work place: Two 20" computer displays in the 'control-room', a keyboard and a mouse. A text-based alarm screen and trendgraphs were available in separate windows. For one group of operators (N=6) the interface was equipped with a mass-data display (MDD) as an additional tool [11]; the other group of 6 operators had no additional mass-data display available. Procedure: All operators participated in three sessions. In the first session they were introduced to the experiment with a demonstration of the simulator. Then, all operators received a manual and were instructed how to use a Computer-BasedTraining course on power plant technology which was developed for this project [12]. This training course provided the operators systematically with the relevant knowledge to control the power plant simulator. In the second session, system control was practiced extensively; the goal was to learn how to operate the simulator. Five scenarios with four of them being fault scenarios, were trained carefully in order to provide the operators with experience in fault diagnosis. In order to guarantee a sufficient level of expertise, a knowledge-test was performed at the end of the second session. The test referred to different types of knowledge as described by Kluwe [13]. As a result, one subject had to be excluded from the sample due to insufficient knowledge. Finally, in the third session subjects were instructed to monitor the plant, to detect faults in time, and to diagnose the faults. In this session the operators were confronted with eight selected scenarios bearing different control requirements. Scenarios were taken from a sample of typical faults when operating coal-fired power plants. All subjects were instructed to think aloud while operating the system. The verbalizations were tape- and videorecorded and transcribed later. In addition, a protocol file of the process parameters as well as the operators' control inputs was made available. Dependent variables: Attributes of the individual control behaviour and of individual control performance were assessed through verbal protocols and behavioural data. At first, all sources of data were combined in one comprehensive protocol describing the operator's behaviour and verbalizations, system messages and changes of process parameters. Secondly, these protocols were rated independently by two raters with regard to several categories of control behaviour (e.g. the integration of perceived information) and control performance (for example, the correctness of the final diagnosis). The interrater reliabilities ranged from moderate to high (median of kappa .78; range .52 to 1.00; N=43 protocols). Finally, the differences in the ratings were identified and analyzed by both raters until they reached consensus.

72

Attributes of the Interface Affect Fault Detection and Fault Diagnosis

4.2 Results Figure 1 illustrates the results for one criterion, the proportion of correct final diagnoses for the eight scenarios. As can be seen, performance differs considerably between the scenarios. Data from both groups, with and without an additional MDD, were analyzed together, as there were no specific hypotheses regarding performance or behaviour differences during fault diagnosis with the exception of fault detection. transition to 750 MW ~ ...........'~-'...... breakdown coal mill 4 feedwater pump breakdown

leak HPP 72 spindle breakage HPP 72 leak HPP 62 spindle breakage HPP 61 sensor failure coal mill 4 0%

25%

5O %

75 %

100 %

Figure 1: P & I Interface: Proportion correct final diagnoses for eight fault scenarios determined on the basis of thinking aloud protocols and behavioural data (N=I 1). The results of more detailed analyses of control performance and control behaviour for three scenarios are reported below: (l) sensor failure in a coal mill, (2) leak in a high-pressure preheater (HPP), and (3) spindle breakage in a HPP. These scenarios were selected on the basis of their specific control requirements. Scenario I was newly introduced to the operators. This scenario was expected to be the most difficult one due to its control requirements. The operator is here confronted with a kind of system behaviour that is normally implausible (i.e. automatic controllers lead to a destabilization of the process); furthermore, the cause of the fault is not visible on the interface. Finally, there are different alternative hypotheses to disprove. It was assumed that scenario 2 and 3 would be difficult, too, although both were experienced in the second session before. These scenarios could be easily confused with each other because they share similar symptoms at the beginning of the fault. Correct diagnoses of both faults requires knowledge about the topology and the functional relations between different components of the system.

Attributes of the Interface Affect Fault Detection and Fault Diagnosis

73

scenario 1: sensor coal mill

failure I 2: leak HPP 62

3:spindle HPP 61

breakage

indicators of control performance Sufficiency

of m m m m m m m m ~ .

mmmmmmmmmm.

mmmmummR 9

registered infor- 73 % (N=8)

9 1 % (N=I 1)

% (N=7)

marion Correct complete

mmmnm~))m 9

mmnn~)mm~B)

36 % (N=4)

18 % (N=2)

18 % (N=2)

9 % (N=I)

cription symptoms,

and m ~ . ~ . ~ . ~ ) ) des- 9 % (N=I)

of

system state and Ixhaviour Correct final diagnosis 0 % (N~) Correct explanation of final 0 % (N=0) diagnosis

m~))nngamn~ a~mmM~lmMlU

0 % (N~)

0 % (N~)

m m m m u ~ u ~

mmmm~mmsm

mmmim~wlt~w~

36 % ( N ~ )

36 % (N=4)

36 % (N=4)

control behaviour Integrated multi-level

I anal~'sis Table 4: Indicators of control performance for three selected scenarios (percentage of subjects; 9 indicates one subject). Table 4 provides the results for several performance variables, and for the control behaviour for the selected scenarios. For scenario 1 none of the operators reported a correct final diagnosis; note, however, that 73% of the operators noticed the information which was assumed to be sufficient for a correct diagnosis, and which was, moreover, actually available on the display. For this scenario only one operator offered a correct and complete description of the fault symptoms, the system behaviour and the system state. Both scenarios 2 and 3 reveal a similar picture as scenario 1. Only 1 or 2 operators proposed correct diagnoses, none of the operators gave a correct explanation of the final diagnosis. Also, the descriptions for these scenarios of the fault symptoms, the system state as well as the behaviour were unprecise or wrong in most of the cases. Again, as for scenario 1, the data reveal that most of the operators noticed the information which was sufficient for a correct diagnosis and also presented on the display. In the case of these scenarios unintegrated multi-level or focussed analyses dominated, i.e. operators did not identify and determine relations between processes in different subcomponents nor

74

Attributes of the Interface Affect Fault Detection and Fault Diagnosis

did they consider the system state or the system behaviour in analyzing what happened in the system. From the results of study 1 it is concluded that operators are more likely to fail in generating correct fault diagnoses when there are more difficult fault scenarios presented by the P&I interface. The reason for unsuccessful fault diagnoses under these conditions is assumed to arise from erroneous interpretation of processes as well as from the lack of integration of information from different sources. It is assumed that the P&I interface does not support these task demands appropriately.

5. Development and Evaluation of modified Interfaces (Study 2) The second study was based on the assumption derived of from study 1, that P&Iinterfaces provide necessary information, but do not support the integration of information for fault diagnosis. An empirical study was designed in order to examine this assumption more closely. The overall goal was to support the integration of information by means of interface modifications. Two variants of the conventional P&I-interface were developed, both taking into account cognitive psychological considerations with respect to the mapping of interface attributes and control demands. In the standard P&I-interface basically the same information is available; the difference, however, is given in form of the representation of the integration of process variables on the interface. The first interface version provided an additional presentation integrating critical relations between mass input, energy input and energy output (E&M-display = energy and mass flow; see figure 2a). The second interface version provided information about functional relations between process parameters combined with the topological representation of the P&Iinterface (F&T-interface = functions and topology; see figure 2b for an example). These interface variants were selected from a sample of various prototypes, since it was assumed that these modifications of the P&I interface would engender a more integrated, multilevel form of information search and analysis by the operator than the standard P&I-interface. Correct fault diagnosis should be more likely under this condition. The P&I-interface variant with the additional E&M-display provides information allowing for inferences about the attainment of the whole system's purpose (i.e. whether energy input and feedwater are in a balance which is indicated by the enthalpy of the steam). According to the abstraction hierarchy [14] [15] this kind of information refers to the most abstract level of the functional purpose of a system. This is assumed to be especially important in the sensor failure scenario. The F&T-interface variant, on the other hand, provides for information which refers to the physical function of the system's components (e.g. increasing temperature in the HPP). This was assumed to support performance in the leak and spindle breakage scenarios which affect primarily the functioning of a specific subsystem.

Attributes of the Interface Affect Fault Detection and Fault Diagnosis

75

main feedwater circle feedwater pump HPP 561 kg/s -~1 561 k g / . [

I

/ (a) F_~M interface

~","~

" i

I- l I

I I

(b) F&T interface

Figure 2: Modifications of the P&I interface; (a) Integrated presentation of energy and mass flow in the E&M-display; (b) presentation of physical functions for a subcomponent. These modifications were added to the P&I-interface.

5.1 Method The empirical design was similar to study 1. A new sample of 12 male operators, all students of technical engineering with at least two years theoretical background, participated voluntarily and were paid. The same hardware equipment was used, but in order to control the use of information, no trendgraphs were available in this study. The theoretical and practical training procedure was similar as in study 1. The interface modifications were only shortly introduced, and there was no specific training of scenarios with these interface variants. One half of the subjects (N=6) began with the P&I-interface with an additional E&M-display while the second half started with the F&T interface. In the second part of the performance session, each group was equipped with the alternative interface. Four fault scenarios were provided under each interface condition. Measures of control performance and behaviour were assessed by two independent raters immediately after the experiment.

5.2 Results In the following, the results for only three scenarios are reported in more detail. These are basically equal to those reported above from study 1. However, the scenarios used in this second study were made more complicated compared to those applied in study 1 by introducing transients before the occurrence of a fault. The difficulty of system control is increased, since it is necessary to monitor the system during a transient between system states. As argued above, it was assumed that in scenario 1 the operators would gain most from the integrated E&M-representation added to the P&I-interface, and in scenarios 2 and 3 from the integrated functional representations provided by the F&T-interface.

76

Attributes of the Interface Affect Fault Detection and Fault Diagnosis

E&M interface 1: sensor failure coal indicators of mill 4 control performance SufF~i~my ~ m m m m m m m m m m m ~ n~giste~ ~ 91%(N=I1) forrnati~ Correct and m m ~ - ~ corr~e~ des- 17 % (N=2) crilXion of symtg~, syaem aate and behavior Cxrrcct

diagnosis

final m m m m m 9

50 % (N=6)

F&T interface 2: leak HPP 72

3: spindle HPP 61

breakage

mmmmmmmmmm)~

m m m m m m ~

83%(N=10)

50%(N=6)

m m m m m . ~

m m m m ~ ~

42 % (N=5)

33 % (N--4)

miimmimmi))~

75 % (N=9)

mmim)~

~~)~))~

33 % (N=4)

Cx.alx:t e.xpla- m m m m ~ . ~ w ~ . ~ ,

mmmmmmw~B~

m m u ~ . ~ . ~

nalion of final 33 % (N=4) diagnosis control behaviour Integrated mmnmmmmmm)~ multi-level 75 % (N=9) analysis

50 % (N=6)

25 % (N=3)

m m m m m m ~

mmmmmm~

50 % (N=6)

50 % (N=6)

t~:J~

Table 5: Indicators of control performance in three scenarios (percentage of subjects, N=12; 9 indicates one subject) Table 5 provides the results with regard to several performance variables and control behaviour when being confronted with the selected scenarios. Overall, the results are in agreement with the expectations. Control performance is improved under the condition of additional integrated representation of information. In scenario 1, most operators noticed sufficient information. The fault descriptions, however, were usually inaccurate (66,7%). They were only correct in two cases (17%). Half of the operators provided a correct final diagnosis, one third of them also a correct explanation of the diagnosis. Compared with the standard P&Iinterface used in study 1 (see table 4), twice as many operators performed an integrated multi-level analysis of the fault as well as its consequences. There is a similar pattern of results in scenario 2. 75 percent of the operators gave a correct diagnosis and 50 percent also a correct explanation, while the respective figures in study 1 consisted of 18 percent correct diagnoses and 0 percent correct explanations. In scenario 3 the effect of the interface is not as clear as in both of the other scenarios, although there is a tendency of improved control performance. One third of the operators provided a correct diagnosis compared with only 9 percent in study I.

Attributes of the Interface Affect Fault Detection and Fault Diagnosis

77

Overall, operators reveal improved control performance, and a more adequate control behaviour when being supported by integrated information representations. When comparing both interface modifications, which is not shown in the table, it becomes evident that the integrated E&M-representation provides better support for the requirements asked for in scenario 1, while the integrated F&T-representation especially supports control performance for scenario 2. This result supports the hypothesis that there is an interaction between task demands and attributes of the interface.

6. Discussion

The goal of this research project was to show that it is possible to support fault detection and fault diagnosis by modifying specific attributes of an interface, taking into account task demands and processing demands given with specific scenarios at the same time. The first study revealed serious shortcomings of the standard P&Iinterface with regard to the necessity to integrate available information for diagnostic judgements. An integrated multi-level analysis was rarely observed under this condition; in addition, operators are likely to fail in difficult scenarios. It is assumed that the information which is actually provided by the interface and which is sufficient to generate a diagnosis is, though noticed, only partially used by the operators. In the second study, two modifications of the P&I interface were developed in order to support the process of fault diagnosis more adequately. Both interface versions were assumed to support the integration of information; they differed with regard to the specific type of functional information with the E&M-display presenting integrated information about the system's functional purpose on the one hand and the F&T-interface presenting information about the physical function of the system's components on the other hand. The results of this study demonstrated that improved control performance and more appropriate control behaviour can be reached whenever integrated information representations are available. However, each interface version was only superior for one class of scenarios. In general, the results of both studies show the shortcomings as well as the facilitatory effect of specific attributes of the interface on control performance and control behaviour.

References Vicente, K. & Rasmussen, J. (1992). Ecological interface design: Theoretical foundations. IEEE Transactions on Systems, Man and Cybernetics, 25, 589-606. [2] Vicente, K. (1995). Ecological interface design. In T. B. Sheridan (ed.), International Federation of Automatic Control Symposium: Analysis, Design and Evaluation of Man-Machine Systems (pp. 623-628). Cambridge: Pergamon Press.

[1]

78

Attributes of the Interface Affect Fault Detection and Fault Diagnosis

[3]

Elzer, P., Weisang, C. & Zinser, K. (1989). Knowledge-based System support for Operator Tasks in S&C Environments. Proceedings of the IEEE Conference on Systems, Man and Cybernetics (Vol. 3, pp. 1078-1083). Boston, MA. Pawlak, W.S. & Vicente, K.J. (1995). Inducing effective operator control through ecological interface design. International Journal of Human-Computer Studies, 44, 653-688. Carroll, J.M. & Rosson, M. B. (1985). Usability specifications as a tool in iterative development. In H. R. Hartson (ed.), Advances in human-computer interaction (pp. 128). Norwood: Ablex. Andriole, S. & Adelman, L. (eds.) (1995). Cognitive system engineering for usercomputer interface design, prototyping, and evaluation. Hillsdale, N.J.: Erlbaum. Whitefield, A. (1990). Human-computer interaction models and their roles in the design of interactive systems. In P. Falzon (ed.), Cognitive ergonomics. Understanding, learning and designing human-computer interaction (pp. 7-25). London: Academic Press. Heinbokel, T., Leimann, E., Willumeit, H. & Kluwe, R.H. (1997). A cognitive psychological framework for the description and evaluation of interfaces. In D. Harris (ed.) Engineering Psychology and Cognitive Ergonomics, Vol. 2 (pp. 437-444). Aldershot: Avebury. Sheridan, T. (1987). Supervisory control. In G. Salvendy (ed.), Handbook of human factors (pp. 1243-1268). New York: Wiley. Grant, S. & Mayes, T. (1991). Cognitive task analysis? In G. R. S. Weir & J. L. Alty (eds.), Human-computer interaction and complex systems (pp. 147 - 167). London: Academic Press. Beuthel, C., Boussoffara, B., Elzer, P., Zinser, K. & Til?,en, A. (1995). Advantages of mass-data displays in process S&C. Preprints of the 6th IFAC / IFIP / IFORS / IEA Symposium on Analysis, Design and Evaluation of Man-Machine Systems (pp. 439444) Cambridge, MA. Willumeit, H. & Heinbokel, T. (1997). Einfiihrung in die Grundlagen der Kraftwerkstechnik [Introduction to power plant engineering]. Hamburg: University of the Federal Armed Forces. Kluwe, R.H. (1997). Acquisition of knowledge in the control of a simulated technical system. Le Travail humain, 60, 61-85. Rasmussen, J. (1983). Skills, rules, and knowledge: signals, signs and symbols, and other distinctions in human performance models. IEEE Transactions on systems, Man and Cybernetics, SMC-13 (3), 257-266. Rasmussen, J. (1986). information processing and human-machine interaction. Amsterdam: North-Holland.

[4]

[5]

[6] [7]

[8]

[9] [10] [1 l]

[12]

[ 13] [14]

[15]

Acknowledgement: Research reported here was supported by a grant from VolkswagenFoundation to the second author.

Evaluation of Interfaces by Means of Experiments: what's behind Taxonomy?

Badi Boussoffara, Peter F. Elzer Institute for Process and Production Control Technology (IPP) Technical University of Clausthal (TUC) Julius-Albert-Str. 6, D-38678 Clausthal-Zeilerfeld, Germany badi @ipp.tu-clausthal.de elzer @ipp.tu-clausthal.de

Abstract: In the context of a research project with the aim of investigating the

influence of human machine interfaces on the error-proneness of situation assessment in technical systems by humans extensive user experiments have been conducted. This paper will report influences of different visualization forms on the classification of process states. The assumed and measured performance will be compared with each other. The results will be discussed as well in a taxonomy context.

1. Introduction The aim of the investigations was the search for a suitable combination of displays or visualization forms that support shortening detection time and reducing error rates during the detection and classification of abnormal process states. The experiments were especially designed for the investigation of the influence of time related and pattern oriented process visualization forms on the error proneness of operators during supervisory tasks (see also [1], [2]). Taking into account measured performance of interfaces the results were discussed in a taxonomy context under the aspects of mapping and integration of information from the different kinds of displays in order to detect and classify abnormal process states correctly. All interfaces that have been investigated were based on P&I (piping and instrumentation) diagrams [3] which are commonly used in S&C of large technical systems.

80

Evaluation of Interfaces by Means of Experiments: what's behind Taxonomy?

The experimental set-up (Fig. 1) consisted of: 9 9 9

a simulator of a coal-fired power station [6] that has been modified in order to provide finer control of critical situations. 2 VDUs and a large screen display. an instructor's workplace.

Figure 1: Experimental set-up. In regular time intervals 4 interfaces were presented to each subject: 9 9 9 9

P&I P&I P&I P&I

diagrams diagrams and M D D diagrams and trend graphs diagrams, MDD and trend graphs.

In each session seven different process states were presented to the subjects. Some of the scenarios were either unknown to the subjects or similar to situations they already knew. To keep the rate of training transfer as low as possible, both the interfaces and the scenarios were presented to the subjects in randomized order. The time for detection of abnormal process states and the number of erroneous classifications were measured. After each session the subjects were asked about the reasons for their classification. They were instructed to think aloud during the experiments as well.

Evaluation of Interfaces by Means of Experiments: what's behind Taxonomy ?

81

As mentioned above the investigated interfaces were based on P&I (Piping and Instrumentation) diagrams. According to the experiment design the subjects were provided with the following additional aids: 9 A Mass Data Display (MDD) [4] which had been designed to particularly support the human ability to quickly recognize and assess changes in graphical patterns. This technique always gives the subject an overview over the entire process. The MDD visualizes the behaviour of the process as a graphical pattern. 9 Trend graphs that visualize time related information about several process variables. During the experiments subjects could freely choose and configure up to eight trend graphs for process variables. During a session subjects could also delete or change the configuration of the trend graphs they had chosen. In this way information about context dependent changes could be acquired.

2. Classification of the Displays and the assumed Performance Looking at the individual displays under the following aspects: 9 effort to be made to access data and information 9 contribution to diagnosis of process states the classification scheme in Fig. 2 has been developed.

0

~3

,.O

O

little ] fair [ much effort to access data and info

Figure 2: A classification scheme for the displays.

82

Evaluation of Interfaces by Means of Experiments: what's behind Taxonomy?

Based on the features of the displays mentioned above it can be assumed that: 9 Using P&I diagrams detection time as well as error rates would be found in the centre of Fig. 3. 9 Using the MDD information about process states in the form of pattern changes is always available. Since the MDD is not bound to any treshold subjects can be alerted at an early stage. Therefore, it can be assumed that the MDD supports subjects to "buy" time during classification and diagnosis of abnormal process states (cf. Fig. 3). 9 Once subjects have selected suitable trend graphs allowing description and diagnosis of an abnormal process state it can be assumed that using trend graphs an exact diagnosis could be made. Therefore, it can also be assumed that error rates using trend graphs would be lower than using P&I diagrams only (cf. Fig.

3).

g

~

OTre~Grlph$

_o short

middle

long

time to identify process states Figure 3: Assumed performance of the interfaces.

3. Observations concerning Introduction of the MDD and Trend Graphs Using the MDD and P&I diagrams it was observed that subjects primarily looked at the MDD for pattern changes. As soon as the process deviated from its normal state the subjects formulated global hypotheses about the disturbed subsystem. Some examples of such hypotheses were: 9 9

"Something is going wrong in the high pressure preheater; It may be a leakage!" "Something is going wrong in coal supply; It may be a coal mill breakdown!"

Evaluation of Interfaces by Means of Experiments: what's behind Taxonomy?

83

It was intended to alert subjects at an early stage and show them that something was not working in the usual way. Thereby, the MDD shall call the attention of the subjects to the subsystems being disturbed. More detailed information about the actual process states must then be acquired in a different way, e.g. by looking at the P&I diagrams. Unfortunately, some subjects tried to formulate more precise hypotheses in a rush. Therefore, the P&I diagram of the corresponding subsystem containing more detailed information about the process variables was only observed to quickly confirm or to reject these hypotheses. However, most of the subjects "stuck" to their first global hypothesis. In the worst case P&I-diagrams were only looked up for information to confirm this apparently fight hypothesis. Thus, discrepant information was ignored. This effect could be explained by a kind of cognitive hysteresis [5]. An explanation of this effect could be that under the given circumstances - observing the behaviour of the MDD and trying to formulate an exact hypothesis about the actual process states - the MDD appears to cause "undesired" pattern matching which leads to misinterpretations. An unknown pattern would be reduced or complemented to a pattern that subjects already know. This effect was particularly observed during detection of similar or unknown process states. These above-mentioned effects -using the MDD for classification and diagnosis and the "fixation" on the "apparently right" hypothesis- generally led to higher error rates than using the other interfaces considered in this experiment. In case of correct classification of abnormal process states detection time was generally shorter. The results will be discussed in the next chapter. Using P&I diagrams and trend graphs it was also observed that subjects primarily looked at the displayed trend graphs for deviations from the normal state. As trend graphs show the exact behaviour of single process variables the subjects formulated more precise hypotheses at a more concrete level. Some examples of such hypotheses were: 9

9

"The level in the high pressure preheater begins to increase, the gain of the control valve also. It could be a leakage ...... Yes it is, what is about emergency valve? Does it open? Yes it does." "The coal supply of one mill breaks down. The other 3 mills compensate the resulting deficiency. What about fresh air supply? It is constant? Yes, coal mill breakdown."

As trend graphs have to be interpreted at first and secondly a relationship between observed process variables has to be interpreted, subjects have to take a fair amount of time to "think" about the actual process states. Furthermore, trend graphs appear to allow prediction of future states to a certain extent. This appears to make interpretation of relationships between the observed process variables easier. Since subjects using trend graphs took the amount of time necessary to diagnose abnormal process states error rates were generally lower than using the other interfaces. On

84

Evaluation of lnterfaces by Means of Experiments: what's behind Taxonomy?

the other hand detection of abnormal process states took longer. This result will be also discussed in the next chapter.

4. Discussion of the Results Although it appeared that subjects using the MDD tended to draw more hasty conclusions, a combination of P&I diagrams and the MDD performed better compared to P&I diagrams alone. At least detection time using the interface with the MDD was shorter (p<0.09). An explanation for shorter detection times could be that the MDD supports alerting subjects at an earlier stage. Thus, the MDD helps to buy time while subjects locate abnormal process states. When subjects only used P&I diagrams they either received a warning from the event list or they detected abnormalities casually in P&I diagrams. However, there is no significant difference between the error rates registered.

C PI m MDD

~ TrendGraphs r P.

"i

_o

short

middle

long

time to identify process states Figure 4: Assumed vs. measured performance. Comparing the interface with trend graphs to P&I diagrams subjects made significantly (p<0.07) less errors. This was expected because of the fact that subjects had to take a fair amount of time to classify process states. A significant difference between detection times could not be found. Although the existing differences in performance were not significant enough (p>0.05) it can be assumed that using P&I diagrams with the abovementioned aids (e.g. trend graphs and MDD) better performance could be achieved compared to

Evaluation of Interfaces by Means of Experiments: what's behind Taxonomy ?

85

using P&I diagrams only. The MDD supports the reduction of detection time of abnormal process states. Trend graphs (eg. time related information) on the other hand support reduction of classification errors. In direct comparison of P&I diagrams and MDD with P&I diagrams and trend graphs the above-mentioned effects can be confirmed. Comparing the interfaces with each other -removing trendgraphs and adding the MDD- subjects detected abnormal process states significantly (p<0.05) faster. On the other hand the error rates increased significantly (p<0.05). Subjects made more errors than when using trend graphs. The main effect is that subjects appear to use the MDD as a means for classification and diagnosis. Finally, the following two combinations of aids must be also discussed: Adding trend graphs to P&I diagrams and the MDD: by introducing time related information (eg. trend graphs) detection time basically did not increase (p<0.1) but error rates were reduced significantly (p<0.04). Adding the MDD to P&I diagrams and trend graphs: by introducing the MDD detection time decreased significantly (p<0.04) but error rates increased significantly (p<0.06). As soon as subjects tried to use the MDD as a means for diagnosis the error rates increased. The effect of drawing hasty conclusions appears to be reconfirmed Concluding the discussions in this chapter all display combinations led more or less to the assumed results with little deviations. The only exception for deviations from the assumed performances was the introduction of the MDD. Subjects using the MDD drew hasty conclusions and therefore, the error rates using the MDD were high. Looking at this effect from another viewpoint and taking into account a taxonomy describing the characteristics of the displays investigated, the effect of drawing hasty conclusions using the MDD might also be explained. For this purpose a taxonomy was developed. Looking at the characteristics of the displays from the viewpoint of how processes are visualized. Related to the characteristics of each display the taxonomy has the following three dimensions: 9 qualitative/quantitative 9 abstract/concrete 9 details/whole These dimensions form a 3D-space shown in Fig. 5.

86

Evaluation of Interfaces by Means of Experiments: what's behind Taxonomy?

r~

Figure 5: A taxonomy of investigated displays. The MDD is an abstract visualization form describing the process behaviour qualitatively and giving an overview over the whole process. Therefore, the MDD is placed at the left upper corner of the cube shown in Fig. 5. Trend graphs are a very quantitative and detailed visualization form and they reflect the behaviour of single process variables. Therefore they can be located in the opposite corner of this cube. With respect to their characteristics P&I diagrams can be found somewhere between the MDD and trend graphs. Since P&I diagrams contain more concrete, quantitative and detailed information than the MDD, they can be placed more in the vicinity of trend graphs. Looking at the taxonomy a gap between the MDD and the other displays can be detected. It appears that, since this gap is very large, subjects are not able to map information from the MDD to other displays like P&I diagrams and trend graphs directly. Particularly during the diagnosis of unfamiliar fault scenarios it was observed often that the subjects were not able to integrate the information appropriately.

Evaluation of Interfaces by Means of Experiments: what's behind Taxonomy ?

87

5. Conclusions Research reported in this paper the influence of pattern matching oriented displays (like the Mass Data Display) and time related information (like trend graphs) on the performance regarding detection of abnormal process states. Generally, it was found that pattern matching displays reduce detection time with the disadvantage that subjects may draw hasty conclusions. Furthermore, it was observed that subjects used the MDD as a means for classification and diagnosis. This led to unsatisfactory results, because it was used outside the domain it had been designed for. Introducing time related information appeared to reduce error rates. It could be concluded that in order to achieve short detection times as well as low error rates new or re-engineered display types can be designed. Therefore, some new interfaces have been and have to be developed systematically. They would have to take into account cognitive considerations with respect to the process of matching interface attributes with control demands. Such displays may be based on experimental results and take into account a suitable taxonomy. They may provide easier access to information for classification of abnormal process states.

References [1]

[2]

[3] [4] [5]

[6]

Bousoffara, B.; Behnke, R.; Elzer, P., 1997, Systematic Evaluation of HCl's by means of User Experiments. B. Borys, G.Johannsen, C. Wittenberg and G. Stratz (eds.). Proceedings of the XVI. European Annual Conference on Human Decision Making and Manual Control. Kassel: University Kassel. Boussoffara, B.; Elzer P., 1998, About Human Pattern Matching and Time Information in S&C of large Technical Systems. In Proceedings of the XVII. European Annual Conference on Human Decision Making and Manual Control. Valenciennes: University Valenciennes, France. DIN 28004: Deutsches Institut fur Normung. Beuth Verlag GmbH, Berlin, KOln. Elzer, P., Beuthel, C., Boussoffara, B., Zinser, K., Tissen, A., 1995, Advantages of mass-data-displays in process S&C. 6th IFAC/ IFIP/IFORS/IEA Symposium on Analysis, Design an Evaluation of Man-Maschine Systems, Boston, MA, pp. 439-444. Norman, D.A., 1986, New Views of Information Processing: Implications for Intelligent Decision Support Systems, Intelligent Decision Support in Process Environments. E. Hollnagel (Ed.), NATO ASI Series Vol. F21. Berlin, Heidelberg: Springer Verlag, pp. 123-136. Weisang, C., Elzer P., 1988, Die Bedeutung eines Prozeflsimulators in der Prototypphase eines wissensbasierten S&C-Systems. Fachtagung "Proze6rechnersysteme "88", Stuttgart.

Acknowledgement: Research reported here was supported by the Volkswagen-Foundation in Germany (Ref. Nr.:I/69 886).

H u m a n Performance and Interface-Design Some Remarks based on Experiments

Peter Stolze ~, Oliver Striiter 2. ' Institut fiir Sicherheitstechnologie, Forschungsgeliinde, D-86748 Garching, [email protected] 2 Gesellschaft fur Anlagen- und Reaktorsicherheit, FOrschungsgel~inde, D-86748 Garching, [email protected] Lehrstuhl fiir Ergonom~e, Technische Universitiit Mtinchen, Boltzmannstr. 15, D-85747 Garching, straeter @lfe.mw.tum.de 3

..

9

Abstract: In this paper we argue that human performance and interface-design cannot be treated as independent dimensions of ergonomic design. If one has identified an inappropriate human ability (e.g., human have difficulties with delayed feedback), this must always be connected to the interface, that was used for confirming this result. Experiments we performed clearly support this hypothesis, showing that human failures and inappropriate human behaviour cannot be separated from the layout of the interface. Even more it looks like experimental results regarding human performance are always a self-prophesying artificial result of the experimental layout rather than self-standing human properties. Consequently, human and interface have to be treated and designed as an overall system with mutual influence. For this, the understanding about the influence of external aspects on human properties has to be investigated further.

1. Introduction In interface design it is often argued that the interface has to be more appropriate to the needs of the potential user. What does this mean? The usual understanding to achieve such a goal is to find certain human properties that makes human fail. The consequence of such thinking is problematic. W e say, for instance, humans have difficulties with time-dynamics. From this experience we draw the conclusion to build an automated system that makes time-dynamics a static one task for the operator (e.g. a button for shutting down the plant). Such a measure rather pushes the operator out of the loop than supporting his abilities. What is to be asked instead is: Is the observed disability really a true one? If we think about living in time and being able to perform complex time-dynamic task like driving a car without any major difficulties, we see that this statement is clearly relative to an implicit

90

Human Performance and Interface-Design - Some Remarks based on Experiments

assumption about the environment, the humans operate in. In this case the assumption is that the philosophy of the control room layout is rather to provide static possibilities for observation and action than dynamic ones. Within a three years project for the VW-Stiftung, ISTec investigated critical characteristics of MMI's that are important for the diagnostic performance of an operator and for his interactions with a technical system. The initial goal of the project was to identify certain human properties that may lead to problems in process control. However, such a simple relationship cannot be supported by the results of this investigation as this paper will outline. We cannot define certain human properties to be supported without linking them to some external (not human related properties) like at least the properties of the tasks to be performed, the properties of the failure modes that should be detected and corrected with the help of the interface, >" the properties of the situational conditions under which the task is to be performed, and finally the properties how the interface considers all these circumstances. In our opinion, any statement attempting to find a human properties that allow a design-suggestion independent from these aspects has to fail. This understanding is not new in ergonomics but we observed that many investigations and ergonomic measures either explicit or implicit ignore these dependencies of human performance. We will first describe the experiment performed and then we will justify this conclusion.

2. Preparatory Tasks for the Experiments In order to achieve realistic scenarios for the experiment, operating experience from nuclear facilities was evaluated from the GRS-database (called BVOR/VERA) according to human abilities and problems (see [3] for the procedure of identification and classification of human error events). The database contains more than 4000 events (about 3000 events on PWR - Pressurized Water Reactors). As a basis for the experimental cases, a PWR was selected and consequently we only considered operational events in PWR. 100 of these events were selected and used to find a detailed classification of problem-areas for human interventions. The following classification scheme for distinguishing the complexity of scenarios resulted from this event analysis: >" Class 1: Malfunction or failure of one component >" Class 2: Malfunction of one component due to faulty activation Class 3: Malfunction of a control or limiting system due to failure of a data acquisition unit

Human Performance and Interface-Design - Some Remarks based on Experiments

91

Class 4: Malfunction of one component combined with or happening after the malfunction of another component According to this classification, we defined 6 scenarios with different degree of difficulty: Simple cases (class 1) to complex cases (class 4). The definition of the 6 Scenarios that also represents the event classification is as follows, see Stolze [2] for details about the scenarios:

Simple scenarios Scenario 1: Stuck open failure of one pressurizer spray valve (class 1) Scenario 2: Unexpected opening of a pressurizer relief valve due to faulty activation (class 2) Scenario 3: Failure of a feedwater control unit due to failure of a data acquisition unit (class 3)

Complex scenarios >" Scenario 4: Failure of a main coolant pump and malfunction of a feedwater valve (class 4) ~" Scenario 5: SCRAM due to manual maloperation of the bypass after a turbine trip (class 4) ~" Scenario 6: Malfunction of bypass and malfunction of a steamgenerator safety valve due to decreased opening pressure (class 4) The scenarios were transferred into a fast running simulation model of German PWR by adapting the scenarios to capabilities of the GRS-Analysis-Simulator, the simulator model, and the simulator interface. Several variants for each scenario were calculated (interaction or no interaction of operator, different types of interactions).

2. Design of the Experiment The experiment was performed in the GRS-Analysis-Simulator. The hypotheses build for the experiment were: >" HI: The higher the training level of the operators, the higher is the diagnostic performance (Factor: Qualification) >" H2: The more possibilities a MMI offers, the better is the diagnostic performance; the simpler the MMI, the worse is the diagnostic performance (Factor: Number of functions of MMI) H3: The more complex the scenario, the better is the diagnostic performance of a MMI that offers more possibilities (Factor: Complexity of situation)

92

Human Performance and Interface-Design Some Remarks based on Experiments -

To investigate these hypotheses, several conditions were introduced for the available parts of the interface (Figure 1 illustrates pictures and experimental conditions). SYNOPS: Permanently in condition 1,2, 3 and 4

Bar-Graphs: Called by User in condition 2 and 4 ~_examF)!e!rom a set)

Subsystems: Called by User in condition ,1,2, 3 and 4/example from::~a set

Trends: Called by User in condition 3 and 4 (example from a set)

Figure 1: Overview about some of the pictures available in each condition. Condition 1: ~" An initial picture (always visible): Overview of primary and secondary loop (socalled SYNOPS-picture) Pictures of subsystems (always available by active call of picture): pressurizer, steam generator, feedwater control, turbine, bypass, reactor safety system Condition 2: ~" All pictures from Condition 1 and ~" Bar graph display of the main system parameters (collection of main variables) Condition 3: ~" All pictures from Condition 1 and ~" Trend-Pictures (available by active call of Trend-Pictures) Condition 4: ~" All pictures from Condition 1 to 3

Human Performance and Interface-Design - Some Remarks based on Experiments

93

Two Groups of probands were used in the experiment: students with basic knowledge about PWRs (Group 1, n=4, called novices) and students with basic knowledge about PWRs plus practical experience on the simulator (Group 2, n=3, called experts). In the GRS-Analysis-Simulator, a block of four 21"-screens (arrangement 2x2) was used. Any picture or trend could be displayed on every screen. The starting picture was permanently displayed on the left screen of the top row. The bargraph display is positioned automatically on the right screen of the top row.

3. Realization of the Experiments Training Phase: Each participant was trained on the use GRS-Analysis-Simulator and on the available pictures. Additionally, a basic knowledge-test performed. Experimental Phase: The scenarios were varied in sequence but every participant had to diagnose every scenario once in every condition (6 scenarios * 4 conditions = 24 trails per operator, random sequence of scenarios and interfaces, each operator performed 2 sessions with 12 trails).The scenarios were demonstrated to the probands up to a predefined halting point. The probands had then been asked to tell their observations by thinking aloud. After the halting point, the probands were asked to give a summary of their observations pointing out malfunctions and failures. At this point they were allowed to use additional pictures and trends if available according to the experimental condition. Furthermore they were asked to give a suggestion for an applicable further measure and to show up the expected consequences of this measure. At last they were asked to tell how they expect the scenario to go on without any measure. Further measurements were: >" Written protocols of diagnostic statements during each session >" Automatic logging of chosen pictures with time-stamp >" Protocols on diagnostic statements after halting point

Post Experimental Phase: After the experiment, the knowledge-test was performed again and a questionnaire on interfaces, strategies, scenarios, and performance in the experiment had to be filled out.

4. Results The results will be described in two steps: First, the results concerning the hypotheses. Second, other observations and evaluations of the experimental data concerning strategies in diagnostic performance.

94

Human Performance and Interface-Design Some Remarks based on Experiments -

4.1. Results Concerning the Hypotheses The following Figures present the results according to the hypotheses mentioned above [2]. In the Figures, "E" means fault correctly recognized, "TW" means fault partly recognized, "NE" means fault not recognized. "VP" means Proband. "Anteil" means portion of success. There is no considerable difference between novices and experts in diagnostic performance. Knowledge about the technical system (here PWR-reactors) seems not to be important to find the correct navigation through the pictures (Figure 2).

..D ~ u ~ Dia.qnosis (vor Haltepun _kt)--

In Total (Gesamte_rgebnis):

Novices (VP1,2,4,5); Experts (VP3,6,7)

Novices (VP1,2,4,5); Experts (VP3,6,7)

Figure 2: Results according to H 1: The higher the training level of the probands, the higher is the diagnostic performance.

Complex scenario:

Simple scenario: S z e r m d o 3: v o r H e l t e p u n l d 80.0%

,,

"~:4~ :

::"

Szen~rto 5: v o r H a ~ p u n l d

"

:"""

- t

=

~,/~.:~..

! :

~!!.~

~':

i'.:/

ruNE

~

20.0%0.0% 1

2

3

4

1

Interface

100.0%

~ . . . . . . .

800.I L :

1

. . . . . . . . .

I!

2

Irdorfsm

3

IntsrM==

3

4

s,nmao 5:Ge=m~rcJa~s

S,~,narlo 3: G e m m t e r g e l o n J s loo.o%

2

.~

~.~.

'i..'i~ '~'2):,~ . ~ " . ?

4

O O .% 1

2 irdoft~o 3

4

Figure 3: Results according to H2: The more possibilities a MMI offers, the better is the diagnostic performance, simple MMI causes worse diagnostic performance

Human Performance and Interface-Design - Some Remarks based on Experiments

95

Diagnostic performance and complexity of the MMI are depending on each other (Figure 3). Interface condition 1 (SYNOPS-overview about the thermohydraulic flows in the plant) seems to be a sufficient source for identifying an abnormality (in the more simple Scenario 3). In the more complex Scenario 5, the SYNOPSoverview even seems to be superior to the other information sources. It seems that the other possibilities of the MMI are rather distracting the proband from generating a diagnosis than supporting him.

In Total: Szer=do 6:Gemmlrg~nls

During_Diagnosis:

Szenado6:~r Hal1~xmld 100.0%

,

.

I

~

]

!

= ~.:..

1

_,oo. 1

t!,:

18~ ~

........

E

I

2 inliflll~

3

4

Figure 4: Results according to H3: The more complex the scenario, the better is the diagnostic performance using a MMI that offers more possibilities However, in the most complex Scenario 6, the SYNOPS-overview is not sufficient (Figure 4). This Scenario can only be detected, when the small deviation gradient (the decreased opening pressure) is realized. For this, the trend-graphs are superior (Interface 3), but this is - of course - reasonable that a decreasing opening pressure can only be observed when presented and best with a trend-graph.

4.2. Strategies in Diagnostic Performance Successful diagnosis is accompanied by an increased number of pictures investigated by the proband (Figure 5). Probands searching more active in the picture-set of the MMI achieve better diagnostic performance. Failure in Diagnosis:

Successful Diagnosis:

i,A

-

Picture,4~o. - ,//, ~ < : ~ . . " ,,

M,, , ,"

i l ,~ /)f/;.-M,":.~' . ....... i~..f,d~,~.7~,,.:

1

,,,,,/i./,

......

~'/

9

!

./

i

..i

~ S t e p . .le.'.... : .

[

~'ilStepof

of ..---~i Picturecall " [

.......~ I

Average of 5,43 Pictures per Scenario

I

i

/

//

!

.../

i i

~

.....................~!et~m.~!......................'.

I

1

4

I

I

Y

I

I

~

II

11

II

Average of 4,87 Pictures per Scenario

Figure 5: Results according to number of used pictures.

96

Human Perfomlance and Interface-Design - Some Remarks based on Experiments

However, diagnostic performance is not assured, if the correct picture is called during this active search. As Figure 6 shows, failed diagnosis appears by some kind of unmotivated use of the potentials of the MMI (in Scenario 5, the failed proband only called three pictures), or by some kind of unmotivated gambling through the pictures. In Scenario 6, the failed proband went through the pictures by calling nearly every picture without any structure in this proceeding. The more successful probands performed an active search strategy (the sequential lines of pictures are above the more split-angle like line of the failed proband). Scenario 6

Scenario 5 I=] ................................

r ............................................................................

,~

,

/

Picture:No. /

/ ~ //" /

/'/'failed proband

\

Step of r-J=A6 .

,

.

,

~

.

.

call .

I

Step'of __.i

.....

_

r = u t u r e

=

I

.

s

9

Pic,t u r e ~ ! `

.

i

.

1.

.

n

,1

~

~

i ,.

Figure 6: Results according to search strategy and sequence of called pictures.

5. C o n c l u s i o n a n d Perspectives None of the (very simplified) hypotheses could be supported by the experiment. This means, we cannot define a simple rule for supporting diagnostic performance. Moreover, the experimental results show that the information has to be appropriate to what is to be detected (i.e., the task and failure mode). The scenario 6 was found as a good example for this: If we do visualize dynamic aspects like the decreased opening pressure inappropriately (e.g. by using a digital number), we do not have to wonder about the result that human have difficulties with such time aspects. But this statement only means: If we do represent time aspects inappropriately then humans will have problems in dealing with them nothing else. If such tautologies are wrongly used in the design (or assessment process), this will not exhibit an improved design of interfaces. Ergonomic layout of computerized CR is not assured by ,,nice pictures". Rather more, we have to design the pictures according to the possible spectrum of tasks and situational conditions under which the operator has to deal with the interface.

5.1. The importance of compatibility of information to the task It is necessary to link both aspects (cognitive Human properties and ergonomic design properties) together according to the tasks of the operators: If the task is to

Human Performance and Interface-Design - Some Remarks based on Experiments

97

consider a failure mode with time-aspects, then the layout of the interface has to be according to this aspect. The system ergonomic approach is known as a proposal to perform this linkage [1]. The presentation of the information has to be compatible to the nature of the parameter (absolute value if the this has a certain meaning vs. analog representation if the sequence of values in time or space have a certain meaning). Especially, if one intends to investigate diagnostic performance, the observations and evaluations of experimental data are showing that such aspects concerning the relationship of ergonomic design and human desires have a big influence.

5.2. The importance of the structure of process information pictures Another more important aspect for diagnostic performance is the structure of the MMI-picture, the picture hierarchy. The structure of process information pictures is of considerable importance to support the search-process of operators. Diagnostic performance is unrelated or even in contrast to the theoretically availability information on the MMI as far as the MMI does not provide a linkage to the correct (failure respective) sub-pictures. Process information pictures have to present the required information for a certain diagnostic step at a given time. Therefore, the interface has also to announce important information independent of the current diagnosis path of the operators. At this point, the fundamental difference between conventional and computerized process control room has to be mentioned: In a conventional control room the information is present in parallel. The operator has all the information present and has only to look around to diagnose. In a computerized control room he has to search actively and sequentially in a hidden structure of pictures. Therefore, he has to remember how to access a certain information, which means an additional cognitive effort for him. This is a clear disadvantage of computerized control rooms. To improve computerized control rooms in the future, it is not needed to concentrate on finding human non-abilities to be supported ergonomically but to find out how abilities are used and selected in a given ergonomic layout and a given failure situation.

References [1] [2]

[3]

Bubb, H. (1993) Systemergonomie. In: Schmidtke, H. (Hrsg) Ergonomie. Hanser. M0nchen. Stolze, P. (1999) Konzeptsammlung und systematische Auswertung theoretischer und empirischer Befunde zu StOrfi~llen in Kernkraftwerken. MeMoVis-T-ISTec-09. ISTec, Garching Str~iter, O. (1997) Beurteilung der menschlichen Zuverltissigkeit auf der Basis yon Betriebserfahrung. GRS-138. GRS. KOln/Gerrnany. (ISBN 3-923875-95-9)

Putting the Normative Decision Model into Practice

Timm Grams University of Applied Sciences, Fulda, Germany e-mail: timm.grams @et.fh-fulda.de

Abstract: The proposed normative model of operator behaviour, called decision event tree (DET), is basically a model of multistage decision under risk. Operator errors are defined to be deviations from the so defined norm. By this the methods of technical risk assessment, economic decision theory and psychology can be utilised for an assessment of operator errors. The proposed model reveals that 1. without a precise definition of what the norm is we don't know what an error is, 2. scenario oriented psychological experiments can be misleading, and 3. good tools are needed for the design of normative models.

1. Introduction O v e r a l l goal. W e want to know the cognitive mechanisms and causes of typical operator errors because we want to avoid these errors through a better design of

man-machine interfaces. M e t h o d of investigation. During simulation experiments the subjects are exposed to a series of scenarios. These scenarios are based on rather realistic operator tasks of a coal-fired power station. A test person representing an operator is called subject. The person who has planned and is conducting the experiment will be named experimentalist. The subjects have to solve a diagnosis task: The experimental conditions are given by introducing failures of certain components of the power station. The task is to find the real causes of the malfunctions and to advise the right mitigation measures. The subjects/operators are asked to think aloud while operating the system. These protocols are then analysed by the experimentalist with respect to possible deviations of the subject's mental model from reality and the technical and psychological causes thereof, the human-machine mismatches [ 1].

100

Putting the Normative Decision Model into Practice

Problems encountered. Such investigations of decision making in complex situations are posing serious difficulties concerning the planning of the psychological experiments and the analysis of the results: The experiments include many hardly controllable extraneous variables. Through complex instructions further extraneous variables like the learning capabilities and the background knowledge of the subjects are introduced. The decision situations are quite complicated and the operator's behaviour cannot be assessed solely from his following a certain predefined linear chain of events. But most important seems to be the experimentalist's risk of falling into the same thinking traps (cognitive deceptions) as the operators and subjects. The experimentalist could overlook or deliberately exclude important failure hypotheses in the evaluation scheme of a scenario. Or he could - according to faulty assumptions upon the frequency or severity of failures - concentrate on a scenario of no significance in reality. To put these mistakes into a generalized setting [2, 3]: We are seeking evidence and confirmation of our assumptions and prejudices thus losing more plausible hypotheses out of sight. By our cognition's tendency towards meaning we often select irrelevant hypotheses on the basis of some rnisleading clues or by applying inappropriate heuristics. Within a taxonomy of operator errors the above thinking traps can be sorted into the classes of inductive reasoning errors and linear cause effect thinking [4].

Proposed solution: normative modelling. To overcome these difficulties it is proposed to base all investigations of operator errors on normative models. A normative model says what the operator should do. An operator error is defined to be a deviation from the normative model. The framework of normative modelling should take into account the normal behaviour of a real process and plant, the various component failures and resulting symptoms, as well as their frequencies and probabilities. There must not only be a simulation model of the plant but also a - perhaps simplified - risk analysis thereof [5], (If we don't know what types of malfunctions we possibly encounter, we cannot know how to act on their symptoms at an as low risk as possible.) The design of an experiment starts with the selection of a scenario following some initiating event, p. e. the breakdown of a coal mill. The symptoms of a malfunction generally cannot be traced back directly to the initiating event because of ambiguity. Other initiating events could result in the same symptoms. The operator should take into consideration all possible causes. Some of the failures taken into consideration may ask for urgent action, others may not. Therefore the normative model should include all known failure hypotheses

Putting the Normative Decision Model into Practice

101

and their a posteriori probabilities, i. e. the conditional probabilities under the symptoms of the scenario. A short introduction into the normative modelling technique will be given. By an example it will be demonstrated how to put this technique into practice, and what can be learned from its application.

2. The Task: Fault Diagnosis of a Coal-fired Power Station The setting. Let the operator's task be the fault diagnosis of a coal-fired power station. Within a specific scenario the operator is confronted with the following symptom: There is a rising level in one of the tanks in the high pressure preheater (HPP). Possible causes (hypotheses) are 1. a small leak in a feedwater pipe or 2. a spindle breakage. The failure rates attributed to a small leak or a spindle breakage, respectively, are known from a technical risk assessment of the power plant. From this the a posteriori probability of each of the hypotheses can be calculated. The probabilities are 0.05 (leak) and 0.95 (break). Without countermeasures a leak possibly results in a bypass activation. In any case the efficiency of the plant will be reduced considerably. The severity of the consequences of the leak can be reduced by mitigation measures. Likewise a breakage results in a reduced efficiency. The severity can be reduced by appropriate countermeasures. The operator may choose to wait and see and he possibly may miss the right time for countermeasures. Multi-stage decision under Risk. According to these observations and hypotheses the operator may choose among the following alternatives of action: do nothing (default), mitigate the consequences of a leak, mitigate the consequences of a breakage, or defer the decision in favour of further examination. After a time consuming investigation it may be possible to discriminate between the hypotheses. If the leak hypothesis is true, there is a loss of feedwater in the main feedwater circle and the temperature of the overheater is rising. This can be observed within 60 s. If the break hypothesis is true, the system stabilizes after a while due to automatic controllers, there is no loss of feedwater and no rising temperature of the overheater. Each of the actions is accompanied by some risk. The risk of an action can be calculated by means of fault tree or event tree techniques. Without countermeasures the following will happen in case of a leak: The level in the HPP increases while the feed water supply to the boiler decreases. The level controller of the HPP becomes active to drain off water from the HPP. If the level continues to increase an emergency valve opens additionally. If these control actions are not able to reduce the level in the HPP two bypass valves become active.

102

Putting the Normative Decision Model into Practice

This results in a reduced efficiency. The bypass activation is assumed to occur with probability 0.5. Even if no bypass activation occurs the efficiency will be reduced considerably. The cost of bypass activation and a reduced efficiency is set to the severity value 1. If an action is taken to mitigate the consequences of the leak the severity is equal to 0. Without countermeasures a breakage can result in a reduced efficiency with probability 1. The severity value is assumed to be 1. The loss can be reduced to 0 if appropriate countermeasures are taken. If the operator chooses to wait and see, he possibly may miss the right time for countermeasures. The decision in favour of a further investigation will be accompanied by a penalty (cost) of let us say 1/10 in the case of a leak. In case of breakage there is a loss of 1/100 due to the deferment.

3. The Normative Model Trees. The Normative Model defines what the operator should know of the processes and of the machine: All hypotheses as well as their a priori probabilities, all scenarios following from these hypotheses, all possible observations and their conditional probabilities, the a posteriori probabilities of the hypotheses, and the severity of all consequences. This knowledge can be represented by a so called tree. The basic terms and definitions for trees as well as the fundamental algorithms on trees can be found in the books on data structures and algorithms [6, 7]. Two types of trees are relevant with respect to the given task: 1. decision trees as known from game theory and from the studies of economic behaviour [8], and 2. event trees as known from technical risk assessment [1]. Event trees are the appropriate means for modelling the consequences of decisions. To meet the requirements of the above multi-stage decision situation these two types of trees will be integrated into one model. We will call such a tree a Decision Event Tree (DET).

By means of DETs we can find the optimum decisions with respect to the minimization of risk. Additionally, DETs serve well as a classification scheme of operator errors [4]. Structure of Decision Event Trees (DET): A DET is defined to be a tree with two distinct kinds of non-terminal nodes: decision nodes and event nodes. In the graph of a tree the decision nodes are represented by rectangles, the event nodes by

Putting the Normative Decision Model into Practice

103

circles. A terminal node or leaf does not need a graphical representation. The connecting line from a parent node to one of its child nodes is called an edge, fig. 1. T I T2

T3 T, T~ Figure 1: Decision event tree with root D~and five terminal nodes (T~..... T~) Adding attributes: An edge may be associated with some cost or loss, called transition loss. These values will be attributed to the child node of the respective edge. For all children of event nodes an additional attribute has to be defined: The (conditional) probability of a transition from the parent node to this node. This probability will be called transition probability. Evaluation of DETs: The calculation of minimal risk can be achieved recursively starting with the terminal nodes. During the evaluation an optimum value will be assigned to all the nodes. Each node is the root of a tree or subtree. The optimum value of a node is defined to be the minimum obtainable risk of the respective tree or subtree. The optimum values of the nodes are defined as follows. 9

Terminal node: zero

9 Decision node: the minimum value of the optimum values of all its subtrees (including the transition loss associated with the edge from the node to the subtree's root) 9 Event node: the mean value of the optimum values of all its subtrees, taking into account the probabilities of the edges leading to the subtrees and the associated transition loss Such an evaluation can directly be implemented by a recursive procedure [6]. Evaluation using spreadsheets. The representation by indentation is a textual alternative to the graphical one [7]. This representation can immediately be transferred into a spreadsheet. The name of a node as well as its attributes (including those of the incoming edge) are the constituent parts of the node record. Each node record is written into one separate row of the table. (The identifiers of all children of a node must be distinct. Identical names are allowed for nodes with different parents.)

104

Putting the Normative Decision Model into Practice

The cells of a node record are arranged in the following manner:

{void} [TransitionProbability] Name [TransitionLoss] Optiraum Value The meaning of the contents can be seen from the cell type, given in italic letters. Square brackets are used to denote optional cells. If the cell type is enclosed by braces there can be any number of adjacent cells of this type. The number of leading void cells depends on the level of indentation: the children of a node are indented by one more cell than their parent node. The transition probability attribute is only applicable to children of event nodes. By the convention of placing the transition probability - a number - in front of the node name the parent node is unequivocally marked as an event node. The transition loss attribute is applicable to all nodes except the root. The calculated optimum value of a node is given in bold numbers at the end of its record.

4. The DET of the Preheater Diagnosis Table 1 shows the spreadsheet version of the DET of the preheater diagnosis. The framed parts plus the extra root node are event trees. These trees describe the events occurring without further interference by the operator. If the operator would do nothing, the consequences are described by the tree with root RisingLevel.Default. All other event trees result from this tree by simple manipulations. (The full name of a node consists of the sequence of all node names along the path from the root up to this node, written in dot notation: "RisingLevel.Default" denotes the child Default of the node RisingLevel. The latter is the root of the entire tree.) For explanation let us have a look at the event tree with the root node RisingLevel Deferment. The optimum value of the deferment node results from the following calculation: With probability 0.05 the rising temperature event follows. This event entails a transition cost of 0.1 due to deferment. With probability 0.95 the no rising temperature event follows. This event entails a transition cost of 0.01 due to deferment. Since the deferment node is an event node (as can be seen from its children) the optimum value of this node is given by the mean value 0.05*(0.1+0)+0.95"(0.01+0), which is equal to 0.0145.

Putting the Normative Decision Model into Practice

~.:

....

105

0.0145

Let,'~:: +::

Default

0 0.05 0.95

LeakMilig

BreakMitig

Deferment

0 0.05 0.95 0 0.05

1 Leak 0.5 0.5 Break 1 0.~3 Leak 1 Break 1 0.05 Leak

0.95

0.5 0.5 Break 1

0

0.0145

LeakMitig

0 Bypass Reduction 0 Reduction

1 1 1 1 1

0 NoReduct 0 Reduction

0 0 1 I

0

1

Bypass Reduction 0 NoReduct

1 1 0 0

0

0

I1 BreakMitig

LeakMitig

0 1

0

i1 BreakMitig

0

I'

Leak 1 1 Leak 0.5 0.5

0 0 0

0 0

00

0

0 NoReduct

0 0

0

l

0 Bypass Reduction

1 1 1

0 0

1 0

Reduction

11

0

I

1

I~11~

0

o

0

j

1

Table 1: DET of the Preheater Diagnosis (Spreadsheet). This is also the optimum value of the entire tree, because all other decisions (default, leak mitigation, and break mitigation) yield higher risk values. Under the given conditions the optimum decision is the deferment decision. This can be seen from the calculated numbers and the evaluation rules given above. This result is strongly dependent from the chosen parameters. Let the penalty for deferment in the case of a breakage be 0.05 instead of 0.01. This would result in an increase of the risk of deferment from 0.0145 to 0.0525. Now the deferment decision is no longer the best choice. An immediate break mitigation would be preferable.

5. The Lessons

Learned

Without a precise definition of what the norm is we don't k n o w what an error is. Errors are defined to be deviations from the norm. Operator errors are defined to be behavioural deviations from the known best behaviour described by a decision event tree. Thus the precise representation of the DET is crucial with respect to the assessment of operator behaviour. The normative model defines unambiguously what we must know of the processes and machines the operator has to control:

106

9 9 9

Putting the Normative Decision Model into Practice

all hypotheses to be included in the model as well as their a priori probabilities all scenarios following from these hypotheses all possible observations (symptoms) and their conditional probabilities, as well as the a posteriori probabilities of the hypotheses

Scenario oriented psychological experiments can be misleading. Sometimes a shorter way to the understanding of behavioural errors is chosen: The actions of the subjects are compared with some predefined optimum actions given by the scenario. By this method optimum decisions could possibly be judged as being faulty. This can be seen from the DET of the preheater diagnosis: Let the penalty for deferment in the case of a breakage be 0.05 instead of 0.01. As shown above the optimum decision under the normative model is an immediate break mitigation. All other decisions are operator errors under the normative model. Let us now consider a scenario following a leak within the HPP. The first observable symptom is identical to that of the breakage scenario (a rising level in one of the tanks of the HPP). Under the narrow view of the scenario only the leak mitigation would be accepted as a correct decision. All other decisions would be classified as an error. But how could the subject expect this very rare scenario? The more likely scenario of a spindle breakage (given the symptom) demands prompt reactions. The mitigation of breakage should be undertaken even if there is a chance of 5 % of a leak instead of a breakage. The useless actions in the view of the currently applied scenario are the rational ones in the view of the normative model. Thus a subject's behaviour during psychological experiments cannot be assessed solely from its following some predefined scenario. Instead, a normative model should be applied during the evaluation process. Good tools are needed for developing normative models. Spreadsheet techniques are error prone. Larger DETs should not be evaluated by means of spreadsheets. An appropriate DET tool should * 9 9 9 9

provide a user interface designed for input error avoidance (for example by plausibility checks) make structural changes of trees easy provide means for easy navigation through the tree offer synoptic representations as well as a zoom facility provide automatic evaluation of DETs

Putting the Normative Decision Model into Practice

107

References [1] [2] [3] [4]

[5] [6] [7] [8]

Leveson, N. G.: Safeware. System Safety and Computers. Addison-Wesley, Reading, Mass. 1995 Salvendy, G. (Edt.): Handbook of Human Factors and Ergonomics (2nd edition). John Wiley, New York 1997 Reason, J.: Human Error. Cambridge University Press 1990 Grams, T.: Operator Errors and their Causes. In: Computer Safety, Reliability and Security. Proceedings of the 17th International Conference, SAFECOMP '98, Heidelberg, Germany, October 1998 (Herausgeber: W. Ehrenberger). Lecture Notes in Computer Science. Springer-Verlag, Berlin Heidelberg 1998 (S. 89-99) Shooman, M. L: Probabilistic Reliability: an Engineering Approach. Robert E. Krieger Publishing Company, Malabar, Florida 1990 Aho, A. V.; Hopcroft, J. E.; Ullman, J. D.: Data Structures and Algorithms. AddisonWesley, Reading, Massachusetts 1983 Knuth, D.: The Art of Computer Programming. Vol. 1: Fundamental Algorithms. Addison-Wesley 1973 Neumann, J. von; Morgenstern, O.: Theory of Games and Economic Behaviour. Princeton University Press, Princeton, New Jersey 1972

Acknowledgement:Preparation of the manuscript was supported by the ,,Volkswagen Stiftung" in Germany.

Discussion Session II

As planned, this discussion session concentrated upon the MeMovis project and - in particular - on the method used for the experiments. The main part was concerned with the 'de'sign' of experiments. This comprised aspects like complexity of scenarios, task analysis, choice of subjects, etc. With respect to complexity a kind of agreement could be observed insofar as this concept as an isolated measure was of limited meaning. Every experiment inevitably reduces the complexity of reality. The important choices that have to be made relate to content, control variables, constraints, etc. - and they have to be made consciously and carefully. They have also to be based upon experience with older experiments. A key property of experiments has to be their 'representativeness' although this may be difficult to quantify or to measure. As an illustration for the importance of this criterion it was e.g. mentioned that a field study - which is usually regarded as representing 'real life'- that is poorly planned may be much less representative than a thoroughly conceived experiment in a laboratory. Aspects that are of importance for the representativeness of an experiment are e.g.: capabilities of the simulator, realism of interfaces, training of subjects, number and kind of possible diagnostic strategies and action options, etc. Broad room was also given to the topic of 'task analysis'- possibly because from the presentations of the project partners it had not become quite clear how this had been done in the project. It turned out that basically two strategies had been combined. One was the 'scenario based' approach, partially using personal experience from industry and earlier experiments about actual and plausible cases of process irregularities and their detection and partially using an explicit analysis of a large database with recorded undesirable process states and appropriate countermeasures. The other one was based upon a 'cognitive analysis' concerned with the information necessary for operators to correctly diagnose a situation. A third strategy was mentioned but had not been used: 'decomposition' of more global tasks into subtasks. The discussion about the mutual advantages of using either professional operators or non-professionals (like e.g. students) - with some special training - as subjects was also very intense and revealed some new aspects. A very interesting one was that careful evaluation of the experiments allows to normalize the measurements in such a way that the results for both groups of persons are comparable. It also turned o u t that in really novel situations the 'experts' may have to resort to basic knowledge and therefore behave like novices. In various cases it had been observed that professionals had mental reservations against the 'new-fangled' interfaces with which they had to work. So, taking also into account the pragmatic point of view that it costs much more to employ professionals as subjects, a reasonable approach

110

Discussion Session H

appears to be that carefully trained non-professionals a subjects are good choice in most cases, but that the results of such experiments should be cross-checked from time to time by experiments with professionals. Triggered by this topic a discussion ensued about the role of user participation in interface design. It was generally agreed to be good method, but with some drawbacks. So e.g. it has been observed that experienced personnel is used to the existing interfaces in such a way that they simply can not think of basic innovations. There have always to be specialists who propose the innovations - and these have to be tested for acceptability in a dialogue with real users. This latter procedure was also recommended for the taxonomy of interfaces proposed by some project partners. It was widely agreed to be a good starting point for a framework for further conceptual work on interface design but it needed further discussion (like the one mentioned~ and - eventually - extension by a fourth dimension: the discrimination between 'active' and 'passive' interfaces. 'Active' meaning in that context that the operator always has the initiative in searching for and identifying relevant information, 'passive' meaning the predominance of signals and cues that guide and influence his behaviour.

Summarized by Peter F. Elzer

Multimedia Interfaces and Process Control: The Role of Expressiveness

James L. Alty IMPACT Research Group, Department of Computer Science Loughborough University, Loughborough, UK Email: [email protected]

Abstract: The concepts of medium and multimedia interfaces are defined and input and output media are examined. The problem of how and when to use alternative renderings in process control interfaces to maximise operator performance is considered, and the idea of expressiveness of a medium is explored. It is related to an Information Signal to Noise ratio idea and some examples of its use in a process control experiment are given.

1. Why Multimedia Interfaces? Many of today's industrial plants are controlled by supervisory and control systems that increasingly reside in software and hardware. The successful and safe operation of a plant depends upon careful engineering of the hardware and the software that drives it. However, the skill of the operators, their understanding of what constitutes an acceptable system state, and the implications of deviations from such states, are also very important. Their understanding clearly depends upon appropriate training and experience, but an additional key factor is the design of the interface between the operators and the system. Do these interfaces clearly communicate the relevant information whilst at the same time avoiding irrelevant information? Are they designed so that they take advantage of human cognitive characteristics rather than compete with them? Are the various media involved in communicating information to the operators being used appropriately and effectively? Recent developments in interface design have included advances in multimedia technology. The question has therefore arisen as to how such technology can be used in interface design to improve operator performance. The term "multi-media" came into common usage in the late 1980's. It describes a set of techniques and technologies for facilitating communication between computers and human beings through the use of a variety of "media". The term "medium" when applied to interface design essentially means a language (or

114

Multimedia Interfaces and Process Control: The Role of Expressiveness

medium) of communication. Since languages consist of a set of allowable symbols (e.g. words in written English), a set of rules for putting these symbols together (e.g. English syntax), semantics (what the word collections actually mean), and pragmatics (conventions and common usage), media can be described in a similar way. Using such a definition one can show the difference between the media of written and spoken language. We often talk about languages as being "expressive". Expressiveness here means being able to communicate what we need to say, with the required subtlety, in a concise manner. In like manner, some media are rich and expressive, whilst others are sparse and very limited in what they can express (compare written language and semaphore). However, richness, or expressive power, does not always mean bestfor-purpose. The expressiveness of the medium should ideally match that of the task to be carded out at the interface [ 1]. The medium should be complex enough to be able to represent the problem, but not so complex as to hide or obscure the meaning. Human beings have employed a variety of media to communicate with each other since the earliest times. It is the degree of success with which human beings use multiple media that has convinced interface designers of the benefits of multimedia communication. Some researchers believe that multimedia communication is a natural product of evolution and corresponds closely with how the brain has developed. Marmollin [2] has described multimedia as exercising "the whole mind". An extension of this viewpoint sees the human brain as having evolved in a multi-sensory environment, where simultaneous input on different channels was essential for survival. The processing mechanisms of the human brain have therefore been fine-tuned to allow for simultaneous sampling and comparison between different channels. When channels agree, a sense of safety and well-being is felt. When channels degrade, input from one channel can be used to compensate another. Thus, input channel redundancy (within limits) may well be an essential part of human-human communication.

2. T y p e s o f M e d i a Media can be subdivided into input and output media. These can then be further divided according to the sense used to detect them - visual, aural or haptic media. A further subdivision can then be effected (for example, into language and graphics for visual output media, or sound and music for aural media, see Table 1).

Multimedia Interfaces and Process Control: The Role of Expressiveness

Aural

Visual

Haptic

Input Media

Natural Sound Spoken Word Synthesised Sound

Video Camera TextScan Diagram Scan Gesture Eye Tracking

Output

Natural Sound Music Synthesised Sound Spoken Word

Written Text Graphics Animation Still Video, Moving Video

Keyboard Mouse Breathing Tube Data Glove Touch Screen Tracker Ball Data Glove Braille Pad

f

Media

115

Table 1: Some Common Media.

2.1 Output Media Many current output media are well-matched to human capabilities. Media such as text, graphics, pictures, video and sound, map well onto books, pictures, and the world we see and hear. Although normal sized VDU screens do not have quite the same properties as A4 paper, the correspondence is close enough. Most human beings therefore have little trouble in adjusting to understanding well-designed visual or aural output media. The problems of designing effective output using these media are similar (but not the same) as those in traditional media design (for example, publishing). Most computers can support quite sophisticated aural output (e.g. music), but this is rarely utilised except in games. Gaver [3] has suggested the use of Auditory Icons. These are well-known, natural, sounds that have common associations. Blattner [4,5] has suggested the use of structured Earcons (based on simple musical motifs. The author [6] has mappings run-time computer algorithms to music, mapping the internal workings of an algorithm to musical structures. Musical mappings have also been suggested to aid computer program debugging [7,8]. Output text has been extended through the creation of Hypertext structures [9]. Hypertext linkages across communication networks have now become commonplace, an obvious example being the Web, and the term Hypermedia is often used to describe the creation and support of linkages between different media. Elements of text may be linked to photographs, movies or even sound sequences either on local systems or across the communication network. One current problem in Hypertext and Hypermedia structures, however, is navigation. Users can easily become lost in Hypermedia space. Non-dynamic links can also inhibit exploratory learning. This is partially because the hypertext approach has no obvious counterpart in normal human activity.

116

Multimedia Interfaces and Process Control: The Role of Expressiveness

2.2 Input Media Current input media are cumbersome and unnatural, requiring skills (such as keyboard or mouse skills) to be used effectively. Input media often need to be coupled with some form of output medium to be useful. Keyboard input is not effective unless the user receives simultaneous output of what is being typed. In a similar manner, input using a mouse requires visual feedback to be effective. This complicates the analysis of input media. Recently, there has been active research on new input media. Developments have been reported on voice recognition (now beginning to reach acceptable levels of performance), gesture and pointing (where the actual visual gestures are tracked by video cameras and interpreted). Other work involves eye-movement (the actual movement of the eye is tracked and used as a selection device), lip motion (to assist in speech recognition), facial expression and handwriting, and even brain wave input. The research is driven by the current primitive state of input media in contrast to human-human communication. An interesting feature of many input media is their impreciseness. Voice recognition is difficult because of extraneous noise, gesture is often vague and ambiguous, and lip motion is not read accurately by most human beings. Such imprecise media are still useful, because human beings process such media in parallel (for example, gesture and lip movement usually accompany speech, exploiting redundancy across these channels). Human beings therefore compare inputs across the different channels for confirmation, or to seek support for the interpretation in one channel by another. Experiments on input media have involved the combination of speech recognition with lip reading, gesture with speech, and speech with handwriting . One experiment concerned the simultaneous input of lip reading and voice input. The acoustic input performance was measured in clean and noisy environments [10]. When the acoustic input was clean, a word accuracy in excess of 90 % was attained. The Lip reading performance, on its own, varied between 32 and 47% accuracy, and, when used in parallel with the acoustic input had minimal effect on overall accuracy. When the noisy acoustic input was used, however, acoustic recognition on sit own fell to around 50% but with lip-reading added in parallel, performance improved to over 70 %. Thus, adding the lip reading input (which had a relatively poor recognition rate on sit own) boosted the recognition rates of acoustic input in the noisy environment.

Multimedia Interfaces and Process Control: The Role of Expressiveness

117

2.3. The Importance of Alternative Renderings Although an important aspect of interface design is choosing the ,,obvious" or "best" medium for particular communication requirements, the deliberate presentation of information in a ,,foreign" medium can deliver new and interesting insights to a problem. Musical harmony is normally presented through the audio channel, yet new insights into harmonic progressions can be obtained by displaying the harmony in a visual medium. A good example of this is the HarmonySpace application of Holland [11]. This tool offers both experts and beginners the opportunity of exploring musical harmony by allowing them to use spatial attributes in their exploration (e.g. nearness, centrality and shape similarity). Similarly, music can be used to assist in the understanding of computer algorithms or physical processes such as turbulence [12]. This use of unusual media for presenting information can be an aid to problem solving in difficult process control situations. It is well-known that operators often get side-tracked into pursuing false explanations for control problems even when the current facts contradict them. They make strenuous attempts to fit every new fact that emerges into their current framework. A new viewpoint using a different medium might shift operators out of these blind alleys.

3. Media Characterisations The Multimedia design issue is not primarily a technical one. The issue is one of choosing an optimal set of media to achieve a particular goal. In process control interfaces, our goals may be: 9 9 9 9 9

performing the task more quickly performing the task with fewer errors making the task easier making learning and remembering the task easier achieving safer operation

The proponents of Multimedia technology hypothesise that good multimedia design, should assist designers in reaching these goals. What is needed is a characterisation which enables designers to map task characteristics to media properties. Media have different capabilities for communicating different types of information. Figure 1 shows a simple space where some properties of various media are illustrated.

118

Multimedia Interfaces and Process Control: The Role of Expressiveness

The labelled axes are: Real to Abstract (by how far is the representation connected with reality?) Static to Dynamic (What role does time play in this medium?) Quantitative to Qualitative. (How far can the medium handle numbers and ratios?)

.......................................S c h = m l j'lcture

Qualitative

.."

fftative

Keal

9

~

AOstract

Figure 1: A Media Space Contrasting the Properties of Different Media. Text is highly Abstract, Static and can be Quantitative. Moving Video is Qualitative, Real and Dynamic. Trend .Graphs are Dynamic, Abstract and Qualitative. Of course these are just typical examples. Actual designation will depend upon their use in any particular situation. For example, a TrendGraph could be reasonably Quantitative and a Video very Abstract. The figure does highlight differences between media and indicate how they might be effective under different condition.

4. The Characteristics of an Acceptable Representation We can assert that all acceptable representations must have at least the following properties: a) all the information required must be currently available (or accessible) in the representation chosen. b) the information should be presented in such a way as to be readily perceived and understood in the right context by the user. This means that the representation should generate, or match, some form of model in the user's head (an internal representation).

Multimedia Interfaces and Process Control: The Role of Expressiveness

119

c) other information, not relevant to the problem solving process, should be kept to a minimum. Designers manipulate the external representation (that is, the set of media chosen to communicate), to make particular operator tasks as easy as possible. To do this the external representations must generate (or match with) the most effective internal representation to enable the operator to solve the problem. What sort of representations might best be used? An obvious start might be to use reality. In the process control case the designer could show actual pictures of the plant and its components and (perhaps) video data of the process in action. In educational programmes the emphasis would be on real pictures and videos (though this might be difficult in Mathematics). Unfortunately, realistic representations tend to fail points a) and c). This is because most of the information provided by realistic representations is not useful for many purposes and tends to add to the information noise. In Process Control, for example, a schematic diagram of a valve is usually more relevant and useful than a picture of it. In educational situations, schematic diagrams are usually more meaningful than realistic pictures. In spreadsheet and word processor applications, the last thing a user wants is a realistic representation of what is happening inside the computer. Process control designers, therefore, normally use representations that are more symbolic than real. However, this provides an immediate problem. Once designers move away from realistic representations, the choice of representations becomes very large. Sometimes the choice may not be too difficult. There may be a set of standardised symbols for an application (Pipe and Instrumentation Diagrams in Process Plants, for example). At other times the choice may be completely open. This therefore is the key issue in Multimedia Design - How does a designer choose a medium (or media) to represent a problem in such a way as to maximise the operator's capability of solving it? There are obvious examples that illustrate this point. Suppose a group of chemists wish to solve a problem involving the gross topology of a complex molecule. They could be presented with a complete listing of the electron density values at every point in space. Alternatively, they could examine a 3-dimensional representation of electron density values where colour is used to show the distribution of electron density in space. In both cases all the information is there to solve the problem. However, the listing would require hours of work, whereas the 3-D view might answer some questions almost instantaneously. Clearly, for this problem, one representation is more appropriate than another. This example tells us that in trying to measure the "effectiveness" or "expressiveness" [ 13] of a medium for a particular task, we are seeking to calculate something like a signal-to-noise ratio for some medium trying to meet some operator objective. In other words, the critical information is the signal and all the additional irrelevant information provided by the representation is the noise. The

120

Multimedia Interfaces and Process Control: The Role of Expressiveness

calculation is not easy since the definitions of signal and noise will vary with the task, and the capabilities of the user. One might think of it simplistically as an Information signal-to-noise ratio: Essential Information required to solve problem. Total Information presented In the molecule example, it is not hard to see that part of the problem is a surfeit of "noise" raising the value of the denominator. Another way of measuring this signal-to-noise ratio might be to determine the work required by an operator to extract the required information to solve the problem. This is still hard to calculate but it does at least provide a way of measuring it. This would be appropriate for performance problems but may not be appropriate for educational situations. In these cases, a better measure might be concerned with how much knowledge was imparted, additional work being part of the learning process.

5. Examples from Process Control Because of space limitations we will provide two examples from work done in the PROMISE project [14] (supported by the EC in the ESPRIT programme). In this work a set of laboratory experiments were carried out using different combinations of media to render the same problem. The problem was the well-known Crossman's water bath [15], and the different media included text, scrolling text, graphics, dynamic trend graphs, sound and speech. A large number of subjects were evaluated carrying out a set of procedures using these different media. The tasks which subjects performed could be classified at three difficulty levels [16] depending upon the task compatibility, described as levels 1,2, or 3 (3 being the most difficult). One experiment concerned a comparison between textual and graphical representations. Three variables were measured - the time taken to stabilise the system from a disturbed state, the number of actions performed, and the number of times the state variables were outside a defined envelope of behaviour (warning situations). Figure 2 shows the results for graphics and text over the three task difficulty levels. It is interesting to note that, as the task complexity becomes higher, the graphical representations start to provide improved performance in all three measures. At low task difficulty there is no difference in performance. Here the expressiveness of the graphical medium provides no advantage over the simpler textual medium at low complexity. As the task difficulty increases, information which previously was "noise" in the graphics now becomes signal and performance improves. For difficult tasks, the expressiveness of the graphical representation better matches the task requirements.

Multimedia Interfaces and Process Control." The Role of Expressiveness

Time

300

Actions

12

j6

200

100

,

.

0

.

0

-

9

1 2 3 Task Difficulty "'-~"

Textual Graphical

Warning

4

situations

/

3

.: .

121

9

/

1 -

1

i,

9

2

-

m

-

|

3

-

9

1

-

9

2

-

m

3

Category 1 - compatible tasks Category 2 - incompatible tasks Category 3 - incompatible tasks

Figure 2: Effect on Performance of Graphics and Text. Another example concerns the use of Sound. The sound of flowing water was added to give an indication of the speed of filling of the tank, and the current water level. When performance (overall) was compared in the sound versus no-sound conditions, the results marginally favoured the no-sound condition (Table 2). Condition

Completion Time

Number of Actions

Sound No Sound

124 107

6.99 4.73

Number of Warning Situations 0.906 0.5

Table 2: The Effects of Sound. However, when the task difficulty was separated out, an interesting Figure 3 Effect of Task Difficulty on Sound Performance result was obtained (Figure 3). A clear cross over can be seen. It appears that at low task difficulty, sound was part of the information noise, but that at greater task difficulty it became part of the information signal. This Information Signal to Noise concept can therefore be useful for describing the usefulness of different media in varying task situations.

122

Multimedia Interfaces and Process Control: The Role of Expressiveness

Actions

Time

300

200

Warning situations

/

6

~7/ 9

9

"1"

! 1

8

J

100

/

10

"2"

"3"

. !

4

9

. 2

9

.

,

3

2 9

"I "

"2 "

'3

Task Difficulty

............

no sound sound

C a t e g o r y 1 - c o m p a t i b l e tasks C a t e g o r y 3 - incompatible tasks) C a t e g o r y 2 - i n c o m p a t i b l e tasks

Figure 3: Effect of Task Difficulty on Sound Performance.

6. Conclusions The definition of expressiveness of a medium is an attempt to capture the complex process of relating medium choice to task requirements in interface design9 Expressiveness is an elusive quality but it is connected with the levels of abstraction afforded by a medium and its relationship with the task needs. We have reformulated the expressiveness idea as a Signal to Noise Ratio9 Using this approach we have been able to offer an explanation of some Process Control results in terms of the relationship of signal to noise and the movement of information between these labels..

References [I] [2]

[3] 14] [5]

Williams, D., Duncumb, I & Alty J.L., Matching Media to Goals: An Approach based on Expressiveness, Proc HCI'96, People and Computers XI, (Kirby, M.A.R, Dix, A.J., & Finlay, J.E., eds.), ,Cambridge University Press, pp 333 - 347, 1996. MarmoUin, H., Multimedia from the Perspective of Psychology", in Kjelldahl, L., (ed.), Multimedia: Systems Interactions and Applications, Springer-Verlag, Berlin, pp 39 52, 1992. Gaver, W.W., Auditory Icons: Using Sound in Computer Interfaces, in Human Computer Interaction, Vol. 2, No. 1, pp 167 - 177. 1986. Blattner, M., Greenberg, R., and Kamegai, M., Listening to Turbulence: An Example of Scientific Audiolisation", in Multimedia Interface Design, Blattner, M., and Dannenberg, R., (eds.), ACM Press, Chapter 6, pp 87 - 102. 1992. Blattner, M9 Sumikawa, D., and Greenberg, R., Earcons and Icons: Their Structure and Common Design Principles, in Human Computer Interaction, Vol 4, No. 1, pp 11 - 44. 1989.

Multimedia Interfaces and Process Control: The Role of Expressiveness

[6]

[7] [8]

[9] [10] [11]

[ 12]

[13]

[14]

[15]

[16]

123

Alty, J.L., Can we Use Music in Computer-Human Communication ?, in People and Computers X, Kirby, M.A.R., Dix, A.J., and Finlay, J.E., (eds.), Proc. of HCI'95, Cambridge Univ. Press, Cambridge, pp 409 - 423. 1995. Francioni, J., Albright, L., and Jackson, J., Debugging Parallel Programs Using Sound", in ACM SIGPLAN Notices, Vol. 26, No. 12, pp 68 - 75. 1991. Alty, J.L., and Vickers. P., The CAITLIN Auralization System: Hierarchical Leitmotif Design as a Clue to Program Comprehension, Proceedings of 4'h ICAD, (Mynatt, E., and Ballas, J.A., eds.), Santa Fe Institute, ISBN 0-9661289-07, pp 89 - 96. 1997. Nielsen, J., Hypertext and Hypermedia, Academic Press, London, pp 263. 1990. Waibel, A., Tue Vo, Minh., Duchnowski, P., and Manke, S., Multimodal Interfaces, The Artificial Intelligence Review, pp 1 - 23. 1995. Holland, S., Interface Design Empowerment: A Case Study from Music, in Multimedia Interface Design in Education, Edwards, (A.D.N., and Holland, S., eds.), NATO AS! series F, Springer Verlag, Berlin, pp. 177 - 194, 1992. Blattner M., Greenberg, R.M., and Kamegai, M., Listening to Turbulence: an Example of Scientific Audiolisation, in Multimedia Interface Design, (Blattner, M., and Dannenberg, R.M., eds.), Chapter 6, pp. 87 - 102, Wokingham: ACM Press, 1992. Williams, D.M.L., and Alty, J.L., Expressiveness and Multimedia Interface Design, Proc. of Edmedia-98, (Ottman, T., and Tomek, I. Eds.), Freiburg, Germany, pp 1505 1510, 1998. Alty J.L., Bergan, M., Dolphin, C., & Schepens, A., Multimedia and Process Control: Some initial Experimental Results, Computers and Graphics, 17 (3), pp 205 - 218. 1993. Crossman, M., and Cooke, J.E., Manual Control of Slow Response Systems, The Human Operator in Process Control, (Edwards, E., and Lees, F., eds.), London, Taylor and Francis, 1974. Sanderson, P.M., and Verhage, A.G., and Fuld, R.B., State Space and Verbal Protocol Methods for Studying the Human Operator in Process Control, in Ergonomics, Vol 32, No. 11, pp 1343 - 1372, 1989.

Acknowledgements: The author wishes to acknowledge a number of useful discussions with Prof. Peter Elzer of Clausthal University, Germany that helped to clarify the idea of expressiveness particularly in the process control domain. These discussions were made possible by sponsorship from the COPES-PROJECT of the EC, and took place during the summer of 1998.

Ecological Interface Design: Some Premises

John M. Flach Psychology Department, Wright State University, Dayton, OH 45435, USA e-mail:j.flach @desire.wright.edu

Abstract: This chapter presents three premises of an ecological approach to humanmachine systems. The first premise is that human-machine systems are dynamic, closed-loop systems that require a circular view of causality. The second premise is that the behaviour of these dynamic systems can best be understood in terms of the constraints in the functional workspace. These constraints include design intentions (e.g., functional goals), physical laws, organizational structure, and physical process and form. The final premise is that the explicit representation of the workspace constraints in the interface will greatly facilitate performance and will enhance the overall stability of the human-machine system.

Despite incredible advances in the development of automated control systems that are capable of closing many of the inner loops in complex work domains (e.g., energy production, advanced manufacturing, or aviation) human operators are ultimately responsible for controlling these work processes. That is, a human operator must monitor the system, compare the state of the system to normative expectations and functional objectives, and ultimately intervene in a way that will compensate for any deviations that are observed. At some level (more likely at multiple levels) the loop is closed through one or more human operators. Thus, stability of the system depends, in part, on the humans' ability to perceive deviations and to act appropriately to correct those deviations. Thus, whenever a system breaks down, it will almost always be possible to trace back and find that human actions were on the error path. That is, the human made an incorrect action, failed to detect a significant deviation, or failed to diagnose the deviation (i.e., correctly compensate for the deviation). Thus, it is tempting to identify human error as the "cause" in many accidents. However, since error trajectories are often unique, it is difficult, based on analysis of the time histories (causal trajectories) of these events, to draw general principles that will help in the design of safer systems. An ecological approach attempts to take a broader holistic view that looks beyond activities (behavioural trajectories) to consider the landscape (i.e., ecology) that shapes trajectories within a work domain. This chapter will consider some of the premises that motivate an ecological approach to the analysis of work domains and to the design of interfaces. The chapter is organized into three major sections. The first section considers the nature of the coupling between perception and action. The second section discusses the identification of constraints as a fundamental goal of analysis and as the semantic foundation for building interfaces. The third section discusses the specification of constraints within representations as a critical factor for skilled control.

126

Ecological Interface Design: Some Premises

1. Perception-Action Coupling Controlled action within complex work domains almost always depends on some degree of feedback. Feedback allows the system to monitor progress with respect to functional objectives and to minimize variations due to disturbances from the environment (e.g., a change in energy demands, a breakdown in equipment, or a change in wind direction). Thus, in complex work domains we are almost always dealing with closed-loop systems. This has several important implications for how we think about these systems. The first issue is the nature of explanation. For example, suppose you were trying to understand why an aircraft suddenly veered away from its normal landing approach path? You could say that this manoeuvre was caused by a particular force to the control stick that was translated into changes in the flight surfaces that altered the aerodynamic forces on the aircraft. You could dig deeper and say that the force on the control stick resulted from a particular pattern of muscle contractions, that in turn were caused by a pattern of electrochemical stimulation, etc. As you follow the causal path deeper into the system it is not clear that deeper insights with respect to the question of "why the aircraft veered suddenly" will result. This analysis helps us to understand "how" the manoeuvre was accomplished, but does not help to answer "why." On the other hand, we could ask about the pilot's intentions. She was trying to avoid collision with another aircraft that had taxied onto the runway. Now we are getting somewhere. Again, we can probe deeper and ask questions about why this manoeuvre and not another? (Is this a satisfactory response to the situation? Is it optimal?) Or we could ask why would she want to avoid a collision (that is what are the costs or values associated with collision compared to alternative outcomes). This line of questioning seems more productive and better suited to the question of why. For physical systems, we are most concerned with "how" the system works. How do the component forces sum to produce the behaviour? But for closed-loop control systems this is not enough. We must also be concerned with the question of "why the system behaves as it does?" This involves consideration of intentions (i.e., goals or purposes) and values (i.e., costs and benefits). This also involves consideration of information and feedback. That is, how can the physical behaviour be compared with goals and values in a way that will result in "error" signals that in turn can be translated into appropriate actions? Thus, we must be concerned about the information coupling that allows co-ordination between system behaviour, intentions, and control actions. This coupling involves both hard/force links (e.g., manipulation of a control stick) and soft/information links (e.g., the problem representation at the interface). Rasmussen [14], [15] provides an important discussion of the need to understand "how" and "why" and of the implications for work analysis. The key point to understand is that information (and the lack of information) is fundamental to system stability (or instability). A second issue, related to the nature of the perception-action coupling, is the very nature of causality. In "physical systems" the future is determined or caused by past

Ecological Interface Design: Some Premises

127

events. Thus, it is often possible to trace backward along behavioural trajectories to identify events in the past (e.g., forces, stimuli) that combine to determine behaviour of the physical system (e.g., motion, responses). However, this type of reasoning breaks down when analyzing closed-loop systems. For example, if a servo-mechanism becomes unstable, it is unlikely that the "cause" of the instability will be found by tracing back along the time history of the process to find an event or events that "caused" the instability. Stability in these systems depends on the coordination between information (feedback) and action. Instability is generally caused by global properties of the system (e.g., forward loop gain in relation to the effective time delay). Identifying these "causes" of instability requires a more global analysis of the system. The organization and the coupling of forces and information must be evaluated in terms of global constraints that generally are not evident from analyses of time histories alone. The key point here is that the linear causal reasoning that works well for the analysis of inanimate physical objects (e.g., the motion of planets) will not work as well for closed-loop control systems. Yes, control systems are deterministic systems. Yes, they are constrained by the same physical laws that constrain inanimate physical systems. However, closed-loop systems have a higher level of organization. This organization introduces global constraints on behaviour that require a different logic of causality --- a logic that is sensitive to the circular coupling of perception and action and the resulting ability to adapt to constraints in the environment [2], [3]. Thus, the ecological approach is a challenge to get beyond the logic of simple stimulus-response chains to examine the role of global (systemic or ecological) constraints in shaping system behaviour.

2. Analysis of Constraint The idea of constraint as a causal factor in determining system behaviour and thus as an object for scientific investigation is perhaps most explicitly shown in the emerging field of non-linear dynamics (e.g., chaos theory). Abraham and Shaw's [1] books illustrated the constraints that determine the behaviour of non-linear systems as three-dimensional landscapes. The hills (repellers) and basins (attractors) in these landscapes represent constraints that shape behavioural trajectories of the dynamical systems. Although Gibson's [6], [7] ecological approach to perception and action predates popular interest in non-linear dynamics, there are obvious parallels between the logic of his approach and the logic that is emerging from the field of non-linear dynamics [11], [12], [17]. This is most clearly seen in Gibson and Crook's [8] analysis of driving, where they introduced the construct of "safe field of travel." The "safe" regions represent basins of attraction and the objects of potential collision are repellers on the ecological landscape that shape driving behaviour. The term affordance was introduced as a construct that reflects the "attractive" or "repelling" nature of the behaviour ecology. The affordances are the opportunities for action in the ecology. The challenge for an

128

Ecological Interface Design: Some Premises

ecological approach is to understand how these basins of attraction and repelling regions (i.e., affordances) arise from the dynamics of perception and action. It is very important to differentiate between the concept of constraint and of state. The states of a system reflect the dimensionality of the process. In other words the states are the coordinates for all points on behavioural trajectories through the workspace. The landscapes (such as those depicted by Abraham and Shaw) show the impact of external constraints on the workspace, but the sources of the undulations in the landscape are not the state variables themselves, but constraints on those variables. For functional work domains these constraints include goals and values (as might be reflected in the cost functional of an optimal control problem). They include physical laws that constrain motion within the state space (as might be reflected in the process or plant dynamics in an optimal control analysis). Note that to determine the "optimal" control law --- the analyst must have an explicit model of both the physical laws (dynamic equations) and the value system (cost functional). Again, Gibson's ecological approach to perception attempted to go beyond pure "state" descriptions of stimulation to better capture the functional constraints that are very important factors when attempting to account for skilled adaptation (i.e., optimal control) to a task ecology. For simple systems (e.g., a laboratory tracking task) the physical laws are known, the goals are clearly specified (e.g., minimize RMS error), and we have good engineering heuristics for approximating the value systems (e.g., the quadratic cost functional). For these laboratory tasks, the task analysis has been done --- that allows application of analytic control models (e.g., the optimal control model) that can provide important intuitions about human and system performance. It is tempting to generalize to complex work domains using these analytical models that worked in the laboratory contexts. However, an ecological approach is a challenge to look past these simple analytic models to the task analyses that went into their construction. The ecological approach challenges researchers to analyze the complex work domains with the same rigor that was given to the analysis of the laboratory tasks. That is, develop valid models of the work dynamics, identify the functional goals and the values or figures of merit that reflect the cost/benefit tradeoffs associated with system behaviour and control action. The expectation is that the "models" for the complex domains may be very different than the models that worked for laboratory tasks. These models will reflect the particularities of the complex work domains. Thus, the ecological approach takes advantage of the analytic toolbox of systems engineering for developing models of the work ecology, but it is sceptical about generalizations based on analytic models derived from simple laboratory task ecologies. A good example is the work on arm protheses that was done by Stassen's group at Delft [16]. The analysis of the prothesis problem included detailed modelling of the physical constraints, but also intensive field investigations to study the ecology of use. Lunteren and Lunteren-Gerritsen [13] describe one of the field analyses:

Ecological Interface Design: Some Premises

129

9 . . an observation method was chosen in which a child with a prothesis is followed over a normal school day from the moment he or she starts dressing until going to bed in the evening. If possible, a day is planned that includes both a physical exercise class and a class involving manual activities. During the day all actions of the child, with or without a prothesis are observed and the ways in which they are executed are noted. This means that the list o f activities is not standardized, which makes a comparison more difficult. However, the only question being asked was f o r what activities do the children use their prosthesis. Moreover, it was felt that in a standardized test situation the child might try to do things differently from his or her normal pattern, i.e., in a way that was thought to please the observer. Besides the functions that were used for a certain action, it was also observed whether the way an action was executed was unobtrusive or involved some unnatural-looking motion (the cosmesis o f using). The way the child moved while not performing any action with the prothesis (the cosmesis of wearing) was also observed (p. 99 - 100)

The work of the Delft group illustrates a commitment to understanding the ecology of use. As opposed to a more common practice were the commitment is to a particular "model," and those aspects of the ecology that don't conform to the model are ignored. Note that a commitment to understanding the ecology does not require throwing away models. Rigorous analytic control models also played an important role in the Delft work. It is simply a matter of priority. Analytic models can be seductive, so that the task analysis reduces to an identification of those aspects of the work domain that correspond to model parameters. An ecological approach gives priority to understanding the domain constraints. This understanding becomes the context for intelligent application of analytic models. Another clear example where priority was given to understanding the task ecology is Hutchins' [10] analysis of ship navigation. While neither Stassen nor Hutchins would typically be identified with an ecological approach --- the commitment to understanding the natural constraints within work domains that their work illustrates is fundamental to an ecological approach. There are important qualitative differences among the kinds of constraints within a work domain that can have important implications for work analyses. These differences, in part, reflect the underlying nature of causality (i.e., the distinction between "how" and "why" discussed in the previous section). For example, the intentional constraints (functional purpose or design purpose) are qualitatively different than the physical constraints (natural laws). Yet, both kinds of constraints interact to determine the underlying topology of the work space. The qualitative differences have implications for both the level of abstraction and the level of decomposition that are most appropriate for analysis and design. Rasmussen [14], [15], [18], [20] has written extensively about these implications and has introduced the Abstraction Hierarchy as a framework that has been useful for analyses of many different work domains. Vicente has utilized the Abstraction Hierarchy very

130

Ecological Interface Design: Some Premises

effectively for the development of graphical interfaces [19] and for analysis of expertise [21]. The key point for this section is that an ecological approach includes the constraints within the work ecology as fundamental targets for analysis. This is consistent with control theoretic and general systems approaches to analysis (particularly evident in the analysis of non-linear systems). However, this insight has often been lost as researchers have tended to focus on the analytic models from control theoretic analyses of laboratory tasks, rather than on the standard practices of control analysis.

3. Degree of Specification In the previous section, the argument that constraints are fundamental to understanding work domains was introduced. This is true from the point of view of the analyst and from the point of view of the operator. For operators to perform skillfully in complex work domains there must be information that allows them to recognize the best opportunities for action. Classically, the recognition problem has been modelled as a computational problem. That is, cognitive psychology has generally assumed that the opportunities for action are underspecified and that logical inferential processes must interpret the sparse information to make informed guesses about the situation. For example, it seemed obvious that a two-dimensional retina could at best only provide hints or cues about the true three-dimensional structure of ecological space. Therefore, it has been assumed that there were powerful information processing systems in the visual system that allows animals to infer the three-dimensional structure from the cues provided by our sensory system. Consistent with this view of human information processing, operator support systems have been designed to either support the inferential process (help the operators to integrate information according to logical rules) or to replace the human (who has a limited capacity for following the dictates of logic) with an automated (expert) system with more powerful and reIiable computational engines. Gibson's theory of direct perception challenged the general assumption that the opportunities for action were always underspecified. For example, he argued that the dynamic optical array (structural properties of reflected light available to a moving observation point) provided direct specification of the three-dimensional ecology in a way that supports many skilful interactions (e.g., control of locomotion). He argued that it was "attunement" to the structure in the optical array (not computational inferences) that allowed skilled interaction (e.g., control of locomotion). The idea that structure in the optic array allowed direct closed-loop control of action is clearly expressed in Gibson's [5] description of visual control of locomotion.

Ecological Interface Design: Some Premises

131

The center of the flow pattern during forward movement of the animal is the direction of movement. More exactly, the part of the structure of the array from which the flow radiates corresponds to that part of the solid environment toward which he is moving. If the direction of movement changes, the center of flow shifts across the array, that is, the flow becomes centered on another element of the array corresponding to another part of the solid environment. The animal can thus, as we would say, "see where he is g o i n g . " . . . To aim locomotion at an object is to keep the center of flow of the optic array as close as possible to the form which the object projects. (p. 155) The construct of Ecological Interface Design (EID) is in part an extrapolation of Gibson's observations about direct perception to the design of graphical interfaces. The challenge is to design representations that provide as direct a coupling between perception and action as possible. The idea is to allow operators to "see where they are going" as they navigate over the functional landscape of the work domain. Again, this does not simply mean that the "states" of the process are represented, but that the whole landscape is made visible (i.e., specified). That is, the operators must be able to see the states in relation to the functional constraints. The representation must reflect the functional goals and values, the physical laws, the organizational constraints, and the physical process constraints that shape action in the workspace. Vicente's work on the DURESS interface is the most systematic and most thoroughly tested and documented application of Ecological Interface Design. A key feature of the DURESS interface is that the constraints on mass and energy balance are explicitly represented in the interface. Tanabe [17], [22] also has an impressive program to apply Ecological Design principles in the development and evaluation of the next generation control room for the nuclear power domain. However, a simpler example from our work in the aviation domain will be used to illustrate the representation of work constraints within an interface. Controlled flight into terrain (CFIT) is a significant problem for high performance military aircraft [9]. CFIT is a situation where an aircraft collides with the ground where there are no obvious mechanical failures, no medical problems, or no unusual weather conditions that might cause the accident. Thus, CFIT appears to be a clear example of operator error. We are currently preparing to test both a hypothesis about how structural properties of optical flow can be a potential contributing factor to these accidents and a display modification to help prevent these accidents. The hypothesis concerns an ambiguity in optical flow fields. As an observer moves through the environment the light reflected from texture surfaces streams by. The faster the observer moves the faster the texture flows by. However, the rate of flow also depends on the distances to the surfaces. The greater the distance the slower is the flow. Thus, the global optical flow rate is directly proportional to observer speed and inversely proportional to distance to the surface. For an observer who is at a constant distance from the ground, the optical flow can provide reliable information about speed of locomotion. However, when altitude and speed are both

132

Ecological Interface Design: Some Premises

varying, global optical flow rate is no longer specific to speed. Thus, a potentially dangerous situation is created when an aircraft simultaneously loses significant altitude and airspeed. The loss of altitude causes an increase in optical flow rate that may mask the loss of airspeed. Loss of airspeed is critical since without adequate airspeed the aircraft looses lift (i.e., stalls). When the pilot pulls up to avoid collision with the ground there is not adequate airspeed and the aircraft stalls and crashes.

Figure 1: The traditional attitude display shows pitch and roll as the relative position of a fixed aircraft icon and a moving horizon. The modification above includes a textured surface with lines that f/ow in a manner similar to stripes on a highway trader a moving vehicle. The rate of flow is proportional to the difference between current air speed and the minimum air speed for controUed flight (i.e., the stall boundary). In most military aircraft, airspeed is displayed either on a linear tape, a rotary dial, or as a digital value. It is not meaningfully configured with any other flight information. A pilot who is confident that airspeed is adequate (due to high optical flow rate) may not consult this instrument and thus may not realize that the aircraft is approaching a stall speed. To help prevent this we have designed a modification to the primary flight display that integrates an indication of airspeed with information about aircraft attitude. Figure 1 shows this primary flight display. The traditional display shows aircraft attitude as the relative position of an aircraft symbol and an indication of the horizon. The standard format in the US is for the plane to be fixed and for the horizon to move in a way that is consistent to the orientation of the aircraft with the horizon. When an aircraft lowers the right wing (banks right). The horizon rotates counter clockwise (banks to the left) until the angle between the plane symbol and virtual horizon matches the orientation of the aircraft and the real horizon. When the aircraft pitches up, the virtual horizon will move down in the display. This is called an inside-out display and was designed to mimic the view through a periscope that has a fixed orientation to the aircraft frame. We have modified the standard flight display to include texture lines on the virtual ground portion of the representation. These line flow downward similar to stripes on a highway flowing

Ecological Interface Design: Some Premises

133

under a vehicle. The flow rate is proportional to the difference between the current flight speed and the minimum speed need to maintain flight (i.e., the stall speed). Thus, the rate of flow is proportional to distance from the stall boundary. We think that there are two principal advantages of this modification. First, by configuring the airspeed with attitude information we hope to make the dynamic interactions among the flight dimensions more readily visible. That is, one potential cause of simultaneous loss of altitude and air speed is unusual orientation of the aircraft (e.g., high degree of banking). Thus, by integrating air speed with the attitude information we hope that pilots will be better able to recognize this interaction. Also, since this is the primary flight display, it is hoped that this will increase the probability that indications of air speed will be within the attentional field of the pilot. That is, we believe the change in flow rate within the primary flight display will be more salient than a change of position on a dial or a change in a digital value. The second advantage is that the flow rate is anchored to the stall speed. That is, the rate of speed is directly proportional to distance from the stall boundary. Thus, the relation between the state variable speed is represented in the context of the aerodynamic constraints that determine lift. The hope is that this will allow the pilot to "see" approach to the stall boundary (as opposed to having to infer this based on knowledge of aerodynamics). Certainly, this is not a difficult inference for skilled pilots. However, unless they recognize that they are approaching the stall boundary, there is no reason for them to sample the air speed. Again, by explicitly modifying the representation to reflect stall constraints this information is made more salient and thus it is more likely to contribute to the pilots' situation awareness. This is one of several modifications that are being considered to improve the interface representations available to pilots [4]. The key point is that providing "feedback" to the operators about state variables is insufficient to insure stable control. The state variables must be represented in the context of the work constraints. These constraints can be integrated within computational engines (e.g., expert systems) or they can be integrated within graphical representations. The ecological approach is most concerned with integrating the constraints into graphical representations. We think that this is the most effective way to utilize the unique skills that a human operator brings to the work domain. We fear that if the constraints are buried within computational engines, then the logic of these automated systems will be opaque to the human operator and thus it will be difficult for the human to trust the expert system. Again, it should be clear that there is an important role for expert systems in complex work domains. However, we think that coordination between these expert systems and human operators will be improved if the internal constraints that drives the computations are made explicit in the interface between human and automated system.

134

Ecological Interface Design: Some Premises

4. Summary and Conclusions In sum, an ecological approach starts with the recognition that the study of human performance in natural work domains is the study of dynamical control systems that are governed by a circular causality. This class of systems can not be understood based on an analysis of state trajectories alone. A broader analysis of the constraint topology (i.e., the ecology) is required. This analysis requires descriptions at multiple levels of abstraction and decomposition. Ecological Interface Design attempts to explicitly incorporate the designer's understanding of the constraint topology within a graphical representation. The goal is to build a representation where the states are represented together with the domain constraints so that the operators can directly "see" the functional implications of any action or event.

References [1] [2] [3] [4]

[5]

[6] [7] [8] [9] [10] [11] [12] [13]

Abraham, R.H. & Shaw, C..D. (1984). Dynamics: The geometry of behavior. Three volumes. Santa Cruz: Aerial Press. Flach, J.M. (1990). Control with an eye for perception: Precursors to an active psychophysics. Ecological Psychology, 2, 83 - 111. Flach, J.M. (1999a). Beyond error: The language of coordination and stability. In Hancock, P.A. (ed.). Human performance and ergonomics. San Diego: Academic Press. Flach, J. M. (1999b). Ready, fire, aim: A "Meaning-processing" approach to display design. In D. Gopher and A. Koriat (Eds.) Attention and Performance XVII: Cognitive regulation of performance: Interaction of theory and application. (197 - 221). Cambridge, MA: MIT Press. Gibson, J.J. (1955/1982). Visually controlled locomotion and visual orientation in animals. British Journal of Psychology, 49, 182 - 194). Also in E. Reed & R. Jones (Eds.) Reasons for realism. Hillsdale, NJ:Eflbaum. Gibson, J.J. (1966). The senses considered as perceptual systems. Boston: HoughtonMifflin. Gibson, J.J. (1979). The ecological approach to visual perception. Boston, MA: Houghton-Mifflin. Gibson, J.J. & Crooks, L.E. (1938/1982). A theoretical field analysis of driving. American Journal of Psychology, 51, 453-471. Also in E. Reed & R. Jones (Eds.) Reasons for realism. Hillsdale, NJ:Erlbaum. Haber, R.N. (1987). Why low-flying fighter planes crash: Perceptual and attentional factors in collisions with the ground. Human Factors, 29, 519 - 532. Hutchins, E. (1995). Cognition in the wild. Cambridge, MA: MIT Press. Kelso, (1995). Dynamic patterns: The self-organization of brain and behavior. Cambridge, MA: M1T Press. Kugler, P.N. & Turvey, M.T. (1987). Information, natural law, and the self-assembly of rhythmic movement. Hillsdale, NJ: Erlbanm. Lunteren, T. van, & Lunteren-Gerritsen, E. van. (1997). In search of design specifications for arm prostheses. In T.B. Sheridan & T. van Lunteren (Eds.) Perspectives on the human controller. Mahwah, NJ: Erlbaum.

Ecological Interface Design: Some Premises

135

[14] Rasmussen, J. (1986). Information Processing and Human-Machine Interaction: An Approach to Cognitive Engineering. New York: North Holland. [15] Rasmussen, J., Pejtersen, A.M., & Goodstein, L.P. (1994). Cognitive Systems Engineering. New York: Wiley. [16] Sheridan, T.B. & van Lunteren, T. (Eds.) (1997). Perspectives on the human controller. Mahwah, NJ: Erlbaum. [17] Tanabe, F., Yamaguchi, Y. & Rasmussen, J. (1998). Simulator experiments with ecological interface systems. JAERI Research Report. Japan Atomic Energy Research Institute. Thelen, E. & Smith (19). [18] Vicente, K.J. (1999). Cognitive work analysis: Towards safe, productive, and healthy computer-based work. Mahwah, NJ: Erlbaum. [19] Vicente, K.J. (1992). Memory recall in a process control system: A measure of expertise and display effectiveness. Memory & Cognition, 20, 356 - 373. [20] Vicente, K.J. & Rasmussen, J. (1990). The ecology of human-machine systems II: Mediating 'direct perception' in complex work domains. Ecological Psychology, 2, 207 - 250. [21] Vicente, K.J. & Wang, J.H. (1998). An ecological theory of expertise effects in memory recall. Psychological Review, 105, 33 - 57. [22] Yamaguchi, Y. Furukawa, H., & Tanabe, F. (1998). Design of subject training on reactor simulator and feasibility study: Toward an empirical evaluation of interface design concept. Paper presented at the Enlarged Halden Program Meeting, Lillehammer, Norway. Acknowledgments: Sincere thanks to Peter Elzer and Badi Boussoffara for the invitation to participate in the international workshop on "Human Error and System Design & Management." They were very kind and generous hosts. John Flach was supported by grants from the Japan Atomic Energy Research Institute (JAERI) and from the Air Force Office of Scientific Research (AFOSR) during preparation of this manuscript. However, John is solely responsible for the views expressed in this chapter.

Ecological Interface Design (EID) and the Management of Large Numbers of Intelligent Agents

John D. Lee, Thomas Geb, Emily Pollack The University of Iowa, 4135 Seamans Center, Industrial Engineering, Iowa City, Iowa, USA e-mail: jdlee @engineering.uiowa.edu

Abstract: Agent-based automation is an emerging technology that presents new

challenges if humans are to manage it effectively. Current automation has been plagued with mode errors, misuse and disuse. The properties of agent-based automation that make it most useful, those of an adaptive self-organizing system may lead to a poor calibration of operators' trust in the automation and a magnification of automation-related problems. The focus of ecological interface design to graphically display the underlying system constraints offers a promising design strategy to promote effective use of agent-based automation.

1. The Emergence of Multi-agent Automation Technology has dramatically changed the role of the human in complex systems, with the human operator's role having evolved from being primarily physical to primarily cognitive. Frequently, automation has been designed to perform single, specific functions and has limited interactions with other elements of automation. Future automation may not be so simple and may introduce new human factors challenges. For example, technological advancement may enable automation to evolve into agents that may become increasingly powerful, autonomous, and interconnected. Agent-based automation offers increased robustness, flexibility, and adaptability; it will be able to adapt and respond to many types of environments. However, understanding how to support the human supervisory control of these agents remains an unresolved issue. A concrete example of this evolution is planetary exploration. The current approach to remote planetary exploration mirrors the current application of many types of advanced automation. In the case of the Mars missions, this involves a single rover. The design of the rovers has been directed primarily towards building single,

138

Ell) and the Management of Large Numbers of IntelligentAgents

complex device. Designers develop the mechanical structure and software to accomplish specific tasks, such navigating a rock-strewn field. The primary limitation of this approach is that the entire mission has a high chance of failure if the demands of the mission depart from the initial assumptions due to unexpected terrain conditions or a rover malfunction. This lack of robustness in the face of unanticipated variability is an important limitation. The concept of multi-agent automation provides an alternate design paradigm that provides substantial redundancy that may make it possible to respond to environment variability while reducing the chance of system failure. Although this emerging design alternative offers substantial benefits, it is not clear how designers should best support the cognitive demands associated with managing agents. Figure 1 shows some of the complex display and control interactions that arise when a human must manage an agent-based automation. Humans may monitor and control individual agents, shown by the thin lines, or they may need to monitor and control groups of robots, shown by the wide arrows.

Figure 1: Many important challenges exist in supporting an effective and synergistic relationship between humans and agents.

1.1. Using multiple agents for effective system control Agent-based automation is an emergent technology that is likely to play an important role in many areas extending from power generation to space exploration. For example, spacecraft and Mars habitats are likely to have multiple agents controlling power, propulsion, navigation, communication, life support, and maintenance. Each of these agents will interact with and direct the actions of suites of more specialized agents. For example, the power system may have different agents monitoring solar panel alignment, fuel cells, power usage, and power generation. These agents may themselves direct multiple agents. In addition, software agents may control scientific experiments and apparatus on the spacecraft. Hundreds of agents may work simultaneously to ensure the safety and success of the mission.

ElL) and the Management of Large Numbers of IntelligentAgents

139

Agent-based automation may be distributed over a large area, enabling multiple jobs or data collection to be done simultaneously. This distributed control system may also be able to quickly adapt to new assignments and environments. This flexibility also allows an operator to redirect them easily . While agents are adaptive, they may also be only partially predictable because they may combine into a self-organizing system. They may interact with each other and the environment to produce emergent behaviours that may not have been foreseen. Specifically, self-organizing systems generate structure at a global level that grows out of interactions of lower level components. Insect colonies frequently exhibit self-organizing behaviour to achieve complex goals. Critical features of selforganizing systems are positive feedback, negative feedback, randomness, and multiple interactions [4]. The characteristics of self-organizing systems could provide great benefits if the supervisory role of the human controller could be effectively supported. Some have suggested an extreme case where agents autonomously control the system and people have no interaction with the agents. A more promising approach assumes a collaborative relationship between many agents and the human supervisor, whereby the human uses the agents as a tool to extend his or her capabilities. The problem is particularly interesting because the emergent behaviour of the multiple agents may not be directly inferred from the behaviour of the individual agent. This presents a substantial challenge to the design of an interface if it is to support effective management of the agent-based automation. 1.2. Example: Swarms of insect robots An important alternative to the current approach single-rover approach to planetary exploration is to engage many simple rovers in exploration. Robot swarms are large collections of small, relatively simple robots that are controlled using the concepts of agent-based automation. In a robot swarm, each robot is a semi-autonomous agent. The swarm concept is modelled after insects such as bees, ants, and termites that demonstrate robust and adaptive behaviour in a wide range of natural environments. Much inspiration has been drawn from the study of these social insects, and the findings associated with these studies have important implications for the design of swarms of insect robots. The success of insects suggests that robot swarms have a tremendous potential to amplify human capabilities in adverse and dynamically changing environments. In a natural environment, such as the pictured lunar surface, the swarm provides multiple viewpoints, distributed data collection, robust performance in the face of mechanical failures, and the ability to adapt to the complex terrain. These and other important features of swarms may be harnessed as a general tool to improve the safety and effectiveness of space and planetary surface missions; however, techniques must be developed to command and control swarm behaviour effectively. A burgeoning community of biology-inspired roboticists is actively exploring the possibilities of employing swarms of insect robots as an alternative to relying on one or two larger robots to assist astronauts in their missions [6], [11]. The insect

140

EID and the Management of Large Numbers of Intelligent Agents

robot concept is based on the notion that small machines with simple reactive behaviours, possibly adaptive or learning intelligence, can perform important functions more reliably and with lower power and mass requirements than can larger robots [3], [5], [10]. Typically, the simple programs running on the insect robot are designed to elicit desirable emergent behaviours in the insect swarm [2]. For example, a large group of small robots might be programmed to search for concentrations of particular mineral deposits by building upon the foraging algorithms of honeybees or ants. Other useful tasks that could be performed by insect robots are machinery inspection, farming, delivering materials, digging, and building. Organization and control of swarm behaviour stems from fundamentally different sources compared to traditional systems currently being monitored and controlled by people. A specific example of the robust and effective group behaviour that selforganizing swarms generate is the foraging behaviour of ants. The effectiveness of this behaviour could easily be generalized to information foraging of robot swarms as they explore a planet's surface, or to collection of contaminants on a spacecraft. Ant foraging involves a trade-off of speed of trail establishment and search thoroughness, a trail that is more quickly established will sacrifice the thoroughness of the search. Parameters that govern this trade-off include the strength of the positive feedback, and the amount of random variation [27]. Alternate means of control include template following (natural gradients/fields), and differential sensitivity of members to control signal. These control mechanisms offer great potential in extending human capabilities, but only if a thorough empirical and analytic investigation identifies the display requirements, viable control mechanisms, and range of swarm dynamics that can be comprehended and controlled by humans. A specific mechanism that underlies the self-organizing behaviour of swarms is stimergy communication, which allows insects to communicate through a dynamically evolving structure. Stimergy is a powerful alternative to a static set of instructions that specify a sequence of activity. Through stimergy, social insects communicate directly through the products of their work. This sort of communication promotes the swarm to evolve into a self-organizing system that can generate many forms of collective behaviour with individuals producing the same behaviour. Interaction between many simple individuals produces complex behaviour for the group [4]. For example, as a self-organizing system, the swarm could dynamically adjust its information or material foraging behaviour to a dynamic environment to accomplish its goals effectively. The performance of the swarm does not depend on the performance of an individual robot. This characteristic of swarm robots, in contrast to conventional automation, has important implications for the control of individual robots and the control of the overall swarm. Understanding the nature of the control mechanism has important implications for the human role in monitoring and controlling swarms. Seeley [25] argues that honey bees adapt to a changing environment at the level of the' group. A colony of

EID and the Management of Large Numbers of lntelligentAgents

141

honey bees functions as a large, diffuse, amoeboid entity that can extend over great distances and simultaneously tap a vast array of food sources [25]. The importance of the emergent behaviour of the swarm implies that the human must receive feedback from, monitor, and control the swarm and not just the individuals. Information and control at the level of individual agents is not likely to prove successful. New display and control paradigms may be required to discover how a human can effectively exert control over a swarm of individual robots. Organization and behaviour of social insects, suggests that control of agent-based automation may be fundamentally different as compared to traditional systems currently being monitored and controlled by people. The specific demands that these most sophisticated systems may impose on operators have not been explored. Failing to understand the cognitive consequences associated with these increasingly sophisticated supervisory control situations may negate any technical benefit they provide.

2. H u m a n Factors Issues of Multi-agent A u t o m a t i o n 2.1. Supervisory control and the new challenges of multi-agent automation Human supervision of multi-agent automation will become increasingly important. As more robotic and software agents are employed to extend the capabilities of humans, the operator's ability to successfully monitor and respond to unexpected events and agent behaviours will be severely tested. Agent-based automation represents a new type of automation that encompasses the human factors challenges witnessed with current automation and introduces new challenges. To be most effective, supervisory control of multiple agents would allow the operator to understand agent behaviours, predict future activity, and maintain control over the agents. In order to understand agent behaviour, the operator needs to identify how an agent's future state is related to its current state. Since this relationship may vary from agent to agent and over time, this information must be easy to assimilate. To predict the future state of the system the operator must be able to estimate the current state of the system and estimate the collective behaviour of the agents. To control the system, the operator must be able to redirect behaviour of either individual agents or groups of agents. To realize the potential of large numbers of interacting agents, agent-based automation must consider the design of the interaction of the agents, their interactions with a human supervisor, and the design of the human-computer interface. Agent behaviour that is counterintuitive, overly complex, or unpredictable may inhibit effective control. To create behaviour that can be effectively controlled requires an understanding of how system complexity is interpreted by people. New

142

EID and the Management of Large Numbers of Intelligent Agents

analytic techniques may be necessary in order to identify human-centered design constraints on the behaviour of increasingly complex automation. Figure 2 shows a series of simple diagrams that illustrate some of the important changes in the relationship between the operator and technology. Each diagram represents an alternate supervisory control situation that is likely to have important implications for the control of multi-agent automation.

Figure 2: Possible methods for use in human supervisory control of multi-agent automation. a) The technology fills a relatively simple role of augmenting the humans perception and control. The operator requests a specific behaviour and the technology responds with a deterministic response. In this situation significant human performance issues emerge as the operator is forced to bridge the gaps between functionally isolated technological aids [ 1], [ 17], [ 18]. b) Here we have a more sophisticated sort of automation, which changes modes in response to the human operator, to other elements of automation, and to the environment. This sort of automation can extend operator capabilities in important ways; however, extensive literature on mode error illustrates the potential for inducing errors and degrading system safety and performance [8], [23], [34]. Providing information regarding both the domain task and the behaviours of the intelligent agent emerge as critical needs [ 19]. c) This scenario shows a qualitative change in supervisory control associated with an increasing number and sophistication of the agents. Because there are many interacting intelligent agents, macro-level behaviour may emerge that cannot be easily predicted by the behaviour of individual agents. The clouds show this macrolevel behaviour. In these situations, mode transitions may surprise the operator even if the operator understands how an agent responds to environmental changes. The operator is unlikely to successfully control this configuration of automation. While this added complexity can undermine the operator's understanding and monitoring efficiency, it supports a much more powerful and adaptive system. The operator is now responsible for managing the overall behaviour of a group of semiautonomous entities that interact in a complex and non-deterministic manner. This group may exhibit properties of a self-organizing system that could have very beneficial consequences, but they may also confuse the operator and lead to inefficient monitoring and management.

EID and the Management of Large Numbers of IntelligentAgents

143

2.2. Complex, counterintuitive agent interactions Multi-agent automation confronts humans with challenges not seen in current forms of automation. Mode errors, misuse, and disuse of automation could dramatically increase with poorly designed multi-agent automation. The factors that contribute to mode errors with current automation include indirect mode changes, inadequate feedback, and inconsistent behaviour [23]. The emergent behaviour of multi-agent automation may exacerbate all of these factors. The fundamental challenge is that multiple agents may interact to produce emergent behaviour that is not an intuitive extension of the behaviour of the individual agents. This emergent behaviour may be a very useful characteristic if it is properly designed and managed. For example, multiple agents could adapt their behaviour in novel ways to preserve overall system performance. The difficulty is trying to anticipate the emergent behaviour that multiple interacting agents may exhibit. Emergent behaviour can provide substantial benefits if it can be focussed on adaptive, goal oriented directions; however, emergent behaviour can also lead to emergent errors. These errors can pose a serious challenge to system performance and safety. Some important emergent error types include:

Loose Cannon - This is when an individual agent assumes an inappropriate influence over one or more subsystems. For example, if the agent monitoring the power reserves causes life support to shut down, the power reserves agent has become a loose cannon. Civil W a r - This occurs when two agents or groups of agents compete for the same resource, causing poor performance. If a laser range-finder on a Mars vehicle wants to steer left to avoid an obstacle, the vision system wants to steer right, and the wheels either oscillate or steer towards center, the range-finder and vision system are in civil war. The interaction among agents may occur directly, through software, indirectly, or through the environment. These interactions may result in Agent-

induced Instabilities. Software Causality - This occurs when an agent silently shuts down, or because of adaptation, it no longer influences the behaviour of the system. When the agent responsible for maintaining power reserves is overridden for immediate needs so often that the power distribution agent adapts for this strategy, the reserves agent may lose its priority and quietly allow the batteries to be depleted. Emergent behaviour of multiple agents introduces the potential for Macro-level and micro-level mode confusion. Attention and understanding of behaviour modes must be distributed between the macro (group behaviour) and micro (individual agent behaviour) levels. An improper balance of focus on one level may allow problems to go undetected on the other. The operator must be able to intervene effectively to stop problems before they become serious. This requires that the operator has

144

EID and the Management of Large Numbers of IntelligentAgents

knowledge of the "normal," expected deviations for both individual and group behaviour. One advantage of agents, as opposed to integrated systems, is that they may be independently designed and tested. However, exhaustive testing, with either integrated systems or agents, is often a serious concern with automation. Testing for numerous failure modes in particular can become prohibitively expensive. When agents are combined, testing becomes exponentially more complex. Agents may interact directly or indirectly through the systems they control, which creates the possibility of a huge variety of errors that are difficult to predict and prevent. Designing interfaces that support effective agent management, particularly in unanticipated situations, may be the most effective means of combating the inevitable failings of multi-agent automation. An operator must be able to understand whether or not agents are acting productively so that he or she can proactively intervene and guide the agents to more productive behaviours.

2.3. Trust in Multi-agent Automation Appropriate reliance is a critical factor in the success of complex, multi-agent automation. Several researchers have suggested that operators' trust in automation may play a major role in guiding this collaborative relationship [12], [15], [16], [21], [26], [32]. Just as the relationships between humans are influenced by trust, so trust may mediate the relationship between the astronaut and the robot swarm. Research has shown that highly trusted automation may be used frequently, whereas operators may choose to control the system manually rather than engage automation they distrust [ 15]. Review of these studies allows the development of a multidimensional definition of trust including four dimensions: foundation of trust, performance, process, and purpose.

Foundation of trust represents the fundamental assumptions of natural and social order that make the other levels of trust possible. Performance rests on the expectation of consistent, stable, and desirable performance or behaviour. Having knowledge of current and historical performance results, the user expects similar ones in the future. This dimension corresponds to a user's perception of reliability or efficiency, having the automation produce results that the user expects. Process corresponds to a user's understanding of how the underlying qualities or characteristics that govern the performance of the automation match the current task demands. Understanding how the automation, algorithms, and control limits operate.

EID and the Management of Large Numbers of IntelligentAgents

145

Purpose rests

on the user's perception of underlying motives or intention, such as why the automation was developed. With humans this might represent motivations and responsibilities. With machines, purpose reflects the user's perception of the designer's intention in creating the system. These dimensions identify general information with which a person calibrates their trust [15]. Ideally, trust in automation is calibrated at a high level of resolution so that people trust the automation when it achieves their goals and do not trust it when the automation cannot achieve their goals. It is difficult to predict the reaction human managers will have to a self-organizing group of agents. It is possible that certain emergent behaviours will induce poor calibration of trust. The consideration of different dimensions of trust and the role of trust on reliance suggests that the interface must be carefully structured to help the operator understand the control system, performance of the agents, and the factors guiding their performance. An effective interface would help calibrate trust by enhancing helping the understanding of emergent behaviours that otherwise might lead the operator to distrust the agent-based automation.

3. Supporting Effective Management of Multi-agent Automation with Ecological Interface Design (EID) Effective management of agent-based automation requires a clear understanding of its behaviour and a well-calibrated level of trust in its capabilities. Excessive trust may lead to misuse and inadequate trust may lead to inappropriate interventions. It may be possible to maintain a well-calibrated level of trust if the human-computer interface reveals the structure and constraints of the system and the agent-based automation. Ecological Interface Design (EID) provides a theoretical basis for interface design that may encourage highly calibrated levels of trust.

3.1. Componentsof EID The essence of the EID philosophy can be summarized by two objectives. First, EID defines the information content required to control the system under normal and abnormal situations. Second, EID defines the form of the information (e.g., digital readouts, bar charts, or integrated objects into which information is embedded) so that it is consistent with the information processing capabilities of the operator. Thus, EID provides a means of identifying the critical information requirements of a good interface and guides the designer in selecting appropriate visual forms. Although EID has not been applied to the displaying of automationrelated information, it is a promising and tractable extension.

146

Ell) and the Management of Large Numbers of Intelligent Agents SystemCharacleristics

Ecological Interface Design

Abslraetion 1 Hierarchy Interface Content Ecological Interface Interface Form 'Skill,rule, and I knowledge-basndI behavior I

J

OperatorCharacterislies

Figure 3: The process of ecological interface design. Figure 3 shows how EID is used to identify the interface content and form and how they combine into the interface design. An important aspect of EID is the cognitive task analysis approach that identifies the physical and functional properties of a system that must be displayed to the operator. These properties are defined by the abstraction hierarchy, which is a description of system constraints at multiple levels of abstraction and detail. Because the abstraction hierarchy describes the system in terms that are consistent with humans natural parsing or chunking of a domain, it offers a powerful technique to describe the information required for system control. Identifying these properties is important because they specify the information required for routine operations as well as unanticipated situations. This type of analysis is particularly important for managing a self-organizing system, where particular behaviour may be impossible to predict, but functional relationships may be possible to specify and display. As a consequence, EID provides a very useful theoretical framework for examining agent-based automation because it defines important functional relationships. Because EID identifies the information requirements of an interface through an analysis of the system goals and the operating environment it has particular promise in helping people control multiagent systems that dynamically adapt to the environment. Even if relevant system constraints are correctly identified, a challenge remains in displaying the large volume of complex information to the operator and to providing intuitive mechanisms for the operator to monitor and command the agents. The skill-, rule-, and knowledge-based distinctions are fundamental characteristics of human performance and they help specify the form of the interface. Skill-based performance governs processes such as patter matching and movement co-ordination, while rule-based performance governs the application of rules-of-thumb and procedures. Knowledge-based performance governs problem solving and generating response to novel situations. Unforeseen situations require the flexibility of knowledge-based behaviour, but knowledge-based behaviour is more demanding than rule or skill-based behaviour. Therefore, the form of the

EID and the Management of Large Numbers of lntelligentAgents

147

interface should support all three levels of cognitive control, without pushing cognitive control to a higher level than is required [29]. This requirement guides interface designers to represent data graphically, making important functional relationships directly visible in the interface. 3.2. Benefits of EID

Several experimental investigations have shown the practical benefits of EID. These investigations demonstrate that ecological interfaces support superior decision making and fault diagnosis, when compared to traditional approaches to interface design [7], [31], [14]. In addition to better diagnosis, experiments have shown that ecological interfaces lead to a better understanding of the system and more efficient control strategies. For example, an ecological interface for monitoring heart activity led to more effective treatment with fewer drug administrations [9]. EID provides a proven foundation that can be built upon to address the challenges of describing the cognitive tasks and demands associated with supervisory control of agents. Another benefit of EID may be that the appropriate use of software agents improves system safety and reliability by providing highly redundant adaptive systems. However, without proper consideration of the human controller, agent-based automation may generate behaviour that is extremely difficult for the human to understand and manage effectively. Various design alternatives may need to be considered to optimize diagnosis performance for a suite of potential, difficult to control emergent errors. Figure 4 contrasts a traditional interface with one based on the principles of ecological interface design. The dramatic differences show the power of EID to generate novel interface concepts. The ecological interface provides operators with a performance advantage by integrating the otherwise disparate data into a meaningful object display. By overlaying that object on the saturation curve for water the individual data points are given meaning in terms of safety critical system states [2], [30]. Cognitive task analysis and display design components of EID will identify an effective human-centered approach to supervisory control of agentbased automation.

148

Ell) and the Management of Large Numbers of Intelligent Agents

/

err ~

PI-I~II W

l

/

I

I

I

/

o,o

1oi

................. _'?_'?'~_~~ '_'~..................................................................................

+.-~._.

"1I

Figure 4: A standard interface and one based on the principles of ecological interface design.

4. Conclusion Agent-based automation is an emerging technology that epitomizes the increasingly complex automation that faces human supervisors. A review of current problems

EID and the Management of Large Numbers of lntelligent Agents

149

with automation suggests that the factors that contribute to mode errors with current automation include indirect mode changes, inadequate feedback, and inconsistent behaviour. An analysis of likely emergent behaviour of agent-based automation, suggests that it may exacerbate all of these factors. Mode errors, misuse, and disuse of automation could dramatically increase with poorly designed agent-based automation. This chapter takes a problem-driven, human-centered perspective in exploring how best to capitalize on the burgeoning field of agent-based automation. To capitalize on the capabilities of agent-based automation, calibration of trust emerged as a critical consideration. One particularly promising approach to enhance the calibration of trust and the effective management of agents is to adapt the theory of Ecological Interface Design (EID) to support effective control of a selforganizing system of agents. This would help develop effective visualizations for displaying agent behaviours, presenting the current state of the system, and the predicting future states. EID is particularly well-suited to this application because EID focuses on identifying and displaying system constraints. Because the behaviour of agents emerges from their interaction with the system EID will represent agent behaviour in a way that makes the emergent behaviour understandable. Using EID as an interface to manage agent-based automation is an important extension to EID, which has produced a series of elegant interface solutions for complex systems. The EID paradigm is particularly well poised to support control of agent-based automation because it emphasizes holistic understanding and pattern recognition that has been successful in supporting the management of other complex systems. Addressing the details of how to adapt the EID principles to the management of an agent-based, self-organizing system, however, remains an important challenge.

References [ 1] [2] [3] [4] [5] [6] [7]

Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775-779. Beltracchi, E. L. (1987). A direct manipulation interface for heat engines based upon the Rankine cycle. 1EEE Transactions on Systems, Man, and Cybernetics, 17(3), 478487. Beni, G., & Wang, J. (1993). Swarm Intelligence in Cellular Robotic Systems, Robots and Biological Systems: Towards a New Bionics. Berlin: Springer-Verlag. Bonabeau, E., Theraulaz, G., Deneubourg, J. L., Aron, S., & Camazine, S. (1997). Selforganization in social insects. Trends in Ecology & Evo|ution, 12(5), 188-193. Brooks, R. A., & Flynn, A. M. (1993). A Robot Being, Robots and Biological Systems: Towards a New Bionics. Berlin: Springer-Verlag. Brooks, R. A., Maes, P., Mataric, M. J., & More, G. (1990). Lunar base construction robots. Proceedings of the 1990 International Workshop on Intelligent Robots and Systems, 389-392. Christoffersen, K., Hunter, C. N., & Vicente, K. J. (1998). A longitudinal study of the effects of ecological interface design on deep knowledge. International Journal of Human-Computer Studies, 48(6), 729-762.

150

[8]

[9] [10]

[11] [12]

[13]

[14]

[15] [16] [17]

[18]

[19] [201

[21] [22]

[23]

[24] [25]

EID and the Management of Large Numbers of Intelligent Agents

Degani, A., & Kirlik, A. (1995). Modes in human-automation interaction: Initial observations about a modeling approach. Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, Vancouver, British Columbia, Canada. Effken, J. A., Kim, N. G., & Shaw, R. E. (1997). Making the constraints visible: Testing the ecological approach to interface design. Ergonomics, 40( 1), 1-27. Fukuda, T., Funato, D., Sekiyama, K., & Ari, F. (1998). Evaluation on flexibility of swarm intelligent system,. Proceedings of the 1998 IEEE International Conference on Robotics and Automation. Johnson, P. J., & Bay, J. S. (1995) Distributed control of simulated autonomous mobile robot collectives in payload transportation. Autonomous Robots, 2(1 ), 43-63. Kantowitz, B. H., Hanowski, R. J., & Kantowitz, S. C. (1997). Driver reliability requirements for traffic advisory information. In Y. I. Noy (Ed.), Ergonomics and Safety of Intelligent Driver Interfaces (pp. 1-22). Mahwah, NJ: Lawrence Erlbaum Associates. Lee, J. D. (1995). Ecological interface design: Applications in the maritime industry. In W. Wittig (Ed.), The influence of the man-machine interface on safety of navigation (pp. 89-95). Bremen, Germany: Verlag TUV Rheinland. Lee, J. D., Kinghorn, R. A., & Sanquist, T. F. (1995). Review of Ecological Interface Design Research: Applications of the design philosophy and results of empirical evaluations. Seattle, WA: Battelle Human Factors Transportation Center. Lee, J. D., & Moray, N. (1992b). Trust, control strategies and allocation of function in human-machine systems. Ergonomics, 35(10), 1243-1270. Lee, J. D., & Moray, N. (1994). Trust, self-confidence, and operators' adaptation to automation. Int. J. Human-Computer Studies, 40, 153-184. Lee, J. D., & Sanquist, T. F. (1993). A systematic evaluation of technological innovation: A case study of ship navigation. IEEE International Conference on Systems, Man, and Cybernetics, 102-108. Lee, J. D., & Sanquist, T. F. (1996). Maritime automation. In R. Parasuraman & M. Mouloua (Eds.), Automation and Human Performance (pp. 365-384). Mahwah, NJ: Erlbaum. Malin, J. T., Schreckenghost, D. L., & Rhoads, R. W. (1993). Making intelligent systems teanz players: Additional case studies (104786): NASA Johnson Space Center. Min, T. W., & Yin, H. K. (1998). A decentralized approach for cooperative sweeping by multiple mobile robots. Proceedings of the 1998 IEEE/RSJ International Conference on Intelligent Robots and Systems. Parasuraman, R., & Riley, V. (1997). Humans and Automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230-253. Rasmussen, J., & Vicente, K. J. (1989). Coping with htunan errors through system design: Implications for ecological interface design. International Journal of ManMachine Studies, 31,517-534. Sarter, N. B., & Woods, D. D. (1992). Pilot interaction with cockpit automation I: Operational experiences with the flight management system. International Journal of Aviation Psychology, 2(4), 303-321. Saner, N. B., & Woods, D. D. (1995). How in the world did we ever get in that mode? Mode error and awareness in supervisory control. Human Factors, 37( 1), 5-19. Seeley, T. D. (1997). Honey bee colonies are group-level adaptive units. American Naturalist, 150, $22-$41.

EID and the Management of Large Numbers of lntelligent Agents

151

[26] Sheridan, T. B. (1975). Considerations in modeling the htanan supervisory controller. Paper presented at the Proceedings of the IFAC 6th World Congress, Boston, Massachusetts. [27] Stickland, T. R., Britton, N. F., & Franks, N. R. (1995). Complex Trails and Simple Algorithms in Ant Foraging. Proceedings of the Royal Society of London Series BBiological Sciences, 260(1357), 53-58. [28] Sugihara, K., & Suzuki, I. (1990). Distributed motion coordination of multiple mobile robots. 5th IEEE International Symposium on Intelligent Control, 138-143. [29] Vicente, K. J., & Rasmussen, J. (1992). Ecological interface design: Theoretical foundations. IEEE Transactions on Systems, Man, and Cybernetics, SCM-22(4), 589606. [30] Vicente, K. J., Moray, N., Lee, J. D., Rasmussen, J. D., Jones, B. G., Brock, R., & Djemil, T. (1996). Evaluation of a rankine cycle display for nuclear power plant monitoring and diagnosis. Human Factors, 38(3), 506-521. [31] Vicente, K. J., Christofferson, K., & Pereklita, A. (1995). Supporting operator problem solving through ecological interface design. IEEE Transactions on Systems, Man, and Cybernetics, SMC-25(4), 529-545. [32] Wickens, C. D. (1998, March). Automation in air traffic control: The human performance issues. Third Human Factors in Automation Conference, Norfolk, VA. [33] Woods, D. D. (1991). Nosocomial automation: Technology-induced complexity and human performance. Proceedings of the International Conference on Systems, Man, and Cybernetics, 1279-1282. [34] Woods, D. D. (1991). Nosocomial automation: Technology-induced complexity and human performance. Proceedings of the International Conference on Systems, Man, and Cybernetics, 1279-1282.

Operator Support in Technical Systems

Heiner Bubb Chair of Ergonomics, Technical University Munich, Boltzmannstr. 15, D-85747 Garching e-mall: [email protected]

Abstract: The idea, the complexity, and the acceptance of operator support in technical systems is explained by the example of driving a motor car. The subjective judgement of future driving aids were investigated in a driving simulator experiment. The new systems are the Heading Control (HC) device, which enables the car to follow the road boundary automatically, and the Adaptive Cruise Control (ACC), which measures the distance to the car ahead and keeps the correct safety distance automatically. In the experimental case of the HC system, a feed back was given to the driver by an artificial reverse moment in the steering wheel. Additionally, the human behaviour was investigated by using a conventional control element in connection with an Active Control Element (ACE), a joystick like control device, which feeds back the dynamic system states of speed and yaw angle velocity by changing the position of the stick. In connection with this ACE, the same feed back can be presented to the driver as in the case of the conventional control elements. The results of the experiment show the superiority of the ACE, although the implementations of the new aids HC and ACC are at present not very well accepted.

1. T e c h n i c a l S o l u t i o n When developing any technical system, the usual procedure is to start with a technical approach and to make improvements step by step according to the experience gained by using this new system. Therefore, planning the use or invention of a machine normally starts with the technical conditions and not with the human preferences. As an example, the familiar processes of driving a motor car may be used. Presumably nobody ever asked how the driver would wish to steer the car. Since a car with four wheels can be moved on a curved path when the axes o f the front wheels have an angle to the ones of the rear wheels, the steering wheel was introduced purely by technical reasons to adjust this angle. The longitudinal dynamic of a car was developed in a similar way. Although we have only two feet, we have to operate three different pedals in a car in this case: the pedals for the accelerator, the brake and the clutch. In addition the gear lever has to be operated by hand. All these control elements must be used and coordinated, just to adjust the one longitudinal movement! What is valid for driving a motor car, is as well valid for the manipulation of other machines or systems.

154

Operator Support in Technical Systems

As an example, just think of the well known fact that up to 80 % of the population is quite incapable to program a video-recorder correctly. As experience shows, an operator does not know how to operate a technical system. The usual consequence is to recommend adequate training and education. Also the observation of accidents leads normally to better safety education of the operators. However, like the experience in public traffic indicates, education campaigns only have a short-term effect and must often be repeated when external conditions change. According to statistics, the most frequent accident categories in all road categories (motorway, country road and city traffic) are deviations from the road to the right or to the left, parallel or opposite collision with vehicles, and front-end collision. Therefore, in opposite to a first impression, it seems to be difficult to control the transversal and the longitudinal dynamic of motorcars. In order to improve the performance of these controlling tasks, several technical assistance systems were

Figure 1: Principal information flow of the electronic stability control (ESC) system in motor cars. developed as for instance ABS (Antiblocking Braking System) and ESC (Electronic Stability Control) which compensate for mistakes of the driver. As Fig. 1 outlines by the example of ESC, the general idea of this kind of system is that the human operator (here the driver) gives an order to the control elements in the same way as it is done with a conventional non aided system (here steering wheel). The actual position of this control element is used to calculate the nominal behaviour of the system (in this example "dynamic calculator II"). By measuring the influencing physical parameters, the actual behaviour is calculated in a parallel branch of the dynamic computer (here "dynamic calculator I"). In the case of an observed

Operator Support in Technical Systems

155

difference between actual and nominal behaviour in a third branch (here "ESC control calculator"), an adequate intervention is calculated. A further technical system, currently on the way to be commercialized, is the Adaptive Cruise Control (ACC). It can be seen as a progressive development of the cruise control system used up to now maintaining a pre-selected speed automatically. The advantage of ACC is the additional radar detection of obstructing objects ahead. The information is used to automatically decrease the speed of the car to keep an adequate distance to the car ahead. Another system still in the phase of research is the Heading Control System (developed within the scope of the PROMETHEUS research project). It is able to define road limitations using a camera. An automatic system keeps the car on the defined trail. In order to keep the driver within the control loop of this system, the actual course given by the driver is compared with the nominal course calculated by the automatic system similar to the one of the ESC-system. A corrective moment is given to the steering wheel in case of deviation. If we summarize all these developments, we can see a general tendency towards automation of the car driving process, which can exclude the driver from the closed loop driver-vehicle system in the end. As a consequence to this technical development, the following questions arise: How is the human operator involved in this new system? What are the tasks left to him now? What are his feelings about and reactions to such a system that imposes inactivity?

2. Ergonomic Analysis and Recommendations The human operator is very adaptive, that means: if he experiences a process to be sufficiently reliable, he does think that the system will ever have a failure or breakdown, and he assumes that the system will work properly at any time. Therefore, one of the most important questions in connection with automation is: What happens if the system is out of order? To answer this question, a system ergonomic analysis [2] of the driving process is to be carried out to comprehend the driver's mental workload on the one hand and to get an idea of the driver's understanding of his role in this process on the other hand. To allow a prediction of the driver's behaviour in the case of breakdown of the system, the system ergonomic approach is used. It starts with a general description of every task and draws ergonomic recommendations for the partial aspects of the task based on experimental experience. It does not create any images about the internal model of the user. The fundamental idea is that the tasks to be performed by the operator may be designed by considering the knowledge about the information transfer between the subsystems man and machine. System ergonomics is concerns three main areas:

156

Operator Support in Technical Systems

Task contents - f e e d back - compatibility." Task contents describes how far the system corresponds to the internal models of the operator, this also means: How big is the effort to learn to operate the system? 9

Feed back answers the question to what extend the operator is able to receive information about his position in the actual state of the system.

9

Compatibility describes the effort of information processing between different information channels that connect the operator to the outside world.

For each of these areas, we know about the specific human workload given by deviations from the demanded lay-out and about the ergonomic rules that can achieve an improvement. For each of these areas, we know about the specific human workload given by deviations from the demanded lay-out and about the ergonomic rules that can achieve an improvement. In order to apply system ergonomics to the driving process, it is useful to consider the hierarchical interconnected levels of driving: navigation, guidance and stabilization [3] (see Fig. 2). These correspond to general hierarchical encapsulated tasks: "planning", "programming and action", and "observing and controlling" [4]. nominal: ~lace me

actual:

. . . ~ , ~. _ ~ s # ' /

.

.

.

lace, ' e

N=~ ':..

(l~ma~)

...

ergonomlr solution proposals: 9 navigationcomputer 9 Detection and indication of and indicationof technical realisable objects ~npost and traffic participants 9 indication or keeping of a - safety distance (e.g. ACC), - overtaking distance, - max. transverse acceleration (e.g. Heading

Contro/)

9 indication of velocity 9 help in unstable driving manoeuvres as for instance ABS and ESP

(Active Control Element)

* ACE

Figure 2: The three level of the driving task [3] and ergonomic aids for the human operator. By the navigation task, the exact route between the starting point and the destination is selected. This can be performed with the help of a brain-stored map or with the help of a hardware map. In any case it is a thought process that needs. The

Operator Support in Technical Systems

157

result of the navigation task is the input for the guidance task. This task is accomplished immediately during the driving process. It consists of the finding of the safe way on the road through the standing and moving traffic participants. For finding this way, geometrical as well as time aspects (i.e. the correct selection of speed, the correct judgement of the moving behaviour of the other traffic participants) must be considered. This task must be accomplished within a very short time frame. It can therefore be called "dynamic task". The outcome of this task is the input for the lowest level, the stabilization task, which often is seen as the real driving task, which determines the manipulation of the control elements (steering wheel, accelerator, brake-pedal, gear shift and clutch pedal) and the forcing function defined by the guidance task into reality. This task is also a dynamic task. The final result must fit to the corresponding tasks on all levels. If a deviation is found anywhere, normally a corrective action must be performed on the next higher level (e.g. if the car is not able to be driven through a desired curve on the level of stabilization, a new way and a new nominal speed must be created on the level of guidance considering the new conditions; if a wrong and not desired way is chosen on the level of guidance, a new corrective way must be found on the level of navigation). If we look at the technical supports being actually in development, we observe that they are especially related to the internal shell "stabilization task" (e.g. ABS and ESC) and to the external shell "navigation task" (in form of the presently offered navigation systems for motorcars) but not for the "guidance task". However, a scientific project currently in progress [5] shows that deficits on this level are the main reason for accidents.

3. Ergonomic Solution Proposals The combination of the Heading Control Device and the ACC-system could be a first approach to support the driver in the guidance task, because this system allows to receive the available area ahead of the driver (HC) and to project a trajectory of the actual path of the car into this area under consideration of obstructing moving objects (principal abilities of the ACC-system). However, this will happen in the far future only. Presently, neither the HC-system nor the ACC-system offers a sufficient technical reliability. Consequently, a speedy introduction of such a system cannot be expected. Apart from the solution of technical problems, the methods of informing the driver as well as means of displaying the system's feedback are objects of major research efforts. This can be done by using a driving simulator that allows to investigate and adjust the technical reliability of new systems and also allows scientific investigations of the related human behaviour. This would already in an early stage allow to foresee future developments. Presently, we are investigating the effect of two concurring systems:

158

Operator Support in Technical Systems

a) Use of conventional control elements: an artificially created reverse moment is added to the steering wheel that informs the driver about the technically determined path. The information effect of this reverse moment is investigated in the form of "narrow guidance" (i.e. every deviation is immediately indicated) and "boundary guidance" (i.e. a reverse moment is applied only, if the driver leaves the safety zone), see Fig. 3.

""::'t"

"

"

=

rl~

~

,~{

reset moment'

reset moment . ..!.'."

AA

left

right

Deviation from nominal course 9 ...... "'= ""~ :'"" ;":": " ] "llll "" ~"

" Ill~fl :'9

j;i

AE

fight iii

left

Deviation from nominal course :... I IlI

Ill'

I

L',.

~'PY "l'"

Figure 3: The two thinkable reset moments in the case of a Heading Car device. In the case of conventional control elements, the information of the longitudinal control intervention is can only be obtained by increasing or reducing the engine power and activating the braking system. In addition, different forms of acoustical or optical indicators are investigated.

b) Use of the so-called Active Control Element (ACE) [6]: the ACE is a specifically-developed control element similar to a joystick that measures the force applied by hand and uses this information for the steering function. The effect of the machine (in case of the motor car it is the actual speed and the actual yaw-speed) is fed back to servo motors of the control elements. By this, the position of the joystick represents always the state of the machine dynamics. The driver holds, so to speak, the machine in his hand (see Fig. 4 and 5).

Operator Support in Technical Systems

Conventional control element task ~ ,.t.o~e

,--.~[~--~~

159

9 result~ machine ~--~ =

Active Control Element(ACE)

.

9: l u . . .

i.

I ,~e conlrolelement,

.!

i

Figure 4: The information flow of the conventional control element in relation to the Active Control Element (ACE).

itudinal velocity

angle velocity

Figure 5: The application of the ACE in motor cars. The ACE is an ergonomic aid on the level of the stabilization task. The effect of it is shown in Fig. 6. The figure illustrates that the closed loop driver-vehicle system shows a outstanding resonance peek at about 2,5 rad/s ( = 0,4 Hz, dotted line) in the case of the conventional steering wheel. In practice, this means that the drivervehicle system shows an instability when the driver makes too fast reactions. The cars seem to go into a skid. Only when the hands are kept off the steering wheel, the car would stabilize itself. The non interrupted line shows the behaviour of the closed loop driver-vehicle system in the case of the ACE. The resonance peek now nearly disappears totally [1].

160

Operator Support in Technical Systems

ZO[.

-

'

m' ~ 00~ m

"

H~I

t,

~

f steedngwheelwithaspeed

I0

..=_

E

activecontrolelementwitha

-1

'~ = -300'

yaw angle dependent position feed back

02

~ 02

0A

lZrll

~

3

]i. . . illlii . . . . . . . . .i'N.I 0,4

frequency

1

3

[rad/s]

Figure 6: Effect of the active control element on the frequency response of the closed loop of the driver-car-system In the same way as it can be used in connection with conventional control elements, this new aid can be combined with the two different types of artificial moments described above, which feed back the information of the technically determined path. The advantage of the ACE in this application is, that the information of the boundaries in transversal and longitudinal direction is transferred to the human operator always in the same way.

4. Investigation of Acceptance Preliminary investigations in a driving simulator where carried out. 5 1 candidates had to drive a round course. During the driving experiments, they additionally had to react on suddenly appearing unexpected tasks like "getting out of the way of a broken down car", "breakdown of HC-System", "coming out of tunnel", "slow car ahead without overtaking opportunity", and "HC triggers a wrong path". They had to participate in two main experiments: one with conventional control elements (steering wheel, accelerator and brake pedal; the simulator was programmed as a car with automatic gear shift) and one with the ACE. In every case, the behaviour "without assistance system", with "narrow guidance" and with "boundary guidance" were investigated. First results are collected in Tab. 1 on the basis of a questionnaire about the experiments. The main questions asked for 9 9

the necessary concentration, the concentration in the case of system brake down,

Operator Support in Technical Systems

9 9 9

161

the judgement of the help that is offered by the new system, the self judgement of the ability for correct reaction, and the individual priority, whether the subjects would buy such a new system. ,, S t e e r i n g

nmry cormer~i~allon in 9t ~ , i ~ e o f

Wh'~.

narrow

guidance

without

assistance

sys-

tembreaktJown

help

,, ,~ ,~ , ,.~.~. ,..

priority

.....':~: ......

without assistance

" "

9: Active Control Elemeiat narrow guidance without assistance, but if assistance then narrow guidance without assistance, but if assistance then narrow guidance

without assistance

without assistance, but if assistance then narrow guidance

without assistance

without assistance, but if assistance then narrow guidance

Table 1: Preferences of the subjects after their experience with the four experimental situations. The table shows, that for both main layouts of the feed back system the "narrow guidance" is preferred. The operator seems to prefer the feeling, that the automatism works. In case of faulty operation or breakdown of the system the unaided hand operation is preferred for both layouts. When a faulty operation or breakdown of the system is taking place, an advantage of "narrow guidance" compared to "narrow guidance" is observed under the condition of ACE . This advantage was never observed with conventional control elements. That means: if we want to benefit from safety from an automatic system, conventional control elements (and indicators) are not appropriate. We have to use new interfaces between man and machine, which specifically take into account ergonomic demands.

References [1] [2] [3]

Bolte, U. (1991): Das aktive Stellteil - ein ergonomisches Bedienkonzept. FortschrittsBerichte VDI, Reihe 17 "Biotechnik", VDI-Verlag, DUsseldorf. Bubb, H. (1988). System Ergonomics as an Approach to Improve ttuman Reliability. Nuclear Engineering and Design 110, S. 233 - 245. Bemotat, R.(1970): Plenary Session: Operation Functions in Vehicle Control, Anthropotechnik in der Fahrzeugf'tihrung, Ergonomics, Vol. 13.

162

[4] [5] [6]

Operator Support in Technical Systems

Sheridan, TB. (1976): Toward a General Model of Supervisory Control. In: Sheridan, T.B., Johannsen, G. (Hrsg.), Monitoring Behaviour and Supervisory Control. New York, London. Reichart, G, (1999): Menschliche Zuverli~ssigkeit im Straflenverkehr; Dissertation an der Technischen Technischen Universitat M0nchen. Bubb, H. (1985): Arbeitsplatz Fahrer - eine ergonomische Studie , Zeitschrift: Automobilindustrie 30, S. 265 - 275.

Interfaces for Every Day Things

Kerstin ROse Institute for Production Automation, University Kaiserslautem, P.O. Box 3049, 67653 Kaiserslautern, Germany e-mail: [email protected] Abstract: Interfaces for Every Day Things - Specialist required? What is the meaning of "Everyday Things" for Industrial Design in general, what for the design of Human-Machine-lnterfaces with high complexity? Based on these questions, this paper aims to show, how important the sensitivity for the design of Everyday Things is and to give an understanding for the correlation between design of Everyday Things and that of Human-Machine-Interfaces. Human operations can generally be seen as being aimed at achieving certain tasks. They can be divided into three main aspects, i.e. type, principles or patterns, and context of operation. The context may vary for different situations, but patterns and types stay principally the same. According to Rasmussen [4], one can distinguish three types of operation: knowledge-based, rule-based and skill-based. Knowledge- and rule-based operations are performed on a conscious level, while skill- based operations are those which are often recurring and thus handled in a increasingly automatic manner, like stimulus-response-operation. The underlying principles, or patterns, of operations can be explained with the Theory of Mental Models 1 [5]: Correlations between input and effect of past operations are stored in the user's memory, together with their basic assumptions. If these assumptions are encountered again, the corresponding mental model is retrieved and used (rule-based action).If it is not possible to transfer the existing mental models directly, the operation cycle is modified. If the desired goal is still not reached, the assumptions are verified. They remain unchanged if they are rated as correct; instead, a new model for the procedure is compiled (knowledge-based action). For more complex tasks, the new model can be a combination of existing 'simple' models. The familiarity with the new mental model increases with the frequency of its use, turning it into a rule-based operation.

1 Mental models represent explanations for proceedings in the real world.

164

Interfacesfor Every Day Things

Research has shown [2,3] that humans possess ,,natural standards", so-called compatibility patterns, which are based on a defined anticipation of a system's reaction to a certain operation, e.g. turn to the right = value increase, a to an inherent system change, e.g. value increase (temperature rise) leads to indicator increase (pointer deflection to the right). Compatibility patterns can be seen as a basic form of mental models. Thus, with the collection of mental models a human being is building a data base, storing effect correlations i.e. operations and related reactions. In this way, the human being is able to retrieve a fitting or modifiable mental model for a number of standard situations. These standard situations are, due to their frequent occurrences, usually encountered in the context of Everyday Things, such as household appliances or cars.

Level of vigilance

Level of automalion

Y

Curve of vigilance in a stressfull situation/by lime

t

I1

Curve of automation operation / by cases of use

Figure 1: Levels of Vigilance and Automation. For operating technical devices, the user employs his mental models and makes assumptions about the necessary input (or action) as well as the subsequent reaction of the device. The user interprets continued confirmation of his assumptions as verification of his mental models. It can be assumed that by use of Everyday Things effect correlations are recognised and tested on everyday technical devices (e.g. HiFi). Consequently, mental models for the functionality of technical devices are established with the use of Everyday Things. It can be assumed that basic knowledge for handling technical devices can be acquired through the use of everyday things. The statement is made that for the use

Interfaces for Every Day Things

165

of Everyday Things and the use of human-machine-interfaces the same mental models are taken as a basis for corresponding operations. Mental models and resulting patterns of operation associated with dealings of Everyday Things are more present in the memory of the user due to the high number of applications. This is an aspect which is especially of importance in routine situations and time critical situations. Time critical situations are encountered in the area of production and process equipment, especially in abnormal conditions. An abnormal condition creates an unforeseen situation in the course of the operation and therefore a stress situation for the user. In stress situations the human operation is mostly based on a simple operation level (often skill-based). In this case the user goes back to application-safe and quickly available operation patterns, i.e. simple mental models, such as those acquired from Everyday Things. If one succeeded to convert these simple mental models into the area of material processing and consider the compatibility pattern, then the user would react more adequately in emergency situations. In this way, human error could be avoided by preventive proceedings in development and design of complex technical systems, together with a general simplification of operation by applying familiar mental models. Therefore, it is apparent that more importance has to be given to the design of Everyday Things, so that the user's operation patterns are already established at an early stage by the use of Everyday Things. They will then form a better basis for operation patterns relevant in complex technical demand situations, especially emergency situations.

References [ 1] [2] [3] [4] [5]

Reason J., 1990, Human Error. Cambridge University Press. Norman D.A., 1988, The Psychology of Everyday Things. Basic Books. Murrell K.F.H., 1971, Ergonomie. l.Auflage, Econ Verlag GmbH, Dilsseldorf, S. 289-310. Rasmussen J., 1983, Skills, Rules, Signals, Signs and Symbols and Other Distinctions in Human Performance. IEEE-Transaction, Vol. SMC-13 Anderson J.R., 1996, Kognitive Psychologie. 2.Auflage, Spektrum Akademischer Verlag, Heidelberg.

Operator Process Interfaces - A Retrospective View of the '90s

Klaus Zinser ABB Utility Automation, Mannheim, Germany e-mail: [email protected]

Abstract: Visualisations for complex dynamic processes represent one of the most demanding fields in the area of developing better operator-process interfaces. This paper describes the comprehensive work done by guiding and integrating advanced university research in the application domain of power plant supervisory control. The results are put into perspective of a decade of many activities in this field.

1. Introduction The last decade was characterised by tremendous advances in the Computer Technology, mostly the CPU and graphics hardware performance. This mostly became apparent in the computer games industry - more power and finesse on ever smaller footprints - an optimized packaged solution for the big business. On the SW side, of course, advances were not that dramatic: X-windows and Motif (the first powerful graphical User Interfaces for Unix, and hence process control systems utilizing this operating system) dominated the first half of the decade, Microsoft Windows the second half (Windows NT being the control system operating system of choice then). Another trend, Artificial Intelligence development environments showed as one its most noticeable effects on computer industry the advancements of object-oriented technology.

2. A p p r o a c h a n d Results 2.1 Visualisations for Process Supervisory Control In ABB, most noticeably in its Corporate Research organisation - together with its Power Plant Control and Network Control divisions, throughout the 90's a

168

Operator Process Interfaces - A Retrospective View of the '90s

continuous effort was made to advance operator interfaces and particularly supervisory control visualisations for the large and dynamic masses of process data. These visualisations encompass: 9 Picture Hierarchy, Picture Pyramid: efforts to arrange a large number of individual process displays in only a few levels of abstraction (from overview to loop) that allow for information panning within a level and so-called information zoom (including dynamic information decluttering) between levels. 9 Context Specific Visualisations: adjustment of the contents of either a process display, or the particular part of a whole Picture Pyramid level according and optimized to the status of the process itself, by support of an expert system for process state assessment and graphical picture composition. 9 Overview Visualisations: one of most demanding challenges is still to visualize the large amount of dynamically changing process data in a way both meaningful to the operator and processable by his cognitive capabilities. The work on Mass Data Displays still represents the most successful and unique work here. 9 Fisheye Views: this name derived from optics stands for the simultaneous, manipulation of overview and detail information presentations, interactively by the operator and with immediate, dynamic graphical feedback. 9 3D process visualisation: most common visualisations today are coherent with the 2-dimensional computer screen (so called 3dimensional displays today are really 2Y2 dimensional, as they only contain 3D effects (shading) of plant components). The 3D process visualisations depicted here were developed to make meaningful use of the 3~ dimension (rotation, distortion) - and at the same time incorporate interactive, dynamic properties of Fisheye Views. All this work initially was done independent of each other. Each visualisation was developed to a level that is difficult to represent in a printed article, and deployed in a simulated process control environment. Eventually, lastly, a taxonomy was developed, on how to bring it all together, including the crucial aspect, on how to navigate between visualisation displays which are individually optimal for a given aspect: the visual momentum. 2.2. A Taxonomy for Process Visualisation This taxonomy describes how process displays should be placed in the AbstractionAggregation Hierarchy along the diagonal that optimally supports problem solving behaviour.

Operator Process Interfaces - A Retrospective View of the '90s

169

AbstractiOn

GoaP

Purpose

Function

physical Appe~.mr

Plant

Syslmm/Subsystem

Component

Degree of DeM#

Figure 1: A Taxonomy for Process Visualisation. The following figure depicts this taxonomy as applied to Power Plant process visualisations (we developed and implemented also displays for Network Control domain).

Goab' Purpose

Function

Physical Appearance

:===:=,. ~ .

Plant

System~ubsystem

Figure 2: The Taxonomy as realised for Power Plant Domain.

t,l~.-~lr

Component

!

.

Ib

Deta,I

170

Operator Process Interfaces- A Retrospective View of the '90s

2.3. Results A series of experiments provided insights how the taxonomy applies in praxis. The major issue of course, the navigation between individual displays of the abstractionaggregation hierarchy, depends on the actual chosen displays. For navigation both aspects of spatial and temporal proximity were investigated, and the convincing results are shown in the following figure.

Goal,

Purpose Function

Physical Appearance SubSystems System Component Figure 3: Navigation in Visualisation Taxonomy -Proof of Concept.

3. D i s c u s s i o n So, given all this work on individual new forms of process displays, integrating them, and performing experimental evaluations, and given the interesting results and findings, the one may wonder why it is not all available in products. As a short summary, there are a variety of reasons, ranging from market aspects such as less steam power plants (primary users of the described approaches) and almost full stop of nuclear plants (traditionally driver of technology), to more popular IPP power plant operators (typically combined cycle power plants that have less supervisory control complexity) - all in the overall scenario of cost pressure due to deregulation, and ranging all the way to soft issues such as conservatism and cultural/education background of plant operators. Then, will we ever get it? - and if yes: when? Given the ubiquitous UI technology, (Microsoft windows UIs, Internet/WWW, palm computers, mobile communication), common DCS technology based on Windows NT that helps to easier implement and leverage investment, and given the next generation operators with more computer literacy, chances are good!

Operator Process Interfaces - A Retrospective View of the '90s

171

References [1]

[2] [3] [4]

[5]

Elzer, P. F., Weisang, C.; Zinser, K., 1990, Knowledege-based System Support for Operator Tasks in S&C Environments, IEEE Conference on Systems, Man and Cybernetics. Zinser, K., 1993, Neue Formen und Medien der Prozeflvisualiserung, atp 9/93, S.499504. Zinser, K, 1995, Fisheye Views - lnteraktive, dynamische Visualisierungen, atp 9/95, S.42-50. Beuthel, C., Elzer, P. F., 1997, Advantages of 3D Displays for S&C of Large Technical Systems, Conference on Human Decision Making and Manual Control, Kassel, Germany. Bums, C; VicenteK., 1996, Comparing Functional Information Content of Displays, Proceedings 28 ~ Annual Conference of Human Factors Association of Canada HFCA, pp. 59-64.

Acknowledgements:The work done in ABB was performed on different aspects of the overall scope with the following organisations who the Author wants to credit here: University of Clausthal, Prof.Dr.Elzer, Dr.Boussoffara, Dr.Beuthel University of Toronto, Prof.Dr.Vicente, Dr.Bums, and last not least the ABB units that supported work with funding and enthusiam over the years.

Acceptance of new Technology and Layout in Control Rooms

J.W. de Vries Dept. of Nuclear Technology, N.V. EPZ NPP Borssele, Belgium e-mail: nucvries @ zeelandnet.nl

Abstract: EPZ, the Electricity Generating Company Ltd. for the Southern Netherlands, owns and operates the Borssele Nuclear Power Plant which is located near Flushing on the Schelde estuary. In 1997 an extensive backfitting programme was carried out at this twin-loop 480 MWe PWR in order to raise nuclear safety to a level comparable with that of a modern plant. One of the key issues in order to accomplish this was the definition of a 'protected zone'. This zone includes the reactor containment and the bunkered building, housing the alternative cooling system, the RPS and the back-up control room. Systems in the protected zone are able to withstand external events as earthquake, airplane crash, gas-cloud explosion and flooding. Of course internal events are taken in account too. From the back-up Control Room it is possible to bring the reactor to a safe and stable cold sub-critical state.

Another major part of the work focused on electrical systems, instrumentation and control and Human-Machine Interaction (HMI). During the project a lot of experience was gathered on (operator) acceptance of new technology and layout in control rooms. Two sub-projects were of importance here, namely the renewal of the main control room and the revamp of the process computer.

1. Revamp Main Control Room Originally there were no plans to revamp the whole of the main control room. However, during the engineering stage it became clear that it would not be possible to accommodate the extended HMI demands in the existing main control room (MCR). Gradual upgrading during the past 20 years had used up any available space on the control desk and wall-panels. A complete renewal of the MCR was called for. In order to ensure that all design-aspects would be covered a HMI-

174

Acceptance of new Technology and Layout in Control Rooms

Working Group was established with participants from relevant departments in the NPP's organization as well as external ergonomic advisers. Its goal was to have a consultative body where departments as operations, training, process-technology and maintenance can - from their own field of expertise- with respect to the HMI, give input and participate in the design of the systems which are modified or newly built. Modern design pays a lot of attention to those engineering design aspects which allow tolerance for delayed human interaction during the course of automatic actions of safety systems. This approach gives breathing space to the operator after initiation of the protection system and leads to a more level-headed reaction on disturbances. After analyzing the shortcomings of the old MCR by walktrough/talk-trough of problems and interviews of the operators, a complete operations shift was made available to design the lay-out of control-desks and panels for both the existing Main Control Room and a new back-up CR. This was done using mock-ups, observing a set of rigorous ergonomic limiting conditions and the operational concept of the NPP. This process took five people working for three months. The Human Machine Interaction Working Group conducted this process. After review by all end-users (shifts) a high degree of acceptance was achieved and two control rooms were ordered. One for the full-scale replica simulator - also part of the back-fitting program- and one for the real NPP. Experiences from test-runs with the first delivered simulator CR were incorporated in the MCR design for the NPP.

2. Revamp Process Computer Another back-fitting sub-project was the revamp of the process computer. The main reason to do this was the extended number of analogue and binary point ID's required by the back-fitting project. The old process computer MMI, requiring a lot of key-board input for information retrieval, was replaced by a modern graphic user-interface with X-Windows for multi-window applications. The operator workstations have a wide variety of displays with modern intuitive user interface. All process displays are based on extensive human factors research conducted in the 1980's. User interaction has been updated to use modern techniques unavailable at the time of the original studies. The old hard-wired integrated plant status overview panel (IPSO) was replaced by a Dr.Seufert computer animated large screen projection system. The new technology replaced the key-board with a trackball or mouse as pointing device and was with its improved operator interface easily

Acceptance of new Technology and Layout in Control Rooms

175

accepted by the control room crew after some training sessions on the simulator. Here again from the start on operational staff was involved in the selection, design and review of the new system displays.

3. Acceptance of new Technology and Layout in Control Rooms Experience gathered during the execution of the project learned that, in order to get good acceptance of new technology and layout in control rooms, one has to start very early in the project working to this goal. Make a timely start with giving project-information to all future users. Involvement in the early project days is the beginning of the acceptance process in the operators heads. Keep in mind that operators in an electrical power station are among the more conservative species in the world. Interview operating staff and other users for early experience, look for the older guys who where there during the plant's first commissioning; they know what was designed in a bad way bur where everyone is accustomed to now; e_g. alarms misplaced vs. panel-seclions, mosaic tiles with control buttons in odd places in the mimic diagram or out of operators reach. Involve future users in the design process of the layout of mimic diagrams for control desks and wall-panels and displays for the computer workstations. Give the operational staff the feeling that the new design is their product. Execute 2 or 3 design reviews by the operational staff during the project in extending order of detail. Talk review comments over with each and every guy and treat both the remarks and the commentators seriously; don't leave grudges behind that way. As an important part of the review process organize usability tests, e.g dry runs of start-up and emergency procedures both on mock-up and simulator. Involvement of users in FAT and SAT works for you in getting commitment and acceptance; people will feel responsible for the end-product that way.

4. Conclusion Involvement of end-users in the process of design, engineering and commissioning of projects with a HMI nature is crucial for the acceptance of new Technology and Layout in Control Rooms.

Advanced User Interface Design for Aircraft Cockpit Devices

JOrg Marrenbach Department of Technical Computer Science, Aachen University of Technology, Ahornstr. 55, D-52074 Aachen, Germany e-mail: marrenbach @techinfo.rwth-aachen.de

Abstract: A Flight Management System (FMS) is a complex computer system that is

used for flight planning in all modem commercial aircrafts. The FMS' humancomputer interface needs substantial improvement to make the best use of the increasing number of features. This article outlines some ideas for a new user interface to replace today's Control and Display Units (CDUs). The alphanumerical flight plan editing is replaced by a graphical user interface. A software prototype of such a CDU has been created, using Statecharts for the definition of this interface. The developed prototype was evaluated analytically as well as in experiments. Overall the pilots envisaged after the experiments that they could use such a system in the future.

I. Introduction A Flight Management System (FMS) is an important part of the automatic flight guidance system of modern aircrafts. The FMS of today has introduced operational advantages and significant cost savings through offering the possibility of an automatic, fuel-efficient flight from take-off to landing and to reduce the pilots workload. However, the FMS with its high level of automation has changed the pilot's role considerably. This has caused dominant problems with respect to human factors. Shortcomings and the most dominant problems of the present FMS are the deterioration of the crew's situational awareness, when out of the guidance loop, a poor ergonomic computer-human interface, and the missing possibility for rapid flight plan changes. Analysis of accidents and training sessions performed in flight simulators show that system behaviour is not always as expected by the crew [1,2,31.

178

Advanced User Interface Designfor Aircraft Cockpit Devices

2. The User Interface of an Advanced FMS In order to make using the FMS easier, especially when changes have to be entered quickly, a demonstrator of an Advanced Flight Management System (AFMS) is developed and evaluated by the Department of Technical Computer Science and other partners in an european research project. The most noticeable change in the user interface is the use of a graphical output device for user inputs, as depicted in figure I. Further, the system-oriented composition of functions is transferred into a much more operational structure. In general, the AFMS provides two ways of access with different functionality: a function-oriented and an object-oriented access mode.

~ IrE] I-;71

lq D IZl IE]DD UIDI-4t E]DB

~k-~--~ ,,~,i,:~;"~"~~ i '"

Figure I: User interface of the Advanced Flight Management System. For the function-oriented mode the operator has to know how the menus and functions are organised. All functions are organised in a so called "menu tree". The user has to select the right "branch" and "sub-branches" with the column and line selection keys to access the function he wants. To inform the pilot, which top level (main menu) he is currently working with, the selected menu selection key will be highlighted until the function tree has left again. The object-oriented mode is quite different and is much more closer to the way a human operator actually thinks, It is created for quick modifications or alterations in flight. This mode offers direct access to the object on that the function has to be executed by moving the cursor of the touchpad to the object. The interaction between the AFMS and the crew is based on the concept of direct manipulation. Due to this, the object-oriented design

180

Advanced User Interface Design for Aircraft Cockpit Devices

References [1] [2]

[3]

Domheim M., 1996, Recovered FMC Memory Puts New Spin on Cali Accident. Aviation Week & Space Technology, 36, pp. 58-61. Salter N.B. and D.D. Woods, 1992, Pilot Interaction with Cockpit Automation: Operational Experiences with the Flight Management System. The International Journal of Aviation Psychology, 2, pp. 303-321. Sarter N.B. and D.D. Woods, 1994, Pilot Interaction with Cockpit Automation II: An

Experimental Study of Pilots'Model and Awareness of the Flight Management System. [4]

[5]

The Intemational Journal of Aviation Psychology, 4, pp. 1-28. Marrenbach J., et al, 1997, Konzept zur ergonomischen Gestaltung der Benutzungsoberfliiche eines zukiinftigen Flight Management Systems. Jahrbuch der Deutschen Gesellschaft ftir Luft- und Raumfahrt, I, Bonn, pp. 497-505. Marrenbach J. and M. Gerlach, 1999, Validierung des Benutzerfiihrungskonzepts eines zukiinftigen Flight Management Systems. Jahrbuch der Deutschen Gesellschaft f(ir Luft- und Raumfahrt, III, Bonn.

180

Advanced User Interface Design for Aircraft Cockpit Devices

References [1] [2]

[3]

Domheim M., 1996, Recovered FMC Memory Puts New Spin on Cali Accident. Aviation Week & Space Technology, 36, pp. 58-61. Salter N.B. and D.D. Woods, 1992, Pilot Interaction with Cockpit Automation: Operational Experiences with the Flight Management System. The International Journal of Aviation Psychology, 2, pp. 303-321. Sarter N.B. and D.D. Woods, 1994, Pilot Interaction with Cockpit Automation II: An

Experimental Study of Pilots'Model and Awareness of the Flight Management System. [4]

[5]

The Intemational Journal of Aviation Psychology, 4, pp. 1-28. Marrenbach J., et al, 1997, Konzept zur ergonomischen Gestaltung der Benutzungsoberfliiche eines zukiinftigen Flight Management Systems. Jahrbuch der Deutschen Gesellschaft ftir Luft- und Raumfahrt, I, Bonn, pp. 497-505. Marrenbach J. and M. Gerlach, 1999, Validierung des Benutzerfiihrungskonzepts eines zukiinftigen Flight Management Systems. Jahrbuch der Deutschen Gesellschaft f(ir Luft- und Raumfahrt, III, Bonn.

AMEBICA - An Auto Adaptive Multimedia Environment Based on Intelligent Collaborating Agents

Chris J. Khalil IMPACT Research Group,Loughborough University, Loughborough, Leicestershire, LEI 1 3TU. [email protected]

Abstract: This paper reports on some of the progress made by the AMEBICA

project. AMEBICA (Auto-adaptive Multimedia Environment Based on Intelligent Collaborating Agents) is an ESPRIT funded project that aims to examine the use of multi-agent controlled auto-adaptive interfaces in the domain of process control. The AMEBICA architecture is described along with conceptual and functional considerations

1. What is AMEBICA? A M E B I C A represents an approach to improving the representation, form and timeliness of data in a complex process control interface. In traditional interfaces, a mapping is made at design time between the process parameters and an appropriate rendering at the interface. This mapping is usually the best all purpose mapping under a set of general constraints. It is not, however, the only mapping - others may have been discarded which represented a better mapping under a different set of constraints. In the general case of the system functioning under normal condition the general mapping may be appropriate. However, if the process moves into a disturbed state, one of the other discarded mappings [Figure 1] may be more appropriate for the new conditions. The goal of A M E B I C A is to capture these other mappings within the system and implement a flexible mapping system, whereby at run-time A M E B I C A makes a decision on which mapping to use, based on the current state of process, the environment, its operator team model and its knowledge of human factors.

182

AMEBICA An Auto Adaptive Multimedia Environment -

[~]~

I ~ ~ [ I~ s

,4, ,,'" ', ProcessState -

I ~ t[-----~" Set of Different _~ Representation . "~ /" Classes .....k, ~.... Possible "~.s.~\ Mapping ProcessState

,v ChosenMapping . . . . . . . . . . -I~ DiscardedMapping

\

Setof Different ProcessEvents

Figure 1: The Set of Possible Mappings Between Process State and Representation.

2. Why Adapt? In modern control rooms, the traditional hard-desk approach has been replaced by a soft-desk approach [1]. In this new approach the operator monitors plant conditions on a large screen or over multiple monitors, but is unable to view all the information simultaneously and must switch between different views as appropriate. The choice of displayed information is set at design time, and any flexibility in the system must be left to the user. It is the aim of AMEBICA to semi-automate the selection and representation of bandwidth limited information. AMEBICA will thus ensure that the main presentation parameters of form, location, and modality correspond to the contents and nature of the information. It is hoped that this will lead to higher predictability of important process occurrences, less information search, less screen cluttering, quicker response times and a generally improved operator effectiveness. ocal )omain ;raphical ttudlo :ngine

ment ~perator sam spects

Figure 2: A More Detailed Viewpoint.

AMEBICA - An Auto Adaptive Multimedia Environment

183

One of Shackel's [2] requirements for usability is flexibility, and as interfaces evolve it is proving crucial. The objective of the AMEBICA system is to introduce an element of system flexibility to a highly complex process control interface. In normal circumstances, it is envisaged that very little adaptivity will be required at the interface (or is indeed desirable, the operators are highly trained, expert users and any unjustified adaptation may well hinder their effectiveness). However, when the system moves into a disturbed state, its role shifts from monitoring system to pro-active alarm handler. This shift is frequently accompanied by alarm flood [3], and consequently information overload. In such situations AMEBICA will assign incoming alarm signals with levels of importance, and use this information along with its knowledge of the current environment, the state of the operators and its knowledge of human factors presentation heuristics to select an appropriate mapping. By dynamically selecting the most appropriate mapping the operator is presented with the most salient information, in the most effective way at the most appropriate time.

3. Conceptual Background AMEBICA has several fundamental issues on which it operates. These issues are crucial to AMEBICA operating in the way envisaged. These can be summarised as follows:

3.1. Timeliness For AMEBICA to work effectively it is crucial that there is as small a delay as possible between signals/measurements arriving from the process, and the rendering at the interface. To this end the system has been designed so as not to process and act directly on the streams, but instead to act on the rendering mechanism that displays the streams. This concept is illustrated in Figure 2, whereby stream information arrives from the process and passes on, unhindered, to the appropriate rendering objects at the interface. As the stream flows to the interface its state values are monitored by the Process Model Agent, which has explicit knowledge of the system and the critical values that justify system adaptation. If the Process Model Agent encounters a condition in a stream that warrants adaptation, it will pass a reference to that stream to the multi-agent system. The agent system then uses its reasoning mechanisms to decide which representation and media best suit a condition of that type and of that importance. The result of this reasoning is AMEBICA altering the form or parameters of the rendering as necessary, by manipulating the local domain graphical/auditory engines.

184

AMEBICA - An Auto Adaptive Multimedia Environment

3.2. Domain independence AMEBICA aims to be a generic adaptation system that maps events of discrete levels of significance - at the input - to appropriate rendering characteristics at the output. To achieve this no direct process knowledge is embedded within AMEBICA. Rather, AMEBICA has two interfaces - the Process Model Agent and the Rendering Interface - that allow it to interact with its environment. These two interfaces contain all the domain dependant information necessary and translate, as appropriate, AMEBICA generic terms to system dependant calls. We see AMEBICA operating in a similar manner as a Java class file in that such a class file is generic and applicable among many systems. However, to make this possible, the class file requires a Virtual Machine (VM) that is highly platform dependent. The VM operates as a translator, changing generic Java calls to systemdependant calls. In the same manner AMEBICA needs a process VM (The Process Model Agent/Rendering Interface) which translates specific process information to a generic format processable by the AMEBICA general system, and vice versa. Thus if an alarm of relatively high importance occurs from a non-critical subsystem, the Process Model might translate this to an low priority AMEBICA alarm. This is then processed and the resultant output of the AMEBICA system is passed to the Rendering which actualises the rendering at the interface. Thus to move AMEBICA between different systems, one simply selects and attaches an appropriate Process Model Agent and Rendering Interface to AMEBICA.

4. AMEBICA Agents To make the correct adaptation it is necessary to take into consideration information from several different sources, including the current environment (lighting levels, operator position etc), the operator team (we use general characteristic of operators rather than individual operators), the process itself and human factors presentation knowledge. Each of these sources is captured as an agent, which takes part in negotiations with other agents to reach an agreement on the best form of adaptation. A Cognitive Resolver agent, whose job it is to query and broker the negotiations, facilitates this process. AMEBICA does not imbue any special AI capabilities within each agent, and adheres to the weak notion of agency [4]. The total system intelligence will therefore be the result of the collective negotiation and communication capacities of the agents.

AMEBICA - An Auto Adaptive Multimedia Environment

185

5. An Example Set of Interactions As an example, consider the actions following a message from the Process Model Agent to a Media Agent. Let us assume that a condition has occurred which requires the information rendered by the Media Agent to be given a much higher priority. (This example is merely one scenario and is highly simplified to illustrate a sample set of interactions.)

Rendering Operator Agent

Humm

Factor Datab~

Proces.q Model Agent

Figure 3: The AMEBICA conceptual architecture. The Process Model detects a condition that may require system adaptation and informs the Media Agent responsible for displaying a condition of that type. The representative Media Agent then queries the AMEBICA system and expects to be returned information on an appropriate rendering and its parameters. To do this it informs the Cognitive Resolver that it has a problem, and that the problem is one of increasing priority for its object (it would probably also describe this as an alarm condition).

186

AMEBICA - A n Auto Adaptive Multimedia Environment

The Rendering Resolution Agent will use its knowledge of context to produce a list of candidate representations. The list is passed to the Media Allocator Agent whose job it is to select the best representation class from the list based on current interface resources usage. It does this by utilising the Presentation Agent which returns information on interface usage. This information is used to determine which of the candidates is most suitable and which can be rendered at the interface. This information is then passed to the Media Agent which implements the rendering and it's parameters. If insufficient interface resources are available, the Media Allocator Agent has the power to alter the configuration of other Media Agents so as to allow the selected Representation to be rendered.

6. Conclusion AMEBICA attempts to deal with the problems of bandwidth-limited interfaces and information overload, by adapting the interface to display the most cogent information at the most appropriate times. It is hoped that by doing so AMEBICA can introduce a key element of flexibility at the interface, and that this flexibility is based on the state of the process, its model of the operator team, the environment and its knowledge of human factors. A multi-agent approach was adopted as it allows all the actors responsible for making decisions about suitable adaptations, to be represented by an autonomous agent. During the remainder of the project many important issues must be addressed such as ways to control and limit adaptation, the effect of positive feedback and ways to capture appropriate heuristics for the governing of agent-agent negotiation. These problems will he addressed along with specifying AMEBICA domain independent behaviour, and characterising domain dependant behaviour within the Process Model Agent and the Rendering Interface.

References [1] [2] [3] [4]

C.R.Dicken (1999) "Soft" Control Desks and Alarm Displays, lEE Computing and Control Engineering Journal, Vol 10, Number I , p l 1-16. Shackel, B. (1990) Human Factors and Usability in Preece, J. and Keller, L. (Eds.) Human-Computer Interaction, Hemel Hempstead, UK: Prentice Hall. M. Bransby and J. Jenkinson (Apr 1998),, lEE Computing and Control Engineering Journal, Vol. 9, Number 2, p61-67. M.Wooldridgeand N.R.Jennings(1995) Intelligent agents: Theory and Practice. The Knowledge Engineering Review, 10(2):1! 5-15.

Safety Culture

Rtldiger Trimpop Departement of Psychology, University of Jena

Abstract: While the influence of technical factors on accidents and disruptions of the work-flow is constantly diminishing, organisational factors become the focus of attention. The basic pattern of values, norms and behaviours underlying the structure and processes as well as decisions is called the culture of an organisation, of which safety climate and safety culture is a subset. Thus, safety culture is determined to a large degree by managerial actions and in turn determines, what behaviour employees show. This text defines the constructs, discusses the role culture and management play in a prevention system and gives examples of how safety culture can both be measured and influenced.

1. Accident Causation Accidents are the result of a multifaceted pattern of behaviours, dysfunctional and inadequate for the given situation at a specific point in time. Thus, the same behaviour can lead to accidents or rewards, e.g. speeding when late for an important meeting. Also, the same behaviour patterns in the same situation, such as two cars running a stop sign at an intersection, can cause a severe accident or nothing at all, if there was a sufficient time-lag between these behaviours. Thus, focussing only on technical features or individual behaviour has only a limited probability of success. Furthermore, as organisational structures, goals, visions and leadership styles are at the core of productivity, quality, health and safety, their combined influence on safety has to be considered as a key factor in both accident causation and prevention. As a consequence, safety culture will be closer examined as to its relation to management, decision-making and behaviour.

2. Safety Culture as a Subset of Organisationai Culture Organisational or corporate culture is a pattern of shared basic assumptions that a group of people learned as it solved its problems of external adaptation and internal integration, that has worked well enough to be considered valid and, therefore, to be

190

Safety Culture

taught to new members as the correct way to perceive, think, and feel in relation to those problems [I]. It manifests itself on three levels: 1. Artefacts: Obvious structures and processes, e.g. safety personnel. 2. Values: Strategies, goals, philosophies, e.g. safety=productivity 3. Basic assumptions: Unconscious, taken-for-granted ideas, opinions, feelings and values e.g., safety works as long as there is no time pressure. While safety cultures, of course, have existed since people engaged in any kind of organised behaviour under uncertainty, the term itself has not been coined or been of major interest until the Tchernobyl nuclear accident in 1986. Before, the concept of safety climate [2] was used, referring to the immediate and situational value system in a working unit as to safety policies and behaviour. Some [3] view climate as a summary of perceptions workers share about their work settings. Climate perceptions summarize an individual's description of his or her organisational experiences rather than his or her affective evaluative reaction to what has been experienced. Some studies examined safety climate empirically. Safety climate was for example measured in 20 industrial organisations in Israel [2], in 10 manufacturing and produce companies in the USA [4], in 9 construction sites in the USA[5] and in 16 construction sites in Finland [6 & 7]. At present, however, the term safety culture has replaced safety climate in the scientific community. A number of different definitions exist that partly lead to different consequences in designing systems and interventions.

2.1. Definitions of Safety Culture The safety culture of an organisation is the product of the individual and group values, attitudes, competencies and patterns of behaviour that determine the commitment to, and the style and proficiency of, an organisation's health and safety programmes. Organisations with a positive safety culture are characterised by communications founded on mutual trust, by shared perceptions of the importance of safety and by confidence in the efficacy of preventative measures. (IEE; Institution of Electrical Engineers of England [8]). Factors are: * leadership and commitment of the CEO 9 executive safety role of line management 9 involvement of all employees 9 effective communications 9 commonly understood and agreed goals 9 good organisational learning 9 responsiveness to change 9 manifest attention to workplace safety and health 9 a questioning attitude 9 a rigorous and prudent approach by all individuals

Safety Culture

191

ASCOT-Guidelines (IAEA [9]): 9 Awareness of safety culture issues 9 Commitment to good safety performance and to continuous safety performance evaluation 9 Commitment to good safety performance as an end in itself and not merely as a mean to satisfy regulatory requirements 9 Investigation of the fundamental causes of events and near misses 9 The examination of activities with potential safety impacts 9 A co-ordinated programme of regular safety audits 9 Efforts to learn from the safety performance of other organisations ILO-Encyclopaedia of Occupational Health and Safety [10] Safety culture is a concept that includes the values, beliefs and principles that serve as a foundation for the safety management system and also includes the set of practices and behaviours that exemplify and reinforce those principles. These beliefs and practices are meanings produced by organisational members in their search for strategies addressing issues such as occupational hazards, accidents and safety at work. These meanings (beliefs and practices) are not only shared to a certain extent by members of the workplace but also act as a primary source of motivated and co-ordinated activity regarding the question of safety at work. INSAG [11]: Safety culture in the nuclear industry is that assembly of characteristics and attitudes in organisations and individuals which establishes that, as an overriding priority, nuclear power plant safety issues receive the attention warranted by their significance. This is determined by organisational policies, managerial action and the response of individuals within the framework. It can be deduced that culture should be differentiated from both concrete occupational safety structures such as the presence of a safety department, of a safety and health committee etc. and existent occupational safety programmes, such as hazards identification activities such as workplace inspections, accident investigation, job safety analysis, etc. and finally integrated safety management systems. While accidents in the nuclear industry so have the potential to cause a major catastrophe, in office jobs the highest accident likelihood is that of stumbling and falling [12]. Thus, safety systems, procedures and management in the nuclear industry have to develop a culture, that makes the system foolproof and focus largely on organisational and technical aspects to reduce the error-influence of operators. Safety is on the constant focus of attention for everybody. In contrast, in office jobs the focus is more on the person him or herself to behave safety-conscious, while safety is not consciously considered often, as there are no obvious dangers, apart from those of day-to-day living. Here it is important to work with the people, making them more aware of dangers. Thirdly, in quick-moving, high-danger professions with many accidents, such as in logging or the construction business, dangers are unavoidable due to the nature of work and those working there have developed a much higher acceptance of

192

Safety Culture

dangerous situations and accidents as well as injuries. Here the underlying safety culture, values and norms have to be generally influenced on all levels of the company and the general type of industry as such.

3. Industrial Examples for Safety Culture Management Trimpop [ 13] differentiates 5 general types of safety management approaches: 9 Participatory Integrated Safety Management, e.g. Dow Chemical 9 Control-oriented safety management, e.g. Dupont 9 Goal-oriented safety management, e.g. Coal-mining industry 9 Individual Safety Leadership, e.g. Small and medium sized companies 9 Technical procedures oriented approach, e.g. Nuclear power plants These types of cultures, interventions and norms have been studied empirically. For example, Eakin [14] showed that in very small businesses, it is common that managers largely delegate responsibility for safety to workers. In a study of about one thousand first-line supervisors, Simard and Marchand [15] showed that a strong majority of supervisors are involved in occupational safety, though the cultural patterns of their involvement may differ. In some workplaces, the dominant pattern is hierarchical involvement and is generally control-oriented; in other organisations the pattern is "participatory involvement", because supervisors both encourage and allow their employees to participate in accident-prevention activities; and in some organisations, supervisors withdraw and leave safety up to the workers. A fairly consistent result of the studies in industrialised and developing countries, emphasises the importance of senior managers' safety commitment and leadership for safety performance [16 & 17]. Moreover, most studies show that in companies with low accident rates (e.g. Dow Chemical or Dupont), the involvement of top managers in occupational safety is as important as their decisions in the structuring of the safety management system, dealing with planning, resources and safety personnel. Indeed, workforce empowerment and active involvement are also documented as factors of successful occupational safety programmes [18]. At the workplace level, some studies offer evidence that effectively functioning joint health and safety committees significantly contribute to the firm's safety performance [19]. Similarly, at the shop-floor level, work groups that are encouraged by management to develop team safety and self-regulation generally have a satisfactory performance [13] Especially in occupational traffic safety, participatory approaches have proven to be very effective means [20 & 21]. Another powerful means of promoting an integrated safety culture among employees is to conduct perception surveys. Workers generally know of many safety problems, but rarely get involved in safety programs. Such a survey can be done using an interview method combined with anonymous questionnaires [22 & 23]. The

Safety Culture

193

survey follow-up is crucial for building an integrated safety culture. Once the data are available, top management should create changes through work groups. This will ensure commitment and adherence to the initiated changes. Such a perception survey should be continuously repeated and adds to a culture of continuous safety and health improvement. One such programme has been studied scientifically and proven successful. It is the TOP 10-Programme of Dow Chemical [18]. Here workers identified the 10 most dangerous behaviours and conditions, developed plans to reduce the dangers, agreed in group discussions among different shifts and with supervisors on the best procedures, trained these procedures and measured their effectiveness. This was considered an important component of the existing safety culture as virtually every employee was involved hunting for safety improvements, training and comparing results. When comparing effectiveness of safety programs at the safety culture level, one has to come up with a set of measures that describe the given cultural features.

4. Measurement of Safety Culture The following list is a sample of measures that have been proven successful in determining organisational culture and safety culture as well. 9 Document analysis: The internal papers, brochures, guidelines of an organisation are examined as to their referral to health and safety, the amount of space, the underlying connotations, the importance and the portrayed value of safety. 9 Guidelines, Code of Conduct: Here it is examined whether procedures and guidelines for machinery, maintenance and repair etc. emphasise health and safety issues, as well as access to and use of these guidelines are measured. 9 Company property, Safety office: The classic Cleanliness and order of a work-place are observed as well as location of safety offices, interconnections and visibility of safety information as well as of safety officers. 9 Questionnaires: Some questionnaires exist to assess safety climate and culture. Zohar [2] developed seven sets of items that were descriptive of organisational events, practices and procedures and differentiated high- from low-accident factories. Brown and Holmes [4] used Zohar's 40-item questionnaire, and found a three-factor model instead of the Zohar eight-factor model. Dedobbeleer and B61and [24] used nine variables to measure the threefactor model of Brown and Holmes. The variables were chosen to represent safety concerns in the construction industry and were not all identical to those included in Zohar's questionnaire. A two-factor model was found.

194

Safety Culture

9 Interviews: Interviews offer the chance to examine causal features for the given circumstances in more depth, but can hardly be performed anonymously. When interpreting interviews one has to adhere to quite rigorous standards for analysing qualitative data to make results comparable. 9 Systematic observations: Here the focus lies on objective observation of certain behaviours, indicative of safety culture. For example, are safety instructions given, how long, by whom and how. Are safety rules adhered to, are supervisors role-model for safety or safety violations. Comparable standards and objective observations allow for the most effective measures but are also of the highest costs, both timewise as well as financially. 9 Hierarchy of procedures: Is there a clear-cut hierarchy of procedures and systems, such as quality before productivity before safety considerations. 9 Involvement in decision processes: How many people from what hierarchical level of the organisation are involved in the decision making process. The higher the degree of participation the higher the likelihood of both detecting errors and safety hazards as well as adherence to self-developed safety standards. 9 Distribution of responsibility: Similar to the decision process responsibility has to be shared by as many hierarchical levels as possible. While the owner is responsible by law and the worker due to suffering from the injuries, all middle level management have to feel responsible and every one in the organisation has to realise that the safety officers are not responsible for producing safety but for giving advice for all other who produce safety or danger with their orders or behaviour. 9 Organisational Charts, Sociogram: An organisational chart shows who, what department and what communication process is officially designed to deal with safety information and co-operation. Very often, however, the theoretically planned relationships do not coincide with reality. That is were a sociogram has its place, plotting what actual communications and cooperations take place and in what kind of quality. 9 Jokes and Stories: Leaving the observable level of indicators for safety culture, jokes and stories can be interpreted as to their meaning. For example, the farewell speech for a retiring manager read as follows: "After he was not capable to work in the production line anymore, he still served the company for many years as the head safety engineer". Stories, heroes (e.g. production hall acrobats without safety net) and jokes reveal the value of safety within an organisational culture.

Safety Culture

195

9 Metaphors, Symbols, Pictures: Similar to jokes and stories, metaphors and symbols reveal underlying thoughts, fears, values and cultural norms. Is a safety system described with a metaphors such as: "All are equal, but some are more equal, or: team work is when everybody does what I want". One can assume a certain quality of culture. The same holds true for pictures and symbols a working unit uses to describe a situation, such as a sinking ship or a shining star. It is important to ask those who give the description as to their interpretation, because investigators themselves might use their own cultural interpretation while the organisational frame at hand has a different interpretative set.

5. Changing Safety Culture Safety culture can be changed on different levels, i.e.: Legislation, regulation Regulatory, supervisory boards Safety officers Employees Management If one looks at the safety regulations within the nuclear industry, or the standards within some Scandinavian countries and compares these to other industrialised and developing countries, the results of legislation and the enforcement of it becomes obvious. Regulations within a company can also make a remarkable difference in culture and safety records, as the chemical company Dupont shows, where a rigorous, control-oriented safety standard leads to excellent safety records and a climate or culture of total safety observation. Regulatory and supervisory boards must set these standards and control them, while the initiative, knowledge and communicative and motivational skills of safety officers lead to major differences both in safety cultures as well as safety records. That employees themselves can make the difference becomes evident in the company Dow Chemical, where virtually every worker is involved in a voluntary health and safety circle and/or task force group, leading to an equally excellent safety record as reached by Dupont with a control-oriented approach. Finally, without management commitment there is little chance of a positive safety culture to develop.

Safety Culture

196

Trimpop [I 3] identifies 5 leverage points to initiate change: 1. 2. 3. 4. 5.

Individual (motivation, identification, information) Group (responsibility, cohesion, team-leadership) Organisation (guidelines, management, structures) Communication (openness, vertical/horizontal) Co-operation (integration, transformation, success)

On each of these leverage points, one can start culture-changing activities but ultimately one has to work on all five levels to create lasting and effective changes. Without organisational structures, new standards and procedures will fall back into the old habits, without individual motivation, new structures have no effect, without communication new values are not transported and without co-operation they are not lived. Finally, each group sets and develops its own sub-culture. If that differs remarkably from the general safety culture, one has to decide which subset is the one aimed for by the organisation as a whole. Petersen [25] provides the following checklist for a positive safety culture with six criteria that can be met regardless of the style of management of the organisation, whether authoritarian or participative, and with completely different approaches to safety: 1. A system must be in place that ensures regular daily pro-active supervisory (or team) activities. 2. The system must actively ensure that middle-management tasks and activities are carried out in these areas: ensuring subordinate (supervisory or team) regular performance ensuring the quality of that performance engaging in certain well-defined activities to show that safety is so important that even upper managers are doing something about it. 3. Top management must visibly demonstrate and support that safety has a high priority in the organisation. 4. Any worker who chooses to should be able to be actively engaged in meaningful safety-related activities. 5. The safety system must be flexible, allowing choices to be made at all levels. 6. The safety effort must be seen as positive by the workforce.

Safety Culture

197

6. Conclusions Rensis Likert [26] showed that the. better an organisation is in core-aspects, the more likely it will be successful in economic success, and thus in safety. These climate variables are as follows: 9 9 9 9 9 9 9 9 9

increasing the amount of worker confidence and managers' general interest in the understanding of safety problems giving training and help where and as needed offering needed teaching as to how to solve problems providing the available required trust, enabling information sharing between management and their subordinates soliciting the ideas and opinions of the worker providing for approachability of top management recognising workers for doing good jobs rather than for merely answering.

Revenue gr0w..th. 9 Employment growth ,Stock Price growth: Net Income growth

Adaptive corporate Non-adaptive corporate cultures cultures 682 % 166% 282 % 36% 901% 74% 756 % 1%

Weak corporate cultures 3% 0% 5% -10%

Table 1: Difference in business success between organisations with a change-oriented, a stability-oriented and a weak corporate culture (N=500; 1977-1988) Indications that this approach holds true come from a large scale longitudinal study [27]. For more than 10 years 500 companies with a strong a weak and an adaptive culture were examined as to the relation of culture and success (see Table 1). The Criteria of Success they found for the adaptive cultures were: 9 A culture of continuous change 9 Openness for changing environments 9 ,,Care" about members, customers and owners 9 Stability in core qualities Transferred to safety culture, none such study exists so far, but if safety culture is a subset of organisational culture the same core qualities should be pursued by safety oriented organisations.

198

Safety Culture

Peterson [25] summarises the importance of safety culture as follows: It is not which elements are used that determines the safety results; rather it is the culture in which these elements are used that determines success. In a positive safety culture, almost any elements will work; in a negative culture, probably none of the elements will get results.

References [1] [2] [3]

[4]

[5] [6] [7] [8] [9] [10] [11] [12] [I3] [14]

[151

[16]

[17]

[18]

Schein, EH. 1985. Organisational Culture and Leadership. S.F.: Jossey-Bass. Zohar, D. 1980. Safety climate in industrial organizations: Theoretical and applied implications. Journal of Applied Psychology 65, 96-102. Dedobbeleer, N and Briand, F. 1989. The interrelationship of attributes of the work setting and workers'safety climate perceptions in the construction industry. In Proc. 22rid Annual Conference of the Human Factors Association of Canada. Toronto. Brown, R and Holmes, H. 1986. The use of a factor-analytic procedure for assessing the validity of an employee safety climate model. Accident Analysis and Prevention 18 445-70. Dedobbeleer, N and Brland, F. 1991. A safety climate measure for construction sites. Journal of Safety Research, 22, 97-103. Mattila, M, Hyttinen, M and Rantanen, E. 1994. Effective supervisory behavior and safety at building sites. International Journal of Industrial Ergonomics, 13, 85-93. Mattila, M, Rantanen, E. and Hyttinen, M. 1994. The quality of work environment, supervision and safetyin building construction. Safety Science, 17, 257-268. IAEA, 1993, ASCOT Guidelines of the International Atomic Energy Commission lEE, 1996, International Electrotechnical Engineers. Safety Culture Simard, M. 1998, Safety culture and management. In J. M. Stellman (Ed.), ILO Encyclopedia of Health and Safety, 4th Ed. Geneva: ILO Press. 59.2-4. INSAG: Safety culture. Wien. 1991 Trimpop, RM 1994. The Psychology of Risk Taking Behavior. Amst.: Elsevier. Trimpop, R.M., 1999 Organisationaler Wandet im Arbeits-, Verkehrs-, Gesundheitsund Umweltschutz. (Nord-West Verlag). Eakin, JM. 1992. Leaving it up to the workers: Sociological perspective on the management of health and safety in small workplaces. International Journal of Health and Safety, 22, 689-704. Simard, M& Marchand, A. 1994. The behaviour of first-line supervisors in accident prevention and effectiveness in occupational safety. Safety Science 19 169-184. Zimolong, B and R Trimpop. 1994. Managing human reliability in advanced manufacturing systems. In Design of Work and Development of Personnel in advanced Manufacturing Systems, edited by G Salvendy and W Karwowski. New York: Wiley. Shannon, H, Waiters, V, Lewchuk, W., Richardson, J. Verma, T. Haines, T and Moran, L. 1992. Health and safety approaches in the workplace. Unpub.report. Toronto:McMaster Jacobus, L., Rohn, S. und Trimpop, R. (1998). Evaluation des Ansatzes "'Faktor Mensch in der Arbeitssicherheit" der DOW Chemical A.G.S. 335-340. In H. von

Safety Culture

[19]

[20]

[21]

[22] [23] [24]

[25] [26] [27]

199

Benda & D. Bratge (Hrsg.). Psychologie der Arbeitssicherheit: 9. Workshop. Heidelberg: Asanger. Chew, DCE. 1988. Quelles sont les mesures qui assurent le mieux la s~curit~ du travail? Etude men6e dans trois pays en d6veloppement d'Asie. Rev Int Travail, 127, 129-145. Trimpop, R., Adolph, L. & Rabe, S., 1996. Evaluation betrieblicher Verkehrssicherheitsmaflnahmen im Rahmen eines integrativen Gesundheitsmanagementansatzes. In B. Ludborzs, H. Nold, & B. Ri~ttinger (Eds.), Psychologie der Arbeitssicherheit. 8. Workshop 1995 (401- 414). Heidelberg: Asanger. Gregersen, N. P., Brehmer, B. & Moren, B. (1996). Road safety improvement in large companies: An experimental comparison of different measures. In Swedish Road & Transport Research Institute (Eds.), Accident Analysis and Prevention, 28, 297- 306. Bailey, C. 1993. Improve safety program effectiveness with perception surveys. Professional Safety, 10, 28-32. Petersen, D. 1993. Establishing good "safety culture" helps mitigate workplace dangers. Occupational Health and Safety, 62, 20-24. Dedobbeleer, N, B61and & German. 1990. Is there a relationship between attributes of construction sites and workers' safety practices and climate perceptions? In: Advances in Industrial Ergonomics and Safety, D Biman (Ed.)Taylor& Francis. Petersen, D., 1998 Safety policy, leadership and culture. In J. M. Stellman (Ed.), ILO Encyclopedia of Health and Safety, 4th Ed. Geneva: ILO Press. 59.2-4. Likert, R. 1967. The Human Organization. New York: McGraw Hill. Kotter, J.& Heskett, J. (1992). Corporate Culture and Performance. Free Press.

Study of Errors by Means of Simulation and Training

Asgeir Dr~ivoldsmo OECD Halden Reactor Project, P. O. Box 173, N- 1751 Halden, Norway, Email: [email protected]

Abstract: The ongoing Human Error Analysis Project (HEAP) was initiated within

the OECD Halden Reactor Project in 1994. Its objectives are to develop a better understanding and explicit model of how and why cognitive errors occur, and to provide design guidance to avoid, or compensate for, cognitive errors. During the project period, results have lead to practical insights concerning method development, and investigation of, simulation as a way of investigating how operators diagnose cognitively challenging scenarios.

1. Introduction A main problem, in the study of complex operating situations, is that it is not clear what should actually be measured. In the special case of a program studying "human error," one important question is how to find good measurable indicators describing the origin of human erroneous actions. This question is independent of whether one chooses to use classification [1], [2], deeper qualitative analysis, or more general performance indicators like, i.e., operator performance [3]; plant performance [4]; situation awareness [5], [6] as the operational tools for investigation of the topic. Fortunately, without entering the ambiguous variety of definitions and classifications of human error, one can agree that a basis for this research lies in collection of good and reliable data with a potential for aggregation upon a level where meaningful analyses can take place. One way of acquiring such data is to use simulations. Data are then depending on a realistic simulations and tasks, and a representative population of operators (subjects). It is important to include these requirements to comply with the validity criteria that should guide methodologies in this domain [7], [8]. This article will describe the HEAP application of simulation as a tool for gathering good and valid data with the purpose of studying human erroneous actions.

180

Advanced User Interface Design for Aircraft Cockpit Devices

References [1] [2]

[3]

Domheim M., 1996, Recovered FMC Memory Puts New Spin on Cali Accident. Aviation Week & Space Technology, 36, pp. 58-61. Salter N.B. and D.D. Woods, 1992, Pilot Interaction with Cockpit Automation: Operational Experiences with the Flight Management System. The International Journal of Aviation Psychology, 2, pp. 303-321. Sarter N.B. and D.D. Woods, 1994, Pilot Interaction with Cockpit Automation II: An

Experimental Study of Pilots'Model and Awareness of the Flight Management System. [4]

[5]

The Intemational Journal of Aviation Psychology, 4, pp. 1-28. Marrenbach J., et al, 1997, Konzept zur ergonomischen Gestaltung der Benutzungsoberfliiche eines zukiinftigen Flight Management Systems. Jahrbuch der Deutschen Gesellschaft ftir Luft- und Raumfahrt, I, Bonn, pp. 497-505. Marrenbach J. and M. Gerlach, 1999, Validierung des Benutzerfiihrungskonzepts eines zukiinftigen Flight Management Systems. Jahrbuch der Deutschen Gesellschaft f(ir Luft- und Raumfahrt, III, Bonn.

Study of Errors by Means of Simulation and Training

203

rate of information about the operators' attention [14], [15]. Used with care, verbal protocols are powerful tools for insight into problem solving [16]. Four different types of verbal data have been investigated. These four are operator concurrent verbal protocols, operator interrupted verbal protocols, operator retrospective verbal protocols (auto-confrontation) and SME concurrent verbal protocols [17], [18], [19], [20]. The evolution of new and better technology for EMT (Eye Movement Tracking) has given the opportunity to use this method in HaMMLab experiments without interfering with the operators' problem solving. This is a very powerful source of information about the operators' voluntary visual information gathering, and combined with verbal protocols, EMT can contribute to better accuracy [21], [22], [23]. Promising results from EMT analysis have also been achieved in the development of measures of tracing cognitive activity [24], and [25].

5. Scoring and Analysis of Performance Data One important lesson learned in HEAP is the need for a close co-operation with SME throughout the whole scoring and analysis phase of an experiment. There is no way the complex process like the one found in a nuclear power station can be sufficiently learned and understood of research staff without hands on experience. Scoring and analysis where process knowledge comes into consideration should always be guided by subject matter expertise. A common goal for the data collection, scoring and analysis of data in HEAP has been to make the performance scores comparable across scenarios, and sensitive to different levels of problem complexity. To achieve flexibility in the scoring, and to meet different demands for details, data have been recorded on the highest practical resolution and attached to a common timeline for all data sources. The cost of data gathering implies that one will always have to live with a limited number of subjects [26]. Data scoring connected to the time line allows expanding the use of within subject design, through use of time-windows based on important events or stages in the scenarios. Within-subject designs used across scenarios have also proven useful in the analysis in situations where effects are scenario dependent, although interpretation can be difficult when such explorative techniques are employed [27].

6. Conclusions HaMMLab experiments are better described as realistic simulations applying a set of experimental control mechanisms, than traditional psychological laboratory

204

Study of Errors by Means of Simulation and Training

experiments carried out in a nuclear power plant simulator. The experimental methodology developed in HEAP has aimed at data collection and meaningful analysis within the natural context of simulation, preserving the access to the powerful tools of experimental designs. At the same time as the hypothetical - deductive approach has been the central paradigm in most HEAP experiments, HEAP can be said to be inductive in its underlying research strategy. Technical development allows more and faster data registration and analysis. Tracing both the operator and the process together, through data from e.g., eye-movement tracking, operator activity logs, process logs, audio and video, have opened a new path into the study of operator problem solving. The HEAP methodology has been developed to satisfy the need for objective and continuous data collection in the complex operating situation of a nuclear power plant scenario. Using a controlled experimental approach, in combination with maximised realism, has not been an obstacle for in-depth study of selected operator activities in a representative situation. Future work in the HRP (Halden Reactor Project) will focus on how to simplify and make the methods developed more easily accessible for use in general simulatortraining situations and control room validation testing. The first attempts in this direction are already taking place in the validation study of the human-machine interface for Oskarshamn Reactor unit one. Several tools have been tested in eight crews, running five scenarios at the training centre KSU in Sweden. Results from this study will be presented after a final data collection in year 2000.

References [ 1] [2] 13] [4] [5] [6]

[71 [8]

Reason, J. (1990). A frcvneworkfor classifying errors, in J. Rasmussen, K. Duncan, and J. Leplat (Eds.), New Technology and Human Error. London: John Wiley. Hollnagel, E. (1993). The phenotype of erroneous actions. International Journal of Man-Machine Studies, 39, 1-32. Skraaning, G. (1998). The Operator Performance Assessment System (OPAS) (HWR538). OECD Halden Reactor Project, Halden, Norway. Moracho. M. J. (1998). Plant Performance Assessment System (PPAS) for crew performance evaluations. Lessons learned from an alarm study conducted in HaMMLab (HWR-504). OECD Halden Reactor Project, Halden, Norway. Endsley, M. R. (1995). Toward a Theo~ of Situation Awareness in Dynamic Systems. Human Factors, 37, 32-64. Hogg, D., Follesr K., Strand-Volden, F. et al. (1995). Development of a Situation Awareness Measure to Evaluate Advanced Alarm Systems in Nuclear Power Plant Control Rooms. Ergonomics, 38( 11), 2394-2413. Fracker, M. L. (1991) Measures of Situation Awareness: Review and Future Directions. AD-A262 672, National Technical Information Service, Springfield,

Virginia, 22161. American Institute of Aeronautics and Astronautics, (1992). Guide to human performance measurement. (A1AA Publication No G-035-1992). Washington, D.C.

Study of Errors by Means of Simulation and Training

[9] [10] [11] [12]

113] [14] [15] [16] [17] [18] [19]

[20]

[21]

[221 [23]

[24]

[25]

[26]

205

Follesr K., Drr A., Kaarstad, M., ct al. (1995). Human error - the Third Pilot Study (HWR-430). OECD Halden Reactor Project, Halden, Norway. Weber, M. J. (1999). Goal Conflicts in the Process Control of a Nuclear Power Plant. Diploma thesis submitted to the Department of Psychology, University of Bern, Switzerland. Braarud, P. 0. (1998). Complexity factors and prediction of crew performance (HWR521). OECD Halden Reactor Project, Halden, Norway. Braarud, P.O. Complexi O' rating of abnormal events and operator perfotmance. CNRA/CSNI Specialists' Meeting on Human Performance in Operational Transients, Chattanooga, Tennessee, 1997. Collier, S. G. (1998). Development of a Diagnostic Complexity Questionnaire (HWR536). OECD Halden Reactor Project, Norway. Newell, A. and Simon, H. A. (1972). Human Problem Solving. Englewood Cliffs, N.J.: Prentice-Hall, 1972 Ericsson, K.A. and Simon, H.A., (1980) Verbal Reports as Data. Psychological Review; 87: 215-251. Pretorius, N. and Duncan, K. D. (1986). Verbal Reports in Psychological investigations: a Logical and Psychological Analysis. Psyke & Logos, 7, 259-287. Kaarstad, M., Kirwan, B., Follesr K. et al. (1994). Human Error - the First Pilot Study (HWR-417). OECD Halden Reactor Project, Halden, Norway. Kaarstad, M., Follesr K., Collier, S. et al. (1995). Human Error - the Second Pilot Study (HWR-421). OECD Halden Reactor Project, Halden, Norway. Hollnagel, E., Braarud, P. O, Dr~ivoldsmo, A. et al. (1996).The Fourth Pilot Study: Scoring and Analysis of Raw Data Types (HWR-460). OECD Halden Reactor Project, Halden, Norway. Braarud, P. 0., Drr A. and Hollnagel, E. (1997). Human Error Analysis Project (HEAP) the Fourth Pilot Study: Verbal Data for Analysis of Operator Performance (HWR-495). OECD Halden Reactor Project, Halden Norway. Hauland, G. (1996). Building a Methodology for Studying Cognition in Process Control: a Semantic Analysis of Visual Verbal Behaviour. Post-Graduate Thesis in Psychology, Norwegian University of Technology and Science, Trondheim, Norway. Kaarstad, M., Follesr K., Collier, S. et al. (1995). Human Error - the Second Pilot Study (HWR-421). OECD Halden Reactor Project, Halden, Norway. Hauland, G. & Hallbert, B. (1995). Relations between visual activity and verbalised problem solving: a preliminary study. In: Leena Norros (Ed.), VTI" symposium 158, the 5 European conference in cognitive science approaches to process control, Espoo, Finland, pp. 99-110. Drr A., Skraaning, J., Sverrbo, M., et al. (1998). Continuous Measures of Situation Awareness and Workload (HWR-539). OECD Halden Reactor Project, Halden, Norway. Skraaning, J. and Andresen, G. (1999). The Function of the Alarm System in Advanced Control Rooms: An Analysis of Operator Visual Activity during a Simulated Nuclear Power Plant Disturbance. Paper at the Enlarged Halden Programme Group Meeting at Loen 1999, OECD Halden Reactor Project, Halden, Norway. Drr A., Follesr K., HolmstrCm, C. et al. (1995). The Halden Project Workshop Meeting on Evaluation Methods and Measurements, and Data Analysis in System Test and Evaluation (HWR-431). OECD Halden Reactor Project, Halden, Norway.

206

Study of Errors by Means of Simulation and Training

[27] Miberg, A. B., Hollnagel, E., Skraaning, G. et al. (1999). The impact of automation on operator performance. An explorative study. Preliminary report. Institute of Protection and Nuclear Safety (IPSN), Paris.

Operator Training and Implication for the Practice

Horst-Gtinther St0renburg KRAFTWERKSSCHULE E.V. (KWS), Klinkestrasse 27-31, D - 45136 Essen, Germany e-mail: [email protected]

Abstract: Innovative technologies call for qualified personnel. Permanent retraining

of the technical personnel is necessary. In the past years, reduction of power plant personnel was intensifying. Training is only possible in an intensive, guided manner. Shift supervisors (Kraftwerksmeister) have been trained at KWS (Kraftwerksschule e.V.) since 1957. On simulators employees have the possibility to learn to apply new power plant technologies in a risk-free environment. The ongoing dialogue between maintenance and operation personnel can be improved by sharing experience at the simulator. New MMIs and supervision and control systems change the work of operators (Kraftwerker) and shift supervisors in the power plant.

1. Introduction Innovative technologies call for qualified personnel with know-how corresponding to the actual state-of-the-art of power plant technology. Only by permanent retraining of the technical personnel it is possible to meet these high requirements on a long-term basis. Due to the change in power plant technology and control systems in the years since 1950 also the methods and tools for training of power plant personnel needed a change. The increasing standard of automation went hand in hand with a reduction of personnel in power plants. Training and education were structured and formalised in this period so that in 1957 the KWS was founded by the utilities in West Germany.

208

Operator Training and Implicationfor the Practice

Fig. I: 1957 (50 MW plant)

Fig. 2: 1975 (300 MW plant) Fig. 3: 1997 (2*450 MW plant) Figure I-3: Design of control panels in steps of 20 years since 1957. In 1977 the first two simulators for nuclear power plants were established in the KWS building in Essen. In 1986 a simulator for fossil fired power plants followed because of the good experience achieved with the two first simulators. Training and education of power plant personnel are divided in two major parts, the theoretical/practical education and the simulator training. To reach a high and efficient level of training, leading personnel from the utilities is permanently involved in developing learning objectives, examination standards and teaching.

2. Theoretical and Practical Training 2.1. Basic Training for Operation and Maintenance Personnel Correct operation and maintenance of the plant is the direct prerequisite for the compliance with legal regulations. All interventions in the power plant process require a proper grasp of the entire plant and an understanding of the functions of plant systems. Training is only possible in an intensive, guided manner. Professional training has a theoretical and a practical part. It covers a broad spectrum of subjects and is structured by the collection of material, learning objectives, fields of employment and examination standards. Unit and shift supervisors employed in the operation of a power plant have leading positions with a high responsibility for safe, environmentally compatible and economic operation. Thus, not only a deepened know-how of the process and its automatisation, but also an insight into economic and legal connections and a suitability for the guidance and training of the committed employees are required.

Operator Training and Implicationfi)r the Practice

209

Operator Training (Kraftwerkerausbildung) Operator training needs three years. The first two years are filled with practical training on site. During this period the shift supervisor is teacher and advisor in one person. After these two years, a four months theoretical training provides the theoretical background. The last year of training is necessary to implement the theoretical knowledge into practical work on site. At the end, an operator is in charge of operating the plant in a correct manner. Shift Supervisor Training (Kraftwerksmeisterausbildung) Shift supervisor training has a duration of one year at KWS in Essen. This training consists of the following elements: 9a professional training part 9a field-comprising part with the subjects costs, law and guidance of employees 9a field-specific part with fundamental and technical subjects followed by an examination at IHK (Chamber of Industry and Commerce).

Fig. 4: KWS-classroom Fig. 5 : KWS-thermo Fig. 6: KWS-electrical training dynamical laboratory laboratory Figure 4-6: KWS facilities for theoretical/practical training in Essen, Germany.

2.2. Enhanced Training Know-how and skills decrease, if they are not used regularly, refreshed and adopted to the state-of-the-art. A couple of advanced courses and seminars are developed to support the brushing up of know-how. These courses reflect the needs of power plants.

3. Simulator Training for Power Plant Personnel Well-trained employees capable of recognising quickly and analytically and eliminating problematic operational situations are a prerequisite for the safe operation of modern power plants. Base load power plants for example, which are especially common in lignite coal mining regions do not offer many training opportunities for start-up and shutdown

210

Operator Training and Implication for the Practice

procedures. The analysis of problematic scenarios enhances the understanding of power plant operations.

Fig. 7: Simulator 2 with hard Fig. 8: Briefing panel (desk panel)

Fig. 9: Simulator 1 with graphical user interface (GUI) Figure 7-9: KWS facilities for simulator training in Essen, Germany. Employees can learn to apply new power plant technologies in a risk-free environment. In addition, employees can practice new operating techniques during start-up and shutdown procedures and practice the handling of problematic scenarios. The ongoing dialogue between maintenance and operation personnel can be improved by sharing experience of normal operations and problem situations at the simulator. Over the past decade, many customers of power plant manufacturers received preparatory simulator training for the operation of their new plants. Simulators have been especially popular for retrofitting and upgrading measures. New Supervision and control systems change the work of operators. This makes it necessary to supply simulators with this modern systems to conform to the state of the art in power plant technology.

References [1] [2] [3]

[4]

Broisch, A., St0renburg, H.-G., Ten Years of Simulator Training for Fossil Fired Power Plants, 1996, VGB-Kraftwerkstechnik 76, D-45136 Essen, Germany Stabler, K., Die Ausbildung des Kraftwerkspersonals, 1994, VGB-Kraftwerkstechnik 74, D-45136 Essen, Germany Schlegel, G., Seibert, D., StOrenburg, H.-G., Die Ausbildung des Betriebspersonals unter besonderer BerOcksichtigung de Umweltschutzes Jahrbuch der Dampferzeugungstechnik; Vulkan-Verlag Essen; 7. Ausgabe 1992 Schlegel, G., St0renburg, H.-G., Die Ausbildung des Betriebspersonals unter besonderer BerOcksichtigungder Simulatorausbildung, Jahrbuch der Dampferzeugungstechnik; Vulkan-Verlag Essen; 6. Ausgabe 1988/8

Function Distribution between Man and Machine: Experiments performed in FANSTIC II

Bemd-Burkhard Borys Systems Engineering and Human-Machine Systems, University of Kassel, 34109 Kassel, Germany. e-mail: borys @imat.maschinenbau.uni-kassel.de

Abstract: In experiments with a Multi-Attribute Task Battery we modified the

number and type of the tasks assigned to a human and took two workload measures and one measure of steadiness of task execution. We found that performance on two tasks in parallel differs from expectations derived from performance on single tasks.

1. Introduction The experiments described below were performed in Brite/EuRAM project FANSTIC II in which we evaluated the impact of changes in future air traffic. The high and increasing degree of aircraft automation combined with the need to keep the pilot in the loop gives room for various different function distributions between pilots and automatic systems. Dependency of performance from function distribution during parallel operation of multiple tasks was the focus of our study. A tool for investigation of human performance on several tasks in parallel is the Multi-Attribute Task Battery (MATB) [ 1].

2. The Experiment In our experiments, we used MATB to vary the number and kind of the tasks assigned to the human. We wanted to find out which combinations of tasks should be allocated to the pilots, leaving the others to the aircraft automation. The following functions usually become automated in aircraft cockpits: (a) attitude control by the autopilot and the vertical and lateral guidance by the flight management system; (b) calculations, like take-off or landing distance; (c) knowledge based operator support, for example for failure diagnostics; and (d) supervisory monitoring, for example by the Ground Proximity Warning System [2]. In accordance to original MATB from [1], we tried to cover this task spectrum with

212

Function Distribution between Mat, and Machine

five independent tasks. A tracking task as well as stepwise selection of new waypoints covered (a) and according to (b), simple arithmetic problems were to be solved. A simulated fuel management during a flight covered (c) and to cover point (d), we implemented a monitoring task. Because interested in workload, we used the calculation task as a secondary task. As the manual control task we implemented compensatory tracking of a second order system. A marker was displaced and to be returned to the centre using a joystick. The management task represented the fuel management of an aircraft with five tanks and eight fuel pumps. Subjects had to hold the tank levels in certain limits by switching the pumps. In the navigation task, subjects had to determine the shortest path to destination and enter the next waypoint. In the supervisory task, subjects had to monitor ten systems states. As the secondary tasks, subjects must perform simple calculations while instructed to work on the secondary task only when time permits, giving the other tasks priority.

3. Implementation and Design Tasks were implemented on three PCs. Their screens were arranged similar to a glass cockpit, a joystick was on the left, special keyboards to the right of the subject, the keys arranged according to the tasks' requirements. For experimental control, data collection, and supervision, we used a fourth PC in an observation room. A stand-alone network connected all four PCs. A first series of experiments involved six pilots. A second series of experiments involved five pilots and five students of mechanical engineering. Most of the 11 pilots held licenses for single-engine planes, three with additional IFR, and two with instructor rating. Three subjects held licenses for commercial and airline transport. We carefully evaluated the question whether it is correct to include students in the experiments. Indeed, students showed different performance on the tasks. Mainly in the manual control task, tracking error (RMS of deviation) was much lower for the students on the cost of higher joystick activity (RMS of deflection - influence of video games?). Also, the measure for scanning irregularity (described below) was significantly lower for pilots. However, we were interested not in absolute values of performance, but in changes of performance and behaviour on variation of function allocation. We used three measures to assess performance: A subjective workload score based on a questionnaire, objective workload ("calculation score") and a score for scanning irregularity, the latter two derived from performance on the secondary task (number of calculations performed and standard deviation of calculation times used). Correctness of the calculation result was of no concern, as only few calculations were incorrect. Indeed, in more than half of the experiments no error was made.

Function Distribution between Man and Machine

213

4. Results All measures showed large inter-individual differences and were Z-transformed resulting in "scores" of 0.0 mean and standard deviation 1.0 for each subject. The secondary task also showed a training effect, which was compensated when calculating the scores. Changes of these scores under different task conditions were evaluated further. The subjective workload score did not show significant differences with the task, when performing one task manually. It did when performing two or three tasks in parallel: Workload was lower when involving the monitoring or navigation task. Nay x

LowWorkload

Mon

Ctr~woTasks SingleTask~ Nl~!-.c~n Mgt ~ _ _ _ _ _ ~ . . ~ - ~

,gh Workload~

Ctr-Mgt

~~ t'

Figure 1: Workload trends in single task and two-task situation Regarding the objective workload measure "calculation score", workload was highest (score was lowest) when performing the management task manually while the other three tasks showed lower workload in the one-task-manual design. However, these experiments showed that the workload of two tasks performed in parallel could not simply be derived from single-task results. The following figure shows calculation scores for the single-task situation (left) and the two-tasks situation (right, on a separate scale) and indicates trends in workload when adding a second manual task by the connecting lines. Combining two of the low-workload tasks (navigation Nay, monitoring Mon, and control Ctr) also results in a low-workload combination. But combining the management task (Mgt) with other tasks showed existing interdependencies between tasks: Although navigation as a single task produced the lowest workload it is not the optimum choice when adding a second task to the management task.

214

Function Distribution between Man and Machine

An evaluation of the scanning irregularity reveals similar results; those combinations showing good results with respect to workload also show regular scanning patterns. For a transition from the two- to the three-task-situation comparable results could be expected. However, the number of combinations was too high compared to the number of experiments that could be performed to show significant differences.

5. Consequences for Future Systems The result shows that finding a suitable combination of manually performed tasks requires both regarding the workload caused by each task alone and regarding interference between the work on the appropriate tasks. Details can be found in [3, 4]. Although exact numbers and even trends cannot be generalised and similar experiments may be necessary for different task combinations these experiments show the importance of mutual influence between different tasks in a multi-task environment. For future evaluations of workload in experimental multi-taskenvironments, we would propose to classify tasks according to physical and cognitive resources needed and estimate changes in workload based on overlap of resource use. For a specific application like the future cockpits or the air traffic management system [5], we proposed to avoid task combinations that use the same resources and evaluate workload for candidate task combinations. With the new possibilities of function distribution between pilots, controllers, and aircraft and ground automation, careful planning of suitable task combinations is necessary.

References [I] [2] [3]

[4] [5]

Hilbum B.G., 1993, The Effect of Long- versus Short-Cycle Schedule on Performance with an Adaptively Automated System. Catholic University of America. Wickens C.D., 1993, Designing for Situation Awareness and Trust in Automation. IFAC Conference on Integrated Systems Engineering, pp 77-82, Baden-Baden. Tiemann M. and Borys B.-B., 1995, Verringemng der Belastung von Piloten durch ver~inderte Aufgabenteilung zwischen Mensch und Maschine. In H.-P. WiUumeit and H. Kolrep (Eds.), Verli~fllichkeit yon Mensch-Maschine-Systemen, pp 139-153. Berlin: Technische Universit~it. ISBN 3-7983-1650-3. Borys B.-B. and Tiemann M., 1995, Experiments for a new Distribution of Function. Chapter 3.5 in [5]. Borys B.-B (Ed.), 1995, Recommendations for a Human-Centred Air Traffic Management Systems. FANSTIC II Sub-Task 2.3 "ATM Implications" Synthesis Report, UKS-2.3-SR001 (Bo007/95). University of Kassel.

The work described herein has been undertaken by the author as part of the European Union FANSTIC 11 project, within the framework of the IMT Community Research Programme, with a financial contribution by the European Commission. The following companies,

Function Distribution between Man and Machine

2 I5

research centres, and universities were involved in the project: AEROSPATIALE, DAIMLER-CHRYSLER AEROSPACE AIRBUS, BRITISH AEROSPACE CA, FOKKER, SEXTANT AVIONIQUE, NLR, CENA, SMITH INDUSTRIES, VDO, ALENIA, CAP'I~C, NATIONAL TECHNICAL UNIVERSITY OF ATHENS, SPACE APPLICATIONS SERVICES, UNIVERSITY OF KASSEL, AIRBUS INDUSTRIES TRAINING, DEFENCE RESEARCH AGENCY, BRITISH AEROSPACE SOWERBY RESEARCH CENTRE, THOMSON-CSF, DORNIER, and FIAR.

Discussion Session V

In the final discussion of the workshop it was tried to deal with such aspects that had been touched upon in more than one paper but not dealt with in detail or that had implicitly surfaced as consequences of what had been said. One of them was the role of 'new technologies' like Virtual Reality, Multimedia etc. Sometimes the impression arises as if they are just applied for their own sake or because their usage by 'computer kids' would guarantee their role as carriers of progress. In the discussion it was widely agreed that this should be avoided. However, the potential of these media should not be underestimated. They may be able to give back to the users to a certain extent what has been taken away from them by the current VTU-based interfaces: non-visual cues like vibration, smell, unidentifiable noises. It was even remarked that in some cases users had not been aware of the existence and usefulness of such cues before they were taken away from them. This observation led to a rather fundamental discussion about the necessity and the proper use of such cues as a means of direct feedback. Examples of devices that triggered this discussion were e.g. the 'playstation joystick' for controlling a power station, the data glove, instruments for minimal invasive surgery, or the 'single joystick control' for cars. It turned out that some deeper questions have to be answered before the usefulness of any particular input device can be discussed. So e.g. it is of interest why a person has to control a technical system - as a professional, for fun, or casually. It is also makes a difference whether the person wants to operate the machine or control the effect of that process. In the first case direct feedback may be regarded as necessary, in the latter case rather indirect supervisory control may be the proper choice. In any case, however, operators want to be able to 'see through' the computer. Finally it was agreed that the discussion about the appropriateness of certain interface designs should neither be based upon a technological view ('the newest is the best') nor on the (rather human) tendency to believe in 'shortcuts' (like e.g. 'Multimedia is the solution', 'situation awareness is the true key to operator performance' or 'ecological interfaces are the only way to go'). Proper interface design should rather be based on a thorough analysis of the information requirements, tedious as this may be, should take into account the results of accident analysis in order to avoid the pitfalls of the past and take an asymptotic approach towards the use of emerging technologies.

Summarized by Peter F. Elzer