Optimal Control: Basics and Beyond

Optimal Control

WILEY-lNTERSCIENCE SERIES IN SYSTEMS AND OPTIMIZATION

Advisory Editors Sheldon Ross Department of Industrial Engineering and Operations Research, University of California, Berkeley, CA 94720, USA

Richard Weber Cambridge University, Engineering Department, Management Studies Group, Mill Lane, Cambridge CB21RX, UK

GITIINS- Multi-armed Bandit Allocation Indices K.ALLIWAILACE- Stochastic Programming KAMP/HASLER- Recursive Neural Networks for Associative Memory KIBZUN/K.AN -

Stochastic Programming Problems with Probability and Quantile Functions

VAN DIJK- Queueing Networks and Product Forms: A Systems Approach WHITILE- Optimal Control: Basics and Beyond WHITILE- Risk-sensitive Optimal Control

Optimal Control Basics and Beyond Peter Whittle Statistical Laboratory, University of Cambridge, UK

JOHN WILEY & SONS

Copyright © 1996 by John Wiley & Sons Ltd Baffins Lane, Chichester, West Sussex P019 IUD, England National 01243 nm1 International ( +44) 1243 TI97TI All rights reserved. No part of this book maybe reproduced by any means, or transmitted, or translated into a machine language without the written permission of the publisher. Cover photograph by courtesy of The News Portsmouth

Other Wiley Editorial Offrces John Wiley & Sons, Inc., 605 Third Avenue, New York, NY 10158-0012, USA Jacaranda Wiley Ltd, 33 Park Road, Milton, Queensland 4064, Australia John Wiley & Sons (Canada) Ltd, 22 Worcester Road, Rexdale, Ontario M9W ILl, Canada John Wiley & Sons (SEA) Pte Ltd, Yl Jalan Pemimpin #05-04. Block B, Union Industrial Building, Singapore 2057

Library of Congress Cataloging-in-PublictJtion Data Whittle, Peter. Optimal control: basics and beyond I Peter Whittle. p. em.- (Wiley-lnterscience series in systems and optimization) Includes bibliographical references and index. ISBN 0 471 95679 I (he : alk. paper). - ISBN 0 471 96099 3 (pb : alk. paper) I. Automatic control. 2. Control theory. I. Title. II. Series. TJ213.W442 1996 629.8- dc20 95-22113 CIP

British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN 0 471 956791; 0 471 96099 3 (pbk) Typeset in 10/12pt Times by Pure Tech India Limited, Pondicherry. Printed and bound in Great Britain by Biddies Ltd, Guildford and King's Lynn This book is printed on acid-free paper responsibly manufactured from sustainable forestation, for which at least two trees are planted for each one used for paper production.

Contents Preface

vii

1 First ideas

BASICS Part 1 Deterministic Models 2 3 4 5 6 7

Deterministic models and their optimisation A sketch of infinite-horizon behaviour; policy improvement The classic formulation of the control problem; operators and filters State-structured deterministic models Stationary rules and direct optimisation for the LQ model The Pontryagin maximum principle

9

11 47 63 91 Ill 131

Part 2 Stochastic Models

167

8 9 10 11 12 13 14 15

169 179 189 215 229 255 269 285

Stochastic dynamic programming Stochastic dynamics in continuous time Some stochastic examples Policy improvement: stochastic versions and examples The LQG model with imperfect observation Stationary processes; spectral theory Optimal allocation; the multi-armed bandit Imperfect state observation

BEYOND Part 3 Risk-sensitive and H00 Criteria

293

16 Risk-sensitivity: the LEQG model 17 The Hoo formulation

295 321

vi

CONTENTS

Part 4 Time-integral Methods and Optimal Stationary Policies

331

18 19 20 21

335 349 357 371

The time-integral formalism Optimal stationary LQG policies: perfect observation Optimal Stationary LQG policies: imperfect observation The risk-sensitive (LEQG) version

Part 5 Near-determinism and Large Deviation Theory

379

22 23 24 25

383 405 415 431

The essentials oflarge deviation theory Control optimisation in the large deviation limit Controlled first passage Imperfect observation; non-linear filtering

Appendices

443

AI Notation and conventions A2 The structural basis of temporal optimisation A3 Moment generating functions; basic properties

443 449 455

References

457

Index

461

Preface Anyone who writes on the subject of control without having faced the responsibility of practical implementation should be conscious of his presumption, and the strength of this sense should be at least doubled if he writes on optimal control. Beautiful theories commonly wither when put to the test, usually because factors are present which simply had not been envisaged This is the reason why the design of practical control systems still has aspects of an art, for all the science on which it now calls. Nevertheless, even an art requires guidelines, and it can be claimed that the proper function of a quest for optimality is just the revelation of fundamental guidelines. The notion of achieving optimality in systems of the degree of complexity encountered in practice is a delusion, but the attempt to optimise idealised systems does generate the fundamental concepts needed for the enlightened treatment of less ideal cases. This observation then has a corollary: the theory must be natural and incisive enough that it does generate recognisable concepts; a theory which ends in an opaque jumble of formulae has served no purpose. 'Control theory' is now understood not merely in the narrow sense of the control of mechanisms but in the wider sense of the control of any dynamic system (e.g. communication, distribution, production, financial, economic), in general stochastic and imperfectly observed. The text takes this wider view and so covers general techniques of optimisation (e.g. dynamic programming and the maximum principle) as well as topics more classically associated with narrowsense control theory (e.g. stability, feedback, controllability). There is now a great deal of standard material in this area, and it is to this which the 'basics' component of the book provides an introduction. However, while the material may be standard, the treatment of the section is shaped considerably by consciousness of the 'beyond' component into which it leads. There are two pieces of standard theory which impress one as complete: one is the Pontryagin maximum principle for the optimisation of deterministic processes; the other is the optimisation of LQG models (a class of stochastic models with Linear dynamics, Quadratic costs and Gaussian noise~ These have appeared like two islands in a sea of problems for which little more than an ad hoc treatment was available. However, in recent years the sea-bed has begun to rise and depths have become shallows, shallows have become bridging dry land. The class of risk-sensitive models, LEQG models, was introduced, and it was

viii

PREFACE

found that the LQG theory could be extended to these, although the mode of extension was sufficiently unevident that its perception added considerable insight. At about the same time it was found that optimisation on the Hoo criterion was both feasible, in that analytic advance was possible, and useful, in that it gave a robust criterion. Unexpectedly and beautifully, these two lines of work coalesced when it was realised that the H 00 criterion was a special case of the LEQG criterion, for all that one was phrased deterministically and the other stochastically. Finally, it was realised that, if large-deviation theory is applicable (as it is when a stochastic model is close to determinism in a certain sense), then all the exact results of the LQG theory have a version which holds in considerable generality. These successive insights revealed a structure in which concepts which had been familiar in special contexts for decades (e.g. time-integral solutions, Hamiltonian structure, certainty equivalence, solution by canonical factorisation) were seen to be closely related and to supply exactly the right view of a very general class of stochastic models. The 'beyond' component is devoted to exposition of this material, and it was the fact that such a connected treatment now seems possible which motivated the writing of this text. Another motivation was the desire to write a successor to my earlier work Optimisation over Time (Wiley 1982, 1983). However, it is not squarely a successor. I wanted to write something much more homogeneous and tightly focused, and the restriction to the control theme provided that tightness. Remarkably, the recent advances mentioned above also induced a tightening, rather than the loosening one might have expected. For example, it turns out that the discounted cost criterion so beloved of exponents of dynamic programming is logically inconsistent outside a rather narrow context (see Section 16.12). In control contexts it is natural to work with either total or time-averaged cost (in terminating or non-terminating situations respectively). The algorithm which emerges as natural is the iterative one of policy improvement. This has intrinsically a clear variational basis; it can also be seen as a Newton-Raphson algorithm (Section 3.5) whose second-order convergence is often rapid enough that a single iteration is enlightening (see Section 3.7 and the examples of Chapter 11); it implies similarly effective algorithms in derived work, e.g. for the canonical factorisations of Chapters 18-21. One very important topic to which we give little space is that of dual control. By this is meant the use of control actions to evoke information as well as to govern the dynamics of the system, with its associated concepts of adaptive control, selftuning regulators, etc. Chapter 14 on the multi-armed bandit constitutes almost the only substantial discussion. Despite the fact that the idea of dual control emerges spontaneously in any effort to optimise the running of a stochastic dynamic system, the topic seems too demanding and idiosyncratic that one can treat it in passing. Indeed, one may say that the treatment of this book pushes a certain line about as far as it can be taken, and that this line necessarily skirts

PREFACE

ix

dual control. In all our formulations of the LQG model, the LEQG model, largedeviation versions and even minimax control we find that there is a certainty equivalence principle. The principle indeed generally takes a more sophisticated form than that familiar from the simple LQG case, but any such principle must by its nature exclude dual control: the notion that control actions affect information gained. Another topic from which we refrain, despite the attention it has received in recent years, is the use of J- factorisation techniques and the like to determine all stabilising controls satisfying some lower bound on performance. This topic is important because of the increased emphasis given to robustness: the realisation that it is oflittle use if a control is optimal for a specified model if its performance deteriorates rapidly with departure from that specification. However, we take reassurance from one conclusion which this body of work establishes: that if a control rule is optimised under the assumption that there is observation error then it is also proofed to some extent against errors in model specification (see Section 17.3). The factorisation techniques which we employ are those associated with the formulation of optimal control as the extremisation of a suitably defined time-integral (even in the stochastic case). This is a class of ideas completely distinct from that of ]-factorisation, and with its own particular elegance. My references to the literature are not systematic, but I have certainly given credit for all recent work for which I knew an attribution. However, there are many sections in which I have worked out my own treatment, very possibly in ignorance of existing work. Let me apologise in advance to authors thus unwittingly overlooked, and affirm my readiness to correct the record at the first opportunity. A substantial proportion of this work was completed before my retirement in 1994 from the Churchill Chair, endowed by the Esso Petroleum Company. I am profoundly indebted to the Company for its support over my 27-year occupancy of the Chair.

CHAPTER I

First Ideas 1 CONTROL AS AN OPTIMISATION PROBLEM One tends to think of 'control' as meaning the control of mechanisms: e.g. the classic stabilisation of the speed of a steam engine by the centrifugal governor, the stabilisation of temperature in a central heating system, or the many automatic controls built into a modern aircraft. However, the controls built into an aircraft are modest compared with those which Nature has built into any higher organism; a biological rather than a mechanical system. This can be taken as an indication that any system operating in time, be it mechanica~ electrical, biological, economic or industrial, will need continuous monitoring and correction if it is to keep on course. In other words, it needs control. The efficient running of the dynamic system constituted by an economy or a factory poses a control problem just as much as does the operation of an aircraft. The fact that control actions may be realised by procedures or by conscious decisions rather than by mechanisms is a matter of implementation rather than of principle. (Although it is also true that it is the higher-level decisions, laying out the general course one wishes the system to follow, which will be taken consciously, and it is the lower-level decisions which will be automated. The more complex the system, the more need there will be for an automated low-level decision structure which ensures that the system actually follows the course decided by higher-level policy.) In traditional control theory the problem is regarded very much as one of stability-that departures from the desired course should certainly be corrected ultimately, and should preferably be corrected quickly, smoothly and effortlessly. Since the mid-century increasing attention has been given to more specific design criteria: control rules are chosen so as to minimise a cost function which appropriately penalises both deviation from course and excessive control action. That is, the design problem is formulated as an optimisation problem. This has virtues, in that it leads to a sharpening of concepts; indeed, to the generation of concepts. It has faults, in that the model behind the optimisation may be so idealised that it leads to a non-robust solution-a solution which is likely to prove unacceptable if the actual system deViates at all from that supposed. However, as is usual when 'theory' is criticised, this objection is not a criticism of theory as such, but criticism of a naive theory. One may say, indeed, that optimisation exposes the weaknesses in thinking which are usually compensated for by soundness of intuition. By this is meant that, if one makes certain assumptions,

2

FIRST IDEAS

then an attempt at optimisation will go to the limit in some direction consistent with a literal interpretation of these assumptions. It is not a bad idea, then, to see how an ill-posed attempt at optimisation can reveal the pitfalls and point the way to their remedy. 2 AN EXAMPLE: THE HARVESTING OF A RENEWABLE RESOURCE A good example of the harvesting of a renewable resource would be the operation of a fishery. Consider the simplest case, in which the description of current fish stocks is condensed to a single variable, x, the biomass. That is, we neglect the classification by species, age, size and location which a more adequate model would obviously require. We also neglect the effect of the seasons (although see Exercise 1) and suppose simply that, in the absence of fishing, biomass follows a differential equation

x= a(x)

(1)

where xis the rate of change of x with time, dx/dt. The function a(x) represents the rate of change of biomass, a net reproduction rate, and in practice has very much the course illustrated in Figure 1. It is initially positive and increasing with x, but then dips and becomes negative for large x, as the demands which a large biomass levies on environmental resources make themselves felt. Two significant stock levels are xo and Xm, distinguished in Figure 1. The stock level x 0 is the equilibrium level for the unharvested population, that at which the net reproduction rate is zero. The stock level Xm is that at which the net reproduction rate is greatest. If stocks are depleted at a rate u by fishing then the equation becomes

x=

a(x)- u.

(2)

X

Figure 1 The postulated form of the net reproduction rate for a population. This rate is maximal at Xm and it is zero at xo, which would consequently be the equilibrium level of the unharvested population.

2 AN EXAMPLE: THE HARVESTING OF A RENEWABLE RESOURCE

3

X

ligure 2 The values Xt and x2 are the possible equilibriii!Tilevels ofpopulation ifharvesting is carried out at a]rxed rate ujor x > 0. These are respectively unstJJb/e and stable, as is seen from the indicated direction ofmovement of:x.

Note that u is the actual catch rate, rather than, for example, fishing effort. Presumably a given effort yields less in the way of catch as x decreases until, when x becomes zero, one could catch at no faster rate than the rate a(O) at which the population is being replenished from external sources (which may be zero). Suppose, nevertheless, that one prescribes a fishing policy by announcing how one will determine u. If one chooses u varying with x then one is showing some responsiveness to the current state; in control terminology one is incorporating feedback. However, let us consider the most naive policy (which is not to say that it has not been used): that which sets u at a definite fixed value for x > 0. An equilibrium value of x under this policy must satisfy a(x) = u, and we see from the graph of Figure 2 that this equation has in general two solutions, x 1 and x2, say. Recall that the domain of attraction of an equilibrium point is the set of initial values x for which the trajectory would lead to that equilibrium. Further, that the equilibrium is stable (in a local sense) only if all points in some neighbourhood of it lie in its domain of attraction. Examining the sign of .X = a(x)- u, we see that the lesser value x 1 has only itself as domain of attraction, and so is unstable. The greater value x 2 has x > x 1 as domain of attraction, and so is stable. One might pose as a natural aspiration: to choose the value of u which is largest consistent with existence of a stable equilibrium solution, and this would seem to be U = Um := a(xm)·

That is, the maximal value of u for which a( x) = u has a solution, and so for which the equilibrium operating point is such that the biomass replaces itself at the maximal rate.

FIRST IDEAS

4

X

Figure 3

Ifthefu:ed harvesting rate is taken as high as Um, then the equilibrium at Xm is only semi-stable.

However, this argument is fallacious, and its adoption is said to be the reason why the Peruvian anchovy fishery crashed between 1970 and 1973 from an annual catch of 12.3 million tons to one ofl.8 million tons (Clark, 1976). As u increases to Um then x 1 and x2 converge to the common value Xm. But Xm has domain of attraction x ~ Xm, and so is only semi-stable (Figure 3). If the biomass drops at all from the value Xm then it crashes to zero. In Exercise 10.4.1 we consider a stochastic model of the situation which makes the same point in another way. We shall see in the next chapter that the policy which indeed maximises the steady-state harvest rate is that which one might expect: to fish at the maximal feasible rate (presumably greater than um) for x > Xm and not to fish at all for x < Xm. This makes the stock level Xm a stable point of the controlled system, at which one achieves an effective harvest rate of a(xm)· At least, this is the optimal policy for this simple model; the model can be criticised on many grounds. .. Exercises and comments

(1) One can to some extent consider seasonal effects by considering a discretetime model Xr+J = a(xr)- Ur in which time t moves forwards in unit steps (corresponding to the annual cycle) rather then continuously. In this case the function a has the form of Figure 4 rather than of Figure 1. The same arguments can be applied as in the continuoustime case, although it is worth noting that it was this model (with u = 0) which provided the first and simplest demonstration of chaotic effects. (2) Suppose that the constant value presumed for u when x > 0 exceeds a( 0 ), with u = 0 for x = 0. Then x = 0 is effectively a stable equilibrium point, with an

3 DYNAMIC OPTIMISATION TECHNIQUES

5

X

Figure 4 The form ofthe year-to-year reproduction rate.

effective harvest rate u = a(O). This is because one harvests at the constant rate the moment x becomes positive, and drives the biomass back to zero again. One has then a 'chattering' equilibrium, at which the alternation of zero and infinitesimally positive values of x (and of zero and positive values of u) is infinitely rapid. The effective harvest rate must balance the immigration rate, a(O). At this level, a fish is caught the moment it appears from somewhere. Under the policy indicated at the end of the section the equilibrium at Xm is also a 'chattering' one. Practical considerations would of course smooth out both operation and solution around this transition point. 3 DYNAMIC OPTIMISATION TECHNIQUES The crudity of the control rule of the previous section lay, of course, in the assumption of a constant harvest rate. The harvest rate must be adapted to current conditions, and in such a way as to ensure that, at the very least, a depleted population can recover. With improved dynamics it may well be possible to retain the point of maximal productivity Xm as the equilibrium operating point. However, one certainly needs a basis for the deduction of good dynamic rules. There are a number of approaches, all ultimately related. The first is the classical design approach, with its primary concern to secure stability at the desired operating point and, after that, other desirable dynamic characteristics. This shares at least one set of techniques with later approaches: the techniques needed to handle dynamic systems (see Chapters 4 and 5). One optimisation approach is that oflaying direct variational conditions on the path of the process; of requiring that there should be no variation of the path, consistent with the prescribed dynamics, which would yield a smaller cost. The optimisation problem is then cast as a problem in the calculus of variations. However, this classic calculus needs modification if the control problem is to be

6

FIRST IDEAS

accommodated naturally, and the form in which it is effective is that of the Pontryagin maximum principle (Chapter 7). This is a valuable technique, but one which would seem to be applicable only in the deterministic case. However, it has a natural version for at least certain classes of stochastic models; see Chapters 16, 18-21, 23 and 25. Another approach is the recursive one, in which one optimises the control action at a given time on the assumption that the optimal rule for later times has already been determined. This leads to the dynamic programming technique, a technique which is central and which has the merit of being immediately applicable also in the stochastic case (see Chapter 8). It is this approach which in a sense provides the spine of our treatment, although we shall see that all other methods are related to it and sometimes provide advantageous variants of it. It is also true that there is merit in methods which display the future options for the controlled process more clearly than does the dynamic programming technique (see the certainty equivalence principles of Chapters 12 and16). One might say that methods which are expressed in terms of the predicted future path of the process (such as the maximum principle, the certaintyequivalence principle and the time-integral methods of Chapters 18-21) correspond to the approach of a chess-player who explores a range of future scenarios in his mind before he makes a move. The dynamic programming approach reflects the approach of the player who has built up a mental evaluation of all possible board configurations, and so can replace the long-term goal of winning by the short-term goal of choosing a move which leads to a higher-value configuration. There is virtue both in the explicit awareness of future possibilities and in the ability to be guided to the same effect by aiming for some more immediate goal. Finally, there is the relatively naive approach of simply choosing a reasonable control rule and evaluating its performance (by, say, determination of the average cost associated with the rule under equilibrium conditions). It is seldom easy to optimise the rule at this stage; the indirect routes to optimisation are more effective and more revealing. However, there is a systematic method of improving such solutions to yield something which is well on the way to optimality. This is the technique of policy improvement (see Chapters 3 and 11), an approach also derived from dynamic programming. Judged either as an analytic or a computational technique, this may be the single most important tool. In cases where optimality may be an unrealistic ambition, even a false one, it offers a way of starting from a humble base and achieving performance comparable with the optimal. The revision of policy that it recommends can itself convey insight. Policy improvement has a good theoretical basis, has a natural expression in all the characteristions of optimality and, as an iterative technique, it shows second-order convergence to optimality.

4 ORGANISATION OF THE TEXT

7

4 ORGANISATION OF THE TEXT Conventions on notation and standard notations are listed in Appendix 1. While much of the treatment of the text is informal. conclusions are either announced in advance or summarised afterwards in theorem-proof form. This form should be regarded as neither forbidding nor pretentious, but simply as the best way of punctuating and summarising the discussion. It is also by far the best form for readers looking for a quick reference on some point. It does create one difficulty, however. There are theorems whose validity is completely assured by the conditions stated-mathematicians could conceive of nothing else. However, there are situations where arguments of less than full rigour have led one to considerable penetration and to what one indeed believes to be the essential insight, but for which the aspiration to full rigour would multiply the length of the treatment and obscure its point. This is particularly the case when the topic is new enough that a rigorous treatment, even if available, is itself not insightful. One would still wish to summarise assertions, however, leaving it to be understood that the truth of these is subject to technical conditions of a nature neither stated nor verified. Such summary assertions should not properly be termed 'theorems: We cover this point by starring the second type. So, Theorem 23.1 is true as its stands. On the other hand, *Theorem 7.2.1 is 'essentially' valid in statement and proof, but both would need technical supplement before the star could be removed. Exercises are in some cases substantial. In others they simply make points which, although important or interesting in themselves, would have interrupted the discussion if they had been incorporated into the main text. Theorems carry chapter and section labels. Thus, Theorem 2.3.1 is the ftrst theorem of Section 3 of Chapter 2. Equations are numbered consecutively through a chapter, however, without chapter label. A reference to equation (18) would thus mean equation (18) of the current chapter, but a reference to equation (3.18) would mean equation (18) of Chapter 3. A similar convention holds for figures.

BASICS PART 1

Deterministic Models

CHAPTER 2

Deterministic Models and their Optimisation 1 STATE STRUCTURE OPTIMISATION AND DYNAMIC PROGRAMMING

The dynamic operation one is controlling is referred to as the 'process' or the 'plant' more or less interchangeably; we shall usually take 'system' as including sensors, controls and even command signals as well as plant. The set of variables which describe the evolution of the process will be collectively termed the process variable and denoted by x. The control variable, whose value can be chosen by the optimiser, will be denoted by u. This is consistent with the notation of Chapter 1. Models are termed stochastic or deterministic according as to whether randomness enters the description or not. We shall see that the incorporation of stochastic effects (i.e. of randomness) is essentially a way of recognising that the values of certain variables may be unknown; in particular, that the future course of certain input variables may be only imperfectly predictable. We restrict ourselves to deterministic models in these first seven chapters. We shall denote time by t. Physical models are naturally phrased in continuous time, when t may take any value on the real axis. However, it is also useful to consider models in discrete time, when tis considered to take only integer values t = ... , -2, -1, 0, 1, 2, .... This corresponds to the notion that the process develops in stages, of equal length. It is a natural view in economic contexts, for example, when data become available at regular intervals, and so decisions tend to be taken at the same intervals. Even engineers operate in this way, when they work with 'sampled data'. Discretisation of time is inevitable if control values are determined digitally. There are mathematical advantages in starting with a discrete-time formulation, even if one later transfers the treatment to the more physical continuous-time formulation. We shall in general try to cover material in both versions. There are two aspects of the model which must be specified if the control optimisation problem is to be properly formulated. The first of these is the plant equation; the dynamic evolution rule that x obeys for given controls u. This describes the dynamics of the system which is to be controlled, and must be derived from a physical model of that system. The second aspect is the performance criterion, which usually implies specification of a cost function. This

12

DETERMINISTIC MODELS AND THEIR OPTIMISATION

cost function penalises all aspects of the path of the process which are regarded as undesirable (e.g. deviations from required path, lack of smoothness, depletion of resources) and the control policy is to be chosen to minimise it. Consider ft.rst the case of an uncontrolled system in discrete time. The plant equation must then take the form of a recursion eXpressing x 1 in terms of previous x-values. Suppose that this recursion is first-order, so taking the form Xt

= a(Xt-1

1

(1)

t),

where we have allowed dynamics also to depend upon time. In this case the variable x constitutes a dynamically complete description of the state of the system, in that the future course {xT; T > t} of the process at time tis determined totally by xr, and is independent of the path{~; T < t} by which x 1 was reached. A model with this property is said to have state structure, and the process variable x can be more strongly characterised as the state variable. State structure for a controlled process in discrete time will also require that the model is, in some sense, simply recursive. It is a property of system dynamics and the cost function jointly. We shall assume that the plant equation takes the form Xt

=

a(Xt-1 1 Ut-1 1

(2)

t).

analogously to (1). Further, if one is optimising over the time period 0 shall assume that the cost function C takes the additive form h-l

h-l

r=O

r=O

c = 2: c(xT, u,., r) + Ch(Xh) = L

Cr

+ ch,

~

t

~

h, we

(3)

say. The end-point h is referred to as the horizon, the point at which operations close. It is natural to regard the terms cr and Ch as costs incurred at time r and time h respectively; we shall refer to them as the instantaneous and closing costs. We have thus assumed, not merely additivity, but also that the instantaneous cost depends only on current state, control and time, and that the closing cost depends only upon the closing state xh. One would often refer to xh and Ch as the terminal state and the terminal cost respectively. However, we shall encounter processes which may terminate in other ways before the horizon point is reached (e.g. by accident or by bankruptcy) and it is useful to distinguish between the cost incurred in such a physical termination and one incurred simply by the state one is left in at the expiry of the planning period. We have yet to define what we mean by 'state structure' in the controlled case, but shall see in Theorem 2.1.1 that assumptions (2) and (3) do in fact imply the simply recursive character of the optimisation problem that one would wish. Relation (2) is of course a simple forward recursion, and the significant property of the cost function (3) turns out to be that it can be generated by a simple backward recursion. We can interpret the quantity

13

1 STATE STRUCTURE h-l

(4)

Ct= Lct+Ch, T=C

as the cost incurred form time t onwards. It plainly obeys the backward recursion

Ct = c(Xt Ut t) 1

1

+ Ct+l

t
(5)

We have the optimisation problem formulated as the problem of choosing {u 1; 0 ::::;; t < h} to minimise the cost function (3), subject to validity of the plant equation (2) over the time interval (0 < t < h). One can think of various approaches to this problem. One would be to solve (2) for x in terms of u and to substitute in (3), leaving a function of the control variables alone to be minimised. However, this is in general impracticable, both mathematically and computationally. It is also unnatural; the dynamic and state character of the model are lost in the reduction. If one could eliminate u instead by solving (2) for u1_, in terms of x 1 and x 1_, then one could reduce the cost function to the form

h-! C= Lg(xT,X-r+!,T)+Ch(xh)

(3')

T=O

for some function g. The study of the extremisation of such forms is exactly the province of the calculus of variations (see Exercise 6), which was for long the classic tool for the treatment of dynamic optimisation problems. However, Equation (2) may simply not be soluble for u, and elimination again implies a loss of structure. Yet a third approach is to regard the problem as one of minimising C with respect to the both sequences {x1 } and {u1 } jointly. The plant equation (2) then constitutes a set of constraints on the variables which can be allowed for by the introduction of Lagrange multipliers. This approach avoids the unnatural elimination of significant variables, and, as always, the multipliers themselves turn out to be significant. It can be regarded as providing one route to the celebrated maximum principle (see Chapter 7), itself a special case of the 'timeintegral' formulation which we shall find fundamental and valuable. However, there is yet a fourth approach, which is to exploit the recursive character of the state formulation to derive a recursive treatment of the optimisation problem. In equation (4) we defined Ct. the cost incurred from time t. Define also the value function F(x 1 , t) as the minimal value ofC 1 with respect to remaining control values u7 ( t ::::;; T < h) for prescribed x 1 . It is useful to regard this as a function F(x, t) of generic variables x and t; the minimal cost incurred from timet if x 1 has the value x. Theorem 2.1.1 Assume the plant equation (2) and the cost function (3). Then: ( i) The value function F obeys the dynamic programming equation

14

DETERMINISTIC MODELS AND THEIR OPTIMISATION

F(x, t)

= inf[c(x, u, t) + F(a(x, u, t), t + 1)] u

t
(6)

with terminal condition F(x,h) = Ch(x).

(7)

(ii) The minimising value ofu in (6) is a/unction ofxand tonly, u(x, t} say, and the optimal value ofcontrol at time tis u 1 = u (X 1 , t). (iii) This expression for u 1 is optimal, no matter how past values ofcontrol have been chosen. Proof Relation (7) follows from the definition of F. The backward recursion (6)

follows from recursion (5) ifwe minimise both sides, firstoverur forT > tand then over u 1• The minimising value of u 1 is optimal, and this is exactly the minimising value in (6). It is clear that both the minimising value of u and the minimised value of the right-hand member in (6) are functions of x and t alone. Assertion (iii) follows simply from the fact that nothing has been assumed of previous history (xr, Uri r < t) except that it is known. 0 The key recursion (6) is referred to as either the dynamic programming equation or the optimality equation. Let us consider the conclusions of the theorem in reverse order. Assertion (iii) follows simply from the way the dynamic programming equation was set up. An optimal control rule that remains optimal no matter how past controls have been chosen is said to be closed-loop optimal, for reasons to be explained in Chapter 4. Other methods of optimisation may well not yield this closed-loop form, since they assume that past as well as future has been optimised (see Section 4). The closed-loop property is closely related to that of feedback: that, just as control u affects the process variable x through the plant equation, sox affects u through the control rule. If it is a general consequence of the recursive approach that past policy is irrelevant, it is a consequence of our specific assumptions that past history is irrelevant. This is the striking implication of assertion (ii): that the optimal value of u1 is a function of x 1 and t alone. However, it is also striking that, to determine this rule, one must solve the dynamic programming Equation (6); i.e. determine the value function and the optimal control jointly. The value function in a way telescopes the future. Let us repeat the analogy of Section 1.3. If one tries to determine the optimal value of u 1 by directly minimising the cost function (3) with respect to all future controls ur(T ~ t) one is in the position of a chess-player who tries to determine his next move by working through all possible future patterns of play in his mind. If one determines the optimal u1 from the dynamic programming equation one is in the position of a chess-player who is so experienced that he can set a value F(x) on any board configuration x and chooses his move so as to improve that value. He

1 STATESTRUCTURE

15

thus replaces the ultimate goal ofwinning by the immediate goal of moving to as favourable a position as possible at the next stage. The plant equation (2) is a forward recursion in time, in that x, is determined in terms of past values. However, the dynamic programming equation (6) is a backward recursion, in that the value function at timet is determined in terms of that at time t + 1. The reason for this is that one cannot optimise the control rule at a given time unless one knows what control policy will be after that time; optimisation runs backwards. As Kierkegaard says, life must be lived forwards but understood backwards. However, all recursions occurring are simple; a consequence of our plant and cost assumptions. One might say that a variable {r is suffiCient for the optimisation problem if (i) both the value function and the optimal control at time t are functions of {1 alone, and (ii) {1 is updated by a simple recursion, in that {, can be calculated as a function of {r-I and u,_ 1 alone. Theorem 2.1.1 then implies that (xc, t) is sufficient, under the assumptions stated. One can then reasonably term x a state variable for the optimisation problem. Sometimes future costs are discounted by modifying the cost function (2) to h-1

C=

L tr" cr + ,&ch

(8)

-r=I

where (3 is a factor lying between 0 and 1, the discount factor. The thought behind this is that compound interest causes money to increase in value by a factor of (3-I per unit time. A unit amount of money available now is thus worth (3-r after time -r. Conversely, a unit amount of money available after time -r has a present value of (JT. Hence the (3-coefficients in expression (8), if we assume that costs are expressed in monetary terms. Discounting is often introduced in this context, partly for the ostensible accounting reasons we have indicated, partly just as a mathematical device to ensure convergence of the total cost C as the horizon h is allowed to recede to the distant future. However, while accounting considerations may be relevant for investment decisions, they are scarcely so for moment-by-moment control decisions, and the unthinking inclusion of a discount can indeed lead to inconsistencies (see Section 16.12). We shall consider the discounting option for the cases for which it is realistic, but shall see in the next chapter that there are other views of the infinite-horizon limit which are perhaps more appropriate in the control context. One could simply absorb the factor {J'" into the -r-dependence we have already allowed in Cr.- However, in general it is useful to distinguish discounting from genuine time-dependence of costs, and to adapt the definition (4) of C, to h-I

c, = "2:::: fJT-lcT + f3h-·rch;

(9)

16

DETERMINISTIC MODELS AND THEIR OPTIMISATION

the present value (at timet) of future costs. With the value function F(x, t) defined in terms of C1 as before, we derive the modified optimality equation

F(x, t)

=

inf[c(x, u, t)

"

+ (3F(a(x, u, t), t + 1)],

(10)

again with terminal condition (7). If the dynamics and instantaneous costs are independent of time, so that the functions a and c are functions a(x, u) and c(x, u) of current state and control variables alone, then the problem is said to be time-homogeneous. In such cases the value function F(x, t) will still be !-dependent in general, because of the effect of a finite horizon h. However, it will depend upon t and h only in the difference s = h- t, the time to go. In such cases we shall often write F(x, t) as Fs(x). If Fis also independent oft (which we expect may be the case for a time-homogeneous model if the horizon is infinite and the infinite-horizon total cost C is well defined) then the problem is said to be time-invariant. We shall consider the continuous-time analogue of this material in Section 6. We shall often use the term control policy, meaning simply a control rule: a rule for determining control values from currently known observables. The clear definition of these terms emerges first when we admit that not everything is observable, and so adopt the stochastic formulation of Chapter 8. A stationary policy is one for which we dispense with the observable 'clock time', and so form the control rule in a time-invariant fashion. Exercises and comments

(1) Retain the plant equation (2) but suppose the cost function (3) modified to C = maxo.;; r
(2) More generally, one could derive a dynamic programming treatment under the assumptions of a simply-recursive plant equation (2) and a cost function for which the cost C 1 from timet obeys a backward recursion C1 = c(x 1, u1, t, Cr+!) for some c, monotonic increasing in the C argument. Recursion (5), linear in costs, then seems very special. However, the case of a linear cost-recursion is virtually the only one which is straightforward for stochastic models. (3) Other variants of the model can be brought to state structure by appropriate transformations. For example, had the plant equation (2) been a second-order recursion x 1 = a(xr-1, Xr-2, ur-I, t), then state structure could have been achieved by adoption of a transformed process variable Xr = (x 1 , Xr-1 ). (4) One can achieve time-homogeneity formally by changing the process variable from x1 to ~~ = (x 1 , t). What is the form of recursions (1) and (2) in terms of~? This standardisation is, although superficial, useful for the compact discussion of, for example, the maximum principle and stopping problems.

2 OPTIMAL CONSUMPTION

17

(5) A state-structured pr?blem can some~es b~ reduc~d ev~n further, so that a lower-dimensional functiOn of the state vanable IS sufficient, m the sense defined above. Suppose that a discrete-time system with a vector state variable has a plant equation which is linear in state, x, = ArXr-1 + b(ur-1, t), an instantaneous cost c(u,, t) which is. independe~t of state a~d a terminal cost at time h which is a function of a hnear funcuon of termmal state, dT xh. Show that t and the 'predicted terminal miss-distance' dT xi') are jointly sufficient at time t, where x~) = AhAh- 1 ... Ar+1x,. Note that xir) is the value Xh would take if the control term b were absent from time t onwards. (6) Suppose that expression (3') for C is differentiable with respect to its xarguments, which we suppose for simplicity can take arbitrary values on the real line. The stationarity conditions (0 < t
(11)

are then necessary if the path {xr} is to minimise C. The continuous-time analogue of expression (3') would be an integral over time

C=

1h

g(x, x, t) dt + · ·.

where x = dx/dt and + · · · indicates possible separate contributions to C from the end-points, t = 0 and h. If we perturb x(t) to x(t) + 8(t) then the perturbation in C is, to first-order terms in 8,

6.C=

1 (og h

0

og ·)

-6+--:-6 dt+···= OX

OX

1h [og 0

(og)] 6dt+···.

d --:-----:OX

dt OX

The requirement of stationarity with respect to variations 6 then leads to the condition that the square-bracketed expression in the final integrand should be zero for 0 < t < h This is the classic Euler condition, the continuous-time analogue of(ll).

2 OPTIMAL CONSUMPTION One can hope to solve the dynamic programming equation (6) analytically, or at least to reduce it in some way that yields insight. Failing this, the recursion can be solved computationally in favourable cases. A tractable example is provided by a simple model of consumption and investment. The model is too simple for realism, but does provide some insight, and is a first version of more substantial models of portfolio management and, indeed, of economic development (see Section 3.4)

18

DETERMINISTIC MODELS AND THEIR OPTIMISATION

We can regard the optimiser as an investor who wishes to split his capital between investment and consumption at each stage. The capital x can be taken as the state variable; a non-negative scalar assumed to follow the plant equation Xr

( 12)

= o:(xr-1- Ur-J).

Here ur is the amount diverted for consumption at time t and o: is the factor by which invested capital appreciates over unit time. It is more natural to phrase the problem as one of maximising the total benefit the investor receives from consumption (which an economist would term utility) rather than of minimising cost. Let r(u) be the utility he derives from consumption u, and suppose that the total utility is just the sum of discounted instantaneous utilities. If debt is not permitted then necessarily x ~ 0 and so ur ~ x 1 ; if consumption cannot be reversed to become production then u ~ 0. Let F.(x) be the maximal utility that can be gained with s time periods to go, starting from capital x. The dynamic programming equation is then

F.(x) = max [r(u) + /3Fs-I (o:(x- u) )] O~u~x

(s>O).

( 13)

The concept of discounting was motivated in the last section by the notion that money invested will grow by compound interest. It may then seem perplexing to see the same feature built in twice: a discounting by a factor {3 as well as the growth of invested capital by a factor o: (as expressed in the plant equation (12)). The more natural interpretation of {3 in this case is as a survival probability; the probability that the investor will survive the next unit of time to enjoy his savings. If we make the assumptions

for the form of the utility functions then (13) is easily solved. Here vis a constant in (0, 1). The form for r(u) is plausible; there are economic reasons for assuming utility to be concave and non-decreasing. That the terminal reward Fo should have the same form is perhaps debatable, but the assumption is a convenient one. We leave the reader to verify the solutions Uopt

= f.s-I/11 X

for the maximal utility and the optimal consumption at horizons. Here {Is} is a sequence of constants obeying

f.sI/11 with

=

1 + ,..,ri/11 iJ s-1

2 OPTIMAL CONSUMPTION

19

so that J}fv =frl/v s

0

+ 1- Y

1-1·

The constant 1 can be interpreted as the rate at which utility grows, so that sacrifice of an amount 8 of current utility can yield utility 8y after a time s (for small8 and with optimal decisions; see Ex. 2). If 1 > 1, so that there is an incentive to delay consumption, then we see from the solution above that

for large s. The second relation implies that, if utility growth is possible, then consumption is indeed delayed to almost the end of the programme. If 1 < 1 then one finds the limiting values Uopt-->

(1- 1)x

ass becomes large. That is, if to delay consumption is unrewarding, then maximal utility over an infinite horizon is finite. Furthermore, the optimal policy then has the stationary form that one reinvests a fraction 'Y of one's capital at all times. However, this stationarity does not imply constant capital; see Exercise 1. This kind of model in general implies the conclusion that one sacrifices consumption to investment (i.e. to growth) if economic growth is possible. The conclusion is one which is modified at the individual level by finite life expectancy, as we have seen. It is also sensitive to the form of the utility function, and of course fails if there are physical limits to growth. Exercises and comments

(1) Under the optimal policy derived above capital grows at each stage by a factor B = CX"'f = ( a{J) lfv. This can be greater or less than unity, consistent with

, < 1. (2) Show that 'Y is indeed the growth rate of utility, in that, if one forgoes a small amount 8 of current utility, one can realise an additional amount {33 0£(l-v)8 = y8 after times. (3) Solve the linear case v = 0. (4) The limit v T 1 yields the case in which r(u) and Fo(x) are proportional to log(u- Umin) and log(x -Xmin) respectively, where Umin and Xmin are minimal subsistence values, below which consumption and final capital must not fall. Solve (13) in this case.

20

DETERMINISTIC MODELS AND THEIR OPTIMISATION

3 PRODUCI'ION SCHEDUliNG This is a version of an inventory problem which allows some useful reduction, but ultimately resorts to computation. We indicate variants of the problem at the end of the section. A factory produces a single commodity and wishes to schedule its production so as to meet a known time-varying pattern of demand in the most economical fashion. Let us define the state x 1 at time t as the stock held at the beginning of the t th day; u1 as production during that day and d1 as demand during that day. In such problems one must indicate relative timing within a stage if one is to be clear about the sequence of events. The plant equation is then

(14) Suppose that the instantaneous cost function for day tis c1(u 1) + c2(x1+1), where CJ (u) is the cost of producing an amount u in a day, and c2(x) is the cost of carrying a stockx from one day to the next. The optimality equation (for costs) is then

F(x, t) = min[c1(u) u

+ c2(x + u- d1) + F(x + u- d1 , t + 1)).

(15)

Plausible simple forms for the cost functions are

CJ(u) = c2(x) =

{0

b a+ u

+oo { ex

(u (u

~

0)

> 0)

(x < 0) (x ~ 0).

(16) (17)

Assumption (16) implies that one can dispose of stock without cost, but that to go into production on a given day implies a base cost a and a unit cost b. Assumption (17) implies that negative stock ('back-logging') is forbidden, and that to carry positive stock incurs a unit cost of c per day. We assume a horizon point h at which negative stock is again forbidden, but positive stock valueless. There is no discounting. It is the occurrence of the terms in a and c which make the problem non-trivial. If the base cost a were zero then one would produce only enough to satisfy current demand. If the storage cost c were zero then one would produce enough to satisfy all future demand in a single production run. With both costs present one must steer between these two extremes. The simplifying feature of the problem, as we have formulated it, can be summarised. Theorem 2.3.1 In an optimal production programme one produces on a given day if and only ifstocks are insufficient to meet that day's demand. When one produces one brings stock up to such a level that it satisfies demand for an integral number ofdays.

3 PRODUCTION SCHEDULING

21

Proof Suppose that under a production programme {u;} one has u; > 0 for a given t. If x ~ d1 and one transferred the production u; to day t + 1 one would save a cost cu; or a+ cu; according as u;+I is zero or positive. Hence the first assertion. For the second point, suppose that on day t one produces enough to satisfy demand before time r, but not enough to satisfy demand at time r as well. That is, u1 = 'r.j::/ ~ - x 1 + 8, where 0 ~ 8 < dr. Then one must produce on day Tin any case. If one decreased by 8 and increased by 8 then one would save a storage cost c( T - t)8, with no other effect on costs. Hence 8 = 0 in an optimal policy. 0

u;

u;

Thus, if x 1 ~ d1 one does not produce and lets stock run down. If x 1 ~ d1 then one produces enough that demand is met up to some time T - 1 exactly (so that x,. = 0), where T is an integer exceeding t. The optimal r will be determined by the optimality equation

+ F(O, r)],

( 18)

c(t, r) =a+ 'l)b + c(j- t)]~.

( 19)

F(x, t) = min [c(t, r) t
~

h

where .,._! }=I

So, after a first production or disposal decision stock comes down to zero at some time, and from that point one operates between recurrent instants of zero stock. These instants are determined by the reduced form of the optimality equation ¢r+I ¢r

=

{

min [c(t, r)

t
+ ¢r]

(dt = 0) (d1 > 0; t
(20)

with ¢h = 0, where ¢ 1 = F(O, t). The cost function (19) is easily calculated; by solving (20) recursively one then calculates the optimal instants at which to produce if initial stock is zero. If one begins with positive but inadequate stocks one produces according to (18). If one begins with more than adequate stock one either lets it run down or disposes of some. The model is, of course, excessively simple. In practice there will be a premium for utilising plant and manpower at a constant rate. The base cost a could be regarded as embryonically representing the cost of switching between different types of production. In Section 7.1 we consider a somewhat related problem whose solution has a nice analytic characterisation. If demand dis characterised statistically then the model can be _given a time-invariant formulation and there is a stationary optimal policy. For example, if the d1 are supposed independently and identically distributed then the well-known 'two-bin policy' is optimal: that when stock falls below a lower critical value one produces so as to bring it to an upper critical value.

22

DETERMINISTIC MODE~ AND THEIR OPTIMISATION

4 LQ REGULATION We come now to a class of models which are much closer to the models of traditional control theory and practice than are those of the last two sections. These are the models for which process and control variables are vector-valued, are constrained only by a linear plant equation and imply quadratic costs. They are for this reason termed 'linear/quadratic', a term abbreviated even further to 'LQ~ LQ models allow very explicit treatment and have a very full theory, which extends to the stochastic case. Not merely are they amenable; they show interesting properties. Indeed, much research in control theory consists in trying (sometimes realistically and sometimes futilely) to find an analogue of LQ theory for non-LQ models. On the other side of the ledger, LQ assumptions are strong. For some applications, the assumptions may not be too unrealistic; for others, they plainly are so. Further, the very simplicity of the models and corresponding strength of conclusions means that there are interesting features which they cannot incorporate. Nonlinearity is one. For another, control actions cannot affect the informativeness of observations. Let us formulate a state-structured LQ model We shall assume that the state variable x is a column n-vector and the control variable u a column m-vector, so that there are m scalar components of control and n scalar components of state. We shall suppose the plant equation

(21)

Xr = AXr-l +Bur-l

and a cost function h-l

C=

L c(x, u,) +! (xTIIxh,

(22)

r=O

where

c(x,u) =!(xTRx+xTsTu+uTSx+uTQu)

=! [~r[~ ~] [~].

Here the h-subscript to the bracket in (22) indicates that all quantities in the bracket, x and II, are to bear the subscript h. The symbols A, B, R, S, Q and IIh denote matrices of appropriate dimension, chosen so that C is non-negative defmite as a quadratic form and Q is positive definite. The inclusion of the factor is a convention which on the whole makes for simplicity. This is by no means the most general state-structured LQ model. For simplicity, we have supposed it time-homogeneous, in that the matrices A, B, R, Sand Q are constant The analysis extends easily to the time-varying case. We could also have included terms of lower degree in both plant equation and cost function, representing disturbances to dynamics and an incentive for x and u to follow

1

4 LQ REGULATION

23

prescribed paths; we re~rn. to these ~atters in Section 9. On present assumptions the cost functiOn IS such as to mduce movement of both x and u to zero; the model is then one of regulation to the set-point (0, 0). One tends to use the term 'regulation' when it is a matter of holding x to a fixed set-point, rather than (for example) of trying to make it follow an external reference signal. We need a simple general result on quadratic forms, and may conveniently take c(x, u) as a specimen such form. Lemma 2.4.1

The quadratic form (23) is minimal with respect to u at

u = -Q- 1Sx and the minimised value ofthe form is

inf c(x, u) u

= !xT (R-

sT Q- 1S)x.

These assertions are readily confirmed. The interest is that the minimising value of u is linear in x and the minimised form remains quadratic in x. A slightly related point is that we can always change variables so as to normalise the matrix Sto zero; see Exerci~e 1. We now come to the central conclusion. Theorem 2.4.2 The assumptions (21 )-( 23) have the consequences that (i) The value function is quadratic in x

F(x, t) = !xTII1 x

(24)

and the time-dependent matrix II 1 obeys the Riccati equation

III= flit+ I

(t
(25)

wherefis an operator having the action

fiT= R

+ ATIIA- (ST + ATIIB)(Q + BTIIB)- 1 (S + BTIIA).

(26)

(ii) The optimal control, in closed-loop form, is (27)

where

(28) Proof Certainly F has the form (24) at time h. Suppose that it does so at time t + 1. Then the optimality equation evaluates F(x, t) as

F(x, t) = inf[c(x, u) +!(Ax+ Bu?II 1+ 1 (Ax+ Bu)]

"

(29)

24

DETERMINISTIC MODELS AND THEIR OPTIMISATION

with c(x, u) given by (23). The minimising value of u is the optimal value of u1• But, appealing to Lemma 2.4.1, we see that the minimising value is that given by (27), (28) and the minimised value has the form (24) with II 1 determined by (25) and (26). The induction is thus complete. 0 The proof is thus a simple one, with the elegant conclusions that the optimal control is linear (in x) and the value function quadratic. The backward recursion (25) for II 1 is celebrated, and referred to as the Riccati equation. (The continuoustime treatment of Section 8 exhibits dii/dt as a quadratic function of II, and it was just equations of this type which Riccati analysed.) A fast algorithm for solving the Riccati equation (at least in the infinite-horizon limit) is derived in Section 3.6. Despite the basis simplicity of the argument and the result, expressions (26) and (28) for the derived matrices may not seem very elegant. However, the structure behind them is simpler than the expressions themselves. One can formally write

fTI = iW[R + sT K + KTS + KT QK +(A+ BK)TII(A + BK)],

(30)

a way of writing the minimisation with respect to the variable u in (29) as a minimisation with respect to a matrix (see Exercise 2). The operator f then appears as the extremum of a linear operation, for which reason it has a particular character. We see from (21) and (27) that the optimal control process obeys the simple recursion

(31) say. If successful regulation is possible then one expects that the solution x 1 of this equation will become smaller with time, and indeed converge to zero in the infinite horizon. The matrix r I is often termed the gain matrix. For the simplest possible example, suppose that x and u are scalar and that the plant equation and cost functions are h-l

Xr = Xr-l

+ Ur-l,

C=!QL!},+!D~. 1=0

The cost function thus penalises deviation from zero of all control variables and of the state variable at termination. The larger DIQ, the greater relative penalty one assigns to terminal deviations in state. One readily finds the evaluation ofii 1 and of the optimal control

QD IIr = Q+sD

D

Ur = - Q+sDXr

(32)

where s = h - t is the time to go. However, suppose we had optimised the path directly, by eliminating u from the cost function and so choosing {x 1} to minimise

5 LQ REGULATION IN THE SCALAR CASE

h-1

C=

!Q 2)xr+l -

x,)

2

25

+ !D~.

1=0

One readily finds that the optimal path is a straight line with constant gradient (and so constant u) given by D Xt+! - Xr = Ut = - Q + hDxo. (33) The control rules (32) and (33) are equivalent, and (33) makes plain a fact not immediately evident from (32): that the control value is constant along the optimal orbit. However, expression (33) gives the correct value of u1 only if the same rule has been applied at earlier times, whereas (32) gives the optimal value of u1 no matter what previous control actions or policy have been. The control rules (32) and (33) are termed respectively closed-loop and openloop, in that they respectively do and do not incorporate feedback from the current state value. The open-loop rule endeavours to guide the process along the desired path by dead-reckoning from the initial position xo. The closed-loop rule corrects the path by reference to the current position x,. However, the presence of feedback is not enough in itself to ensure the stronger property of closed-loop optimality: that the rule is optimal whatever past control policy may have been. The distinction between the open- and closed-loop forms of a rule becomes vital when the path of the process is subject to unpredictable disturbances; see Section 9.1.

Exercises (1) Suppose that one works in terms of a revised control variable u• = u + Q- 1 Sx. Show that the matrix S of the cross-term is reduced to zero by this change. What are the revised values of R. Q, A and 81 (2) The infimum with respect to Kin expression (30) implies that one symmetric matrix can be 'bigger' than another. The ordering is that of positive-definiteness: we write P, > P2 if the matrix P, - P 2 is non-negative definite. (3) Complete the deduction of the open-loop control (33). Note that the open- and closed-loop forms of control agree at t = 0. This will always be true: the two forms of the optimal rule will agree at the starting point, and so the closed-loop form can be deduced from the open-loop form if one formulates general initial conditions (i.e. a start from a general value ofstatex at a general timet).

5 LQ REGULATION IN THE SCALAR CASE One can gain useful insights by following through the material of the last section in the simplest case, when both x and u are scalar. We shall return to cases of greater physical interest in the continuous-time formulation of Section 8.

26

DETERMINISTIC MODELS AND THEIR OPTIMISATION

rn

n Figurt! 1 Convergence ofthe sequenceJ(•l to the unique positive root of ll =/II.

Assume then the scalar case, with S normalised to zero. The Riccati equation (25) reduces to

A2 QII,+t II, = fllt+t = R + Q + B2IIt+t .

(34)

and lit = f 9 Ilh, where s = h- t is time to go and fs is the sth iterate of the operator f. As a function of II the quantity fii has the character depicted in Figure 1; it is monotone increasing and concave. We shall make these facts the basis of an informal proof of the following theorem; a formal proof will be given for the vector case in Chapter 6.

Theorem 2.5.1 Suppose that Q and Rare positive, that B is non-zero and that Ilh is non-negative. Then the sequence lit converges monotonically with increasing time to go h-t to the unique positive solution II ofthe equilibrium Riccati equation II =/II.

(35)

The corresponding limit gain factor r is the numerically smaller root ofthe quadratic ,!!quation (36) Proof It is obvious graphically that (34) has only a single positive root. The sequencef9Ilh is given by the cobweb diagram of Figure 1, and it is again evident graphically that this converges to II with increasing s. We leave formalisation of these assertions to the reader. We find that (37)

5 LQ REGULATION IN THE SCALAR CASE

27

Eliminating II from (34) and (37) we deduce that r satisfies (36), which has reciprocal real solutions. It also follows from (34) that expression (37) can be rewritten as

r

=~[II~

R]

(38)

and from (37) and (38) that If I < 1, whatever the magnitude of A. The steady-state gain factor r can thus be identified with the numerically smaller root of (36). 0 The conclusion is then what one might have hoped: that in the infmite horizon the optimal regulation rule becomes time-homogeneous. Further, this timehomogeneous rule is stable in that lr! < 1, with the implication that the solution x 1 of the optimally controlled plant equation (31) converges to zero with increasing t. The stability is, of course, induced by the penalties of instability. The conclusion is sensitive to various assumptions, however; see Exercise 1. Suppose that control is expensive, in that Q/R is large. We see then from (36) that r will be close to the numerically smaller of A and A- 1. That is, K ~ 0 if the plant is stable (i.e. if lA I < 1) and K ~ B- 1 (A-! - A) if the plant is unstable (i.e. if IAI > 1). So, if the plant is stable and control is expensive then virtually no control will be exerted, which is not surprising. However, if the plant is unstable and control is expensive then a control will be exerted which effectively replaces the gain factor A for the uncontrolled plant equation by its reciprocal The scalar case is one for which we can optimise the trajectory directly. If we use the plant equation (21) to eliminate the control variable then the cost function can be written 00

C =! L:[&; + QB- 2(xr+1 - Ax,)

2

]

+ i IIh~·

(39)

t=O

Minimisation with respect to x 1 yields an equation which we can write as [Q{l - AS")(l - AS"- 1)

+ B2R]xr = 0

(0 < t
Here § is the backward translation operator, with effect

§x,

= Xr-1,

{40)

so that s-- x 1 = Xt+l· The solution x 1 of the second-order difference equation (39) will be a linear combination of z~ and z~, where z1 and z2 are the roots of 1

Q(l- Az)(l- Az- 1 )'+ _BlR = 0. But we know from (36) that these roots are just r±t, so that the optimal trajectory has the form X 1 = cr 1 + c'r-r, (41)

28

DETERMINISTIC MODELS AND THEIR OPTIMISATION

where the coefficients c and c' are determined by initial and terminal conditions. The initial condition is prescription of xo, and the terminal condition is, in the infinite-horizon case, effectively that x 1 ---> 0 with increasing t. Solution (41) thus becomes x 1 = fl x 0 , consistent with the recursive analysis. Exercises and comments 1 Consider the following separate variations of the assumptions of the theorem. (i) If B = 0 (so that the process is uncontrolled) then (35) has a finite solution if the plant is strictly stable; otherwise it has no finite positive solution. (ii) If R = 0 and the plant is stable then (35) has the unique solution II= 0. If R = 0 and the plant is unstable then there is a solution II = 0 and another positive solution. These two solutions correspond respectively to infinite horizon limits for the two cases IIh = 0 and IIh > 0. (iii) If Q = 0 then II = R. In this case the optimal control takes whatever value is needed to bring the state variable to zero at the next step.

6 DYNAMIC PROGRAMMING IN CONTINUOUS TIME Control models are usually framed in continuous time, and all the material of Section 1 has a continuous-time analogue. If we look for state structure then the analogues of relations (2) and (3) are a first-order differential equation i = a(x, u, t),

(42)

as plant equation and a cost function

IC =

1h

c(x, u, T)dT + C(x(h), h).

(43)

of integral form. Here i is the time rate of change dx / dt of the state variable x. It thus seems that an assumption is forced upon one: that the possible values and course of x are ·· such this rate of change can be defined; see Exercise 1. We shall usually suppose x to be a vector taking values in !Rn. We shall write the value of x at time t as x( t) rather than x 1 , although often the time argument will be suppressed. So, it is understood that the variables are evaluated at time tin the plant equation (42), and at time T in the integral of (43). The quantity c(x, u, T) now represents an instantaneous rate of cost, and the final term in (43) again represents the closing cost incurred at the horizon point h. The general discussion of optimisation methods in Section 1 holds equally for the continuous-time case: there seem to be just two methods which are widely applicable. One is that of direct trajectory optimisation by Lagrangian methods, which we develop in Chapter 7 under its usual description of the maximum principle. The other is the recursive approach of dynamic programming.

6 DYNAMIC PROGRAMMING IN CONTINUOUS TIME

29

We can derive the continuous-time dynamic programming equation formally [rom the discrete-time version (6). The value function F(x, t) is again defined as the minimal future cost incurred if one starts from time t with state value x. Considering events over a positive time-increment 8t we deduce (cf. (6)) that

F(x, t) = inf[c(x, u, t)8t + F(x + a(x, u, t)8t, t + bt)] u

+ o(8t).

(44)

Letting 8t tend to zero in this relation we deduce the continuous-time optimality equation

.

[

1~f c(x, u, t)

+

oF(x, t) ot

+

oF(x, t) ] ox a(x, u~ t) = 0

(t
(45)

Here oF j ox is the row vector of first derivatives ofF with respect to x (see the conventions listed in Appendix 1). As in the discrete-time case, the minimising value of u in (45) is the optimal value of u(t), and there is a terminal condition

F(x,h) = C(x,h). In the discounted case the cost function will become C = 1h e-ar c(x, u, T)dT

+ e-"hC(x(h), h)

where a is the discount rate. The value of Fin the right-hand member of (44) is then discounted by a factor f3 = e-" 61 , with the consequence that the optimality equation (45) is modified to

.

mf[c(x, u, t) - aF(x, t) u

+

oF(x, t) 0t

+

oF(x, t) a(x, u, t)] = 0. 0X

( 46)

There are many questions of rigour. One is: whether the postulated differentials exist, and whether it matters if they do not (see Exercise 2 and Section 10.7). Another is: whether fulfilment of the optimality equation is either necessary or sufficient for optimality. These we shall pass over, discussing only points which arise in particular applications, when they usually have intuitive content. Exercises and comments

(1) In Section 1 we made no assumptions at all concerning the sets within which the variables x and u may vary. In the continuous-time case restrictions are implied by the seeming necessity that the rate of change dx/dt should be meaningful. The point arises even in the uncontrolled case: could one have a state-structured continuous-time deterministic model for which the state variable took values in a discrete set (say)? If the transitions between the possible x-values are governed by chance then the model is stochastic; see Section 9.1 (and it was the seeming necessity that a quantised system should be stochastic which Einstein refused to accept). If the transitions are governed by the value of some

DETERMINISTIC MODELS AND THEIR OPTIMISATION

30

underlying continuous variable ('hidden variables') then x is no longer a state variable. If the transitions in x are determined by x alone then, either they are instantaneous (in which case they occur at an infinite rate) or at some time delay (in which case one has reverted to something like a discrete-time formulation-a so-called discrete-event system or Petri net). (2) Consider the following simple first passage problem. Both x and u take values on the real line; the plant equation is x = u and the instantaneous cost function is c(x, u) = ! (L + Qzl). Termination can take place at either x = 0 or x = 1, with corresponding terminal costs C0 and C1 . Effectively, then, from an intermediate point one heads for whichever of these two termination points is the more favourable, on the basis of control, journey-time and termination costs. The problem is plainly time-invariant and the optimality equation is infu[! (L + Qu2 ) +uFx] = 0, where we have used the subscript notation for partial derivatives: Fx = 8F/8x. ThisissubjecttoboundaryconditionsF(O) =Co andF(l) = C1• Show that u = -Q- 1Fx, that Fx = ±y'IQ and that F(x) =min[ Co+ x..;rQ, cl + (1- x)y'LQ] for (0 <X< 1). One heads for the termination point corresponding to the minimising option at constant speed ,jiJQ. Thus, either F(x) consists of two linear segments or of a linear segment with a discontinuity at one of the end-points (see Figure 2). In the first case, Fx does not exist at the break-point (the point at which the two options are equally costly). In the second, it does not exist at one termination point (that for which it would be less costly to travel to and terminate at the other termination point, if one were permitted). In neither case does it matter: Fx exists in the directions in which one is both able and wishful to move. (3) A rather more general first-passage problem raises all kinds of physical associations. Let the vector x denote the Cartesian coordinates of a particle moving in a Euclidean space of n dimensions. When at position x the particle moves with speed v(x) and in a direction which can be chosen. The equation of motion is thus x = v(x)u, where u is a unit vector which can vary arbitrarily with F(x)

F(x)

(a)

X

(b)

X

Figure 2 The form of the value/unction F(x)for the problem ofminimal-cost passage. In case (a) there is a discontinuity ofF" at the break point at which the two termination points are equally costly. In case (b) there is a discontinuity ofFat a boundary which is never optimal for termination.

31

7. OPTIMAL HARVESTING

tiJne. Let F(x) be the minimal time taken for the particle to reach a stopping set !I' from a point x outside it. Show that the dynamic programming equation is 1 jVF(x)l = v(x)- ; i.e.

Ln (8F)1 -8. = }=I

v(x)-z.

XJ

This is the eikonal equation of geometric optics, a short-wavelength form of the wave equation, to be solved subject to F = 0 in Y'. How is the optimal direction at a given point determined from F ? (4) Consider a version of Exercise 3 in which the magnitude of u may vary. Suppose that the plant equation is i = u and that the integral of the

instantaneous cost c, (x) to be minimised.

+ c2(x)!ul 2 from initial position to termination in Y' is

7. OPTIMAL HARVESTING Let us take as an example the model with which we began: the harvesting example of Section 1.2. The plant equation is i = a(x) - u, where xis biomass, a(x) is the natural net growth rate and u the catch rate. Let us assume a reward function

IR. =

1oo e-aT u dT.

We could define a cost function as the negative of this quantity, but the more natural course is to consider maximisation of a reward, so that F(x, t) will now be the maximal value at time t of the present value of future reward starting from a current biomass of x. The integral will certainly converge if u is uniformly bounded. The problem has a totally time-invariant formulation, so it may be presumed that the value function is a function F(x) of initial state alone. The optimality equation will then be

s~p

[u-

aF(x)

+ (a(x)- u) ~~x)] 8

= 0.

(47)

Let us assume that practical considerations set an upper bound M on catch rate. Note that u occurs linearly in the bracketed expression, with coefficient 1 - Fx (where we use the subscript notation for differentials, see Appendix 1). We can thus say that the optimal value of u is zero for values of x such that Fx > 1 and is M for values of x such that Fx < 1. That is, one does not fish at all if the effective marginal cost in terms of decreased stock exceeds unity (the market price of the catch), and one fishes at the maximal achievable rate if the marginal cost is less than unity. A policy in which the control adopts the extreme values in its permitted set in this way is termed a bang-bang policy.

DETERMINISTIC MODELS AND THEIR OPTIMISATION

32

It seems plausible then that the optimal policy should take the form

u=

{0

(x c)

M

(48)

for some threshold value c. In fact we shall demonstrate that, if the function a(x) is strictly concave and bounded above by M, then the policy (48) is optimal if the threshold has the value determined by d(c) =a

(49)

where d denotes the differential. Note that we have not specified what u should be at the threshold c itself. In fact, specification is pointless. For x < c biomass increases under the policy, for x > c biomass decreases. Under the policy biomass thus converges to the equilibrium value c, and at this value natural growth and catch necessarily balance, so that effectively u = a( c). We say 'effectively' because u could be 'chattering' infinitely rapidly between the two extreme values as xis nudged either side of threshold. However, the time average of catch rate at c must be a( x) over any non-zero time- interval. To prove the assertions above, let V(x) be the value function under the policy (48), for a threshold as yet undetermined. Then plainly V(c) = a(c)/a. For x < c there is zero catch and the biomass grows at rate a(x) until x equals c, which happens first after time c

TJ (x)

=

dy a(y)

1 x

Thus (x < c).

(50)

Correspondingly, one finds that

V(x) = M +a( c)- M a a

e-72(x)

(x

~c)

(51)

where

dy x ( ). r2(x) = , M-ay

1

The term in Min (51) reflects the return from a catch at this rate until the time r 2 when biomass has been brought down to the value c. One readily verifies that the value of c which maximises either of expressions (50) or (51) is that determined by (49). If we can now show that V.~ (with this choice of c) is greater than or less than unity according as xis less than or greater than c then the implication will be that V satisfies the dynamic programming

8 LQ REGULATION IN CONTINUOUS TIME

33

equation (47). We shall see in Exercise 3.1.1 that this implies that V can be identified with F and that the policy is optimal. Verification is direct. It follows from expression (50) that for x ~ c - a V -a( c) -rt(x) Vx - ax ( ) - ax ( )e - exp

[1c x

a'(y)- ad ] ( ) y ;::: I ay

with equality only at x = c. Correspondingly, one finds from (51) that for x ;::: c

v_,

= exp[l" d(y)- a dy] c M- a(y)

~I

with equality only at x = c. Demonstration is thus complete. Note that both Fand Fx are continuous at the optimal threshold. Continuity of Fx constitutes the tangency condition, which we discuss more generally in Exercise 10.7.2. The policy may not seem a very practical one, in that fishing effort would have to vary between extreme values as stock varied in a small interval about the operating point c. However, this would be changed if one chose the performance criterion to put a premium also on continuity of operation. The main issue is to recognise an operating point, with complete cessation of fishing effort as soon as stocks fall markedly below it. The fishing fleet could then be reduced to match the effort required at this point. A more realistic model would also distinguish between 'catch' and 'effort' -the effort per unit catch increasing with decreasing x. Note that the solution makes sense even in the undiscounted case, despite the fact that the future reward IR is then infinite. In this case relation (49) yields a'(c) = 0, so that cis just the value Xm maximising a(x). That is, the operating point is just that chosen for the crude policy of Section 1.2. However, there is the difference that this operating point is now stable, because of the cessation of fishing once stock falls below it. As the discount rate a increases so the operating point-the equilibrium stock level-falls. The reason for this is that, if x exceeds the value recommended, then it is worth while to reap the profits of a quick harvest, even if by so doing one reduces stocks to a lower-yielding equilibrium. The most extreme case is that in which a> a'(O), when the recommendation is to fish the population to virtual extinction. In this the model demonstrates the crudity of the narrow accounting view with its neglect of wider social, economic or environmental considerations. We consider stochastic versions of the model in Chapter 10, and are forced by these to face the issues. 8 LQ REGULATION IN CONTINUOUS TIME In analogy to assumptions (21) and (22) of the discrete-time case we suppose the time-homogeneous forms of plant equation

34

DETERMINISTIC MODELS AND THEIR OPTIMISATION

x= Ax+Bu

(52)

c(x, u)dT + ~[xTnx](h)

(53)

and cost function C=

~

1h

where the instantaneous cost rate c(x, u) still has the form (23). Then the analogues of the discrete-time assertions of Theorem 2.4.2 follow immediately, either by passage to the continuous limit from these or by solution of the continuous-time dynamic programming equation (45). The cost function is quadratic in state

F(x, t) = ~xTn(t)x

(54)

and the time-dependent matrix ll obeys the Riccati equation (0 ~ t
(55)

This is to be regarded as a backward equation for ll(t) with terminal specification ofll(h). The optimal control rule is linear in current state (56) all quantities being evaluated at a common timet. There is thus something of a simplification over the discrete-time case, in that the inverse matrix (Q + BTllB)- 1 appearing in the Riccati equation and control rule is replaced simply by Q- 1• If S is normalised to zero then the Riccati equation (55) reduces to (57)

where J = BQ- 1BT. The matrix J turns out to play an important role; we shall term it the control-power matrix. It is kind of matrix ratio of control effectiveness to control cost, and so measures the effectiveness of control on a cost basis. ·· In most control contexts we are interested in the time-invariant solution ll of (55) and the corresponding stationary policy (56). We shall develop efficient numerical methods for the determination of ll in the next chapter. There are a few classic problems which can be treated explicitly, however. One of these concerns the stabilisation of a pendulum, to either the hanging or the standing ('inverted') position. Let a be the angle of deviation of the pendulum from the vertical. The equation of motion in linearised form (i.e. for small a) is

a= aa +bu. Here the term aa represents the force due to gravity (where a is negative or positive according as a is measured from the hanging or the inverted position) and u is

8 LQ REGULATION IN CONTINUOUS TIME

35

the control force applied to the bob. One can consider stabilisation of a to zero in both cases; one tends to speak of 'the pendulum' or 'the inverted pendulum' according as one is speaking of the first or the second. If we adopt the state variable with components a and ci: then the coefficients in the standard form (52) of the plant equation are

B=

[~].

Let us set

and assume, for simplicity, that Sis zero and that R is diagonal with entries r1 and r2 . The Riccati equation (55) then becomes, in full,

i + r1 + 2ag -

Jg'-

g +I+ ah - Jgh h + '2 + 2g - Jh 2

=0 =

0

(58)

= 0,

whereJ = b /Q. We seek an equilbrium solution of these equations which is non-negative definite and stable. (We discuss stability systematically in Section 5.1). If the problem is properly posed such a solution will exist, and will be both fmite and unique. We find that 2

g=

a+ Ja 2 +Jr1 J ,

h=

J'2 ~2g,

I= h(Jg- a)

(59)

is an equilibrium solution, and is in fact the only one satisfying the conditions for positive definiteness: that f, g and I h - g2 must all be non-negative. Positivity of h implies that we take the positive square root in the expression for h above. Positivity of I implies that Jg - a ~ 0 which implies that we take the ositive square root in the expression for g.We find thatlh- g2 = (r 1 + r2 a 2 + 6r 1)/J so we require that J > 0. It is evident from the solution above that we require J > 0 if TI is to be finite, but we shall see in Section 16.9 that the condition acquires special force for stochastic versions of the model. The solution thus determined is also a stable solution of the Riccati equation; see Exercise 5.1.5. The optimal control is u = -bQ- 1(go: + hci:). Note that this involves ci: as well as a; we shall see that this is necessary for stability. Note also that if r 1 > 0 then r2 can well be zer<~-a control which reduces o: to zero automatically achieves the same fora A rather more substantial version of this model is the classic problem of the cart and the pendulum, illustrated in Figure 3. A cart of mass M can be moved horizontally under an applied control force u, without friction. An inverted

36

DETERMINISTIC MODELS AND THEIR OPTIMISATION

Figure 3 The cart and the pendulum.

pendulum of length L and bob mass m stands on the cart. One observes the horizontal displacement q of the cart and the angle a of the pendulum from the vertical, and the aim is to choose a control rule which stabilises these to zero, say. The difficulty of the problem is the indirectness of the means by which control is exerted on the pendulum. An effective control can be achieved, however, and it is impressive to see this realised; the pendulum stands firmly upright with no perceptible motion of any part of the system. The Newtonian equations of motion are second-order differentials in time, so the state vector x can be chosen as having components (q, q, a, a). It is shown in Exercise 5.2.3 that the linearised (small a) the equations of motion for the system have the form (52) with 00 0I A= 0 0 [ 0 0

-!30 0 "(

0] 0 I ' 0

B-

1/M 0

[

-1/~LM)

l

(60)

where f3 = gm/ M and '"Y = (g/ L) (I + m/ M) and g is the acceleration due to gravity. Let us consider a numerical example with the time and length scales chosen so that g = I, L = 1 and suppose that M = 2, R = /, S = 0 and Q = I. Nothing is gained for illustrative purposes by choosing more exotic values. Solution of the Riccati equation, either directly or by the policy improvement methods of the next chapter, leads to the numerical evaluations II=

4.44 9.37 [ 11.32 11.36

9.37 30.33 38.97 39.21

11.32 38.97 67.06 63.71

11.36] 39.21 63.71 ' 61.86

K

= [1.00

4.44

12.37

11.33].

The relatively large coefficients associated with the angle variables in both matrices is an indication of the difficulty of controlling the pendulum at one

37

8 LQ REGULATION IN CONTINUOUS TIME

remove. It is not surprising that the coefficients of the angle variables inK (and so in the control rule) are positive, since we see from the form of B that angle is affected negatively by u; a push of the cart to the right tips the pendulum to the left. However, since the push indeed displaces the cart to the right (as indicated by the positive entry in B) it is surprising that the coefficients of the displacement variables in the control rule are also positive-it seems that the position correction has the wrong sign. However, if one wishes to move the cart to the right from a position of rest and does so directly, then the pendulum falls to the left. One has first to move the cart to the left, so that the pendulum falls to the right, and to then move the cart to the right in such a way as to both achieve the required displacement and bring the falling pendulum to the vertical again. This is a mild example of the counter-intuitive behaviour of more complex systems (although those with experience of stick balancing will have developed the required sophistication of intuition). Exercises and comments

(1) One again normalises Sto zero by choice of u + Q- 1Sx as transformed control variable. Show that the transformed values of A and R are A - BQ- 1S and R - sT Q- 1S, while those of Band Q are unchanged. (2) Consider the scalar case with S normalised to zero. The equilibrium Riccati equation is R + 2AII = J II2 and the gain matrix r + BK satisfies r 2 = A 2 + JR. Complete the analogue of the analysis of Section 5.

(3) The continuous-time analogue of the problem treated at the end of Section 4 would have a plant equation x = u in scalar x and u, an instantaneous cost rate Qu 2 and a terminal cost Dx2 at time h. Show that the value function and optimal control with time s to go are

!

!

QDxl F

= 2(Q+Ds)'

Dx

u =- Q+Ds·

One could regard x as the lateral displacement of a projectile which will strike its target at time h, and for which the only course deviation explicitly penalised is the terminal miss-distance x(h). However, if the projectile is regarded as having inertia then one should rather take the model of Exercise 4. (4) Consider the 'inertial' version of Exercise 3 in which the two components of the state vector (lateral displacement and its rate of change) obey x1 = x2 and .X2 = u, the instantaneous cost is Qul and the terminal cost is D.xi at time h. Show that the value function and optimal control with times to go are

!

!

D

u = - Q + Ds 3

13 s(xl + sx2).

38

DETERMINISIIC MODELS AND THEIR OPTIMISATION

- x +sx2 . · · d d the predicted terminal miss-distance (in. the absence of ts m ee . Here 1 further control) so the conclusion is consistent With that of Exercise 1.4. Note that for a given value of this quantity the optimal co~trol becomes small for large s • .L lenty oftime remains to make the correction) and also for small.s.(when a (~wuenp given change in x2 scarcely affects terminal XI). In. rocket contexts _It IS more realistic to take the instantaneous cost as proportional to lui, which affects optimisation profoundly; see Section 7.7.

9 OPTIMAL TRACKING FOR A DISTURBED LQ MODEL

In Section 4 we considered an LQ version of the regulation problem, for which the aim was to bring the path to the set point x = 0, u = 0 in an optimal fashion. However, the problems faced in applications are considerably more general. For one thing, the set point could be arbitrary; e.g. one might wish to be able to set a central heating system to stabilise at any prescribed room temperature in a certain range. For another, one may wish to follow a variable set point in time; i.e. to follow a command signal. For example, one may wish a yacht to follow a desired course, taking account of the winds and currents which are expected, or a pottery kiln to follow a desired temperature profile. We shall then generalise the model of Section 4 to incorporate two features: the feature of tracking, i.e. of aiming to follow a command signal, and the feature of disturbances to the dynamics, such as the winds and currents of the yachting example. Both the disturbances and the command signal may well not be known in advance. The sea and weather conditions of the yachting example will certainly be only imperfectly predictable. Further, if the vessel's aim is not to follow a prescribed course, but rather to shadow an unpredictable rival, then the command signal is likewise unpredictable. We shall not be able to handle such issues properly until we allow the effects of stochastic inputs (Chapter 10) and imperfect state observation (Chapter 12). For the moment we shall have to assume that both disturbances and command signal are known in advance, although we can presage the later treatment somewhat if we assume that these also are generated by a model. We begin with the discrete-time version, and can then deduce the continuoustime analogue immediately (see Exercise 3). Let us then assume the plant equation (21) modified to Xr

= AXr-I

+Bur-l+

(61}

d,

where {d,} is a known disturbance sequence. We assume the instantaneous cost function (23) modified to C(Xr, Ur,

_ 1[XU -_ x?] T[RS ST] [xU -_ x?~ ] ~ Q

t) - 2

(62}

t

1

9 OPTIMAL TRACKING FOR A DISTURBED LQ MODEL

39

where {r,'} and {uD are the desired state and control paths, known in advance. Normally one would regard x~ as the command signal and assume that the ideal value for u1 would be the 'minimal effort' value, probably zero. However, we can as well allow prescription of a general control profile. The terminal cost function generalises correspondingly to ~ [(x- r?II(x- x")]h, where the h subscript indicates that all quantities in the bracket are evaluated at time h. We can somewhat reduce the problem by changing to the variables x- 1

=

X 1 -

c x 1 , Ur

=u

1 -

uc1

representing deviation from the desired path. The plant equation (61) then holds in these new variables if we substitute

d, = d1 - d~ = d,- x~ + Ax~-l + Bu~-l for d1• This normalised d represents the effective disturbance in 'deviation' terms; it is the old disturbance less the term d~

= x~- Ax~-l -

Bu~_ 1 .

(63)

This modifying term represents the 'unnaturalness' of the prescribed path for the plant, and is zero if the desired path itself satisfies the undisturbed plant equation. This is a point which will manifest itself repeatedly. The dynamic programming treatment is straightforward and yields interesting conclusions, but by somewhat laborious calculations. We shall only sketch these, because a much more powerful and insightful approach will emerge in Chapter 6. Assume xc and uc normalised to zero, as above. One finds then that the quadratic form (24) of the value function is replaced by a non-homogeneous quadratic form

F(x 1 , t) = ~xTII 1 x 1 - aT x, +Or

(64)

The matrix II1 obeys exactly the same equation as before, the Riccati equation (25) I (26), and the optimal control is

u1 = K 1x 1 - D~ 1 BT(IIr+ldr+l

+ ar+l)·

Here the feedback matrix K 1 also has exactly its previous value (28) and D 1 = Q + BTIIr+JB.

The coefficient a 1 of the linear term, which reflects the effect of disturbances and command signals, obeys the backward recursion

a,= rT(ar+l- IIr+ldr+J) where r, is the gain matrix defined in (31).

40

DEfERMINISTIC MODELS AND THEIR OPTIMISATION

The term 61 in (64) is also interesting; it is that component of the cost of future operations which is independent of the effect (presumably transient) of the value of initial state. For the regulation case of Section 4 it had the value zero; in the present case it reflects the total penalty for tracking error incurred up to the horizon point. One can derive a recursion for it, but we shall not do so. The term does not affect the control rule, we shall find other ways of assessing tracking error, and shall in any case see that the recursion can be expressed much more elegantly in a more general formulation (see Section 18.2). From these conclusions w~ can progress to the final conclusions, stated for the unreduced formulation. Theorem 2.9.1 Assume the plant equation (61) and instantaneous cost function (62). Then the optimal open-loop control rule and the optimally controlled plant equation take the respective forms h-1-i 1 u~-u~=K~(x~-x~)- n~ BT r"J+ 1r"J+ 2 .. -ri+1rri+J+i(di+J+l- d~+J+i) (65)

L

j=O

-BD~ BT 1

h-1-i

L

j=O

r"J+ 1 ri+ 2 ···fi+1 rr~+J+l(dt+J+l-d~+J+l)

(66)

where~ isdefmedin (63).

We see that the optimal control (65) now has afoedback-feedforward form. The feedback term K 1(x 1 - ~) is exactly as previously, except that x 1 has been replaced by the deviation x 1 - x~. The sum represents the feedforward term; a term which expresses an optimal anticipation of future disturbances and command signals. If we consider the infinite-horizon case and assume that infinite-horizon limits exist then the rule (65) will simplify to 00

ul- u~

= K(x1- x~)- n- 1BT L(rT)fii(di+J+l- d~+J+ 1 ).

(67)

J=O

We shall show in Section 6.1 that existence of the infinite-horizon limit for II is usually associated with the fact that the limit gain matrix r is a stability matrix. By this we mean that its powers rJ decay to zero with increasing j; in fact, exponentially fast. One can then see the feedforward term in (67) as incorporating an automatic 'matrix discounting' of the future. Intuitively: the further ahead that a problem lies, the less urgent is it. For a simple example, suppose that the amount of water x in a reservoir obeys the recursion

9 OPTIMAL TRACKING FOR A DISTURBED LQ MODEL

Xr = Xr-! - Ur-!

41

+ dr,

where u represents draw-off (to be chosen) and d represents inflow (supposed known in advance). The instantaneous cost function is R(x- r) 2 + Q(u- uc/. That is, one would like to secure uniformity of utilisation rate by holding u as near to the constant value uc as possible. The x-component of cost represents an attempt to enforce the constraints on x: that it should lie between zero and the capacity of the reservoir. One can enforce these constraints rigidly (see Section 7.13) but in an LQ formulation one can merely penalise deviations from half-capacity, say. Note the function of a reservoir: to act as a buffer which smooths out variations in inflow to deliver a more uniform outflow Applying the above theory we find that the optimal infinite-horizon rule is

!

!

Ur

= (J- r) (xr- Xc +

f:

fldr-'-j+l)

(68)

}=0

where r is, as in Section 5, the smaller root of Q(I - f) 2 = Rr. Note that this rule is independent of uc. The average level of outflow must equal the average level of inflow, whatever that may be, and there is no point in trying to prescribe it. The situation would be different if the two did not necessarily balance in the long run; see Exercise I. It is interesting that the 'discounting' matrix in (67) should just be the transpose of the gain matrix r. Indeed, there is a duality between past and future, as will emerge in Chapter 6. For validity of the infinite-horizon version (67) it is certainly necessary that the sum should converge. Suppose that the elements offi converge to zero as pi with increasingj, for a positive scalar p. The sum in (67) will then certainly converge if d1 - d; grows as p, 1 with jp,j < p- 1• However, for successful tracking we would really demand rather more than that. Ideally, the tracking errors should converge to zero with increasing t. A less stringent demand is that they should remain uniformly bounded, so that costs accumulate at a bounded rate. We see from (67) and the corresponding infinite-horizon version of (66) that these demands will holdiftheyhold for the sum 6 1 = L:~(rT) 1 TI(dr+J- d;~)· Ifthe disturbance d1 is uniformly bounded in value then its contribution to 6 1 will also be uniformly bounded; a conclusion which we can strengthen in Chapter 10, when disturbance is stochastic in nature. However, the path x~ which one is attempting to follow may well go off to infinity at some rate and the question is: what rates are consistent with vanishing or with bounded tracking errors? For simplicity, consider the case when both d1 and u~ are zero. If d~ is of order p, 1 for large t (with 0 ::::;; p, < p- 1) then so is 6 1 • For tracking errors to converge to zero with increasing tit is thus necessary that d~ = x7- A_r,-_ 1 should converge to zero with increasing t; a conclusion which we could express as follows.

42

DETERMINISTIC MODELS AND THEIR OPTIMISATION

Theorem 2.9.2 A necessary and suffzcient condition that the component of tracking error due to variation in the command signal should tend to zero with increasing time is that all unstable modes of the command signal should satisfy the uncontrolled plant equation. The continuous-time analogues of these assertions are immediate; see Section 6.2. In developing LQ theory further in Chapter 6 we relate it to classic control theory, among other things, so continuous time then provides the more natural frame. Exercises and comments (1) Work through the reservoir example in the case when the plant equation is x 1 = /3Xr-l -Ur-I+ d1• Here f3 is a coefficient lying between 0 and 1; reflecting the fact that a fraction 1 - f3 of the water is lost over a stage by leakage or evaporation. (2) Suppose that the inflow d is itself known to obey an equation d1 - v = o:(dr-1 - v). The relation (68) yields the control rule c o:(I - r) u1 = v + (1- r)(xr- X)+ l _ o:r (dr- v),

provided lo:rl < 1. (3) Suppose that d1 and u~ are identically zero, that S has been normalised to zero and that infinite-horizon limits hold Show that the optimal control is 00

ur = Kxr- D-

1BT ~)rT)j R.x~+j+l· j=O

Suppose that the command signal is generated by r, = Cw 1 where w1 satisfies a recursion w, = A 1Wr-1· Show then that u 1 = Kx 1 + K1 w1, where K1 = - D- 1BT

f:}:o(rT)j RcA{+ 1•

10 OPTIMAL EQUiliBRIUM POINTS: NEIGHBOURING OPTIMAL CONTROL A minimal aspiration would seem to be: to locate the optimal point at which a controlled deterministic process should settle in equilibrium. That is, we suppose a stationary control rule, suppose that it has a stable equilibrium point, and then ask what would be the optimal location of this equilibrium. This is to be contrasted with the more ambitious programme of (i) fmding an optimal control rule, (ii) establishing that it has stationary limit form in the infmite horizon, and (iii) establishing that there is a stable equilibrium under this stationary policy.

43

10 OPTIMAL EQUILIBRIUM POINTS

One hopes that the two paths lead to the same conclusion, but the first certainly easier. Consider the dynamic programming equation F(x, t) = inf[c(x, u) u

+ j3F(a(x, u), t + 1)]

IS

(69)

for a discrete-time discounted state-structured process under the assumptions of a time-homogeneous plant equation Xr+l = a(x 1 , u 1) and instantaneous cost [unction c(x, u). We shall assume that both x and u are vector-valued and unconstrained and that the functions a, c and F have as many derivatives as required. (In fact, the strongest assumption required is that differentials of second order should exist and be continuous). Actually, both c and a could be considered time-dependent in what follows, except when we demand existence of a static equilibrium. If x and u are understood as column vectors, then recall that, by the conventions of Appendix 1, the vector of first derivatives Fx(x, t) is a row vector. Define (70) the negative transpose of Fx evaluated at (x,, t). We shall consider the case when {x 1} is the optimal orbit for prescribed x 0 , so that { >. 1 } is then the sequence of differentials (70) defined on this optimal orbit. If we set x = x 1 in (69) then the minimality condition with respect to u1 will imply a stationarity condition

(71) where the row vector Cu and matrix au of derivatives are evaluated at time t. Differentiation of (69) with respect to x 1 yields the companion equation

(72) Theorem 2.10.1 (Discrete time) Assume that the differentials above exist and that the optimally controlled process has an equilibrium point. Then the values ofx, u and>. at an optimal equilibrium must satisfy the three equations x=a,

(73)

where arguments x and ufor a, c and their derivatives are understood. This follows simply by taking the equilibrium condition x = a together with the equilibrium forms of equations (71) and (72) above. Note the necessity for introduction of the conjugate variable >.; it was because of this that we had to establish the dynamic equations (71) and (72) first (see Exercise 1). One can, of course, eliminate >.to obtain the pair ofequations

44

DETERMINISTIC MODELS AND THEIR OPTIMISATION

(74)

x= a,

The fact that the optimal equilibrium point varies with f3 is something we have already observed for the continuous-time model of Section 7, for which the optimal equilibrium point c was determined in terms of the discount rate a by equation (49). Discounting has the effect of encouraging one to realise return early in time, so there is an inducement to take a quick immediate yield from the system, even at the cost of sinking to a lower-yielding equilibrium. Equations (73) may have several solutions, of course, and the eligible solution must at least be stable. For a case in point, one can consider an optimal fishing policy under a model which allows for age structure in the fish population. For certain parameter choices, there is an equilibrium in that the optimal fishing pattern is constant from year to year. For other parameter values, one should opt for the non-stationary policy of 'pulse fishing', under which one allows the population to build up for a period before harvesting it; a static equilibrium solution may or may not be unstable under these circumstances. For the continuous-time analogue of this analysis the plant equation· is .X= a(x, u) and the dynamic programming equation is inf[c - aF +a;+ Fxa] u

ut

= 0,

(75)

where a is the discount rate. The equations analogous to (71) and (72) are (76)

(77) We shall encounter these again in Chapter 7 when we develop the Pontryagin maximum principle; relations (76) and (77) are then seen as conditions that the orbit be optimal. For present purposes, we deduce the analogue of Theorem 2.10.1. Theorem 2.10.2 (Continuous time) Assume that the differentials above exist and that the optimally controlled process has an equilibrium point. Then the values ofx, u, and .X at an optimal equilibrium must satisfy the three equations a

=0,

(78)

The question that should now really be faced is: what should the optimal control policy be in the neighbourhood of the equilibrium point? That is, if the equilibrium values are denoted by .X and u, then how should u vary from u as x varies slightly from x? To determine this we must consider second-order effects and obtain what is in essence an LQ approximation to the process in the neighbourhood of equilibrium. More generally, one can do the same in the neighbourhood of an optimal orbit.

10 OPTIMAL EQUILIBRIUM POINTS

45

Consider again the discrete-time case, and define II 1 as the value of the square matrix of second-order derivatives Fxx on an optimal orbit at time t. Let 6.x 1 denote a given perturbation of Xr from its value on the orbit at this point and 6.ur the corresponding perturbation in ur (whose optimal value is now to be determined). Theorem 2.10.3 (/)iscrete time) Assume that all differentials now invoked exist. Define the Hamiltonian at timet, H(xr, ur, Ar+t) = -c(xr, ur) + /3A"[+ 1a(xr. u 1 ) and the matrices Ar =ax, Br =au, Rr = -Hxx. Sr = -HUX> Qr-'-1 = -Huu; these being evaluated on the original optimal orbit at time t. Then the matrix Dr satisfies the Riccati recursion

Dr= [R + j3ATIIr+1A- (ST

+ /3ATIIt+tB)(Q + j3BTIIr+IB)- 1(S + /3BTIT 1+tA)Jc (79)

where all terms in the square bracket save II are to bear the subscript t. The perturbation in optimal control is, to first order, 6.u 1 = K16.x 6 where

(80) Proof Set x = x 1 + 6.x 1 and u = u 1 + 6.ur in the dynamic programming equation (69), where x 1 and u1 are the values on the original optimal orbit, and expand all expressions as far as the second-order terms in these perturbations. The zerothorder terms cancel in virtue of the equation (69) itself, and the first-order terms cancel in virtue of relations (76) and (77). One is then left with a relation in second-order terms which is just the equation

!xTII 1x

= inf[c(x, u) + !!3(Ax + Bu? IIr+l (Ax+ Bu)] u

with x and u replaced by 6.x 1 and 6.u 1 and A, B, R, S and Q replaced by the tdependent quantities defined above. The conclusions thus follow. 0 The interest lies in the replacement of the cost function c(x, u) (with cost matrices R, Sand Q) by the Lagrangian-like expression c(x, u)- /3ATa(x, u). This is the negative of the Hamiltonian which will play such a role in the discussion of the maximum principle in Chapter 7. The additional term -j3ATa(x, u) would in fact contribute nothing at this point if a(x, u) were linear: it is the non-linearity in the plant equation which adds an effective supplementary cost. The negative signs which occur in our definition of A come from a desire to be consistent with convention; these signs would all be positive if one had phrased the problem as one of reward maximisation rather than of cost minimisation. This perturbation calculation is, of course, of no value unless the perturbed orbit continues to lie close to the original optimal orbit. So, either one must be

46

DETERMINISTIC MODELS AND THEIR OPTIMISATION

considering events over a horizon which is short enough that perturbations remain small, or the perturbed orbit must in fact converge back to the original orbit. The implication in the latter case is that the original orbit is an attractor under the optimal control rule. This would be a rather special circumstance, except in the particular case when the original orbit had itself settled to a stable equilibrium value. In such a case the matrices TI and K will also be independent oft.

The continuous-time analogue follows fairly immediately; we quote only the undiscounted case. Theorem 2.10.4 (Continuous time) Assume that all differentials now invoked exist. Define the HamiltonianH(x, u, .h) = -c(x, u) + .hTa(x, u) and the time-dependent matrices A= ax, B =a,., R = -Hxx• S =-flux, Q = -H,.,.; these being evaluated on the original optimal orbit at time t. Then the matrix TI = Fxx ~valuated on the original orbit) satisfies the Riccati equation

(81) The perturbation in optimal control is, to first order, 6-u time-dependent value

= K6.x,

where K has the

(82) For the harvesting model of Section 7 the Hamiltonian is linear in u, so that Q = 0 and the above analysis fails. Such cases are spoken of as singular. As we see from this example, an optimal control with an equilibrium may nevertheless very well exist. The singularity reflects itself in the discontinuous nature of the control rule. Exercises and comments (1) One could have derived the conditions of Theorem 2.10.1 in the undiscounted case simply by minimising c(x, u) with respect to x and u subject to the constraint x = a(x, u). The variable A then appears as a Lagrange multiplier associated with the constraint. The approach of the text is better for the discounted case, however, and necessary for the dynamic case.

CHAPTER 3

A Sketch of Infinite-horizon Behaviour; Policy Improvement Suppose the model time-homogeneous. In the control context one will very frequently consider indefinitely continuing operation, i.e. an infinite horizon. If the model is also such that infinite-horizon operation makes physical sense then one will expect that the value function and the optimal control rule will have proper infinite-horizon limits, and that these are indeed time-independent. That is, that the optimal control policy exists and is stationary in the infinite-horizon limit. In this chapter we simply list the types of behaviour one might expect in the infinite-horizon limit, and that typifY applications. Expectations can be false, as we illustrate by counter-example; dangers are best put in proportion if they are identified. However, we defer any substa~tial analysis until Chapter 11, when more examples have been seen and the treatment can be incorporated in that of the stochastic case. Coupled with this discussion is introduction of the important and central technique ofpolicy improvement. The infinite-horizon limit should not be confused with the limit of equilibrium behaviour. If the model is time-homogeneous and the control rule stationary then one can expect behaviour to tend to some kind of equilibrium with time, under suitable regularity conditions. It would then be more appropriate to regard equilibrium as an infinite-history limit.

1 A FORMALISM FOR THE DYNAMIC PROGRAMMING EQUATION We suppose throughout that the model is state-structured, so that we can appeal to the material of Sections 2.1 and 2.6. Consider the discrete-time case first. We shall suppose the model time-homogeneous, so that the time argument can be dropped from a(x, u, t) and c(x, u, t), but we shall allow the possibility of discounting. The value function F(x, t) can then be written Fs(x), where s = h - t is the time to go. We achieve considerable notational simplification if we write the dynamic programming equation (2.10) simply as

Fs = ff'Fs-1 where 2 is the operator with action

(s > 0)

(1)

48

A SKETCH OF INFINITE HORIZON BEHAVIOUR

!l'¢(x)

= inf[c(x, u) + f3¢(a(x, u))] u

(2)

on a scalar function of state ¢(x ). The interpetation of !l' ¢(x) is that it is the cost incurred if one optimises a single stage of operation: so optimising u1, say, knowing that x 1 = x and that one will incur a closing cost ¢(x1+1) at time t + 1. The operator !l' thus transforms scalar functions of state into scalar functions of state. We shall term is the forward operator of the optimised process, since it indicates how minimal costs from timet (say) depend upon a given cost function at time t + 1. The term is quite consistent with the fact that the dynamic programming equation is a backward equation. Let us also consider how costs evolve if one chooses a policy 7l" which is not necessarily optimal. If the policy 7l" is Markov in that u1 depends only t and the value x of current state x 1 then the value function from time t will also be a function only of these variables, V( 1!", x, t), say. If the policy is also stationary then it must be of the form u, = g(x 1) for some fixed function g. In this case the policy is often written as 7!" = gCool, to indicate indefinite application of this constant rule, and one can write the value function with times to go as V5 (g(oo), x). The backward cost recursion for this fixed policy is Vs+I(g(ool,x) = c(x,g(x))

+ /3V5 (g(ool,a(x,g(x))),

(s > 0)

(3)

a relation which we shall condense to

(4) Here L(g) is then the forward operator for the process with policy g(oo). If it can be taken for granted that this is the policy being operated then we can suppress the argument g and write the recursion (4) simply as

Vs+l = LV5 • The operators L and !l' transform scalar functions of state to scalar functions of state. If one applies either of them to a function ¢, then it is assumed that ¢is a scalar function of state. One important property they share is that they are monotonic. By this we mean that !l'¢ ~ !l'.,P if¢> ~ .,P; similarly for L.

Theorem 3.1.1 (i) The operators !l' and L are monotonic. (ii) If F 1 ~ (::::;; )Fo then the sequence {Fs} is monotone non-decreasing (nonincreasing); correspondingly for { V5 }. Proof Lis plainly monotonic. We have then ifwe take u = g(x) as the particular control rule induced by formation of !t'¢, i.e. the minimising value of u in (2} Assertion (i) is thus proven.

2 INFINITE-HORIZON LIMITS FOR TOTAL COST

49

Assertion (ii) follows inductively. IfF, ~Fs-l then Fs+l = fil F, ~ fil Fs-l =F,.

0 In continuous time we have (with the conventions of Section 2.6) the analogues of relations (I) and (4): oF= vHF

OS

,

oV

OS

= MV

(s > 0).

(5)

Here F and V are taken as functions of x and s and the operators vlt and M = M(g) have the actions . o¢(x) vlt¢(x) = mf[c(x,u)- a¢(x) +--a(x,u)], u 0X o¢(x) M¢(x) = c(x,g(x))- a¢(x) + ---ax-a(x,g(x)).

Exercises and comments (1) Consider a time-homogeneous discrete-time model with uniformly bounded instantaneous cost function and strict discounting (so that f3 < 1). Suppose also (for simplicity rather than necessity) that the state and control variables can take only finitely many values. Define the norm 11¢11 of a scalar function ¢(x) of state by supx I¢(x) I, the supremum norm. Let f!J be the class of functions bounded in this norm. Show that for¢ and 1/J in f!J we have II fil ¢ - ff'¢!1 ~ .BII ¢ -1/JII. Hence show that the equilibrium optimality equation F = fil F has a unique solution Fin f!J, identifiable with lim5 _, 00 fi7 5'¢ for any ¢ of f!J, and that the uj x relation determined by ff' F defines an optimal infinite-horizon control rule.

2 INFINITE-HORIZON LIMITS FOR TOTAL COST The fact that one faces an infinite horizon, i.e. envisages indefinite operation, does not mean that the process may not terminate. For example, if one fires a guided missile, it will continue until it either strikes some object or falls back to Earth with its fuel spent. (For simplicity we exclude the possibility of escape into space.) In either case the trajectory has terminated, with a terminal cost which is a function IK(i) of the terminal value .X of state. The problem is nevertheless an infinite-horizon one if one has set no a priori bound on the time allowed. If one had set a bound, in that one considered the firing a failure if the missile were still in flight at a prescribed time h, then presumably one should assign a cost Ch(xh) to this contingency. The cost Ch might then more appropriately be termed a closing cost, to distinguish it from the terminal cost IK, the cost of natural termination.

50

A SKETCH OF INFINITE HORIZON BEHAVIOUR

In the infmite-horizon case h is infinite and there is no mention of a closing cost One very regular case is that for which the total infinite-horizon cost is well defined, and the total costs V(x) and F(x) (under a policy gCco) and an optimal policy respectively) are finite for prescribed x. If instantaneous cost is nonnegative then this means that the trajectory of the controlled process must be such that the cost incurred after time t tends to zero with increasing t. One situation for which this would occur is that envisaged above: that in which the process terminates of itself at some finite time and incurs no further cost Another is that discussed in Exercise 1.1, in which instantaneous costs are uniformly bounded and discounting is strict, when the value at time 0 of cost incurred after time t tends to zero as /3 1• Yet another case is that typified by the LQ regulation problem of Section 2.3. Suppose one has a fixed policy u = K.x which 1 is stabilising, in that the elements of (A+ BK) tend to zero as p1 with increasing 1 t. Then x 1 and u1 also tend to zero as p , and the instantaneous cost c(x1 , u1) tends to zero as fl'-'. Although there is no actual termination in this case, one approaches a costless equilibrium sufficiently fast that total cost is finite. One would hope that finiteness of total cost would imply that V and F could be identified respectively as the limits of Vs = Lsch and of Fs = 2sch ass -+ oo, for any specified closing cost Ch in some natural class CC. One would also hope that these cost functions obeyed the equilibrium forms of the dynamic programming equations F = !i'F, (6)

V=LV

(7)

and that they were the unique solutions of these equations (at least in some natural class of functions). Further, that the minimising value ofu in (6) would determine a stationary optimal policy. That is, that if

F= .ff'F=L(g)F (8) then g
51

2 INFINITE-HORIZON LIMITS FOR TOTAL COST

timet under the policy g(oo) for a prescribed value of initial state. Then the assertion of the theorem simply amounts to the statement s

00

L c =lim Lc sfoo 1

0

1,

0

valid for a sum of non-negative terms, whether convergent or not. Theorem 3.2.2 Suppose that c ~ 0 and that the function cf;.(x) is such that¢ L(g)¢ <¢.Then V(g(oo)) < ¢.

0 ~

0 and

Proof Here by L(g)¢ < ¢we mean that inequality holds for all x and strict inequality for some x. Let us denote L(g) and V(g(oo)) simply by Land V Monotonicity of L implies that

L'O:::;; L'¢ < ¢

(s= 1,2,3, ... ).

Lettings tend to infinity, we deduce the assertion of the theorem.

0

A function ¢(x) such that L(g)¢ :::;; ¢ is termed L(g)-excessive. It can be regarded as a cost function with the property that, if one has the options of either continuing with the policy g(oo) or of stopping and incurring a cost ¢(x) (where x is the current state), then one never does worse by continuing. The argument of Theorem 3.2.2 then implies that V(g(oo)) is the least non-negative L(g)-excessive function. This observation implies the following stronger statement. Theorem 3.2.3 Suppose that the instantaneous cost function is non-negative. Then V(g(oo)) is the minimal non-negative solution of V = L(g) V. Correspondingly, Fis the minimal non-negative solution ofF = .!£F.

Exercises and comments The following counterexamples to the regular behaviour envisaged in the text illustrate various types of behaviour. One is that the incurral of some element of cost can be indefinitely postponed but not ultimately avoided; this explains the instability shown in Exercise 1. Another is that a notional closing cost can affect costs and decisions at all horizons; this explains the non-uniqueness shown in Exercise 2 and 3. (In fact, it is taken as part of the infinite horizon specification that the closing cost is zero. This sets an absolute origin to cost, so that costs cannot necessarily be normalised by addition of a constant.) Finally, the case in which one maximises a positive reward ('positive programming') has the feature that an improvement in policy takes one away from the fixed bound of zero rather than towards, and this has consequences; see Exercises 4 and 5.2. Some exceptional cases are more easily illustrated by a stochastic example; see Exercises 11.1.1, 11.1.2 and 11.2.2.

52

A SKETCH OF INFINITE HORIZON BEHAVIOUR

The reader is invited to investigate the effect of discounting or of the addition of a time penalty (in the fonn of a constant positive component of instantaneous cost) on all examples. (1) Consider a process whose state space consists of the non-negative integers j = 0, I,2, ... and a supplementary state a. In state a one can move to any integer state j > 0; in state j > 0 one can move only to j - 1; state 0 is absorbing. All transitions are costless, except that from 1 to 0, which carries unit cost. Then F3 (a) is zero, because one can move from a to a j so large that the transition I ----> 0 does not occur before closing. Thus F 00 (a) := limF3 (a) = 0. On the other hand F( o:) = I, because the transition I ----> 0 must occur at some time under any policy (i.e. choice of move in state o:). The fact that Fr:10 #- F means that optimisation does not commute with passage to the infinite-horizon limit, and is referred to (unfortunately) as instability. (2) Suppose that one can either continue, at zero cost, or terminate, at unit cost. There is thus effectively only one continuation state; if F is the minimal cost in this state then the equation F =IfF becomes F = min(F, 1). The solution we want is F = 0, corresponding to the optimal policy of indefinite continuation. However, the equation is solved by F = K for any constant K ~ 1; that for K = 1 is indeed consistent with the non-optimal policy that one chooses termination. K can be regarded as a notional closing cost, whose value affects costs and decisions at all horizons. It is optimal to continue or to terminate according as K ~ I or K ~ 1. In fact, K = 0, by the conventions of the infinite-horizon formulation, but the non-uniqueness in the solution of the dynamic programming equation reflects a sensitivity to any other specification. (3) A more elaborate version of the same effect is to assume that x and u may take integral values, say, and that the plant equation and cost function are such as to imply the equilibrium dynamic programming equation.

F(x) = min[lul u

+ F(x- u)].

=

The desired solution is F = 0, u 0, corresponding to a zero closing cost. However, there are many other solutions, as the reader may verifY. corresponding to a non-zero notional closing cost. (4) Consider a process on the positive integers x = 1, 2, 3, ... such that when, inx, one has the options of either moving to x + 1 at zero reward ('continuation') or retiring with reward 1 - I/ x ('termination'). This then is a problem in 'positive programming': one is maximising a non-negative reward rather than minimising a non-negative cost If G(x) is the maximal infinite-horizon reward from state x then the dynamic programming equation is G(x) = max[ G(x + 1), 1 - 1/ x]. This is solved by any constant G ~ 1, corresponding to the specification of some x-dependent closing reward which exceeds or equals the supremum terminal

3 AVERAGE-COST OPTIMALITY

53

reward of 1. However, we know that there is no such closing reward, and we must restrict ourselves to solutions in G ~ I. The only such solution is G = I, attained for indefinite continuation. But indefinite continuation is non-optimal-one then never collects the reward. In short, this is a case for which there is a g such that !l' F = L(g)F for the correct F, but g(oo) is nevertheless non-optimal. In fact, no optimal solution exists in this case. If one decides in advance to terminate in state x, then there is always an advantage in choosing x larger, but x may not be infinite.

3 AVERAGE-COST OPTIMALITY In most control applications it is not natural to discount, and the controlled process will, under a stationary and stabilising policy, converge to some kind of equilibrium behaviour. A cost will still be incurred under these conditions, but at a uniform rate 1, say. The dominant component of cost over a horizon h will thus be the linear growth term [h, for large h. For example, suppose we consider the LQ regulation problem of Section 2.4, but with the cost function c(x, u) modified to! (x - q) T R(x- q) +! uT Qu. One is thus trying to regulate to the set point (q, 0) rather than to (0, 0). At the optimal equilibrium a constant control effort will be required to hold x in at least the direction of q. One then incurs a constant cost, because of the constant offset of (x, u) from the desired set point (q, 0); see Exercise I. More generally, disturbances in the plant equation will demand continuing correction and so constitute a continuing source of cost, as we saw in Section 2.9. With known disturbances cost is incurred at a known but time-varying rate. One could doubtless develop the notion of a long-term average cost rate under appropriate hypotheses, but a truly time-invariant model can only be achieved if disturbances are specified statistically. For example, we shall see in Section 10.1 that, if the disturbance takes the form of 'white noise' with covariance matrix N, then the minimal expected cost incurred per unit time is 1 = !tr(NII). Here IT is the matrix of the deterministic value function derived in Section 2.4. In general, there are many aspects of average-cost optimality--<:oncepts and counterexamples-which are best discussed in the stochastic context, and which we shall defer to Chapter 11. Let us denote the cost per unit time for the policy g(oo) and for an infinitehorizon optimal policy by lg and 1 respectively. It is as yet not clear how one is to tackle infinite-horizon cost evaluation or optimisation; if costs are going to build up at a constant rate then the total cost over an infinite horizon is certainly infinite. One way of reducing the situation to that of the last section is to subtract lg or 1 from the cost c. With this normalisation one has a problem for which total cost may again be well defined, under appropriate regularity conditions. The effect of the normalisation will be to change the dynamic programming equation V = LVunderthepolicyg(oo) to

54

A SKETCH OF INFINITE HORIZON BEHAVIOUR

rg + v =

Lv.

(9)

Here v(x) is the infinite-horizon value function for the reduced-cost problem, presumed finite. We interpret v(x) as the component of infinite-horizon cost arising from the fact that the operation started in state x rather than in equilibrium; we shall term it the transient cost. One presumes that equation (9) and the condition that v be finite determines rg· However, the absence of discounting means that (9) determines v only to within an additive constant. This constant can be regarded as arising from an irrelevant closing cost, and can be normalised by prescribing the value of v(x) for somex. Actually, if the controlled process has several equilibrium points then both average cost rg and the arbitrary 'constant' of integration will depend upon the particular equilibrium to which the process converges, and so will be independent of starting point x only for starting points with the same terminal equilibrium. Otherwise expressed, they are constant within the domain of attraction of a given equilibrium point, but will in general vary between domains. We expand on this point in discussion of the continuous-time case below. The analogue of the dynamic programming equation (6) for the optimally controlled process will be r+f

= !Ef

( 10)

where r is the minimal average cost andf(x) the minimal transient cost. If this optimally controlled process has distinct equilibria of different average cost then it must be that there is no finite-cost manoeuvre which will bring the state value from a domain ofhigher r to one of oflower r· If there were, then one would use it. We have the continuous-time equivalents of (9) and (10)

rg

= Mv,

1 = .Af

with rg and 1 still interpretable as rates of cost per unit time. We can write the first of these equations more explicitly as

8v(x)

rg = c(x,g(x)) + ~a(x,g(x)).

( 11)

Suppose that i is an equilibrium point of the process under policy g(oo), so that a(i, g(i)) = 0. If we assume that v(x) has finite differential at i then it follows

from (11) that the average cost is determined, not surprisingly, as

rg = c(i,g(i)).

(12)

If v( x) is indeed to be interpretable as the increase in total cost incurred by a start from x rather than from the equilibrium value i then it follows that the constant of integration is fixed by the condition v(i) = 0. Suppose now that we try to choose the policy optimally. We see from the above that there are two levels of optimality. First of all, one could try simply to choose

4 GROWTH OPTIMALITY

55

the control rule u = g(x) so that the equilibrium value xis such as to minimise expression (12). This is the criterion of average cost optimality; the equilibrium values of x and u which achieve it are determined in Theorem 2.10.2 (with the discount rate a set equal to zero). However, this does not determine the control rule away from equilibrium. A necessary property is that the rule should stabilise the equilibrium; a desirable property if that it should minimise the cost of passage to equilibrium. The full dynamic programming equation 'Y = .Af achieves this, by minimising the costf(x) of passage to equilibrium as well as the cost"' per unit time at equilibrium. The distinction between optimisation at and optimisation to the equilibrium point generally disappears both in the stochastic case and in practice, when one must assume that the process is constantly disturbed from what would be its deterministic equilibrium point. Exercises and comments (1) Consider the problem of regulation to (q, 0) considered in the text with S = 0. Show that the actual equilibrium value of xis x =(I- R- 10)q and the minimal average cost is"(= qTOq, where

4 GROWTH OPTIMALITY Suppose that the cost (or reward) incurred over a horizon oflengthh is of order/', where p > 1. That is, cost (reward) grows exponentially fast. This is the type of behaviour encountered in economic contexts, when p is the rate of economic growth. Maximisation of total reward over a long horizon amounts primarily to maximisation of the growth rate, just as it amounted to minimisation of the cost rate in the last section. This is not the kind of behaviour one encounters in conventional control contexts, but we sketch a few ideas so as to complete the gallery of types: processes for which (in rough terms) total cost remains bounded, grows linearly or grows expone~tially as the horizon recedes. The example we consider is the simplest possible, but serves to illustrate a number of points. Suppose that one can operate a number of activities, each of which both consumes and produces various commodities at rates proportional to the intensity with which the activity is pursued Let the JCh element of the column vector x, denote the intensity with which activity k is pursued at time t. Let ajk denote the amount of commodity j consumed by activity kat unit intensity and bjk the amount produced. Let b1 denote the amount of commodity j which is naturally available per unit time. One then has the constraints x, ;;:: 0 and

56

A SKETCH OF INFINITE HORIZON BEHAVIOUR

Ax1 ~ b + Bx1-J. (13) Inequality (13) reflects the fact that at time t one cannot choose an activity pattern which consumes more of any commodity than is available. Here A and B are the matrices and b the column vector with elements a1k, bJk and b1 respectively. One must regard (13) as the plant equation. The latitude of choice in the fulfilment of this inequality represents the latitude of choice of decision: the choice of the intensity vector x 1• Suppose one aims to maximise 2:~ r(x1 ), where r(x) is the utility associated with an intensity vector x. Briefly, one can imagine that some of the production is siphoned off into personal consumption, which yields satisfaction (i.e. utility). However, personal consumption can be regarded as just another activity, albeit an unproductive one. The choice of rate of personal consumption can thus be lumped in with choice of the intensity vector x. Suppose now that one can find a non-negative vector x and a scalar p > 1 such that pAx~ Bx.

That is, one can achieve an expanding economy, x 1 oc p 1x, even in the absence of the external resources represented by b, at least if initial conditions are such that all activities represented in x can be started up. If r(.\x) ~ +oc for this x and with increasing positive scalar ,\ (so that personal satisfaction is guaranteed by the programme) then one sees that, in the limit of large h, optimisation of reward amounts primarily to maximisation of the growth rate p. The maximal value of p consistent with (13) is indeed a solution of the eigenvalue problem (pA- B)x = 0, p being the maximal solution of jpA- Bi = 0 and x the corresponding eigenvector. The fact that the elements of A and B are non-negative implies that the same is true of x. One requires that p > 1 if the system is to be productive and that x should satisfy the conditions above if one is to be able to start up the programme and satisfy the consumer. This maximal-growth path constitutes the 'turnpike' which one will largely follow in an optimal programme of expansion from some prescribed initial state to some desired terminal state. This model is a special case of the so-called von Neumann-Gale model (see e.g. Gale, 1960, 1967, 1968) which has itselfbeen much generalised since. The notion that an economy can grow indefinitely is, of course, fallaciousessential resource limitations will manifest themselves at some point. However, there may be a phase during which the idea is not unrealistic.

5 POLICY IMPROVEMENT Consider again the total-cost case of Section 2, which we shall consider only for the case of non-negative instantaneous cost c(x, u). Solution of equation (7) for the

57

5 POLICY IMPROVEMENT

value function V with a prescribed policy must be regarded as relatively straightforward, because this is linear in V. At the worst, it can be solved computationally if the number of states is finite. The dynamic programming equation (6), which determines the optimal policy, is another matter, however. It involves a combination of linear and extremal operations. One could approximate the determination of both Fand an optimal policy ever more closely by calculating the iterates Fs = .!l'Fs-1 for s = I, 2, .... This is the method of value iteration. It has the merit of reproducing the effect of an increasing horizon, but is generally very slow to converge. A much faster method is that of policy improvement. This is also iterative, proceedin~ in stages which we shall label by i. Suppose that at st~e i one has a policy g/')(). Denote the corresponding infinite-horizon cost V(g~00 ) by V;. (This is a bad notation, in that V; is liable to be confused with Vs; the cost at horizon s. We hope that strict adherence to the symbols i and s in the two contexts will prevent confusionJ We know Vi to be the minimal non-negative solution of Vi= L(g;) V1• Having determined V;, one then determines the g1+ 1 which yields an improved policy from L(g;+I) V; = .!l'V;. That is, g;+I (x) is the minimising value of u in .!l' V;(x), and so is the optimal value of control if one can optimise for one step before reverting to the policy g~""l.

f{

Theorem 3.5.1 The inequality .!l' V1 ::;;;; L(g;) V; holds. strict inequality holds for some x then the policy gj:'{ is a strict improvement on gioo . Proof The inequality first asserted holds in virtue of the definition of .!l'. If strict inequality holds (by which we mean that equality does not hold everywhere) then V; > .!l'V;

= L(gi+i)V; ~ L(gt+tYV; ~ L(gi+Ir(o)-+ Vi+!·

That is, V1 > V;+I and the iteration has produced a strict improvement.

0

One would like to assert that, if equality holds, so that V; = .!l' V;, then V; = F and the policy gi"") is optimal. This will be true under the assumptions of Exercise 1.1, for example, but is not in general true without qualification. One may say that value iteration carries out the operations of improvement of policy and passage to the infinite-horizon limit simultaneously. Policy improvement realises these two operations alternately, with passage to the limit telescoped to solution of the linear equation system V = LV. Typically, policy improvement indeed approaches optimality (when it does so) in very few iterations. The following observation partly explains why. Theorem 3.5.2 The policy improvement algorithm is identical with application of the Newton-Raphson algorithm to the equilibrium dynamic programming equation F=Ii'F.

58

A SKETCH OF INFINITE HORIZON BEHAVIOUR

Proof Recall what is meant by the Newton-Raphson (henceforth NR) algorithm. If we have an approximate solution F = V; ofF = 2 F then we improve it to a solution V;+t = V; +~;by setting (14) expanding the right-hand member as far as first-order terms in ~; and solving the consequent linear equation for ~;. Suppose that!£ V; = L(gi+l) V;. Because u is optimised out in the application of!£ the variation in u induced by the variation ~i of V; has no first-order effect on the value of 2( V; +~;),and

2(V; +

~;) = L(g;+t)(V;

+ ~;) + o(~;)

It follows that the linearised version of equation (14) is just V;+' = L(gi+,) vi+ I·

That is, V;+t is just the value function for the policy g~~1l, where g;+t was deduced D from the value function V; exactly by the policy improvement procedure. This is a useful observation, because the NR algorithm is such a natural one, as we shall find. The algorithm now has a direct variational justification in this context. The equivalence also has implications for rates of convergence: in regular cases, at least, value iteration and policy improvement show respectively first-and second-order convergence; see Exercise I. Policy improvement can be used also for the average-cost formulation. If we assume, for simplicity, that there is only a single equilibrium point under the various policies, then in discrete time it takes the form: determination of the average cost/; and transient cost function v;(x) from /i

+

v;

= L(g;)v;

followed by determination of the improved control rule u = g;+t (x) from L(gi+t)v; = 2v;.

By 'improvement' is meant that either the average cost has been decreased, or it is unchanged and the transient cost has been decreased. One generally hopes for improvement in the first and stronger sense; see Section 11.3. The continuoustime versions of both total- and average-cost procedures will be plain. The techniques of value iteration and policy improvement were formalised by Howard (1960). The equivalence of policy improvement and the NR algorithm was demonstrated in the LQ case by Whittle and Komarova (1988); in this case it holds in a tighter sense. However, we see from the last theorem that it holds generally. Puterman (1994) attributes the observation of the equivalence to Kalaba (1959) and Pollatschek and Avi-Itzhak (1969). However, it is only in recent years that the point and its application have bitten: see Chapter 18.

6 POLICY IMPROVEMENT AND LQ MODELS

59

Exercises and comments (1) Consider solution of the equation y = f(y) for a scalar y. Let y be the desired solution and {yi} a sequence of approximate solutions. Denote the error y;- y by E;. The sequence of approximations is said to show rth-order convergence if Ei+I = 0( ~) for small Ei and all sufficiently large i. Suppose one generates the sequence by iteration: Yi+I = f(y;). Then, if convergence toy indeed occurs, one fmds that Ei+I = f'E; + o(Ei), where f' is the first derivative off at y. If the sequence is generated by the NR algorithm one finds that E;+I

!"

=- 2(1- f')

2

Ef + o(t:; ).

Granted the convergence and differentiability assumptions, we see then that the two methods show first- and second-order convergence respectively. (2) In the case of positive programming the analogue of the least-excessive property of Theorem 3.2.2 does not hold, with the consequence that policy improvement may not work. Consider Exercise 2.4, where one has the option of continuation or termination in each state. If one chooses termination in all states then the corresponding value function (in reward terms) is V(x) = 1- 1/x. If one then performs policy improvement (i.e. chooses the action corresponding to the larger of V(x+ 1) or the retirement reward 1- 1/x) then one chooses continuation in all states. However, indefinite continuation is non-optimal. 6 POLICY IMPROVEMENT AND LQ MODELS

Policy improvement can be regarded as both an analytic and a computational technique. For LQ models it is a combination of both, in that it provides an excellent way of solving the Riccati equation iteratively. Consider the problem ofundiscounted LQ regulation to zero treated in Section 2.3. For notational simplicity we shall assume the matrix S normalised to zero. We know that the optimal policy has the form u1 = Kx 1 • Assume then that the policy at stage i has the form u1 = K;x 1, so that the corresponding infinitehorizon value function has the form Vi(x) = -!xTII;x. Then the two steps of the policy improvement algorithm reduce to: (i) Determine II; from the linear equation (15)

This will have a finite solution if the matrix A + BK; is a stability matrix; i.e. if the assumed control law is stabilising. (ii) Determine the matrix Ki+I of the improved control law as ( 16)

A SKETCH OF INFINITE HORIZON BEHAVIOUR

60

For a numerical example, consider the case

A= [

n,

~

B

=

[n,

R= [

~ ~],

Q

= 1,

S

= 0,

which can be regarded as a discrete-time version of the stabilisation to the origin of a point-mass moving on a line-Newtonian in that its acceleration is proportional to the control force exerted on it. If we suppose initially a control rule with K = -[0.25 1.00], then successive iterations modify this to -[0.329 1.224] and then -[0.328 1.220]. The smallness of the change on the second iteration indicates that the first iteration had already brought the control very close to optimality. The value of II corresponding to the last (and virtually optimal) rule is

n=

[3.715 3.044

3.044] 8.264 .

The continuous-time versions of (15) and (16) are

R + K( QK; + (A+ BK;) TIT;+ IT;(A + BK;) K;+t

=

= 0,

-Q- 1BTII;.

(17) (18)

For example, consider the pendulum regulation problem of Section 2.8 with

The ± option in A corresponds to consideration of the inverted or the hanging pendulum respectively. Of course, we solved this problem analytically in Section 2.8, but it is of interest to see how quickly the policy improvement algorithm will yield convergence to the known solution. For the hanging case we have the optimal solution II=[2../2

I

1]

../i '

K= -[1

J2]

where /2 = 1.414 to three decimal figures. The initial choice K = -[1 iterates to -[1 1.5] and -[1 1.416] on the first and second steps. For the inverted version the optimal solution is

II_ [4.898 -

3.000

3.000] 2.449 '

K = -[3.000

1]

2.449],

to three decimal figures. The initial choice K = -[2 1] iterates to -[3.5 4.0], -[3.05 2.76] and -[3.001 2.467] on successive steps. The numerical solution of the cart/pendulum problem was given in Section 2.8. The difficulty with this problem is to find a stable policy with which to start the

61

7. POLICY IMPROVEMENT FOR THE HARVESTING EXAMPLE

algorithm-one cannot do so until one realises that the coefficients of the displacement terms in the control rule must have the sign opposite to that which one would expect on naive grounds, as noted in Section 2.8. The problem of regulation to zero can be regarded as a total-cost problem, since a policy u = Kx which stabilises the equilibrium x = 0 achieves finite total cost from any finite starting point x. If we bring in disturbances or tracking as in Section 2.9 then cost does indeed build up indefinitely with time, although we have to go to a statistical formulation of the disturbances and the reference signal before the average cost has a simple definition; see Section 10.1. However, in all of these cases the central problem remains the solution of the Riccati equation for the matrix TI, which does not differ from that for the simple regulation problem above.

7. POLICY IMPROVEMENT FOR THE HARVESTING EXAMPLE Consider again the harvesting example of Section 1.2. The naive policy considered in that section turned out to be a very short-sighted one; the optimal policy (under the extreme simplifying assumptions of the model) was deduced in Section 2.7. Let us see how successful one stage of policy improvement is in taking us from the first policy towards the second. If V(x) is the infinite-horizon cost under a policy u = g(x) then V satisfies the equation

u- a V

+ (a(x)

- u) Vx = 0

(19)

with u indeed given the value g(x). Recall that xis the stock size, a(x) the natural net reproduction rate, u the catch rate and a the discount rate. For the naive policy u is given the constant value urn = a(xm) for a~l x > 0, where Xm is the value maximising a~). For the optimal policy one harvests at the maximal feasible rate M for x > X 0 and not at all for x ~ Xa, where X 0 is the root of a' (x) = a. It follows from (19), with u = g(x), that

Vx = 1 + a(x) -a V(x). g(x) - a(x)

(20)

It also follows that, for the improved policy, one will set u equal to zero or M according as Vx is greater than or less then unity. The choice in the case of equality is immaterial; we might as well set u equal to zero for definiteness. We see then from (20) that this implies that we should take u equal to zero or M according as a(x)- aV(x) and g(x)- a(x) have the same sign or not (with u zero in the transitional case). For the naive policy it follows from (20), and is otherwise obvious, that

(21)

62

A SKETCH OF INFINITE HORIZON BEHAVIOUR

where T(x) is the time taken for the stock to drop from x to zero under the policy. For x ~ Xm this time is infinite; for x < Xm

r

dy T(x) = Jo a(xm) - a(y) ·

(22)

For the naive policy it is also true that the expression g(x) - a(x) = a(xm)a(x) is nonnegative for all positive x. It thus follows that u will be zero in the improved policy for all x such that a(x) ~ a V(x). Appealing to evaluations (21) and (22) we find this to be the case if a

x 1

dy I a(xm)- a(O) ::::; og = o a(xm) - a(y) a(xm) - a(x)

1x

a'(y) d y, o a(xm) - a(y)

or x a- a'(y) -:---:---'---':-"'7" d y ::::; 0. o a(xm) - a(y)

1

(23)

The value of x separating the regimes of zero and maximal harvesting rate in the improved policy is that for which equality holds in (23). One sees that this root certainly lies between Xm and x 0 • Repetition of the argument shows that the recommended transition value converges rapidly to the optimal value x 0 with iteration of the improvement. This argument was of course unnecessary, as the optimal policy is known and simple. However, the example is perhaps reassuring, in that it shows how a single stage of policy improvement may change a conspicuously stupid policy to one qualitatively close to the optimal.

CHAPTER 4

The Classic Formulation of the Control Problem: Operators and Filters 1 THE CLASSIC CONTROL FORMULATION By the 'classic' control formulation we mean the formulation adopted when design was achieved by art (or craft) rather than by optimisation. The two approaches necessarily have a good deal of common ground, however, some of which we explore in this chapter. A prototype example is that of a mariner trying to make a ship follow a desired course; we may as well suppose it a sailing ship, and so emphasise the classical nature of the example. The block diagram of Figure 1 represents the situation. The ship is affected by the actions taken by the mariner, in changing sail or helm. It is also subject to other and unintended disturbances, such as weather and current. The mariner observes how the ship reacts to these combined forces and revises his actions accordingly, thus closing the control loop. His actions must be based upon what he can observe: e.g. apparent speed and heading of the ship, roll and pitch, the state of the sea and weather, the hazards of the topography and, of course, the specified intended course. Actual course

Wind and sea

-

Ship

r----

Helm and set of sail

Observations

f---'----

Mariner Desired course

Figure 1 The block diagram for a system in which a mariner controls the course of his ship by the operation of helm and sails, endeavouring to follow a set course against the discurbances offered by weather and sea.

64

THE CLASSIC FORMULATION OF THE CONTROL PROBLEM X

d Plant

,-

c7

r------

'

y

u

I '----

Controller

r--

X

w

Figure 2 An abstract version of the block diagram of Fig. 1, corresponding to equations ( 1) and(2).

In more general language the situation is that of Figure 2. The physically given system which is to be controlled is termed the plant. In our navigation example the plant is primarily the ship (although see the comments of Exercise 1), which is subject to the disturbances d of sea and weather as well as to the deliberate control actions u. The mariner becomes the controller, the unit which determines control actions u on the basis of current information. (This unit could then equally well be a person taking conscious decisions or a device following the rules built into it. The formal view is simply that a control policy formulates an action rule for all foreseeable situations. This view fails if situations arise which are not foreseeable or which are poorly quantifiable-outside our domain!) The current information will include all available observations y on the state of the plant and also the command signal w which specifies those aspects of the course which one wishes the plant to follow. The block diagram of Figure 2 is equivalent to a set of mathematical relations. The plant is regarded as a system whose inputs are the control sequence u and disturbances d and whose outputs are the actual performance x and the observation sequence y. However, the only output upon which decisions can be based are the observations. For control purposes the plant is thus described by an operator~. known and given, which converts input into output: y = ~(u,d).

(1)

The physical variables of the process (such as the process or state variable x) are buried in the dynamics of the system which determine the relation (1). This formulation is then often referred to as an input-output formulation, to be compared with the more explicit state-structured dynamic models of later chapters. Of course, relation (1) is not just an 'instantaneous' relationship. It is the consequence of a dynamic mechanism, and so will in general represent y at a given time as being dependent upon all past u and d.

65

I THE CLASSIC CONTROL FORMULATION

One chooses a control rule, necessarily in terms of observables, u = %(y, w)

(2)

where the operator% is to be chosen to secure good properties for the controlled system. A simplified version of this model which is often treated is u = %{w-

y).

(3)

That is, it is supposed that the disturbance is simply added to the control input, that the only observable plant output y is the very variable which is required to follow the command signal w and that the control signal is determined purely from the tracking error e = y- w. These assumptions are represented in the simplified block diagram of Figure 3. At least in the case when plant and control operators are linear, the system (3) has the formal solution y =(.I+ <;9%)- 1 <;9(%w +d),

(4)

for the plant output y in terms of system inputs. Here .I is the identity operator. Relation (4) can be regarded as the input/output relationship for the controlled system. It yields, in particular, the expression

(5) for the tracking errore = y - w. Now, what one certainly requires of the controlled system is that the tracking error should be small. Consider, for simplicity, the case d = 0, for which the tracking error is the solution of the equation

(.I+ <;9%)e = -w. d

+

(6) y

Plant

c9 y

u

Controller

X

+

w

Figure 3 The special case ofthe system ofFig. 2 in which the controller acts on the deviation of desired course wfrom actual course y, and in which the disturbanced is simply superimposed on the control u. This corresponds to the pair ofequJltions (3).

66

THE CLASSIC FORMULATION OF THE CONTROL PROBLEM

Then the minimal demand of standard control theory is that the dynamic system specified by (6) should behave well, in that its solution e should be small and even converge to zero as time passes for all w in some class of typical command signals. It is such considerations which generate the classical concepts of transient response, stability, dynamic lags etc. In developing these ideas we shall have to more specific about the nature of signals such as u, dandy and of operators such as t§ and %. We should also note that there are considerations other than the reduction of error e. This reduction should be achieved with reasonable economy of control effort Further, any control policy specified should be robust in that it works satisfactorily, not merely for a variety of inputs wand d, but also under some degree of mis-specification of the given system (i.e. of the plant operator t§). Exercises and comments (1) In a higher-level formulation one will have a mathematical model of the sea and the weather, which will in this case also be included in the 'plant: Although one has no hope of controlling sea or weather they form part of the physical picture, and can be predicted to some degree. In such a case the disturbance d would represent just those forces driving sea, weather etc. which are 'primitive', in that they are not explainable by a model. A primitive signal is a combination of two extremes, either fully known in advance (in which case no model is needed) or unknown and completely random (so that a model can achieve no further reduction). Equally, the command signal may be specified in advance (a specified course for the ship) or not (the intended course may have to be revised in the light of contingencies, or one may be following an unpredictable target vessel). In the latter case the plant may include also a model for the evolution of the command signal, and in either case the command input to the system will be reduced to its 'primitive' component.

(2) R~lation (4) follows by elimination of u from relations (3), and leads essentially to the demand that (J + t§%) should have a stable inverse. If we eliminate y instead then we obtain the equation (J + ft§)u = f(t§d- w) with the implication that the operator (J + ft§) should have a stable inverse, if the control u is not to build up with time. If the dimension of u is smaller than that of y then the second formulation has the advantage of working in a lower dimension. (Note that the dimension of J will then differ in the two cases.) The operator t§% is the loop operator in that it gives the effect on a signal of being passed successively through controller and plant. The operator .Y('t§ is equally a loop operator, but calculated by starting from the plant input port rather than the controller input port.

2 FILTERS IN DISCRETE TIME: THE SCALAR (SISO) CASE

67

2 FILTERS IN DISCRETE TIME: THE SCALAR (SISO) CASE The blocks of an engineer's block diagram (such as those of Figures 1-3) are units specified only by their inputs and outputs and the operation which converts one into the other. Let us for the moment discuss this matter generally, so that symbols such as x and~ refer only loosely to the control context. For our purposes a signal x is simply a function of time-the speed of a flywheel, the angle of an aileron, the price of fish, the pattern of power demand over the country, the picture on a television screen. In discrete time we shall write its value at timet as x 1 , so that {x1 } is a sequence. In continuous time we shall write rather x( t), if the time argument is indicated at all. The action of the unit represented by one of these blocks is then specified by a relationship X=~d

between input d and output x, say. The quantity~ is the operator which represents the effect of the block, transforming input signals into output signals. Such a unit or block is also often referred to as a 'filter', since in electronic contexts so many of the operations applied to a signal are regarded as filters in that they may, for example, pass frequencies only in some prescribed band. In those contexts filters are realised by electronic hardware. The response of a car to bumps and hollows in the road is equally well the response of a filter, realised mechanically by the suspension linked to the mass of the car. In the same way one can see filters which are realised by economic mechanisms (e.g. the response of imports to demand) or biological mechanisms (e.g. the response of heart- rate to exertion). So, for our purposes, 'filter' and 'operator' are respectively concrete and abstract expressions of the same thing. A signal could well take qualitative values (e.g. the colour of a traffic light or the identity of a telephone caller) but we shall suppose it quantitative. A filter will in general have several inputs and several outputs, but, by taking signals as vectorvalued, we can combine these into a single vector input and a single vector output, usually of different dimensions. Treatment of the vector case scarcely needs more formal machinery than does that of the scalar case (i.e. that of a single scalar input and a single scalar output) if one uses the right formalism. We shall consider the two cases separately, nevertheless, since this gives us the chance to approach the topic in two different ways. In considering the scalar case we shall concern ourselves simply with what conclusions hold; in considering the vector case we shall also ask why they hold. What we have termed the scalar and vector cases are often referred to as the SISO and MIMO cases in the engineering literature. The terms are abbreviations of 'single input/single output' and 'multiple input/multiple output' respectively. Although we start with the SISO case in this section, we shall in general take the MIMO case as the standard one. That is, we take it for granted that signals are vector-valued. We shall consider discrete-time filters first.

68

THE CLASSIC FORMULATION OF THE CONTROL PROBLEM

One important sub-class of filters is constituted by the linear filters, for which indeed the relation between input and output is linear:

(7) The coefficient gtr is then the response of the ftlter at time t to an input consisting of a unit pulse at time r. For this reason it is termed the transient response or the impulse response of the ftlter. Two significant properties of a ftlter are causality and stability. A filter is causal (or non-anticipatory) if its output at time t cannot depend upon its input after time t. In the linear case (7) this would imply that

(8) One expects that any physical mechanism (driven by an input d to produce an output x) will be causal. For example, consider the disturbed plant equation (2.61), and suppose it uncontrolled, in that u is set identically equal to zero. Then the solution x 1 of this equation is dependent (linearly, in fact) upon the disturbance input dT only for r:;:;; t. Non-causal fllters can occur, however, if physical mechanisms are supplemented by human intervention. Consider the same system, but with the optimal control (2.65) applied. This makes allowance for future disturbances, and so the solution x 1 of the controlled system will depend upon disturbances r4 for all r. That is, the corresponding filter is non-causal Note, however, that this can come about only because future d-values are assumed known. Stability, understood in a wide sense, is simply the requirement that the filter output should lie in some regular class of signals if the input does so-e.g., the class of uniformly bounded signals. There are as many definitions of stability as there are of regularity. To be more specific we should consider a more specific class of filters, the translation-invariant filters, which are stationary in their action. Consider the backward translation operator !Y, with with effect

(9) This then achieves a time delay of a single time unit. The operator !Y2 achieves a time delay of two units. More generally, the operator !Y' achieves a time delay of r units, where r is an integer of either sign. If r is negative then the effect is of a time advance of -r. A filter is said to be translation-invariant or time-homogeneous if a time-shift in its input merely produces the same time-shift in its output. This can be expressed as the commutation relation

(10) which in fact implies that

( 11)

69

2 FILTERS IN DISCRETE TIME: THE SCALAR (SISO) CASE

Theorem 4.2.1 Fora translation-invariant linear filter relation (7) takes the form

(12)

Proof Applying both sides of (10) to the input consisting of a unit pulse at time t'- 1, with f§ having the linear form (7), we deduce that grr' = gr-l,t'-l· Iterating this relation, we deduce that g11, = g1-r,o = gr-r, say. 0

The coefficient g1 is then interpretable as the transient response of the filter a time r after application of a unit pulse to the input, actual clock time being irrelevant. We shall abbreviate the description 'translation-invariant linear' to TIL. A stability condition for TIL filters is that of of lq-stability: that ( 13)

We see that It-stability ensures that uniformly bounded input yields uniformly bounded output, and so is sometimes referred to as BIBO stability. The demand of /2-stability will prove to be a natural one when stochastic inputs are considered. Stability has in fact been assumed in some degree when one writes down a relation such as (7) or (12). There is an implication behind this that the filter has been operating since the indefinite past (as there is no upper limit on r) and that the output is zero if the input is zero. In other words, if the filter represents a physical mechanism, then there is an assumption that any effects from the startup of this mechanism in the remote past have died away (see Exercise 3.1). If a pulse input reveals the transient response, then, for TIL filters, an exponential input x 1 = z- 1 for scalar z reveals another simple response pattern. One sees readily that the output is just G(z)z-r where G(z) is the generating function of the transient response:

G(z) =

Lg,z',

(14)

at least if the sum (14) is convergent. That is, the effect of the filter on such an input is simply to multiply it by a factor G(z). The function G(z) is termed the transfer function of the filter. One sometimes expresses the relationship (14) by saying that the transfer function G(z) is the z-transform of the transient response {g,}. Note that we could write f§ = G(!l) in the TIL case. Note also that, iff§! and f§2 are TIL filters, then f§tf§ 2 is a TIL filter with transfer function Gt (z)G2(z). Inputs of the form x 1 = e1w1 are of special interest, being both exponential and bounded. We can well consider complex-valued signals, and are indeed forced to do so if we require both boundedness and exponential behaviour. The transfer function for this input is G( e- 1w), the frequency response at frequency w. If we write this as p(w)e-iO(w) then p and e (assumed real) represent respectively the

70

THE CLASSIC FORMULATION OF THE CONTROL PROBLEM

multiplication of amplitude and the shift in phase which the filter induces on the signal eiw'. One may ask: to what extent are the transient response, transfer and frequency response functions mutually determining? This raises questions both of mathematical nicety (essentially, the whole theory of Fourier/Laplace transforms and their inversion) and of physical understanding. We shall escape as lightly as we can with the first and attempt to clarify the second. The transient response function certainly determines the other two in principle; the question then is: for what values of z is the series expression (14) for G(z) meaningful? Stability conditions of the type of (13) certainly ensure that G(z) is defined for z on the unit circle in some sense, and so that the frequency response function G( e-iw) is correspondingly defined. If q = 1 then the series defming G(z) is absolutely convergent on the unit circle. If q = 2 then G(z) is defined on the unit circle in an L2, sense. If the frequency response function is given, then by hypothesis it has a Fourier expansion

( 15) and the Fourier coefficients (the transient response) are then determined by the inverse relation g,

= -1

271'

/ " G( e"")e'wr . . dw.

( 16)

-7T

One can then determine G(z) and continue the original specification of G(z) off the unit circle. If we are given the transfer function G( z) then it does not necessarily follow that the transient response is determined by (15), (16), for reasons we shall see in a moment. In discussing these matters it is helpful to avoid the subtleties of Fourier theory by making some strong (but not unrealistic) convergence conditions. Suppose that the filter is stable in the very strong sense that the transient response g, converges exponentially fast to zero as r __. ±oo, in that it behaves as P'i for r large and positive and as P2 for r large and negative, where 0 ~ Pl < 1 < Pz ~ oo. Then G(z) converges for P2' < lzl < pj 1 and is analytic in this annulus. This argument can be reversed. Suppose that the response function G(z) is ~iven and is analytic in an annulus a2 ~ lzl ~ a1 • Then in this annulus G(z) IS represented by the series 2:::, g,z' (a Laurent series) where g, is determined by the contour integral g, =

2~i /

G(z)r_,_, dz

(17)

and the contour of integration is a circuit of the origin inside this annulus. This reduces to relation (16) in the case that the contour of integration is the unit

2 FILTERS IN DISCRETE TIME: THE SCALAR (SISO) CASE

71

circle. Furthermore, g, converges to zero at least as fast as o:!' for large positive r and as o:2' for large negative r. However, this g, is not necessarily the transient response of the physical model. There may be several such annuli of analyticity, and, if they are separated by a singularity of G, then the series representations will differ. We shall see just how this happens at the end of the next section. The appropriate representation will be determined on physical grounds: again, those of stability and causality.

Theorem 4.2.2 Suppose that the filter response function G(z) is analytic in an annulus 0:2 ~ lzl ~ O:J. Then (i) The filter output is defined for inputs ofthe form z-r for z in this annulus, andfor such inputs the filter behaves as one with the transient response g, determined in (17). (ii) If 0:2 < 1 < O:J then the filter thus determined is lq-stable for any positive q. (iii) If 0:2 = 0 then the filter thus determined is causal. These conclusions follow from the discussion before the theorem. Their implications go something like this. We try to determine a filter with prescribed transfer function G(z) by determining its transient response {g, }. If we look for a filter which is stable then we expand G(z) in its annulus of analyticity which includes the unit circle (if it has one). If we look for a filter which is causal then we take the contour of integral (17) so that all singularities of G(z) lie outside it. We can certainly find a filter which is both causal and stable if all its singularities lie in some region izl ;;:: 0:1 > 1. When we spoke of a filter with a given frequency response function there was an implied assumption that the filter was stable to sinusoidal inputs eiwr, so that (16) gave the appropriate inversion. The class of filters whose transfer function G(z) is rational is practically irpportant, for reasons which we shall see shortly. They are also mathematically straightforward: the transient response is a finite linear combination of exponentials, G( z) has singularities only at a finite number of poles, and one sees easily that the conditions of Theorem 4.2.2 are necessary as well as sufficient.

Theorem 4.2.3 Suppose that a causa/filter has a transfer function G(z) which is rational in z. Suppose furthermore than G(z) has been so normalised that numerator and denominator contain no common factor and no term in z raised to a negative power. Then the filter is stable (in the lq sense for any finite positive q) ifand only if the poles of G lie strictly outside the unit circle. The transient response is then determined by (16) ~r. equivalently, by (17) with the unit circle as integration contour). Proof The simplest proof is an elementary one, relying on the partial fraction expansion

72

THE CLASSIC FORMULATION OF THE CONTROL PROBLEM

G(z) =

l:c,z + :L:Ld1k(z1 -z)-k r

j

k

of G(z). Here all the sums have finite range,j and k are positive integers and the z1 are the finite non-zero poles. Causality requires us to seek an expansion (15) of G(z) in non-negative powers, and we certainly cannot find one unless c, = 0 for r < 0. That is, G must not have a pole at z = 0. Stability certainly requires that g, should tend to zero with increasing r. The term in (z1 - z)-k generates a contribution tog, which is of order rk- 1z1-r, so g, will certainly converge to zero (and exponentially fast) with increasing r if fz1I > 1 for all j. The condition is also necessary, because the functions rk- 1z1-r ofr are linearly independent (for varyingj and k), and g, could 0 then not possibly converge to zero if lz1I : : ; 1 for any j. 3 FROM DYNAMIC MODEL TO THE INPUT/OUTPUT RELATION: FILTER INVERSION The specification of a filter by its operator !§or (in the TIL case) by its transfer function G amounts to an input-output description in that it indeed specifies what the output should be for a given input. However, something has been lost in such a specification: the dynamic mechanism by which the filter was realised. A return to the physical model both clarifies a number of questions (notably, that of stability) and raises others. Consider an earlier example which we instanced in passing; that of a car driving over a bumpy road. The dynamic equations governing the response of the car are differential equations in the output variables x (the vertical displacement of the car body, etc.) driven by the input variables d (the irregularities in the road). These equations express input d in terms of output x, and so in effect specify the operation 1§- 1 of the inverse filter. It can be said that, in solving the equations, one is inverting this given filter to determine the action!§ of the direct filter. For example, suppose that the dynamics of the filter are expressed by the pthorder difference equation p

LArXt-r = dt.

( 18)

r=O

We can alternatively write this as

A(.r)x =d.

(19)

where A(ff) = I:~=<> A,ff'. If this is to be equivalent toy= G(ff)x then we must have A(.r)G(.r) = J, or, in terms of response functions, G(z)A(z) = 1. From this and Theorem 4.2.3 we thus deduce

3 FROM DYNAMIC MODEL TO THE INPUT/OUTPUT RELATION

73

Theorem 4.3.1 The transfer function ofthefilter d---> x implied by relation (18) is G(z) = A(z)- 1 . The cau.salformofthefuter is stable ifand only ifall zeros ofA(z) lie strictly outside the unit circle. Inversion of the operator thus amounts, in the TIL case, to inversion of the transfer function in the most literal sense: G(z) is simply the reciprocal of the transfer function A(z) of the inverse filter. However, to this observation must be added some guidance as to how the series expansion of A(z)- 1 is to be understood. We do indeed demand causality if equation (18) is to be understood physically: as a forward recursion in time whose solution cannot be affected by future input. We shall see in the next section that the theorem continues to hold in the vector case. In this case A(z) is a matrix, with elements polynomial in z, which implies that A(zf 1 is rational. That is, the transfer function from any component of the input to any component of the output is rational. This can be regarded as the way the rational transfer functions make their appearance in practice: as a consequence of finite-order finite-dimensional (but multivariable) linear dynamics. The simplest example is that which we have already quoted in the last section; the uncontrolled version x, = Ax,_ I

+ d,

(20)

of relation (2.61). We understood this as a vector relation in Section 2.9, and shall soon do so again, but let us regard it as a scalar relation for the moment. We have then A(z) = 1 - Az, with its single zero at z = A- 1• The necessary and sufficient condition for stability is then that lA I < 1. The solution of the equation for prescribed initial conditions at t = -r is t-r-1 Xt

=

L A'd,_, +AI-r

Xr.

(t

~

-r)

(21)

r=O

whatever A. From this we see that the stability condition IAI < 1 both assures stability in what one would regard as the usual dynamic sense (that the effect of initial conditions vanishes in the limit -r ___, -oo) and in the filter sense (e.g. that the filter has the BIBO property). If lA I ~ 1 then solution (21) is still valid, but will in general diverge as t - -r increases. We can return to the point now that the transfer function has, in the present case G(z) = ( 1 - Az) -I, two distinct series expansions. These are -1

00

G(z)

= _EA'z',

G(z) =- l:A'z', -oc

0

1

1

valid respectively for !zl < IAI- and Jz! > IAJ- . The first of these corresponds to a filter which is causal, but stable only if !AI < 1. The second corresponds to a

74

THE CLASSIC FORMULATION OF THE CONTROL PROBLEM

filter which is stable if [A[ > 1, but is non-causal. Indeed, it corresponds to a solution 00

Xt =

L( -At' dt+r r=l

of (20). This is mathematically acceptable if [AI < 1 (and d uniformly bounded, say), but is of course physically unacceptable, being non-causal.

Exercises and comments (1) Consider the filter d - x implied by the difference equation (18). The leading coefficient Ao must be non-zero if the filter is to be causal. Consider the partial power expansion. t-1

p-1

A(z)- 1 = Lg,z' +A(z)- 1 Lctd+k, k=O

ro=O

in which the two sums are respectively the dividend and the remainder after t steps of division of A(z) into unity by the long-division algorithm. Show, by establishing recursions for these quantities, that the solution of system (18) for general initial conditions at t = 0 is just p-1

1-1

Xt = Lg,d,_, + LCtkXt-k· k=O

ro=O

Relation (21) illustrates this in the case p = 1. (2) Suppose that A(z) = II~=t (1- a1z), so that we require jo1[ < 1 for allj for stability. Determine the coefficients c1 in the partial fraction expansion p

A(z)- 1

=

L c1(1 -

a1z)- 1

}""'I

in tbe case when the o1 are distinct. Hence determine the coefficients g, and c1k of the partial inversion of Exercise 1. (3) Model (20) has the frequency response function (1 - Ae-iw)- 1. An input signal d, = eiw1 will indeed be multiplied by this factor in the output x if the filter is stable and sufficient time has elapsed since start-up. This will be true for an unstable filter only if special initial conditions hold (indeed, that x showed this pattern already at start-up).What is the amplitude of the response function?

4 FILTERS IN DISCRETE TIME; THE VECTOR (MIMO) CASE Suppose now that input d is an m-vector and output x an n-vector. Then the representations (7) and (12) still hold for linear and TIL filters respectively, the

4 FILTERS IN DISCRETE TIME; THE VECTOR (MIMO) CASE

75

first by definition and the second again by Theorem 4.2.1. The coefficients g 1r and g, are still interpretable as transient response, but are now n x m matrices, since they must give the response of all n components of output to all m components of input. However, for the continuation of the treatment of the last section, let us take an approach which is both more oblique and more revealing. Note, first, that we are using !!/ in different senses in the two places where it occurs in (11). On the right, it is applied to the input, and so converts m-vectors into m-vectors. On the left, it is applied to output, and does the same for nvectors. For this reason, we should not regard !!/ as a particular case of a filter operator r§; we should regard it rather as an operator of a special character, which can be applied to a signal of any dimension, and which in fact operates only on the time argument of that signal and not on its amplitude. We now restrict ourselves wholly to the TIL case. Since r§ is a linear operator, one might look for its eigenvectors; i.e. signals ~~ with the property r§~ 1 = .\~ 1 for some scalar constant .\. However, since the output dimension is in general not equal to the input dimension we look rather for a scalar signal 0'1 such that r§~u 1 = ryu 1 for some fixed vector 1J for any fixed vector~- The translation-invariance condition (ll) implies that, if the input-output pair {~0' 1 , 1]0'1 } has this property, then so does {~O'r-J,1J0' 1 _I}. If these sequences are unique to within a multiplicative constant for a prescribed ~then one set of signals must be a scalar multiple of the other, so that ~0' 1 _ 1 = z~rJ 1 for the some scalar z. This implies that a1 ex: z- 1, which reveals the particular role of the exponential sequences. Further, 1J must be then be linearly dependent upon ~. although in general by a z-dependent rule, so that we can set 1J = G(z)~ for some matrix G(z). But it is then legitimate to write r§

= G(3"),

(22)

since r§ has this action for any input of the form ~z- 1 or a linear combination of such expressions for varying~ and z. If G(z) has a power series expansion 'L,g,z' then relation (22) implies an expression (12) for the output of the filter, with g, identified as the matrix-valued transient response. We say 'a' rather than 'the' because, just as in the SISO case, there may be several such expansions, and the appropriate one must be resolved by considerations of causality and stability. These concepts are defined as before, with the lq-condition (13) modified to

wheregrJk is thejkth element of g,. The transient responseg, is obtained from G(z) exactly as in (17); by determination of the coefficient of z' in the appropriate expansion of G(z). If causality is demanded then the only acceptable expansion is that in non-negative powers of z.

76

THE CLASSIC FORMULATION OF THE CONTROL PROBLEM

If G(z) is rational (meaning that the elements of G(z) are all rational in z) then the necessary and sufficient condition that the filter be both causal and stable is that of Theorem 4.2.3, applied to each element of G. Again, one returns to basics and to physical reality if one sees the filter as being generated by a model. Suppose that the filter is generated by the dynamic equation (18), with x and d now understood as vectors. If this relation is to be invertible we shall in general require that input and output be of the same dimension, so that A(Y) is a square matrix whose elements are polynomial of degree p in §". The analysis of Section 3 generalises immediately; we can summarise conclusions.

Theorem 4.4.1 The filter C§ determined by (18) has transfer function G(z) = A(z)- 1. The causa/form ofthe filter is stable ifand only if the zeros ofiA(z) I lie strictly outside the unit circle. Here IA(z)l is the determinant of A(z), regarded as a function of z. The first conclusion follows from A(z)G(z) = I, established as before. The elements of 1 A(zr are rational in z, with poles at the zeros of the determinant IA(z)j. (More exactly, these are the only possible poles, and all of them occur as the pole of some element.) The second conclusion then follows from Theorem 4.2.3. The fact that z = 0 must not be a zero of jA(z)l implies that Ao is non-singular. This is of course necessary if (18) is to be seen as a forward recursion, determining x 1 in terms of current input and past x-values. The actual filter output may be only part of the output generated by the dynamic equation (18). Suppose we again take the car driving over the bumpy road as our example, and take the actual filter output as being what the driver observes. He will observe only the grosser motions of the car body, and will in fact observe only some lower-dimensional function of the process variable x. Exercises and comments

(1) Return to the control context and consider the equation pair X1

= Axt-i + Bu 1-J,

Yt

=

CXt-i·

(23)

as a model of plant and observer. In an input-output description of the plant one often regards the control u as the input and the observation y as the output, the state variable x simply being a hidden variable. Show that the transfer function u ---'* y is C(I- Az) -I Bz2 • What is the condition for stability of this causal filter? If a disturbance d were added to the plant equation of (23), then this would constitute a second input. If a control policy has been determined then one has a higher-order formulation; dis now the only input to the controlled system. (2) The general version of this last model would be

77

S COMPOSITION AND INVERSION OF FILTERS; Z-TRANSFORMS

dx + f:Mu

= 0,

y

+ CCx = 0,

where d, !16 and ct are causal TIL operators. If d 1 function u----+ y is C(z)A(z)- B(z).

= A (ff) etc.

then the transfer

5 COMPOSITION AND INVERSION OF FILTERS; z-TRANSFORMS We assume for the remainder of the chapter that all filters are linear, translationinvariant and causal. Let us denote the class of such filters by CC. If filters ~~ and ~2 are applied in succession then the compound filter thus generated also lies in CC and has action (24) That is, its transient response at lag r is L:vK2vKl,r-v. in an obvious terminology. However, relation (24) expresses the same fact much more neatly. The formalism we have developed for TIL filters shows that we we can manipulate the filter operators just as though the operator .r were an ordinary scalar, with some guidance from physical considerations as to how power series expansions are to be taken. This formalism is just the Heaviside operator calculus, and is completely justified as a way of expressing identities between coefficients such as the A, of the vector version of (18) and the consequent transient response g,. However, there is a parallel and useful formalism in terms of z-transforms (which become Laplace transforms in continuous time). This should not be seen as justifying the operator formalism (such justification not being needed) but as supplying useful analytic characterisatioQ.s and evaluations. Suppose that the vector system p

(25)

LArXc-r =de. r=O

does indeed start up at time zero, in that both x and dare zero on the negative time axis. Define the z-transforms IX)

IX)

x(z) =

l:xcz',

d(z) = A

t=O

L.,; d1zI

(26)

" '

t=O

for scalar complex z. Then it is readily verified that relation (25) amounts to

A(z)x(z) = d(z) with an inversion

x(z) = A(z)- 1d(z).

(27) 1

This latter expression amounts to the known conclusion G(z) = A(z)- if we understand that A(z)- 1 and d(z) are to be expanded in non-negative powers of z.

78

THE CLASSIC FORMULATION OF THE CONTROL PROBLEM

The inversion is completed by the assertion that x 1 is the coefficient of Z1-in the expansion of x(z). Analytically, this is expressed as Xr

= 2 ~i

J

x(z)z-t-l dz

(28)

where the contour of integration is a circle around the origin in the complex plane small enough that all singularities of x(z) lie outside it. Use of transforms supplies an alternative language in which the application of an operator, as in A(ff)x, is replaced by the application of a simple matrix multiplier: A(z)x. This can be useful, in that important properties of the operator A(ff) can be expressed in terms of the algebraic properties of A(z), and calculable integral expressions can be obtained for the transient response g, of (17). However, it is also true that both the operator formalism and the concept of a transfer function continue to be valid in cases where the signal transforms .X and d do not exist, as we shall see in Chapter 13. Exercises and comments

(1) There is a version of (27) for arbitrary initial conditions. If one multiplies relation (25) by z1 and then sums over t ~ 0 one obtains the relation

x(z) = A(z)-l

[d(z)-

tL

AjX-kZj-kl·

(29)

k=l; ;;>k

This is a transform version of the solution for x 1 of Exercise 3.1, implying that solution for all t ~ 0. Note that we could write it more compactly as

(30) where the operator [ ]+ retains only the terms in non-negative powers of z from the power series in the bracket. It is plain from this solution that stability of the filter implies also that the effect of initial conditions dies away exponentially fast with increasing time. (2) Consider the relation x 1 = god1 + g 1dr-1, for which the transient response is zero at lags greater than unity. There is interest in seeing whether this could be generated by a recursion of finite order, where we would for example regard relation (25) as a recursion of order p. If we define the augmented process variable .X as that with vector components x 1 and d1 then we see that it obeys a recursion.X 1 = Ax1_ 1 + Bd1, where

A=

[00

KI] 0 ,

6 FILTERS IN CONTINUOUS TIME

79

The fact that this system could not be unstable under any circumstances is reflected in the fact that II - Azl has no zeros, and so G(z) has no poles.

6 FILTERS IN CONTINUOUS TIME

In continous time the translation operator f!T' is defined for any real r; let us replacer by the continous variable T. However, the role of the unit translation f!T must be taken over by that of an infinitesimal translation. More specifically, one must consider the rate of change with translation g) = lim T -l [1 - f!T'"], which is just the differential operator g) = d/dt. The relation TlO

.( ) . x(t-T+Ot)-x(t-T) 1liD =X t-T 6t10 c5t amounts to the operator relation d

--f71"=~r.

dT

Since f/

0

= 1 this has formal solution arr

.:7

=e -T!?J ,

(31)

which exhibits !!) as the infinitesmal generator of the translations. Relation (31) can be regarded as an expression of Thylor's theorem

e-1"gjx(t) =

't (-~~'/ }=0

x(t) = x(t- T) = f71"x(t).

(32)

1·

Note, though, that translation x(t- T) of x(t) makes sense even if x is not differentiable to any given order, let alone indefinitely. As in the discrete-time case, a translation-invariant linear filter ~ will modify an exponential signal x(t) =~est merely by a multiplicative matrix factor, G(s), say, so that (33)

G(s) is then the transfer function of the continuous-time filter. We can legitimately write ~=

G(Ei')

(34)

in analogy to (22), since, by (33), r§ has this action on any linear combination of exponential signals. However, use of the formalism (34) does not imply that the differentials £}r x( t) need exist for any r for a function x( t) to which~ is to be applied. We have already seen in (32) that the translation e_.,.g x( t) is well-defined, even if x( t) is not differentiable at all.

80

THE CLASSIC FORMULATION OF THE CONTROL PROBLEM

In fact, identification (31) demonstrates that, if G(s) has a Fourier-Laplace representation,

1

00

G(s)

=

e-srg(r) dr,

(35)

then the filter relationship x = '§d can be written

whence we see that we can identify g( r) with the transient response of the filter. However, we must be prepared to stretch our ideas. For example, one could envisage a filter x = ~' d which formed the rth time differential of the current input. One could represent this in the form (36) only by setting g(r) = (-~)' 6( r), the rth differential of a delta-function. We have taken the integral only over non-negative Tin (35) on the supposition that the filter is causal. If the integral (35) is absolutely convergent for some real positive value a of s then it will define G(s) as an analytic function for all s such that Re(s) ? a.

7 DYNAMIC MODELS: THE INVERSION OF CONTINUOUS-TIME FILTERS As we have already emphasised in Sections 3 and 4, one must look beyond the input-output specification of a filter to the dynamic mechanism behind it. Suppose that these equations take the form .fllx =d.

(37)

The simplest finite-order TIL assumption is that .fll is a differential operator of order p, say: p

s(

=A(~)= LA,~'.

(38)

r=O

(For economy of notation we have denoted the matrix coefficients by A, as for the discrete-time version (18), but the two sets of coefficients are completely distinct.) The system (37), (38) then constitutes a set of differential equations of degree p at most. This is to be regarded as a forward equation in time determining the forward course ofthe output y. In discrete time this led to the requirement that Ao should be non-singular. The corresponding requirement now is that the matrix coefficient of the highest-order differentials should be non-singular. That is, if the kth individual output Xk occurs differentiated to order rk at most in system (37), (38) then the matrix A. whose jkth element is the jkth element of A,k (for all relevant}, k) must be non-singular.

7 THE INVERSION OF CONTINUOUS-TIME FILTERS

81

Just as for the discrete-time case of Sections 3 and 4 the actual filter d ---> x, 1 obtained by inversion of the relation (37), has A(s)- as transfer function. We must also suppose the filter causal, if relation (37) is su!.fosed on physical grounds to be a forward relation in time. Thus, if A(sr has the Laplace representation (39) then g( T) is the transient response of the filter, and the solution of equation (37) is

1

00

x(t)

=

g(T)d(t- T) dT

(40)

plus a possible contribution from initial conditions. There will be no such contribution if the system starts from a quiescent and undisturbed past or if the filter is stable and has operated indefinitely. The Laplace transform is the continuous-time analogue of the z-transform of Section 5. Suppose the system (37) quiescent before time zero, in that both x(t) and d(t) are zero for t < 0. If we multiply relation (37) at time t by e-st and integrate over t from 0 to infinity then we obtain

A(s)x(s) = d(s)

(41)

where x(s) is the Laplace transform of x( t ):

x(s) =

roo e-

lo-

31

x(t) dt.

(42)

The reason for emphasising inclusion ofthe value t = 0 in the range of integration will transpire in the next section. However, relation (41) certainly implies that .Y(s) = A(s)- 1x(s) forallsfor which bothx(s) and A(s)- 1 are defined, and this is indeed equivalent to the solution implied by the evaluation of the transient response implicit in (39). In a sense there is no need for the introduction of Laplace transforms, in that the solution determined by (39), (40) remains valid in cases when d(s) does not exist. However, the Laplace formalism provides the natural technique for the inversion constituted by relation (39); i.e. for the actual determination of g( T) from G(s) = A(s)- 1• Exercises and comments (I) Show, by partial integration, that

82

THE CLASSIC FORMULATION OF THE CONTROL PROBLEM

(Here ~r x( 0) is a somewhat loose expression for the rth differential of x at time 0.) For general initial conditions at t = 0 relation (41) must thus be replaced by

r-l

A(s).Y(s) = x(s)

+ LAr Ls'l~r-q-ly(O). q=O

r

This is the continuous-time analogue of relation (29). (2) A stock example is that of radioactive decay. Suppose that a radioactive substance can decay through consecutive elemental forms j = 0, 1, 2, ... , and that xj(t) is the amount of element} at timet. Under standard assumptions the Xj will obey the equations

(J= 1,2, ... ). where J.lj is the decay rate in state}. Here we have supposed for simplicity that only element 0 is replenished externally, at rate d(t). In terms of Laplace transforms these relations become

(s + J.lo)xo =

d,

(s+J.lj)xj=J.lj-lXj-1

(}=1,2, ... ),

if we assume that xj(O) = 0 for all}. We thus find that A

.X1 -

/-lO/-ll'''J.l'-1 1

d- - -d- ' A

(s+J.lo)(s+J.lt)···(s+J.lj)

Pj(s)

say. If J.lQ, I-ll, . .. , /-lj are distinct and positive then this corresponds to a transient response function (ford ~ xj) : j

Gj(T) =

e-JIF

L f)/( ) . k=O .rj /-lk

(43)

Suppose thatj =pis the terminal state, in that 1-lp = 0. The term corresponding to k = p in expression (43) for j = p is then simply 1. This corresponds to a singularity at s = 0 in the transfer function from d to Xp. The singularity corresponds to a rather innocent instability in the response of Xp to d: simply that all matter entering the system ultimately accumulates in state p.

8 LAPLACE TRANSFORMS The Laplace transform (42) is often written as !f'x to emphasise that the function x(s) has been obtained from the function x(t) by the transformation!£'. (This is quite distinct from the forward operator defined in Section 3.1-we use !f' to denote Laplace transformation only in this section.) The transformation !f' is linear, as is its inverse, which is written !f'- 1. One of the key results is that the inversion has the explicit form

83

9 STABILITY OF CONTINUOUS-TIME FILTERS

x(t)

= _p-l_x = - 1 . 111+ioo e31 x(s) 21fl

ds

(44)

u-ioo

x

where the real number u is taken large enough that all singularities of lie to the left of the axis of integration. (This choice of integration path yields x( t) = 0 for t < 0, which is what is effectively supposed in the formation (42) of the transform. It is analogous to the choice of integration path in (17) to exclude all singularities of G(z), if one wishes to determine the causal form of the filter.) Inversion of a transform then often becomes an exercise in evaluation of residues at the various singularities ofthe integrand. The glossary of simple transform-pairs in Table 4.1 covers all cases in which .X(s) is rational and proper (a term defined in the next section). The reader may wish to verify validity of both direct and inverse transforms. In all cases the assumption is that x( t) is zero for negative t. Table4.1 X

I t"/n! e-<>1

t"e-"'ljn! 6(r)

s-1 s-n-1

(s + a)- 1 (s + ct)-n-1 I

If s corresponds to the operation of differentiation then s- 1 presumably corresponds to the operation of integration. This is indeed true (as we saw in Exercise 7.2), but the operation of integration is unique only to within an additive constant, to be determined from initial conditions. That initial conditions should have a persistent effect is an indication of instability. A very useful results is the final value theorem: that if limrroo x(t) exists, then so does lims!O sx(s), and the two are equal. This is easily proved, but one should note that the converse holds only under regularity conditions. Note an implication: that if limrroc ~x(t) exists for a given positive integer j, then so does limslO si+l x(s), and the tWO are equal.

9 STABILITY OF CONTINUOUS-TIME FILTERS Let us consider the SISO case to begin with, which again sets the general pattern. Lq-stability for a realisable filter requires that fooc !g(rW dr

< oo.

L1-stability is then again equivalent to BIBO stability, and implies that G(s) is analytic for Re(s) ~ 0. This condition certainly excludes the possibility that g could have a differentiated delta-function as component, i.e. that the filter would

84

THE CLASSIC FORMULATION OF THE CONTROL PROBLEM

actually differentiate the input at any lag. A bounded function need not have a differential at all, let alone a bounded one. A filter for which IG(s) I remains bounded as lsi --+ oo is said to be proper. This excludes s' behaviour of G( s) for any r > 0 and so excludes a differentiating action for the filter. If IG(s)l --+ 0 as lsi ---+ oo then the filter is said to be strictly proper. In this case even delta-functions are excluded in g; i.e. response must be smoothly distributed. A rational transfer function G(s) is now one which is rational ins. As in the discrete-time case, this can be seen as the consequence of finite-order, finitedimensional linear dynamics. The following theorem is the analogue of Theorem 4.2.3.

Theorem 4.9.1 Suppose that a causa/filter has a response function G(s) which is rational and proper. Then the filter is stable ifand only ifall polesofG(s) have strictly negative real part {i.e. lie strictly in the left halfofthe complex plane). Proof This is again analogous to the proof of Theorem 4.2.3. G(s) will have the expansion in partial fractions G(s) =

L c,s' + L L djk(s- Sjrk-l r

j

k

where the ranges of summation are finite, j and k are non-negative integers and the s1 are the non-zero poles of G(s). Negative powers s' cannot occur, since these would imply a component in the output consisting of integrated input, and the integral of a bounded function will not be bounded in general. Neither can positive powers occur, because of the condition that the filter be proper. The first sum thus reduces to a constant c0. This corresponds to an instantaneous term c0 d(t) in filter output t'§d, which is plainly stable. The term in (s- s1 )-k-l gives a term proportional to-r" exp(s1r) in the filter response; this component is then stable if and only if Re(s1) < 0. The .condition of the theorem is thus sufficient, and necessity follows, as previously, 0 from the linear independence of the components of filter response. The decay example of Exercise 7.1 illustrates these points. The transfer function Gj(s) for the output x1 had poles at the values -Jlk (k = 0, 1, ... ,j). These are strictly negative fork < p, and so G1 determined a stable filter for j < p. The final filter had a response singularity at s = -p,P = 0. This gave rise to an instability corresponding, as we saw, to the accumulation of matter in the final state.

Exercises and comments (I) The second stock example is that of the hanging pendulum-a damped harmonic oscillator. Suppose that the bob of the pendulum has unit mass and

10 SYSTEMS STABILISED BY FEEDBACK

85

that it is driven by a forcing term d. The linearised equation of motion (see Section 5.1) for the angle of displacement x of the pendulum is then A(£»)x = d where A(s) = s'- + a,s + ao. Here (non-negative) represents damping and ao (positive) represents the restoring force due to gravity. If a 1 = 0 then the zeros of A(l') have the purely imaginary values ±iylaO (corresponding to an undamped oscillation of the free pendulum, with an amplitude determined by initial conditions). If 0 < < 2yla0 then the zeros are complex with negative real part (corresponding to a damped oscillation of the free pendulum~ If a 1 ~ 2y"QQ, then they are negative real (corresponding to a damped non-oscillatory motion ofthe free pendulum). The equivalent filter is thus stable or unstable according as the pendulum is damped or not. A damped harmonic oscillator would also provide the simplest useful model of our car driving over a bumpy road, the output variable x being the vertical displacement of the car body. If the suspension is damped lightly enough that the car shows an oscillatory response near the natural frequency of vibration wo = ylaO then the response function A(sr' will be large in modulus at s = ±iwo. This can be observed when one drives along an unsealed road which has developed regular transverse ridges (as can happen on a dry creek bed). There is a critical speed which must be avoided if the car is not to develop violent oscillations. The effect is enhanced by the fact that the ridges develop in response to such oscillations!

a,

a,

10 SYSTEMS STABILISED BY FEEDBACK We shall from now on often specify a filter simply by its transfer function, so that we write G rather than r§. In continuous time the understanding is then that G is understood as denoting G(PJ) or G(s) according as one is considering action of the filter in the time domain or the transform domain. Correspondingly, in discrete time it denotes G(Y) or G(z), as appropriate. We are now in a position to resume the discussion of Section 1. There the physically given system (the plant) was augmented to the controlled system by addition of a feedback loop incorporating a controller. The total system thus consists of two filters in a loop, corresponding to plant and controller, and one seeks to choose the controller so as to achieve satisfactory performance of the whole system. Optimisation considerations, a consciousness that 'plant' constitutes a model for more than just the process under control (see Exercise 1.1) and a later concern for robustness to misspecification (see Chapter 17) lead one to modify the block diagram of Figure 2 somewhat, to that of Figure 4. In this diagram u and y denote, as ever, the signals constituted by control and observations respectively. The signal ( combines all primitive exogenous inputs to the system: e.g. plant noise, observation noise and command signals (or the noise that drives command signals if these are generated by a statistical model).

86

THE CLASSIC FORMULATION OF THE CONTROL PROBLEM

l;

~

G

y

u

K

Figure 4 The block diagram corresponding to the canonical set ofsystem equations (45). The plant G. understood in a wide sense, has as outputs the actual observations yand the vector of deviations .6.. It has as inputs the control u and the vector (of 'primitive' inputs to the system.

The signal D.. comprises all the 'deviations' which are penalised in the cost function. These would for example include tracking error and those aspects of the control u itself which incur cost. Here the plant G is understood as including all given aspects of the system. These certainly include plant in the narrow sense-the process being controlled-but also the sensor system which provides the observations. They may also include subsidiary models used to predict, for example, sea and weather for the long-distance yachtsman of Section 1, or the future inflow to the reservoir of Section 2.9, or the command signal constituted by the position of a vehicle one is attempting to follow. The optimiser may be unable to exert any control upon these aspects, but he must regard them as part of the total given physical model. As well as the control input u to this generalised plant one has the exogenous input(. This comprises all quantities which are primitive inputs to the system; i.e. exogenous to it and not explained by it. These include statistical noise variables (white noise-which no model can reduce) and also command sequences and the like which are known in advance (and so for which no model is needed). It may be thought that some of these inputs should enter the system at another point; e.g. that observation noise should enter just before the controller, and that a known command sequence should be a direct input to the controller. However, the simple formalism of Figure 4 covers all these cases. The input (is in general a vector input whose components feed into the plant at different ports. A command or noise signal destined for the controller can be routed through the plant, and either included in or superimposed upon the information streamy. As far as plant outputs are concerned, the deviation signal D.. will not be completely observable in general, but must be defined if one is to evaluate (and optimise) system performance.

10 SYSTEMS STABILISED BY FEEDBACK

87

If we assume time-invariant linear structure then the block diagram of Figure 4 is equivalent to a set of relations D.= G11( + G12u y = G12( + Gzzu (45) u=Ky. We can write this as an equation system determining the system variables in terms of the system inputs; the endogenous variables in terms of the exogenous variables:

(46) By inverting this set of equations one determines the system transfer function, which specifies the transfer functions from all components of the system input ( to the three system outputs: D., y and u. The classic demand is that the response of tracking error to command signal should be stable, but this may not be enough. One will in general require that all signals occurring in the system should be finite throughout their course. Denote the first matrix of operators in (46) by M, so that it isM which must be inverted. The simplest demand would be that the solution of (46) should be determinate; i.e. that M(s) should not be singular identically in .r. A stronger demand would be that the system transfer function thus determined should be proper, so that the controlled system does not effectively differentiate its inputs. A yet stronger demand is that of internal stability; that the system transfer function should be stable. Suppose all the coefficient matrices in (46) are rational in s. Then the case which is most clear-cut is that in which the poles of all the individual transfer functions are exactly atthe zeros of IM(s) I, i.e. atthe zeros of II- Gzz(s)K(s)l. In such a case stability of any particular response (e.g. of error to command signal) would imply internal stability, and the necessary and sufficient condition for stability would be that II- Gz2(s)K(s) Ishould have all its zeros strictly in the left half-plane. In fact, it is only in quite special cases that this pattern fails. These cases are important, however, because performance deteriorates as one approaches them. To illustrate the kind of thing that can happen, let us revert to the model (3) represented in Figure 3, which is indeed a special case of that which we are considering. The plant output y is required to follow the command signal w; both of these are observable and the controller works on their difference e. The only noise is process noised, superimposed upon the control input. Let us suppose for simplicity that all signals are scalar. Solution (5) then becomes

e = (l

+ GKf 1(Gd-

w)

(47)

88

THE CLASSIC FORMULATION OF THE CONTROL PROBLEM

in the response function notation. Suppose that G(s) and and K(s) are rational. Then the transfer function -(1 + GK)- 1 of e tow is rational and proper and its only poles are precisely at the zeros of I + G(s)K(s). It is then stable if and only if these zeros lie strictly in the left half-plane. The same will be true of the response function (I + GK) -I G of e to d if all unstable poles of G are also poles of GK. However, suppose that the plant response G has an unstable pole which is cancelled by a zero of the controller response K. Then this pole will persist in the transfer function (1 + GK)- 1 G, which is consequently unstable. To take the simplest numerical example, suppose that G = (s- 1) -I, K = 1 - s- 1. Then the transfer functions

-(1

+ GK)- 1 =

__s_

s + 1'

(I

-1

s

+ GK) G = (s + I\)(s- I)

are respectively stable and unstable. One can say in such a case that the controller is such that its output does not excite this unstable plant mode, so that the mode seems innocuous. The mode is there and ready to be excited, however, and the noise does just that. Moreover, the fact that the controller cannot excite the mode means that it is also unable to dampen it. These points are largely taken care of automatically when the controller is chosen optimally. If certain signal amplitudes are penalised in the cost function, then those signals will be stabilised to a low value in the optimal design, if they can be. If inputs are such that they will excite an instability of the total system then such instabilities will be designed out, if they can be. If inputs are such that their differentials do not exist then the optimal system will be proper, if it can be. One may say that optimality enforces a degree of robustness in that, as far as Physical constraints permit, it protects against any irregularity permitted in system input which is penalised in system output. Optimisation, like computer programming, is a very literal procedure. It supplies all the protection it can against contingencies which are envisaged, none at a]] against others. Exercises and comments

(1) Application of the converse to the final value theorem (Section 8) can yield Useful information about dynamic lags-the limiting values for large time of the tracking errore or its derivatives. Consider a scalar version of the simple system of Figure 3. If the loop transfer function has the form ks-N ( 1 + o(s)) for smalls, then k is said to be the effective gain of the loop and N its type number: the effective number of integrations achieved in passage around the loop. Consider a :ommand signal w which is equal to t" j n! for positive t and zero otherwise. Then J.V ::::: s-n-J, and it follows from (47) and an application of the converse to the final Value theorem (if applicable) that the limit of qj efor large tis lims!O 0(~-n+J). It

10 SYSTEMS STABILISED BY FEEDBACK

89

thus follows that the limit offset in the jth differential of the output path y is zero, finite or infinite according as n is less than, equal to or greater than N + j. So, suppose that w is the position of a fleeing hare and y the position of a dog pursuing it. Then a zero offset for j = I and n = 2 would mean that, if the hare maintained a constant acceleration (!) then at least the difference in the velocities of the dog and the hare would tend to zero with time. It appears then that an increase in N improves offset. However, it also causes a decrease in stability, and N = 2 is regarded as a practical upper limit.

CHAPTER 5

State-structured Deterministic Models In the last chapter we considered deterministic models in the classic inputoutput formulation. In this we discuss models in the more explicit state-space formulation, specialising rather quickly to the time-homogeneous linear case. The advantage of the state-space formulation is that one has a physically explicit model whose dynamics and whose optimisation can both be treated by recursive methods, without assumption of stationarity. Concepts such as those of controllability and observability are certainly best developed first in this framework. The advantage of the input-output formulation is that one can work with a more condensed formulation of the model (in that there is no necessity to expand it to a state description) and that the transform techniques then available permit a powerful treatment of, in particular, the stationary case. We shall later move freely between the two formulations, as appropriate. 1 STATE-STRUCTURE FOR THE UNCONTROLLED CASE: STABILTY; LINEARISATION Let us set questions of control and observation to one side to begin with, and simply consider a dynamic system whose course is described by a process variablex. We have already introduced the notion of state structure for a discrete-time model in Section 2.1. The system has state structure if x obeys a simple recursion X1

= a(Xt-l, t),

(1)

when x is termed the state variable. Dynamics are time-homogeneous if they are governed by time-independent rules, in which case (1) reduces to the form

(2) We have said nothing of the set of values within which x may vary. In the majority of practical cases xis numerical in value: we may suppose it a vector of finite dimension n. The most amenable models are those which are linear, and the assumption of linearity often has at least a local validity. A model which is statestructured, time-homogeneous and linear then necessarily has the form Xt

=

Axt-l

+b

(3)

92

STATE-STRUCTURED DETERMINISTIC MODELS

where A is a square matrix and ban n-vector. If the equation(/- A)x = b has a solution for x (see Exercise 1) then we can normalise b to zero by working with a new variable x- x.lfwe assume this normalisation performed, then the model (3) reduces to (4) Xr = Axr-1· The model has by now been pared down considerably, but is still interesting enough to serve as a basis for elaboration in later sections to controlled and imperfectly observed versions. We are now interested in the behaviour of the sequence x, = A'xo generated by (4). It obviously has an equilibrium point x = 0 (corresponding to the equilibrium point x = x of (3)). This will be the unique equilibrium point if I -A is nonsingular, when the only solution of x = Ax is x = 0. Supposing this true, one may now ask whether this equilibrium is stable in that x, ~ 0 with increasing t for any xo. Theorem 5.1.1 The equilibrium of system (4) at x = 0 is stable eigenvalues ofthe matrix A have modulus strictly less than unity.

if and only if all

Proof Let A be the eigenvalue of maximal modulus. Then there are sequences A'.xo which grow as A1, so IAI < 1 is necessary for stability On the other hand, no such sequence grows faster than t"- 1IAI 1, so IAI < 1 is also sufficient for ~~

0

A matrix A with this property is termed a stability matrix. More explicitly, it is termed a stability matrix 'in the discrete-time sense', since the corresponding property in continuous time differs somewhat. Note that if the equilibrium at zero is stable then it is necessarily unique; if it is not unique then it cannot be stable (Exercise 2). Note thatg1 = A 1 is the transient response function of the system (4) to a driving input. The fact that stability implies exponential convergence of this response to zero also implies lq·Stability of the filter thus constituted, and so of the filter of Exercise 4.4.1. The stability criterion deduced there, that C(I- Azf 1B should have all its singularities strictly outside the unit circle, is implied by that of Theorem 5.1.1. All this material has a direct analogue in the continuous-time case, at least for the case of vector x (to which we are virtually forced; see the discussion of Exercise 2.1.1). The analogue of (2), a state-structured time-homogeneous model, is

.X= a(x).

(5)

(For economy of notation we use the same notation a(x) as in (2), but the functions in the two cases are quite distinct.) The normalised linear version of this model, corresponding to (4~ is

l STATE-STRUCTURE FOR THE UNCONTROLLED CASE

93

i =Ax.

(6)

The analogue of the formal solution x 1 = A 1xo of (4) is the solution

x(t) = e1Ax(O)

(tAY 2:::-.x(O) J=O ]· co

:=

1

(7)

of (6). The stability criterion is also analogous.

Theorem 5.1.2 The equilibrium of system (6) at x = 0 is stable eigenvalues ofthe matrix A have real part strictly less than zero.

if and only if all

Proof Let O" be the eigenvalue of maximal real part. Then there are functions x(t) = e1A x(O) which grow as eu1, so Re(O") < 0 is necessary for stability. On the other hand, no such function grows faster than f!-IeRe(ulr, so Re(O") < 0 is also sufficient for stability. D A matrix A with this property is a stability matrix in the continuous-time sense. If a(x) is nonlinear then there may be several solutions of a(x) = 0, and so several possible equilibrium points. Recall the definitions of Section 1.2: that the domain ofattraction of an equilibrium point is the set of initial values from which the path would lead to that point, and that the equilibrium is locally stable if its domain of attraction includes a neighbourhood of the point. Henceforth we shall take 'stability' as meaning simply 'local stability'. For non-linear models the equilibrium points are usually separated (which is not possible in the linear case; see Exercise 2) and so one or more of them can be stable. Suppose that x is such an equilibrium point, and define the deviation .6.(t) = x(t) - x from the equilibrium value. If a(x) possesses a matrix ax of first derivatives which is continuous in the neighbourhood of x and has value A at x then equation (6) becomes

(8) to within a term o (.6.) in the neighbourhood of x. The state variable x will indeed remain in the neighbourhood of x if A is a stability matrix, and it is by thus testing A = ax(x) that one determines whether or not xis locally stable. The passage from (5) to (8) is termed a linearisation of the model in the neighbourhood of x, for obvious reasons, and the technique of linearisation is indeed an invaluable tool for the study of local behaviour. However, one should be aware that nonlinear systems such as (2) and (5) can show limiting behaviour much more complicated than that of passage to a static equilibrium: e.g. limit cycles or chaotic behaviour. Either of these would represent something of a failure in most control contexts, however, and it is reasonable to expect that optimisation will exclude them for all but the most exotic of examples.

94

STATE-STRUCTURED DETERMINISTIC MODELS

We have already seen an example of multiple equilibria in the harvesting example of Section 1.2. If the harvest rate was less than the maximal net reproduction rate then there were two equilibria; one stable and the other unstable. The stock example is of course the pendulum; in its linearised form the archetypal harmonic oscillator. If we suppose the pendulum undamped then the equation of motion for the angle a: of displacement from the hanging position is

a+ w2 sin 0: =

0'

(9)

where J is proportional to the effective length of the pendulum. There are two static equilibrium positions: a: = 0 (the hanging position) and a: = 1r (the inverted position). Let us bring the model to state form and linearise it simultaneously, by defining D. as the vector whose elements are the deviations of a: and a from their equilibrium values, and then retaining only first-order terms in D.. The matrix A for the linearised system is then

A=[±~ ~], where the + and - options refer to the inverted and hanging equilibria respectively. We find that A has eigenvalues ±w in the inverted position, so this is certainly unstable. In the hanging position the eigenvalues are ±iw, so this is also unstable, but only just-the amplitude of the oscillation about equilibrium remains constant. Of course, one can calculate these eigenvalues simply by determining which values a are consistent with a solution a:( t) = ea1 of the linearised version of equation (9). However, we shall tend to discuss models in their state-reduced forms. Discrete-time models can equally well be linearised; we leave details to the reader. We shall develop some examples of greater novelty in the next section, when we consider controlled processes. Exercises and comments

(1) This exercise and the next refer to the discrete-time model (3).1f (I- A)x = b has no solution for .X then a finite equilibrium value certainly does not exist. It follows also that I - A must be singular, so that A has an eigenvalue .>. = 1. (2) If (I- A)x = b has more than one solution then, again, I- A is singular. Furthermore, any linear combination of these solutions with scalar coefficients (i.e. any point in the smallest linear manifold At containing these points) is a solution, and a possible equilibrium. There is neutral equilibrium between points of At in that, once x 1 is in At, there is no further motion.

I STATE-STRUCTURE FOR THE UNCONTROLLED CASE

95

(3) Suppose that the component x11 of the vector x 1 represents the number (assumed continuous-valued) of individuals of age j in a population at time t, and that the x1, satisfy the dynamic equations CX>

Xot = LaixJ,t-1, J=O

(j > 0).

The interpretation is that a1 and b1 are respectively the reproduction and survival rates at age j. One may assume that b1 = 0 for some fmite j if one wishes the dimension of the vector x to be finite. Show that the equilibrium at x = 0 is stable (i.e. the population becomes extinct in the course of time) if all roots>. of CX>

Lbob1 · · ·bJ-IaJ>..-1-I = 1 j=O

are strictly less than unity in modulus. Show that the root of greatest modulus is the unique positive real root. (4) A pattern observed in many applications is that the recursion (2) holds for a scalar x with the function a(x) having a sigmoid form: e.g. :x?- /(1 + x) 2 (x ~ 0). l 0 and Jg >a are thus necessary and sufficient for stability of this equilibrium.

(6) The eigenvalues and eigenfunctions of A are important in determining the 'modes' of the system (4) or (6). Consider the continuous-time case (6) for definiteness, and suppose that A has the full spectral representation A = H AJI- 1, where A is the diagonal matrix of eignenvalues >.1. and the columns of H (rows of n- 1) are the corresponding right (left) eigenvectors. Then, by adoption of a new state vector x = Hx, one can write the vector equation (6) as the n decoupled scalar equations iJ = >.1. xi, corresponding to the n decoupled modes of variation. An oscillatory mode will correspond to a pair of complex conjugate eigenvalues. (7) The typical case for which a complete diagonalisation cannot be achieved is that in which A takes the form

96

STATE-STRUCTURED DETERMINISTIC MODELS

for non-zero J.l· One can imagine that two population groups both reproduce at net rate >., but that group 2 also generates members of group 1 at rate J.l· There is a double eigenvalue of A at >.,but Ai = )j-1

[>- }J.L] 0

>. '

At _

e

-e

>.c [ l

0

j.J,t]

1 .

One can regard this as a situation in which a mode of transient response e>-.1 (in continuous time) is driven by a signal ofthe same type; the effect is to produce an output proportional to te>-. 1• If there are n consecutive such stages of driving then the response at the last stage is proportional to t"e>.'. In the case when >. is purely imaginary (iw, say) this corresponds to the familiar phenomenon of resonance of response to input of the same frequency w. The effect of resonance is that output amplitude increases indefinitely with time until other effects (non-linearity, or slight damping) take over. 2 CONTROL, IMPERFECT OBSERVATION AND STATE STRUCTURE

We saw in Section 2.1 that achievement of 'state structure' for the optimisation of a controlled process implied conditions upon both dynamics and cost function. However, in this chapter we consider dynamics alone, and the controlled analogue of the state-structured dynamic relation (1) would seem to be Xt

= a(xr-l,ur-l,t),

(10)

which is indeed the relation assumed previously. Control can be based only upon what is currently observable, and it may well be that current state is not fully observable. Consider, for example, the task of an anaesthetist who is trying to hold a patient in a condition of light anaesthesia. The patient's body is a dynamical system, and so its 'physiological state' exists in principle, but is far too complex to be specifiable, let alone observable. The anaesthetist must then do as best he can on the basis of relatively crude indicators of state: e.g. appearance, pulse and breathing. In general we shall assume that the new observation available at timet is of the form

y,

= c(xr-1, Ur-I, r).

( 11)

So, if the new information consisted of several numerical observations, then y, would be a vector. Note that y, is regarded as being an observation on immediate past state Xx-r rather than on current state x,. This turns out to be the formally natural convention, although it can certainly be modified. It is assumed that the past control u,_ 1 is known; one remembers past actions taken. Relation (11) thus

2 CONTROL, IMPERFECT OBSERVATION AND STATE STRUCTURE

97

represents an imperfect observation on Xr-1, whose nature is perhaps affected both by the value chosen for u1_ 1 and by time. Information is cumulative; all past observations are supposed in principle to be available. In the time-homogeneous linear case x, u and y are vectors and relations (10) and (11) reduce to Xr

= Axr-1 + Bur-1

Yt = Cxr-1

(12) (13)

Formal completeness would demand the inclusion of a term Dur-1 in the righthand member of (13). However, this term is known in value, and can just as well be subtracted out. Control does not affect the nature of the information gained in this linear case. System (12), (13) is often referred to as the system [A, B, C ], since it is specified by these three matrices. The dimension of the system is n, the dimension ofx. The linear system is relatively tractable, which explains much of its popularity. However, for all its relative simplicity, the [A, B, C] system generates a theory which as yet shows no signs of completion. Once a particular control rule has been chosen then one is back in the situation of the last section. Suppose, for example, that current state is in fact observable, and that one chooses a control rule of the form u1 = Kx1 • The controlled plant equation for system (12) then becomes x 1 =(A+ BK)x1-J,

whose solution will converge to zero if A + BK is a stability matrix. The continuous-time analogue of relations (10), (11) is

x=

a(x, u, t),

y = c(x, u, t),

(14)

x=Ax+Bu,

y = Cx+Du.

( 15)

and of (12), (13)

with D usually normalised to zero. Note that, while the plant equation of (14) or (15) now becomes a first-order differential equation, the observation relation becomes an instantaneous relation, non-differential in form. This turns out to be the natural structure to adopt on the whole, although it can also be natural to recast the observation relation in differential form; see Chapter 25. In continuous time the system (15) with D normalised to zero is also referred to as the system [A, B, C ].

One sometimes derives a linear system (15) by the linearisation of a timehomogeneous non-linear system in the neighbourhood of a stable equilibrium point of the controlled system. Suppose that state and control values fluctuate about constant values x and u, so that y fluctuates about y = c(x, u). Defining the transformed variables :X= x- x, u' = u- u andy= y- y, we obtain the

98

STATE-STRUCTURED DETERMINISTIC MODELS

system (15) in these transformed variables as a linearised version of the system (14) with the identifications

D

=Cu.

Here the derivatives are evaluated at x u, and must be supposed continuous in a neighbourhood of this point. The approximation remains valid only as long as x and u stay in this neighbourhood, which implies either that (.X, u) is a stable equilibrium of the controlled system or that one is considering only a short time span. Subject to these latter considerations, one can linearise even about a timevariable orbit, as we saw in Section 2.9. Exercises and comments (1) Non-uniqueness of the state variable. If relations (12), (13) are regarded as just a way of realising a transfer function C(/- Azr 1Bz2 from u to y then this realisation is far from unique. By considering a new state variable Hx (for square nonsingular H) one sees that the system [HAH- 1, HB, cn- 1] realises the same transfer function as does [A, B, C ]. (2) A satellite in a planar orbit. Let (r, B) be the polar coordinates of a particle of unit mass (the satellite) moving in a plane and gravitationally attracted to the origin (where the centre of mass of the Earth is supposed situated). The Newtonian equations of motion are then

.. ·2 -2 • r = r0 - 1r --r- u,,

,r-

2 represents the gravitational force and u, and ua are the radial and where tangential components of a control force applied to the satellite. A possible equilibrium orbit under zero control forces is the circle of radius r = p, when the angular velocity must bee = w = r I rJ3). Suppose that small control forces are applied; define x as the 4-vector whose components are the deviations of r, r, f) and from their values on the circular orbit and u as the 2-vector whose elements are u, and ua. Show that for the linearised version (15) of the dynamic equations one has

J(

e

A_

-

[3~0 0

1

0

0 0 0 0 -2w 0

-~] 1 0

'

Note that the matrix A has eigenvalues 0, 0 and ±iw. The zero eigenvalues correspond to the neutral stability of the orbit, which is one of a continuous family of ellipses. The others correspond to the periodic motion of frequency w.

3 THE CAYLEY-HAMILTON THEOREM

99

In deriving the dynamic equations for the following standard examples we appeal to the Lagrangian formalism for Newtonian dynamics. Suppose that the system is described by a vector q of position coordinates flJ, that the potential and kinetic energies of the configuration are functions V(q) and T(q, q) and that an external force u with components Uj is applied. Then the dynamic equations can be written (j=1,2, ... ) (3) The cart and the pendulum. This is the celebrated control problem formulated in Section 2.8: the stabilisation of an inverted pendulum mounted on a cart by the exertion of a horizontal control force on the cart. In the notation of that section we have the expressions

v=

Lmg cos

Q,

Show that the equations of motion, linearised for small n, are

(M + m)q + mLii = u,

q+Lci =ga,

and hence derive expressions (2.60) for the matrices A and B of the linearised state equations. Show that the eigenvalues of A are 0, 0 and ±y'g(l + m/ M) / L. The zero eigenvalues correspond to the 'mode' in which the whole system moves at a constant (and arbitrary) horizontal velocity. The positive eigenvalue of course corresponds to the instability of the upright pendulum. (4) A popular class of controllers is provided by the PID (proportional, integral, differential) controllers, for which u is a linear function of current values of tracking error, its time-integral and its rate of change. Consider the equation for the controlled pendulum, linearised about the hanging position: & + u?n = u. Suppose one wishes to stabilise this to rest, so that a is itself the error. Note that a purely proportional control will never stabilise it. The LQ-optimal control of Section 2.8 would be linear in a and 0:, and so would be of the PD fonn. LQoptimisation will produce something like an integral term in the control only if there is observation error; see Chapter 12.

3 THE CAYLEY-HAMILTON THEOREM A deeper study of the system [A, B, C ] takes one into the byways oflinear algebra. We shall manage with a knowledge of the standard elementary properties of matrices. However, there is one result which should be formalised. Theorem 5.3.1 Let A be an n x n matrix. Then the first n powers /,A, A2 , •.• , A"- 1 constitute a basis for all the powers A' of A, in that scalar coefficients c,1 exist such that

100

STATE-STRUCTURED DETERMINISTIC MODELS n-1

(r = 0, 1,2, ... ).

A'= LCrjAi

(16)

)=0

It is important that the coefficients are scalar, so that each element of A' has the same representation in terms of the corresponding elements of I, A, ... , A"- 1•

Proof Define the generating function 00

«P(z) = L(AzY = (I- Az)- 1 j=O

where z is a scalar; this series will be convergent if lzl is smaller than the reciprocal of the largest eigenvalue of A. Writing the inverse as the adjugate divided by the determinant we have then II- Azi«P(z) = adj(/- Az)

( 17)

Now II- Azi is a polynomial with scalar coefficients a1 :

"

II- Azl = L:a1zi, }=0

say, and the elements of adj(I- Az) are polynomials in z of degree less than n. Evaluating the coefficient of z' on both sides of (17) we thus deduce that

" LaJAr-J

= 0

(r

~

n).

(18)

}=0

Relation (18) constitutes a recursion for the powers of A with scalar coefficients a1. It can be solved for r ~ n in the form (16). 0 The Cayley-Hamilton theorem asserts simply relation (18) for r = n, but this has the extended relation (18) and Theorem 5.3.1 as immediate consequences. It is sometimes expressed verbally as 'a square matrix obeys its own characteristic equation', the characteristic equation being the equation L,'J=o a1>..n-J = 0 for the eigenvalues >... Exercises and comments (1) Define the nth degree polynomial P(z) - II- Azl = L,'J=o aizi. If we have a discrete-time system x 1 = Ax1-1 then Theorem 5.3.1 implies that any scalar linear function~~ = cT x 1 of the state variable satisfies the equation P(5")~ 1 = 0. That is, a first-order n-vector system has been reduced to an nth-order scalar system.

4 CONTROLLABIUTY (DISCRETE TIME)

101

One can reverse this manipulation. Suppose that one has a model for which the process variable~ is a scalar obeying the plant equation P(§)~1 = bu 1_ 1 ( *) with ao = l. Show that the column vector x, with elements (~" ~r-l, ..• , ~r-n+ 1) is a state-variable with plant equation x 1 = Ax,_ 1 + Bu,_ 1, where

A=

[-~, 0

-a2 0 1

0

0

-an-I 0 0

-an 0 0 0

The matrix A is often termed the companion matrix of the nth-order system ( * ). (2) Consider the continuous-time analogue of Exercise 1. If x obeys x = Ax then it follows as above that the scalar ~ obeys the nth-order differential equation P(~)e

= o.

Reverse this argument to obtain a companion form (i.e. state-reduced form) of the equation P(~)~ =bu. Note that this equation must be regarded as expressing the highest-order differential of~ in terms of lower-order differentials, whereas the discrete-time relation (*) expresses the /east-lagged variable~~ in terms oflagged variables.

4 CONTROLLABILITY (DISCRETE TIME) The twin concepts of controllability and observability concern the respective questions of whether control bites deeply enough that one can bring the state variable to a specified value and whether observations are revealing enough that one can indeed determine the value of the state variable from them. We shall consider these concepts only in the case of the time-homogeneous linear system (12), (13~ when they must mirror properties of the three matrices A, B and C. The system is termed r-controllable if, given an arbitrary value of xo, one can choose control values u0 , u1 , •.• , u,_ 1 such that x, has an arbitrary prescribed value. For example, if m = n and B is non-singular then the system is 1controllable, because one can move from any value of xo to any value of x1 by choosing uo = ,B- 1 (x 1 - Axo). As a second example, consider the system X1

=

[

all

a21

0]

a22

Xr-l

+

[1] O

Ut-I

for which m = 1, n = 2. It is never 1-controllable, because u cannot affect the second component of x in a single step. It is uncontrollable if a21 = 0, because u can then never affect this second component. It is 2-controllable if a21 =/=- 0, because

102

STATE-STRuCTURED DETERMINISTIC MODELS

x2

-

2

A xo = Bu1

+ ABuo

= [

~ ~~: ] [ ~],

and this has a solution for uo, ur if a21 =!= 0. This argument generalises. Theorem 5.4.1 matrix

Then-dimensional system

[A, B, ·]is r-controllahle if and only if the ( 19)

hasrankn.

We write the system as [A, B, ·]rather than as {4, B, q since the matrix Cevidently has no bearing on the question of controllability. The matrix (19) is written in a partitioned form; it has n rows and mr columns. The notation T;on is clumsy, but short-lived and motivated by the limitations of the alphabet; read it as 'test matrix of size r for the control context: Proof If we solve the plant equation (12) for x, in terms of the initial value x 0 and subsequent control values we obtain the relation

(20) The question is whether this set of linear equations in uo, u 1, •.. , u,_ 1 has a solution, whatever the value of then-vector x,- A'xo. Such a solution will always exist if and only if the coefficient matrix of the u-variables has rank n. This matrix is just r,' 00 , whence the theorem follows. D If equation (20) has a solution at all then in general it has many. We shall find a way of determining 'good' solutions in Theorem 5.4.3. Meanwhile, the CayleyHamilton theorem has an important consequence. Theorem 5.4.2 Ifa system of dimension n is r-controllable, then it is also s-controllahlefor s ~min (n, r). Proof The rank of r,an is non-decreasing in r, so r-controllability certainly implies s-controllability for s ~ r. However, Theorem 5.3.1 implies that the rank of r;on is constant for r ~ n, because it implies that the columns of r,on are then linear combinations (with scalar coefficients) of the columns of ~ 00 . The system is thus n-controllable if it is r-controllable for any r, and we deduce the complete assertion. 0

If the system is n-controllable then it is simply termed controllable. This is a reasonable convention, since Theorem 5.4.2 then implies that the system is

4 CONTROLLABILITY (DISCRETE TIME)

103

controllable if it is r-controllable for any r, and that it is r-controllable for r ~ n if it is controllable. One should distinguish between controllability, which implies that one can bring the state vector to a prescribed value in at most n steps, and the weaker property of stabilisability, which requires only that a matrix K can be found such that A + BK is a stability matrix, and so that the policy ur = Kx 1 will stabilise the equilibrium at x = 0. It will be proved incidentally in Section 6.1 that, if the process can be stabilised in any way, then it can be stabilised in this way; also that controllability implies stabilisability. That stabilisability does not imply controllability follows from the case in which A is a stability matrix and B is zero. This is uncontrolled, and so not controllable, but stable, and so stabilisable. Note, however, that stabilisability does not imply the existence of a control which stabilises the process to an arbitrary prescribed equilibrium point; see Sections 6.2 and 6.6. Finally, the notion of finding a u-solution to the equation system (20) can be made more definite if we require that the transfer from x 0 to x, be achieved optimally, in some sense.

Theorem 5.4.3

(i) r-controllability is equivalent to the demand that the matrix r-1

~n = LABQ-IBT(AT)i j=O

(21)

should be positive definite. Here Q is a prescribed positive definite matrix. (ii) If the process is r-controllable then the control which achieves the passage from prescribed xo to prescribed x, with minimal control cost! 'L:01 u'J Qur is

(0

~ r

< r).

(22)

Proof Let us take assertion (ii) first. We seek to minimise the control cost subject to the constraint (20) on the controls (if indeed this constraint can be satisfied, i.e. if controls exist which will effect the transfer). Free minimisation of the Lagrangian form r-1

r-1

r=O

r=O

!Lu;QUr + AT(x,- A'xo- LAr-r-I Bu.r) yields the control evaluation

in terms of the Lagrange multiplier A. Evaluation of A by substitution of this solution back into the constraint (20) yields the asserted control (22).

104

STATE-STRUCTURED DETERMINISTIC MODELS

However, we see that (22) provides a control rule which is acceptable for general x 0 and x, if and only if the matrix cr;on is non-singular. The requirement of non-singularity of cr;on must then be equivalent to that of controllability, and so to the rank condition of Theorem 5.4.1. This is a sufficient proof of assertion (i), but we can give an explicit argument. Suppose ~on singular. Then there exists an n-vector w such that wT cr;on = 0, and so wT cr;cnw = 0, or r-1

L(wT£B)Q-1(wT£B)T =0. j=O

But the terms in this sum are individually non-negative, so the sum can be zero only if the terms are individually zero, which implies in turn that wT Ai B = 0 (j = 0, 1, ... , r - 1). That is, wTr;on = 0, so that r;on is of less than full row-rank, n. This chain of implications is easily reversed, demonstrating the equivalence of the two conditions: T~on is of rank n if and only if a;:on is, i.e. if and only if the non-negative definite matrix ~on is in fact positive definite. 0 The matrix a;:on is known as the control Gramian. At least, this is the name given in the particular case Q =I and r = n. As the proof will have made clear, the choice of Q does not affect the definiteness properties of the Gramian, as long as Q is itself positive definite. Consideration of general Q has the advantage that we relate the controllability problem back to the optimal regulation problem of Section 2.4. We shall give some continuous-time examples in the exercises of the next section, for some of which the reader will see obvious discrete-time analogues.

5 CONTROLLABiliTY {CONTINUOUS TIME) Controllability considerations in continuous time are closely analogous to those in discrete time, but there are also special features. The system is controllable if, for a given t > 0, one can find a control {u(T);O::;;,; T < t} which takes the state value from an arbitrary prescribed initial valuex(O) to an arbitrary prescribed terminal value x(t). The value oft is immaterial to the extent that, if the system is controllable for one value oft, then it is controllable for any other. However, the smaller the value of t, and so the shorter the interval of time in which the transfer must be completed, the more vigorous must be the control actions. Indeed, in the limit t ! 0 infinite values of u will generally be required, corresponding to the application of impulses or differentials of impulses. This makes clear that the concept of r-controllability does not carry over to continuous time, and also that some thought must be given to the class of controls regarded as admissible.

5

CONTROLLABIUTY (CONTINUOUS TIME)

105

It follows from the Cayley-Hamilton theorem and the relation 00

eAr = ~)At/ fj!.

(23)

j=O

implicit in (7) that the matrices I, A, A 2 , ... , A"- 1 constitute a basis also for the family of matrices {eAr; t ~ 0}. Here n is, as ever, the dimension of the system. We shall define r;on again as in (19), despite the somewhat different understanding of the matrix A in the continuous-time case.

Theorem 5.5.1 (i) The n-dimensional system [A, B, · ] is controllable ifand only the matrix r;on has rank n. (ii) This condition is equivalent to the condition that the control Gram ian G(t)con

=

1'

eAT BQ-1 BTeATT dr

if

(24)

should be positive defmite (for prescribed positive t and positive definite Q ~ (iii) If the system is controllable, then the control which achieves the passage from prescribed x(O) to prescribed x( t) with minimal control cost! J~ uT Qu dr is (25)

Proof If transfer from prescribed x(O) to prescribed x(t) is possible then controls {u( T); 0 ~ T < t} exist which satisfY the equation x(t)- eArx(O) =

1' eA(r-T)Bu(~)

dr,

{26)

analogous to (20). There must then be a control in this class which minimises the control cost defined in the theorem; we find this to be (25) by exactly the methods used to derive (22). This solution is acceptable if and only if the control Gramian G(tt00 is non-singular (i.e. positive definite); this is consequently the necessary and sufficient condition for controllability. As in the proof of Theorem 5.43 (i): if G{trn were singular then there would be ann-vector w for which wT G{t)con = 0, with the successive implications that wTeA'B = 0 (t ~ 0), wTAiB = 0 (j = 0, 1, 2, ... ) and wT ~on = 0. The reverse implications also hold. Thus, an alternative necessary and sufficient condition for controllability is that r:;on should have full ~~

0

While the Gramians G( tt00 for varying t > 0 and Q > 0 are either all singular or all non-singular, it is evident that G(trn approaches the zero matrix as t approaches zero, and that the control (25) will then become infinite.

106

STATE-STRUCTURED DETERMINISTIC MODELS

Exercises and comments (1) Consider the satellite example of Exercise 2.1 in its linearised form. Show that the system is controllable. Show that it is indeed controllable under tangential thrust alone, but not under radial thrust alone.

(2) Consider the two-variable system x1 = >..ixi + u (j = 1, 2). One might regard this as a situation in which one bas two rooms, roomj having temperature xi and losing temperature at a rate ->..1x1, and beat being supplied (or extracted) exogenously at a common rate u. Show that the system is controllable if and only if >.. 1 =f. >.. 2. Indeed, if >..1 = >..2 < 0 then the temperature difference between the two rooms will converge to zero in time, however u is varied. (3) The situation of Exercise 2 can be generalised. Suppose, as in Exercise 1.5, that the matrix A is diagonalisable to the form s- 1AH. With the change of state variable to the set of modal variables .i = Hx considered there the dynamic equations become j_;j

= }v.ij + 2.: bjkUk, k

where bfk is tbejktb element of the matrix HB. Suppose all the >..1 distinct; it is a fact that the square matrix withjkth element >..j'- 1 is then no~-singular. Use this fact to show that controllability, equivalent to the fact that r;on has rank n, is equivalent to the fact that HB should have no rows which are zero. In other words, the system is controllable if and only if there is some control input to any mode. This ~ssertion does not, however, imply a failure of controllability if there are repeated eigenvalues. 6 OBSERVABILITY The notion ofcontrollability rested on the assumption that the initial value of state was known. If, however, one must rely upon imperfect observations, then it is a question whether the value of state (either in the past or in the present) can be determined from these observations. The discrete-time system [A, B, C] is said to be r-observable if the value ofxo can be inferred from knowledge of the subsequent observations Y1.Y2 ... ,y, and subsequent relevant control values u 0 , u1 , ... , ur-2· Note that, if xo can be thus determined, then x 1 is also in principle simultaneously determinable for all t for which one knows the control history. The notion of observability stands in a dual relation to that of controllability; a duality which indeed persists right throughout the subject. We have the determination Yr

=

C

l

T-2 2.: Ar-j- Bui + Ar-i xo [ 2

J=O

6 OBSERVABILITY

107

of y.,. in terms of Xo and subsequent controls. Thus, if we define the reduced observation

y.,. =

-r-2

y.,. - C

L .A.'T-j- Buj 2

j=O

then xo is to be determined from the system of equations

y.,. =

C.A..,._ 1xo

(0 < T:::::;; r).

{27)

These equations are mutually consistent, by hypothesis, and so have a solution The question is whether this solution is unique. This is the reverse of the situation for controllability, when the question was whether equation (20) for the u-values bad a solution at all, unique or not. Note an implication of the system (27): that the property of observability depends only upon the matrices A. and C; not all upon B. We define the matrix

I I CA.

~bs r

=

CA c2

CA'._

1

(28)

'

the test matrix of size r for the observability context.

Theorem 5.6.1 (i) Then-dimensional system [A,·, C] is r-observable if and only if the matrix T~bs has rank n. (ii) Equivalently. the system is r-observable ifand only ifthe matrix r-1

~bs

= L(C.A.T-I?M-ICA-r-1

(29)

r-=0

is positive definite, for prescribed positive de]znite M. (iii) Ifthe system is r-observable then the determination ofXo can be expressed r-1

xo =

(~bs)-IM-1 L(CAT-I)Ty.,..

{30)

r-=0

(iv) Ifthe system is r-observable then it is s-observablefor s ;;:::: min (n, r). Proof If system (27) has a solution for Xo (which is so by hypothesis) then this solution is unique if and only if the coefficient matrix r;bs of the system bas rank n, implying assertion (i). Assertion (iv) follows by appeal to the Cayley-Hamilton theorem, as in Theorem 5.4.2. If we define the deviation ry, = y,. - CA.,.- 1xo then equations {27) amount to ry,. = 0 (0 < T:::::;; r). Ifthese equations were not consistent we could still define a

108

STATE-STRUCTURED DETERMINISTIC MODELS

'least-square' solution to them by minimising any positive-definite quadratic form in these deviations with respect to xo. In particular, we could minimise :E~-=.b ?JJM- 11Jr. This minimisation yields expression (30). If equations (27) indeed have a solution (i.e. are mutually consistent, as we suppose) and this is unique then expression (30) must equal this solution: the actual value of XQ. The criterion for uniqueness of the least-square solution is that G,?bs should be non-singular, which is exactly condition (ii). As in Theorem 5.4.3, equivalence of 0 conditions (i) and (i) can be verified directly, if desired. Note that we have again found it helpful to bring in an optimisation criterion. This time it was a question, not of finding a 'least cost' solution when many solutions are known to exist, but of finding a 'best fit' solution when no exact solution may exist. This approach lies close to the statistical approach necessary when observations are corrupted by noise; see Chapter 12. Matrix (29) is the

observation Gramian. The continuous-time version of these results will now be apparent, with a proof which bears the same relation to that of Theorem 5.6.1 as that of Theorem 5.5.1 does to the material of Section 4.

Theorem 5.6.2 (i) Then-dimensional continuous-time system [A 1 • 1 C] is observable ifand only ifthe matrix T~bs defined by (28) has rank n. (ii) This condition is equivalent to the condition that the observation Gramian 1

G(t)obs

= 1 (Ce.A.'~"?M- 1 Ce-4-'~" dr

(31)

should be positive definite (for prescribedpositive t and positive definite M ). (iii) Ifthe system is observable then the determination ofx(O) can be written

where ji(t) =y(t)

-1'

CeA(t-'T)Bu(r)dr.

A way of generating real-time estimates of current state is to drive a model of the plant by the apparent discrepancy in observation. For the continuous-time model (15) this would amount to generating an estimate x(t) of x(t) as a solution ofthe equation

.i =Ax+ Bu+ H(y- Cx)

1

(32)

where the matrix His to be chosen suitably. One can regard this as the realisation of a ftlter whose output is the estimate x generated from the known inputs u and

109

6 OBSERVABILITY

y. Such a relation is spoken of as an observer, it is unaffected in its performance by the control policy. We shall see in Chapter 12 that the optimal estimating relation in the statistical context, the Kalmanfrlter, is exactly ofthis form. Denote the estimation error x(t)- x(t) by ~(t} By subtracting the plant equation from relation (32) and settingy = Cx we see that

A= (A -HC)~. Estimation will thus be successful (in that the estimation error will tend to zero with time) if A - HC is a stability matrix. If it is possible to find a matrix H such that this is so then the system [A,·, C] is said to be detectable; a property corresponding to the control property of stabilisability.

Exercises and comments (1) Consider the linearised satellite model of Exercise 2.2. Show that state x is observable from angle measurements alone (i.e. from observation of 0 and 0) but not from radial measurements alone. (2) The scalar variables xi ( j equations

X)

= 1, 2, ... , n)

= 2(1 + Xnr 1 -X)+ u,

Xj

= Xj-1

of a metabolic system obey the -

Xj

(j

= 2, 3, ... In).

Show that in the absence of the control u there is a unique equilibrium point in the positive orthant. Consider the controlled system linearised about this equilibrium point. Show that it is controllable, and that it is observable from measurements of x 1 alone.

Notes We have covered the material which is of immediate relevance for our purposes, but this is only a small part of the very extensive theory which exists, even (and especially) for the time-homogeneous linear case. One classical piece of work is the Routh-Hurwicz criterion for stability, which states in verifiable form the necessary and sufficient conditions that the characteristic polynomial I>J - A I = 0 should have all its zeros strictly in the left half-plane. Modern work has been particularly concerned with the synthesis or realisation problem: can one find a system [A, B, C] which realises a given transfer function G? If one can find such a realisation, of finite dimension, then it is of course not unique (see Exercise 2.1). However, the main consideration is to achieve a realisation which is minimal in that it is of minimal dimension. One has the important and beautiful theorem: a system [A, B, C] realising G is minimal if and only if it is both controllable and observable. (See, for example, Brockett (1970) p. 94)

llO

STATE-STRUCTURED DETERMINISTIC MODELS

However, when we resume optimisation, the relevant parts of this further theory are in a sense generated automatically and in the operational form dictated by the goal. So, existence theorems are replaced by explicit solutions (as Theorem 5.4.3 gave greater definiteness to Theorem 5.4.1), the family of 'good' solutions is generated by the optimal solution as the cost function is varied, and the conditions for validity of the optimal solution provide the minimal and natural conditions for existence or realisability.

CHAPTER 6

Stationary Rules and Direct Optimisation for the LQModel The LQ model introduced in Sections 2.4, 2.8 and 2.9 has aspects which go far beyond what was indicated there, and a theory which is more elegant than the reader might have concluded from a first impression. In Section 1 we deal with a central issue: proof of the existence of infinite-horizon limits for the LQ regulation problem under appropriate hypotheses. The consequences of this for the LQ tracking problem are considered in Section 2. However, in Section 3 we move to deduction of an optimal policy, not by dynamic programming, but by direct optimisation of the trajectory by Lagrangian methods. This yields a treatment of the tracking problem which is much more elegant and insightful than that given earlier, at least in the stationary case. The approach is one which anticipates the maximum principle of the next chapter and provides a natural application of the transform methods of Chapter 4. As we see in Sections 5 and 6, it generalises with remarkable simplicity; we continue this line with the development of time-integral methods in Chapters 18-21. The material of Sections 3, 5, 6 and 7 must be regarded, not as a systematic exposition, but as a first sketch of an important pattern whose details will be progressively completed. 1 INFINITE-HORIZON UMITS FOR THE LQ REGULATION PROBLEM One hopes that, if the horizon is allowed to become infinite, then the control problem will simplify in that it becomes time-invariant, i.e. such that a time-shift leaves the problem unchanged. One hopes in particular that the optimal policy will become time-invariant in form, when it is referred to as stationary. The stationary case is the natural one in a high proportion of control contexts, where one has a system which, for practical purposes, operates indefinitely under constant conditions. The existence of infinite-horizon limits has to be established by different arguments in different cases, and will certainly demand conditions of some kind-time homogeneity plus both the ability and the incentive to control. In this section we shall study an important case, the LQ regulation problem of Section 2.4. In this case the value function F(x, t) has the form !xTII,x where II obeys the Riccati equation (2.25). It is convenient to write F(x, t) rather as Fs(x) = !xTII(s)X where s = h- tis the 'time to go'. The matrix II(o) is then that

112

STATIONARY RULES AND DIRECT OPTIMISATION

associated with the terminal cost function. The passage to an infinite horizon is then just the passage s --t +oo, and infinite-horizon limits will exist if lies) has a limit value II which is independent ofiico) for the class of terminal cost functions one is likely to consider. In this case the matrix K 1 = K(s) of the optimal control rule (2.27) has a corresponding limit value K, so that the rule takes the stationary form u1 = Kxt. Two basic conditions are required for the existence of infmite-horizon limits in this case. One is that of sensitivity: that any deviation from the desired rest point x = 0, u = 0 should ultimately carry a cost penalty, and so demand correction. The other is that of controllability: that such any such deviation can indeed be corrected ultimately. We suppose S normalised to zero; a normalisation which can be reversed if required by replacing R and A in the calculations below by R - sT Q- 1S and A - ST Q- 1B respectively. The Riccati equation then takes the form

(s

= 1,2, ... ).

(1)

where/ has the action /II= R

+ ATTIA- ATTIB(Q + BTIIB)- 1BTIIA.

(2)

Lemma 6.1.1 Suppose that II(o) = 0, R ;;::: 0, Q > 0. Then the sequence {II(s)} is non-decreasing (in the ordering ofpositive-definiteness). Proof We have F1 = xT Rx;;::: 0 = Fo. Thus, by Theorem 3.1.1, Fs(x) = xTII(s)X is non-decreasing ins for fixed x. That is, II(s) is non-decreasing in the matrix-definiteness sense. Lemma 6.1.2 Suppose that II(o) = 0, R;;::: 0, Q > 0 and that the system [A, B, ·]is either controllable or stabilisable. Then {II(s)} is bounded above and has a finite limit II. Proof To demonstrate boundedness one must demonstrate that a policy can be found which incurs a finite infinite-horizon cost for any prescribed value x of initial state. Controllability implies that there is a linear control rule (e.g. that suggested in the proof of Theorem 5.4.3) which, for any x 0 = x, will bring the state to zero in at most n steps and at a finite cost xTII• x, say. The cost of holding it at zero thereafter is zero, so we can assert that (3) for all x. Stabilisability implies the same conclusion, except that convergence to zero takes place exponentially fast rather than in a finite time. The nondecreasing sequence {II(s)} is thus bounded above by II* (in the positive-definite

I LIMITS FOR THE LQ REGULATION PROBLEM

113

sense) and so has a limit II. (More explicitly, take x = e1, the vector with a unit in the jth place and zeros elsewhere. The previous lemma and relation (3) then imply that 1rjj8, the jjth element of II(s)• is non-decreasing and bounded above by rrj. It thus converges. By then taking x = e1 + ek we can similarly prove the convergence of rr1kJ.) 0 We shall now show that, under natural conditions, II is indeed the limit of II(s) for any non-negative definite II(o)· The proof reveals more.

Theorem 6.1.3 Suppose that R > 0, Q > 0 and the system [A, B, ·]is either controllable or stabilisahle. Then (i) The equilibrium Riccati equation II

=JII

(4)

has a unique non-negative definite solution II. (ii) For any finite non-negative definite II(o) the sequence {II(!)} converges to II (iii) The gain matrix r conesponding to II is a stability matrix. Proof Define ll as the limit of the sequence f(slo. We know by Lemma 6.1.2 that this limit exists, is finite and satisfies (4). Setting u1 = Kx1 and Xt+i = (A + BK)x1 = rx, in the relation

(5) where K and rare the values corresponding to II, we see that we can write (4) as

(6) Consider the form

V(x) and a sequence Xt

=

xTnx

(7)

= r 1Xo, for arbitrary X(). Then V(xt+i) - V(x 1) = -xJ (R + KT QK)x 1 ~ 0.

(8)

Thus V(x 1) decreases and, being bounded below by zero, tends to a limit. Thus

xJ(R+KTQK)x1 - 0

(9)

which implies that x 1 - 0, since R + KT QK ~ R > 0. Since .xo is arbitrary this implies that rt - t 0, establishing (iii). We can thus deduce from (6) the convergent series expression for II: 00

II =

:l)rTy (R + KT QK)rl. j=O

( 10)

114

STATIONARY RULES AND DIRECT OPTIMISATION

Note now that, for arbitrary finite non-negative definite ll(o).

ll(s) = fsln(o) ~ f(s)o

--+

n.

(11)

Comparing the minimal s-horizon cost with that incurred by using the stationary rule Ut Kx1 we deduce a reverse inequality

=

s-i

ll(s) ~ 2)rTy (R + KT QK)fi + (rTyn(o)rs

--+

IT.

(12)

j=O

Relations (11) and (12) imply (ii). Finally, assertion (i) follows because, if another fmite non-negative definite solution of(4), then fi = f(slfi --+ IT.

fi

is 0

It is gratifying that proof of the convergence llc1J -+ IT implies incidentally that is a stability matrix. Of course, this is no more than one would expect: if the optimal policy is successful in driving the state variable x to zero then it must indeed stabilise the equilibrium point x = 0. The proof appealed to the condition R > 0, which is exactly the condition that any deviation of x from zero should be penalised immediately. However, we can weaken this to the condition that the deviation should be penalised ultimately.

r

Theorem 6.1.4 The conclusions of Theorem 6.1.3 remain valid if the condition that R > 0 is replaced by the condition that, if R = LTL then the system [A 1 • 1 L] is either observable or detectable. Proof ·Relation (9) now becomes

(Lx 1 )T(Lx1) + (Kx 1)TQ(Kx1)--+ 0 which implies that Kx1 --+ 0 and Lx 1 --+ 0. These convergences, with the relation x, =(A+ BK)xt-1 imply thatx1 ultimately enters a manifold for which 1

Lx,

=0

1

Xt

= AXt-i·

(13)

=

The observability condition implies that these relations can hold only if x 1 0. The detectability condition implies that we can find an H such that A - HL is a stability matrix, and since relations (13) imply that x 1 = (A - H L )xr-i 1 then again x, --+ 0. Thus x 1 --+ 0 under either condition. This fact established, the proof continues as in Theorem 6.1.3. 0 We can note the corollaries of this result, already mentioned in Section 5.4.

Corollary 6.1.5 (i) Controllability implies stabilisability. (ii) Stabilisability to x = 0 by any means implies stabilisability by a control of the linear Markov form u1 = Kx1 •

2. STATIONARY TRACKING RULES

115

Proof (i) The proof of Theorem 6.1.3 demonstrated that a stabilising policy of the linear Markov form could be found if the system were controllable. (ii) The optimal policy under a quadratic cost function is exactly of the linear Markov form, so, if such a policy will not stabilise the system (in the sense of ensuring a finite-cost passage to x = 0), then neither will any other. 0 2. STATIONARY TRACKING RULES

The proof of the existence of infinite horizon limits demonstrates the validity of the infinite-horizon tracking rule (2.67) of Section 2.9, at least if the hypotheses of the last section are satisfied and the disturbances and command signals are such that the feedforward term in (2.67) is convergent. We can now take matters somewhat further and begin, in the next section, to see the underlying structure. In order to avoid undue repetition of the material of Section 2.9 and to link up with conventional control ideas we shall discuss the continuous-time case. The continuous-time analogue of (2.67) would be u- uc

= K(x- xc)- Q- 1BT

1

00

erT-rii[d(t + T)- dc(t + T)] dT

( 14)

where II, K and rare the infinite-horizon limits of Section 1 (in a continuous-time version) and the time argument t is understood unless otherwise stated. We regard (14) as a stationary rule because, although it involves the time-dependent signal ( 15) this signal is seen as a system input on which the rule (14) operates in a stationary fashion. A classic control rule for the tracking of a command signal r in the case uc = 0 would be simply ( 16)

where u = Kx is a control which is known to stabilise the equilibrium x = 0. We see that (14) differs from this in the feedforward term, which can of course be calculated only if the future courses of the command signal and disturbance are known. Neither rule in general leads ultimately to perfect following, ('zero offset') although rule (14) does so if d - ~. defined in (15), tends to zero with increasing time. This is sometime expressed as the condition that all unstable modes of (xc, uc, d) should satisfy the plant equation. There is one point that we should cover. In most cases one will not prescribe the course of all components of the process vector x, but merely that of certain linear functions of this vector. For example, an aeroplane following a moving target is merely required to keep that target in its sights from an appropriate distance and angle; not to specify all aspects of its dynamic state. In such a case it is better not

116

STATIONARY RULES AND DIRECT OPTIMISATION

to carry out the normalisation of x, u and d adopted in Section 2.9. If we assume that uc = 0 and work with the raw variables then we find that the control rule (14) becomes rather 00

u = Kx- Q- 1BT [

erTT[lld(t + r)- (R- sT Q- 1s)xc(t + r)] dr

(17)

Details of derivation are omitted, because in the next section we shall develop an analysis which, at least for the stationary case, is much more direct and powerful than that of Section 2.9. Relation (17) is exactly what we want. A penalty term such as (x- r l R(x- r) is a function only of those linear functions of (x- r) which are penalised. The consequence is then that Rr and Sr are functions only of those linear functions of r which are prescribed. If we consider the case when S, uc and dare zero and r is constant then relation (17) reduces to {18) which is to be compared with relation (16) and must be superior to it (in average cost terms). When we insert these two rules into the plant equation we see that x settles to the equilibrium value r- 1BKx = -r- 1JI1xC for rule (16) and r- 1J(rT)- 1Rr for rule (18). Here Jis again the control-power matrix BQ- 1BT. We obtain expressions for the total offset costs in Exercise 1 and Section 6. Exercises and comments (1) Verify that the offset cost under control (16) (assuming S zero and r constant) is! (Ar) T P(Ar), where P

= (rT)- 1(R + KT QK)r-l = -nr- 1 -

(rT)- 1ll.

We shall come to evaluations under the optimal rule in Section 6. However, if R is assumed non-singular (so that all components of r are necessarily specified) then location of the optimal equilibrium point by the methods of Section 2.10 leads to the conclusion that the offset cost under the optimal rule (18) again has the form !{Ar?P(Ar), but now with P = (AR- 1AT +BQ- 1BT)- 1. This is generalised in equation (43). 3 DIRECI' TRAJECTORY OPTIMISATION: WHY THE OPTIMAL FEEDBACK/FEEDFORWARD CONTROL RULE HAS THE FORM IT DOES Our analysis of the disturbed tracking problem in Section 2.9 won through to a solution with an appealing form, but only after some rather unappealing

117

3 DIRECT TRAJECTORY OPTIMISATION

calculations. Direct trajectory optimisation turns out to offer a quick, powerful and transparent treatment of the problem, at least in the stationary case. The approach carries over to much more general models, and we shall develop it as a principal theme. Consider the discrete-time model of Section 2.9, assuming plant equation (2.61) and instantaneous cost function (2.62). Regard the plant equation at time T as a constraint and associate with it a vector Lagrange multiplier >.7 , so that we have a Lagrangian form 0 = L[c(xr, U71 r) + >.'J(xr- AXr-l- B~-l- d7 )] +terminal cost.

( 19)

T

Here the time variable T runs over the time-interval under consideration, wb.ich we shall now suppose to be h 1 < T < h2 ; the terminal cost is incurred at the horizon point T = hz. We use T to denote a running value of time rather than t, and shall do so henceforth, reserving t to indicate the particular moment 'now'. In other words, it is assumed that controls U7 for T < t have been determined, not necessarily optimally, and that the time t has come at which the value of ur is to be determined in the light of information currently available. We shall refer to the form 0 of (19) as a 'time-integral' since it is indeed the discrete-time version of an integral. We shall also require of a 'time-integral' a property which 0 possesses, that one optimises by extremising the integral.free/y with respect to all variables except those whose values are currently known. That is, optimisation is subject to no other constraint. The application of Lagrangian methods is certainly legitimate (at least in the case of a fixed and finite horizon) if all cost functions are non-negative definite quadratic forms; see Section 7.1. We can make the strong statement that the optimal trajectory from time t is determined by minimisation of 0with respect to (x7 , ~) and maximisation with respect to .\7 for all relevant T ~ t. This extremisation then yields a linear system of equations which we can write

R

s

[ I - AfT

ST

Q -Bfl

[3],

(t~r
where fl is the backwards translation operator defined in (4.9) and normalised disturbance

d is

(20) the

(21)

already introduced in Section 2.9. We have added a superscript (t) to emphasise that this is an optimisation from timet onwards; nothing is assumed of the (x, u) path before time t except that it is known. The effect of this is that the optimal control rule, when we deduce it, will be in closed-loop form.

118

STATIONARY RULES AND DIRECT OPTIMISATION

Let us write equation (20) as (22) where {(T} is then known, and { D.~l}, the course of the deviations from the desired path of the (henceforth) optimally-controlled process, is presumably determined by (20) plus initial conditions at T = t and terminal conditions at T = h. Note that the matrix ci> is Hermitian, in that if we define the conjugate of ci> = ci>(ET) as if>= ci>(ff- 1)T then if>= ci>. Suppose that ci>(z), with z a scalar complex variable, has a canonical factorisation (23) where ci>+(z) and ci>+(z)- 1 can be validly expanded on the unit circle wholly in non-negative powers of z and ci> _ ditto for non-positive powers. What this would mean is that an equation such as (24) for D. with known v (and valid for all T before the current point of operation) can be regarded as a stable forward recursion for D. with solution (25) 1

Here the solution is that obtained by expanding the operator ci>+(ff)- in nonnegative powers of§", and so is linear in present and past v; see Section 4.5. We have taken recursion (24) as a forward recursion in that we have solved it in terms of past v; it is stable in that solution (25) is certainly convergent for uniformly bounded v. Factorisation (23) then implies the rewriting of (22) as (26) so representing the difference equation (22) as the compositiOn of a stable f<;>rward recursion and a stable backward recursion. The situation may be plainer in terms of the scalar example of Exercise l. One finds generally that the optimisation of the path of a process generated by a forward recursion yields a recursion of double order, symmetric in past and future, and that if we can represent this double-order recursion as the composition of a stable forward recursion and a stable backward recursion, then it is the stable forward recursion which determines the optimal forward path in the infinite-horizon case (see Chapter 18). Suppose we let the horizon point h2 tend to +oo, so that (26) holds for all T ~ t. We can then legitimately half-invert (26) to (27)

3 DIRECT TRAJECTORY OPTIMISATION

119

if(.,. grows sufficiently slowly with increasing r that the expression on the right is 1 is expanded in non-positive powers of :Y. We thus have convergent when if>_ ( an expression for A.,. in terms of past A and present and future(. This gives us exactly what we want: an expression for the optimal control in the desired feedback/feedforward form.

.rr

Theorem 6.3.1 Suppose that d.,. grows sufficiently slowly with T that the semi-inversion (27) is legitimate. More specifically, that the semi-inversion

~+(~) ~ ~ ~r ~~-(~)-![it [

(27')

is legitimate in that the right-hand member is convergent when the operator if>_ ( :Y) -I is expanded in non-positive powers offf. Then

(i) The determination ofu1 obtained by setting T = tin relation (27') constitutes an expression ofthe infinite-horizon optimal control rule infeedbacklfeedforwardform. (ii) The Hermitian character of if> implies that the factorisation (26) can be chosen so that if>_ = «<> +, whence it follows that the operator which gives the feedforward component is just the inverse ofthe conjugate ofthe feedback operator. Relation (27') in fact determines the whole future course of the optimally controlled process recursively, but it is the determination of the current control u1 that is of immediate interest The relation at r = t determines u1 (optimally) in terms of x 1 and d1(r ~ t); the feedbacklfeedforward rule. Furthermore, the symmetry in the evaluation of these two components explains the structure which began to emerge in Section 2.9, and which we now see as inevitable. We shall both generalise this solution and make it more explicit in later chapters. The achievement of the canonical factorisation (23) is the equivalent of solving the stationary form of the Riccati recursion, and in fact the policyimprovement algorithm of Section 3.5 translates into a fast and natural algorithm for this factorisation. The assumptions behind solution (27) are two-fold First, there is the assumption that the canonical factorisation (23) exists. This corresponds to the assumption that infmite-horizon limits exist for the original problem of Section 2.4; that of optimal regulation to zero in the absence of disturbances. Existence of the canonical factorisation is exactly the necessary and sufficient condition for existence of the infinite-horizon limits; the controllability/sensitivity assumptions of Theorem 6.1.3 were sufficient, but probably not necessary. We shall see in Chapter 18 that the policy-improvement method for deriving the optimal infinite-horizon policy indeed implies the natural algorithm for determination of the canonical factorisation. The second assumption is that the normalised disturbance d.,. should increase sufficiently slowly with r that the right-hand member of (27), the feedforward

120

STATIONARY RULES AND DIRECT OPTIMISATION

term, should converge. Such convergence does not guarantee that the vector of 'errors' !1 1 will converge to zero with increasing t, even in the case when all components of this error are penalised. There may well be non-zero offsets in the limit; the errors may even increase exponentially fast. However, convergence of the right-hand member of (27) implies that d,. increases slowly enough with time that an infinite-horizon optimisation is meaningful. Specifically, suppose that the zeros of 1~-(z)! do not exceed p in modulus (where certainly p < l) and that the components of (Lr grow no faster than 7" in modulus. Then the feed-forward term in (27) is convergent if P'Y < 1. On the other hand, some components of !l,. may indeed grow at rate 7". For the particular ~ of (20), corresponding to the state-structured case, one finds a canonical factorisation (28) where

and nand K are the infinite-horizon limits of these quantities. Factorisation (28) differs slightly from (23) in that there is the interposing constant matrix ~0 , but this only slightly modifies the semi-inversion (27). In the case when r is only partially prescribed we would write the equation system (20) rather as

R

s

ST

Q [ 1-Aff Bff

[-AT ff-BTff- 1 0

1

]

[x]

(t)

u

=

>. ,.

and perform the same semi-inversion on this system as previously. The command

signal r occurs only in the combinations Rr and Sr, which are functions only of those components of r which are prescribed.

Exercises and comments (1) Consider the simple regulation problem in the undisturbed scalar case, when we can write the cost function as l:,.[Q.s- 2 (x,. - Ax,_I) 2 + R.x;J + terminal cost. The problem is thus reduced, since we have used the plant equation to eliminate u, and so need not introduce .>..The stationarity condition on x,. yields cl>(ff)x" = 0 (*), where ci>(z) = Q(l - Az)(l - Az- 1) + B2R. (This is not the ci> of (22~ but a reduced version of it)

!

4 AN ALTERNATIVE FORM FOR THE RICCATI RECURSION

121

The symmetric (in past and future) second-order equation (•) can be legitimately reduced to a first-order stable forward equation, which determines the infinite-horizon optimal path from an arbitrary starting point, so yieldin.g in effect an optimal control rule in open-loop form. Suppose the canonical factorisation il>(z) <X (1- rz)(l- rz- 1) where r is less than unity in modulus. (Note that this is exactly the determination of the optimal gain factor r given by equation (2.36)). Then division of (*) by the 'future' factor leaves the equation ( 1 - r ff)x.,. = 0, or x.,. = rx.,._ 1, which we know to be indeed the plant equation under optimal infinite-horizon control. (2) Use relations (27')-(29) to verify the determination (2.67) of the optimal

control rule. (3) The continuous time version of equation (22) (for the state-structured model (252)/(2.53)) is il>(f0)~ =(,where

R it>(s) = S [ sl -A

sT

Q -B

If we define the conjugate of it>(s) as il>(s) =it>( -s)T then it> is evidently selfconjugate. The analogue of the canonical factorisation (26) is it>(s) = it>_(s) il>+(s), where both it>+ and il>:;: 1 have a Laplace representation valid for Re(s) ~ 0 which involves only exponentials e-st for non-negative t (and so have all singularities in the left half-plane). it>_ has the complementary definition, and will in fact be the conjugate of it>+. Solution by the semi-inversion (27) then proceeds analogously. (4) Show that the roots of lit>-(z)l = 0 are just the eigenvalues of the optimal infinite-horizon gain matrix r.

4 AN ALTERNATIVE FORM FOR THE RICCATI RECURSION One could extremise the time-integral (19) recursively, and by doing so one is led to the expanded form of the optimality equation

F(x, t) = inf sup[c(x, u, t) Xr+loU

).

+ >.T(xt+l- Ax- Bu) + F(xt+ 1 , t + 1)].

Here we have written x 1, Ut and >.t+I simply as x, u and.>., and F(x, t) is the usual value function at timet, which we know to have the form !xTJI1x. Let us suppose for simplicity that we are in the time-homogeneous case when c(x, u) is given by (2.23) and that S has been normalised to zero. This normalisation can be reversed by replacing A and R in the expression below by A - BQ- 1S and R - sT Q- 1S respectively. If we extremise >. out first in the equation above then this is reduced to the usual optimality equation {2.29), leading to expressions (2.25) and (227) for the Riccati

122

STATIONARY RULES AND DIRECT OPTIMISATION

recursion and the optimal control. If we extremise the variables out in the order xt+ 1, u, >. then we find that the right-hand member of the equation undergoes the successive transformations

!(xTRx + uT Qu) - ..\T(Ax+ Bu) - !>.Trr;+\ ..\ --+ lxT Rx- >.TAx- 1>.Trr-1 >. - 1>.T J >. 2 2 t+l 2 --+

!xT[R +AT (J +II~\ f Ajx. 1

where J = BQ- 1BT, as ever. That is, the Riccati equation is recovered in the alternative form

(31) Further, if we trace through the extremal value of u1 which emerges from the above sequence of transformations we find that it is u 1 = K 1x 1 , as ever, but with the gain matrix K 1 having the evaluation

(32) instead of (2.28). We shall find a use for these alternative forms (31) and (32) in Chapter 16. In continuous time the two forms coalesce; expressions (2.55) and (2.56) for the continuous-time Riccati equation and control matrix are (with the normalisation to S = 0) the limit forms of both discrete-time versions. However, the alternative forms above have in common with the continuous-time forms that they reveal the role of the control-power matrix J = BQ- 1BT.

5 DIRECT TRAJECTORY OPTIMISATION FOR HIGHER-ORDER

CASES The approach of Section 6.2 generalises immediately. Suppose that the plant equation is generalised to

(33)

dx+!Jiu = d

where .91 = A(ff) = L:~=O A,§"' and !11 = B(ff) = 2:~ 1 B,§"', corresponding to pth-order dynamics. The time-integral (19) then generalises to 0=

2._)c(x.,, Un T) +

>.; (dx., + !Jiu.,- dr )] +terminal cost

(34)

T

and the equation system (20) correspondingly to

[!

sT Q

!11

-?] [x-xcl(r) = [0] !11 0

u-uc >.

0

T

d

(35) T

6 THE CONCLUSIONS IN TRANSFER FUNCTION TERMS

123

Here an expression such as .i1 again denotes A (y- 1) T, the conjugate of d. The normalised disturbance d now bas the definition d- de= d- d r - !Muc. We can again write system (35) as (22); the matrix of operators .P(ff) thus defined again bas the self.conjugacy property ~ = .P. Now, provided that a canonical factorisation (23) of .P(z) exists (which is again ensured by appropriate controllability and sensitivity hypotheses) th.en the manipulations (26)-(27) go through exactly as previously to yield the optimal control rule implied in (27). This is expressed in terms of present and past process variables, and so we must suppose these observable. The case of imperfect observation is best left until we treat the full stochastic case (see Chapters 12 and 20 for the cases p = 1 and p unrestricted respectively). Because we have restricted the dynamics to finite order p the canonical factors are polynomial of degree p and a fast iterative method of factorisation based upon the policy improvement algorithm is still available; see Chapter 18. The continuous time results are analogous, with, for example, .91 having a representation A(!?~) and its conjugate .91 a representation A( -!?I) T_ The canonical factors have the characterisation given in Exercise 3.3.

6 THE CONCLUSIONS IN TRANSFER FUNCTION TERMS The analysis of Sections 3 and 5 is almost in transfer function terms as it stands; completion of the view raises some interesting points. Consider the continuous-time version of system (20): (36) This then constitutes a filter (the optimal filter, as a transformation from input (to output D.) with transfer function .P(s)- 1• This conclusion seems so simple that one wonders whether the subtleties of canonical factorisation etc. were necessary. However, they were indeed so. For one thing, the symmetry of .P implies that the system (36) is unstable as a forward dynamic system, and that the corresponding filter cannot be both causal and stable. That it is not causal is, of course, because the optimal control bas a feed-forward component, anticipating the effect of future disturbances. The stable inversion must be of the form

D.(t)

=I:

g(T)((t- T) dT

(37)

where g( T) is determined from the Fourier inversion (38) (i.e. by taking the contour of integration as the imaginary axis in s-space when inverting the Laplace transform; cf. Theorem 4.2.2(ii)). The transient response

124

STATIONARY RULES AND DIRECT OPTIMISATION

g( r) for r > 0 (r < 0) is made up of contributions from the poles ofthe integrand corresponding to zeros of l~(s) I in the left (right) half of the complex plane. It is the separation of these two sets of singularities which corresponds to the canonical factorisation of ~- The equivalent of a canonical factorisation cannot be avoided, and remains to be faced in the apparently simple inversion (37). The second point is that the filter view of relation (36) implies that it holds for all time r, whereas we suppose that it holds only for r ~ t, where t is the 'now' of the optimiser. That is, in order to develop the optimal control rule in closed-loop form, we assume that optimisation of control from time t does not presume optimality of control before time t. This is the reason why the set of optimality 1 conditions (36) is inverted only partially, to ~+(.qJ)~ = ~-(Et)- ( rather than to ~ = ~(Et)- 1 (. A control rule deduced from the second relation would be openloop, even if optimal. However, if this complete filter inversion is not useful for determining policy, it is useful for determining some aspects of performance. Consider the particular and d are both identically zero and the command signal r is case in which prescribed in all components, so that, as in Sections 4.1 and 4.10, we consider the response of tracking error x- r to command signal r. In this case the system (36) reduces to

rr

where the vector ~ on the left can be regarded as system error, the quantity that one would wish to tend to zero with the passage of time. If the optimal control has been used at all times and

r(r)

= LWjeSfT

(39)

j

then the contribution to system error ofthejth term is

We can thus assert Theorem 6.6.1 The process command signal (39) will ultimately be followed with zero error by the optimally controlled process if A(sj)Wj = 0 for all j such that Re(sj) ~ 0.

This of course is simply a restatement of the conclusion already reached ~n Section 2.9: that there will be zero offset in the limit if all unstable modes of the command signal satisfy the uncontrolled plant equation.

6 THE CONCLUSIONS IN TRANSFER FUNCTION TERMS

125

The conclusion extends to the case when only certain functions of x are subject to command. Suppose that only the linear combination Dxc is prescribed, and has the form

Dxc(T) =

L

(40)

v1es1r.

j

Theorem 6.6.2 The partial process command signal (40) will ultimately be followed with zero error by the optimally controlled process iffor each j such that Re(s1) ~ 0 one can find a vector w1 such that A(s1 )w1 = 0 Dw1 = v1 . 1

This corresponds to a completion of r to a form (39) which is consistent with prescription (40) and with zero limiting tracking error. Let us calculate what the asymptotic offsets would be in the case of constant ::xc, uc and zero d. The asymptotic values of x and u are then determined by

[

R S

STQ AJ"l [x] BJ u

Ao

Bo

0

=

[li

Sr ++ STucl Q~ .

>.

0

Here the matrix in the left-hand member is just <)(0), so that Ao and Bo are the absolute terms in A(s) and B(s). Formulae condense considerably if we adopt a notation which implies a viewpoint which we shall increasingly see as natural. Suppose that we lump process and control variables into a single vector, :r =

[:]

1

the system vector. In

terms of this we can write the last equation system as

(41) Now, if <)(s) is to have a canonical factorisation then it cannot be singular on the imaginary axis, so that the matrix in (41), identifiable with <)(0) is certainly nonsingular. 1

Theorem 6.6.3

The equilibrium rate ofoffset cost under optimal control is ~ (:r _ ;rc)T9l(:r _ r) = ~;rcT(9l- 9lP9l);rc

(42)

where Pis the top left matrix in the partitioned inverse ofthe matrix ofsystem (41). /f9l is non-singular then this reduces to

We leave verification to the reader. Expression (42) involves :rc only in the combinations 9lr and rT 9lr as it must if only certain components of xc have 1

126

STATIONARY RULES AND DIRECT OPTIMISATION

been specified and deviations from them costed I( on the other hand, 9l is nonsingular, then all components of ~ have been specified. We see that expression (44) is zero if ~0 ~ = 0, which is the familiar statement that the constant values X = r and U = tf should satisfy the plant equation.

7 DIRECT TRAJECTORY OPTIMISATION FOR THE INPUTOUTPUT FORMULATION: NOTFS OF CAUTION A higher-order model such as (33) can be regarded as a reduction of a statestructured model, but it is still a full dynamic model It is natural to ask whether these Lagrangian techniques of trajectory optimisation still apply for the ultimate reduction; the case when the plant is simply specified by its transfer function, so that one has an input-output specification rather than a dynamic specification. In fact, this is not a good idea. An optimisation by any method of the problem in this form is in fact fraught with subtle complications, and these reveal themselves in a Lagrangian attack. One basic reason is that the significant physical variable of the problem, the process variable, is no longer directly observable in general, and this lack of direct observation has consequences which we can list as follows. (i) The calculation in effect tries to achieve two objects at the same time: to determine what the optimal control rule would have been had the process variable been known and to reconstruct (estimate) this variable optimally from the observations. The whole analysis is much simplified, both in concept and in execu~on, if one reverts to a process description, when these two aspects separate clearly and explicitly. (ii) In the input-output description the plant equation has already been 'solved', but the point of a Lagrangian approach is to avoid premature elimination of variables (which is what solution amounts to), and certainly to avoid solution of an equation which, in isolation, is unstable. Plant instabilities greatly complicate the direct optimisation oftrajectory if plant has been specified in input-output -form. (iii) The optimal control rule deduced under an input-output specification can well be open-loop, and so sensitive to perturbation or mis-specification. (iv) The factorisation algorithm based upon policy improvement is straightforward only for polynomial transfer functions, i.e. for systems expressed by finite-order dynamics. Some analysis will bear out these points. The analysis will propel us somewhat ahead of ourselves, in that it will indicate the necessity for the statistical treatment of unobservables, but that is no bad thing. We adopt the formulation of Section 4.10:

(44)

127

7 DIRECT TRAJECTORY OPTIMISATION

y

= Gz1( + Gzzu.

(45)

Here (subsumes all exogenous signals; i.e. inputs to the system from outside such as command signal, process noise and observation noise. The signal y is the observable plant output and 6. the deviation which is to be minimised in the sense that one minimises a total cost! Lr 6.J9t6.r. (We revert to discrete time for simplicity.) Consider then the minimisation of the Lagrangian form 0 = l:)!6.T9t6. + J.?(y- Gz,(- Gzzu)]r +end terms

(46)

r

with expression (44) substituted for 6.. Here the multiplier at time r has been denoted by J1.r rather than >-r· This is because relation (45) is an observation relation rather than a plant equation-the plant equation has been 'lost' jn the input-output specification. The 'end terms' comprise terminal cost and also an initial term reflecting beliefs about initial values. Expression (46) is a form in the variables (, u, y, and f.L· Let us suppose that at time t observations Yr are available for r < t. We suppose that the only information on the system input ( is that derived from the observations plus a statistical characterisation which is incorporated in the penalty function! 6..T9t6. (We shall see from Chapter 12 and even more clearly from Chapter 16 that it can be so incorporated.) The form (46) then has to be extremised with respect to the values of all variables which are unobserved and all decisions which are unmade at timet. For r ~ tone can extremise out Yr. leading to the conclusion that J1.r = 0 and so to the equation system

(r

~

t).

(47)

derived by extremisation with respect to (r and ur- Here the superscript ( i) indicates that all variables in the vector are optimised values based on information at timet. So, (~') is the effective estimate of (T based on information at timet. For r < t the variables y and u are known and so cannot be extremised ouL One then has the equation system

[ G~~~u

-~21 ][;

r)

= [

~/ -G~:Gl2 ][ ~

L

(r < t).

(48)

derived by extremisation with respect to (T and JJ.r. This is essentially concerned with the estimation of past (; the equation system (47) with the combined optimisation of future u and prediction of future (. We say 'essentially' because the two systems are coupled by the occurrence of common variables around the present timet.

128

STATIONARY RULES AND DIRECT OPTIMISATION

One presumes that a canonical factorisation of the operator for each equation system will achieve the same kind of reduction that we achieved in the passage from (26) to (27). However, this presumption is in general mistaken. Consider, for simplicity, the situation in which the whole course of ( is in fact known. The only optimality condition is then the stationarity condition for present and future u:

('r

~

t).

(49)

Let us specialise even further: to the simplest version of the regulation problem of Section 2.4, supposing that all variables are scalar, and that Sis zero. There is no system input, so (is absent, and the components of A (i.e. the signals which enter the cost function AT9lA = R:x? + Qu 2 ) are u itself and

x = (1- A.?T)- 1B.?Tu. This last equation expresses plant in input-output form, and minimisation of l::,.(li + Qu2 ) with x thus expressed leads to the version of (49) for this case:

If we write the left-hand member as lfl(Y)u,. then IJl can be expressed

lfl(z)

1

(X

(1 - fz)(1 - rz- ) (1- Az)(1- Az-1)

where r = A + BK is the optimal gain factor determined by (2.36). Now, the recursion in u which corresponds to the optimal control rule, u,. = Kx,. = K(l- A.?T)- 1B.?Tun is (50) But the operator on u in this equation is the canonical factor of lfl(Y) only if !AI< 1, i.e. if the plant is stable. It cannot be assumed, then, that division of relation (49) by the 'future' canonical factor of the operator on the left-hand side will reduce it to the correct forward recursion for u. Recursion (50) would in any case be impracticable in the case of unstable plant. It is then itself an unstable recursion, which would be upset by the least rounding error in specification of past u, let alone by misspecification of the model; see Exercise 1. That the cases of stable and unstable plant differ qualitatively is well recognised in the literature (see e.g. Zames, 1981). It is sometimes coped with decomposing the controller into an arbitrary stabiliser followed by an optimising compensator. We shall return to direct trajectory optimisation in Chapters 18-21, but with the supposition that the plant is specified by explicit dynamic equations (possibly stochastic and not necessarily state-structured) rather than by an input-output

7 DIRECT TRAJECTORY OPTIMISATION

129

relation. It will turn out that the canonical factorisation treatment is then, as in Section 5, direct and problem-free. Exercises and comments (1) Note that a control rule deduced by any argument from (49) (such as the correct rule (50)) would in any case be an open-loop rule, and so not robust to perturbation or misspecification. The supposition, explicit or implicit, that ( is known and that the model assumed is correct, leads to the conclusion that feedback is unnecessary. The control rule properly deduced from (47) will have some element of feedback, because of the· assumption that ( is unknown and must be estimated-indeed, re-estimated at each stage. This in turn implies a robuster rule; see Chapter 17.

Notes on the literature Canonical factorisations of operators have been used in many contexts, notably that of Wiener-Kolmogorov prediction theory (see Section 13.4). However, their use for trajectory optimisation in the precise sense of Sections 3 and 5 is rather different; the first work on this line known to the author is that of Hagander (1973). The author (Whittle, 1983b) used the technique for the optimisation of both estimation and stochastic control, assuming state structure but allowing the feature of risk-sensitivity which we shall consider in Chapter 16. This work was extended to dynamics of general order in Whittle and Kuhn (1986) and further developed in Whittle (1990a). The approach leads one inevitably to a 'system' approach in which (x, u) is seen as a joint variable constrained by the plant equation. Willems (1991, 1992, 1993) comes to the same view when he develops his 'behavioural' approach. In our notation for the continuous-time case, he asserts that the system is controllable if and only ifm:(s) has the same rank for all complex s, and that a certain reduction of CI.> possesses a canonical factorisation if it is strictly positive on the imaginary axis. The second assertion would certainly need modification in our Lagrangian formalism, when our (mtentionally) unreduced time-integral is not positivedefinite as a quadratic form, but of mixed character. This mixed character is intrinsic and, in the later risk-sensitive context, significant.

CHAPTER 7

The Pontryagin Maximum Principle The Pontryagin maximum principle states a method of direct trajectory optimisation which we have in fact already seen in two contexts and by two approaches. These are the approaches which yield both a rapid development of the formalism and a feeling for its meaning. In Section 2.10 we considered perturbations from an optimal trajectory and, in the continuous-time case, deduced the first-order relations (2.76) and (2.77) by dynamic programming methods. These can be regarded as necessary conditions for optimality of the trajectory, at least when the derivatives invoked exist. Then, in the last chapter, we found Lagrangian methods efficacious for a direct optimisation of trajectory. This was for the LQ case, but perhaps generalises. The maximum principle was first seen as replacing the classical calculus of variations, which it superseded because it could deal with cases for which the optimal control turned out to be discontinuous. One finds such discontinuous controls in the 'bang-bang' control of the Bush problem (Section 6), rocket thrust programming (Section 7) and even our fishing problem (Section 2.7). The principle also revealed an attractive formalism, a Hamiltonian structure analogous to that of classical mechanics. The reason for this is that a Hamiltonian structure appears whenever an incomplete specification of dynamics is supplemented by an extremal principle. In the control case the dynamical specification is incomplete because the plant equation determines process dynamics for given control values, but gives no guide for the determination of these control values. The guide is supplied by the extremal principle of costminimisation. A fully rigorous derivation of the principle would be, at the present, both lengthy and unattractive. We shall rather adopt a heuristic approach which reveals the structure very quickly and which provides a machinery for the quick generation, in any particular case, of the assertions which one may expect to hold. There is one interesting point, however. Versions of the principle hold in both discrete and continuous time, but in fact under milder conditions in the continuous-time case, for reasons explained in Section 1. Because of this and because of the greater importance of the continuous-time case in the control context we shall devote most of the treatment to this case.

132

THE PONTRYAGIN MAXIMUM PRINCIPLE

1 THE PRINCIPLE AS A DIRECT LAGRANGIAN OPTIMISATION We saw in the last chapter how effective direct trajectory optimisation was for the LQ model, especially in the infinite-horizon limit. It is then natural to ask whether the same methods could not be carried over to other models. That is, consider a discrete-time model with the state-structure implied by the plant equation (2.2) and the cost function (2.3) and with a vector-valued state variable. Regard the plant equation at timeT as a constraint on the {x, u} path with which can be associated a vector Lagrangian multiplier A.,.. The Lagrangian form

h-1 L)c(x,., u,., T)

+ A;(x,.- a(Xr-1, Ur-1, 7)] + Ch(xh)

(1)

r=O

should then be extremised with respect to the {x, u, A} path, for given initial conditions. We have hitherto supposed the horizon point h prescribed, with h = +oo as a limit case. It is conceivable, however, that the Lagrangian approach could be valid under other stopping rules; e.g. that h is the first time at which x first enters some set. For instance, the flight of an aircraft terminates at the moment when it comes to rest on the ground, which may be at a time and in a manner unscheduled. The formalism that one derives this way is exactly the formalism of the socalled 'discrete maximum principle', the maximum principle in a discrete-time formulation. However, the conclusions are correct under only quite restrictive conditions, because the strong form of Lagrangian methods to which we wish to appeal is valid only under such conditions. Briefly, we would like to assert that necessary and sufficient conditions for optimality of the {x, u} path are that the Lagrangian form (1) should be minimal with respect to {x, u} and maximal with respect to {A}. This assertion is certainly true under the following hypotheses: that the set of permitted paths { x, u} is convex, that c is convex in its x, u arguments, that a is linear in its x, u arguments, that the horizon is fixed and finite, plus growth conditions ensuring boundedness of optimal path and cost. These hypotheses were all satisfied for the LQ model of the last chapter, at least so long as the horizon was held finite. However, if they are weakened then one can be sure of nothing without further investigation. The continuous-time analogue of the Lagrangian form (1) would be

1h [c(x, u, T)

+ AT(x- a(x, u, T))]dT + C(x(h), h)

(2)

in the notation of Section 2.6. One is now has a continuous infinity of variables, and so would expect additional reasons for possible failure of the Lagrangian approach. Greater care is indeed necessary, but the continuous-time case presents one significant simplification. Suppose that the. control variable u is also vector-valued, but with its values restricted to a set lliJ which may very well not be

I THE PRINCIPLE AS A DIRECT LAGRANGIAN OPTIMISATION

133

convex. However, by varying u rapidly relative to the rate at which xis changing one can effectively achieve any value of u which is an average of values in d/1. That is, d/1 is effectively replaced by its convex hull, a convex set. This is the intuitive content of the so-called 'chattering lemma: This differing behaviour manifests itself in that the dynamic programming equations yield a version of the maximum principle in continuous time which can only be equalled in strength in discrete time if one makes restrictive assumptions. We shall then associate the principle almost entirely with the continuous-time case. The LQ model remains the outstanding example of a case in which the discrete-time maximum principle is valid and useful; we indicate a couple of others in the exercises. Exercises and comments

(1) Economic growth Consider the dynamic allocation problem of Section 3.4, where x 1 is the 'activity vector' at time, t, the vector of intensities at which the various activities are pursued. Thus x 1 ~ 0, and resource limitations enforce the plant equation Ax1 ~ b + Bxt-1· Suppose that utility is linear: that one wishes to cT x 1, where the kth component of cis the rate at which activity k maximise delivers 'utility'. Application of Lagrangian techniques is valid; the Lagrangian form would be

2::7

h

L[cT x 1+ ).:[(b + Bxt-1- Axt- Zt)] I= I

Here z1 ~ 0 is the margin of inequality in the plant equation; the vector of amounts of resources unused at stage t. The multiplier has the familiar price interpretation; thejth element of A1 is the effective unit price ofresourcej at time t. Maximisation with respect to z1 yields the conclusion A1 ~ 0 with equality in those components for which the corresponding component of z1 is positive. Maximisation with respect to x 1 yields c + BT At+! -AT A1 ~ 0, with equality in those components for which the corresponding component of x 1 is positive. If the system is self-sufficient in that it can maintain itself in the absence of external supplies then, at times remote from both the beginning and the end of the optimisation period, it settles on to a maximal growth path (the turnpike of the economists), for which x 1 is of the form p1x for some fixed x and some p ~ 1. The maximal growth rate p and the direction x of the optimal path are determined by

(3) so that p isjustthe maximal root of IPA -

Bl =

0.

134

THE PONTRYAGIN MAXIMUM PRINCIPLE

The von Neumann-Gale model (see e.g. Gale, 1960, 1967, 1968) generalises this in that it relaxes the assumptions of linear dependence of consumption, production and utility on x. Convexity assumptions must be retained, however, or the two extremal operations in the analogue of (3) no longer commute, and the concept of an optimal growth rate has to be qualified. (2) Optimal dosage Consider the plant equation x, = Ax1-1 +__!!! in scalar variables, where the controls u are to be chosen to minimise ~}(xr - X,:) 2 subject only to u ;;;:: 0. This would then be an LQ tracking problem but for the fact that control costing has been replaced by the positivity condition u ;;;:: 0. One might regard x as the concentration of a drug in a patient's body, attenuating at rate A in the absence of further administration, but maintained by dosage u. The sequence { u~} is the desired concentration profile. (It is convenient to make a couple of temporary changes of convention: we write u1 rather than u,_ 1 in the plant equation, and shall change the sign of A.) The Lagrangian form is

!

h

L!(x1 - x~)

2

+ A1(Ax1_, + u1 -

x 1)].

1=1

One then fmds the conditions A1 ;;;:: 0 with equality if u 1 > 0 and

(1

~

t
(4)

with end conditions Ah+I = 0 and specification of x 0• Relation (4) and the terminal condition on Ayield h-1

0 ~At=

j~

say, so that Xh+l

h-1

l:Aix,+j- l:AiX;+i =X,- x;', j=O

= Xh+l = 0. Note then that (1 - Aff)(l- Aff- 1 )X1 = u1 ;;;:: 0.

Complete the argument to show that, if we define .flld X 1 = (I - Aff) (I- Aff- 1 )X1 = (1 + A 2 )Xx- AXt-1- AX1+1, then the optimal solution corresponds to the sequence {X1} which is minimal subject to the conditions

(1

~

t ~h).

The course of X corresponding to the optimal solution will consist of 'free' segments for which .flld X 1 = 0 and X 1 ;;;:: (at which no dose is administered), interpersed by points at which X1 = x,c and u1 = .9191 X1 > 0 (at which a dose is administered). One can be more explicit about the algorithm. Denote the times t at which X, = ~ by II, t2, ... , going backwards in time. Then t 1 = h + 1, when Xh+ 1 = 0.

x;

2 THE PONTRYAGIN MAXIMUM PRINCIPLE

135

The solution in the range tr+l ~ t ~ t, has the 'free solution' form X 1 = CJiA 1+ CziA -t. The coefficients c are to be chosen so that Xr = for t = lj' ti+ I. Once t 1, t2 , ••• , t, have been determined then ti+l is determined as the smallest value of t ( ~ 1) such that the free solution agreeing with xc at t and t1 is not smaller than XC at any intermediate point. If this value oft seems to bet= 1 then indeed it is provided that the value of xo is such that d if X1 > 0, i.e. such that medication should begin immediately. If, on the other hand, d .it X1 ~ 0, then medication begins first at t 1• We have used the notation d to indicate potential generalisations of the argument. The problem has something in common with the production problem of Section 4.3. In the case A = 1 it reduces to the so-called 'monotone regression' problem, in which one tries to approximate a sequence {~}as closely as possible by a non-decreasing sequence {x 1 }. In this case X, is the smallest concave sequence which exceeds~ and its 'free segments' are straight lines.

x;

2 THE PONTRYAGIN MAXIMUM PRINCIPLE The maximum principle (henceforth abbreviated to MP) is a direct optimality condition on the path of the process. It is a calculation for a fixed initial valuex of state, whereas the DP approach is a calculation for a generic initial value. It can be regarded as both a computational and an analytic technique (and in the second case will then solve the problem for general initial value). The proof of the fact that derivatives etc. exist in the required sense is a very technical and lengthy matter, which we shall not attempt. It is much more important to have a feeling for the principle and to understand why it holds, coupled with an appreciation that caution may be necessary. We shall give a heuristic derivation based upon the dynamic programming equation, which is certainly the directest and most enlightening way to derive the conclusions which one may expect to be valid. A conjugate variable p will make its appearance. This corresponds to the multiplier vector A, the identification in fact being p = .XT, so that p is a row vector. The row notation p fits in naturally with gradient and Hamiltonian conventions; the column notation .X is better when, as in equation (6.20), we wish to write all the stationarity conditions as a single equation system. We shall refer top as either the 'conjugate variable' or the 'dual variable'. Note the conventions on derivatives listed in Appendix 1: in particular, that the vector of first derivatives of a scalar variable with respect to a column (row) vector variable is a row (column) vector. Consider first a time-invariant formulation. The state variable x is a column vector of dimension n; the control variable u may take values in a largely arbitrary set llli. We suppose plant equation x = a(x, u), instantaneous cost function c(x, u), and that the process ,.stops when x first enters a prescribed stopping set!/, when a terminal cost lk\(x) is incurred. The value function F(x) then obeys the dynamic programming equation

136

THE PONTRYAGIN MAXIMUM PRINCIPLE

inf(c + Fxa) = 0 ...

(x ¢. !/),

(5)

with the terminal condition

(x E !/).

F(x) = IK(x)

(6)

The derivative Fx may well not exist if xis close to the boundary of a forbidden region (within which F is effectively infinite) or even if it is close to the boundary of a highly penalised but avoidable region (when F will be discontinuous at the boundary). We have already seen examples of this in Exercise 2.6.2 and shall see others in Section 10. However, let us suppose for the moment that x is on a free orbit, on which any perturbation ox in position changes F only by a term Fxox + o(ox). Define the conjugate variable P = -Fx

(7)

(a row vector, to be regarded as a function of time p(t) on the path) and the Hamiltonian

H(x, u,p) = pa(x, u)- c(x, u)

(8)

(a scalar, defined at each point on the path as a function of current x, u and p).

*Theorem 7.2.1 (The Pontryagin maximum principle on a free orbit; time-invariant version) ( i) On the optimal path the variables x and p obey the equations

[= a(x, u)]

(9)

( 10) and the optima/value ofu(t) is the value ofu maximising H[x(t), u,p(t)]. (ii) The value of His identically zero on this path. *Proof Only assertions (9) and (10) need proof; the others follow from the dynamic programming equation (5) and the definition (7) of p. Assertion (9) is obviously valid. To demonstrate (10), write the dynamic programming equation in incremental form as F(x) = inf[c(x, u)bt + F(x + a(x, u)bt)] + o(ot).

( 11)

"

Differentiation with respect to x yields

-p(t) whence (10) follows.

= CxOt- p(t + ot)[l +ax ot] + o(ot) 0

2 THE PONTRYAGIN MAXIMUM PRINCIPLE

137

The fact that the principle is such an immediate consequence of the dynamic programming equation may make one wonder what has been gained. What has been gained is that, instead of having to solve the partial differential equation (5) (with its associated extremal condition on u) over the whole continuation set, one has now merely to solve the two sets of ordinary differential equations (9) and (10) (with the associated extremal condition on u) on the orbit. These conditions on the orbit are indeed those which one would obtain by a formal extremisation of the Lagrangian form (2) with respect to x, u and >.; as we leave the reader to verify. Note that the equations (9) and (10) demand only stationarity of the Lagrangian form with respect to the >.- and x-paths, whereas the condition with respect to u makes the stronger demand of maximality. It is (9) and (10) which one would regard as characterising Hamiltonian structure; they follow by extremisation of an integral J[px- H(x,p)] dtwith respect to the (:x,p) path. A substantial question, which we shall defer to the next section, is that of the terminal conditions which hold when x encounters 9. Let us first transfer the conclusions above the the time-dependent case, when a, c, 9 and II< may all be !dependent. The DPequation for F(x, t) will now be inf(c + F, + Fxa) = 0

..

(12)

outside 9, with F(x, t) = IK(x, t) for (x, t) in 9. However, we can reduce this case to a formally time-invariant case by augmenting the state variable x by the variable t.We then have the augmented variables P ~ [p Po].

(13)

where the scalar po is to be identified with - F,. However, we shall still preserve the same definition (8) of Has before, so that, as we see from (12), the relation

H=F,= -po.

(14)

holds on the optimal orbit.

Theorem 7.2.2 (The Pontryagin maximum principle on a free orbit) (i) The assertions ofTheorem 7.2.1 ( i) stillhold, but equation (10) is now augmented by the relation

Po= -H, = c,- pa 1•

( 15)

(ii) H +Po is identically zero on an optimal orbit. Suppose the system timehomogeneous in that a and c are independent oft. Then His constant on an optimal orbit.

138

THE PONTRYAGIN MAXIMUM PRINCIPLE

Proof All assertions save the last are simple translations of the assertions of Theorem 7.2.1. If a and c are independent oft then we see from (15) that Po is constant on an optimal orbit, whence the fmal assertion follows. 0 However, the essential assertions of the maximum principle are those expressed in Theorem 7.2.1 (i) which, as we see, transfer to the time-dependent case unchanged. Note that His now a function oft as well as ofx(t), u(t) andp(t). Exercises and comments

(1) As indicated above, one can expect H to be identically zero on an optimal orbit when the process is intrinsically time- invariant and the total cost F~) is welldefined The case of a scalar state variable is then particularly amenable. By eliminatingp from the two relations, that His identically zero and that it is maximal with respect to u, one derives the optimal control rule in closed-loop form. (2) Suppose that the process is time-invariant and has a well-defined average cost 'Y· The total future cost is then F(x, t) = f(x) - 1t plus an 'infinite constant' representing a cost of 'Y per unit time in perpetuity. We thus have H -p 0 =

=

F1 = -1, so that the constant value of H can be identified with the average reward rate -~. In the scalar case the optimal control rule can be determined, at least implicitly, from H + 7 = 0 and the u-maximality condition. The equilibrium point is then determined from Hx = Hp = 0; 'Y can then be evaluated as the value of-Hat this equilibrium point. 3 TERMINAL CONDITIONS The most obvious example of departure from a free orbit is at termination of the path on the stopping set. Since the path is continuous, there is the obvious matching condition: that the terminal point is the limit point along the path. However, if one may vary the path so as to choose a favourable terminal point, then there will also be optimality conditions. The rigorous statement of these terminal conditions can be quite difficult if one allows rather general stopping sets and terminal costs. We shall give only the assertions which follow readily in the most regular cases. However, even more difficult than termination is the case when parts of state space are forbidden, so constraining the path. (For example, an aircraft may be compelled to avoid obstacles, or an industrialist may not be allowed to incur debt, even temporarily.) In such a case the optimal path must presumably skirt the boundary of the forbidden region for a time before resuming a free path. The special conditions which bold on entry to and exit from such re-stricted phases are termed transversality conditions; we shall consider them in Section 11. Consider first the fully time-invariant case. One then has the terminal condition F(x) = K(x) for x in the stopping set !1'. However, one can appeal to

139

3 TERMINAL CONDITIONS

this as a continuity condition, that F(x) ---+ IK(i) as x (outside!/) approaches x (inside Sf'), only if x is the optimal termination point for some free trajectory. Obviously x must lie on the boundary 8!/ of !/, since motion is continuous. However, we shall see from the examples of the next section there may be points on this boundary which are so costly that they are not optimal terminal points for any trajectory terminating in !/. Let f/ opt denote the set of possible optimal termination points. Partial integration of the Lagrangian expression for cost minimisation throws it into the form

1 1

(pi- H) dr + IK(x) =

-1

1

(px +H) dr + px + IK(x)- p(O)x(O)

(16)

where the overbar indicates terminal values. Let a be a direction from x into f/ opt. in that there is a value x< = x + ea + o( e) which lies in f/ opt for all small enough values of the positive scalar £ If x is an optimal termination point for the trajectory under consideration then we deduce from (16) that px + IK(x) :::::; px< + IK(x<). In the limit of small Ethis yields

(p + IKx)O" ~ 0

( 17)

where the derivative Kx is evaluated at x.

*Theorem 7.3.1 Let x and p be the terminal values of state and dual variable on an optimal trajectory for a time-invariant problem. Then the terminal optimality condition (17) holds for all directions a into f/ opt/rom x. lfx is interior to f/ opt. the derivative IKx is continuous at x and a tangent plane to f/ exists at i then (17) can be strengthened to

(p + IKx)O" = 0. for all directions in the tangent plane. This can be expressed: the vector (p normal to the boundary off/ at .X.

( 18)

+ Kx) Tis

The strengthening of (17) to (18) follows because, under the conditions stated, the inequality (17) holds for all directions in the tangent plane, and in particular holds for -a if it holds for a. If we transfer these conclusions to the problem with state variable € = (x, t) then we obtain the generalisation to the time-variable case. Note that f/ will now be a set of €values and IK a function of €.

*Theorem 7.3.2 tory. Then

Let (x, t) and (p,p 0 ) be the terminal values on an optimal trajec-

(19)

140

THE PONTRYAGIN MAXIMUM PRINCIPLE

for all directions (0", r) into Y'opt from (x, T).If(x, t) is interior to 9' opt• the derivatives ofKare continuous there and a tangent plane to 9' exists there, then equality holds in (J9)forall (O", r) in this tangent plane.

4 MINIMAL TIME AND DISI'ANCE PROBLEMS It is useful to discuss a problem whose solution is intuitively obvious before tackling a substantial control problem. In this way one gets some feeling for application of the maximum principle, and also for how it should be modified when the orbit encounters a constraint. The first is an MP treatment of the problem already discussed in Exercise 2.6.3. Suppose that x is the coordinate of a particle inn-dimensional space. The particle moves at velocity v(x1 but its direction of movement can be chosen at wilL The plant equation is thus x = uv(x) where u is a unit vector to be chosen. Suppose that the cost associated with an orbit is the time taken to reach the stopping x-set 9' plus a function IK(x) of the terminal coordinate .X. That is, c = 1 and a terminal cost function IK(x) is defined on 9'. The problem is thus time-invariant with H = puv(x) - I. Since v(x) is a positive scalar then H is maximal with respect to u when u is chosen in the direction ofp T; i.e. (20)

On an optimal free orbit p obeys

jJ

= -puv:x = -!Pivx.

(21)

The termination condition (18) becomes p

+ Kx .L 89'

(22)

(where the termination values are understood). If IK is constant in 9' then this simply amounts to ul.89'; that the optimal trajectories meet the stopping set orthogonally to its boundary. · Consider now the particular case when the velocity is constant; we may as well suppose that v = 1. We see then from (21) that p is constant and so from (20) that an optimal path has a constant direction. That is, optimal free orbits are straight lines. This is no surprise; the minimisation of time now amounts to the minimisation of distance, and the minimal-distance path to 9' is a straight line which meets the boundary of 9' orthogonally. We can now visualise the optimal trajectories very easily, even in constrained cases, if we see them as a string stretched tightly between initial and terminal points. The value function would now have the form F(x)

= inf[lx- .XI + K(x)J X

4 MINIMAL TIME AND DISTANCE PROBLEMS

141

Figure 1 The minimal-distance path to a stopping set Y, meeting the boundary of fl' normally.

with x of course constrained to Y'. Consider the particular case when the stopping set consists of just two points, x and x', say, with zero terminal cost. Then F(x)

= min[lx- x'l, lx- x"l],

and one simply moves in a straight line to the nearer terminal point. The interest of the example is that Fx does not exist at either of the possible terminal points or at points equidistant from x and x'. This does not matter at all; the relevant directional derivative always exists in directions in which it would be optimal to move. The loci of discontinuity in Fx correspond to break-points in optimal control; e.g. points at which the optimal direction of movement changes discontinuously with x. A second example illustrates the effect of a discontinuity in IK(x); the type of behaviour against which the conditions of the last section were intended to guard. Consider a two-dimensional example, x having components x 1 and x 2 , with stopping set x2 = 0 and terminal cost equal to unity for negative x1 and zero for non-negative x 1. That is, the stopping set is the x1-axis, and the positive half-axis is preferred to the negative half-axis. The optimal path is a straight line to the nearest point in one of the two half-axes; we leave the reader to show that

Here the three regions fl:; are marked in Figure 2; their boundaries are determined by continuity of F(x). So, in fi 1(which is just Xi ~ 0) one moves straight to the nearest point on the Xiaxis. In fi 2 one still aims for the nearest point on the non-negative x1-axis, which is the origin. However, if x 1 is sufficiently large and negative for a given value of x 2 (i.e. in fi 3) one gives up the struggle and goes for the nearest termination point, despite the fact that this will now be on the more heavily penalised negative axis.

142

THE PONTRYAGIN MAXIMUM PRINCIPLE

Figure 2 Optimal pathsfor termination on the x 1 axis when termination on the negative part ofthe axis carries unit penalty. Termination on -1 < x 1 < O.from a starting point offthe axis is never optimal.

Figure 3 .Minimal-distance paths to!/2 when fl' 1 is effectivelyforbidikn. Thefree parts ofthe optimal trajectory are straight lines; otherwise they hug the boundary of fl' 1·

Note that the segment -1 < x1 < 0 of !/' is never entered by an optimal trajectory. Related to this is the fact that both Fx and the action rule are discontinuous on the~ z/ ~ 3 boundary. The situation for which a part of state space is forbidden is equivalent to that for which passage into it is. so heavily penalised as to be effectively forbidden. So, consider the example of Figure 3, for which the terminations set breaks up into two parts: !/' 1 , which is so heavily penalised that it is effectively forbidden, and !/'z, within which I<= 0. Optimal paths will take the shortest route to !/'2 which avoids !/' 1; this avoidance may take the form of skirting the boundary of!/' 1 for a time, as we see from the diagram.

S SOME MISCELLANEOUS PROBLEMS The maximum principle has found extensive application in economics. For what is probably the simplest example, consider the situation of a monopolist who holds an amount x of a non-renewable resource which he releases at a time-

5 SOME MISCELLANEOUS PROBLEMS

143

variable rate u. Because he has a monopoly this release rate determines the current unit price of the commodity, p(x). He wishes to maximise the total discounted return

The plant equation is x = -u. We shall take J\ as the conjugate variable (a scalar), since we are already usingp to denote price. Discounting makes the problem time-dependent. (This time-dependence is removed in a dynamic programming approach only by a renormalisation to present value as time advancesJ The instantaneous value of u is such as to maximise the Hamiltonian H = e-a 1up(u) - J\u.

(23)

We have >. = -Hx = 0, so that J\ is constant. If J\o is the conjugate variable associated with time then J\o + H = 0. At termination we have J\ 0 + K 1 = 0. But K is identically zero, so J\ 0 is zero at termination, which implies that H = 0 at termination. This certainly holds if u =. 0 at termination, and this turns out to be the only way of satisfying the terminal condition. The initial condition is

1

00

X=

U

dt.

(24)

Consider first the case p(u) = u-"'~, which in fact makes the problem a version of the consumption problem considered in Section 2.2. Maximisation of H gjves the rule u = ke-ath, for some constant k. If t is the termination time then the condition u(t) = 0 is satisfied only for t infinite; the resource is never released completely in finite time. Condition (24) gives the evaluation k = (ax h), so that the optimal release rule is u = (ax h) e-ath. Here xis the initial stock and this is the open-loop release rule. If xis taken as current stock then the optimal rule in closed-loop form is u = axj-y. If we insert the open-loop expression for u into the reward function above we find the maximal return F(x) = (-yfatx 1 -"'~. These conclusions are consistent with those of Section 2.2, in the case when infinitehorizon limits existed. Consider now the case p(u) = (1 - u/2)+. The rate of return up(u) thus tends to zero as u ! 0; we shall see that this has as consequence that the resource is released in a finite time. The maximal instantaneous rate of return is attained for u = 1, and we certainly may assume that u < 2, since higher values yield zero rate of return. The optimal release rate is u = 1 - J\ea1• The condition u(t) = 0 then implies that J\ = e-ai, and the optimal release rule in open-loop form is u

= 1- e-a(i-t).

(25)

144

THE PONTRYAGIN MAXIMUM PRINCIPLE

Condition (24) yields the determining condition fort

x=t- (I -e-Q1)/a.

(26)

and substitution of expression (26) for u into the reward function yields

F(x)

= (1- e-Q1) 2 j(2a.).

(27)

The determinations (25) and (27) of the optimal u and the value function are only implicit; they are expressed in terms oft which is only implicitly determined as a function of x by (26). The dynamic programming equation, had we taken that route, could not have been solved more explicitly. Indeed, it would have taken some ingenuity to have spotted this implicit form of the solution.

Exercises and comments

(1) Zerme/o's problem A straight river has a current of speed c(y ), where y is the distance from the bank from which a boat is leaving. The boat then crosses the river at a constant speed v relative to the water, so that its position in downstream/cross-stream coordinates (x,y) satisfies x = v cos(u) + c(y), y = v sin u where u is the heading angle indicated in Figure 4. (i) Suppose that the boatman wishes to be carried down-stream as little as possible in crossing. Show that he should follow a beading U =COS-I ( -

c~)).

(Note the implication, we must have c(y) ~ v for ally. Otherwise the boatman could move upstream in the slack water as far as he liked) (ii) Suppose the boatman wishes to reach a given point on the opposite bank in minimal time. Show that he should follow the heading

u =cos

-1 (

pv

)

- 1 + pc(y) ,

where p is a constant chosen to make the path pass throught the target point. y

c(y}

+ X

F.,ure 4 The Zerme/o problem of&. 5.1: the optimal crossing ofa stream.

6 THE BUSH PROBLEM

145

6 THE BUSH PROBLEM This is one of the celebrated early problems of optimal control: to bring a mass to rest in a given position in minimal time, using a force of bounded magnitude. One example would be that of bringing the rollers of a rolling mill to rest in standard position in minimal time, the angular momentum of the rollers corresponding to the linear momentum of the mass. In the one-dimensional case the solution turns out to be simple: to apply maximal force first in one direction and then in the other, the order and duration being such that the mass will come to rest and to the desired position simultaneously. However, to prove the optimality of this manoeuvre, extreme and discontinuous in character, was beyond classical methods of variational calculus. Consider the one-dimensional case; let x denote the coordinate and v the velocity. Thus (x, v) is the state variable, with the plant equation

x= v,

v=

u,

(28)

where u can be interpreted as the force applied per unit mass. We suppose that lui~ M, c = 1 and that!/ consists of the single point (x, v) = (0, 0). That is, the mass is to be brought to rest at the origin in minimal time. Ifp, q are taken as the variables conjugate to x, v respectively, then H =pv+ qu- 1

(29)

q= -p

(30)

P=O,

u = M sgn (q)

(31)

If the terminal values ofp and q are denoted o and f3 then (30) implies that

p=o, in terms of time-to-go s. Since u relation H = 0 implies that

q=/3+os

=M

(32)

sgn(/3) and v = 0 at termination, the

1/31 = 1/M. Consider first the positive option, f3 = I j M. If o ~ 0 then q ~ 0 for s ~ 0, so that u = M, and backwards integration along the orbit yields v = -Ms,

Thus x, v lie on the parabolic locus

(x

~

0, v ~ 0)

This is the lower half of the switching locus drawn in Figure 5. If, on the other band, o is negative, then q changes sign at s = /3/lol =so, say, as then does u. If we follow the optimal path in reverse time then in this case it

146

THE PONTRYAGIN MAXIMUM PRINCIPLE

X

M FigureS T1ri! Bush problem The two half-parabolae through lhe origin constitute lhe switching locw; we illustrate a path which begins with maximal deceleration, then switches to maximal acceleration when it meets the locus.

leaves the switching locus at s = so and then follows the lightly-drawn parabola in Figure 5. In forward time, if one started at a point on this path then one would apply maximal deceleration u = - M and hold it until the switching locus was reached. One would then apply maximal acceleration u = M and move along the switching locus to the origin. The case j3 = 1/M leads correspondingly to the other half of the switching locus:

(x::::;; 0, v ~ 0). The maximum principle yields the relations (31) and (32) and from these we have deduced the optimal rule in closed-loop form, expressed in terms of the switching locus. If one starts off the locus then one applies maximal deceleration or acceleration depending on whether one is above or below it Once the locus is reached one applies maximal deceleration or acceleration depending upon whether one is upon the upper or the lower branch. Once the origin is reached then the force is of course removed. The control rule we have deduced is an example of what is termed 'bang-bang' control. That is, the control variable u in general takes extreme values in its permitted set ~. sometimes switching between these in a discontinuous fashion, as in this example. Small heating systems are usually run this way, with the gas flame either fully on or fully off; intermediate settings being impracticable. Fuel economy certainly requires that rocket thrust should be programmed this way: to operate in general either at full thrust (in some direction) or at zero thrust. In the Bush example the control force was not cos ted, merely limited. In the rocket case

7 ROCKET THRUST PROGRAMME OPTIMISATION

147

there is a linear costing in addition to limitation; this implies the bang-bang character of the optimal rule, as we shall see in the next section. The treatment of the Bush problem generalises to some degree to the ndimensional case. If we work from the vector generalisations of (30) and (31) then we deduce the partial characterisation

u = M (sa. + {3) T

[sa.+ !3! of the control law, for fixed a., {3 subject to 1!31 = 1/ M. As an optimal path is followed we see that the direction of the control force now in general varies all the time, with up to n reversals of direction in some component. One generates all optimal orbits by variation of a. and {3. However, to deduce the closed-loop form of the control in this way seems difficult. Somewhat simpler is the LQ version of the problem, outlined in Exercise 1. Exercises and comments (1) Consider the vector version of the Bush Rroblem with no bound on u, but with an instantaneous cost function! (L + Q!u! ), penalising respectively time taken and control energy consumed during passage to termination at x = v = 0. Suppose, to begin with, that we prescribe the termination time t. Show (and we shall treat a more general case in Section 7) that the optimal control is

2 (3x + 2vs) = - 6 (x u = -~

~

vs) --,v +2 s

(33)

and that the control cost incurred is

(34) where s = t - tis time to go. The first term in the final expression of (33) induces correction of final position (with an inference that the effective average velocity over the remaining time interval is v/2) and the second term induces correction of final velocity. One can now take account of the time-cost by choosing s to minimise Ls + F(x, v, s). This will give an (x, v)-dependent evaluation ofs and so oft, but the evaluation oft is necessarily constant along the optimal orbit.

!

7 ROCKET THRUST PROGRAMME OPTIMISATION The solution of the Bush problem of Section 5 is very much what intuition might suggest, but is nevertheless remarkable for its extreme and discontinuous character. It is just for this reason that the problem was so resistant to classical optimisation methods, such as as the calculus of variations. The maximum

148

THE PONTRYAGIN MAXIMUM PRINCIPLE

principle has been significant in that it provided a technique for just such cases. A source of significant problems since the 'forties has been the incentive to determine a rocket thrust programme which is optimal in the time/fuel consumption needed to achieve some given manoeuvre. We shall assume the rocket to be effectively a point mass, so that its state is specified by its coordinates in physical space (x), its velocity (v) and its mass (m). That is, we neglect rotation, yaw, vibration or any aspect of the rocket other than translation of its centre of mass through space and the wasting of its mass through fuel consumption. Note that x in this case is not the state vector, but simply that part of the state vector (x, v, m) which describes the Euclidean coordinates of the rocket in physical space. Suppose that the rocket jet has a backward_vector velocity k relative to the rocket (so that the absolute velocity of the material of the jet is v-k) and that the rocket is subject to external forces of vector magnitude h = h(x, v, m, t). Then the condition of conservation of linear momentum over a time interval of length 8t gives the equation (m- 8m)(v

+ 8v) + (v- k)8m =

mv + h 8t

or

mv =km+h,

(35)

the so-called rocket equation. We shall assume that the direction of the jet vector k can be freely controlled and that the rate mat which mass can be expelled in the rocket jet can also be controlled within limits. If we set

km=u, then we can regard the thrust vector u as the control vector, to be chosen freely subject to iui~M,

(36)

say. With these definitions the collective plant equation for the rocket can be written

mv = u +h.

m=

(37)

-ciui

If conjugate variables p, q, r are associated with x, v, m respectively, then equations (10) become for this case . p

qh

= -m "''

. q= - p - qh - ., m

(38)

7 ROCKET THRUST PROGRAMME OPTIMISATION

149

If we suppose that the cost function is purely terminal then the optimal control u should be such as to maximise the expression qu

--crlul m

(39)

For a given value of lui the thrust vector u will then be chosen in the direction ofqT:

u=qT~ lql, and lui will be chosen to maximise x:lul, where

That is,

lui

~ ;~t~re. {

}

accordillg as

~ ~ ~. {

(40)

The vector q is often termed the primer; its direction determines the optimal thrust direction and the magnitudes of q and r determine the optimal thrust magnitude. The paths on which lui is respectively M, 0 or intermediate in value are often called maximal thrust, null thrust and intermediate thrust arcs respectively, or simply MT, NT and IT arcs. Equations (38) simplify somewhat if the rocket is moving in a purely gravitational field, when h = m7, where 'Y is the gravitational vector. One is thus neglecting aerodynamic effects such as drag. The vector 'Y will have the form 7(x) = VJ if the field is a conservative one, V(x) being the potential energy associated with the gravitational field. In this case equations (38) reduce to

p = -q"(x,

q= -p,

and the primer obeys the equation

q = q"(x = qV;u

{41)

where Vxx is the matrix of second differentials of x. Equations (38) become particularly simple if Vmay be assumed quadratic, when Vxx is independent of x. As a very special case, consider the problem of maximising the height reached by a sounding rocket, assuming constant gravity and neglecting (implausibly!) effects such as aerodynamic drag. The problem can be taken as a onedimensional one in which x represents height measured upward from the starting point (the ground). The quantities v, p, q and r will all be scalar, and 'Y = - g, where g is the gravitational constant.

150

THE PONTRYAGIN MAXIMUM PRINCIPLE

The problem is one of maximising x at termination. Let mo denote the mass of the rocket structure, so that w = m - mo is the mass of fuel remaining. The terminal conditions holding are then p = l, q = 0, and r is non-positive or nonnegative according as the fuel reserve w is positive or zero. We find from equations (38) that p = 1, q = s (=time to go) so that (42) But u, if non-zero, is in the direction of q = s and so positive (upward-directed). Thus K: = -m- 1 and "' must decrease strictly through time. We thus see that the thrust programme must take the form of a phase of maximal thrust followed by a phase of null thrust, either or both of these phases possibly being of zero duration. Denote the terminal value of r by r0 . If r0 > 0 (so that w = 0 at termination) then it follows from (40) and (42) that thrust is zero for s < cmoro and maximal for larger s, and the condition H = 0 implies that v = 0 at termination. This is the usual case, in which an MT phase which exhausts fuel is followed by an NT phase during which the rocket coasts to its maximal height, when its velocity is zero. If r0 ~ 0 then "' > 0 at termination. There is thus no NT arc; maximal thrust is applied throughout. If r 0 = 0 then it follows again from H = 0 that v = 0 at termination. If r 0 < 0 then v < 0 at termination. These are cases in which the thrust is insufficient to lift the rocket. If initially the rocket happens to be already rising then maximal thrust is applied until the rocket is on the point of reversing, which is taken as the terminal instant. If the rocket happens to be already falling then termination is immediate. This last discussion illustrates the literal nature of the analysis. In discussing all the possible cases one comes across some which are indeed physically possible but which one would hardly envisage in practice. Exercises and comments (1) An approximate reverse of the sounding rocket problem is that of soft landing: to land a rocket on the surface of a planet with prescribed terminal velocity in such a way as to minimise fuel consumption. It may be assumed that gravitational forces are vertical and constant, that there is no atmosphere and that all motion is vertical. Note that equation (42) remains valid. Hence show that the thrust programme must consist of a phase of null thrust followed by one of maximal thrust upwards (the phases possibly being of zero duration). How is the solution affected if one also penalises the time taken?

8 PROBLEMS WHICH ARE PARTIALLY LQ LQ models can be equally well treated by dynamic programming or by the maximum principle; one treatment is in fact only a slightly disguised version of

151

8 PROBLEMS WHICH ARE PARTIALLY lQ

the other. However, there is a class of partially LQ models for which the roaximum principle quickly reveals some simple conclusions. We shall treat these at some length, since conclusions are both explicit and transfer in an interesting way to the stochastic case (see Chapter 24). Assume vector state and control variables and a linear plant equation

x=Ax+Bu.

(43)

Suppose an instantaneous cost function

c(u) = !uT Qu,

(44)

which is quadratic in u and independent of x altogether. We shall suppose that the only state costs are those incurred at termination. The analysis which follows remains valid if we allow the matrices A, Band Q to depend upon time t, but we assume them constant for simplicity. However, we shall allow termination rules which are both time-dependent and non-LQ, in that we shall assume that a terminal cost K(e) is incurred upon first entry to a stopping set of ~values [1', where is the combined state/time variable = (x, t). We assume that any constraint on the path is incorporated in the prescription of [I' and K, so that € values which are 'forbidden' belong to fl' and carry infinite penalty. The model is thus LQ except at termination. The assumption that state costs are incurred first at termination is realistic under certain circumstances. For example, imagine a missile or an aircraft which is moving through a region of space which (outside the stopping set fl') is uniform in its properties (i.e. in gravitational force and air density). Then no immediate position-dependent cost is incurred. This does not mean to say that spatial position is immaterial, however; one will certainly avoid any configuration of the craft which would take it to the wrong target or (in the case of the aircraft) lead it to crash. In other words, one will try to so maneoeuvre the craft that flight terminates favourabl~ in that the sum of control costs and terminal cost is minimal. This will be the interest of the problem: to chart out a course which is both economical and avoids hazards (e.g. mountain peaks) which would lead to premature termination. The effect of such hazards is even more interesting in the stochastic case, when even the controlled path of the craft is not completely predictable. It is then not enough to scrape past a hazard; one must allow a safe clearance. The analysis of this section has a natural stochastic analogue, which we pursue in Chapter24. The Hamiltonian for the problem is

e

e

H(x,u,p) = A.T[Ax+BuJ-!uTQu if we take p = A.T as the multiplier. It thus follows that on a free section of the optimal path (i.e. a section clear of 5")

THE PONTRYAGIN MAXIMUM PRINCIPLE

152

u = Q-!BT)..

(45)

A=-AT>..

(46)

e

Consider now the optimal passage from an initial position = (x, t) to a terminal = (x, t) by a path which we suppose to be free. We shall position correspondingly denote the terminal value of>.. by .X, and shall denote time-to-go t - t by s. It follows then from (40) and (41) that the optimal value of u is given by

e

(47) Inserting this expression for u back into the plant equation and cost function we find the expressions

x- eAsx = V(s).X,

(48)

= !_xT V(s)A

(49)

F(e, ~)

for terminal x and total cost in terms of .X, where

V(s) =los ttfrJeATrdr.

(50)

and J = BQ- 1BT, as ever. In (50) we recognise just the controllability Gramian. Solving for Afrom (48) and substituting in (47) and (49) we deduce

Theorem 7.8.1 Assume the model specified above. Then (i) The minimal cost offree passage from ~toe is F(e, e)= !(x- eA 3 x)TV(s)- 1(x- eA 8 x),

(51)

andtheopen-loopformoftheoptimal control ate= (x, t) is u = Q-1 BTeATsv(s)-l(x- eAsx). wheres = t- t. (ii) The minimal cost ofpassage from e to the stopping set [/, by an orbit which is free before termination, is

(52)

ifthis can be attained (53)

Expression (52) still gives the optimal control rule, with ~ determined by the minimisation in (53). This value of~ will be constant along an optimal path. We have in effect used the simple and immediate consequence (47) of the maximum principle to solve the dynamic programming equation. Relation (52) is indeed the closed-loop rule which one would wish, but one would never have

9 CONTROL OF THE INERTIALESS PARTICLE

153

imagined that it would imply the simple course (47) of actual control values along the optimal orbit. Solution of (43) and (47) yields the optimal orbit as

x(r) =eA.,. x + V(r)eAT(i-T) X

(t ~

T

~ l)

where x = x(t) and X is determined by (48). For validity of evaluation (53) it is necessary that this orbit should not meet the stopping set !/before timet. Should it do so, then the orbit will have to break up into more than one free section, these sections being separated by grazing encounters with !/ at which special transition conditions will hold. We shall consider some such cases by example in the following sections. 9 CONTROL OF THE INERTIALESS PARTICLE

The examples we now consider are grossly simpler than any actual practical problem, but bring out points which are important for such problems. We shall be able to generalise these to the stochastic case (see Chapter 24), where they are certainly non-trivial. Let x be a scalar, corresponding to the height of an aircraft above level ground. We shall suppose that the craft is moving with a constant horizontal velocity, which we can normalise to unity, so that time can be equated to horizontal distance travelled. We suppose that the plant equation is simply

x=u,

(.54)

i.e. that velocity equals control force applied. This would represent the dynamics of a mass moving in treacle: there are no inertial effects, and it is velocity rather than acceleration which is proportional to applied force. We shall then refer to the object being controlled as an 'inertialess particle'; inertialess for the reasons stated, and a particle because its dynamic state is supposed specified fully by its position. It is then the lamest possible example of an aircraft; it not merely shows no inertia, but also no directional effects, no angular inertia and no aerodynamic effects such as lift. We shall use the term 'aircraft' for vividness, however, and as a reminder of the physical object towards whose description we aspire by elaboration ofthe model. We have A = 0, B = l.We thus see from (45)/(46) that the optimal control value u is constant along a free section of the orbit, whence it follows from the plant equation (54) that such sections of orbit must be straight lines. We find that V(s) = Q- 1s, so that F(~, ~) = Q(x- x) 2 /2s. Suppose thatthe stopping set is the solid, level earth, so that the region of free movement is x > 0 and the effective stopping set is the surface of the ground, x = 0. The terminal cost can then be specified as a function IK( t) of time (i.e. distance along the ground). The expressions (53) and (52) for the value function and the closed-loop optimal control rule then become

154

THE PONTRYAGIN MAXIMUM PRINCIPLE

F(x, t)

= ~~~[~ + IK(t + s)].

(55)

u = -x/s.

(56)

Here the time-to-go s must be determined from the minimisation in (55~ which determines the optimal landing-point 1 = t + s. The rule (56) is indeed consistent with a constant rate of descent along the straight-line path joining (x, t) and (0, t). However, suppose there is a sharp mountain between the starting point and the desired terminal point 1 determined above, sufficiently high that the path determined above will not clear it That is, if the peak occurs at coordinate t 1 and has height h then we require that x(tl) >h. If the optimal straight-line path determined above does not satisfy this then the path must divide into two straight-line segments as illustrated in Figure 6. The total cost of this compound path is

(57) where Fis the 'free path' value function determined in (55). It is more realistic to regard a crash on the mountain as carrying a high cost, K,, say, rather than prohobited. In the stochastic case this is the view that one must necessarily take, because then the crash outcome always has positive probability. If one resigns and chooses to crash then there will be no control cost at all and the total cost incurred will be just K 1• One will then choose the crash option if one is in a position (x, t) for which expression (57) exceeds K 1, i.e. for which

x
(58)

where dis the constant (2/Q)[K, - F(h, t 1)]. For K1 large this can occur only if the craft is much closer to the mountain than it is to the peak; see figure 6.

X

t Figure 6 The straight-line segments ofa path surmounting a peak The curve is the switching locus (determined by equality in (58)) on which one switchesfrom endeavouring to clear the

peak to accepting a crash.

10 CONTROL OF THE INERTIAL PARTICLE

155

10 CONTROL OF THE INERTIAL PARTICLE The feature that must most urgently be added to the model above is that of inertia for vertical motion. Let us take the state variable as having components x and v, where x is again height and v is vertical velocity: rate of increase of height. We are thus guilty of an inconsistency in that x denotes a component of the state vector rather than the full vector. However, continuity with the discussion above makes this desirable. We assume then the plant equation

v= u,

x =v,

(59)

and again the instantaneous cost function! Qtl. Any coefficients which might have occurred in the second equation have been normalised to unity by scalechanges in the variables. We have then e

As= [

1 s] 0 I ·

It follows then from (45)/(46) that the optimal control is linear in time, and so from the plant equation (59) that, along the optimal orbit, the height is cubic in time:

Here a and /3 are coefficients to be determined from the terminal conditions. To determine the control rule in closed-loop form, we note first the evaluation

V(s) = Q-

1

{

[~j; ~

2

].

Expressions (51) and (52) for minimal cost of passage from and optimal control at ~ then become

F(~, ~)

= (6Q/;)[i- x-! (v + v)s] 2 + (Q/2s)(v- v) 2, u = (2j?)[3(i- x)- 2vs- vs].

(60) (61)

If one is considering passage to f/ rather than to a prescribed ~ then ~ is again determined by the minimisation in (53). The control rule (61) is in the closed-loop form that one would wish, but indeed one would never have guessed from this that the optimal control in fact varies linearly with time. These evaluations are of course valid only if passage is free. To consider what happens in other cases, suppose we indeed consider an optimal 'landing' in that ~ = (i, v, t) is prescribed as (0, w, t). That is, one is required to land at timet (corresponding to prescription of a position on the ground) with vertical velocity w (necessarily non-positive). Then a cubic path satisfying initial and terminal conditions (and so the unique optimal free path) could take any of the forms

156

THE PONTRYAGIN MAXIMUM PRINCIPLE

(c)

(d)

figure 7 The various possibilities for the free trajectory of the optimally controlled inertial particle.

(i}-(iv) of Figure 7. Cases (i) and (ii) are acceptable, because the orbit meets the stopping set (the ground) first at the desired terminal point. In cases (iii) and (iv) the aircraft begins in so violent a dive that it overshoots the plane x = 0 before it finally approaches the desired terminal point in the prescribed fashion. But such an 'overshoot' is a crash-a premature encounter with 9'-and is effectively forbidden if the penalty is set high enough. Note that case (iv) could not occur if w < 0; it is possible only if w = 0, and can be regarded as a limit version of case (iii). In either of these cases the actual optimal orbit must break into two parts, as illustrated in Figure 8. First the pilot makes strenuous efforts to come out of the dive, which he does at time t1, say.

figure 8 The case in which the optimal path to a desired terminal configuration at t breaks into two parts: a grazing escapefrom crash at tl/ollowed by afree trajectory to termination.

10 CONTROL OF THE INERTIAL PARTICLE

157

Economy of control dictates that he only just escapes a crash, in that as he comes out of the dive (i.e. v = 0) he only just misses the ground (i.e. x = 0+). Having come out of the dive he approaches the desired terminal point ~by an optimal path which is free. The two sections of the path (those before and after ft) are indeed distinct in that they are distinct cubic curves. We have to work out how the optimal grazing point t 1 is determined, and the transition conditions obtaining there. Optimisation of t 1 means that the pilot has the desired terminal configuration~ in mind and comes out of the dive in such a way as to optimise costs after as well as before t 1• But this is to demand too much. In practice a pilot will simply concentrate on the immediate emergency, knowing that this is the priority. Consideration of what is to happen once the emergency is over is generally deferred, partly because one does not have the processing power to do anything else and partly because later events are secondary in cost terms. We shall find support for the latter assertion. Let us first consider, then, the problem of pulling out of a dive as economically as possible, without consideration of what is to happen afterwards. The cubic path must then take the form of Figure 9, touching x = 0 from above at a point t to be determined. We use the notation t because this is a terminal point as far as this phase of the operation is concerned. Theorem 7.10.1 Suppose the initial configuration is (x, v, t) with v < 0. Then the closed-loop control rule which minimises the cost ofpulling out ofthe dive is

2v2

(62)

u=3x·

This pulls thecraftoutofthediveaftera times= t- tequal to -3x/v. and at a cost of

-(2Qv3 /9x). Proof We may as well set t = 0 and t = s. We know from (60), (61) that the minimal cost of free passage from (x, v, 0) to (0, 0, s) is F = (2Q/})(3.x2

t,

+ 3xvs + ;;)

't

Figure 9 The presumedform ofan optimal crash-avoiding trajectory

(63)

158

THE PONTRYAGIN MAXIMUM PRINCIPLE

and that the closed-loop rule which achieves it is

u = -(2/s'1)(3x + 2vs).

(64)

We can optimise with respect to s. The value of s minimising expression (63) for v negative is s = -3xjv; at this value the cost F and the control u have the evaluations asserted above. We have to confirm that the path thus generated does not in fact cross x = 0 before times. This path x( r) is a cubic with x(O) = x, (0) = v, and a double zero at -3xjv. These conditions determine the cubic, and its third zero is found also to lie at -3xjv. 0

x

Note a consequence of the last observation: that -3xfv is in fact the largest value which could be chosen for s; any larger and the path would have crossed the axis before times. At this point one will switch from control rule (62) to a zero control if crash avoidance is all that is demanded. The uncontrolled path now avoids f/', if only just We can now return to the landing problem, and the question of determining whether the path to landing is free or is broken by a grazing encounter with the ground. In this latter case, we should also determine the point at which break occurs. We shall now use i to denQte the time of landing (prescribed) and shall suppose that if there is an earlier grazing then it occurs at t,.

Theorem 7.JQ2 Consider the landing problem enunciated above; sets = t - t. Then the optimalpath is afree one unless both the folwwing inequalities hold: 2

3x+ vs < 0,

(3x_+__;c..__ vs) w >_..:..._ 4xs

(65)

If these inequalities do indeed both hold then the optimal path suffers a grazing en~ounter

with the ground after a time s1 = t 1 - t determined as the unique positive

root of (3x + VS! ) 2

w2

= (s- si) 2

(66)

which is less than - 3x/ v. The first condition of (65) states that the prescribed landing time must be later than the time at which it would be optimal to pull out of the dive without consideration of the sequel. The second condition implies that, even if the frrst holds, the optimal path to the termination point will still be free if the required terminal rate of descent is large enough. That is, if one is willing to accept a crash landing at the destination!

10 CONTROL OF THE INERTIAL PARTICLE

159

Proof If the path is a free one then it has the cubic form given above. We may as well normalise the time origin by setting t = 0 and so t = s in this relation. The coefficients a and {3 in the cubic are then determined by the terminal conditions x(s) = 0, x'(s) = w. The cubic then has a root at T = s, and one finds that theremaining two roots are the roots of the quadratic

-?- 2aT+ b =

0,

(67)

where 2

a=

-ilx

s(x+vs) 2x+ (v+ w)s'

b = -::-----;----:2x + (v + w)s ·

The only case (consistent with x > 0, w < 0) in which the optimal path is not free is the case (iii) of Figure 7, so this is the one we must exclude. This will be the case in which the quadratic (67) has both roots between 0 and s. This in turn is true if and only if the quadratic expression is positive at both 0 and s, and has a turning point inside the interval at which it is negative. That is:

b < 0,

?-

2as + b

> 0,

0
c? >b.

We find that the first two of these conditions are both equivalent to 2x + (v

This last with the inequality a

+ w)s < 0.

> 0 implies that x+ vs

The condition a

(68)

<0

(69)

< sis equivalent to 3x + vs + 2ws

and the final condition, a

2

< 0,

(70)

> b, is equivalent to (3x + vs)

2

+ 4xws > 0.

(71)

The free path is non-optimal if and only if all of relations (68)-(71) hold. Relations (70) and (71) give the bounds on w 3x + vs 2s

(3x + vs) 4xs

2

- - - < - w < -'--------,---------'The upper bound in this relation exceeds the lower bound by (3x + vs)(x + vs)/ (4xs). It follows from (69) that the interval thus determined for w is empty unless 3x + vs < 0, a relation which implies (68), (69) and (70). We are thus left with the pair of conditions (65) asserted in the theorem. In the case that both these conditions are fulfilled the optimal path cannot be free, and is made up of two free segments meeting at time t 1 = t + s 1. We choose s 1 to minimise the sum of the cost incurred on the two free segments, as given by expression (60); the stationarity condition (66) emerges immediately. It follows

160

THE PONTRYAGIN MAXIMUM PRINCIPLE

Figure 10 A graphical illustration ofthe solution ofequation (66) for the optimal timing s1 of the grazing point.

from the observation after Theorem 7.10.1 that the root s 1 must be less than -3x/v. Indeed, equation (66) has a single such root as we see from Figure 10; the left- and right-hand members of (66) are respectively decreasing and increasing, as functions of s,, in the interval 0 ~ s 1 ~ - 3xjv. 0 Indeed, we can determine s 1 explicitly. In taking square roots in (66) we must take the negative option on one side, because 3x + vs 1 is positive whereas w is negative. The appropriate root of the resulting quadratic in s 1 is (3x- vs)

s, =

+ V(3x + vs) 2 2(w~v)

12xws '

at least if w - v > 0, which we may expect. This approaches - 3x/vas w tends to zero. 11 AVOIDANCE OF THE STOPPING SEf: A GENERAL RESULT

The conclusions of Theorem 7.10.1 can be generalised, with implications for the stochastic case. We consider the general model of Section 9 and envisage a situation in which the initial conditions are such that the uncontrolled path would meet the stopping set !/'; one's only wish is to avoid this encounter in the most economical fashion. Presumably the optimal avoiding path will graze !I' and then continue in a zero-cost fashion (i.e. subsequently avoid !I' without further control). We shall speak of this grazing point as the 'termination' point, since it indeed marks the end of the controlled phase of the orbit. We then consider the linear system (43) with control cost (44). Suppose also that the stopping set !I' is the half-space

11 AVOIDANCE OF THE STOPPING SET: A GENERAL RESULT

161 (72)

!/'={x:aTx:::::;b}

This is not as special as it may appear; if the stopping set is one that is to be avoided rather than sought then its boundary will generally be (n- I)dimensional, and can be regarded as locally planar under regularity conditions. Let us denote the the linear function aT x of state by z. Let F(~, [)be the minimal cost of transition by a free orbit from an initial point ~ = (x, 0) to a terminal point [ = (x, t), already evaluated in (51). Then we shall find that, under certain conditions, the grazing point [of the optimal avoiding orbit is determined by minimising F( ~. [) with respect to the free components of [at termination and maximising it (at least locally) with respect to t. That is, the grazing point is, as far as timing goes, the most expensive point on the surface of !/ on which to terminate from a free orbit. The principal condition required is that aTB = 0, implying that the component z is not directly affected by the control. This is equivalent to aTJ = 0. Certainly, if one is to terminate at a given timet then the value of z is prescribed as b, but one should then optimise over all other components of x. Let the value of F(~, [) thus minimised be denoted G(~, t). The assertion is then that the optimal value oft is that which maximises G( ~' t). We need a preparatory lemma. Lemma 7.11.1 Add to the prescriptions (43), (44) and ( 72) ofplant equation, cost function and stopping set the conditions that the process be controllable and that aTB = O.Then

(73) Proof Optimisation ofF(~,[) with respect to the free components of x will imply that the Xof (47) is proportional to a, and so can be written Oa for some scalar e. The values of() and tare related by the termination condition z = b, which, in virtue of (48), we can write as

aT e-41x

+ ()aT V(t)a = b.

(74)

We further see from (49) that G(~, t)

= !fPaT V(t)a.

(75)

Let us write V(t) and its derivative with respect to t simply as V and V. Controllability then implies that aTVa > 0. By replacing T by s- T in the integrand of (50) we see that we can write Vas

(76)

V=J+AV+ VAT.

Differentiating (74) with respect tot we fmd that (aTVa)(d()jdt) +aT AeAia + OaTVa

= 0,

(77)

162

THE PONTRYAGIN MAXIMUM PRINCIPLE

so that

aG = at

te aTVa + OaTVa(dOjdt) = -OaTAeAix- !OlaTVa. 2

Finally, we have :Z = aT(Ax +1~) =aTAx= aTAeAix +eaT AVa= aT AeAix+!OaTVa. The second equality follows because aT J = 0 and the fourth by appeal to (76). We thus deduce from the last two relations that aGI ar = -Oz. Inserting the evaluation of 0 implied by (74) we deduce (73).

Theorem 7.11.2 The assumptions of Theorem 7.11.1 imply that the grazing point ~ ofthe optimal!/-avoiding orbit is determined by first minimising F (~, ~) with respect to x subject to aTx = band then maximising it, at least locally. with respect tot. Proof As indicated above, optimality will require the restricted i.-optimisation; we have then to show that the optimal t maximises G(~, !). At any value t for which the controlled orbit crosses z = 0 the uncontrolled orbit will lie below z = 0, so that aT eAi x - b < 0. If, on the controlled orbit, z decreases through zero at timet, then one will increase tin an attempt to find an orbit which does not enter !/.We see from (73) that G(x, t) will then increase. Correspondingly, if z crosses zero from below then one will decrease t and G(x, t) will again increase. If the two original controlled orbits are such that the t values ultimately coincide under this exercise, then i = 0 at the common value, so that the orbit grazes !/ locally, and G(x, t) is locally maximal with respect to t. The true grazing point (i.e. that for which the orbit meets !/ at no other point) will be found by repeated 0 elimination of crossings in this fashion. One conjectures that G(x, t) is in fact globally maximal with respect tot at the grazing point, but this bas yet to be demonstrated. We leave the reader to confirm that this criterion yields the known grazing point t = -3xjv for the crash avoidance problem of Theorem 7.10.1, for which the condition aT B = 0 was indeed satisfied. We should now determine the optimal !/-avoiding control explicitly.

Theorem 7.11.3

Define

D. = D.(x, s) = b - aTeAs x,

a = a(s) =

J

aT V(s)a.

Then, under the conditions of Theorem 7.11.1, the optimal!/-avoiding control at xis (78)

163

12 CONSTRAINED PATHS: TRANSVERSALITY CONDITIONS

where s is given the value s which maximises (tl. I a) 2 • With this understanding tl. I u1 is constant along the optimal orbit (before the grazing point) ands is the time remaining until grazing. Proof Let us set ~

= (x, 0), ~ = (x, s) so that x is the value of state when a time

s remains before termination. The cost of passage along the optimal free orbit from~to ~is

(79) where V = V(s) is given by (50) and 6 = x- eAsx. The optimal control at time t = 0 for prescribed ~is (80) The quantity v- 16 is the terminal value of .A and is consequently invariant along the optimal orbit. That is, if one considers it as a function of initial value then its value is the same for any initial point chosen on that orbit Specialising now to the f/ -avoiding problem, we know from the previous theorem that we determine the optimal grazing point by minimising F ( with respect to x subject to z = b and then maximising it with respect to s. The first minimisation yields

e

e, ()

(81) so the values of s at the optimal grazing point is that which maximises (tl.l a) 2 • Expression (78) for the optimal control now follows from (80) and (81~ The identification of D.lil with v- 16 (with s determined in terms of current x) demonstrates its invariance along the optimal orbit (before grazing). D

12 CONSTRAINED PATHS: TRANSVERSALITY CONDITIONS We should now consider the transition rules which hold when a free orbit enters or emerges from a part of state space in which the orbit is constrained. We shall see that conclusions and argument are very similar to those which we deduced for termination in Section 3. Consider first of all the time-invariant case. Suppose that an optimal path which begins freely meets a set ~ in state-space which is forbidden. We shall assume that ~ is open, so that the path can traverse the boundary a~ of~ for a while, during which time the path is of course constrained. One can ask: what conditions hold at the optimal points of entry to and exit from 8ff? Let x be an entry point and p and p' be the values of the conjugate variable immediately before and after entry. Just as for the treatment of the terminal problem in Section 3, we can partially integrate the Lagrangian expression for minimal total cost up to the transition point and so deduce an expression whose primary dependence upon the transition value x occurs in a term (p- p')x.

THE PONTRYAGIN MAXIMUM PRINCIPLE

164

Suppose we can vary x to x + eO'+ o( e), a neighbouring point in 8/F. Then, by the same argument as that of Theorem 7.3.1 we deduce that (p - p') f7 is zero if x is an optimal transition value. That is, the linear function pO' of the conjugate variable is continuous at an optimal transition point for all directions f7 tangential to the surface of :F at x. Otherwise expressed, the vector (p - p') T is normal to the surface of :F at x. We deduce the same conclusion for optimal exit points by appeal to a timereversed version of the problem. Transferring these results to the time-varying problem by the usual device of taking an augmented state variable~= (x, t) we thus deduce *Theorem 712.1 Let :F be an open set in (x, t) space which is forbidden. Let (x, t) be an optimal transition point (for either entry to or exit from 8/F) and (p,po) and p',p~) the values of the augmented conjugate variable immediately before and after transition. Then

(82) fora// directions (0', r) tangential to 8/F at (x, t). In particular, if t can be varied independently of x then the Hamiltonian H is continuous at the transition. Proof The first assertion follows from the argument before the theorem, as indicated. If we can vary tin both directions for fixed x at transition then (82) implies that po is continuous at transition. But we have p 0 + H = 0 on both sides of the transition, so the implication is that His continuous at the transition. 0

One can also develop conclusions concerning the form of the optimal path during the phase when it lies in 8/F. However, we shall content ourselves with what can be gained from discussion of a simple example in the next section. An example we have already considered by a direct discussion of costs is the steering of the inertial particle in Section 10. For this the state variable was (x, v) and :F was x < 0. The boundary 8/F is then x = 0, but on this we must also require that v = 0, or the condition x ~ 0 would be violated in either the immediate past or the immediate future. Suppose that we start from (x, v) at t =:: 0 and reach x = 0 first at time t (when necessarily v = 0 as well). Suppose p, q are the variables conjugate to x, v, so that the Hamiltonian is H = pv + qu - Qu 2 /2, and so equal to pv + q2 j2Q when u is chosen optimally. Continuity of Hat a transition point, when vis necessarily zero, thus amounts to continuity of q2• Let us confirm that this continuity is consistent with the previously derived condition (66). lfp and q denote the values of the conjugate variables at t 1 - , just before transition, then integration of equation (46) leads to p( r) = p, u( r) = -Q- 1 (qs+p~j2), x(r) = Q- 1 (q~/2+ps 3 j6), T. The values of p and q are determined by the prescribed initial

Q- 1q(r) = Q- 1(q+ps), v(r)

where s =

t1 -

=::

13 REGULATION OF A RESERVOIR

165

values x and v. We find q proportional to (3x + vt) / tl, and one can find a similar expression for q immediately after transition in terms of the values 0 and w of terminal height and velocity. Assertion of the continuity of q2 at transition thus leads exactly to equation (66~

13 REGULATION OF A RESERVOIR This is a problem which the author, among others, has discussed as 'regulation of a dam: However, purists are correct in their demur that the term 'dam' can refer only to the retaining wall, and that the object one wishes to control, the mass of water, is more properly referred to as the 'reservoir: . Let x denote the amount of water in the reservoir, and suppose that it obeys the plant equation = v- u, where vis the inflow rate (a function of time known in advance) and u is the draw-off rate (a quantity at the disposition of the controller~ One wishes to maximise a criterion with instantaneous reward rate g(u), where g is concave and monotonic increasing. This concavity will (by Jensen's inequality) discourage variability in u. One also has the natural constraint u ~ 0. The state variable x enters the analysis by the constraint 0 ~ x ~ C, where C is the capacity of the reservoir. We shall describe the situations in which x = C, x = 0 and 0 < x < Casfull empty and intermediate phases respectively. One would of course wish to extend the analysis to the case for which v (which depends on future rainfall, for example) is imperfectly predictable, and so supposed stochastic. This can be achieved for LQ versions of the model (see Section 2.9) but is difficult if one retains the hard constraints on x and u and a non-quadratic reward rate g( u). We can start from minimisation of the Lagrangian form J[-g(u) + p(xv + u)] dr, so that the Hamiltonian is H(x, u,p) = g(u) + p(v- u). A price interpretation would indeed characterise p as an effective current price for water. We then deduce the following conclusions.

x

Theorem 7.13.1 An optimal draw-offprogramme shows the following features. (i) The value of u is the non-negative value maximising g(u) - pu, which then increases with decreasing p. (ii) The value ofu is constant in any one intermediate phase. (iii) The value ofp is decreasing Qncreasing) and so the value ofv = u is decreasing Qncreasing) in an empty (full) phase. (iv) The value ofu is continuous at transition points. Proof Assertion (i) follows immediately from the form of the Hamiltonian and the nature of g. Assertions (ii) and (iii) follow from extremisation of the Lagrangian form with respect to x. In an intermediate phase x can be perturbed either way, and one deduces that jJ = 0 (the free orbit condition for this particular case). Hence p, and so u, is constant in such a phase. In an empty phase perturba-

166

THE PONTRYAGIN MAXIMUM PRINCIPLE

tions of x can only be non-negative, and so one can deduce only that P ~ 0. Thus p is decreasing, and so u is increasing. Since u and v are necessarily equal, if x is being held constant, then v must also be increasing. The analogous assertions for a full phase follow in the same way; perturbations can then only be nonpositive. The fmal assertion follows from continuity of Hat transition points. With u set equal to its optimal value H becomes a monotonic, continuous function of p. Continuity of Hat transition points then implies continuity ofp, and so of u. 0 The example is interesting for its appeal to transversality conditions, but also because there is some discussion of optimal behaviour during the empty and full phases (which constitute the boundary 8/F of the forbidden region !F: the union of x < 0 and x > C). 'frivially, one must have u = v in these phases. However, one should not regard this as the equation determining u. In the case x = 0 (say) one is always free to take a smaller value of u (and so to let water accumulate and so to move into an intermediate phase). The optimal draw-off rate continues to be determined as the value extremising g( u) - pu; it is the development ofp which is constrained by the condition x = 0. Although the rule u = v is trivial if one is determined not to leave the empty phase, the conclusion that v must be increasing during such a phase (for optimality) is non-trivial.

Notes Pontryagin is indeed the originator of the principle which bears his name, and whose theory and application has been so developed by himself and others. It is notable that he held the dynamic programming principle in great scorn; M.H.A. Davis describes him memorably as holding it up 'like a dead rat by its tail' in the preface to Pontryagin et al. (1962). This was because of the occasional nonexistence of the derivative Fx in the simplest of cases. However, as we have seen, it is a rat which alive, ingenious, direct, and able to squeeze through where authorities say it cannot The material of Section 11 is believed to be new.

PART 2

Stochastic Models

CHAPTER 8

Stochastic Dynamic Programming A difficulty which must be faced is that of incompleteness of information. That is, one may simply not have all the information needed to make an optimal decision, and which we have hitherto supposed available. For example, it may be impossible or impracticable to observe all aspects of the process variable-the workings of even a moderate-sized plant, or of the patient under anaesthesia which we instanced in Section 5.2, are far too complex. This might matter less if the plant were observable in the technical sense of Chapter 5, so that the observations available nevertheless allowed one to build up a complete picture of tlle state of affairs in the course oftime. However, there are other uncertainties which cannot be resolved in this way. Most systems will have exogenous inputs of some kind: disturbances, reference signals or time-varying parameters such as price or weather. If the future of these is imperfectly predictable, as is usually the case, then the basis for the methods we have used hitherto is lost There are two approaches which lead to a natural mathematical resolution of this situation. One is to adopt a stochastic formulation. That is, one arrives somehow at a probability model for plant and observations, so that all variables are jointly defined as random variables. The variables which are observable can then be used to make inferences on those which are not. More specifically, one chooses a policy, a control rule in terms of current observables, which minimises the expectation of some criterion based on cost. The other, quite as natural mathematically, is the minimax approach. In this one assumes that all unobservables take the worst values they can take Uudged on the optimisation criterion) consistently with the values of observables. The operation of conditional expectation is thus replaced by a conditional maximisation (of cost). The stochastic approach seems to be the one which takes account of average performance in the long run; it has the completer theory and is the one usually adopted The minimax approach corresponds to a worst-case analysis, and is frankly pessimistic. We shall consider only the stochastic approach, but shall find minimax ideas playing a role when we later develop the idea of risk-sensitivity. Lastly, there is a point which should be made to maintain perspective, even if it cannot be followed up in this volume. The larger the system (i.e. the greater the number of individual variables) then the more unrealistic becomes the picture that there is a central optimiser who uses all currently available information to make all necessary decisions. The physical flow of information and commands

170

STOCHASTIC DYNAMIC PROGRAMMING

would be excessive, as would the central processing load. This is why an economy or a biological organism is partially decentralised: some control decisions are made locally, on the basis of local information plus central commands, leaving only the major decisions to be made centrally, on aggregated information. Indeed, the more complex the system, the greater the premium on trading a loss in optimality for a gain in simplicity-and, perhaps, the greater the possibility of doing so advantageously, and of recognising the essential which is to be optimised. We use the familiar notations E(x) and E(xiy) for expectation and conditional expectation, and shall rarely make the notational distinction (which is only occasionally called for) between random variables and particular values which they may adopt. Correspondingly, P(x) and P(xiy) denote the probability (unconditional and conditional) of a particular value x, at least if x is discretevalued. However, more generally and more loosely, we also use P(x) to denote simply the probability law of x. So, the Markov property of a process {x 1} would be expressed, whatever the nature of the state space, by P(x1+1IX1) = P(xt+1lx1), where X1 is the history {xT; T ~ t}. 1 ONE-STAGE OPTIMISATION A special feature of control optimisation is that it is a multi-stage problem: one makes a sequence of decisions in time, the later decisions being in general based on more information than the earlier ones. For this very reason it is helpful to begin by considering the single-stage case, in which one only has a single decision to make. For example, suppose that the pollution level of a water supply is being monitored. One observes pollution level y in the sample taken and has then the choice of two actions u: to raise the alarm or to do nothing. In practice, of course, one might well convert this into a dynamic problem by allowing sampling to continue over a period of time until there is a more assured basis for action one way or the other. However, suppose that action must be taken on the basis of this single observation. A cost C is incurred; the costs of raising the alarm (perhaps wrongly) or of not doing so (perhaps wrongly). The magnitude of the cost will then depend upon the decision u and upon the unknown 'true state' of affairs. Let us denote the cost incurred if action u is taken by C(u1 a random variable whose distribution depends on u. One assumes a stochastic (probabilistic) model in which the value of the cost C(u) for varying u and of the observable yare jointly defined as random variables. A policy prescribes u as function u(y) of the observable u; the policy is to be chosen to minimise E[C(u(y) )]. Theorem 8.1.1 The optimal decision function u( y) is determined by choosing u as the value minimising E[C(u)iy]. Proof If a decision rule u(y) is followed then the expected cost is

1 ONE-STAGE OPTIMISATION

E[C(u(y))]

171

= E{E[C(u(y))ly]} ~ E{inf E[C(u)ly]} u

and the lower bound is attained by the rule suggested in the theorem.

0

The theorem may seem trivial, but the reader should understand its point: the reduction of a constrained minimisation problem to a free one. The initial problem is that of minimising E[C(u(y)J with respect to the jUnction u(y), so that the minimising u is constrained to be a function of y at most. This is reduced to the problem of minimising E[C( u) IY]freely with respect to the parameter u. One might regard u as a variable whose prescription affects the probability distribution of the cost C 1 just as does that of y, and so write E[C (u) IYJ rather as E[qy, u]. However, to do this is to blur a distinction between the variables y and u. The variable y is a random variable whose specification conditions the distribution of C. The variable u is not initially random, but a variable whose value can be chosen by the optimiser and which parametrises the distribution of C We discuss the point in Appendix 2, where a distinction is made by writing the expectation as E[Ciy; u], the semicolon separating parametrising variables from conditioning variables. However, while the distinction is important in some contexts, it is not in this, for reasons explained in Appendix 2. The reader may be uneasy: the formulation of Theorem 8.1.1 makes no mention of an important physical variable: the 'true state' of affairs. This would be the actual level of pollution in the pollution example. It would be this variable of which the observation y is an imperfect indicator, and which in combination with the decision u determines the cost. Suppose that the problem admits a state variable x which really does express the 'true state' of affairs, in that the cost is in fact a deterministic function C (x, u) of x and u. So, if one knew x, one would simply choose u to minimise C(x, u). However, one knows only y, which is to be regarded as an imperfect observation on x. The joint distribution of x andy is independent of u, because the values of these random variables, whether observable or not, have been realised before the decision u is taken.

Theorem 8.1.2 Suppose that the problem admits a state variable x, that C(x 1 u) is the cost jUnction and f (x 1 y) the joint density of x and y with respect to a product measure Jll (dx)JLl(dy). Then the optimal value of u is that minimising C(x, u) f(x, Y)JLI d.x.

J

Proof Let us assume for simplicity that x and y are discrete random variables with a joint distribution P(?c,y); th.e formal generalisation is then clear. In this case E[C(u)iy]

=L X

C(x 1 u)P(xly)

<XL C(x u)P(x,y) = L C(x, u)P(x)P(yJx), 1

X

X

(1)

172

STOCHASTIC DYNAMIC PROGRAMMING

where the proportionality sign indicates a factor P(y) -I, independent of u. The third of these expressions is the analogue of the integral expression asserted in thetheorern. [] We give the fourth expression in (1) because P(x) and P(yjx) are often specified on fairly distinct grounds. The conditional distribution P(yjx) of observation on state is supplied by one's statistical model of the observation process, whose mechanism may be fairly clear. The distribution P(x) constitutes the 'prior distribution' of state and its specification may be debatable; see Exercise 1. Exercises and comments (1) The so-called two-hypothesis two-action case is the simplest, but is both illuminating and useful. Consider the pollution example of the text, and suppose that serious pollution of the river, if it occurs, can only be due to a catastrophic failure at a factory upstream. There are then only two 'pollution states', that this failure has not occurred or that it has, corresponding to x equal to 0 or 1, say. Denote the prior probabilities P(x) of these by ?rx and the probability density of the observation y conditional on the value of x by fx(Y~ Suppose that there are just two actions: to raise the alarm or not. The cost of raising the alarm is Co or zero according as x is 0 or 1; the cost of not raising the alarm is zero or C1 according as xis 0 or 1. It follows then from the last fonn of the criterion that one should raise the alarm if ?ro/o (y) Co < 1rJ/i (y) C1. That is, if the likelihood ratio fi (y) /lo (y) exceeds the threshold value 7roCo/7ri C1. (2) Risk-sensitivity and hedging. The new effects that a stochastic element can bring can be demonstrated on a one-stage model. Suppose that, if one divides a sum of money x so that an amount x1 is invested in activity j, then one receives a total return of r = E1 CJXJ. That is, c1 is the rate of return 6n activity j. If the c1 are known then one maximises r by investing the whole sum in an activity for which the return rate c1 is greatest. (If there are several activities achieving this rate then the investment can be spread over them, but there is no advantage in such diversification). If the c1 are unknown, and regarded as random variables, then one maximises the expected return by investing the whole sum in an activity for which the expected rate of return E(c1) is maximal, where this is an expectation conditional on one's information at the time of the investment decision. However, suppose one chooses rather to extrernise the expected value of exp( Or), maximising or minimising according as the risk-sensitivity parameter(} is positive or negative. This criterion takes account of variability of return as well as of expectation, variability being welcome or unwelcome according as (} is positive or negative (the risk-seeking and risk-averse cases; see Chapter 16~

2 MULTI-STAGE OPTIMISATION

173

For simplicity, let us assume (rather unrealistically) that the random variables Cj are independent. Then one chooses the allocation to maximise L,1 e- 1Fj(&xj). where lj(o) = log{E[exp(ocj)]}. The functions lj(o) are convex (see Appendix 3). It follows then that the functions e- 1Fj(Oxj) are convex or concave according as one is in the risk-seeking or risk-averse case. In the first case will invest the whole sum in an activity j for which e- 1Fj(Ox) is maximal. In the second case one spreads the investment (hedges against uncertainty) by choosing the Xj so that FJ (Ox1 ) ~ >., with equality if Xj is positive (j = I, 2, ... ) . Here >. is a Lagrange multiplier chosen so that L,j x1 = x. Hedging is a very real feature in investment practice, and we see that it is induced by the two elements of uncertainty and riskaverseness. (3) Follow through the treatment of Exercise 2 in the case (again unrealistic!) of normally distributed cb when lj· (o) = J.LjO + Vjo 2 • Here J.Lj and v1 are respectively the mean and variance of c1 (conditional on information at the time of decision).

!

2 MULTI-STAGE OPTIMISATION; THE DYNAMIC PROGRAMMING EQUATION If we extend the analysis of the last section to the multi-stage case then we are essentially treating a control optimisation problem in discrete time. Indeed, the discussion will link back to that of Section 2.1 in that we arrive at a stochastic version of the dynamic programming principle. There are two points to be made, however. Firstly, the stochastic formulation makes it particularly clear that the dynamic programming principle is valid without the assumption of state structure and, indeed, that state structure is a separate issue best brought in later. Secondly, the temporal structure of the problem implies properties which one often takes for grapted: this structure has to be made explicit. Suppose, as in Section 2.1, that the process is to be optimised over the time period t = 0, I, 2, ... , h. Let W0 indicate all the information available at time 0; it is from this and the stochastic model that one must initially infer plant history up to time t = 0, insofar as this is necessary. Let x 1 denote the value of the process variable at time t, and X, the partial process history {xi, x2, ... , x 1 }. Correspondingly let y 1 denote the observation which becomes available and u1 the control action taken at time t, with corresponding partial histories Y, and U1• Let W1 denote the information available at time t; i.e. the information on which choice of u, is to be based. Then we assume that W, = { W0 , Y, U,_J}. That is, current information consists just of initial information plus current observation history plus previous control history. It is taken for granted, and so not explicitly indicated, that W 1 also implies prescription of the stochastic model and knowledge of clock timet.

174

STOCHASTIC DYNAMIC PROGRAMMING

A realisable policy 1r is one which specifies u1 as a function of W 1 for t = 1, 2, ... , One assumes a cost function C. This may be specified as a function of Xh and Uh-l, but is best regarded simply as a random variable whose distribution, jointly with that of the course of the process and observations, is parametrised by the chosen control sequence Uh-1· The aim is then to choose -rrto minimise E... (C). Define the total value function Uh-l·

G( W1) = inf E..-[CI W,], ...

(2)

the minimal expected cost conditional on information at time t. Here E... is the expectation operator induced by policy 1t: We term G the total value function because it refers to total cost, whereas the usual value function F refers only to present and future cost (in the case when cost can indeed be partitioned over time). G is automatically t-dependent, in that W1 takes values in different sets for different t. However, the simple specification of W, as argument is enough to indicate this dependence.

*Theorem 8.2.1 (The dynamic programming principle) The total value function G( W 1) obeys the backward recursion (the dynamic programming or optimality equation)

G(W,) = inf E[G(Wt+I)I Wr, Ur]

"'

(t

= 1 ' 2, ... ' h - 1)

(3)

with closing condition

(4) and the minimising value ofu1 in (3) is the optimal value ofcontrol at timet. We prove these assertions formally in Appendix 2. They may seem plausible in the light of the discussion of the previous section, but demonstration of even their formal truth requires a consideration of the structure implicit in a temporal optimisation problem. They are in fact rigorously true if all variables take values in finite sets and if the horizon is finite; the theorem is starred only because of possible technical complications in other cases. That some discussion is needed even of formal validity is clear from the facts that the conditioning variables W, in (2) and ( W, u1) in (3) are a mixture of random variables Y, which are genuinely conditioning and control histories Ur-I or U1 which should be seen as parametrising. Further, it is implied that the expectations in (3) and (4) are policy-independent, the justification in the case of (4) being that all decisions lie in the past These points are covered in the discussion of Appendix 2. Relation (3) certainly provides a taut and general expression of the dynamic programming principle, couched, as it must be, in terms of the maximal current observable W1•

3 STATE STRUCTURE

175

3 STATE STRUCTURE The two principal properties which ensure state structure are exactly the stochastic analogues of those assumed in Chapter 2. (i) Markov dynamics. It is required that the process variable x should have the property (5) P(xt+tiX,, U,) = P(xt+tlx,, u,). where X, U1 are now complete histories. That is, if we consider the distribution of x 1+ 1 conditional on process history and parametrised by control history then it is in fact only the values of process and control variables at time t which have any effect. This is the stochastic analogue of the simply-recursive deterministic plant equation (2.2~ and specification of the right-hand member of (5) as a function of its three arguments amounts to specification of a stochastic plant equation. (ii) Decomposable cost function. It is required that the cost function should break into a sum of instantaneous and closing costs, of the form

c=

h-1

h-1

1=0

1=0

L c(x, u,, t) + Ch(xh) = L c, + ch,

(6)

say. This is exactly the assumption (2.3) already made in the deterministic case. We recall the definition of sufficiency of a variable ~~ in Section 2.1 and the characterisation of x as a state variable if (x1 , t) is sufficient. These definitions transfer to the stochastic case, and we shall see, by an argument parallel to that of the deterministic case, that assumptions (i) and (ii) do indeed imply that x is a state variable, if only it is observable. A model satisfying these assumptions is often termed a Markov decision process, the point being that they define a simply recursive structure. However, if one is to reap the maximal benefit of this structure then one must make an observational demand. (iii) Perfect state observation. It is required that the current value of state should be observable. That is, x 1 should be known at the time t when u1 is to be determined, so that W, =(X,, U,_t)· As we have seen already in the deterministic case, assumption (i) can in principle be satisfied if there is a description x of the system which is detailed enough that it can be regarded as physically complete. Whether this detailed description is immediately observable is another matter, and one to which we return in Chapters 12 and 15. We follow the pattern of Section 2.1. Define the future cost at timet h-1

IC 1 =

L:cT +Ch T=t

and the value function F(W,) =infE.,.[C,IW,] 7(

(7)

176

STOCHASTIC DYNAMIC PROGRAMMING

so that G( W1) = I:~~~ Cr + F( W1). Then the following theorem spells out the sufficiency of~~ = (x1 , t) under the assumptions above.

Theorem 8.3.1 Assume conditions (i)-(iii) above. Then (i) F( JV;) is a function of x 1 and t alone. If we write it F(x 1 , t) then it obeys the dynamic programming equation

(t

~h)

(8)

with terminal condition (9) (ii) The minimising value ofu1 in (8) is theoptimalvalueofcontrolat timet, which is consequently also a function only of x 1 and t.

Proof The value of F(Wh) is Ch(xh), so the asserted reduction of Fis valid at time h. Assume it valid at time t + 1. The general dynamic programming equation (3) then reduces to F(W1 ) = inf{c(x1, u1 , t) u,

+ E[F(xt+I, t + l)JX,, U,]}

(10)

and the minimising value of u1 is optimal. But, by assumption (i), the right-hand member of (10) reduces to the right-hand member of (8). All assertions then D follow by induction. So, again one has the simplification that not all past information need be stored; it is sufficient for purposes of optimisation that one should know the current value of state. The optimal control rule derived by the minimisation in (8) is again in closed-loop form, since the policy before timet has not been specified. It is in the stochastic case that the necessity for closed-loop operation is especially clear, since continuing stochastic disturbance of the dynamics makes use of the most recent information imperative. At least in the time-homogeneous case it is convenient to write (8) simply as

F(·,t) = fi?F(-,t+ 1)

( 11)

where £? is the operator with action £? ¢(x)

= inf {c(x, u) + E[¢(xt+l) Jx1 = x, u1 = u]}. u

(12)

This is of course just the stochastic version of the forward operator already introduced in Section 3.1. As then, 2 ¢(x1) is the minimal cost incurred if one is allowed to choose u1 optimally, in the knowledge that at time t + 1 one will incur a cost of ¢(xt+ 1). In the discounted case£? would have the action 2¢(x) = inf {c(x, u) u

+ ,8E[¢(xt+l )Jx1 =

x, u1 = u]}.

(13)

177

4 THE EQUATION IN CONTINUOUS TIME

4 THE DYNAMIC PROGRAMMING EQUATION IN CONTINUOUS

TIME It is convenient to note here the continuous-time analogue of the material of the last section and then to develop some continuous-time formalism in Chapter 9, before progressing to applications in Chapter 10. The analogues of assumptions (i}-(ili) of that section will be plain; we deduce the continuous-time analogues of the conclusions by a formal passage to the limit. It follows by the discrete-time argument that the value function, the infimum of expected remaining cost from time t conditional on previous process and control history, is a functionF(x, t) of x(t) = xand t alone. The analogue of the dynamic programming equation (8) for passage from t to t + 8t is

F(x, t)

= inf{c(x, u, t) + E[F(x(t + 8t), t + 8t)lx(t) = x, u(t) = u]} + o(5t). u (14)

Defme now the infinitesimal generator A(u, t) of the controlled process by A(u, t)rp(x)

1 = lim(8t){E[¢(x(t + 8t))lx(t) = x, u(t) = u]- ¢(x)}. 6t!O

(15)

That is, there is an assumption that, at least for sufficiently regular ¢(x),

E[rp(x(t + 8t))lx(t)

= x, u(t) = u] = ¢(x) + [A(u, t)¢(x)]8t + o(8t).

The form of the term of order 8t defines the operator A; to write the coefficient of 8t as A(u, t)rp(x) emphasises that the distribution of x(t + 8t) is conditioned by the value x of x(t) and parametrised by t and the value u ofu(t).We shall consider the form of A in some particular cases in the next chapter.

*Theorem 8.4.1 Assume the continuous-time analogues of conditions (i)-(iii) of the Section 3. Then x is a state variable and the value function F (X. t) obeys the dynamic programming equation

i~ [c(x, u, t) +oF~~· t) + A(u, t)F(x, t)]

= 0

(t
(16)

The minimising value ofu is the optimal value ofu( t). Equation (16) follows formally from equation (14) in the limit of small 5t. We have starred the theorem because, to the technical complications which can arise when x and u may vary in infinite sets and when also h may later allowed to be infinite are added those which may appear in the passage to continuous time. However, any invalidity of (16) in particular cases is usually well signalled by one's realisation of some anomaly in the mathematical representation of the physical picture.

178

STOCHASTIC DYNAMIC PROGRAMMING

If costs are discounted at rate a then equation (16) becomes

¥[c-aF+': +AF] =0

(t < h).

(17)

Here the arguments of the various functions have been left understood, as is often possible without ambiguity.

CHAPTER 9

Stochastic Dynamics in Continuous Time In general we assume the reader acquainted with basic probability theory. However, we shall devote a short chapter to continuous-time stochastic processes, since these show special features and are of special relevance in control contexts. Some of the material also prepares the way for that of Chapter 22. We assume a state-structured model with state variable x, but shall for simplicity consider only the time-homogeneous uncontrolled case. These limitations are easily removed later, by simply making the various rates which occur depend also upon t and u.

1 JUMP PROCESSES One extreme case is that in which the state space fl is discrete, so that changes in state must be discontinuous. For example, if x is the size of a queue then x can only take the values 0, 1, 2, ... and a transition (from x to x ± 1) takes place whenever a customer arrives or leaves. Similarly, if we consider a model of a population which is recognised as being made up of individuals, then population size can only be integral. Notation is simplified if we assume that the values of state x are integral, taking values j = 0, 1, 2, ... , say. If one later wishes to attach other numerical values to the states then there is no trouble in doing so. Suppose now that

P[x(t + 8t) = klx(t) = j] = >..1k8t + o(8t)

(k =1- j)

(1)

for small positive 8t. This is a regularity condition which turns out to be selfconsistent. The quantity >..1k is termed the probability intensity of transition from j to k, or simply the transition intensity. The assumption itself implies that the transition has been a direct one: the probability of its having occurred by passage through some other state is of smaller order in 8t (see Exercise 1).

Theorem 9.1.1 The process with transition intensities >..1k has infinitesimal generator A with action

A¢(j)

=

L AJk[¢(k)- ¢(j)]. k

(2)

180

STOCHASTIC DYNAMICS IN CONTINUOUS TIME

Proof It is a consequence of (1) that E[¢(x(t + 8t))- ¢(x)ix(t) = j) =

L AJk[¢(k)- ¢(j)]8t + o(8t) kf.j

whence (2) follows, by the definition of A. To include the case k = j in the D summation plainly has no effect. We can return to the controlled time-dependent case by making the transition intensity a function >.1k(u, t) ofu and t, when the dynamic programming equation (8.16) becomes

i~f [c(j, u, t) +oF~, t) + ~ >.1k(u, t)[F(k, t)- F(j, t)]l

= 0.

(3)

Exercises and comments (1) It follows from (1) and the Markov character of the process that

P[x(t + 8t) = i,

x(t + 2 8t) = klx(t) = j] = >.1;Aik(bt) 2 + o[(8t)

2

]

fori distinct from bothj and k. This at least makes it plausible that the probability of multiple transitions in an interval oflength 8t is o(8t). 2 DETERMINISTIC AND PIECEWISE DETERMINISTIC PROCESSES The deterministic model for which x is a vector obeying the plant equation x = a(x) is indeed a special case of a stochastic model. The rate of change of ¢(x) in time is ¢xa, so that the infinitesimal generator A has the action A¢(x) = ¢x(x)a(x)

(4)

where ¢xis the row-vector of differentials of¢ with respect to the components of x. Consider a hybrid of this deterministic process and the jump process of the last section, in which the x-variable follows deterministic dynamics x = a1(x) in the jth regime, but transition can take place from regime j to regime k with intensity >.1k(x). Such a process is termed a piecewise deterministic process. The study of such processes was initiated and developed by Davis (1984, 1986, 1993). For example, if we consider an animal population, then statistical variability can occur in the population for at least two reasons. One is the intrinsic variability due to the fact that the population consists of a finite number of individuals: demographic stochasticity. Another is that induced by variability of climate, weather etc.: environmental stochasticity. If the population is large then it is the second source which is dominant: the population will behave virtually deterministically under fixed environmental conditions. If we suppose, for

3 THE DERIVATE CHARACTERISTIC FUNCTION

181

simplicity, that the environmental states are discrete, with well-defined transition intensities, then the process is effectively piecewise deterministic. In such a case the state variable consists of the pair (j, x): the regime labelj and the plant variable x. We leave it to the reader to verify that the infinitesimal generator of the process has the action

A¢J(j,x) = ifJx(j,x)aJ(x)

+ LAJk(x)[ifJ(k,x)- ¢J(j,x)]. k

3 THE DERIVATE CHARACTERISTIC FUNCTION Recall that the moment-generating JUnction (abbreviated MGF) of a random column vector xis defined as M(a) = E(eax), where the transform variable a is then a row vector. Some basic properties of MGFs are derived in Appendix 3. One would define M (iO) as the characteristic jUnction of x; this always exists for real 0. The two definitions then differ only by a 900 rotation of the argument in the complex plane, and it is not uncommon to see the two terms loosely confused. Suppose from now on that the state variable x is vector-valued. We can then define the function (5) of the column vector x and the row vector a. It is known as the derivate characteristic fUnction (abbreviated to DCF). We see, from the interpretation of A following its definition (8.15), that Hhas the corresponding interpretation

E(e 06xlx(t) = x) = 1 + H(x, a)8t + o(8t),

(6)

where 6x = x( t + 6t) - x( t) is the increment in x over the time interval ( t, t + 6t]. The DCF thus determines the MGF of this increment for small 6t; to this fact plus the looseness of terminology mentioned above it owes its name. For example, consider a process for which xcanjump to a value x + d;(x) with probability intensity >.1(x)(j = 1, 2, ... )_ For this the infmitesimal generator has the action

AifJ(x) = L >.1(x)[¢J(x + d;(x))- ifJ(x)]

(7)

j

and the DCF has the evaluation

H(x,a) = I:>.J<x)[eadJ(x)- 1].

(8)

j

Comparing these last two relations we see that we can make the assertion, at least for processes of this type:

182

STOCHASTIC DYNAMICS IN CONTINUOUS TIME

"'Theorem 9.3.1 Relation (5) between the derivate characteristic/unction Hand the infinitesimal generator Ahas the inversion A= H(x, a/ax)

(9)

if it

is understood that, when we form M>(x) = H(x, a/ax)(x), the differential operator a1ax acts only on the x-argument of.

This indeed follows from comparison of (7), (8) and appeal to the 'Taylor' formula

e(8f8x)d(x) = (x +d), Here (a/ax)dwemean the inner product I:k dka/axk' where dk andxk arethekth components of the vectors d and x respectively. We shall return to the important identity (9) in Section 22.3. 4 PROCESSES OF INDEPENDENT INCREMENTS, AND PROCESSES DRIVEN BY THEM A process {x(t)} is one of independent increments if the increments in x over disjoint time intervals are statistically independent. If, in addition, the distribution of the increment depends only upon the length of the interval then it is a homogeneous process of independent increments (abbreviated HPII). Such processes are important because their 'time derivative' (should it exist) provides the continuous-time equivalent of a sequence of liD random variables. The abbreviation HPII is not a standard one, but it will serve us for this and the next section. Let us define the MGF

M(a, t) = E[e"'[x(t)-x(Ollj. Then the HPII property is obviously equivalent to the property

M(a, t1 "'Theorem 9.4.1

+ tz) =

M(a, ti)M(a, tz)

(10)

lf {x(t)} is an HPJ/then there exists a function ,P(a) such that M(a, t) = e1t/l(a).

( 11)

dx(t)}] = exp{j ,P(a(t)) dt },

(12)

Furthermore E[exp{j a(t)

for prescribed functions a (t) for which the right-hand member is defined.

4 PROCESSES OF INDEPENDENT INCREMENTS

183

*Proof Relation (11) is indeed a consequence of (10). It follows from (11) and the independence of increments that, if {t1} is an increasing sequence of timepoints, then E [exp{

~ aJlx(tJ)- x(tJ-d}] = exp{ ~(tJ- tJ-1)1/J(aJ) }·

Relation (12) then follows by a limiting argument. *Theorem 9.4.2 ofx, so that

D

The process {x(t)} is an HPII ifand only ifH(x, a) is independent

A= 1/J(o/ox),

(13)

say. One can then identify 1/J with the 1/J of(ll). *Proof If the process is an HPII so that (11) holds then one indeed verifies that H(x, a) = 1/J(a), implying(13). To establish the reverse conclusion, define the conditional MGF M(a, x, t) = E[e<>x(r)lx(O) = xJ This obeys the backward equation

a

otM(a,x, t) = 1/J(o/ox)M(a, x, t)

which has a solution M(a,x,t) = e=+h/1(<>).

(14)

But relation (14) is sufficient to establish both relation (10) and the representation (11). D So, H(x, a) = 1/J(a) is both necessary and sufficient for the HPII property, with the 1/J( a) of (11) identified with the DCF. It is natural to ask: what functions 1/J are possible? which are interesting? In fact, the HPII property implies that exp['I/J(a)J must be the MGF of an infintely divisible distribution. The Uvy-Khintchine theorem then implies that an HPII process must be a superposition of independent Poisson and Wrener processes. It is a quick matter to describe these processes and verify the HPII property for them. The simplest HPII process is the Poisson process. For this x takes only nonnegative integer values, the only transitions are of the form x -+ x + 1 and these all have the same intensity A. One could regard x(t) - x(O) as the number of events which have taken place in the time interval (0, t], if events take place independently with probability intensity A. These events might be insurance claims, traffic accidents or arrivals of cosmic particles in a chamber. The DCF of the process is 1/J(a) = A(e<>- 1). We see then from (11) that the distribution of the number of events in a time interval of length t is Poisson with expectation At, whence the name of the process. One speaks of the events occurring in a Poisson stream ofrate >..

184

STOCHASTIC DYNAMICS IN CONTINUOUS TIME

We can generalise the process by superimposing Poisson streams. Consider the scalar process {w(t)} generated by

w(t) =

L.>1x1(t)

(15)

j

where the a1 are constants and the {x1(t)} are independent Poisson processes of respective rates >y. For example, thejth stream might be interpretable as a stream of particles each carrying charge a1; the variable w(t) is then the total charge which has accumulated by time t. Such a process is termed a compound Poisson process. It is plainly HPII with DCF t/J(o)

= L}y(eaai- 1).

(16)

j

One obtains an interesting class of models by driving a deterministic model by

anHPil Theorem 9.4.3 Consider the vector process {x(t)} generated by the differential equation (written in incrementa/form) bx = a(x) bt + b(x) bw

(17)

where {w( t)} is a vector-valued homogeneous process ofindependent increments with Then {x(t)} is a MarkovprocesswithDCF

DCFt{J(o~

H(x, o)

.

= oa(x) + t/J(ob(x)).

(18)

The assertions follow immediately if one regards relation (6) as providing the effective definition of the DCF and inserts expression (17) for bx into it. The 'plant equation' (17) is interesting as the continuous-time version of a process driven by 'noise' which is totally random, in the sense that its values at different times are statistically independent. One is tempted to rewrite it as .X = a(x)

+ b(x)~:

(19)

where the 'noise' variable ~: is the time derivative of w. The process {w(t)} generated by (15) plainly does not have a derivative in the usual sense. If one admitted the idea one would have to regard it as a random sequence of impulses, those of magnitude a1 occurring with intensity >..b independently of other impulses. This is the 'shot noise' process formulated by physicists early in the century. The concept is proper in that integrals of shot noise are proper; one can write (12) as E[exp{j

o(t)~:(t) dt}]

= exp{j t/J(o(t)) dr}.

(20)

185

5 WIENER PROCESSES (BROWNIAN MOTION)

5 WIENER PROCESSES (BROWNIAN MOTION), WHITE NOISE AND DIFFUSION PROCESSES If we consider a sequence of shot noise processes in which the impulses become ever weaker but ever more frequent then we reach the classic limit process associated with Brownian motion and white noise.

*Theorem 9.5.1 Consider the scalar HPJ/ { w( t)} with DCF 1/J(a.) = (eaa + e-aa)/2a2.

(21)

The limit process for small a is then an HPII with DCF and infinitesimal generator

A=

1/J(8f8x) = ~(:x)

2

(22)

Further, since E[exp{j a.(t)dw(t)}] =E[exp{j a.(t)E(t)dt}]

=exp{~j a.(t) 2 dt}. (23)

increments ofthe process arejointly normally distributed. The assertions are formally immediate. They constitute in fact a process form of the central limit theorem, given operational meaning by the fact that the argument can be phrased in terms of the statistics of well-behaved functionals such as the integral I a.( t) dw( t). We recognise in (21) a compound Poisson process consisting of two streams, each of rate 1/2a2 and carrying 'charges' of size a and opposite signs. If a is small then the charge carried is small, but particles arrive fast. The 'derivative' f of the limit process is an infinitely dense sequence of infinitesimal independent impulses, whose integral has zero expectation but is normally distributed The limit process { w( t)} is a time-homogeneous process of independent increments which also has the property of being Gaussian. As a function of time it in fact continuous, and is the only continuous HPII. It is in all senses classical: sometimes referred to as Brownian motion and written B(t); sometimes referred to as the Wiener process and written W(t). Its formal differential E(t) is termed 'white noise' by engineers for reasons explained in Section 13.2 It is improper in having infinite variance, but proper in that integrals of white noise are proper, as we see in (23). The argument of Theorem 9.5.1 can be repeated in a vector version. We then arrive at the notion of a vector Wiener process w(t) and vector white noise E(t) with the property that the linear form I a.(t) dw(t) = I a.(t)E(t) dt is normally distributed with zero mean and covariance matrix

186

STOCHASTIC DYNAMICS IN CONTINUOUS TIME

cov(j o(t)e(t) dt)

=j

o(t)No(t)T dt

(24)

for some non-negative deftnite matrix N, at least for functions o(t) such that this last integral is convergent. Relation (24) implies that

( 1

t+6t

cov {6t)- 1

)

e(r) dr = Nj6t,

1

which has an infinite limit as 6t tends to zero. This is an indication that c: itself has an irregular character; its components have inftnite variance. Such a conclusion is inevitable if one requires property (24) for non-zero N The matrix N indeed gives the statistical scale of c: but is not its covariance matrix. It is more appropriately to as the power matrix. As we have seen, one can derive white noise as the 'limit' of realistic physical processes. Mathematically, the concept is regularised in the so-called theory of generalised processes (see e.g. Gihman and Skorohod, 1972, 1979} White noise is to conventional processes what the Dirac 6-function is to conventional functions. Like the 6-function, it can be seen as a limit of conventional processes, and the operational characterisation in terms of linear functionals above is analogous to the operational characterisation I o(t)6(t) dt = o(O) of the 6-function. Initial discussions of the 6-function took the circumspect route of writing I o(t)6(t) dt as I o(t)dH(t) where the Heaviside function H(t) is the formal integral of the 6-:-function; the function which is constant except for a single unit step at the origin. This is now seen as a crutch which can be kicked away, but the same caution leads to the writing of I o(t)c:(t) dt as I o(t) dw(t) where {w(t)} is a Wiener process. The noise-driven processes (19) which are most considered, for various reasons, are those for which the driving noise e is white; such a process is termed a diffusion process for reasons which will soon become clear. If one takes a ·deterministic process as a first crude approximation to a given stochastic process, then there are good reasons for taking a diffusion process as the second; see Section 22.3.We can specialise Theorem 9.4.3 to the diffusion case.

Theorem 9.5.2 Consider the vector process {x(t)} generated by the differential equation 6x = a(x)t + b(x) 6w

(25)

or

x=

a(x) + b(x)c:

(26)

where c: is a white noise process (wa Wiener process) ofpower matrix N Then {x( t)} is a Markov process with DCF

5 WIENER PROCESSES (BROWNIAN MOTION)

187

H(x,a) = aa(x) +!aN(x)aT

(27)

where N(x) = b(x)Nb(x)T. We see from (13) and (27) that the generator of the process has the action A¢ = ¢xa(x)

+!

tr[N(x)¢xx]

(28)

where ¢xx is the square matrix of second differentials of ¢. It is because A is a second-order differential operator in x that one terms the process a diffusion process. More exactly, the corresponding equation j = ATf for the probability density fof x(t) is a diffusion equation in the full technical sense, representing the diffusion of probability mass in state space. The deterministic term a(x) in (26) simply induces a transfer of probability mass along the deterministic path, but the random term in f induces a spreading of that mass. The vector a(x) and matrix N(x) are referred to as the drift and diffusion coefficients respectively. It is indeed the generator (27) which determines the forward dynamics of the process, and so implies a specification of the 'stochastic plant equation: However, in this case the stochastic differential equation (26) can be regarded as providing an alternative specification, and one expressed as a dynamic equation in the state variable itself. For a controlled time-homogeneous diffusion process the dynamic programming equation for the value function F(x, t) would be inf[c(x, u, t) u

+ F + Fxa(x, u, t) +! 1

tr(N(x, u, t)F=)I = 0.

(29)

CHAPTER 10

Some Stochastic Examples 1 LQG REGULATION WITH PLANT NOISE Let us return to the LQ regulation example of Section 2.4 with the sole modification that the plant equation is changed to Xr

= AXr-1 +Bur-l+ fr

(1)

where t. 1 is a random disturbance term. This term would often be referred to as plant noise-'plant' because it occurs in the plant equation and 'noise' because in audio-electronic contexts the disturbance manifests itself as a rush of background noise. Plant noise is indeed often to be regarded as random in that it is only partially predictable. The thermal noise of electronic circuits or the gusts to which an aircraft is subject provide examples of disturbances which are scarcely predictable at all. The mention of predictability is a reminder that the stochastic character of the noise must be specified. A common starting assumption is that the noise is as unpredictable as possible; that it is white in that the distribution of t. 1 conditional on process history before timet is independent of both process history and t. This implies that the process bas the Markov property (i) of Section 8.3. It also implies that the variables € 1 are independently and identically distributed. (See Sections 9.5 and 13.2 for a discussion of the special character of white noise in continuous time and of the reason for the description 'white~ Process history before time t is specified by (X1_ 1, Ur-1), since this determines fr forT < t. Actually, we do not need to demand complete 'whiteness', but simply that (2)

for a constant matrix N. That is, that the conditional mean of f 1 should be constant (and normalised to zero, by convention) and that the conditional covariance matrix of f 1 should also be constant. A stronger assumption which is often made is that the noise should be Gaussian as well as white, i.e. that the noise variables should be jointly normally distributed. The resultant model is then termed an LQG model, indicating linear dynamics, quadratic costs and Gaussian noise. However, as far as discrete-time models are concerned, we need the Gaussian assumption first when we come to consider imperfect state observation in Chapter 12.

190

SOME STOCHASTIC EXAMPLES

Theorem 10.1.1 Assume the stochastic plant equation specified by (1~ (2) and the quadratic cost structure specified in equations (2.22) and ~23). Then the value function has the form

F(x, t)

= !xTII,x + 6

(3)

1

and the optimal control the form u1 =K1x 1•

(4)

Here the matrices II, and K1 have the same evaluations as in Theorem 24.2 and

(5)

6, = 6r+1 +! tr(Nilr+I)

with terminal condition 6h = 0. That is, the matrix II, is the same solution of the Riccati equation (2.25) as before, and K, is determined from it by the same relation (2.28) as before. Indeed, the control rule (4) is exactly what it was in the noise-free case, and the only effect of the noise is to add a term 61 to the cost. The interpretation is that the quadratic term in (3) represents the cost caused by the initial deviation of state x from zero and the term 61 represents the cost caused by the continued disturbance of state by noise. In the infinite-horizon limit (if one exists) the presence of noise induces a cost of! tr(NII) per unit time.

Proof The course of the proof is inductive, as ever. Relation (3) holds at time h. Assume then that it holds at time t + 1, so that F(W,)

= inf{c(x, u) +!E((Ax + Bu + fr+I?nr+I(Ax +Bu + fr+I)IXr,Ur]} "

+ 61+11

where x, u and e are the values holding at time t. Assumptions (2) imply that the conditional expectation in this relation reduces to

!(Ax+ Buliir+I (Ax+ Bu) +! tr(NIIr+l) whence all assertions follow.

0

The admission of plant noise with the 'second-order white' properties (2) thus affects the optimal control rule not at all (at least, if this is expressed in closed-loop form) and increases costs only by a term independent of state or polic}: It may seem strange that, for our first stochastic example, the optimal control was unaffected (at least in its closed-loop form) by the presence of the stochastic element. This is a consequence of the very special assumptions. The essence of optimal control generally is that one exploits the statistical characteristics of

1 LQG REGULATION WITH PLANT NOISE

191

noise variables to the full. For example, suppose that the plant equation (1) is modified to x 1 = Ax,_1 + Buc-1 + v,

(6)

where v1 is a stochastic noise signal of some more general form. One will often suppose that it can be generated as the output v, = G(f/)£. 1 of a linear filter driven by white noise, so that one still has in effect a system driven by white noise, but more complex than (1). This can be coped with in various ways, but the most insightful is that which follows from the certainty equivalence principle of Chapter 12. This principle implies that, under some further assumptions on the linear/Gaussian nature of observations, the optimal control at time t for system (6) would be the same as that for the deterministic system X.,.=

A Xr-1

+ B U.,.-1 + v.,.(1)

(r

> t),

(7)

where v~') is an appropriate linear predictor of v.,. based on information available at time t. That is, at time t one replaces future stochastic noise vT( r > t) by an 'equivalent' deterministic disturbance v~1 ) and then applies the methods of Sections 2.9 or 6.3 to deduce the optimal feedback/feedforward control in terms of this predicted disturbance. We shall see in Chapter 12 that similar considerations hold if the state vector xis itself not perfectly observable. It turns out that E~) = 0 (r > t) for a white-noise input f. which has been perfectly observed up to timet. This explains why the closed-loop control rule was unaffected in case (1). Once we drop LQG assumptions then treatment of the stochastic case becomes much more difficult. For general non-linear cases there is not a great deal that can be said We shall see in Section 7 and in Chapter 24 that one can treat some models for which LQG assumptions hold before termination, but for which rather general termination conditions and costs may be assumed. Some other models which we treat in this chapter are those concerned with the timing of a single definite action, or with the determination of a threshold for action. For systems of a realistic degree of complexity the natural appeal is often to asymptotic considerations: e.g. the 'heavy traffic' approximations for queueing systems or the large-deviation treatment oflarge-scale systems. Exercises and comments (1) Consider the closed- and open-loop forms of optimal control (2.32) and (2.33)

deduced for the simple LQ problem considered there. Show that if the plant equation is driven by white noise of variance N then the additional cost incurred from time t = 0 is DQN (Q + sD)- 1 or hDN according as the closed- or the open-loop rule is used. These then grow as log h or as h with increasing horizon.

2:::-ci

192

SOME STOCHASTIC EXAMPLES

2 OPTIMAL EXERCISE OF A STOCK OPTION As a last discrete-time example we shall consider a simple but typical financial optimisation problem. One has an option, although not an obligation, to buy a share at price p. The option must be exercised by day h. If the option is exercised on day t then one can sell immediately at the current price x" realising a profit of x 1 - p. The price sequence obeys the equation Xr+! = x, + e1 where the e1 are independently and identically distributed random variables for which Elel < oo. The aim is to exercise the option optimally. The state variable at time t is, strictly speaking, x 1 plus a variable which indicates whether the option has been exercised or not However, it is only the latter case which is of interest, sox is the effective state variable. If Fs(x) is the value function (maximal expected profit) with times to go then

Fo(x) = max{x- p, 0}

= (x- p)+

and

F.r(x) = max{x- p, E[F.r-!(x +e)]}

(s=l,2, ... ).

The general character of Fs(x) is indicated in Figure 1; one can establish the following properties inductively: (i) Fs(x) -xis non-increasing in x; (ii) Fs(x) is increasing in x; (iii) Fs(x) is continuous in x; (iv) Fs(x) is non-decreasing ins. For example, (iv) is obvious, since an increase ins amounts to a relaxation of the time constraint. However, for a formal proof:

F1(x)

= max{x- p,E[Fo(x +e)]}~

max{x- p,O}

= F0 (x),

x-p Figure I The valuefonction at horizon sfor the stock option example.

3 A QUEUEING MODE

193

whence Fs is nondecreasing ins, by Theorem 3.1.1. Correspondingly, an inductive proofof(i) follows from Fs(x)- x =max{ -p, E[Fs-1 (x + t:)- (x + t:)]

+ E(t:)}.

We then derive Theorem 10.2.1 There exists a non- decreasing sequence {as} such that an optimal policy is to exercise the option first when x ): as, where xis the current price and sis the number ofdays to go before expiry ofthe option. Proof From (i) and the fact that Fs(x) ): x- pit follows that there exists an as such that Fs(x) is greater than x- p if x < as and equals x- p if x ): as· It follows from (iv) that as is non-decreasing ins. 0

The constant as is then just the supremum of values of x for which Fs(x) x-p.

>

3 A QUEUEING MODEL Queues and systems of queues provide a rich source of optimisation models in continuous time and with discrete state variable. One must not think simply of the single queue ('line') at a ticket counter; computer and communication systems are examples of queueing models which constitute a fundamental type of stochastic system of great technological importance. However, consideration of queues which feed into each other opens too big a subject; we shall just cover a few of the simplest ideas for single and parallel queues in this section and the next chapter. Consider the case of a so-called MIMI! queue, with x representing the size of the queue and the control variable u being regarded as something like service effort. If we say that customers arrive at rate A and are served at rate p,(u) then this is a loose way of stating that the transition x --+ x + 1 has intensity Aand the transition x--+ x- 1 has intensity p,(u) if x > 0 (and, of course, intensity zero if x = 0). We assume the process time-homogeneous, so that the dynamic programming equation takes the form

i~ [c(x, u) +oF~~· t) + A(u)F(x, t)]

= 0.

(8)

Here the infinitesimal generator has the action (cf. (9.2))

A(u)¢(x) = A[¢(x + 1)- ¢(x)]

+ p,(u,x)[¢(x- 1)- ¢(x)]

where p,( x, u) equals p,( u) or zero according as xis positive or zero. If we were interested in average-optimisation over an infinite horizon then equation (8) would be replaced by

194

SOME STOCHASTIC EXAMPLES

'Y = inf[c(x, u) u

+ A(u)f(x)]

(9)

where .\ and f(x) are respectively the average cost and the transient cost associated with the average-optimal policy. In fact, we shall concern ourselves more with the question of optimal allocation of effort or of customers between several queues than with optimisation of a single queue. In preparation for this, it is useful to solve (9) for the uncontrolled case, when J.L(u) reduces to a constant J.L and c(x, u) to c(x). In fact, we shall assume the instantaneous cost proportional to the number in the queue, so that c( x) = ax. We leave it to the reader to verify that the solution of equation (9) is, in this reduced case, a.\

"(=--,, j.L-/\

!( X )

= _a_x(x + 1) J.L- A

2

.

(10)

We have assumed the normalisation f(O) = 0. The solution is, of course, valid only if.\ < p., for it is only then that queue size is finite in equilibrium.

4 THE HARVESTING EXAMPLE: A BIRTH-DEATH MODEL Recall the deterministic harvesting model of Section 1.2, which we shall generally associate with fisheries, for definiteness. This had a scalar state variable x, the 'biomass' or stock level, which followed the equation

x=

a(x)- u.

(11)

Here u is the harvesting rate, which (it is supposed) may be varied as desired. The rate of return is also supposed proportional to u, and normalised to be equal to it. (The model thus neglects two very important elements: the age structure of the stock and the x-dependence of the cost of harvesting at rate u.) We suppose again that the function a(x), the net reproduction rate of the unharvested population, has the form illustrated in Figure 1.1; see also Figure 2. We again denote by Xm and xo the values at which a(x) is respectively maximal and zero. An unharvested population would thus reach an equilibrium at x = x 0 . We know from the discussion of Section 2.7 that the optimal policy has the threshold form: u is zero for x ~ c and takes its maximal value (M, say) for x > c. Here c is the threshold, and one seeks now to determine its optimal value. If a(x) > 0 for x ~ c and a(x)- M < 0 for x > c then the harvested population has the equilibrium value c and yields a return at rate 'Y = a( c). If we do not discount, and so choose a threshold value which maximises this average return A, then the optimal threshold is the value Xm which maximises a( c). A threshold policy will still be optimal for a stochastic model under corresponding assumptions on birth and death rates. However, there is an effect which at first sight seems remarkable. If extinction of the population is impossible then one will again choose a threshold value which maximises average

4 THE HARVESTING EXAMPLE: A BIRTH-DEATH MODEL

195

return, and we shall see that, under a variety ofassumptions, this optimal threshold indeed approaches Xm as the stochastic model approaches determinism. However, if extinction is certain under harvesting then a natural criterion (in the absence of discounting) is to maximise the expected total return before extinction. It then turns out that the optimal threshold approaches xo rather than Xm as the model approaches determinism. There thus seems to be a radical discontinuity in optimal policy between the situations in which the time to extinction is finite or infinite (with probability one, in both cases). We explain the apparent inconsistency in Section 6, and are again led to a more informed choice of criterion. We shall consider three distinct stochastic versions of the model; to follow them through at least provides exercise in the various types of continuous-time dynamics described in the last chapter. The first is a birth-death process. Letjbe the actual number offish; we shall set x = j f K where K is a scaling parameter, reflecting the fact that quite usual levels of stock x correspond to large values of j. (We are forced to assume biomass proportional to population size, since we have not allowed for an age structure). We shall suppose that j follows a continuous-time Markov process on the nonnegative integers with possible transitions j --+ j + 1 and j --+ j - 1 at respective probability intensities >..1 and 1-LJ· These intensities thus correspond to population birth and death rates. The net reproduction rate >..1 - J.ti could be written a1, and corresponds to K.a(x). Necessarily /-LO = 0, but we shall suppose initially that >..o > 0. That is, that a zero population is replenished (by a trickle of immigration, say), so that extinction is impossible. Let 1ri denote the equilibrium distribution of population size; the probability that the population has size j in the steady state. Then the relation 1r1>..1 = 7rj+I J..l.i+I (expressing the balance of probability flux between states j and j + 1 in equilibrium) implies that 1fJ ex Pi• where

Pi= >..o>..l ••. )..j-1

(

j= 0 ,1,2, ... ) ,

(12)

J.I.I/J2 · · • J..l.i

(with p0 = 1, which is consistent with the convention that an empty product should be assigned the value unity). A threshold c for the x-process implies a threshold d ~ KC for thej-process. For simplicity we shall suppose that the harvesting rate M is infmite, althcugh the case of a finite rate can be treated almost as easily. Any excess of population over d is then immediately removed and one effectively has >..d = 0 and Pi = 0 for j >d. The average return (i.e. expected rate of return in the steady state) on the xscale is then

(13)

SOME STOCHASTIC EXAMPLES

196

the term 7rdAd representing the expected rate at which excess of population over c is produced and immediately harvested. Suppose now that the ratio ej = /-Lj I >.j is effectively constant (and less than unity) for j in the neighbourhood of d. The effect of this is that 7rd-j ::::> 1rdO~ (j ~d), so the probability that the population is an amountj below threshold falls away exponentially fast with increasingj. Formula (13) then becomes 1

::::>

~t- 1 >.d(1 - Od) = ~t- 1 (>.d- /-Ld) = ~t- 1 ad =a( c).

The optimal rule under these circumstances is then indeed to choose the threshold cas the level at which the net reproduction rate is maximal, namely, Xm. This argument can be made precise if we pay attention to the scaling. The nature of the scaling leads one to suppose that the birth and death rates are of the forms >.j = ~t>.(jl~t), /-Lj = K.J.L(jl~t) in terms of functions >.(x) and J.L(x), corresponding to the deterministic equation .X= >.(x) - J.L(x) = a(x) in the limit of large K.. The implication is then that >.j I /-Lj varies slowly with j if K. is large, with the consequence that the equilibrium distribution of x falls away virtually exponentially as x decreases from d = cI K.. The details of this verbal argument are easily completed (although anomalous behaviour atj = 0 can invalidate it in an interesting fashion, as we shall see). The theory of large deviations (Chapter 22) deals precisely with such scaled processes, in the range for which the scale is large but not yet so large that the process has collapsed to determinism. Virtually any physical system whose complexity grows at a smaller rate than its size generates examples of such processes. Suppose now that >.0 = 0, so that extinction is possible (and indeed certain if, as we shall suppose, passage to 0 is possible from all states and the population is harvested above some finite threshold.) Expression (12) then yields simply a distribution concentrated onj = 0. Let F;· be the expected total return before extinction conditional on an initial population ofj. (It is understood that the policy is that of harvesting at an infinite rate above the prescribed threshold value d.) The dynamic programming equation is then

(0 <j
(14)

with the effective boundary conditions Fo = 0 and (Fd+l - Fd) = 1. This last relation follows because any excess over threshold is immediately removed and converted into yield. Theorem 10.4.1 Assume that passage to 0 is possible from all states and that the threshold dis finite. Then the relevant solution of ( 14) is

Fj =lid

(t

/-L1/-L2 · · · /-Lk-1) k=l >.1>.2 . .. >.k-1

(j

~d)

(15)

S BEHAVIOUR FOR NEAR-DETERMINISTIC MODELS

197

where (16)

Proof We can write (14) as >.1!11+1 = J.Lill.l where !11 = Fj - FJ-I· Using this equation to determine !11 in terms of l1ci+1 = 1 and then summing to determine Fj, we obtain the solution (15). D

We see from (15) that the d-dependence of the Fj occurs only through the common factor ILl, and the optimal threshold will maximise this. The maximising value will be that at which Ad/ J.Ld decreases from a value above unity to one below, so that ad = A.t - J.Ld decreases through zero. That is, the optimal value of c is Xo, the equilibrium level of the unharvested population. More exactly, it is less than x0 by an amount not exceeding K:- 1 . This means in fact a very low rate of catch, even while the population is viable. The two cases thus lead to radically different recommendations: that the threshold should be set near to Xm or virtually at x 0 respectively. We shall explain the apparent conflict in the next two sections. It turns out that the issue is not really one of whether extinction is possible or not, but of two criteria which differ fundamentally and are both extreme in their way. A better understanding of the issues reveals the continuum between the two policies. Exercises aod comments

(1) Consider the naive policy first envisaged in Chapter 1, in which the population was harvested at a flat rate u for all positive x. Suppose that this translates into an extra mortality rate of v per fish in the stochastic model. The equilibrium distribution tr1 of population size is then again given by expression (12), once normalised, but with J.LJ now modified to J.Li + v. The roots x 1 and x2 of a(x) = u, indicated in Figure 1.2 and corresponding to unstable and stable equilibria of the deterministic process, now correspond to a local minimum and a local maximum of tr1. It is on this local maximum that faith is placed, in that the probability mass is supposed to be concentrated there. However, as v (and sou) increases this local maximum becomes ever feebler, and vanishes altogether when u reaches the critical value Um = a(xm)· S BEHAVIOUR FOR NEAR-DEI'ERMINISIIC MODELS In order to explain the apparent stark contradiction between the two policies derived in Section 4 we need to obtain a feeling for orders of magnitude of the various quantities occurring as K: becomes large, and the process approaches determinism. We shall follow through the analysis just for the birth-death model

198

SOME STOCHASTIC EXAMPLES

of the last section, but it holds equally for the alternative models of Sections 8 and

9. Indeed, all three cases provide examples of the large deviation theory of Chapter22. Consider then the birth-death model in the case when extinction is possible. Since most of the time before extinction will be spent near the threshold value if K is large (an assertion which we shall shortly justify) we shall consider only Fck the expected yield before extinction conditional on an initial valued ofj. Let 1j denote the expected time before extinction which is spent in state j (conditional on a start from d). Then, by the same methods which led to the evaluation (15), we find that

1j = Ild

[t

J.i.IJ.l-2 · · · J.i.k-1] [J.l.j+IJ.i.J+2 · · · J.i.d].

k=l .A1.A2 ... >-k-1

.Ai.AJ+I ... .Act

( 17)

which is consistent with expression (15) for Fd = .AdFd. When we see the process in terms of the scaled variable x = j I K we shall write Fj = ~tF(x) and 1j = T(x).lfwe define

R(x)

=fox log(.A(y)lp,(y)J dy

( 18)

then we deduce from expression (15) that

F(c) =

e"R(c)+o(~<)

( 19)

for large K. We thus see that F( c) grows exponentially fast with K. Indeed, the same holds true for the occupation times:

T(x) =

e"R(x)+o(~<)

(20)

for any fixed x. Evaluations of the type of (19) and (20) are what we shall see in Chapter 22 as large deviation evaluations. They can be refined, but the only more exact relation which we shall need is

1j = (PJI >..o)

[t

J.i.IJ.l- 2 ... J.i.k-ll k=l >..1.>.2 ... >-k-1

~ SpJI >.o,

(21)

say, where

S= t[J.i.IJ.i.2···J.i.k-l] k=l .A2.A2 ... Ak-1

(22)

In the next section we shall interpret SI >.o as the expected time taken for a population which has been extinguished to be reseeded and grow to viability. Since the terms in the sum S decline exponentially fast with increasing k, the relative error in approximation (21) is exponentially small.

6 THE UNDERSTANDING OF THRESHOLD CHOICE

199

The scaling argument will again imply that 01 = JJ.J I >.1 varies only slowly if K is large, with the implication from (21) that Td-J "'

Td()~

for fixedj; an an~§Ue of the corresponding assertion for 1r1 in the last section. If we defme Text = I:j.. 1 1), the expected time to extinction, then (23) so that 1 - Bd is a measure of the proportion of time before extinction which is actually spent at threshold Moreover, since :E1 ..;; /<JC 1j IText decreases to zero with increasing K we can just as well interpret Text as the expected time needed to escape permanently from any neighbourhood of d. In other words, the time to extinction is asymptotic to the time at which harvesting becomes commercially nonviable. Suppose that 11.!1 is the actual total yield (on the x-scale) before extinction and :Y the actual time to extinction, so that these are random variables with respective expectations F(c) and Text (conditional on a start from x =c). Then a closer analysis shows that in fact 11.!1 I F(c) and :Y IText converge to unity in almost any stochastic sense as K -+ oo. The point is that x recurs to c a great many times before it ultimately drops to zero, and the contributions to 11.!1 or :Y from each of these excursions away from c and back are independent and identically distributed random variables. The consequence is that the expectation of the average rate of return before extinction, 11.!1 I :Y, can be replaced by the ratio of expectations:

E(11.!1 I:Y)

~

E(11.!1)IE(:Y) = F(c)IText,

the relation becoming exact in the limit of large evaluation

F(c)IText

= K-

FdiText::::: K- 1~(1 - Bd)

1

K.

1

Relation (23) then implies the

= K- (>.d- JJ.d) =

>.(c)- JJ.(c) =a( c) (24)

for the expected average return over the period before extinction. This demonstrates that the expected rate of return averaged over the time before extinction converges to the equilibrium rate of return for the deterministic process as K -+ oo.

6 THE UNDERSTANDING OF THRESHOLD CHOICE Expressions (16) and (24) are again maximal for c near to x 0 and Xm respectively. We see that the discrepancy is not a consequence of differing assumptions, because these two evaluations have been made for the same process; one for which ultimate extinction is certain. It is a consequence of differing criteria. To

200

SOME STOCHASTIC EXAMPLES

ask for maximal total return and to ask for maximal average return over the time to extinction (of stock or, almost equivalently, of commercial viability) are two very different things. They differ so because the exponential dependence of Text upon K means that it varies extremely rapidly with c, whereas the average return (24) varies only moderately. The maximisation of expected total return then amounts virtually to the maximisation of expected survival time, with the rate of return playing a role which actually becomes ever less significant as K increases. Indeed, the recommendation is virtually that one should not harvest. In fact, both criteria are extreme, one taking the yield rate before extinction and the other the time to extinction as virtually the sole consideration. A balanced criterion would be one which chose the threshold c to maximise yield rate 'Y =a( c) subject to a prescribed lower bound on Text· Since this last expression depends exponentially upon K (which is what induces the sensitivity of its dependence upon c) one might consider rather the normalised expression

L(c) = lim

K-l

log

Text·

lt-CXl

This in fact bas the evaluation

L(c) = { R(c) R(xo)

(c

~

xo) (c ~ xo)

(25)

where R is indeed the function defined in (18). The two quantities a( c) and L(c) vary jointly with threshold cas indicated in Figure 2; a(c) increases with cup to the value Xm and declines thereafter; L~) increases with c up to the value xo and is constant thereafter. Thus, if one prescribes the value of Text as at least Tmin• then L(c) must be at least K- 1 log Tmin· For small enough K one will thus have to take xo as threshold, but as K increases the recommended threshold will decrease quite quickly until it reaches Xm. L (c)

Figure 2 The characterisers ofperformance for a stochastic fishing model operating at threshold c in the asymptotic (near-detenninistic) case. The graphs ofa( c) the average yieldrate, and ofL(c) the normalised logarithm ofexpected extinction time, as functions ofc.

201

7 CONTINUOUS-TIME LQG PROCESSES

Interestingly, one can reach this same view by returning to the first stochastic model of Section 4: the birth-death model without extinction. It follows from (21) that we can write expression (13) for the yield rate in the steady state as

..\(c)Td 'Y;:::; (S/..\o) +Text;:::; a(c)[l + S/..\oText)],

(26)

where the terms neglected are o(l) inK. Here Text is now to be interpreted as the time to first extinction for this immortal population. Now, ..\o is the rate at which a population which has been reduced to zero is restarted by some external mechanism, so that ..\01 is the expected 'seeding' time. This seeding does not amount to an effective restart until numbers have been brought up to viability; we can regard (S / ..\o) as the expected time needed for this to occur: the expected restart time. Let us then denote it by Trcs, so that expression (26) for 'Y becomes 'Y;:::; a(c)[l + (Tres/Text)r'.

This is then approximately a(c) or a(c)(Text/Tres) according as to whether the restart time is small or large relative to the extinction time. Since Tres is independent of c, these two extreme evaluations are equivalent to those obtained for the two extreme criteria in the case when extinction was possible. As one varies the value assumed for Tres one moves monotonically between these two extremes, just as one did by varying the prescribed value Tmin in the constrained optimisation of yield rate. Ifwe consider discounted criteria then the possibility of extinction is seemingly irrelevant, because any accounting horizon is orders of magnitude smaller than a prudent lower bound for the extinction time-the conservation horizon. As above, then, the constraint of such a lower bound must be imposed explicitly. That is, suppose that a is the discount rate and that F(c, a) is the discounted future yield at an operating level (and threshold) of c, calculated on the basis of a deterministic model. Then one should choose c to maximise F(c, a), subject to prescription of a lower bound Tmin on Text· As in the undiscounted case above, the effect of this constraint will weaken rapidly as the scale parameter K increases, and the model approaches determinism.

7 CONTINUOUS-TIME LQG PROCESSES AND PASSAGE TO A STOPPING SET One of the most tractable controlled diffusion processes is, as one might expect, the continuous-time version of the LQG regulation problem treated in Section 1. We assume the quadratic cost structure of Section 2.8 but modify the plant equation to

x= Ax+Bu+ f

(27)

SOME STOCHASTIC EXAMPLES

202

where e is white noise of power matrix N. The dynamic programming equation is then

inf[c(x, u) II

+ F1 + Fx(Ax + Bu) +!

with terminal condition F( x, h)

tr(NFJCJC)] = 0

(t
(28)

= ~ x Tn (h)x. One readily verifies the solutions

F(x, t) = !xTTI(t)x + 6(t),

u(t) = K(t)x(t)

(29)

for value function and optimal control, where TI and K are exactly the matrices of the deterministic treatment of Section 2.8. The term 6(t), representing the future cost attributable to process noise, has the evaluation

(30) in analogue to the discrete-time relation (5). For a modification from the pure LQG form, consider a stochastic version of the first-passage problem of Exercise 2.6.2, in which one attempted to reach one end or the other of the interval [0, 1] with the optimal compromise between economy of time and of control effort. As first-passage problems go this is about the simplest, but it raises a number of interesting points. We asume the model modified only in that the plant equation becomes .X = u + e, where e is scalar white noise of power N. One's intentions are then to some extent frustrated by random disturbances to the path. The dynamic programming equation (28) now becomes inf[! (L + Qrr) u

+ uFx + !NFxx] =

0

(xI= 0, 1).

(31)

This holds up to termination, which occurs on passage into either of the values x = 0 or x = 1, with respective terminal costs

F(O} =Co,

F(1)=C,.

(32)

We can actually perform the minimisation in (31) to deduce U

= -Q- 1Fx

(33)

and the reduced version (34} of the dynamic programming equation. The non-linear equation (34) can in fact be linearised by the transformation

'1/J(x) =

e-F(x)/QN

to the form QN 2'1/lxx - L'l/J = 0.

7 CONTINUOUS-TIME LQG PROCESSES

203

This has the general solution

,P(x) = c1e"'x

+ Cze-=

(0 ::::;;

X ::::;;

1)

(35)

where a= JLjQN2 and the coefficients c; are determined from the boundary conditions (32). In contrast to the piecewise-linear expression ofF (x) for the deterministic case derived in Exercise 2.6.2, F(x) now has a single analytic expression over the whole interval 0 ::::;; x ::::;; 1. Mathematically, the fact that F now obeys a secondorder equation means that both boundary conditions can be met by a single analytic expression. Physically, the fact that noise can carry one off the intended course means that one might end up at either boundary point, and induces a blurring of costs which makes F smooth. The evaluation of Fis now both greater and smoother than in the deterministic case, as indicated in Figure 3. In case (a) there was formerly (i.e. in the deterministic case) a change in goal at an intermediate value of x and there still is, at the point at which Fis maximal. However, F is now smoothly differentiable at this break-point, because noise can carry one across it in either direction. In case (b) Fx increases as one approaches the undesirable termination point x = 1, reflecting one's increasing efforts to avoid it. Avoidance is never certain,

(a)

0

1

X

(b)

0

X

Figure 3 The stochastic version of Fig. 22 The upper curves give the value jUnction F(x)for the stcchastic version ofthe optima/frrst-passage problem of Ex. 2.6.2 The two cases correspond, as previously. to those in which the optimal deterministic policy has a break-point in the interior ofthe continuation region or on its boundary.

SOME STOCHASTIC EXAMPLES

204

however, as reflected in the fact that Fis now continuous at this boundary. This might be termed the 'fly-paper' effect, if one regards the moving point as a fly trying to avoid the fate of becoming stuck on fly-paper at x = 1. A deterministic fly, whose path is fully under its own control, can actually approach arbitrarily close to the fly-paper with impunity, knowing that he can avoid entrapment Hence the discontinuity of Fat x = 1, reflecting the difference between being close and being stuck. A stochastic fly cannot guarantee his excape; the nearer he is to the paper, the more certain it is that he will be carried on to it. Hence the continuity of Fat x = I in this case. This also explains why the fly tries so much harder in the stochastic case than in the deterministic case to escape the neighbourhood of the fly-paper (as manifested by the greater magnitude in the stochastic case of Fx, and so of -u for x near 1). One may say that the penalty of ending on the fly-paper 'propagates' into the free-flight region in the stochastic case, causing the fly to take avoiding action while still at a distance from the paper. A rigorous treatment of the dynamic programming equation in cases where the postulated differential Fx does not exist everywhere can be based on the concept of a 'viscosity colution' (see Fleming and Soner, 1992). In at least some cases this can be envisaged as obtained by the addition of a little stochasticity to a previously deterministic problem. As above, this has the effect of smoothing the solution, and the limit of this stochastic solution as one approaches the deterministic case is in fact the correct deterministic solution. This approach must be regarded as justificatory rather than constructive, however; we shall see in the large deviation treatment of Chapter 22 that the boot is often on the other foot, and that the stochastic solution is, in a sense, approximated by a deterministic solution. To return to the stochastic fly: it is only the difference in penalties at the two termination points which matters, so we may as well set Co = 0. Suppose we now let C1 tend to +oo, SO that absorption at X = 1 carries infinite penalty and is tO be avoided at all costs. One would imagine that the controlled process would then become improper in some sense, because there is a contingency, presumably of positive probability, which carries infinite penalty. This is not the case, however. The limit process is perfectly proper with a control rule which, although also perfectly proper, becomes vigorous enough as x = 1 is approached that absorption at this point in fact has zero probability. (Stronger than that: the expectation of cost due to a contingency of infinite cost but zero probability could be anything. In fact, it is zero in this case.) In this limit we must have '1/J(O) = 1, '1/J( 1) = 0, and the solution (35) which satisfies these boundary conditions is e

-F(x)fQN

a(l-x)

= '1/J(x)

a(x-1)

e - e =-e"- e-<>

(36)

The optimal control rule is U

= -Q-I Fx =

e<>(l-x)

N'I/Jx/'1/J

= -.JL!Q e<> (I

-X

)

+ e<>(x-1) -

e<>

(

X-

I)·

7 CONTINUOUS-TIME LQG PROCESSES

205

For N small (and so a: large) u has approximately the constant value - VLfQ for x between zero and a value bounded away from 1. However, as x j 1 then u ! - oo, indicating a strenuous effort to avoid passage to x = 1. That this effort is successful is indicated by the fact that F(x), as given by (31), is finite for x < 1. This example is a particular case of a class of controlled processes which we shall investigate in Chapter 24. There is another point which comes out of this example. Suppose, for a general controlled diffusion, that F(x, t) is such that the fmal term tr[N(x, u, t)Fxx] in the dynamic programming equation (28) is independent of x and u, so that it can be written as a function J.L(t) of time alone. Then the value function F(x, t) will differ from that for the deterministic case Fdet(x, t) only by a function of time:

F(x, t)

= Fdet(X, t) +

1

J.L(r) dr,

and the optimal control will be the same as for the deterministic case. This was true for the LQG example; it would be approximately true in the neighbourhood of a stable equilibrium of the deterministic process. One may expect it to be approximately true if N is small, but to show failure at points where Fxx is large. This is exactly what we observe in the first-passage example above. The optimal policy in the stochastic case deviates seriously from that for the deterministic case exactly at those points where Fxx is large for some reason, which correspond to those points at which Fx failed to exist in the deterministic case. These were points at which there was a discontinuity in policy, either chosen or forced. These matters are related to the search for a 'stochastic maximum principle' which we shall join in Chapter 23. Exercises and comments

(1) Note that the solution of (34) for x ~ 0 or x ~ 1 will equally well be of the form (35), but with the c-coefficients now determined by the boundary condition at one stopping point and a growth condition at infinity. Determine it. (2) Optimal stopping and tangency conditions. The value function will satisfy matching conditions at a boundary where the process changes character in some sense, and will in addition satisfy optimality conditions if the position of the boundary itself can be optimised. Even a non-rigorous discussion of the forms such conditions can take can be quite intricate, as a great deal depends upon continuity of the path, the directions in which movement is possible, etc. (see Whittle 1983a, pp.l02-108 and 201-207). However, an important case which appears in various guises is that of optimal stopping, where the optimiser can choose between the actions of continuation or termination, so that the stopping set itself can be optimised The example of Section 2 was one such. Consider a discrete-time formulation in which the value

206

SOME STOCHASTIC EXAMPLES

function F obeys F = LF in the continuation region C and F = IK in the stopping region 9J. The state-variable argument xis understood, and we shall suppose, for simplicity, that there are no actions to be optimised other than the continuation/ stopping decision. We shall suppose an infinite-horizon time-invariant formulation. The optimality equation is then F = min(LF, IK), with F

= LF < IK

(~);

where the brackets indicate the set of x within which the assertion holds. Here ~ and 9J have been chosen optimally and we have closed 9J by choosing the stopping option in cases of indifference. Define the accessible stopping set g;•; that part of 9J which can be reached by a single transition from some part of~- The essential matching condition is then that F = IK in ~·. Define also the continued cost F*; the solution ofF* = LF* (holding everywhere) which agrees with Fas much as it can: on ~ u g;• (which is the only set of x which matters). We have then, from the relations above and the definition ofF*,

F* < IK

(~);

F* = IK

(9J*);

LF" :::;; LF

(9J*).

These relations imply optimality conditions which imply a tangency condition, that F* - IK should have zero derivative at the stopping point, if we can go to a continuous limit in space and time. Suppose for example that x takes integer values j; let us write F ( x) as Fj etc., and suppose that LFJ = Cj + pjF;- 1 + qjF} + rjFj+ 1 where (p, q, r) constitute a distribution. A continuous-limit version of this assumption could thus yield either a deterministic process or a diffusion process. Suppose that the value j = k marks an optimal stopping boundary, in that k - 1 is in~ and k in~- We suppose that r > 0, so that movement in the direction of increasingj is possible. Then the last set of relations above, asserted at k - 1, k and k respectively, yields Fk-1

< IKk-1,

F; = IKb

Fk+l ~ IKk+l·

That is, F* - IK has a turning point at the stopping boundary, which will imply zero derivative at this point in a continuous-limit version. Note that we have already demonstrated validity of the tangency condition at the optimal threshold value for the deterministic harvesting model of Section 2.7.

8 THE HARVESTING EXAMPLE: A DIFFUSION MODEL Suppose the deterministic harvesting equation (11) modified to the stochastic differential equation

.X= a(x)- u+ t:

(37)

where t: is white noise of power v( x) /"'"We introduce the factor K. so that the model can be made to approach determinism by allowing K. to become large. The

207

8 THE HARVESTING EXAMPLE: A DIFFUSION MODEL

introduction of K in this way amounts to a scaling assumption which, as we shall see in Chapter 22, fits into a general pattern consistent with the scaling of the birth-death process in Section 4. If the unharvested version of the model (ie. that for which u O) has an equilibrium distribution then this has probability density

=

(38) where

R(x)

=2

jx a(y)v(yr

1

dy.

(39)

(see Exercise 1). We assume that x is kept to the positive half.axis by some condition such as v(x) l 0 as x l 0. If we allow harvesting for x > c then it is best to assume initially that the harvesting rate is finite and equal toM, say, so that a(x) is modified to a(x)- Mfor x >c. We thus deduce

Theorem 10.8.1 The average rate ofreturn for the diffusion model (37) is

h

7r(x)e-,;MQ(x) d.x fo 7r(x) dx + 7r(x)e-,;MQ(x) dx M

'"f=

c

r

(40)

where 71" has the evaluation (38) and Q(x) =

21x v(yr

1

dy.

(41)

Evaluation (40) becomes 7r(c)v(c) 'Y- 2fo7r(x) dx

(42)

in the limit oflarge M. Evaluation (40) is direct and (42) the limit version of it for large M. These expressions must be maximised with respect to c. Let us confirm what one would expect: that they approach the known deterministic value a( c) asK - oo and the model approaches determinism. If we assume R(x) increasing in x for x < c then we find the evaluation of expression (42)

'Y "'! v(c)R'(c) =a( c). See Exercise 2 for the case of a finite harvest rate. Suppose however that extinction is possible, in that the state value x = 0 is absorbing for the process (37). We follow an analysis similar to that of Section 4. If F(x) is the total expected return before extinction conditional on an initial value x then this obeys the dynamic programming equation

208

SOME STOCHASTIC EXAMPLES

u +(a- u)F' + (2K-r 1vF" = 0.

(43)

where the harvest rate u takes the values 0 or m according as x ::;:;; c or x > c.

Theorem 10.8.2 Suppose that extinction is certain for the harvested process. Then the relevant solution of (43) is F(x)

=

eiCR(c)

(fox e-~
(0::;:;; x::;:;; c),

(44)

where F' (c) has the evaluation F'(c) =2M

1oo e~e[R(y)-R(c)-MQ(y)lv(y)-l dy.

(45)

and R(x), Q(x) are the functions defined in (39) and (41). If M is infmite then F'(c) = 1. Proof We obtain solution (44) as we obtained (15); by solution of (43) for x ::;:;; c. The evaluation (45) then follows by solution of (43) for x > c and appeal to the condition of certain extinction. The last assertion follows from this evaluation. D The analogue of the final conclusion of Section 4 then follows.

Theorem 10.83 The optimal value of c is that maximising the expression ei c. Show then that expression (40) has the evaluation M/(MfZ- R') 1 ;::;; (1/ R')

+ Ij(M(!- R') =a,

for large "" where all functions are evaluated at c.

9 A MODEL WITH A SWITCHING ENVIRONMENT

209

9 THE HARVESTING EXAMPLE: A MODEL WITH A SWITCHING ENVIRONMENT As indicated in Section 9.2, we can use a piecewise-deterministic model to represent the effects of environmental variation. Suppose that the model has several environmental regimes, labelled by i = 1, 2, . . . . In regime i the population grows deterministically at net rate a;(x), but transition can take place to regime h with probability intensity IW;h· This is then a model whose stochasticity is of quite a different nature to that of the birth-death or diffusion models of Sections 4 and 8. It comes from without rather than within, and represents what is termed 'environmental stochasticity', as distinct from 'demographic stochasticity'.Whether this affects conclusions has to be determined. The equivalent deterministic model would be given by equation (11) but with

(46) where p; is the steady-state probability that the system is in regime i. The model converges to this deterministic version if transitions between regimes take place so rapidly that one is essentially working in an 'average regime: This occurs in the limit oflarge, "'• so that l'i. again appears as the natural scaling parameter. A fixed threshold would certainly not be optimal for such a multi-regime model. It is likely that the optimal policy would be of a threshold nature, but with a different threshold in each regime. Of course, it is a question whether the regime is known at the time decisions must be made. Observation of the rate of change of x should in principle enable one to determine which regime is currently in force, and so what threshold to apply. However, observation of x itself is unreliable, and estimation of its rate of change fraught with extreme error. If one allowed for such imperfect observation then the optimal policy would base action on a posterior distribution (i.e. a distribution conditional on current observables) of the values of both x and i (see Chapter 15). An optimal policy would probably take the form that harvesting effort should be applied only if an estimate of the current value of x exceeded a threshold dependent on both the precision of this estimate and the current posterior probabilities of the different regimes. We shall consider only the fixed-threshold policy, desperately crude though it must be in this case, and shall see how the optimal threshold value compares with that for the equivalent deterministic model. We shall consider a two-regime case, which is amenable to analysis. A value of x at which at (x) and a2 (x) have the same sign cannot have positive probability in equilibrium. Let us suppose then that at (x) = >.(x) ~ 0 and a2(x) = -f.J-(x) ~ 0 over an interval which includes all x-values of interest. We shall set llt2 = 111 and 112t

=

112.

210

SOME STOCHASTIC EXAMPLES

Suppose initially that extinction is impossible, so that the aim is to maximise the expected rate of return 'Y in the steady state. We shall suppose that the maximal harvest rate M is infinite. For the deterministic equivalent of the process we have, by (46),

a(x) = v2.A(x) - v1JL(x) . VJ

(47)

+1/2

We shall suppose that this has the character previously assumed, see Figure 2. We also suppose that JL(x) = 0 for x ~ 0, so that xis indeed confined to x ~ 0. The question of extinction or non-extinction is more subtle for this model. Suppose, for example, that .A(O) = 0 (so that a population cannot be replenished) and that JL(x) is bounded away from zero for positive x. Then extinction would be certain, because there is a non-zero probability that the unfavourable regime 2 can be held long enough that the population is run down to zero. For extinction to be impossible in an isolated population one requires that JL(x) tends to zero sufficiently fast as x decreases to zero; the exact condition will emerge. Let p 1(x) denote the probability/probability-density of the ilx pair in equilibrium. These obey the Kolmogorov forward equations

(0

~ x


The second equation continues to hold at x = c, but the first does not. If (i, x) = (1, c) then x cannot continue to increase, since all excess over c is immediately cropped. The state (1, c) can be left only by transition of i from 1 to 2; a discrete event which occurs with intensity K.v 1• The effect of this is that an atom of probability forms at (1, c). If this has magnitude q in equilibrium then the balance of probability flux into and out of the state implies the equation

Theorem10.9.1 Pt(x)

(49)

(x =c).

Api = K.VJq

(i) The equilibrium distribution of(i, x) is given by

= k.A(x)- 1e"R(xl,

pz(x) = kJL(xt 1et
q = k(K.vJ)- 1e"R(c)

(50)

where k is a normalising constant and R(x)

= fox[V2JL(y)- 1 -

vt.A(yt 1J dy.

(51)

(ii) The average return under the c-thresho/d policy is 'Y =

whereg(x)

=

K.VJ

.A( c) K.R(c)]g(x) dx + 1

foc exp[K.R(x) -

.A(x)- 1 + JL(xt 1•

(52)

211

9 A MODEL WITH A SWITCHING ENVIRONMENT

Proof The balance of probability flux at x in equilibrium implies that >..pt

= ILP2

(O:s;;x
(53)

Using this equation to eliminate P2 from the first equation of (48) we deduce an equation I

Pt

lt!l] AV2) + 'I + T - -;- PI ()..'

= 0.

This leads to the solution for p 1 asserted in (50). The solutions for p 2 and q then follow from (53) and (49). Assertion (ii) follows from (50) and the relation 'Y

= qa(c).

0

The condition of non-extinction is just that the distribution (50), when normalised, should not have all its probabili~ mass at the origin. This sets a bound on the rate at which 1121-L(x)- 1 - v1>-.(x)- grows as x! 0.

Theorem 10.9.2 In the deterministic limit expression (57) reduces to 'Y =a( c), with a(x) specified by (52} Proof We may suppose that a(x) ;;:=: Oin the interval [0, c), so that R is increasing in the interval In the limit oflarge tt expression (52) then becomes

y"'

.).. (vi/ R')(>-. -I + 1-L- 1)

+1

= vz.A111

!IJIJ.

+ 112

=a

with the argument c understood throughout.

0

The optimal value of c thus converges to Xm in the deterministic limit, as one would expect Suppose now that extinction is possible, in that .A(O) = 0 and that the population can be run down to zero in a bounded time in the unfavourable regime 2. It will then in fact be certain under communicationlboundedness conditions. Let F1(x) denote the expected total return before absorption conditional on a start from (i, x).We have then the dynamic programming equations

(0 < x< c). Since escape from x = 0 by any means is impossible we have Ft (0) However, the real assertion is that

(54)

= F2 (0) =

0.

(55) where F1(0+) = limxLoF1(x) and¢ is an as yet undetermined positive quantity. The point is that, if x is small and positive, then it has time to grow in regime 1 and time to decline to zero in regime 2 (before there is a change in regime). The

SOME STOCHASTIC EXAMPLES

212

second equation of (54) continues to hold at x = c, but for the first we must substitute (56) Relation (56) follows from the fact that escape from (1, c) is possible only the transition of i from 1 to 2; this takes an expected time (~~:11 1 )- 1 during which return is being built up at rate >.(c). Theorem 10.9.3

The value functions F; (x) have the evaluations

F1 (x) = ¢

+ ((3/vz)

Fz(x) = ((3/vi)

1x

1x >.(y)- e-~
1

J.L(Y)- e-"'R(y) dy

dy, (0

~ x
(57)

where¢= (3/ 11:,

(3 =

llj""l >.(c)e"'R(c)

(58)

and R(x) is the function (51) so normalised that R(O) = 0. Proof We deduce from equations (54) that

(59)

Using this relation to eliminate F 2 from the first equation of (54) we obtain an equation for F( similar to that for p 1 (x) above, with solution Ff = (r/vz)>. -Iei.(c)e"'R(c)_ For 11: large this amounts to the maximisation of R(c), i.e. to the equation a(c) = 0, with a~) having the determination (47). That is, the optimal threshold again approaches the value x 0 . To be exact, the stationarity condition with respect to cis

If we assume that >.(k) is increasing with x then we see that, at least for sufficiently large 11:, the optimal threshold c lies somewhat above the value xo. For the two previous models it lay below. It is in this that the nature of the stochasticity (environmental rather than demographic) reveals itself. In the previous examples

9 A MODEL WITH A SWITCHING ENVIRONMENT

213

there would virtually never have been any harvesting if c had been set above the equilibrium value .xo. The effect in this case is that x can indeed rise sufficiently above x 0 during the favourable regime 1, and one waits for this to happen before harvesting. Notes on the literature The fact that the threshold c should seemingly be set at the unharvested equilibrium level x 0 if one sought to maximise the expected total return before a point of certain extinction was first observed by Lande, Engen and Saether (1994, 1995). for the case of a diffusion process. The analysis of Section 8 expands this treatment. The material of Sections 4-6 and 9 appears in Whittle and Horwood (1995).

CHAPTERJJ

Policy Improvement: Stochastic Versions and Examples 1 THE TRANSFER OF DETERMINISTIC CONCLUSIONS In Chapter 3 we considered patterns of infinite-horizon behaviour and the technique of policy improvement for the deterministic case. All conclusions reached there transfer as they stand to the stochastic case if we appropriately extend the definitions of the forward operators L and !l' and their continuoustime analogues. As in Chapter 3, attention is restricted to the state-structured time-homogeneous case. In discrete time we define the conditional expectation operator

E(u)cjl(x)

= E(cjl(xt+dlxt =

x, Ut

The forward operators L(g) and !l' for the policy respectively are then defined by

L(g)cjl(x)

= u].

g(oo)

and an optimal policy

= c(x,g(x)) + {3E(g(x))¢(x),

!l'cjl(x) = inf[c(x, u) u

(1)

+ {3E(u)¢(x)].

(2)

(3)

The corresponding dynamic programming equations for the value functions Vs = Vse((oo)) and Fs then again take the forms

(s > 0),

(4)

with the x-argument of these functions understood. The material of Chapter 3 is now valid as it stands with these extended definitions. Explicitly: Theorem 3.1.1, asserting the monotonicity of the forward operators and the monotonicity (in s) of the value functions under appropriate terminal conditions, still holds. If we assume the instantaneous cost function c(x,u) non-negative then Theorems 3.2.1-3.2.3 on total cost still hold, as does Theorem 3.5.1 on policy improvement The continuous-time analogue of the operator (1) is the infinitesimal generator A( u) for the controlled process, defined in (8.15). In terms of this the stochastic versions of the differential forward operators M (g) and Jt of Section 3.1 take the forms

216

POLICY IMPROVEMENT: STOCHASTIC VERSIONS

M(g)rf>(x) = c(x,g(x))- a¢(x) + A(g(x))rf>(x) vltrf>(x) = inf[c(x, u)- a¢(x) + A(u)rf>(x)].

" The assertions of Chapter 3 for the continuous time case then also transfer bodily. Exercises and comments

(1) We can supplement the example of instability given in Exercise 3.2.1 by the classic stochastic example of the simplest gambling problem. Suppose a gambler has a capital x which takes integral values, positive as well as negative, and has the choice of ceasing play with reward x, or of placing a unit stake and continuing. In this second case he doubles his stake or loses it, each with probability 112. If his aim is to maximise expected reward then the dynamic programming equation is

Gs(x)

= max{x,!(Gs-t(X -1) + Gs-t(x+ l)]}

(s > 0)

where Gs(x) is his maximal expected reward if he has capital x with s plays remaining. If at s = 0 he only has the option of retiring then Go (x) = x, and so Gs(x) = x for all s. However, the infinite-horizon version of this equation also has a solution G(x) = +oo. If the retirement reward xis replaced by min (a, x) for integral a then the equation has a solution G(x) =a for x ~a. This corresponds to the policy in which the player continues until he has a capital of a (an event which ultimately occurs with probability one for any prescribed a) and then retires. The solution G(x) = +oo corresponds to an indefinitely large choice ofa. Investigate how infinite-horizon conclusions are modified if any of the following concessions to reality is admitted: (i) debt is forbidden, so that termination is enforced in state 0; (ii) rewards are discounted; (iii) a constant positive transaction cost is levied at each play. (2) An int~resting example in positive programming is that of blackmail. Suppose there are two states: those in which the blackmailer's victim is compliant or resistant. Suppose that, if the blackmailer makes a demand of u (0 ~ u ~ 1), then a compliant victim pays it, but becomes resistant with probability u2. A resistant victim pays nothing and stays resistant If Gs is the maximal expected amount the blackmailer can extract from a compliant victim in s further demands, then Go = 0 and

Gs+l

= sup(u + (1- ~)Gs] = 1/J(Gs), "

1

say. Here the optimising value of u is the smaller ofl and (2Gs f and 1/J( G) is 1 or G + 1j (4G) according as G is less than or greater than!Show that Gs grows as s 112 and the optimal demand decreases as s- 112 for large s. There is thus no meaningful infinite-horizon limit, either in total reward or in

2 AVERAGE-COST OPTIMALITY

217

optimal policy. The blackmailer becomes ever more careful as his horizon increases, but the limiting policy u = 0 is of course not optimaL

(3) Consider the discounted version of the problem, for which the infinite-horizon reward G obeys G = ,P({3G). Show that, if! ~ {3 < 1, then G = (2y'{3(1 - /3) )- 1 .

2 AVERAGE-COST OPTIMALITY The problem of average-cost optimisation is one for which the stochastic model in fact shows significant additional features. Because of its importance in applications, it is also one for which we would wish to strengthen the discussion of Chapter 3. We observed in the deterministic contexts of Section 2.9 and 3.3 that one could relatively easily determine the value of control at an optimal equilibrium point, but that the determination of a control which stabilised that point (or, more ambitiously, optimised passage to it) was a distinct and more difficult matter. The stochastic case is less degenerate, in that this distinction is then blurred. Consider the case of a discrete state space, for simplicity. Suppose that there is a class of states 91 for which recurrence is certain under a class of control policies which includes the optimal policy. Then all these states will have positive probability in equilibrium (under any policy of the class) and, in minimising the average cost, one also optimises the infmite-horizon control rule at every state value in Pit. Otherwise expressed: since equilibrium behaviour still implies continuing variation (within 9l) in the stochastic case, optimisation of average cost also implies optimisation against transient disturbance (within Pit). These ideas allow us to give the equilibrium dynamic programming equations (3.9) and (3.10) an interpretation and a derivation independent of the sometimes troublesome infinite-horizon limit. Consider the cost recursion for the policy g(ool: 'Y + v = L(g)v,

(5)

where 'Y is the average cost under the policy and v(x) the transient cost from x, suitably normalised. (These are both g-dependent, but we take this as understood, for notational simplicity.) Suppose that the state space is discrete and all states are recurrent under the policy. Then 'Y can be regarded as an average cost over a recu"ence cycle to any prescribed state (see Exercise 2~ Equation (5) can be very easily derived in this approach. which completely avoids any mention of infinite horizon limits, although it does imply indefinite continuation. The natural normalisation of vis to require that E[v(x)] should be zero, where the expectation is that induced by policy in equilibrium. This is equivalent to requiring that the total transient cost over a recurrence cycle should have zero expectation. If there are several recurrence classes under the policy then there will be a separate equation (5) for each recurrence class. These recurrence classes are

218

POLICY IMPROVEMENT: STOCHASTIC VERSIONS

analogous to the stable equilibrium points of the deterministic case (although see Exercise 1). In the optimised case the equation

'Y+f=2f

(6)

has a similar interpretation, the equation holding for all x in a given recurrence class under the optimal policy, and"' and f being the minimal average cost and transient cost in this class. Whether this equation is either necessary or sufficient for optimality depends again upon uniqueness questions: on whether the value function could be affected by a notional closing cost, even in the infmite-horizon limit. The supposition of a non-negative cost does indeed imply thatfmay be assumed bounded below, and so that the relevantfsolution of (6) is the minimal solution exceeding an arbitrary specified bound However, one must frequently appeal to arguments more specific to the problem if one is to resolve the matter.

Theorem 11.21 Suppose that (6) can be strengthened to

"1+ f

=

2/ =

L(g)f.

(7)

Then

(8) where dn = n- 1 (/(xo)- E[f(xn)!Wo]), foranypolicyn withequalityifn = g(ool. If dn ~ 0 with increasing nfor any 1r andfor any Wo one can then assert that the policy g( 00 l is average-cost optimal. Proof Relation (7) is a strengthening in that the second equality asserts that the infimum with respect to u implied in the evaluation of !Rf(x) is attained by the choice u = g(x). If we denote the value of c(x, u) at timet by c1 then (7) can be written "f + f(x,) ~ E.,..[ct

+f(xt+I)! Wt]

for any policy 7; with equality if 11' = g(oo). Taking expectations on both sides conditional on Wo and summing overt from 0 to n - 1 we deduce that "l' +f(xo) .; E, (

~ c, +f(x.) IWo}

whence the assertions from (8) onwards follow.

0

2 AVERAGE-COST OPTIMALITY

219

If cis uniformly bounded in both directions then we may assume the same off, and it is then clear that dn has a zero limit. In other cases special arguments are required to establish the result. Exercises and comments

(1) One might think of the recurrence classes as being analogous to the domains of attraction of the deterministic case, but this is not so. The notion that occupation of the states should actually recur is important: in the deterministic case this would true only of the points of the so-called attractor. Furthermore, the transient states in the stochastic formulation may communicate with many of the recurrence classes. A different set of ideas altogether (and one with which we are not concerned for the moment) is encountered if a process is 'scaled' so that it approaches a deterministic limit as a parameter "' is increased. In this case states which are in the same recurrence class may communicate ever more feebly as "' is increased, until they belong to distinct domains of attraction in the deterministic limit.

(2) A controlled Markov process with a stationary Markov control policy is simply a Markov process with an associated cost function. Suppose the statespace discrete, that c( x) is the instantaneous cost in state x and that p( x, y) is the transition probability from state x to state y. Suppose we define a modified cost function c(x) -1 and define v(x) as the expected sum of modified costs over a path which starts at state x and ends at first entry (or recurrence) to a specified state, say state 0. Then

v(x)

=

c(x) -1 + I>(x,y)v(y). yfO

Suppose we choose 1 so that v(O) = 0, i.e. so that the sum of modified costs has zero expectation over a recurrence cycle. Then this last equation becomes simply

v(x) = c(x) -1 + LP(x,y)v(y).

(9)

y

which we can identify with (5). We have 0 = v(O) sum is over a recurrence cycle to state 0. That is,

= E[L;,(c(x,) -1)], where the

where r is the recurrence time; this exhibits ~f as the average cost over a recurrence cycle. On the other hand, we deduce from (9) that 1 = L:x 1r(x)c(x), where {1r(x)} is the equilibrium distribution over states. The second interpretation is the 'infinite horizon' one; the first is purely in terms of recurrence, with no explicit appeal to limiting operations.

POLICY IMPROVEMENT: STOCHASTIC VERSIONS

220

(3) The blackmail example of Exercise 1.2 showed infmite total reward, but is not rephrasable as a regular average-reward problem. The maximal expected reward over horizons grows as s 112 rather than s, so the average reward is zero and the transient reward infmite. Indeed, an average reward of "Y and a transient reward off(x) are associated with a development

F(x) =

1

2/3 +f(x) + o(l - /3)

of the discounted value function for 1 - /3 small and positive. This contrasts with the (1 - {3) -I 12 behaviour observed in Exercise 1.2. 3 POLICY IMPROVEMENT FOR THE AVERAGE-COST CASE The average-cost criterion is the natural one in many control contexts, as we have emphasised. It is then desirable that we should obtain an average-cost analogue of Theorem 3.5.1, establishin~ the efficacy of policy improvement. 00 We assume a policy g} at stage i. Let us distinguish the corresponding evaluations of -y, v(x), L and expectation E by a subscript i. The policyimprovement step determines Kt+l by !t'v; = L;+ 1 v;. Let us write the consequent equation "Yi + V; ~

ft'V;

= Lt+! V;

(10)

in the form "Yi

+ v; ~ !t'v; + 6 =

L;+l v; + 6

( 11)

where 6 is some non-negative constant If 6 can in fact be chosen positive then one has an improvement in a rather strong sense, as we shall see.

Theorem 11.3. 1 Relation (11) has the implication (12)

where d, = n- 1 (v;(xo)- E;+l[v;(x,)Jxo]).

q d,

--t 0 with increasing n it then follows that: (i) there is a strict improvement in average cost if 6 is positive; vz) average-optimality has been attained ifequality holds in (11).

The proof follows the same course as that of Theorem 11.2.1. Note that, effectively, the only policies considered are stationary Markov. However, we expect the optimal policy to lie in this class.

4 MACHINE MAINTENANCE

221

In continuous time the optimality equation (7) is replaced by 1 = .Af, i.e. 1 = inf[e(x, u) u

+ A(u)f(x)].

( 13)

4 MACHINE MAINTENANCE As an example, consider the optimisation of the allocation of service effort over a set of n machines. The model will be a very simple one, formulated in continuous time. Consider first a model for a single machine which, with use, passes through states of increasing wear x = 0, 1, 2, .... The passage from x to x + 1 has intensity >.for all x, and the machine incurs a cost at a rate ex while in state x, this representing the effect of wear on operation. A service restores the machine instantaneously to state 0. Suppose the machine is serviced randomly, with probability intensity f-L· The dynamic programming equation under this blind policy will be

1 =ex+ >.[!(x +I)- f(x)] + f-L[f(x)- /(0)]

( 14)

if costs are undiscounted. Here 1 is the average cost under the policy and fo) the transient cost in state x. The equation has the unique solution f(x) = exjp,,

1

= >.eff-L.

( 15)

if we make the normalising assumptionf(O) = 0. Suppose now that we haven such machines and distinguish the parameters and state of the ith machine by a subscript i. The average cost for the whole system will then be 1

= L 'Yi =

L >.;c;jf..Li

( 16)

i

with the f-L; constrained by (17)

iff-Lis the intensity of total service effort available. Specification of the f-L; amounts to a random policy in which, when a machine is to be serviced, the maintenance man selects machine i with probability t-L;/ f-L· A more intelligent policy will react to system state x = {x;}. One stage of policy -improvement will achieve this. However, before considering policy improvement, one could simply optimise the random policy by choosing the f-L; to minimise expression (16) subject to constraint (17). One readily finds that this leads to an allocation f-L; ex y'>:ic/. Policy improvement will recommend one to next service the machine i for which

222

POLICY IMPROVEMENT: STOCHASTIC VERSIONS

I)cixi + AJ[jj(xJ+i)- jj(xJ)]}

+ J..L[.fi(O)- f(x;)]

j

is minimal; i.e. for which

Ji(x;) = C;Xi/ j..£; is greatest. If we use the optimised value of J..£; derived above then the recommendation is that the next machine i to be serviced is that_ for which Xt y'CJ>:; is greatest. . Note that this is an index rule, in that an index x,-JCJ):; is calculated for each machine and that machine chosen for service whose current index is greatest. The rule seems sensible: degree of wear, cost of wear and rapidity of wear are all factors which would cause one to direct attention towards a given machine. However, the rule is counter-intuitive in one respect: the index decreases with increasing A;, so that an increased rate of wear would seem to make the machine need service less urgently. However, the rate is already taken account of in the state variable x; itself, which one would expect to be of order A; if a given time bas elapsed since the last service. The deflation of x; by a factor ...;>::;is a reflection of the fact that one expects a quickly wearing component to be more worn, even under an optimal policy. An alternative argument in Section 14.6 will demonstrate that this policy is indeed close to optimal. 5 CUSTOMER ALLOCATION BETWEEN QUEUES Suppose there are n queues of the type considered in Section 10.3. Quantities defined for the itb queue will be given subscript i, so that x;, A;, J..£; and a;x; represent size, arrival rate, service rate and instantaneous cost rate for that queue. We suppose initially that these queues operate independently, and use

(18) to denote the total arrival rate of customers into the system.

However, suppose that arriving customers can in fact be routed into any of the queues (so that the queues are mutually substitutable alternatives rather than components of a structured network). We look for a routing policy 1r which minimises the expected average cost "f.,= E.,[_L; a;x;]. The policy implied by the specification above simply sends an arriving customer to queue iwith probability A;/ A; the optimal policy will presumably react to the current system state {x1}. The random routing policy achieves an average cost of "'(=

~ ~aX ~ "(j= ~--'-'; ; J..£;-A;

(19)

6 ALLOCATION OF SERVER EFFORT BETWEEN QUEUES

223

· (see (10.10)). As in the last section, before applying policy improvement we might -- " ·as well optimise the random policy by choosing the allocation rates At to minimise expression (19) subject to (18), for given A. One readily finds that the optiroal choice is

(20) where 8 is a Lagrange multiplier whose value is chosen to secure equality in (18). So, speed of service and cheapness of occupation both make a queue more attractive; a queue for which a1 I Jl.; > 1I 8 will not be used at all. Consider one stage of policy improvement It follows from the form of the dynamic programming equation that, if the current system state is {x 1}, then one will send the next arrival to that queue i for which

A[.fi(x; +I)-.fi(x;)] + LJJ.Axi)[jj(xJ -1)- jj(xj)]

(21)

I

is minimal. Here the fi, are the transient costs under the random allocation policy, determined in (10.10). That is, one sends the arrival to that queue i for which

1':(

Ji Xt

+ l) - Jl1':(X; ) -_ a1(x 1 +, 1) JJ.j-l'lj

is minimal. If the A; have already been optimised by the rule (20) then it follows that one sends the new arrival to the queue for which ( (x1 + 1) ..;a:Tii;) is minimal, although with i restricted to the set of values for which expression (20) is positive. This rule seems sensible: one tends to direct customers to queues which are small, cheap and fast-serving. Note that the rule again has the index form.

6 ALLOCATION OF SERVER EFFORT BETWEEN QUEUES We could have considered the problem of the last section under the assumption that it is server effort rather than customer arrivals which can be switched between queues. The problem could be rephrased in the form in which it often occlirs: that there is a single queue to which customers of different types (indexed by z) arrive, and such that any customer can be chosen from the queue for the next service. Customers of different types may take different times to serve, so it is as well to make a distinction between service effort and service rate. We shall suppose that if one puts service effort u 1 into serving customers of type i (i.e. the equivalent of a; servers working at a standard rate) then the intensity of service completion is the service rate a1JJ.;. One may suppose that a customer of type i has an exponentially distributed 'service requirement' of expectation JJ.j 1, and that this is worked off at rate a 1 if service effort a 1 is applied.

224

POLICY IMPROVEMENT: STOCHASTIC VERSIONS

As in the last section, we can begin by optimising the fixed service-allocation policy, which one will do by minimising the expression -y=

a;>.,

L. "Yt= L. U;J.L;-At I

I

with respect to the service efforts subject to the constraint

LU;=S,

(22)

I

on total service effort. The optimal allocation is u 1 = J.Lj 1[>..1+ yf8a 1>.1J.L1]

(23)

where 8 is a Lagrange multipler, chosen to achieve equality in (22). An application of policy improvement leads us, by an argument analogous to that of the previous section, to the conclusion that all service effort should be allocated to that non-empty queue i for which J.L;[f;(x,- 1)- f;(x 1)] is minimal; i.e. for which a;J.L 1x 1

v1(x1) = J.L;[f;(x1)- f;(x1 - 1)] = U;J.L; _

>.;

(24)

is maximal. If the fixed rates J.Lt have been given their optimal values (23) then the rule is: all service effort should be concentrated on that non-empty customer class i for which x 1 a1J.Lt/ >.. 1 is maximal It is reasonable that one should direct effort towards queues whose size x 1 or unit cost a1 is large, or for which the response J.Li to service effort is good However, again it seems paradoxical that a large arrival rate >..1 should work in the opposite direction. The explanation is analogous to that of the previous section: this arrival rate is already taken account of in the queue size x 1 itself, and the deflation of x 1 by a factor .../)..1 is a reflection of the fact that one expects a busy queue to be larger, even under an optimal policy. Of course, the notion that service effort can be switched wholly and instantaneously is unrealistic, and a policy that took account of switching costs could not be a pure index policy. Suppose that to switch an amount of service effort u from one queue to another costs c!u!. Suppose that one application of policy improvement to a policy of fixed allocation {u1} of service effort will modify this to {~}.Then the~ will minimise

J

2)tcd~- u,!- ~vt]

(25)

i

subject to

(26)

7 REWARDED SERVICE RATHER THAN PENALISED WAITING

225

and non-negativity. Here v; = v;(x;) is the index defined in (24) and the factor 4 occurs because the first sum in (25) effectively counts each transfer of effort twice. If we take account of constraint (26) by a Lagrangian multiplier (} then the differential of the Lagrangian form Lis

oL

{ L;+ := a~ = L;- :=

!c -!c -

Vj -

(a-;> 0";)

(}

Vj -

(}

(a-; < 0";)

We must thus have ~ equal to a-;, not less than a-;, not greater than a-; or equal to zero according as L;- < 0 < L;+, Lt+ = 0, L;_ = 0 or L;- > 0. This leads us to the improved policy. Let 2l+ be the set of i for which v; is maximal and 2l_ the set of i (possibly empty) for which v; <max v1 -c. 1

(27)

Then all serving effort for members of 2l_ should be transferred to members of 2l+· The definitions of the v; will then of course be updated by substitution of the new values of service allocation. Such a reallocation of effort will occur whenever discrepancies between queues become so great that (27) holds for some i. The larger c, the less frequently will this occur.

7 REWARDED SERVICE RATHER THAN PENALISED WAITING Suppose the model of the last section is modified in that that a reward r; is earned for every customer of type i whose service is completed, no cost being levied for waiting customers. This is then a completely different situation, in that queues can be allowed to build up indefinitely without penalty. It has long been known (Cox and Smith, 1961) that the optimal policy under these circumstances is to serve a customer of that type i for which r;p,; is maximal among those customers present in the queue. This is intuitively right, and in fact holds for service requirements of general distribution We shall see that policy improvement leads to what is effectively just this policy. For a queue of a single type served at unit rate we have the dynamic programming equation in the undiscounted case

"'=

~~(x + 1)

+ p,(x)[r- ~(x)],

(28)

where x is queue size, ~ is arrival rate, p,(x) equals the completion rate p, if x is positive and is otherwise zero, r is the reward and ~(x) is the increment f(x) - f(x- 1) in transient cost. Equation (28) has the general solution for~ _ "!- p,r ~') ~(x)- A-P, ' + ("/' P,-A

[l!_]x-1 ' .

(29)

/1

If p, > ~ then finiteness implies that the second term must be absent in (29), whence we deduce that

226

POLICY IMPROVEMENT. STOCHASTIC VERSIONS

6-(x)

= r.

(30)

This is the situation in which it is the arrival rate which limits the rate at which reward can be earned. In the other case, J.L < A, it is the completion rate which is the limiting factor; the queue builds up and we have

6-(x)

=

(J.L/.XY- 1 .

(31)

The total reward rate for a queue of several types under a fixed allocation {17;} of service effort is then "f

=L

r1

min[.X;, 17;J.Li].

j

If we rank projects in order of decreasing r;J.Li then an optimal fixed allocation is that for which 17; = .X;/ J.L; for i = 11 2 1 • • • 1 m, where m is as large as possible consistent with (22), to allocate any remaining service effort to type m + 1 and zero service effort to remaining types. Now consider a policy improvement step. We should direct the next service to a customer of type i which is present in the queue and for which i maximises J.L1[r1 - .6.;(x;)]. It follows from solutions (30), (31) that this expression is zero for i = 11 2, ... 1 m and equal to r;J.L;[1- (17;J.L;/.X;)x;-l] fori> m. That is, one will serve any of the first m types if one can. If none ofthese are present, then in effect one will serve the customer type present which maximises r 1J.L;, because the fact that the traffic intensity for queue m + 1 exceeds unity means that Xm+l will be infinite in equilibrium. It may seem strange that the order in which one serves members of the m queues of highest effective value r1J.L 1 appears immaterial. The point is that all arrivals of these types will be served ultimately in any case. If there were any discounting at all then one would, of course, always choose the type of highest value among those present for fust service. 8 CALL ROUTING IN LOSS NETWORKS Consider a network of telephone exchanges, with the nodes of the network (the exchanges) indexed by a variable j = 11 2, .... Suppose that there are mik lines ('trunks') on the directed link from exchange j to exchange k, of which Xjk are busy at a given time. One might think then that the vector ! = { Xjk} of these occupation numbers would adequately describe the state of the system, but we shall see that this is not quite so. Calls arrive for a jk connection in a Poisson stream of rate )vk. these streams being supposed independent Such calls, once established, terminate with probability intensity J.Ljk· When a call arrives for ajk-connection, it need not be established on a directjk link. There may be no free trunks on this link, in which case one can either look for an alternative indirect routing (of which there may be many) or simply not

227

8 CALL ROUTING IN LOSS NETWORKS

accept the call. In this latter case we assume that the call is simply lost-no queueing facility is provided, and the caller is assumed to drop back into the population, resigned to disappointment. We see that a full description of the state of the network must indicate how many calls are in progress on each possible route. Let n, be the number of calls in progress on route r. Denote the vector with elements n, by !! and let !! + e, denote the same vector with n, increased by one. Let us suppose that the establishment of a jk-connection brings in revenue Wjk. and that one seeks a routing policy which maximises the average expected revenue. The dynamic programming equation would then be

'Y =

L L -\kmax{O, j

k

Wjk

+max[!(!! + e,)- fC!!)]} r

(32)

+ 2:n,p,[f(_!!- e,)- /(!!)),

where 'Y andfindicate average reward and transient reward respectively. Here the r-maximisation in the first sum is over feasible routes which establish a jkconnection. The zero option in this sum corresponds to rejection of the call on the grounds that connection is either impossible or uneconomic. (The difference f(!!) - f(!! + e,) can be regarded as the implied cost of establishing an incoming call along router. If this exceeds Wjk then the connection uses capacity which could be more profitably used elsewhere. We can take as a convention that this cost is infinite if the route is infeasible-i.e. requires non-existent capacity.) In the second sump, is taken to equal /ljk if router begins inj and ends ink. The term indicated is included in the sum only if!! - e, ~ 0; i.e. if there is indeed at least one call established on route r. Solution of this equation seems hopeless. However, the value function can be determined for the simple policy in which one uses only direct routes, accepting a call for this route if and only if there is a free trunk. We shall then apply one stage of policy improvement. The average and transient rewards on one such link (for which we shall drop the jk subscripts) are determined by ~f =

-X[w + f(x

+ 1)-f(x)] + px[f(x- 1)- f(x)]

(0 < x < m).

(33)

For x = 0 this equation holds with the term in p missing; for x = m it holds with the term in A missing. Let us define the quantity

A(x) = /(x)- f(x

+ 1)

which can be interpreted as the cost of accepting an incoming call if x trunks are currently busy. One finds then the elegant solution

A( x ) = w B(m, 8) B(x,fJ)

L..l.

(0 < x < m);

'Y = ,\w [1- B (m,ull)] ,

228 where(}=

POLICY IMPROVEMENT: STOCHASTIC VERSIONS

>.J J.Lis the traffic intensity on the link. and B(m, 0) is the Erlangfunction B(m, B)

~ (If" /m!) (t If' /x!)

-I

This is the probability tha~ all trunks are busy, so that an incoming call cannot be accepted: the blocking probability. The formula for w1k thus makes sense. Returning to the full network, define

Then we see from (32) that one stage of policy improvement yields the revised policy: If a call comes in, assign it the feasible route for which the sum of the costs x along the route is minimal, provided that there is such a feasible route and that this minimal cost does not exceed WJk· In other cases, the call must or should be rejected The policy seems sensible, and is attractive in that the effective cost of a route is just the sum of individual components x. For this latter reason the routing policy is termed separable. Separability ignores the interactions between links, and to that extent misses the next level of sophistication one would wish to reach. The policy tends to assign indirect routings more readily than does a more sophisticated policy: This analysis is taken, with minor modifications, from Krishnan and Ott (1986). Later work has taken the view that, for a large system, policies should be decentralised, assume very limited knowledge of system state and demand little in the way of real-time calculation. One might expect that performance would fall well below the full-information optimum under such constraints, but it seems, quite remarkably, that this need not be the case. Gibbens eta/. (1988, 1995) have proposed the dynamic alternative routing policy under which, if a one-step route is not available, two-step routes are tested at random, the call being rejected if this search is not quickly successful One sees how little information or processing is required, and yet performance has been shown to be close to optimal

CHAPTER 12

The LQG Model with Imperfect Observation 1 AN INDICATION OF CONCLUSIONS When we introduced stochastic state structure in Section 8.3 we assumed that the current value of the state variable was observable, a property generally referred to as 'perfect observation: Note that this is not the same as 'complete information', which describes the situation for which the whole future course of the process is predictable for a given policy. If the model is state-structured but imperfectly observed (i.e. if the current value of state is imperfectly observable) then the simple recursive treatment of Section 8.3 fails. We shall see in Chapter 15 that, in this case, an effective state variable is supplied by the so-called information state: the distribution of x 1 conditional on W 1• This means that the state space !!£with which we have to work has perforce been expanded to the space of distributions on !!£, a tremendous increase in dimensionality. However, LQG processes show a great simplification, in that for them these conditional distributions are always Gaussian, and so are parametrised by the conditional mean value x1 and the conditional covariance matrix V1• Indeed, matters are even simpler, in that V1 turns out to develop in time by deterministic and policy-independent rules. The only point at which we appeal to the observed course of the process is then in the calculation of x1 • One can regard x1 as an estimate of the current state x 1 based on current information W1.It is interesting that the formulation of a control rule forces one to estimate unobservables; even more interesting that the optimisation of this rule implies criteria for the optimisation of estimation. LQG processes have already been defined by the properties of linear relations between variables, quadratic costs and Gaussian noise. We shall come shortly to a definition which expresses their essence even more briefly and exactly. If we add the feature of imperfect observation to the LQG regulation problem of Section 10.1 then we obtain what one might regard as the prototype imperfectly observed state-structured LQG process in discrete time. For this the plant and observation relations take the form Xt

= Axt-I +But- I+ ft

(1)

+ 1/t

(2)

Yt = Cxt-I

230

THE LQG MODEL WITH IMPERFECT OBSERVATION

where y 1 is the observation which becomes available at timet. We suppose that the process noise E and the observation noise TJ jointly constitute a Gaussian white noise process with zero mean and with covariance matrix (3)

Further, we retain the instantaneous and terminal cost functions c(x,u)

=![~r[~ ~][~],

(4)

of Section 9.1. Ifwe can treat this model then treatment of more general cases (e.g. incorporating tracking of a reference signal) will follow readily enough. We shall see that there are two principal and striking conclusions. One is that, if u, = K 1x 1 is the optimal control in the case of perfect state information, then the optimal control in the imperfect-observation case is simply u 1 = K,x,. This is a manifestation of what is termed the certainty equivalence principle (CEP). The CEP states, roughly, that one should proceed by replacing any unobservable by its current estimate, and then behaving as if one were in the perfect-observation case. It turns out to be a key concept, not limited to LQGmodels. On the other hand, it cannot hold for models in which policy affects the information which is gained. The other useful conclusion is the recursion for the estimate x1

(5) known as the Kalman filter. This might be said to be the plant equation for the effective state variable x1; it takes the form of the original plant equation (1), but, instead of being driven by plant noise e1, is driven by the innovation y 1 - Cx1_ 1 • The innovation is just the deviation of the observation y 1 from the value E(yd W1_1) that one would have predicted for it at time t - I; hence the name. The matrix H 1 is calculated by rules depending on V,_ 1. . The Kalman filter provides the natural computational tool for the real-time determination of state estimates, a computation which would be realised by either a computational or an analogue model of the plant. Finally, LQG structure has really nothing to with state structure, and essential ideas are indeed obscured if one treats the state-structured case alone. Suppose that X, U and Y denote process, control and observation realisations over the complete course of the process. The cost function C will then be a function q X, U). Suppose that the probability density of X and Y for a control sequence U announced in advance is

f(X, Yl; U)

= e-D(X,Yf;U)

(6)

(so that U is a parametrising variable). This must be a density relative to an appropriate measure; we return to this point below. We shall term [ll

2 LQG STRUCTURE AND IMPERFECT OBSERVATION

231

the discrepancy, since it increases with increasing improbability of plant/ observation realisations X and Y for given U The two functions C and ID of the variables indicated characterise the cost and stochastic structure of the problem. One can say that the problem is LQG if both the cost function C and the discrepancy [) are quadratic in their arguments when the density (6) is relative to Lebesgue measure. This characterisation indeed implies, very economically, that dynamic relations are linear, costs quadratic and noise Gaussian. It also implies that policy cannot affect information, in that the only effect of controls on observations is to add on a term which is linear in known controls, and so can be corrected out. It implies in addition that the variables take values freely in a vector space (since the density is relative to Lebesgue measure) and that the stopping rule is specification of a horizon point h (since histories X, Y, U are taken over a prescribed time interval). It can be asserted quite generally (i.e. independently ofLQG ideas) that the two quantities C and ID between them characterise the model. One might say that both C and ID should be small (relative to what they could be), in that one wishes to make C small and expects ID to be small. One can define the discrepancy for any random variable; i.e. ID(x) for x alone or ID(x!y) for x conditioned by y. The fact that the discrepancy is the negative logarithm of a density relative to a measure which may be normalised in different ways means that it is determined only up to an additive constant. We shall assume it so normalised that inf ID(x) = 0. X

So, if xis a vector normally distributed with mean f-L and covariance matrix V, then ID (x) is just the quadratic form! (x - f-L) T v- 1 ( x - f-L). Note the general validity of formulae such as

ID(x,y) = ID(y)

+ ID(x!y).

It is often asked why the observation relation is taken in the form (2), with Yt essentially being an observation on immediately previous state rather than on current state. The weak answer is that things work out nicely that way. Probably the right answer is that, if one regards (x 1 , y 1 ) as a joint state variable, then y 1 is a function of current state, uncorrupted by noise. This is an alternative expression of imperfect observation which has a good deal to recommend it: that one observes some aspect of current state without error.

2 LQG STRUCTURE AND IMPERFECT OBSERVATION The treatment of imperfectly observed LQG models involves a whole train of ideas, which authors order in different ways as they seek the right development.

232

THE LQG MODEL WITH IMPERFECT OBSERVATION

We shall start from what is surely the most economical characterisation of LQG structure: the assumed quadratic character of the cost C(X, U) and the discrepancy D(X, Yl; U) as functions of their arguments. We also regard the treatment as constrained by two considerations: it should not appeal to state structure and it should generalise naturally to what is a natural extension of LQG structure: the risk-sensitive models of Chapters 16 and 21. The consequent treatment is indeed a very economical and direct one; completed essentially in this section and the next. Sections 6-8 are added for completeness: to express the results already obtained in the traditional vocabulary oflinear least square estimates, innovations, etc. Section 9 introduces the dual variables, in terms ofwhich the duality of optimal estimation to optimal control finds its natural statement. Note first some general points. The fact that we have written the cost C as C( X, U) implies that the process has been defined generally enough that the pair (X,, U1) includes all arguments entering the cost function Cup timet (such as values of reference signals as well of plant itself). The dynamics of the process and observations are specified by the probability law P(X, Yl; U), which is subject to some natural constraints. We have

P(X, Yl; U)

h-1

h-1

1=0

1=0

=II P(xr+I,Yr+IIXr, Y,; U) =II P(xr+I,Yr+J!X,, Y,; U,)

(7)

the second equality following from the basic condition of causality; equation (A2.4). Further

D(x,+J,Yr+JiX,; U,) = D(xr+J!X,, Y,; U,) + []l(y,+J!Xr+l, Y,; U,) = D(xr+J!X,; U,) + D(y,+JiXr+l• Y,; U,)

(8)

the second equality expressing the fact that plant is autonomous and observation subsidiary to it, in that the pair (X1, Y1) is no more informative for the prediction of X 1+1 than is X 1 alone. Relations (7) and (8) then imply

Theorem 12.2.1 The discrepancy has the additive decomposition h-1

D(X, Yl; U) = L[D(xr+IIXr; U,)

+ D(Yr+IIXr+l, Y,; U,)].

(9)

1=0

Consider now the LQG case, when all expressions in (9) are quadratic in their arguments. The dyamics of the process itself are specified by h-1

D(XI; U) =

L D(xr+IIX

1;

t=O

U1)

(10)

2 LQG STRUCTURE AND IMPERFECT OBSERVATION

233

and the conditional discrepancies will have the specific form D(xr+IIX,; U,)

=! (xr+l -

dr+l - A,X,- B, U,) T N,--)1(xr+l- dr+l- A,X,- B, U1)

(11) for some vector d and matrices A, B, N, in general time-dependent. Relations (10) and (11) between them imply a stochastic plant equation Xr+l = A,X, + B,U, + dr+!

+ tr+l

(0 ~ t
(12)

for which the plant noise variables t are independently and normally distributed withzeromeanandwithcov(t,) = N,. One must view this as a canonical plant equation rather than necessarily the actual, physical plant equation; see Exercise 2. However, the deterministic version of the process would undoubtedly be that for which t was set equal to zero in relation (12). This is exactly the relation which would be obeyed by the most probable path for given U; i.e. that followed by the values of x,(1 ~ t ~h) which minimise [)(XI; U).

Theorem 12.22 Suppose LQG structure. The total value function Gdct(X, U,_I) for the deterministic (and perfectly observed) process is equal to the minimal value of C(X, U) with respect to u,.(r ;;;Ji: t1 when for X-r(r > t) have been substituted the values which minimise [)(XI; U). The value ofu, thus determined, Udct(X, Ur-t1 is the optimal value of utfor the deterministic process in closed-loop form. Proof All that we have done is to use the deterministic version of (12) to express future process variables X-r (r > t) in the cost function in terms of control variables U and current process history X1, and then minimise the consequent expression for C with respect to the as yet undermined controls u,.( r ~ t). However, the point of the theorem is that this future course ofthe deterministic process is determined (for given U) by minimisation of [)(XI; U) with respect to these future variables. 0 Otherwise expressed, the 'deterministic future path' is exactly the most probable future path for given X, and U In optimising the 'deterministic process' we have supposed current plant history X, known; a supposition we now drop.

Exercises and comments (1) Note that state structure would be expressed, as far as plant dynamics go, by

P(XI; U) = IT,P(xt+tlx,; u,). (2) Relation (12) should indeed be seen as a canonical plant equation rather than necessarily the 'physical' plant equation. The physical plant equation might have the form

234

TIIE LQG MODEL WITH IMPERFECT OBSERVATION

where the plant noise e• is autocorrelated. However, we can substitute f;+I = E( E7+ dX1 , U1) + ft+ I· This standardises the equation to the form (12), since the expectation is linear in the conditioning variables and f has the required orthogonality properties. The deterministic forms of the two sets of equations will be equivalent, one being derivable from the other by linear operations.

3 THE CERTAINTY EQUIVALENCE PRINCIPLE When we later develop the ideas of projection estimates and innovations then the certainty equivalence principle is rather easily proved in a version which immediately generalises; see Exercise 7.2. Many readers may find this sufficient. However, we find it economical to give a version which does not presume this apparatus, does not assume state-structure and which holds for a more general optimisation criterion. The LQG criterion is that one chooses a policy n to minimise the expected cost E'li"(C). However, it is actually simpler and more economical for present purposes to consider the rather more general criterion: that nshould maximise E,.[e-9"C] for prescribed positive B. Since this second criterion function is 1 - OE'li"(C) + o(B) for small 9, we see that the two criteria agree in the limit of small 9. For convenience we shall refer to the two criteria as the LQG-criterion and the LEQG-criterion respectively, LEQG being an established term in which the EQ stands for 'exponential of quadratic~ The move to the LEQG criterion induces a measure of 'risk-sensitivity'; of regard for the variability of C as well as its expectation. We shall see in Chapters 16 and 21 that LQG theory has a complete LEQG analogue. Indeed, LQG theory appears as almost a degenerate case of LEQG theory, and it is that fact which we exploit in this section: LEQG methods provide the insightful and economical treatment of the LQG case. We wish to set up a dynamic programming equation for the LEQG model. If we defme the somewhat transformed total value function e-G(W,)

= f( Yt) sup£'71"[e-9CI Wt], 1r

where f is the Gaussian probability density, then the dynamic programming equation takes the convenient form

(t
(13)

We lose no generality by assuming perfect information at the horizon point h; this cannot affect policy since no further actions are to be taken. Thus Wh = (X, Y, U) and the terminal condition for equation (13) is

3 THE CERTAINTY EQUIVALENCE PRINCIPLE

235

G( Wh) =[])(X, Yl; U) + OC(X, U).

(14)

The following simple lemma expresses a key result.

Lemma 12.3.1 Suppose that Q( z, y) is a quadratic function ofvectors z andy, positive definite in y. Then

j exp[-Q(z,y)]dy = exp[q- itf Q(z,y)] where q is a constant having the evaluation log[ (211")'/ 21Qyy 1-l/2], r is the dimension of yand Qyy is the matrixofsecond-orderderivatives ofQ(z, y) with respect toy. The point ofthe lemma is that, if one replaces an integration with respect toy by a minimisation of Q with respect to y, then the result is correct as far as terms dependent on the second argument z are concerned.

Proof If y is the minimising value (which will in general be z-dependent) then one can write Q(z,y)

= Q(z,y) +! (y- .Y? Qyy(y- y).

The lemma now follows by integration with respect to the variable y -

y.

0

We can apply the Lemma to the solution of (13).

Lemma 12.3.2 Assume LEQG structure. Then the dynamic programming equation (1 3) with terminal condition (1 4) has the solution

G( W1) = inf

Ur:T ;;. I

inf inf[ID(X, Yl; U)

yT:T>I

X

+ 8C(X, U)] + g1

(15)

where g1 is a policy-independent function oft alone. The value of u1 thus determined is the LEQG-optimal value ofthe control at timet. Proof We see from (14) that G( WT) is quadratic in its arguments at time T = h; suppose this so at time t + 1. Then evaluation of the integral in (13) by appeal to the lemma leads to the conclusion that G( W1) = inf inf G( W1+1) u,

Yr+l

+ ...

(16)

where + ... indicates a policy-independent function oft alone, and we know the infimising u1 to be optimal. The quadratic nature of G is thus established inductively, and solution (15) follows from (14), (16). In making this last deduction we have re-ordered the extremal operations; they indeed commute because they are all infimising operations. The ..¥-minimisation occurs because, in the

236

THE LQG MODEL WITH IMPERFECT OBSERVATION

dynamic programming equation at time h - 1, the integration over Yh amounts to an integration over all variables of the full description which are not determined from Wh-1· 0 The restrictions on the form of BJI which we deduced in the last section imply a strengthening of the last theorem.

Theorem 12.33 Assume LEQG structure. Then the dynamic programming equation (13) with terminal condition (14) has the solution G( W,) = inf{BJI(Xr.

x,

+ u,.:Tinf I ~

Y,l; Ur-1) inf [D(xr+I. ... ,xhiX1; U)

Je.:T>t

+ OC(X, U)]} + g,

(17)

where g, is a policy-independent function oft alone. The value ofu1 thus determined is the LEQG-optimal value of the control at time t. If the value of u, minimising the square bracket is denoted u(X1 , Ur-1) then the LEQG-optimal value of u, is u(xi'>, Ur-1) where X~'> is the value ofX, determined by the final X,-minimisation. The transformed form (17) of (15) follows by appeal to the reduction of []) expressed in (9). The final assertion of the theorem follows from it. This assertion is very significant; it is actually the risk-sensitive form of the certainty equivalence principle (CEP), at least for the case 0 ;;.: 0. We shall develop it in Chapter 16. The point for the moment is that the CEP we seek follows immediately from it

Theorem 12.3.4 (The Certainty Equivalence Principle for the LQG case) Assume LQGstructure. Then the optimal valueofur is Udet(Xi'>, Ur-1), where Udet(X,, Ur-1) is the optimal control for the deterministic case determined in Theorem 12.2.2 and X 1(r) isthevalueofX,minimisingD(Xr. Y,l; U,_i). Proof This is just the final assertion of the previous theorem in the limit 0! 0. Suppose 6 small, and consider first the extremisations of the square bracket in (17). The values of ~(r;;.: t) minimising the square bracket for given U are, to within a term O(O),just the values minimising BJI(xt+h ... , xhiX1; U). That is, just the future course of the deterministic version of the process (for given X1 , U) to within a term 0(0). These yield a value for D(xt+b ... Xh!X,, U) which is 0(6'-), and the u-minimisations which follow are essentially minimisations of OC(X, U) with future process variables { ~; r > t) given their deterministic prediction. The value u(X,, U,_,) of u, which minimises the square bracket is thus Udct(Xr. U,_J)+ 0(6). The minimal value of the square bracket in (17) is then 0(6), with the 1 consequence that the value of minimising the curly bracket is just > + 0(6), with the definition of x?> modified to that stated in the theorem. The final

X,

Xi

4 APPLICATIONS: THE EXTENDED CEP

237

determination of the minimising value of u, which we know to be LEQGoptimal at(}, is then udet(x?l, Ur-i)+ 0(9), whence the assertion of the theorem ~~

0

The quantity x?l is effectively an estimate of X1; the superscript ( t) is intended to indicate that it is an estimate based on the information available at time t. It indeed makes sense as an estimate; in this LQG case it is the value of X 1 which minimises IG(X,, Yr\i Ur-i), and so the value maximising P(X1, Y1 \; Ur-i)· Otherwise expressed, it is the value maximising P(X1 \ Y1; Ur-i) = P(X1\ W1), and so is the most probable value of X1 conditional on current information W,. We shall for the moment term it the conditionally most probable (CMP) estimate, and shall see in Section 6 that it has other characterising properties. Theorem 12.3.4 asserts that the optimal value of u1 is calculated exactly as in the completely observed deterministic case, except that the fact that that process history Xc is now in general unobservable is overcome by replacing X1 by its current estimate x,Ul. This is what one would in general mean by the CEP; that it is optimal to behave as though the estimate were the true value. Of course, for this to be true the estimate must be that which is 'optimal' on a criterion induced by the control criterion. Exercises and comments

(1) Suppose that current process history X 1 is indeed observable. Then the CEP asserts that the optimal control is just Udet(Xr, U,_i); exactly what it is in the deterministic case. That is, the presence ofuncorrelated noise fin the canonical plant equation (12) does not affect the closed-loop expression for the optimal control. This is what we have already observed in Section 9.1 for the statestructured case. 4 APPLICATIONS: THE EXTENDED CEP The most immediate application of the CEP would be to the state-structured regulation problem of Section 9.1. The known optimal control u1 = K 1x 1 now 1 1 l, wholly in terms of the current estimate l of current state becomes u1 = x 1• We shall consistently write this estimate simply as x, and shall show in the next section that it is generated by the Kalman til ter (5). If one is regulating to zero when the plant equation is subject to a known disturbance sequence {d1} then we know from Sections 2.9 or 6.5 that the optimal control takes the feedback/feedforward form

K,xl

xl

00

Ut

= Kx, + 2::= Ljdt+j+i j=O

(18)

238

THE LQG MODEL WITH IMPERFECT OBSERVATION

for known matrices L1, where we have for simplicity written out only the timeinvariant form of th.is relation. If one penalises the departure of x and u from prescribed reference signals then we know (see the sections quoted) that we can reduce the optimal control to the form (18), when d is a linear combination of actual disturbances and reference signals. In practice, the future course of disturbances and reference signals will in general be unknown, but, under LQG assumptions, the CEP will be applicable, and one can assert that the optimal control rule will be just

Ut = Kx~t)

00

+ L Ljd~J+l

(19)

j=O

Here d~ 1 is an effective forecast of the value of dt+J formed at timet, in that it is the most probable value of dr+J conditional on information available at time t. If one does not feel confident enough of the statistics of the situation to form a probability model then the 'most probable' forecasts in (19) could be replaced by the best informal or subjective forecast available. One could not say much about performance of the rule in such a case, but at least one could be sure that the rule was of a form known to be optimal for a fully specified model. Although we derived the CEP by appeal to the dynamic programming recursion (in particular, in the passage from (13) to (17)), it could also be applied to a direct trajectory optimisation (as is indeed implicit in Theorem 12.2.2). That is, at ti:me t one just substitutes conditionally most probable estimates for unobservables and then carries out, by whatever means convenient, a direct optimisation of the future trajectory thus predicted. The determination of the current control obtained in this way is then optimal. This extended form may be useful if one wishes to restore the mental picture of the future course of the process wh.ich is largely lost in a recursive formulation. The values u~'l for T > t of later decisions formed in th.is way are not in general the optimal values of these quantities, but they are indeed the best estimates one can form at timet of what these optimal future decisions will be (an assertion we make explicit in Exercise 7.1). This is one advantage of the extended formulation; that optimisation is seen as part of a provisional forward plan. This plan will of course be updated as time moves on and one gains new information, but the fact that it has been formed gives one a much more conscious appreciation of the course of events and the contingencies against which one is guarding. It is a view of optimisation which lies close to the intuitive methods by which we cope with daily life. Again, one can imagine formal estimates being replaced by informal estimates in cases where one does not have the basis for a complete stochastic model. Of course, the point about the recursive approach is that it is economical: one does not have to hold a whole future in one's mind (or processors). The chess analogue we have already given in Section 1.3 brings home the contrast between

5 CALCULATION OF ESTIMATES: THE KALMAN FILTER

239

the two views. To play chess by the extended method is to choose one's current Illove after analysing all likely sequences of play some time into the future. To play chess by the recursive method is to give every board configuration x a value F ( x) and to look for a move which changes the configuration to one of greater (or Illaximal possible) value. In other words, an ultimate goal (winning) is replaced by an immediate goal (improvement of configuration). Of course, the chess example is complicated by the fact that there are two players, optimising for different goals. However, there are plenty of other examples. One can say that Nature has allowed for the limited intelligence (and processing power) of her creations by replacing ultimate goals by intermediate goals. An animal does not eat in order that it may not die (a distant goal) or in order that the species may survive (even more distant) but in order that it may not reel hungry (immediate goal). Evolution has implanted a mechanism which both senses relevant aspects of state (a need for food) and evaluates them (so that a condition of extreme hunger is unpleasant). 5 CALCULATION OF ESTIMATES: THE KALMAN FILTER

x

In order to implement a control rule such as as u, = K 1 1 for an imr,erfectly 1 observed state-structured system we have to be able to calculate 1 = x 1 ). LQG assumptions will imply that the distribution of x 1 conditional on W, is normal, with mean 1 and covariance matrix V, say. The conditional mean value .X, is indeed identifiable with the conditionally most probable value, because it is the value which maximises P(x1 1W1). If the observation relation is also 'state-structured' in the appropriate sense then one can develop recursions which express .X, and V, in terms of Xt-1. V,_ 1, y, and Ut-1· The recursion for 1 is the ubiquitous Kalman filter (5). Such recursions supply the natural way of calculating the estimates, whether control is realised by digital or analogue means. Before developing them we shall clarifY notation and some properties of the normal distribution. Suppose that x and y are a pair of random vectors (so defmed as joint random variables). Let us suppose for convenience that they have zero means: E(x) = 0, E(y) = 0. Then the matrix E(xyT) with jkth element E(x1yk) is the covariance matrix between x and y (note that the order is relevant). We shall denote it variously

x

x

x

E(xyT) = cov(x, y)

= Vxy·

(20)

We shall denote cov(x, x) simply by cov(x); the covariance matrix of the random vector x. If x is scalar then cov(x) is simply the variance of x, which we shall write as var(x). If cov( x, y) = 0 then one says that x andy are mutually uncorrelated, and writes this x_Ly. The relation is indeed an orthogonality relation under conventions natural to the context.

240

-~

THE LQG MODEL WITH IMPERFECT OBSERVATION

·-)

Lemma 12.5.1 Suppose the vectors x andy are jointly normally distributed with zero i means. Then the distribution of x conditional on y is normal with mean and covar.·: iance matrix given by .' cov(xly) =

Vu -

Vxy V~ 1 Vyx·

(21)

Proof Denote the value asserted for E(xly) in (21) by x; we shall later see this notation as consistent. Then one readily verifies that x - x and y are independently (and normally) distributed, so that the unconditioned distribution of x - x is the same as its distribution conditional on the value of y. But one again verifies that the unconditioned distribution of x - x is normal with zero mean and covariance matrix equal to the expression for cov(xly) asserted in (21). The 0 conclusion of the theorem thus follows.

v,,

We have implicitly assumed non-singular, but the assertions of the theorem remain meaningful even if this is not the case; see Exercise 1. We come now to the principal conclusions.

Theorem 12.5.2 Assume the imperfectly observed state-structured model specified in relations (1)-(4) of Section I. Suppose initial conditions prescribe xo as normally distributed with mean x0 and covariance matrix Vo. The model is then LQG and the distribution of x, conditional on W 1 is normal with mean x1 and covariance matrix v,,say. (i) The estimate x1 obeys the updating relation

x,

=Ax,_, + Bur-1 + H,(y, -

Cx,_t)

(22)

(the Kalman filter), where

(23) (ii) The covariance matrix V1 obeys the updating relation V, = N +A V,_,AT- (L + AV,_, CT)(M + CV1_ 1CT)- 1(LT

+ CV,_,AT). (24)

Proof There are many proofs, and we shall indeed give one in Section 8 which dispenses with the assumption of normality. However, this assumption has been intrinsic to the LQG formulation and makes for what is by far the most economical proof. The preliminary assertions of the Theorem follow from previous discussion. If we denote the estimation error x 1 - x1 by b., then V, is exactly the covariance matrix of b.,. The quantities Xr-1 - Xr-1, x,- Axr-1 - Bu,_I, and y 1 - Cx,_, are

5 CALCULATION OF ESTIMATES: THE KALMAN FILTER

241

identifiable with At-1, f.t and T/t respectively. These are jointly normally distributed with zero mean and covariance matrix

[~

0 N LT

L0 M

l .

where we have written Vt-1 simply as V. This effectively gives us the joint distribution of Xt-i> Xt and Yt conditional on Wt-1, Ut-I· What we must do is to integrate out the unobservable Xt-1 and then condition the distribution of x 1 by the value of Y~> so obtaining the distribution of x1 conditional on ( W 1_ 1, u1_ 11 y 1) = w;, as desired. Integration over values of Xt-1 is equivalent to the substitution Xt-1 = Xt-1 + At-1· The quantities (; = Xt- Ait-1- But-1 and ( =y1 Cit-! are then identifiable with f.t +A At- I and TJt +CAt- I and so 1 are jointly normally distributed with zero mean and covariance matrix N + A VAT [ LT + CVAT

L +A VCT ] M + CVCT

(25)

It then follows by an appeal to Lemma 12.5.1 that the distribution of(; conditional on the value of (t is normal with mean Ht(t and covariance matrix V11 where H 1 and V1 have the values asserted in (23) and (24). Converting this into a statement of the distribution of Xt conditional on Yt (and w;_ 1) we deduce the assertions (i) and (ii) oftheTheorem. 0 Recursion (24) is a forward recursion of Riccati type, analogous to the backward Riccati recursion of control optimisation. The analogy is based on a real duality, as we shall see in Section 9. The recursion is deterministic; it is only i which needs the observations for its updating. However, we need the value of V to determine the i updating, as we see from (23). As observed in Section 1, the Kalman filter (22) is just the plant equation with the estimate i replacing the actual state value x and a term in the innovation ( replacing the plant noise f. It is this latter term which of course provides a correction when the observation Yt turns out to differ from the value which would have been predicted. All this material has an immediate continuous-time version. We shall simply quote results, since these can be obtained either by analogous arguments or by a formal passage to the continuous limit. The standard state-structured model specifies the plant and observation relations x=Ax+Bu+€,

y = Cx+ TJ

(26)

where f. and TJ (plant noise and observation noise) are jointly white, with zero mean and power matrix [

[r i:t] .Note that the plant relation is in differential form,

242

THE LQG MODEL WITH IMPERFECT OBSERVATION

the observation relation in instantaneous form. This is generally regarded as the natural formulation, although we shall have occasion to recast it in Chapter 25.

Theorem 12.5.3 Assume the imperfectly observed state-structured model specified above. Suppose that initial conditionsprescribe x( 0) as normally distributed with mea1L .i:(O) and covariance matrix V(O). Then the distribution ofx(t) conditional on W(t) is normal with mean x( t) and covariance matrix V( t), say. determined as follows. (i) The estimate .X obeys the updating relation ~=Ax+Bu+H(y- Cx)

(27)

H = (L + VCT)M- 1•

(28)

(the Kalman ftlter), where

( ii) The covariance matrix Vobeys the updating relation

V=

N

+ AV +VAT- (L+ VCT)M- 1(LT + CV).

(29)

Exercises and comments

(1) Suppose that Vyy is singular, so that vectors c exist such that Vyyc = 0. These are elements of the null space..¥ of Vyy· For such vectors var( cTy) = cT Vyyc = 0, so that cT y is zero in mean square. Show that VxyC = 0 for such c. An equation Vyya = b for vector a will have a solution, which we shall denote by v;:,I b, only if b is orthogonal to all c of..¥, and this solution is arbitrary to within addition of an element of .K. Show that different evaluations of the expression for E(xjy) in (21) obtained in this way differ by a term cTy for some c of..¥ (and so are zero in mean square) and that expression for cov(xjy) is uniquely determined.

6 PROJECTION ESTIMATES Our LQG analysis has generated effective estimates, the most probable values of unobservable random variables conditional on the values of observables, under the assumption that all random variables are jointly normally distributed However, the estimates thus generated have distinctive properties if one makes no distributional assumptions at all, apart from specification of first and second. moments ofthevariables concerned This weakening of hypotheses leads to a new view of the conclusions derived under Gaussian hypotheses in Sections 1-5, and a valuable one. We shall assume notations (for covariance matrices, etc.) and definitions (of orthogonality etc.) listed in Appendix I. Suppose that xis a random vector of zero mean and covariance matrix Vxx· Then we can still define the quadratic form

6 PROJECTION ESTIMATES

243

the form which would have occurred in the exponent of the normal density had x been normally distributed, and which we would then have termed the discrepancY. Let us continue to do so (although it is only when x is normally distributed that we can identify D(x) with [])(x)). For non-singular A of appropriate dimension we have D(x) =!(Ax)T(AVxxAT)- 1(Ax). But A VxxAT is just the covariance matrix of the transformed vector Ax. We thus deduce the simple but significant conclusion:

Lemma12.6.1 Suppose that D(x) can be written! (Ax) T v- 1 (Ax). Then necessarily V = cov(Ax). From this we further deduce the following lemma.

Lemma12.6.2 infxD(x,y) = D(y). Proof Let the minimising value be denoted x, a linear function of y. Then D( x, y) can be written as the sum of quadratic forms (30) for some V1 , V2. It then follows from expression (30) and the previous proposition that x- x and y are mutually orthogonal, with V1 = cov(x- x) and V2 = cov(y). Since plainly infxD(x,y) equals the final form in (26), the 0 conclusion then follows. Note that the xthus determined is just the estimate of x in terms of ywith which we are familiar, the conditionally most probable value if distributions are normal. Let us now term it the minimal discrepancy estimate. Suppose now that y has been observed and x has not, and one would wish to form an estimate of x in terms of y. The simplest estimate would be a linear estimate, of the form

x=Hy.

(31)

and the most immediate concept would be that of a linear least-square estimate: to choose the coefficient H in (31) so that the mean-square error E[(x- .X) T G(x -.X) is minimal for prescribed positive-definite G. One readily finds that nothing is gained by the addition of a constant term to (31) (i.e. the optimal value of the constant is zero) and that optimisation with respect to Hyields the condition X-

X j_

y,

(32)

244

THE LQG MODEL WITH IMPERFECT OBSERVATION

independent of G. That is, that the estimation error is orthogonal to the observations. Let us term a linear estimate satisfYing (32) a projection estimate. Then we can assert that condition (32) is sufficient for the linear least-square property as well as necessary, in that a projection estimate also has the linear least-square property. In fact, we can assert a number of equivalences.

Theorem 12.6.3 The following characterisations of an estimate x of x in terms of y are equivalent. q) It is a projection estimate, i.e. a linear function ofy for which x - x ..L y. ~z) It is a linear least-square estimate. Qiz) It is the value ofx which freely minimises the discrepancy D(!c, y). ~v) It has the property that cov(x- x) ~ cov(x- x).

(33)

for any linear estimate x. All such estimates are effectively identical, in that they are equal in mean square. Proof Let us abbreviate 'linear least-square' and 'minimal discrepancy' to LLS and MD respectively. We know that the LLS estimate is a projection estimate and it follows from the proof of Lemma 12.6.2 that the same is true of the MD estimate. If x = Hy then the orthogonality condition (32) implies that

0 = E[(Hy - x)yT] = HVyy - Vxy·

(34)

So, if Vyy is non-singular, then the projection estimate has the unique determination

.

X=

vxy v-I yy Y·

(35)

(cf. (21)). If Vyy is singular then the analysis of Exercise 5.1 shows that the solution (35) is still meaningful and different evaluations of it are equivalent in mean square. However, proof that a projection estimate has the stronger LLS property (34) both provides the reverse implications (i) ::::} (iv) ::::} (ii) and demonstrates mean square equivalence. Note first that x - x ..L x - x, because x - x is a linear function of y. We have then cov(x- x) = cov(x- x + x- x)

= =

cov(x- x) + cov(x- x, x- x) + cov(x- x, x- x) + cov(x- x) cov(x- x) + cov(x- ,\") (36)

Now cov( 8) ~ 0 for any random vector 8, and in particular for the vector 8 = x- x. Thus inequality (33) holds.

6 PROJECTION ESTIMATES

245

If x is also a projection estimate then the argument above also establishes the reverse inequality to (33), and so equality. If equality holds in (33) then cov(x- x) = 0, so that x = x in mean square. An application of Cauchy's inequality then shows that cov(x - x, y) equals cov(x- x, y), which is zero, by hypothesis. Thus xis also a projection estimate. 0 We have glossed over one point. There is trouble with the definition of D(x, y), and so with the concept of the minimum discrepancy estimate, if the joint covariance matrix of x andy is singular. The point is actually a very significant one, because the means by which we deal with it (see Exercise 1) provide the passage to the 'right' formulation of the whole control problem; see Section 9. We verify that all projection estimates have the error covariance matrix cov(x- x) =

Vxx -

Vxy

v;/ Vyx,

(37)

immediately if Vyy is non-singular and in the more circumspect sense of Exercise 5.1 otherwise. We shall sometimes write the projection estimate x of x in terms of yas

x =
(38)

to emphasise the role of the two variables. This can be regarded as a translation of the expression E(xjy), which would be appropriate if the variables were normal, to the case when only second-order properties are assumed.

= A
and


+ x2jy) =
for x 1 and x2 vectors of the same dimension. However, the essential point is that it is indeed a projection operator, enjoying the following characteristic properties.

Theorem 12.6.4

~) <6'2

= <6', in that
=
= H
Proof Assertion (i) follows immediately from C(Hyjy)

Exercises and comments (1) The matrix identity xTv- 1x = sup>.[2.>lx- _ATV.A] for non-negative definite V is fundamental, and valid even if V is singular (in that the evaluation is +oo

THE LQG MODEL WITH IMPERFECT OBSERVATION

246

unless x lies in the orthogonal complement of the null space of V). It gives us the evaluation of the discrepancy D (x, y) =

r[ ][ l

s~y [ATx + JLTy -! [~ ~;: ~;~ ~]

The minimising values of Aand JL are the differentials of D with respect to x andy. If x has the discrepancy-minimising value x then A must be zero, so the extremal equations with respect to Aand JL become x = VxyJL andy = Vyy/L· This amounts just to the equation (35). The analogues of A and JL in the dynamic context are just the Lagrange multipliers associated with plant and observation equations, and it is by their introduction that a direct trajectory optimisation finds its natural completion (see Section 9) as it did already in the case of perfect observation (Section 6.3). 7 INNOVATIONS The previous section was concerned with a one-stage situation. In the temporal context we shall have a multi-stage situation, in which at every instant of time t (= 0, 1, 2, ... ) one receives a new vector observation y 1• The observation history attime tis then Y1 = {y.,.; 0 ~ r ~ t}. Define the innovation ( 1 at timet by

(o =Yo- E(yo),

( 1

=y1 -19'(y 1 JY,_l)

(t=l,2,3, ... ).

(40)

The innovation is thus the deviation of the actual observation y 1 at timet from the projection forecast one would have made of it the moment before, at time t - 1. It thus represents the 'new' information gained at time t, in that it is that part of y 1 which could not have been predicted from earlier information. Theorem 12.Zl

The innovations are mutually uncorrelated.

Proof Certainly ( 1 l_ Yr-l· It follows then that ( 1 linear function of Yr-l·

l_

(,.-forT < t, because(,.- is a 0

Now define

the projection estimate of a given random vector x based on information at time t. We shall use this superscript notation consistently from now on, to denote the optimal determination (of the quantity to which the superscript is applied, and under a criterion of optimality which may vary) based on information at timet. It follows, by appeal to (39), that x(t) = ltf(xj Y,_l)

+ ltf(xj(,) =

x(t-1)

+H

1( 1

(41)

247

7 INNOVATIONS

where the matrix H1 in fact has the determination

H1 = cov(x, ( 1)[cov((1}t 1.

(42)

Equation (41) shows that the estimate of x based on Y1_ 1 can be updated to the estimate based on Y1 simply by the addition of a term linear in the last innovation. This is elegant conceptually, and turns out to provide the natural algorithm in the dynamic context. Equation (37) of course implies further that I

x(tl

=L

I

8(xi(r)

r=O

=L

Hr(r.

r=O

which is nothing but the development of a projection in terms of an orthogonal basis. However, the innovations are more than just an orthogonalisation of the sequence {Yr}; the fact that the orthogonalisation is achieved by the timeordered rule (40) means that ( 1 indeed has the character expressed in the term 'innovation'. The innovations themselves are calculated by the forward recursion 1-1

(r = Yt-

L tS'(yll(r)·

(43)

r=1

Alternatively, we can take the dual point of view, and calculate them by minimising the discrepancy backwards in time rather than by doing least square calculations forward in time.

Theorem 127.2 The value ofy1 minimising D( Y1) is <8'(y1 Y1-1) and the minimised value is D( Y1-1 ). Further, 1

(44) where M 1 = cov( ( 1). Proof The first pair of assertions follows from the theorems of the last section, and relation (44) is nothing but relation (30) transferred to this case.

D

We plainly then have the additive decomposition of the discrepancy

D(Y1) = !L~1

(;M; 1(r.

Exercises and comments (1) Let u~l be the estimate of the optimal value of Ur formed at time t by the application of the extended CEP in Section 4. Show that indeed u~l = 8( u~r) IW, ).

248

THE LQG MODEL WITH IMPERFECT OBSERVATION

(2) A proof of the certainty equivalence principle. Consider the state-structured LQG model expressed in equations (1)-{4), the criterion being minimisation of expected cost Let F(x 1 , t) and u(x1 , t) be the value function and oftimal control 1 rule at time t under perfect state-observation. Note that x~t+l = x~ ) + z1 = x1 + z1 where, conditional on ( W1, u1), the random variable z1 has zero expectation and a covariance matrix independent of policy or process history. Hence show, by an inductive argument, that the value function and optimal control in the case of imperfect observation are F(x1 , t) + · · · and u(x1 , t) respectively, where + · · · indicates a term independent of policy or process history. 8 THE KALMAN F1LTER REVISITED We can now give a quick derivation of the Kalman filter under purely secondorder assumptions, when x1 is defined as the projection estimate tB'(x 1 W 1). The proof is in some ways more enlightening than that of Section 5, although it needs more in the way of preliminary analysis. We have 1

X1

= tf(Ax1-1 +Bur-l + Erl Wr) =Bur-l+ tf(Axr-1

+ e,l Wr-1) + tf(Axt-1 + Erl(r)

(45)

= AXe-l + Bur-1 + Hr(r for appropriate H 1• Here ( 1 is the innovation

(, = Yt -

8(y,l W,_I)

= Yt- C.Xr-1,

(46)

so the form (22) of the Kalman filter recursion follows from (45) and (46). Further H,

= cov(Ax,_ 1 + Er, (r)[cov((,)r 1 = cov((;, (,)[cov((,)r 1 ,

(47)

where

C=

A(xr-1 - Xr-1)

+ e1•

(48)

c;

But, as in Section 5, and ( 1 can be written as e1 + A~t-l and TJ1 + C~r-1, respectively, and are jointly normally distributed with zero means and with covariance matrix (25} The expression (23) for H 1 thus follows, as does expression (24) for V1 =

cov(~,) = cov(Hr(r - (;).

(49)

The quantity ( 1 is certainly the innovation in the observations. The quantity (; equals x; - 8(x 11 W,_I), and so can be regarded as the plant innovation. 9 ESTIMATION AS DUAL TO CONTROL The recursion (24) for the error covariance matrix V1 stands in obvious parallel to the Riccati recursion (2.25)/ (2.26) of LQ control optimisation. The analogy is

9 ESTIMATION AS DUAL TO CONTROL

249

complete, with only the 'dualising' modifications that transposes of covariance matrices replace cost matrices and the recursion goes forward instead of backward. In fact, there is a duality between control and estimation which is constantly in evidence, but is revealed cleanly only by adoption of the Lagrangian methods of Section 6.3. The stochastic model must be completed by prescription of initial conditions, which in the present state-structured case means prescription of the distribution of xo conditional on Wo; normal with mean .Xo and covariance matrix Vo. The expression for [b, = [b(X,, Y,j W0 ; U,_J), the discrepancy for the realisation up to timet, is then

where .6., t: and 1J are understood as expressed in terms of x, y and u by appeal to the plant and observation relations (1), (2) and the initial relation .6.o = xo- xo.

(51)

The projection estimate of the course of the process up to timet, {x~'l; 0 ~ T::;;;_ t}, is obtained by minimising [b, with respect to the corresponding x.,.-values. However, the nearer the model is to being deterministic, the less natural does this formulation seem. The covariance matrices whose inverses appear in (11) and (50) will be singular if any noise component is identically zero, and this plainly presents problems in discussing and minimising these forms. The resolution turns out to be a continuation of the Lagrangian reformulation of Section 6.3 for the deterministic case, which indeed carries over naturally to the stochastic imperfectly observed model. That is, we view the plant and observation equations together with the initial condition (51) as constraints which are best taken care of by the introduction of Lagrange multipliers. Let us introduce multipliers /.,.to take account of constraint (51) in the case T = 0 and the plant equation at time Tin the case T > 0; correspondingly, a multiplier m.,. to take account of the observation relation at time T. One then has the Lagrangian form I

[b, -JQ(xo- xo- .6.o)- L[l;(x.,.- Ax.,._I- Bu.,._I- t:.,.) 'T=l

+ m~(y.,.- Cx.,._I -7].,.)], to be minimised with respect to x and the noise variables and to be maximised with respect to the multipliers. Properly, the multipliers should have a superscript t to indicate that the extremisation is one based on information at time t. However, for notational simplicity we shall include this superscript only when the point needs emphasis.

250

THE LQG MODEL WITH IMPERFECT OBSERVATION

Theorem 12.9.1 If the noise variables are minimised out ofthe Lagrangian form then the problem is reduced to that ofrendering the time integral I

0(/,m,x) =[!/TV/+ /T(x-

x)Jo + L[v(IT,mT)

(52)

T=!

+ l;(xr- AXr-1- BUr-!)+ m".;(yr-

CXr-i)]

maximal with respect to the x variables and minimal with respect to the (J, m) variables. Here

(53) and the noise estimates are related to the multipliers by _ • (1) v.o l0(t) -_ uA 0(1) -Xo-x 0 ,

(54)

(55)

Proof Equations (54) and (55) are just the stationarity conditions for the Lagrangian form with respect to the noise variables. At the values thus determined the Lagrangian form reduces to -0 (I, m, x), and remaining assertions follow. D Note that the transformation clears the form of matrix inverses. Note also that 0 is the analogue of the Lagrangian form (6.19) for the deterministic control case, with (1, m, x) replacing (x, u, .A). The quadratic form v(l, m) is just the information analogue of the instantaneous cost c(x, u) defined in (4). In general, we shall see that the primal and dual variables switch roles when we move from the control problem defined on the future to the estimation problem defined on the past . . The multipliers I and m are to be regarded as differentials of []) 1 with respect to the values of noise variables, which in turn implies the relations (54) and (55) between multipliers and noise estimates. By taking the extremisations of the form 0 in various orders we can obtain the analogues of various familiar assertions. Firstly, by writing down all the stationarity conditions simultaneously we obtain the equation system

[% ~ !][:f+[~l~o

(1

~ T ~

(56)

t).

together with the end conditions

Vl~1 )

=

io-

xg>,

1~1 )

= AT1~ 1 ) + CTm\1>,

(/,m)~1 )

=0

(T

> t).

(57)

9 ESTIMATION AS DUAL TO CONTROL

251

In (56) we have again used the operator notation

.PI= I- AY', with .il = I - AT .r- 1 etc. Note also that the translation operator acts on the subscript r, not on the superscript t, which is regarded as fixed for the moment. In equation (56) we see the complete dual ofthe corresponding system (6.20) for the control problem. We can use the operator and operator factorisation approach just as we did in Section 6.3 and 6.5; the analogue of an infmite horizon will now be the passage from a start-up point at r = 0 to r --> -oo. We shall follow through these ideas explicitly and for a more general case in Chapter 20.

Theorem 12.9.2 If one extremises out the x-variab/es in 0then the problem is reduced to that ofminimising theform I

!WVZ) 0 + L[v(IT,m,.)- iJBu.,.-1 +mJyTJ

(58)

7"=1

with respect to the variable (1, m), subject to the backward equation

(0

~

'T

~

t)

(59)

and end-condition (l,m)T = 0 (r > t). The assertion is immediate. Its point is that exhibits the estimation problem as the complete analogue of the control problem originally posed in Section 2.4 Equation (59) is the analogue of the plant equation, v(l, m) is the analogue of the instantaneous cost, and the initial term! (fT Vl) 0 is the analogue of the terminal cost. The difference lies only the occurrence of a few terms linear in (I, m) in the sum (58)-the observation and control variables constitute effective input terms which lead to a non-homogeneity. However, we are in general not interested in estimating all of process history, but merely the value of current state x 1• For this we resort to recursive arguments, just as the optimal control was deduced by the recursive argument of the dynamic programming equation.

Theorem 129.3 The extremum of form (52) for prescribed x, and I, is zT VI + IT (X - X)], + ... , where + ... indicates terms independent of these variables.

[!

Proof Let us drop the t-subscripts for the moment The constrained extremum is certainly of the form ! zT PI + [T (x - J1.) + · · · for some P, p.. If we minimise with respect to I then this becomes -! (x .- p.)T p- 1(x- p.) + · · ·. But this must be identical with-! (x- x? v- 1(x- x) + ... ,whence the assertion follows. 0

252

Theorem 12.9.4

THE LQG MODEL WITH IMPERFECT OBSERVATION

The recursion

holds, with lr-1 =AT/+ cTm.

(61)

The Kalman filter (22) and the forward Riccati recursion (24)for Vfollow. Proof It follows from the previous theorem and expression (52) for the time integral that we have the recursion

[!IT VI+ IT (x- x)], =ext{[! fT VI+ IT (x- x)],_ 1 + v(/1, m1)

+ l{(xr- Axr-1- Bu,_i) + m"J(Yr- Cx,_i)},

(62)

where the extremum is with respect to lr-1,Xr-1 and m 1• The extremum with respect to x 1_ 1 yields the condition (61) and the recursion reduces to (60). The Kalman and Riccati relations then follow immediately. 0 The reader may ask why we should bother with yet a third derivation of the Kalman filter. Well, the derivation is incidental to the goal; a formalism· which is claimed as significant could hardly be regarded as living up to that claim unless it delivered the Kalman filter in passing. One point is that the Lagrangian approach, already seen as valuable for the opt~isation of deterministic control, is now seen to extend to this stochastic imperfectly observed case. More significantly, it achieves the goal which has loomed ever more clearly as desirable: the characterisation of the optimisation problem as the free extremisation of a timeintegral such as (52) with respect to its variables. Constraints can of course not be eliminated, but they can be exhibited as natural consequences of a free optimisation in a higher-level problem. This higher-level formulation was achieved in two steps. The first was the deduction of the certainty equivalence principle, which absorbed the constraint constituted by the realisability condition (that control must depend on current observables alone). The second was the introduction of the dual variables I and m, which absorbed the constraints constituted by plant and observation equations. Comparison of the equation systems (6.20) and (56) demonstrates the complete symmetry between control/future and estimation/past, with primal variables (x, u) and dual variables (/, m) switching roles in the two cases. There are other byproducts in terms of insight and solution: reduction of these equation systems by the canonical factorisation methods of Section 6.3 solves the stationary optimisation problem for stochastic dynamics of any order (see Chapter 18-21). Remarkably, insight becomes complete first when we consider the risk-sensitive models of Chapters 16 and 21, for which past and future time-integrals are

~;,

9 ESTIMATION AS DUAL TO CONTROL

253

~corporated in a si~gle_ integral, and t~e Lagrange multipliers I and >. associated ;~th the plant equation lD the past

and 1n the future can be related.

~-;::.

Exercises and comments (1) Alternative forms. By varying the order in which the extrema are taken in the last equation, demonstrate that the Kalman filter (22) holds with the alternative evaluation of Hand recursion for Y.

(63)

v, = N +A(CTM- 1C+ v,-=_11)- 1.AT.

(64)

Here A= A- LM- 1C and N = N- LM- 1LT. Relations (63)/(64) have the same continuous-time limit version (28) I (29) as do (23)/ (24), just as was the case for the corresponding control relations.

CHAPTER13

Stationary Process; Spectral Theory 1 STATIONARITY: THE AUTOCOVARIANCE FUNCTION Consider a system which is intrinsically time-homogeneous and for which a stationary control rule is adopted. If this control rule is successful, in that it is stabilising, then the system will eventually settle down to steady-state behaviour. A stochastic process showing such steady-state behaviour is itself termed stationary. More technically, a process is stationary if it is statistically invariant under time-translation. To appreciate what is meant by this, consider first the discretetime case. Denote the process by {x 1}, so that x is the process variable, and denote a realisation (i.e. the actual course of the whole process in a particular case) by X. Recall the backward translation operator §' introduced in Section 4.2, which has the effect that the tth term in the transformed sequence ff X is x 1_ 1 rather than x 1• More generally, the tth term in the sequence ff' X is x1_,, for any integral r. One says then that the process {x 1} is stationary if

E[¢(:1"' X)]

= E[¢(X)]

(1)

for any integral r and for any scalar functional ¢ of the realisation for which the right-hand member in (1) is defined. This is exactly what is meant by 'statistical invariance under time-translation~ Note that (1) will in fact hold for all r if it holds for r = ±1. Our approach to control optimisation hitherto has been to optimise in the finite horizon initially and then see if the optimal control tended to a stationary form in the infinite-horizon limit. The process may then itself converge to stationarity in the course oftime (i.e. in the infinite-history limit). However, one could move to the stationary situation immediately, by asking for the stationary rule which minimises an expected cost per unit time in the stationary state. We make some observations on this point in Exercise 2, but it is clear, in any event, that there is interest in the study of the stationary case. Indeed, under LQ assumptions there are two principal bodies of technique: one dealing with the state-structured case by recursive methods and the other dealing with the stationary case by transform methods. If we consider LQ models then there is an assumption that x is vector-valued (a column vector of dimension n, say) and that the only functions of X which we need consider are linear/quadratic. Indeed, if the process is Gaussian, then its statistics are completely characterised by its first-and second-order moments.

256

STATIONARY PROCESSES; SPECTRAL THEORY

As far as first-order moments are concerned, the stationarity condition (1) will imply that E(x,) is independent oft, so that E(x,) = J1-

(2)

for some constant n-vector Jl-. As far as second-order moments are concerned, (1) will imply that that the covariance between x, and x,_, is a function of r alone: cov(x 1, x,_,) =

v,,

(3)

say, for integral r. The quantity v, is termed the autocovariance at lag r of the process. It is an n x n matrix whose jkth element is the covariance between the jth element of x, and the kth element of x,_,. It provides the only measure one has of dependence between values of the process variable at different times, if one is restricted to knowledge of first and second-order moments. It is therefore the only measure there is at all in the Gaussian case. If one regards v, as a function of the lag r then it is termed the autocovariance function. The full expression

v, = E[(x,- Jl-)(x,_,-

Jl-lL

(4)

for v, and stationarity imply that v_, = v~, or - v_,. T v,-

(5)

That is, the autocovariance function is invariant under the combined operations of transposition and lag-reversal. This plus the further property indicated in Exercise I are in fact the characterising properties of an autocovariance function. Commonly one supposes that the process has been reduced to zero mean (by adoption of a new variable x - J.l-). In this case (4) reduces to v, = E[x,xJ_,]. A degenerate but important stationary process is vector white noise, {e,}, for which v, is zero for non-zero r. One normally supposes that this has zero mean, and, if vo = V, then one speaks of {e,} as 'white noise of covariance matrix V~ The corresponding assertions for the case of continuous time are largely immediate; one simply allows the time variable t and the lag variable r to take any value on the real line. The one point that does need special discussion is the character of white noise, already covered to some extent in Section 9.5. Discretetime white noise {e,} of covariance matrix Vhas the property that the linear form I:;, o.,e, has zero mean and covariance matrix I:;, o., V a.J, at least for all sequences of matrix coefficients { o.,} such that this last sum is convergent. The corresponding characterisation of continuous-time white noise is that the linear form o.(t)eJt) dt is normally distributed with zero mean and covariance matrix Ja.( t) V a.( t) dt. The additional property of normality follows from the assumed independence and stationarity properties. If e had autocovariance function v(r) then the relation

J

2 THE ACTION OF FILTERS: THE SPECTRAL DENSITY MATRIX

cov(l o:(t)E(t) dt)

=I I

257

o:(tJ)v(tt- t2)o:(t2)T dt1 dt2

would hold in regular cases. The evaluation asserted implies tb.en that v(r) = Vc5(r), where c5 is the Dirac delta function; an indication of the exceptional cb.aracter of continuous-time white noise. The matrix V is the covariance matrix of the integral off over a unit time interval; it is appropriately referred to as the power matrix of f. Exercises and comments 2

(1) Note that L:i L:k aJ "i-kak = E(L:, aJ x, ) ;::J: 0, for any sequence of column vectors {a,} for which the first sum coverges. (2) Note that there is no assumption in the dynamic programming equation that the controlled process is stationary, even when one is considering an assumed infinite-horizon limit (as with the simple rule u, = Kx1). One may say that this is because the control thus derived, although stationary, optimises passage to the steady state as well as performance in the steady state. If one tries to determine an optimal steady-state rule directly then one denies oneself an instrument (the reaction against transients) and may derive a rule less effective in dealing with such transients. For example, the rule u1 0 is optimal for the deterministic LQ problem of Section 2.4 if the equilibrium x 1 = 0 has already been reached. It will not be optimal if x is not zero, and will not even be stabilising unless x = 0 is a stable equilibrium of the uncontrolled process. This is an extreme example of a policy which is 'average-cost optimal' but not optimal If non-degenerate plant noise is present (as in Section 10.1, for N > 0) then there is only one policy which is average-cost optimal. This is the infinitehorizon optimal policy, which is optimal for arbitrary initial conditions, and so also copes optimally with transients.

=

2 THE ACTION OF FILTERS: THE SPECI'RAL DENSITY MATRIX The use of z-transforms and Fourier transforms is as natural in this context as it was in Chapter 4, and for the same reason: because of the translation-invariant character of the process. Consider again the discrete-time case and define the autocovariance generating function (abbreviated AGF) 00

g(z) =

L

v,?!,

(6)

r=-oo

where z is a complex scalar. The AGF is then ann x n matrix whose elements are functions of z. Property (5) implies that g(z) = g(z- 1)T, a relation which we shall write as g = g, consistent with the usage of Sections 6.3 and 6.5. It follows then

258

STATIONARY PROCESSES; SPECTRAL THEORY

that, if the doubly infinite series (6) converges at a given value of z, then it also converges at z- 1• In particular, if v, decays as p' with increasing r, for some scalar pin [0, 1), then g(z) converges in the annulus p < lzl < p- 1• Note that a white noise process of covariance matrix Vhas a constant AGF: g( z) = V. The AG F may well not converge anywhere. The standard example is the simple sinusoid x 1 = sin(wot - '1/J ), which defines a stationary process if the phase '1/J is assumed to be a random variable uniformly distributed over (-1r, 1r). Its autocovariance function is v, = cos( wor), so thatthe series (6) indeed converges nowhere. A regular class of stationary process is constituted by those for which x is the output of a stable system obeying finite-order linear dynamics and driven by white noise. We shall see below that g(z) is then rational and necessarily convergent in an annulus p < lzl < p- 1• The important property of the AGF is that it transforms very much more pleasantly under the action of a filter on the process than does the autocovariance function itself. Suppose that a process {y1} is derived as the output of a translation-invariant linear filter with stationary input {x 1}:

!

(7) Here we have used the notation of Chapter 4, so that b, is the transient response of the filter and B(z) = Ls b,z' its generating function; the transfer function of a discrete-time filter. If the sum in (7) is convergent (the appropriate sense here being that of convergence in mean square) then the output {y 1} is defined and also stationary. Since we are dealing with more than one process then autocovariances etc. must be labelled by the process to which they refer. Let us denote the autocovariance and the AGF for the process {x1} by v~x) and g(x)(z), etc. Theorem 13.2.1 Suppose that the functions g(x)(z) and B(z) are both analytic in an annulus p < lzl < p- 1• Then the sum (7) is mean-square convergent, the AGFg(yl(z) ofthe output is analytic in the same annulus and output statistics are related to input statistics by (y) - '"' '"'b

v,

(x)

bT

- ~ ~ 'ivr-j-j+k k' j

(8)

k

(9) Proof Relations (8) and (9) are those which would be established by formal argument: assertion (8) follows from (7) and the definition of an autocovariance; assertion (9) then follows by the formation of generating functions. Convergence of the sum (7) and validity of these manipulations is amply justified by our

2 THE ACTION OF FILTERS: THE SPECTRAL DENSITY MATRIX

very strong assumptions: that v~x) and

ing,,,.

br

259

converge to zero as plrl with increas-

0

The assumptions of the theorem are excessively strong, but make for simplicity, and are justified in the rational case which largely covers our needs. Some relaxation is indicated in Exercise 2, but substantial relaxation takes one deep into Fourier theory. The point of the theorem is that the transformation rule (8) for the autocovariances themselves is not particularly pleasing, but the rule (9) for the transformation of AGFs is both compact and suggestive. We shall often write it simply as

A prime example is a statistical version of a model already considered in Section 4.4:

y=Cx,

A(ff)x =e.

Here A(ff) is polynomial in :T and e is white noise of covariance V, so that the output y is a linear function of a variable x generated by a stochastic difference equation. Stability requires that jA(z)j should have all its zeros strictly outside the unit circle. Under these circumstances the processes converge to stationarity with g(yl(z)

= CA(zri V A(zfi cT.

This is certainly rational, and free of poles in a neighbourhood of the unit circle. The AG F can be given a more physical basis, if, as in Section 4.2, we see the power series (6) as a Fourier series. Let us indeed define

L 00

f(w) = g(e-iw) =

v,e-irw,

(10)

r--oo

where the variable w is to be regarded as 'frequency'. The transformation (10) with w real amounts to considering g(z) on the unit circle, where we expect it to converge if it converges anywhere. Suppose x scalar, for simplicity. We have 0 ~h-I Ej

h-1

h-1 h-I

h

t""O

j,, k=O

rz-h

L x,e-itwl2 =h-I L L Vj-kei(k-j)w = L (1 -

jrjjh)v,e-irw.

This last expression will converge to f(w) with increasing h if regularity conditions are satisfied; a sufficient condition is that Er jv,j < oo. We see then that, under such conditions,J(w) is real and positive and can be identified with the expected squared modulus of the Fourier transform of the sequence {x,}, in an appropriately normalised and limiting sense. This argument leads to an

260

STATIONARY PROCESSES; SPECTRAL THEORY

interpretation of/(w) as the density of the expected distribution of 'power' in the process {x 1} over the frequency spectrum. For this reason it is termed the spectral density function in the scalar case, and the spectral density matrix in the vector case. For the sinusoidal process sin(wot - 1/J) mentioned above the Fourier series (10) of course does not converge, but limiting arguments evaluate it as /(w) == ~ [o(w- wo) + o(w + wo)]. That is, the energy of the process is indicated as being concentrated in the frequencies ±wo, which is indeed consistent with the nature of the process. Relation (9) becomes, in terms of spectral densities,

(11) where {3(w) = '£, b,e-irw is the frequency response function of the filter (see Section 4.2). This could be regarded as the consequence E[((yl((yl] == {3E[(<xl(Cxl]j3 of a relation ((yl(w) = {3(w)(<xl(w), where (<xl(w) is the (random) Fourier amplitude at frequency w in a Fourier decomposition of x, into frequencies and (Cx) (w) its transposed complex conjugate. In fact, such a decomposition is a delicate matter and one must be circumspect in speaking of a 'Fourier amplitude', but the view is nevertheless a correct one; see Exercise 1. Note that we expect the relation

1 v, = 271"

j" ei'"'f(w) dw,

(12)

-11"

inverse to (10). The continuous-time results are formally analogous, with sums overt replaced by integrals over the whole time axis and integrals over w replaced by integrals over the infinite frequency axis ( -oo, oo). Thus, the mutually inverse pair of relations (10), (12) is replaced by

1

00

v(r)

= -1

271"

eiwrf(w) dw.

-oo

These relations are certainly valid if/(w) exists and is integrable. If we consider the important special case (in continuous time) ofrationalf(w), thenf(w) can certainly have no singularity on the real w-axis, so integrability cannot fail for this reason. However, integrability over the infinite w-axis will also require thatf(w) should tend to zero with increasing w, and should do so sufficiently quickly. The effect of this condition is to exclude a white noise component, since white noise has a constant spectral density. If f(w) is rational and tends to zero with increasing w then it must indeed tend to zero at least as w- 2, and so is integrable. The filter x---+ y expressed by y = B(~)x has transfer function B(s) and frequency response function {3(w) = B(iw). With these understandings, relation (11) holds in regular cases.

3 MOVING-AVERAGE AND AUTOREGRESSIVE REPRESENTATIONS

261

. For a white noise process the SDFf (w) has the constant value V (where Vis the covariance matrix or power matrix, according as to whether the process is in discrete or continuous time). One interprets this as an indication that the energy of the process is uniformly distributed over the range of meaningful frequencies (i.e. (-71', 11') in discrete time and ( -oo, oo) in continuous time). The process was then given the name 'white' noise in the mistaken belief that white light has such a uniform energy spectrum. It certainly does not, even over the visible range, but the term 'white noise' has stuck.

Exercises and comments (1) Circulant processes. The structure of a stationary process can be understood much better if one makes the time axis finite by assuming it periodic, with period m. That is, the complete realisation X consists of the sequence {xo 1 x 1 , x2 , ••• 1 Xm- 1}, and the shift operator !T effects a cyclic permutation, so that !TX = {xm-!.Xo 1 X!, ... ,Xm-2}· 'Stationarity' then just means statistical invariance under $", and so under any cyclic permutation. Assume stationarity in what follows. Show that cov(x, Xr-r) is a function v, of r alone with period m. Suppose, for simplicity, that xis scalar, and let Vbe the covariance matrix of them-vector X Show that V has eigenvalues jj = L:, v,o-jr and that the kth element of the corresponding right eigenvector is fPk, where(}= exp(211'i/m) and sums run over any m consecutive integers. That is, the jth eigenvalue is a 'spectral density' eval~ated at w = 211'j /m 1 and can be i_dentified with El(th wher~ (j is the finite Four1er transform ofthe sequence Xm that (j = m- 112 6 1 XrO-J'. Furthermore, these Fourier amplitudes are uncorrelated, in that E((J(k} = 0 for distinctj, kin the set 0 1 11 ••• 1 m - 1. The effect of a filter with operator B(!T) = Lr b,!T' is to multiply (J by B({}i).

(2) Consider a discrete-time SISO filter driven by white noise, and so with output The necessary and sufficient condition that this expression should converge in mean square (and so that the output should have fmite second-order moments) is that 2::, /?, is finite. The same condition is therefore sufficient if one considers an input whose SDF is bounded above. The condition will also be necessary if the SDF of the input is bounded away from zero.

L:, b,f1_,.

3 MOVING-AVERAGE AND AUTOREGRESSIVE REPRESENTATIONS: CANONICAL FACTORISATIONS Consider the process {x1} obtained as the output of a filter with a white noise input: Xr

=

L

b,fr-r·

(13)

262

STATIONARY PROCESSES; SPECTRAL THEORY

In the time-series literature such a process would be referred to as a moving average process, the words 'average' and 'moving' being a loose indication of the properties 'linear' and 'translation-invariant: If the white-noise process has covariance matrix V then it follows, as a special case of (9), that {x 1 } has AG F

g(z)

= B(z) VB(z)

(14)

If the filter is causal then b, is zero for r negative, so that B(z) is a series in nonnegative powers of z. One then speaks of the moving average relation itself as being 'causal' or 'realisable: Suppose, as in Section 4.4, that the filter is specified as an input-driven linear difference equation 00

A(ff)x1 = L:a,x1-r = ft

( 15)

r=O

We regard (15) as a relation determining x 1 in terms of f 1 and past x-values, so that

a, is indeed zero for r negative and ao is non-singular. If the relation (15) is stable then the output {x1} generated is stationary, and can indeed be represented as a realisable moving average of the input. That is, it can be represented in the form 1 (13) with B(z) = A(z)- , where these expressions are understood as power series in non-negative powers of z (see Section 4.4). Its AGFis then

g(z) = A(z)- 1 V A(z) - 1•

(16)

In the time-series literature a process generated by a relation such as (15) is termed an autoregressive process, the relation itself being an autoregression. The term conveys, again loosely, the idea that the variable is linearly dependent upon its own past. The relation (15) is of course just a stochastic difference equation, linear and with constant coefficients, and of order p if a, is zero for r > p. Let us now reverse these deductions. If the AGF has the form (14) then the process could have been generated by a moving average relation (13); if the AGF has the form (16) then the process could have been generated by an autoregressive relation (15). In other words, one would have, respectively, moving average and autoregressive representations of the process. The point of this manoeuvre will appear in the next section. Let us make it more explicit.

Theorem 13.3.1 Suppose that the AGF g(z) of a process x 1 can befactorised in the form (14), where both B(z) and B(z) -I are analytic in /z/ ::;;; 1. Then the process has a causal moving average representation (JJ), with V identified as the covariance matrix ofthe white noise process { f 1}. Furthermore, this representation is invertible, in that it can be inverted to the autoregressive relation (/5), withA(z) = B(zf 1• Proof Define a process { f 1 } by

3 MOVING-AVERAGE AND AUTOREGRESSIVE REPRESENTATIONS

263

where it is understood that the expansions are in non-negative powers of§. It follows from the assumptions of the theorem that this quantity is well-defined, so that the process { fr} is stationary and both the relations (13) and (15) hold between the two processes. We see now by appeal to relation (9) that g(E)

= AgA = B- 1gir 1 = V,

so that the process {fr} is white, with covariance matrix V.

0

The factorisation (14) demanded in the theorem is a canonical factorisation of gt). This type of factorisation will occur repeatedly as we look, essentially, for a moving-average or autoregressive representation of the process. The factorisation is indeed possible if g(z) is an AGF and is analytic in a non-empty annulus p < izl < p-1. We can restate the theorem in what is a completely equivalent form, but with an emphasis whose difference will prove significant. Theorem 13.3.2 Suppose that the AGF g(z) ofa process x 1 is such that its inverse as a matrix can be factorised in the form

(18) where both A(z) and A(z)- are analytic in izi ~ 1. Then the process has a stable autoregressive representation (15), with V identified as the covariance matrix of the white noise process { fr}. 1

This is indeed just a variation of the statement of the previous theorem. Note, however, that it is expressed in terms of a canonical factorisation of g(zr 1 rather than of g(z), and that the order of factors is reversed, in that it is the final factor rather than the initial factor which is required to have regularity properties inside the unit circle. Of course, one factorisation immediately implies the other, but the distinction between the two versions will prove significant. Roughly, when it comes to state estimation (as it will) then factorisations (14) and (18) are respectively equivalent to the two characterisations of a projection estimate as a linear least square estimate or as a minimum discrepancy estimate. When we deal with processes whose dynamics are of finite order (as we shall) then factorisation (18) is a polynomial factorisation, whereas (14) is not. The fr defined by (17) are in fact just the innovations of the x-process (assuming this to have begun in the infinite past) and the realisable moving-average representation (13) is just a representation of the process in terms of its own innovations. It is a special case of a representation deduced by Wold (1938). The Wold representation breaks x into the moving-average term deduced above (the purely non-deterministic component) and a term which can be perfectly predicted

264

STATIONARY PROCESSES; SPECTRAL THEORY

by linear relations. Our factorisation hypotheses exclude the possibility of this second, deterministic, component. 4 THE WIENER FILTER Suppose that one has observed a process {x 1} up to time t, and so knows its partial history X 1 = {x 7 ; T :::::; t}. Suppose that one wishes to use the observations to estimate the value of some unknown random variable f It is natural to consider the projection estimate ~(t) = th"(~JX1 ), which we know, from the last chapter, to have optimality properties. We suppose all variables reduced to zero mean.

Theorem 13.4.1 Suppose that the AGFofthe process {x 1} has canonicalfactorisation (13). Then the projection estimate of~ in terms of X 1 is given by DO

c(t)-'"'

<,

-

( 19)

L.....J rrXt-Tl r=O

where

~ ,,zF = ( ~ ~,zF) v- B(z) 1

(20)

-I

and

(21)

Proof We can equivalently project~ upon the E-history, so obtaining DO

DO

DO

~(t) = Lth"(~IEt-r) = 2:~,V- 1 Et-r = 2:~,5""'V- 1 B(ffr xt 1

r=O

r=O

r=O

whence (19) follows. The first equality is a consequence of the orthogonality of the E1• D Essentially, one obtains the estimate by projecting upon x-innovations and then expressing these innovations in terms of observed x. The filter (19) which determines ~(t) from X 1 is the Wiener filter. It is generally stated for the particular case ~ = xt+m, when one is attempting to predict what the value of the process variable will be m steps in the future. Exercises and comments

(1) Show that, for the prediction problem last indicated, solution (20) for the ,_ coefficients becomes E~o ,,z' = [z-m B( z) l+B( z I

r

0

5 OPTIMISATION CRITERIA IN THE STEADY STATE

265

(2) Denote the predictor of Xt+m thus determined by x~2m Show that one could have e~uivalently have calculated these predictors by application of the relations E a,x}l, = 0 (r > t). That is, by repeated appeal to the autoregressive relation, future E set equal to zero. This is indeed an immediate consequence of the fact that 8(E.,.IX1) = 0 forT> t, which is itself a consequence of the white-noise assumptions.

wifu

5 OPTIMISATION CRITERIA IN THE STEADY STATE Optimisation criteria will often have a compact expression in terms of spectral densities in the stationary regime (i.e. the steady-state limit). Whether one best goes about steady-state optimisation by a direct minimisation of these expressions with respect to policy is another matter, but the expressions are certainly interesting and have operational significance. Consider first the LQG criterion. Let us write an instantaneous cost function such as (2.23) in the 'system' form (22) Here A is the vector of 'deviations' which one wishes to penalise (having, for example, x - r and u - rr as components) and 9t is the associated matrix (which would be just [ ~

~]

in the case (223)). Suppose that { A 1} has auto-

covariance generating function g(z) in the stationary regime under a given stabilising policy -n; and corresponding spectral density f(w). Then the aim is to choose 1r to minimise the average cost 'Y = E(!AT9tA] = !E{tr[9tAAT]} =! tr[9t cov(A)].

(23)

Here E denotes an expectation under the stationary regime for policy 1r and tr(P) denotes, as ever, the trace of the matrix P. For economy of notation we have not explicitly indicated the dependence of,, E and g(z) upon 1r. Appealing to the formula cov(A) =

2~ Jf(w) dw,

we thus deduce Theorem 135.1

The criterion (23) can be expressed 'Y = E(!AT9tA] =

~

J

tr[9t.f(w)J dw

(24)

where f(w) is the spectral density function of the A-process under the stationary regime for policy 11; and the integral is is over the realfrequency interval [-1r, n] in the case ofdiscrete time and over the whole rea/line in the case ofcontinuous time.

266

STATIONARY PROCESSES; SPECTRAL THEORY

In the discrete-time case it is sometimes useful to see expression (24) in power series rather than Fourier terms, and so to write it

I -y=!Abs{tr[mg(z)]}=-.

47rl

j

dz tr(mg(z)]-.

z

(25)

Here the symbol ~bs' denotes the operation of extracting the absolute term in the expansion of the bracketed term in powers of z upon the unit circle, and the integral is taken around the unit circle in an anticlockwise direction. If we had considered a cost function

(26) up to a horizon h then we could have regarded the average cost (23) as being characterised by the asymptotic relation

E(C) = h-y + o(h)

(27)

for large h. Here E is again the expectation operator under policy 1r, but now conditional on the specified initial conditions. The o(h) term reflects the effect of these conditions; it will be zero if the stationary regime has already been reached attime t = 0. Consider now the LEQG criterion introduced in Section 123. We saw already from that section that the LEQG model provided a natural embedding for the LQG model; we shall see in Chapters 16, 17 and 21 that it plays an increasing role as we bring in the concepts of risk-sensitivity, the Hoo criterion and largedeviation evaluations. For this criterion we would expect a relation analogous to (27):

(28) Here -y(0) is a type of geometric-average cost, depending on both the policy 1r and the risk-sensitivity parameter 0. The least ambitious aim of LEQG-optimisation in the infinite-horizon limit would be to choose the policy 1r to minimise -y(O). ('Least ambitious', because a full-dress dynamic programming approach would minimise transient costs as well as average costs.) We aim then to derive an expression for -y(O) analogous to (24).

Theorem B.5.2

The average cost -y( 0) defined by (48) has the evaluation -y(O) =

!j

4 0

logll + (}mf(w)i dw

(29)

for values of 0 such that the symmetrisation of the matrix I+ (}mf(w) is positive definite for all real w. Here f (w) is the spectral density function of the D.-process under the stationary regime for the specified policy 1r (assumed stationary and

5 OPTIMISATION CRITERIA IN THE STEADY STATE

267

linear), and the integral is again over the interval [-1r, 1r] or the whole real axis in discrete or continuous time respectively. Here IPI denotes the determinant of a matrix P; note that expression (29) indeed reduces to (24) in the limit () ---+ 0. We shall prove an intermediate lemma for the discrete-time case before proving the theorem. Suppose that the AGF of {6. 1} has a canonical factorisation (16) with A(z) analytic in some annulus lzl :::;; 1 + b for positive b. That is, the 6.process has an autoregressive representation. It is also Gaussian, since the policy is linear. Let us further suppose this representation so normalised that Ao =I. The probability density of 6. 1 conditional on past values is then f(6.rl6.r; T < t)

=

[(27r)miVI]- 112 exp[-! E; v- 1 E1]

where t:: 1 = A(ff)6. 1 and m is the dimension of 6.. This implies that

f(6.o, 6.1' ... '6.h-d6.r; T < 0) = [(21ft! Vlrh12 exp [ -!

~ [(2~)'1 v1r'

12

exp [

-l

t,

~ El v- 1Er] (30)

lli M(ff)ll, + o(h)]

where M(z) = A(z) v- 1A(z) = g(zr 1• Since the multivariate density (30) integrates to unity, and since the normalisation Ao = I implies the relation logl VI = dloglg(z)l, we see that we can write conclusion (30) as follows.

Lemma 13.5.3 Suppose that M (z) is self-conjugate, positive definite on the unit circle and analytic in a neighbourhood ofthe unit circle. Then

j j ... j exp[-!~xi"M(ff)x,]dxodxl .. . dxh-1 =

(21f)hmfZexp{ -(h/2)Abs[logiM(z)IJ

(31)

+ o(h)}.

*Proof of Theorem 13.5.2 The expectation in (28) can be expressed as the ratio of two

multivariate integrals of type (31), with the identifications 1 + ()9l and M(z) = g(z)- 1 in numerator and denominator respectively. We thus have

M(z)

= g(zr

E(e-ec) = exp[-(h/2)Abs{logjg(z)- 1 + 09lj + logjg(z)l} + o(h)] whence the evaluation

'"'!(B) =

1

20

Abs[logl/ + B9tg(z) I}

268

STATIONARY PROCESSES; SPECTRAL THEORY

and its alternative expression (29) follow. The continuous-time demonstration is analogous. 0 The only reason why we have starred this proof is because the o(h) term in the exponent of the final expression (30) is in fact 6.-dependent, so one should go into more detail to justify the passage to the assertion (31).

CHAPTER14

Optimal Allocation; The Multi-armed Bandit 1 ALLOCATION AND CONTROL This chapter is somewhat off the principal theme, although on a topic of considerable importance in its own right, and the reader may bypass it if he wishes. Allocation problems are concerned with the sharing of limited resources between various activities which are being pursued. This not the kind of problem envisaged in classical control theory, but of course is indeed a control problem if this allocation is being varied in time to meet changing conditions. For example, the adaptive allocation of transmission capacity in a communications network provides just such an example; clearly of fundamental technological importance. The classic dynamic allocation problem is the 'multi-armed bandit', henceforth referred to as MAB. This is the situation in which a gambler makes a sequence of plays of any of n gambling machines (the 'bandits'), and wishes to choose the machine which he plays at each stage so as to maximise the total expected pay-off (perhaps discounted, in the infinite-horizon case} The pay-off probability of the ith machine is a parameter, (Ji say, whose value is unknown. However, the gambler builds up an estimate of (Ji which becomes ever more exact as he gains more experience of the machine. The conflict, then, is between playing a machine which is known to have a good value of (Ji and experimenting with a machine about which little is known, but which just might prove even better. It is in order to resolve this conflict that one formulates the problem as an optimisation problem. As an allocation problem this is quite special, on the one hand, but has features which confuse the issue, on the other. The resource which one is allocating is one's time (or, equivalently, effort) in that one can play only one machine at a time, and must decide which. In more general models one will be able to split the allocation at a given time. The problem also has a feature which is fascinating but is irrelevant in the first instance: the 'state' of the ith machine at a given time is not its physical state, but is the state of one's information about the value of 01, an 'informational state~ We shall see how to handle this concept in the next chapter, but this aspect should not divert one from the essential problem, which is to decide which machine to use next on the basis of something which we may indeed term the current 'state' of each machine.

270

OPTIMAL ALLOCATION; THE MULTI-ARMED BANDIT

In order to sideline such irrelevancies it is useful to formulate the MAB problem in greater generality. We shall suppose that one has n 'projects', the ith of which has state value x;. The current state of all projects is assumed known. One can engage only one project at a time; if one engages project i then X; changes to a value by time-homogeneous Markov rules (i.e. the distribution of conditional on all previous state values of all projects is in fact dependent only on x;), the states of unengaged projects do not change and there is a reward whose expectation is a function r;(x;) of x; and i. If one envisages indefinite operation starting from time t = 0 and discounts rewards by a factor {3 then the aim is to choose policy 1r so as to maximise E,.[:L::::o {31R 1], where R 1 is the reward received at time t. The policy is the rule by which one decides which project to engage at any given time. One can generalise even this formulation so as to make it more realistic in several directions, as we shall see. However, the problem as stated is the best first formalisation, and captures the essential elements of a dynamic allocation problem. The problem in this guise proved frustratingly difficult, and resisted sustained attack from the 'forties to the 'seventies. However, it had in fact been solved by Gittins about 1970; his solution became generally known about 1981, when it opened up wide practical and conceptual horizons. Gittins' solution is simple to a degree which is found amazing by anyone who knew the frustrations of earlier work on the topic. One important feature which emerges is that the optimal policy is an index policy. That is, one can attach a index v;(x;) to the ith project which is a function of the project label i and the current state x; of the project alone. If the index is appropriately calculated (the Gittins index), then the optimal policy is simply to choose a project of currently greatest index at each stage. Furthermore, the Gittins index v; is determined by the statistical properties of project i alone. We shall describe this determination, both simple and subtle, in the next section. The MAB formulation must be generalised if one is to approach a problem as complicated as, for example, the routing of telephone traffic through a network of exchanges. One must allow several types of resource; these must be capable of allocation over more than one 'project' at a time; projects which are unengaged may nevertheless be changing state; projects may indeed interact. We shall sketch one direction of generalisation in Sections 5-7. The Gittins solution ofthe MAB problem stands as the exact solution of a 'pure' problem. The inevitable next stage in the analysis is to see how this exact solution of an idealised problem implies a solution, necessarily optimal only in some asymptotic sense, for a large and complex system.

x;

x;

2 THE GITTINS INDEX The Gittins index is defined as follows. Consider the situation in which one has only two alternative actions: either to operate project i or to stop operation and

2 THE GITTINS INDEX

271

receive a 'retirement' reward of M. One has then in fact an optimal stopping problem (since once one ceases to operate the project, its state will not change and there is no reason to resume operation). Denote the value function for this problem by ¢;(x;, M), to make the dependence on the retirement reward as well as on the project state explicit. This will obey the dynamic programming equation

(1) a relation which we shall abbreviate to

¢; = max[M, 4¢;).

(2)

Here x; and~ are, as above, the values of the state of project i before and after one stage of operation. As a function of M for fixed x; the function ¢; has the form indicated in Figure 1: non-decreasing and convex, and equal to M for M greater than a critical value M;(x;). This is the range in which it is optimal to accept the retirement reward rather than to continue, and M 1(x1) is the crossover value, at which M is just large enough that the options of continuing or terminating are equally attractive. Note that M;(x 1) is not the fair buy-out price for the project when in state x 1; it is more subtle than that. It is the price which is fair (in state x 1) if an offer is made which is to remain open, and so which the project operator is free to accept at any time in the future. It is this quantity which can be taken as the Gittins index, although usually one scales it to take

v;(x,)

= (1 -

{3)M;(x;)

(3)

as the index. One can regard vas the size of an annuity that a capital sum M would buy, so that one is rephrasing the problem as a choice between the alternatives of either operating the project with its uncertain return or of moving to the 'certain' project which offers a constant income of v.

Figure 1 The graph of¢>1(x 1, M) as afunction of M Here X; and Mare respectively the state of project i and the reward ifthe retirement option is accepted, and ¢ 1(x;, M) is the valuefunction ofthe corresponding optimal stopping problem.

272

OPTIMAL ALLOCATION; THE MULTI-ARMED BANDIT

Note that the index v,(x,) is indeed evaluated in terms of the properties of project i alone. The solution of then-project problem is phrased in terms of these indices: the optimal policy (the Gittins index policy) is to choose at each stage one of the projects of currently greatest index. One may indeed regard this as a reduction of the problem which is so powerful that one can term it a solution, since solution of the n-project problem has been reduced to solution of a stopping problem for individual projects. We shall prove optimality of this policy in the next section, but let us note now an associated assertion. Let x denote the composite state (x1, x2, ... , Xn) of the set of all n projects, and let «<»(x, M) denote the value function for the problem of choosing optimally between any one of these projects and the additional option of total retirement with a terminal reward M. Let A(l - {3) and B(I - {3) be uniform lower and upper bounds (presumed finite) on the reward rates r 1(x1), so that A and B are lower and upper bounds on the total expected discounted reward obtainable if there is no retirement option. Then we shall show that 4>(x, M) has the evaluation in terms of the one-project value functions q)j if»(x,M)

= B-

r

}M

rr3¢;(x;,m) dm. Om

i

(4)

Solution of this augmented problem for general M implies solution of the nproject problem without the retirement option, because if M < A then one will never accept the retirement option. Exercises and comments (1) Prove that ¢1(x1, M) has indeed the character asserted in the text and illustrated in Figure 1.

3 OPTIMALITY OF THE GITTINS INDEX POLICY The value function «J»(x, M) will obey the dynamic programming equation if»= max[M,maxLtCil]

(5)

I

where the operator Lt defined implicitly by comparison of (1) and (2) will act only on the x 1-argument of Cll. We shall prove validity of (4) and optimality of the Gittins policy by demonstrating that expression (4) is the unique solution of (5) and that the Gittins policy corresponds to the maximising options in (5). More explicitly, that one operates a project of maximal index vi if this exceeds M (1 - !3) and otherwise accepts the retirement option. Many other proofs of optimality have now been offered in the literature which do not depend upon dynamic programming ideas; one particular line is indicated in the exercises

3 OPTIMALITY OF THE GITTINS INDEX POLICY

273

below. However, these are no shorter, most of them do not yield the extra conclusion (4), and it is a matter of opinion as to whether they are more insightful.

Lemma 14.3.1 Expression (4) may alternatively be written ll>(x, M) = rPi(x1, M)P1(x, M) where

+ Loo ¢ 1(x;, m) dmP1(x, m)

·=II 8¢j(Xj, aM M)

P,·( x, M) .

(6)

(7)

jf.i

is non-negative, non-decreasing in M and equal to unity for

M > M(i) := mJX Mi.

(8)

Jrl

Proof Note that quantities such as M 1 and M(i) have a dependence upon xwhich we have suppressed. Equation (6) follows from (4) by partial integration. Since ¢ 1, as a function of M, is non-decreasing, convex and equal to M for M ~ M 1, then 8¢;/oM is non-negative, non-decreasing and equal to unity for M;:::: M 1• The D properties asserted for P1 thus follow. Consider the quantity

81(x1,M)

=

¢1(x 1,M)- L 1¢ 1(x1,M)

and note that that 61 ;:::: 0, with equality for M ~ M 1• We are interested for the moment in the dependence of the various quantities on M for fixed x, and so shall for simplicity suppress the x-argument.

Lemma 14.3.2 Expression (6) satisfies the relations II>~

with equality if M

~

(9)

M

maxMj, and 1

II>(M)- L;II>(M) with equality if M 1 =

=

max~ ~

J

8;(M)P;(M)

+!Moo 8;(m)dmP;(m) ~ 0

(10)

M.

Here dmP1(m) is the increment in P1(m) for an incrementdm in m.

Proof Inequality (9) and the characterisations of the equality case follow from (6) and the properties of P;. The first relation of (10) follows immediately from (6). The non-negativity of the expression follows from the non-negativity of 8; and the non-negative and

274

OPTIMAL ALLOCATION; THE MULTI-ARMED BANDIT

non-decreasing nature of P;. We know that o;(M) = 0 for M ~ M; and that dmP;(m) = 0 form;:?!: M(i)• so that expression (10) will be zero if M ~ M; and Mul ~ M;. This pair of conditions is equivalent to those asserted in the lemma.o

Theorem 14.3.3 The value function (x, M) of the augmented problem indeed has the evaluation (4) and the Gittins index policy is optimal. Proof The assertions of Lemma 14.3.2 show both that expression (4) satisfies the dynamic programming equation (2) and that the Gittins index policy (augmented by the recommendation of termination if M exceeds max;M;) provides the maximising option in (2). But since (2) has a unique solution and the maximising option indicates the optimal action (see Exercise 3.1.1, valid also for the stochastic case) both assertions are proved. Exercises and comments We indicate an alternative line of argument which explains the form of solution (4) and avoids some of the appeal to dynamic programming ideas. (1) (Whittle, 1980). Consider a policy which is such that project i is terminated as soon as x; enters a write-off set S; (i = 1, 2, ... , N) and retirement with reward M takes place the moment all projects have been written off. We assume that there is some rule for the choice from the set of projects still in use, which we need not specify. Let us term such a policy a write-offpolicy, and denote the value functions under a given such policy for theN-project and one-project situations by F(x, M) and.fi(x;, M) respectively. Then

oF oM= E(,B'"Ix)),

o.fi ) oM= E (/3.,., Ix;,

where 1 is the (random) time taken to drive all N projects into their write-off sets and 1; the time taken to drive project i into its write-off set. But 1 is distributed as 2:; 7'; with the 7'; taken as independently distributed. Hence it follows that

oF = oM

II o.fi

. oM'

I

which would imply the validity of (4) if it could be asserted that the optimal policy was also a write-off policy. (2) (Tsitsiklis-Weber). Denote the value function for the augmented problem if i is restricted to a set I of projects by V(I). This should also show a dependence on M and the project states which we suppress. Then Vhas the sub modular property

V(I)

+ V(J)

;:?!: V(I U J)

+ V(I n J).

(11)

4 EXAMPLES

275

Prove this by induction on the time-to-go s and appeal to the fact that choice of a one project from each of I u J and I n J can be seen as a choice of one project from each of I and J, but that the converse assertion is not in general true. (3) (fsitsiklis, 1986). Take I as the set of all projects which are written off (in an optimal policy for the full set of n projects) and J as its complement. Then relation (11) becomes M + V(J) ;;?!: 41 + M, or 41 ~ V(J). But plainly the reverse inequality holds, so that

4 EXAMPLES The problem has been reduced to determination of the index v(x) for an individual project, so we can drop the project index i. Determination of v(x) requires solution of the stopping problem for that project, and one may well have to resort to numerical methods at this point. The fact that analytic solution is possible in only relatively few cases does not invalidate the fact that the MAB problem is essentially solved by the reduction of then-project problem to the oneproject problem (with a retirement option). We shall list some examples which in fact permit rapid and transparent treatment. Let us say that a project is deteriorating if ¢(x(t), M) is necessarily nonincreasing in t. One may, for example, have a machine whose state is sufficiently indicated by its age, and whose performance deteriorates with age. We leave the reader to show, from the dynamic programming equation (2), that v(x) = r(x) simply, where r(x) is the instantaneous reward function. If all projects were deteriorating then the optimal policy is a one-step lookahead policy: one chooses the project i for which the expected immediate reward r;(x;) is maximal This will ultimately lead to the situation in which the r; have been roughly equalised for all projects, and one then switches projects to keep them so. That is, it is as if one kept changing the tyres on one's car with the spare so as to keep all five tyres in an equal state of wear. Switching costs will ensure that one in fact tolerates a degree of inequality. The opposite situation is that of an improving project, for which ¢(x(t),M) is non-decreasing with time. For example, the performance of a machine may improve with age in the sense that, the longer it lasts, the more likely it is to be a good one. We leave to the reader to confirm that, in this case,

v(x)

~ (1- fl)E[t,th(x(t))lx(O) ~ x].

276

OPTIMAL ALLOCATION; THE MULTI-ARMED BANDIT

That is, the index is the discounted constant income equivalent of the expected discounted return from indefinite operation of the project. If all projects are improving then, once one adopts a project, one stays with it. However, mention of 'lasting' brings home the possibility that a machine may indeed 'improve' up to the point where it fails, and is thereafter valueless. Let us denote this failure state by x = 0, presume it absorbing and that r(O) = 0. Suppose that the machine is otherwise improving in that ¢(x(t), M) is nondecreasing in t as long as the state x = 0 is avoided. Let a denote the random failure time, the smallest value oft for which x( t) = 0. In this case

v(x) E[L~~~ ,B'r(x(t))lx(O) = x] 1 - {3 = 1 - E[f3o-lx(O) = x] If all projects follow this 'improving through life' pattern then, once one adopts a project, one will stay with it until it fails. Another tractable example is provided by a diffilsion process in continuous time. Suppose that the state x of the project takes values on the real line, that the project yields reward at rate r(x) = x while it is being operated, that reward is discounted at rate a, and that x itself follows a diffilsion process with drift and diffilsion coefficients J1. and N. This conveys the general idea of a project whose return improves with its 'condition', but whose condition varies randomly. The equation for ¢(x, M) is then

(x >e)

(12)

e

where is the optimal breakpoint for retirement reward M. We find the solution of (12) to be

¢(x, M)

= (xfa) + (J1./o2 ) + cePJC

(13)

where p is the negative solution of

! Npl + Jl.P- a = 0. and c is an arbitrary constant. The general solution of (12) would also contain an exponential term corresponding to the positive root of this last equation, but this will be excluded since ¢ cannot grow faster than linearly with increasing x. The unknowns c and ~ are determined by the boundary conditions ¢ = M and ¢x = 0 at X= (see Exercise 10.7.2} If we substitute expression (13) into these two equations then the relation between M and which results is equivalent to M = M(e). We leave it to the reader to verify that the calculation yields the determination

e

e

( ) M( x=x+ vx=a )

J1. +

J J1.2 + 2aN . 20

5 RESTLESS BANDITS

277

The constant added to x represents the future discounted reward expected from future change in x. This is positive even if J.L is negative-a consequence of the fact that one can take advantage of a random surge against trend if this occurs, but can retire if it does not.

5 RESTLESS BANDITS One desirable relaxation of the basic MAB model would be to allow projects to change state even when not engaged, although of course by different rules. For example, one's knowledge of the efficacy of a medical treatment used to combat a particular infection improves as one uses it. However, it could actually deteriorate when one ceased to use the treatment if, for example, the virus causing the infection were mutating. For a similar example, one's information concerning the position of an enemy submarine will in general improve as long as one tracks it, but would actually deteriorate if one ceased tracking. Even if the vessel were not taking deliberate evasive action its path would still not be perfectly predictable. As a final example, suppose that one has a pool of n employees of whom exactly m are to be set to work at a given time. One can imagine that employees who are working produce, but at a decreasing rate as they tire. Employees who are resting do not produce, but recover. The 'project' (the employee) is thus changing state whether or not he is at work. We shall speak of the phases when a project is in operation or not as active and passive phases. For the traditional MAB model a project was static in its passive phase. As we have seen, for many problems this is not true: the active and passive phases produce contrary movements in state space. For submarine surveillance the two phases induce gain and loss of information respectively. For the labour force the two phases correspond to tiring and recovery. We shall refer to a project which may change even in the passive phase as a 'restless bandit', the description being a literal one for the submarine example. The work-force example generalised the MAB model in another respect: one was allowed to engage m of then projects at a time rather than just one. We shall allow this, so that, for the submarine example, we could suppose that m ( < n) aircraft are available to track the n submarines. It is then a matter of allocating the surveillance effort of the m aircraft in order to keep track of all n submarines as well as possible. We shall specialise the model in one respect: we shall assume rewards undiscounted, so that the problem is that of maximising the average reward. This makes for a much simpler analysis; indeed, the known solutions of a number of standard problems are greatly simpler in the undiscounted case. As we have maintained earlier, and argue on structural grounds in Section 16.9, the undiscounted case is in general the realistic one in the control context.

278

OPTIMAL ALLOCATION; THE MULTI-ARMED BANDIT

Let us label the phases by k; active and passive phases corresponding to k = I and k = 2 respectively. If one could operate project i without constraint then it would yield a maximal average reward "f; determined by the dynamic programming equation 'Yi

+/;(x;)

= max{rik(x;) k

+ Ek[l;(.i;)lx;]}.

(14)

Here r;k(x;) is the expected instantaneous reward for project i in phase k, Ek is the expectation operator in phase k and /;(x;) is the transient reward for the optimised project. We shall write (14) more compactly as 'Yi

+ /; = max[L;1 /;, Li2/;]

(i= 1,2, ... ,n).

( 15)

Let m(t) be the number of projects which are active at time t. We wish to optimise operation under the constraint m(t) = m for prescribed m and identically in t; that exactly m projects should be active at all times. Let Ropt(m) be the optimal average return (from the whole population of n projects) under this constraint. However, a more relaxed demand would be simply to require that E[m(t)]

= m,

( 16)

where the expectation is the equilibrium expectation for the policy adopted. Essentially, then, we wish to maximise E("'£; r;) subject to E("'£; l;) = n- m. Here r; is the reward yielded by project i (dependent on project state and phase) and l; is the indicator function which takes the value 1 or 0 according as to whether project i is in the passive or the active phase. However, this is a constraint we could take account of by maximising E["'£;(r; + vl;)], where vis a Lagrangian multiplier. We are thus effectively solving the modified version of (15) "f;(v)

+/; =

max[L;I/;, v +La/;]

(i= 1,2, ... ,n)

(17)

where/; is a function/;(x;, v) of x; and v. An economist would view vas a 'subsidy for passivity', pitched at just the level (which might be positive or negative) which ensures that m projects are active on average. Note that the subsidy is independent of the project; the constraint (16) is one on total activity, not on individual project activity. We thus see that the effect of relaxing the constraint m(t) = m to the averaged version (16) is to decouple the projects; relation (17) involves project i alone. This is also the point of the Gittins solution of the original MAB problem: that the nproject problem was decomposed into none-project problems. A negative subsidy would usually be termed a 'tax: We shall use the term 'subsidy' under all circumstances, however, and shall refer to the policy induced by the optimality equations (17) as the subsidy policy. This is a policy optimal under the averaged constraint (16). If we wish to be specific about the value of v we shall refer to the policy as the v-subsidy policy. For definiteness, we shall close

S RESTLESS BANDITS

279

the passive set That is, if x1 is such that Ltt fi = v + La fi then project i is to be rested. The value of v mwt be chosen so that indeed m projects are active on average. This induces a mild recoupling of the projects.

Theorem 14.5.1 The maximal average reward under constraint (16) is R(m)

= ¥[~ l't(v)- v(n- m)]

(18)

and the minimising value ofv is the required subsidy level. Proof This is a classic assertion in convex programming (that, under regularity conditions, the extrema attained in primal and dual problems are equal) but best seen directly. The average reward is indeed the square-bracketed expression in (18~ because the average subsidy paid must be subtracted. Since

L 11(11) = supE,. [I:(r; + 11l;)l i

,.

I

(where 1r denotes policy) then

8 av4: 1 (11) 1

I

= E1r

4:/

1

= (n- m).

I

This equation relates m and v and yields the minimality condition in (18). The condition is indeed one ofminimality, because I: f';(ll) is convex increasing in 11. The function R(m) is concave, and represents the maximal average reward for any min [0, n]. D Define now the index v1(x1) of project iwhen in state x 1 as the value ofv which makes 4t fi = 11 + L 12 fi in (17). In other words, it is the value of subsidy which makes the two phases equally attractive for project i in state x 1• This is an obvious analogue of the Gittins index, to which it indeed reduces when passive projects are static and yield no reward. The interest is that the index (which Gittins characterised as a fair 'retirement income') is now seen as the Lagrange multiplier associated with a constraint on average activity. The index is obviously meaningful: the greater the subsidy needed to induce one to rest a project, the more rewarding must it be to operate that project. Suppose now that we wish to enforce the constraint m(t) = m rigidly. Then a plausible policy is the index policy; at all times to choose the projects to be operated as the m projects of currently greatest index (i.e. the first m on a list ranking projects by decreasing index). Let w denote the average return from this policy by Rind(m).

280

OPTIMAL ALLOCATION; THE MULTI-ARMED BANDIT

Theorem 14.5.2 Rmd(m)

~

Ropt(m)

~

R(m).

(19)

Proof The frrst inequality holds because Rapt is by definition the optimal average return under the constraint m(t) = m. The second holds because R(m) is the optimal average return under the relaxed version of this constraint, E~~=m

0

The question now is: how close are the inequalities (19), i.e. how close is the index policy to optimality? Suppose we reduce rewards to a per project basis in that we divide through by n. The relation

Rind(m)/n

~

Ropt(m)Jn ~ R(m)/n

(20)

then expresses inequalities between rewards (under various policies) averaged over both time and projects. One might conjecture that, if we let m and n tend to infinity in constant ratio and hold the population of projects to some fixed composition, then all the quotients in (20) will have limits and equality will hold throughout in this limit. This conjecture has in fact been essentially verified in a very ingenious analysis by Weber and Weiss (1990). However, there are a couple of interesting reservations. Let us say that a project is indexable if the set of values of state for which the project is rested increases from the empty set to the set of all states as v increases. This implies that, if the project is rested for a given value of subsidy, then it is rested for all greater values. It also implies that, if all projects are indexable, then the projects i which are active under a a v-subsidy policy are just those for which

vi(x;) > v. One might think that indexability would hold as a matter of course. It does so in the classic MAB case, but not in this. Counter-examples can be found, although they seem to constitute a small proportion of the set of all examples. An example given in Whittle (1988) shows how non-indexability can come about. Let D(v) be the set of states for which a given project is rested under the v-subsidy policy. say) enters D as v increases. It can be that paths Suppose that a given state (x starting from with in D show long excursions from D before they return. This implies a surrender of subsidy which can become non-optimal once v increases through some higher value, when will leave D. Another point is that asymptotic equality in the second inequality of (20) can fail unless a certain stability condition is satisfied (explicitly, unless the solution ofthe deterministic version of the equations governing the distribution of index values in the population under the index policy converges to a unique equilibrium~ However, the statistics of the matter are interesting. In an investigation of over 20000 randomly generated test problems Weber and Weiss found that about 90% were indexable, but found no counterexamples to average-optimality (i.e. of

e e

=e.

e

6 AN EXAMPLE: MACHINE MAINTENANCE

281

instability of the dynamic equations). In searching a more specific set of examples they found counterexamples in fewer than one case in IQ- 3, and for these the margin ofsuboptimality was of the order of one part in 1o- 5. The assertion that the index policy is average-optimal is then virtually true for all indexable cases; it is remarkable that an assertion can escape absolute validity by so little. The result gives some indication that asymptotic optimality can be achieved in large systems by quite simple policies.

6 AN EXAMPLE: MACHINE MAINTENANCE The machine maintenance problem considered in Section 11.4 constitutes a good first example. This is phrased in terms of costs rather then rewards, so that it is now natural to think of vas a cost for action rather than a subsidy for inaction; i.e. to identify it as the cost of a machine overhaul. In the notation of Section 11.4, the dynamic programming equation for a single machine analogous to equation (17) is then )' = min{v +ex+ /(0) - f(x), ex+ >.[f(x + 1) - /(x)].

(21)

If we normalise /(0) to zero and conjecture that the optimal policy is to service the machine if X ;;::: for some critical value then (21) implies the equations

e

e

'Y + f(x) = v + ex

)'=ex+ >.[!(x + 1)- /(x)] These have solution

f(x) = ex+ v- )' f(x) =

I:hj=O

ex)/>.= )'X- ex(x- 1)

(o ~ x ~e).

2).

).

e

The identity of the two solutions at X= thus provides the determining equation for )' in terms of

e

(22) Here we have neglected discreteness effect by replacing ~(e- 1) bye_ Now ~ is determined by optimality; i.e. by the requirement that 'Y should be minimal with respect to This is equivalent to requiring that the derivatives with respect to~ of the two sides of relation (22) should be equal (see Exercise 10.7.21 i.e. to the condition >.e,...., 'Y- ee. Substituting this evaluation of 'Y into (22) we deduce a relation between v and~ equivalent to v "' v(e). In this way we find the evaluation

e.

ex?-

v(x) "'e(x + >.) + 2I,

282

OPTIMAL ALLOCATION; THE MULTI-ARMED BANDIT

accurate to within a discreteness effect The last (and dominant) term in this expression is essentially the index which was deduced by policy improvement in Section 11.4.

7 QUEUEING PROBLEMS Despite the fact that the technique of policy improvement produced plausible index policies for a number of queueing problems in Sections 11.5-11.7, the restless bandit technique fails to do so. Consider, for example, the costing of service, in the hope of deducing an index for the allocation of service between queues. With the cost structure of Section 11.5 the dynamic programming equation corresponding to (17) would be 'Y = ex+ min{ >.~(x + 1), 11 + >.~(x + 1) - J.t(x)~(x)}

Here

11

is the postulated cost of employing a server,

~(x)

is the increment

f(x)- f(x- 1) in transient cost and J.t(x) equals J.t or zero according as x is positive or zero. One must assume that J.t > >. if 'Y is to be finite. One finds (and we leave this as an exercise to the reader) that for any non-negative 11 the active set-the set in which it is optimal to employ a server-is the whole set x > 0 of positive x. One thus does not have a decision boundary which changes with changing v; the very feature on which the definition of the index was based in Section 5. One can see how this comes about: there is no point in considering policies for which, for example, there is no service in a state X = for some ~ > 0. Such policies would present the same optimisation problem as ever for the queue on the Set of States X~ ~'but With the feature that one is accepting a base-load of and so incurring an unnecessary constant cost of c~. One might argue that variation of J.t allows varying degrees of engagement, so that one might allow J.' to vary with state with a corresponding proportional s_ervice cost However, one reaches essentially the same conclusion in the undiscounted case: that an optimal policy serves all states x > 0 at a common rate. The special character of such queueing problems becomes manifest if one considers the large-system limit n --+ oo envisaged in Section 5. It is assumed that the traffic intensity for the whole system is less than unity, so that there is sufficient service capacity to cover all queues. In that case all this capacity is directed to a queue the moment it has customers, so that all queues are either empty or (momentarily) have one customer in the course of service. This rather unrealistic immediacy of response can be avoided only if response is inhibited by switching costs or slowed down by delay in either observation or transfer. Such considerations do not hold for the pure reward problem of Section 11.7. We leave the reader to confirm that the methods of Section 6 indeed lead to the known optimal policy.

e

e

7 QUEUEING PROBLEMS

283

Notes oo the literature

The subject has a vast literature. Although Gittins had arrived at his solution by 1970 it was published first in Gittins and Jones (1974). His later book (Gittins, 1989) gives a collected exposition of the whole subject. The proof of optimality associated with solution (4) for the value function was given in Whittle (1980). Restless bandits were introduced in Whittle (1988) and their analysis completed by Weber and Weiss (1990).

CHAPTER 15

Imperfect State Observation We saw in Chapter 12 that, in the LQG case, there was a treatment of the case of imperfect observation which was both complete and explicit. By 'imperfect observation' we mean that the current value of the process variable (or of the state variable, in state-structured cases) is not completely observable. In practice, observation will seldom be complete. However, the LQG case is an exception in that it is so tractable; this tractability carries no further than to the LEQG case (see Chapter 16). There are very few models with imperfect observation which can be treated both exactly and explicitly. Let us restrict attention to the state-structured case, which one might regard as the standard normalised form. Then the central formal result which emerges if the state variable is imperfectly observed is that there is still a simply recursive dynamic programming equation. However, the argument of the value function is no longer the physical state variable, but an 'informational' state variable: the distribution of the current value of physical state conditional on current information. This argument is then, not as formerly an element of the state space PI, but a distribution on PI. This great increase in cardinality of the argument makes analysis very much more difficult. The simplification of the LQG case is that this conditional distribution is always Gaussian, and so parametrised by its current mean x1 and covariance matrix V1• The value function depends then on these arguments alone, and the validity of the certainty equivalence principle and associated ideas implies even further simplification. Certainly, there is no general analogue of the certainty equivalence principle, and it must be said that this fact adds interest as well as difficulty. In general the choice of control will also affect the information one gains, so that control actions must be chosen with two (usually conflicting) goals in mind: to control the system in the conventional sense and to gain information on aspects of the system for which it is needed. For example, to 'feel the steering' of a strange car is effective as a long-term control measure (in that one gains necessary information on the car's driving characteristics) but not as a short-term one. Similarly, it is the conflict between the considerations of profit-maximisation on the basis of the information one has and the choice of actions to improve this information base that gives the original multi-armed bandit problem its particular character. Control with this dual goal is often referred to as dual control. (A 'duality' quite distinct from the mathematical duality between control/future and estimation/

286

IMPERFECT STATE OBSERVATION

past which has been our constant theme.). An associated concept is that of adaptive control: the system may have parameters which are not merely unobservable but also changing, and procedures must be such as to track these changes as well as possible and adapt the control rule to them. A procedure which effectively estimates an unknown parameter will of course also track a changing parameter. The theory of dual and adaptive control requires a completely new set of ideas; it is subtle, technical and, while extensive, is as yet incomplete. For these reasons we shall simply not attempt any account of it, but shall merely outline the basic formalism and give a single tractable example. 1 SUFFICIENCY OF THE POSTERIOR DISTRIBUTION OF STATE Let us suppose, for simplicity of notation, that all random variables are discretevalued-the formal extension of conclusions to more general cases in then obvious. We shall use a naive notation, so that, for example P(x11 W1) denotes the probability of a value x 1 of the state at time t conditional on the information W1 available at time t. We are thus not making a notational distinction between a random variable and particular values of that random variable, just as the 'P' in the above expression denotes simply 'probability of' rather than a defined function of the bracketed arguments. We shall use a more explicit functional notation when needed. Let us consider the discrete-time case. The structural axioms of Appendix 2 are taken for granted (and so also their implication: that past controlsparametrising variables-can be unequivocally lumped in with the conditioning variables). We assume the following modified version of the state-structure hypotheses of Section 8.3. (i) Markov dynamics It is assumed that process variable x and observation y have the property

(I) (ii) Decomposable cost function It is assumed that the cost function separates into a sum of instantaneous and terminal costs, of the form /r-1

C=

L {3 c(x,, u, t) + (iCh(xh)· t=O 1

(2)

(iii) lriformation It is assumed that W1 = ( Wo, Y,, U,_t) and that the information available at time t = 0 implies a prior distribution of initial state P(xol Wo). It is thus implied in (iii) that Yr is the observation that becomes available at time t, when the value of u 1 is to be determined. Assumption (i) asserts rather more than Markov structure; it states that, for given control values U, the stochastic

I SUFFICIENCY OF THE POSTERIOR DISTRIBUTION OF STATE

287

process {x,} is autonomous in that the distribution of x 1 + 1 conditional on X 1 and Y, is in fact dependent only on X 1• In other words, the causal dependence is one-way; y depends upon x but not conversely. This is an assumption which could be weakened; see Exercise 1. We have included a discount factor in the cost function (2) for economy of treatment. Assumptions (i) and (iii) imply that both state variables and observations can be regarded as random variables, and that the stochastic treatment can be started up, in that an initial distribution P(xol Wo) is prescribed for initial state Xo (the prior distribution). The implication may be more radical than it seems. For example, for the original multi-armed bai:ldit the physical state variable is in fact the parameter vector = {Oi}; the vector of unknown success probabilities. The fact that this does not change with time, under usual assumptions, makes it no less a state variable. However, the change in viewpoint which allows one to regard this parameter as a random variable is non-trivial. Generally, the formulation above implies that unknown plant parameters are to be included in the state variable and are to be regarded as random variables, not directly observable but of known prior distribution. That is, one takes the Bayesian point of view to inference on structure. The controversy among statisticians concerning the Bayesian formulation and its interpretation has been a battle not yet consigned to history. We shall take the pragmatic point of view that, in this context, the only formulations which lead to a natural recursive mathematical analysis of the problem are the Bayesian formulation and its minimax analogue. We shall refer to the distribution

e

P, = {P,(x,)} := {P(x,j W,)}

(3)

as the posterior distribution of state. More explicitly, the posterior distribution of x 1 at time t, in that it is the distribution of x 1 conditional upon the information that has been gathered by timet. It obeys a natural forward recursion.

Theorem 15.1.1 (Bayes updating of the posterior distribution) Under the assumptions (i)-(iii) listed above the posterior distribution P1 obeys the updating formula

(4)

Proof We have, for fixed W,+l and variable x,+"

P(xt+d Wt+l) ex P(xt+I, Yt+d W,, u,)

= LP(x,,x,+l,Yt+h jw;,u,) x,

288

IMPERFECT STATE OBSERVATION

= LP(xrl W,, u1 )P(xt+l,Yt+d W1, x 1, u1) x,

= LP,(x,)P(xr+i 1Yt+dx 1 ,u1). X

The last step follows from (3) and the implication of causality: that P(x1 1 W1, u1) :::::: P(xrl W, ). Normalising this expression for the conditional distribution of Xr+i we deduce recursion (4). 0 Just as the generic value of x 1 is often denoted simply by x, so shall we often denote the generic value of P1 simply by P. We see from (4) that the updating formula for P can be expressed

P(x)--+ P'(x) :=

I:z P(z)a,(x,y,lz,u) L:x L:z P(z)a (x,ylz, u)

(S)

1

where a1 (x, ylz, u) is the functional form of the conditional probability

P(xt+l = x, Yt+l = ylx, = z, u, = u). Recall now our definition of a sufficient variable €1 in Section 2.1. Theorem 8.3.1 could be expressed as an assertion that, under the assumptions (i)-(iii) of Section 8.3, the pair (x1, t) is sufficient, where x 1 is the dynamic state variable. What we shall now demonstrate is that, under the imperfect-observation versions of these assumptions expressed in (i)-(iii) above, it is the pair (P1 , t) which is sufficient, where P, is the posterior distribution of dynamical state x 1 • For this reason Pis sometimes referred to as an 'informational state' variable or a 'hyperstate' variable, to distinguish it from x itself, which still remains the underlying physical state variable.

Theorem 15.1.2 (The optimality equation under imperfect state observation) Under the assumptions (i)-( iii) listed above, the variable (P1 , t) is sufficient, and the optimality equation for the minimal expected discounted future cost takes the form F(P,t) =i~f [LP(x)c(x,u,t) X

+.BLLLP(z)a,(x,ylz,u)F(P',t+ X

y

1)]

(6) (t
Z

where P' is defined by (5) and the terminal condition for system (6) is F(P, h) = L P(x)Ch(x).

(7)

X

The proof follows by the now-familiar backward induction; confirmation is left to the reader.

2 MACHINE SERVICING AND CHANGE-POINT DETECTION

289

Recursion (6) may seem unattractive in view of the 'fractional linear' dependence (5) of P' on P. In fact, this reduces effectively to a linear relation. ·Recall that a function ¢(P) is homogeneous ofdegree r in P if ¢(>.P) = >.' ¢(P) for any positive scalar>.. We shall find it sometimes convenient to write F(P, t} as F(( { P(x)}, t) if we wish to indicate how P(x) transforms for a given value ofx.

Theorem 15.1.3 The value jUnction F(P, t) can be consistently extended to unnormalised distributions P by the requirement that it be homogeneous ofdegree one in P. when the dynamic programming equation (6) simplifies to the form

Proof Recursion (6) would certainly reduce to (8) if F(P, t + 1) had the homogeneity property, and F(P, t) would then share this property. But it is evident from (7) that F(P, h) has the property. 0 The conclusion can be regarded as an indication of the fact that it is only the relative values of P(x 1 W1) (for varying x 1 and fixed W1) which matter, and that the normalisation factor in (5) is then irrelevant. 1

Exercises and comments (1) An alternative and in some ways more natural formulation is to regard (x1 ,y1) as jointly constituting the physical state variable, but of which only the component y 1 is observed. The Markov assumption (i) of the text will then be weakened to

P{xt+t,Yr+IIXr, Yr. Ur) = P(xt+t.Yt+dxr.Yr.Ur) consistent with the previous assumption (i) of Section 8.3. Show that the variable (P 11 y 1 , t) is sufficient, where P 1 = {P1(x1)} = {P(x1 1 Wr)} is updated by the formula

PHI (Xr+J) oc

L Pr(xr)P(xt+t.Yt+tlxr.Yr. Ur)· x,

2 EXAMPLES: MACHINE SERVICING AND CHANGE-POINT DEfECTION One might say that the whole of optimal adaptive control theory is latent in equation (8), if one could only extract it! Even something like optimal statistical communication theory would be just a special case. However, we shall confine our ambitions to the simplest problem which is at all amenable.

290

IMPERFECT STATE OBSERVATION

Suppose that the dynamic state variable x represents the state of a machine; suppose that this takes integer values j = 0, 1, 2, . . . . Suppose that the only actions available are to let the machine run (in which case x follows a Markov chain with transition probabilities PJk) or to service it (in which case the machine is brought to state 0~ To run the machine for unit time in statej costs CJ> to service it costs d Al each stage one derives an observation y on machine state. Suppose, for simplicity, that this is discrete-valued, the probability of an observation y conditional on machine statejbeingp1(y). Let P = {P1} denote the current posterior distribution of machine state, and let 'Y and f(P) denote the average and transient cost under an optimal policy (presumed stationary). The dynamic programming equation corresponding to (8)is then

where o(P) is the unnormalised distribution which assigns the entire probability mass L;1 P1 to state 0. The hope is, of course, to determine the set of P-values for which the option of servicing is indicated. However, even equation (9) offers no obvious purchase for general solution. The trivial special case is that in which there are no observations at all. Then P is a function purely of the time which has elapsed since the last service, and the optimal policy must be to service at regular intervals. The optimal length of interval is easily determined in principle, without recourse to (9~ A case which is still special, but less trivial, is that in which the machine can be in only two states, x = 0 or I, say. We would interpret these as 'satisfactory' and 'faulty' respectively. In this case the informational state can be expressed in terms of the single number P1

rr---- Po+P1' This is the probability (conditional on current information), that the machine is faulty. Let us suppose that Pol = p = 1 - q and p 1o = 0. That is, the fault-free machine can develop a fault with probability p, but the faulty machine cannot spontaneously correct itself. If we setf(P) = ¢{rr) and assume the normalisation ¢(0) = 0 then equation (9) becomes, in this special case,

(10) Here we have assumed that c0 = 0, and have defined

2 MACHINE SERVICING AND CHANGE-POINT DETECTION

p(y)

= (1 -1r)qpo(Y) + (p+1rq)pi(y),

291

(11)

Formula (11) gives the updating rule 1r---+ n'. The optimal decision will presumably be to service when 1r exceeds some threshold value, this value being in principle determinable from {10). This two-state model can also represent the problem of change-point detection mentioned in Section 8.1. Suppose that state 0 is that in which a potential pollution source (say, a factory or a reactor) is inactive, and state 1 that in which it is active. One must decide whether to 'service' (i.e. to give the alarm and take antipollution measures) or not on the basis of observations y. Giving the alarm costs d; delaying the alarm when it should be given costs c per unit time. Two particular cases of this model have been analysed in the literature: those in which the observations y are Poisson or normal variables respectively, with parameters dependent upon pollution state j. The problem of determining the critical threshold value of 1r from (10) under these assumptions is soluble, and is referred to as the Poisson or Gaussian disorder problem, respectively.

BEYOND PART 3

Risk-Sensitive and H 00 Criteria

CHAPTER 16

Risk-sensitivity: The LEQG Model 1 UTILITY AND RISK-SENSITIVITY Control optimisation has been posed in terms of cost, whereas economists work largely in terms of reward, i.e. negative cost. Since we shall be invoking some economic concepts, let us conduct the discussion in terms of reward for the moment, before reverting to the cost formulation. Suppose that R is the net monetary reward from some enterprise. One then wishes to conduct the enterprise (i.e. choose a policy) so as to maximise IR. However, if IR is a random variable, then it does not necessarily follow that one will wish to maximise £.11 (R) with respect to policy 1r; one might choose rather to maximise E"(U(R)], where U is some non-linear function, usually monotone increasing. For example, if U had the concave form of Figure l then this would be an indication that a given increment in reward would be of less benefit if IR were already large than if it were small. The function U is termed a utility function, and is virtually defined by the fact that, on axiomatic or behavioural grounds, one has decided that E,.[U(IR)] is the quantity one wishes to maximise. The necessity for choice of a utility function arises because one is averaging over an uncertain outcome, but it would also arise if one wished to characterise the benefit derived from a reward which was distributed over time or over many enterprises. In cost terms, one could generalise the minimisation of expected cost E,. (C) to the minimisation of the criterion E,.[L(C)]. Here Lis a disutility function, again

u(R)

Figure 1 The graph ofa utility function U (IR) of the concave increasing form usually considered. The concavity expresses a decreasing marginal rate ofreturn of utility with reward, which induces a ri.sk-averseness on the part ofthe optimiser.

296

RISK-SENSITIVITY: THE LEQC MODEL

Figure 2 A convex disutility fimction, implying an effective risk-averse attitude or pessimism

presumably monotone increasing. One gains a feeling for the implications of such a generalisation if one considers the two cases of Figures 2 and 3. In the case of Figure 2 L is supposed convex, so that every successive increment in cost is regarded ever more seriously. In this case Jensen's inequality implies that E[L(C)] ~ L[E(C)] so that, for a given value of E(C), a certain outcome is preferred to an uncertain outcome. That is, an optimiser with a convex disutility function is risk-averse, in that he dislikes uncertainty. The concave disutility function of Figure 3 corresponds to the opposite attitude. In this case L is supposed concave, so that successive increments in cost are regarded ever less seriously. Jensen's inequality is then reversed, with the implication that an optimiser with a concave disutility function is risk-seeking, in that he positively welcomes uncertainty. In the transition case, when Lis linear, the optimiser is risk-neutral in that he is concerned only by the expectation of cost and not by its variability. All other cases correspond to a degree of risk-sensitivity on his part. One can interpret risk-seeking and risk-averse attitudes on the part of the optimiser as manifestations of optimism or pessimism respectively, in that they

Figure 3 A concave disutillty function, implying an effective risk-seeking attitude or optimism.

1 UTILITY AND RISK-SENSITIVITY

2fJ7

imply his belief that uncertainties tend to his advantage or disadvantage respectively. This conclusion will emerge rather more explicitly in the next section. The attitude to risk is revealed also if one converts the criterion back on to a cost scale by defining

(1) Here L -l is the function inverse to L, which certainly exists if L is strictly monotonic. If L is monotone increasing then minimisation of x,r is of course equivalent to minimisation of the expected disutility. Suppose now that under policy 71' the cost C has expectation m and a small variance ll Expansion of L(C) in powers of C - m then leads to the conclusion that E,.[L(C)] = L(m) + iL"(m)v + o(v) (under regularity conditions on differentials and moments). This in turn implies that

L"(m) v

x,. = m + L'(m) :2 + o(v).

(2)

That is, variability is weighted positively or negatively in the criterion according as the disutility function is convex or concave. This argument is less convincing than that based on Jensen's inequality, in that it makes unnecessary assumptions. However, it is illuminating in other respects; see Exercise 1. There are now good reasons for paying particular attention to the exponential disutility function e-SC, where ()is a parameter, the risk-sensitivity parameter. This function should be minimised for () negative (when it is monotone increasing) and maximised for fJ positive (when it is monotone decreasing~ However, if we normalise back on to a cost scale as in (1), then we find that all cases are covered by the assertion that the normalised criterion

(3) should be minimised with respect to 71'. The criterion reduces in the case () = 0 to the classic risk-neutral criterion E,..(C); it corresponds to increasingly riskseeking or risk-averse attitudes as() increases through positive values or decreases through negative values respectively. Note that relation (2) now becomes

x,.(fJ)

= m- fJv/2 + o(v).

(4)

The exponential disutility function is attractive for two reasons. (i) The parameter () places the optimiser naturally on a scale of optimism-pessimism. (ii) In this case, and essentially in this case alone, the coefficient of v in (2) is independent of m (see Exercise 1). That is, in this case there is an approximate decoupling of the aspects of expectation and variability of cost. However, the more compelling reasons are what one might term mathematical/pragmatic in character; such reasons have a way of turning out to be fundamental. (iii) Under LQG assumptions the exponential criterion leads to a

298

RISK-SENSITIVITY: THE LEQC MODEL

complete and attractive generalisation of LQG theory. Essentially, the expectation in (3) is then the integral of something resembling a Gaussian density. (iv) If LQG assumptions are replaced by what one might term 'largescale' assumptions and if an exponential criterion is adopted then large-deviation theory becomes immediately applicable. It is striking that the economic concept of risk-sensitivity, interesting in itself, should mesh so naturally with the mathematics. We shall explore the LQG generalisation in the remainder of this chapter. Large-deviation concepts open a complex of ideas to which the final part of the book is devoted. Exercises and comments

(1) Show that L"(m)/ L'(m) is independent of m if and only if L(m) is a linear function of an exponential of m. A utility function is of course not significantly changed by a linear transformation, because such transformations commute with the operation of expectation. (2) Note an implication of Exercise 8.1.3: that relation (4) is exact (i.e. there is no remainder term) if L(m) is an exponential function of a normal variable. (3) A classic moment inequality asserts that, if x is a non-negative scalar random variable, then (Ex) 1/' is non-decreasing in r. From this it follows that x. . (9) is non-increasing in 9. 2 THE RISK-SENSITIVE CERTAINTY-EQUIVALENCE PRINCIPLE The combination of the LQG hypotheses of Section 12.2 and the exponential criterion (3) leads us to the LEQG model already discussed in Section 12.3, and to which this chapter is devoted. In fact, we found in Section 12.3 that the most economical way of proving the certainty-equivalence principle (CEP) in the LQG case was to do so first for the LEQG model in the risk-seeking or 'optimistic' case 9 > 0. Let us slightly rephrase the conclusions summarised there in Lemma 12.3.2 and Theorem 12.3.3. Define the stress

(5) the linear combination of cost and discrepancy which occurs naturally in the evaluation of expectation (3). Let us also define the modified total-cost value function G( W1) as in Section 12.3 by e-IJG(W,) = f( Y1)extE,..[e-8Cj W,J. (6)

...

where the extremisation is a maximisation or a minimisation according as (} is positive or negative. Then the conclusions of Lemma 123.2 and Theorem 12.3.3 can be rephrased as follows.

2 THE RISK-SENSITIVE CERTAINTY-EQUIVALENCE PRINCIPLE

299

Theorem 16.2.1 (The risk-sensitive certainty equivalence theorem for the riskseeking (optimistic) case) Assume LEQG structure with () > 0. Then the total value function has the expression G(W,)

= g + u,.:r;.. inf inf inf§ Jr:r>t X 1 = g, + inf{0D(X,, Y,j; Ur-I) x, 1

1

(7)

where g1 is a policy-independentfunction oft alone. The value ofu1 thus determined is the LEQG-optimal value of the control at time t. If the value of u1 minimising the square bracket is denoted u(X1, U,_I) then the LEQG-optimal value of u1 is u( X,(t), Ur-I) where X,(t) is the value of X 1 determined by the]mal X1-minimisation. The first equality of (7) asserts that the value function at time t is obtained by minimising the stress with respect to all process/observation variables currently unobservable and all decisions not already formed, and that the value of u1 determined in this way is optimal. This may not immediately suggest the expression of a CEP, but it is a powerful assertion which achieves what one might term conversion to .free form. By this we mean that an optimisation with respect offunctions u1(W1 ) has been replaced by an unconstrained minimisation with respect to unobservables and undetermined controls. That is, the constraint that the optimal u1 should depend only upon W1 has been achieved automatically in a free extremisation. This process of effectively relaxing constraints will be taken to its conclusion in Chapters 19-21, when we reach the full time-integral formulation. The final assertion of the theorem looks much more like a CEP, in that it asserts that the optimal value of uf is just the optimal value for known X1 with X 1 replaced by an estimate X,Ct . The risk-neutral CEP is often confused with the separation principle, which asserts (in terms not well agreed) the separateness of the optimisations of estimation and controL There is certainly no such separation in the LEQG case. Control costs (both past and future) affect the value of the estimates and noise statistics affect the control rule (even if the process variable is perfectly observed) However, we see how a separation principle should now be expressed. If X 1 is provisionally assumed known then the evaluation of the two terms inside the curly brackets in the final expression of (7), which can be regarded as concerned with estimation and control respectively, can proceed separately. The two evaluations are then coupled by the final minimisation with respect to X1, which also yields the final estimate x,. The effectiveness of this separation is much clearer in the state-structured case considered below. The CEP in the risk-averse case, (J < 0, differs interestingly, and requires more careful statement. The distinction arises from the fact that relation (12.16) now becomes

300

RISK-SENSITIVITY: THE LEQC MODEL

G(W,) =sup infG(Wr+1) + · · · u,

(8)

Yr+l

and that the order of the two extremal operations cannot in general be reversed. The consequence is that the analogue of Theorem 16.2.1 is Theorem 16.2.2 (The risk-sensitive certainty equivalence theorem for the riskaverse (pessimistic) case) Assume LEQG structure with () < 0. Then the total value function has the expression

G(W,) = g1 + inf sup ... inf sup§ u,

YtT I

llb-l

1 = g, + stat{0- 1D(X,, X,

+stat Jlr:T

~ l

Yb

Y,l; Ur-1)

stat [C(X, U) +

x.,.:r>t

(9)

o- 11D(xr+1, ... ,xhiX,; U)]}

where g, is a policy-independent function oft alone. The value of u, thus determined is the LEQG-optimal value of the control at time t. If the value of u1 extremising the square bracket is denoted u(X,, Ur-1) then the LEQG-optima/ value of u, is u( x,
Here Yh can be regarded as yielding complete information, and so can be identified with X, and the operator 'stat' simply renders the expression to which it is applied stationary with respect to the variable indicated. The first relation in (9) follows as in the risk-seeking case, but the order of the extremisations must now be observed. However, all that is necessary in applying recursions such as (8) is that quantities being minimised (maximised) should be convex (concave) in the relevant argument. The cases in which this requirement fails prove interesting; see Sections 3 and 4. The extremisation conditions in the first expression of (9) will yield linear relations which can be solved (i.e. variables eliminated) in any order. The rearrangement of extremal operations in the final expression of (9) corresponds to just such a reordering, but with the characterisations of maximality or minimality now weakened to stationarity of some kind. Although state-structure plays no part in this formalism (and it is for this reason plus economy that we have dragged the reader through the general case) the situation does indeed become more transparent if state structure is assumed. In such a case cost and discrepancy will have the additive decompositions h-1

c=L

c(x,, u,, t)

1=0

+ Ch(xh) =

h-1 L c,

+ ch

h

ID

(10)

1=0

= D(x,) + LD(x,,y,lxr-1.y,_ 1;u,_J) =Do+ 1=1

h

z::::n,

(11)

1=1

say. Here we have not discounted explicitly, and D(x0 ) is the discrepancy derived from the prior distribution of Xo conditional on W0 .

301

2 THE RISK-SENSITIVE CERTAINTY-EQUIVALENCE PRINCIPLE

Define now thefoture stress F1 = F(x1, t) and the past stress P, the values of h-i

= P(x1, W,) as

t-1

l_)c.- + e- 1D.-+ I)+ Ch

and

e- 1Do+ L(c.-+B- 1D.-+J) r=O

r=t

with all variables except W, and x, extremised out. That is, all controls from timet onwards are minimised out and all unobservables at time t except x 1 itself are minimised or maximised out, according as Bis positive or negative.

Theorem 16.2.3 (Recursions and certainty-equivalence for the state-structured Assume a state-structured LEQG model. Then the future and past stresses are determined by the recursions and terminal conditions

case)

F(x, t) =inf ext [c(x, u11 t) Ut

Xt+i

+F(x,+i,t+l)]

+ e- 1D(x,+Jix1; u,)

(12)

(O~t
(13)

F(xh) = Ch(xh)

+ P(Xt-io W,_i)J P(xo, Wo)

(0 < t

(14) ~h)

= e- 1D(Xo)

(15)

where ext indicates a minimisation or a maximisation according as 0 is positive or negative.lftheminimisingva/ueofu, in (!.2) is denoted u(x, t) then the optimal value of u1 isu(x11 t), where x1 isthevalueofx1extremising P1 + F, = P(x, W,) + P(x,, W1 ). All the assertions are immediate consequences of the previous two theorems and the definitions. Results now look much more familiar; the last assertion is plainly certainty-equivalent in character and in equations (12) and (14) we recognise respectively analogues of the dynamic programming equation and the updating equation for the conditional distribution P(x 11W1). Because of the quadratic nature of cost and discrepancy these equations will reduce to Riccati and Kalman relations of the type familiar from Sections 24 and 12.5. We see how the separation principle manifests itself. Provisional postulation of the value x, of current state decouples the evaluations of past and future stress, so that recursions (U) and (14), concerned with control/future and estimation/past can be solved separately for F, and P1 as functions of observables W, and the (generally unobservable) x 1• The calculations are then recoupled by the 1 minimisation of P1 + F, with respect to x 1, so yielding the fmal estimate = x~ ) and the certainty-equivalent optimal control u(x, t). The pattern which emerges is then close to that of the LQG case with, however, a mixing of cost and stochastic effects. The discrepancy term in (12) enforces a

x,

302

RISK-SENSITIVITY: THE LEQC MODEL

stochastic plant equation but also induces a dependence of the optimal control upon the covariance matrix N of plant noise, even if state observation is perfect The cost'term in (14) has an effect upon estimates, while the fmal estimate 1 shows a dependence on future costs as well as past costs. The future stress F(x 11 t) indeed differs from the normalised value function for the case of perfect state observation only by a term dependent on t alone. This term represents the cost due to plant noise, and will be evaluated in Section 10. Finally, a word on notation.. We shall term 1 the minimal stress estimate of x 1, even if indeed the extremisation of stress is more complicated than a simple minimisation. It is identical with what we have before written x~t) and shall do again: the best estimate of x 1 based upon information at time t. We shall use 1 to denote the estimate of x 1 based upon information at time t which extremises past stress. In the risk-neutral case the two estimates coincided, because future costs had no effect upon state estimates. However, we have now to make a distinction.

x

x

x

3 SOME SIMPLE EXAMPLES It is helpful to gain a feeling for the effects of risk-sensitivity by examining a couple of examples which are simple in that they are concerned purely with estimation. Consider the uncontrolled Markov model x 1 = Axr-l + f. 1• We suppose that x is scalar and that the plant noise c:: is white and Gaussian with variance N. The cost function is C = L::O li, and it is supposed that the total information available at time 0 is knowledge of xo. The stress function is consequently

!

CXl

§ =

!L[Rx; + (9N)- (x,- Ax -t)2]. 1

1

t=O

At time 0 this is to be extremised with respect to the current unobservables, which are just Xt, x 2, x3, . . . . We thus have a risk-sensitive formulation of a pure prediction problem from time t = 0. In the risk-neutral case the predictor is 0 ) evidently just the projection estimate: tl(x 1/Xo) = A 1xo. The values extremising stress satisfY

xl

A(x(O) (1 + A 2 + 9NR)x(O)t t-l

+ x(O) )= x+l

0

(t

;?;

0).

Assume, for simplicity, that 0
(16) If 9 is positive then 0 < o < A, and the forecast is optimistic in that it tends to zero faster (and so incurs a smaller cost) than does the risk-neutral predictor. Better expressed: the forecaster is optimistic, in that he is effectively assuming that the noise E will be such as to deflect the path of the process in a favourable direction.

3 SOME SIMPLE EXAMPLES

303

If 0 is negative then a > A and the forecast is pessimistic. One can regard the prediction as having broken down altogether if 0 is so negative that equation (16) has no root smaller than unity, when the extremal stress is infinite. This occurs when n

17

~ 17ii ·= -(I - A)2

"'

NR

•

At the critical point iJ the stress function, which is now being maximised with respect to unobservables, is no longer concave in its arguments, and has extremal value +oo. One can regard this as the point at which the forecaster has become so pessimistic that he has essentially reached a breakdown point: his apprehensions ofloss outweigh the reassurance of statistics. For an even simpler example, consider a risk-sensitive version of the one-stage estimation problem of Section 12.6. Suppose that x and y are jointly normal random vectors with zero mean and covariance matrix

l

cov[; = [

~;: ~;;

rl

and that there is a cost function! xTRx. The variable y has been observed and xis to be estimated. In a risk-sensitive formulation one derives the estimate x by minimising 20 §

= OxT RX + [X] T[lxx y

lyx

[X]

lxy] lyy y

with respect tox, thus deriving

x=

-(lxx + OR)- lxyY· 1

This estimate of course equals the projection estimate if 0 is zero, and approaches the minimal-cost value 0 as 0 increases. The matrix l:x:x is the information matrix of x conditional on y; the effect of risk-sensitivity is to modify this to lxx + OR. That is, one 'gains' information if 0 is positive (in that one is happy to let costpressures supply the 'information' that x is nearer to its cost-minimising value 0 than the data would have suggested). One loses 'information' if 0 is negative (in that fear of costs makes one doubt the data). When 0 becomes so negative that lxx + OR is singular then estimation fails altogether: the optimiser has become pessimistic to the point of neurotic collapse. The critical value iJ at which this occurs is the greatest root of llxx + OR! = 0. In both examples optimisation failed only for 0 sufficiently large in a negative direction. This was because the cost function C was bounded below by zero in both cases, and stress-extremisation broke down only when it yielded an infinite positive cost This occurred when the optimiser was sufficiently pessimistic, and one might term the breakdown that of neurosis. If C can take infinite negative

304

RISK-SENSITIVITY: THE LEQC MODEL

values (i.e. if an infinite reward is possible) then breakdown can also occur if 9 is sufficiently large and positive. That is, stress-extremisation can yield an infinite negative cost (i.e. an infinite positive reward) if the optimiser is sufficiently optimistic; one might term this breakdown that of euphoria. A simple example of control is supplied by consideration of the one-stage problem, whose solution presages something of the character of the multi-stage problem. Suppose that the cost function is C uT Qu +!(xi -X:) TR(xi -X:), where Xi =Ax+ Bu + e, so that x and u are just the values of x 0 and uo. The variables u and e are then to be chosen to extremise the stress C + !eT(oNrie. We obtain results in the most convenient form if we write this as

=!

§

= inf![uT Qu + fT (9Nrie- ).T R-i). + 2).T(Ax + Bu + f- xc)]. A

Extremising first with respect to u and e and then with respect to ). we find that the optimal u and the extremal § are given by

where P = R-i + BQ-iBT + 9N = R-i + J(9), say. The effect of increasing optimism (i.e. increasing 9) is indeed to make P larger [m the positive-definite sense) and to make both the optimal control and corresponding stress value smaller. These extremisations (and our implicit commutation of them) are legitimate as long as P is positive definite, i.e. as long as 9 > 9, where 0 is the largest root of IR-i + J(9)1 = 0. Equivalently, the critical value 0 is the largest root of II+ RJ(O)I = 0, an evaluation which is correct even if R is singular.The value 0 thus determined is indeed the point of 'neurotic' breakdown. It is always non-positive if Qand Rare non-negative defmite. Note that u and e both behave as controls in this example, in that they both appear linearly in the expression for Xi and quadratically in the stress function. However, whereas u is always chosen to minimise stress, e is chosen to to minimise it or maximise it according as 9 is positive or negative. That is, this au"xiliary 'control' is chosen to help or frustrate the optimiser according as he is risk-seeking or risk-averse. We shall expand upon this point in the next section. If we modify R to-R in the cost function then the term! (xi - X:) T R(xi -X:) is to be interpreted rather as a reward, which we wish to be large. The calculations above are still valid as long as P (modified by the change of sign of R) is negative definite, i.e. as long as 9 does not exceed the smallest root of I/- RJ(9)1 = 0. This upper bound represents the point of 'euphoric' breakdown. It can be of either sign. 4 THE CASE OF PERFECT STATE OBSERVATION Most of the remainder ofthe chapter is devoted to the case of principal interest: the state-structured regulation model in the standard time-homogeneous undiscounted formulation of equations (12.1)-(12.4). The exception is Section 12,

4 THE CASE OF PERFECT STATE OBSERVATION

305

where we find ourselves forced to take a fundamental look at the question of discounting. Let us first of all assume perfect state observation. The problem reduces then to the solution of the equation (12) for the future stress; essentially the dynamic programming equation. This can be written

F(x11 t) = inf ext [c(xr, ur) + !0- 1(e:T N- 1e:)t+ 1 + F(xr+b t + 1)] u,

{17)

.Xt+l

where 'ext' denotes a minimisation or a maximisation according as 0 is positive or negative, and ft+l and Xr+l are related by the plant equation (12.1). In virtue of this last fact we can rewrite (12) as

F(x 11 t) = infext[c(xr, ur) + !0- 1(e:TN- 1e:),+ 1 + F(Axr + Bu1 + ft+l• t + 1)] u,

ft+l

{18) But in this form we see that e: can be seen as a subsidiary control variable, entering the plant equation linearly just as u does and carrying a quadratic cost just as u does. It is chosen to help the optimiser if 0 is positive and to oppose him if 9 is negative. One might regard u and e: as the controls which are wielded by the optimiser and by Nature respectively. The optimiser effectively assumes that Nature is working with him or against him according as he is risk-seeking or riskaverse-a fair characterisation of optimism or pessimism respectively. Note that Nature makes its move first, at each stage. Ofcourse, e: does not appear as a control in the actual plant equation, but only in the predicted course of the optimally controlled process. In the actual plant equation it is simply random process noise, as ever. It appears as a 'control' for the predicted process because this prediction is generated by a stress extremisation; the extremisation which determines the optimal value of current control. The LQ character ()f recursion (18) implies a solution on the familiar Riccati lines, which can indeed be expressed in terms of the solution of the risk neutral case.

· Theorem 16.4.1 The solution ofequation Q8) for the future stress has the quadratic form F(xr, t) =! (xTITx) 1 (19)

if it has this form for t = h, and the optimal control then has the linear form (20) Here II, is determined by either of the alternative forms (2.25) or (6.31) of the riskneutral Riccati equation and K1 by either of the corresponding alternative equations (2.27) or ~32) if ITt+ 1 is replaced in the right-hand side ofthese equations by

{21)

306

RISK-SENSITIVITY: THE LEQC MODEL

Validity of these conclusions is subject to the proviso that the matrix J(B) + ll;+'1 should be positive defznite for all relevant t, where J( B) is the augmented controlpower matrix J(B) = BQ- 1BT + BN. (22) Proof This is inductive, as ever. Suppose that relation holds at time t + 1. We leave the reader to verify the identity T

I

T

T-

ext[(€ (BN)- f +(a+ f) II( a+ E)]= a ITa

•

(23)

If we perform the f-extremisation in (23) we thus obtain · F(x1 , t) = mf[c(x, u,)

u,

+ 2I (Ax,+ Bu,) T-ITt+ I (Ax 1 + Bu1)].

But this is just the inductive relation which held in the risk-neutral case with the substitution ofilr+l for llr+l· The question ofvalidity is covered in the following discussion. 0 If we consider the solutions of the risk-neutral case in the alternative forms (6.32) then we see that the only effect of risk-sensitivity is simply to replace the control-power matrix J = BQ- 1BT by the augmented form (22~ so that, for example, the Riccati equation becomes (6.31~

(24) if S has been normalised to zero. That is, the control-power matrix is augmented by a multiple of the noise-power matrix. This illustrates again the role of noise as an effective auxiliary control, working with or against the intended control according as ()is positive or negative. H the final maximisation with respect to >. of the second set of displayed equations in Section 6.4 is now to be valid then we require the final condition of the. theorem. This sets a lower bound on 8: see Exercise 1. The optimal (i.e. stress-extremising) values of u1 and ft+I are matrix multiples of x,; if they are given these values then the quantities Ax1 + Bu1 and Ax,+ Bu, + ft+l can be written r,x, and f,x,. We can regard r, =A+ BK1 as the actual gain matrix for the optimally controlled process and f 1 as what one might call the predictive gain matrix: the gain matrix that would hold if fr+l really did take the role of an auxiliary control variable and take the value predicted for it by stress extremisation. By appealing to the alternative derivations of Section 6.4 one fmds the evaluations

r, =A- J(J(B) +II~\ )- 1A,

r,- =A- J(B)(J(B) + TI,+-11) -1 A

(25)

if S has been normalised to zero. In other cases A should be replaced by A - sT Q- 1B. If infmite horizon limits exist then it is f which is necessarily a

5 THE DISTURBED (NON-HOMOGENEOUS) CASE

307

stability matrix; r may not be if the optimiser is excessively optimistic. Note the relation

Exercises and comments (1) In the scalar case the infinite-horizon form of the Riccati equation (24) becomes

A2 II II = R + 1 + 1(9)II where 1(9) is given by (22). Assume that R > 0. Show that the equation has a finite non-negative solution iff 1(9) > 0 in the case IAI ~ 1 and 1(9) > -(1 -IAI 2)/ R in the case lA I < 1. That is, the critical lower bound ii is - B 2f N Q or - B 2 / N Q( 1 - IAI 2 )/ NR according as the uncontrolled plant is unstable or stable. 5 THE DISTURBED (NON-HOMOGENEOUS) CASE If the plant equation (12.1) is modified by the addition of a deterministic disturbance d, : x 1 =Ax,_ I + Bu,_l + d, + e1 then we shall expect the future stress to have the non-homogeneous quadratic form (26) F,(x) = ixTII,x- ui x + · · · where + · · · indicates terms independent of x. We can generalise relation (23) to obtain extE[(eT(9Nf 1e +(a+ e)TII(a +e)- 2uT(a +e)]= aTfla- 25-Ta + · · · where + · · · indicates terms independent of a, the matrix fi is again given by (21), and

From this we deduce

Theorem /6.5./ If the plant equation includes a deterministic disturbance d then the future stress has the form (26) with II, obeying the modified Riccati equation indicated in Theorem 16.4.1 and u 1 obeying the backward recursion (27) Here i' 1 is the predictive gain matrix defined in (25).

308

RISK-SENSITIVITY: THE LEQC MODEL

Verification is immediate. We see that recursion (27) differs from the corresponding risk-neutral recursion of Section 2.9 only by the substitution of of f' 1 for the risk-neutral evaluation of r ,. This is indeed as it must be, since we have replaced optimisation with respect to a single control u by optimisation with respect to the pair (u, ~). The same argument leads to the evaluation of the optimal control as

u, = K,x, + (Q + BTftt+1B)- 1BT(I + Ollr+IN)- 1(O'r+l- II,+ldr+J).

(28)

The combination of (28) and the recursion (Z7) gives one the explicit feedbackfeedforward formula for the optimal control analogous to (2.65). As for the risk-neutral case, all these results emerge much more rapidly and cleanly (at least in the stationary case) when we adopt the factorisation techniques of Sections 6.3; see Section 11 and Chapter 21. 6 IMPERFECT STATE OBSERVATION: THE SOLUTION OF THE P-RECURSION In the case of imperfect observation one has also to solve the forward recursion (14) for the function P(x,, W,). Just as the solution of the F-recursion implies solution of the control-optimisation problem in the case of perfect observation, so solution of the P-recursion largely implies solution of the estimation problem. 'Largely', because the P-recursion produces only the state estimate xpased upon past stress.. However, the full minimal-stress estimate is then quickly derived, as we shall see in the next section. For the standard regulation problem as formulated in equations (12.1)-(12.4) we can write the P-recursion (14) as

x,

OP(x,, W,) = min[Ocr-1 +D, +8P(x1-l, Wr-1)]

(29)

Xr-1

where

and t:, 1J are to be expressed in terms of x, y, u by the plant and observation relations (12.1) and (12.2} Now, if we take the risk-neutral limit 0 -+ 0 then ()p will have a limit, D, say, which satisfies the risk-neutral form of (29)

D(x,, W,) = min[D, +D(xr-~, Wr-1)]

(30)

Xr-l

and has the interpretation

D(x,, W,)

= D(x,, Y,j; U,_I) = D(x,j W,) + · · · = i[(x- x?V- 1(x- x)], + · · · (31)

6 IMPERFECT STATE OBSERVATION

309

Here+··· indicates terms not involving x 1 (in fact identifiable with D( Y1j; U1_ 1)), and .X, and V1 are respectively the mean and covariance of x, conditional on W,. In this risk-neutral case xis identifiable with x and, as we saw from Section 12.5, relation (31) implies the recursive updatings of xand Y. the Kalman filter and the Riccati recursion. We can in fact utilise the known solution of (30) to determine that of (29). In the risk-sensitive case we can again establish that OP has the quadratic form exhibited in the final member of (31), so that

P(x, W1) =

~ [(x- x) v- 1(x- x)], + · ··,

(32)

where + · · · again indicates terms independent of x, irrelevant for our purposes. The quantity x1 can now be identified as the estimate of xr which extremises past stress at time t, and V1 can be said to measure the precision of this estimate in that (Ov,r' measures the curvature of past stress as x 1 varies from x,. Relation (29) now determines the updating rules for these quantities.

Theorem 16.6.1 Under the assumptions above the past stress has the quadratic form ()2) with 0 and V, identified as the prescribed mean and variance of xo conditional on W0 . The valuesofx,and V, are determined/rom thoseofx1_ 1and V1_ 1 by the Kalman filter (1 2.22)1 (12.23) and the Riccati recursion (12.24) with the modifications that x, is replaced by x,, v,_, by

x

(33)

andxt-t by (34)

in the right-hand sides ofthese relations. For validity ofthese recursions it is necessary that V1:_11 + cT M- 1C+ should be positive definite for all relevant t. Proof The quadratic character (32) of P follows again inductively from (29). Recursion (29) differs from (30) only in the addition of the term Oc,_,. If we assume that P(x 1_ 1 , W,_ 1) has the quadratic form asserted then wefmd that fJc,_ 1 + OP(xt-1, W,_ 1)

=! [(x- xlv- 1(x- .i)],_ 1 +terms independent of x,_,

whence it follows that

But this is just the recursion of the risk-neutral case with the substitution of Xt-1 and V,_, for x,_, and V,_,. 0

310

RISK-SENSITIVITY: THE LEQC MODEL

The modified recursions are again more transparent in the alternative forms

(12.59), (12.60). One sees that, as far as updating of Vis concerned, the only effect of risk- sensitivity is to modify cTM- 1c to cT M- 1c + () R. That is, to the information matrix associated with a single y-observation one adds the matrix

9R, reflecting the 'information' implied by cost-pressures (positive or negative according to the sign of 9). The passage (34) from x1_ 1 to Xr-1 indicates bow the estimate of Xr-1 changes if we add present cost to past stress. In continuous time the two forms of the modified recursions coincide and are more elegant; we set them out explicitly in Section 8. 7 IMPERFECT STATE OBSERVATION: RECOUPLING In the risk-neutral case the recipe for the coupling of estimation and control asserted by the CEP is so immediate that one scarcely gives it thought: the optimal control is obtained by simple substitution oftbe estimate 1 for x, in the perfect-observation form of the optimal rule. It is indeed too simple to suggest its risk-sensitive generalisation, which we know from Theorem 16.23. This is that the optimal control at time t is u(x 1 , t), where u(x, t) is the optimal perfectobservation (but risk-sensitive) control and 1 is the minimal-stress estimate of xr: the value of x 1 extremising P(x 1, W1) + F(x, t). It was the provisional specification of current state x 1 which allowed us to decouple the evaluations of past and future stress; the evaluations are recoupled by the determination of 1• Now that the separate evaluations {26) and (32) ofF and P have been made explicit in the last two sections, the recoupling can also be made explicit.

x

x

x

Theorem 16.7.1 The optimal control at timet is given by u1 = K1x 1 , where

.x, =(I+ 9v,rr,r 1(x, + 9V,u,).

(35)

Here IT, K1 and CT1 are the expressions determined in Theorems 16.4.1 and16.5.1 and V1, x1 those determined in Theorem 16.6.1. This follows immediately from the CEP assertion of Theorem 16.2.3 and the evaluations of the last two sections. Note that .X1 is an average of the value 1 of x 1 which extremises past stress and the value II~ 1 u1 which extremises future stress. As one approaches the risk-neutral case then the effect of past stress swamps that of future stress.

x

8 CONTINUOUS TIME The continuous-time analogue of all the LEQG conclusions follows by passage to the continuous limit, and is in general simpler than its discrete-time original. The

7 IMPERFECT STATE OBSERVATION: RECOUPLING

311

analogues of the CEP theorems of Section 1 are obvious. Note that the u and f extremisations of equation (18) are now virtually simultaneous: the optimiser and Nature are playing a differential game, with shared or opposing objectives according as 8 is positive or negative. The solutions for past and future stress in the state-structured case simplify in that the two alternative forms coalesce. The solution for the forward stress for the disturbed but otherwise time-homogeneous regulation problem is

F(x, t)

= !xTilx- qT x + · · ·

where Il and u are functions of time. The matrix Il obeys the backward Riccati equation

This reduces to

IT+ R + ATil + IlA -

ITJ(8)1I = 0

(37)

if Shas been normalised to zero, where the augmented control-power matrix J( 8) again has the evaluation

J(8)

= BQ- 1BT + 8N.

The vector u obeys the backward linear equation

&+f'Tu-Ild=O where f' is the predictive gain matrix

The optimal control in the case of perfect state observation is

u = Kx + Q-IBTu

(38)

where the time-dependence of u, K, x and u is understood, and

K= -Q- 1(S+BTII).

{39)

In the case of imperfect state observation the past stress has the solution P(x, W)

= ;8 (x- x?v- 1 (x- x) + ...

where the time-dependence of x, W, forward Riccati equation

xand Vis understood The matrix Vobeys the

312

RISK-SENSITIVITY: THE LEQC MODEL

which reduces to

if L has been normalised to zero. The updating formula for Kalman filter) is

x (the risk-sensitive

di:jdt =Ax+ Bu + d + H(y- Cx)- OV(Rx+ STu)

(40)

where H = (L + VC)M- 1 . Recoupling follows the discrete-time pattern exactly. That is, the optimal control is u = Kx + Q- 1BTu where K is given by (39) and xis the minimal-stress estimate of x:

x =(I+ OVIIf 1 (x+ OVu). 9 SOME CONTINUOUS-TIME EXAMPLES The simplest example is that of scalar regulation, the continuous-time equivalent of Exercise 4.1. Equation (37) for II can be written :

=

R + 2AIT- J(O)IT2

= f(IT),

(41)

say, where J(()) = B 2 Q- 1 + ()N, and s represents time to go. Let us suppose that R > 0. We are interested in the non-negative solutions of f(IT) = 0 which moreover constitute stable equilibrium solutions of(41), in thatf'(IT) < 0. In the case J(O) > 0 there is only one non-negative root, and it is stable; see Figure 4. In the case J(O) = 0 there is such a root if A < 0 and no non-negative root at all if A ~ 0. If J(O) < 0, then there is no non-negative root if A ~ 0, but there can be one if A is negative and -J( 0) not too large; see Figure 5. In fact, there is a root of therequiredcharacterifA < OandR-A 2 jJ(O) < O.Tosummarise:

Fipn 4 The graph offll in the CtlSe J > 0; it has a single positive zero.

9 SOME CONTINUOUS-TIME EXAMPLES

313

II Figure 5 The graph offiT in the case J < 0. A < 0; there is a positive zero if J exceeds a critical value.

Theorem 16.9.1 Assume the scalar regulation problem set out above, with S = 0 and R and Q both positive. Then both II and the magnitude of K decrease as (J increases. IfA ~ 0 (i.e. the uncontrolled plant is unstable) then the breakdown value iJ equals - B2 IN Q and II becomes infinite as() decreases to this value. If A < 0 (ie. the uncontrolled plant is stable) then the breakdown value iJ equals -B2 INQ- A 21NR. The non-negative equilibrium solution II of (41) at()= iJ is finite, but is unstable to positive perturbations. A second-order example is provided by the linearised pendulum model of Section 2.8 and Exercise 5.1.5. In a stochastic version the angle of deflection o: from the vertical obeys o = ao:+bu+€ where € is white noise of power N and the coefficient a is negative or positive according as one seeks to stabilise to the hanging or the inverted position. The cost function is! (r 1o:2 + r2 2 + Qif), with r 1 and Q strictly positive. The analysis of Section 2.8 applies with J = b'-Q- 1 +ON. It follows from that analysis that there is a finite equilibrium solution II of the Riccati equation if and only if J > 0, and that this solution is then stable. The breakdown value is thus 0 = -1 IN Q, whatever the sign of a. The greater stability of the hanging position compared with the inverted position is reflected in the relative magnitudes of II in the two cases, but the hanging position is still only neutrally stable, rather than truly stable. Finally, a second-order example which has no steady state is provided by the inertial missile example of Exercise 2.8.4. The solution for the optimal control obtained there now becomes

a

Ds(xi + xzs) u = - -==Q_+_D=-(:-':-1-+-=e~N=-=Qc:--)s"731-=3 '

314

RISK-SENSITIVITY: THE LEQC MODEL

e

where sis time to go. The critical breakdown value is = -1 IN Q - 3INDs3' and so increases with s. The longer the time remaining, the more possibility there is for mischance, according to a pessimist. 10 AVERAGE COST

The normalised value function F(x, t) defmed by

e-BF(x,t)

= extuE[e-liC'ix1 = x, Ur = u].

(42)

in the state-structured case should be distinguished from the future stress defined in Section 2, which is in fact the x-dependent part of the value function. The neglected term, independent of x, is irrelevant for determination of the optimal control, but has interest in view of its interpretation as the cost due to noise in the risk-sensitive case. Let us make the distinction by provisionally denoting the value function by Fv(x, t).

Theorem 16.10.1 Consider the LEQG regulation problem in discrete time with perfect state observation and no deterministic disturbance (ie. d = 0). Then the normalised valuefunction F, has the evaluation

F,(x, t) = F(x, t)

+ 81

(43)

where F(x, t) is the future stress, evaluated in Theorems 16.4.1 and 16.5.1, and (44) The proof follows by the usual inductive argument, applied now to the explicit form exp[-9Fv(x, t)] =

e~t(11')-" 12 iNr 1 / 2 j

exp[- !fTN- 1€ - 9c(x, u)

- 9Fv(Ax + Bu + E1 t + 1)] d€ of recursion (42). The evaluation (44) of the increment in cost due to noise indeed reduces to the corresponding risk-neutral evaluation (10.5) in the limit of zero 9. It provides the evaluation 1 (45) 1(9) = logl/ + 9NIII

29

of average error in the steady state, where II is now the equilibrium solution of the Riccati equation (24) (with A and R replaced by A- ST Q- 1Band R- ST Q- 1S if S has not been normalised to zero). More generally, it provides an evaluation of the average cost for a policy u(t) = Kx(t) which is not necessarily optimal (but stabilising, in an appropriate sense) if II is now the solution of

315

10 AVERAGE COST

IT= R

+ KTS + ST K + KT QK +(A+ BK)T(II- 1 + 9Nr 1(A + BK).

(46)

For more general models the average cost is best evaluated by the methods of Section 13.5. Recall that we there deduced the expression

7(8) =

!J

4 9

logl/ + 99tf(w)l dw

(47)

for the average cost under a policy which is linear, stationary and stabilising, but otherwise arbitrary. Here f(w) is the spectral density function of the deviation vector D. under the policy and 9t the associated penalty matrix. This expression is valid for a general linear time-invariant model; there is no assumption of state structure or of perfect process observation. On the other hand, the task remains of determining f(w) in terms of the policy and determining the class of f(w) which can be generated as this policy is varied. It is by no means evident that the evaluation (47) reduces to that determined by (45) and (46) in the state-structured regulation case with the policy (supposed stabilising) u = K.x. The reconciliation in the risk-neutral case is straightforward. That for the risk-sensitive case is less so and, as one might perhaps conjecture from the form of (47), follows by appeal to a canonical factorisation: see Exercise 20.2.1. A view of system performance which has become popular over recent years is to consider the deviation vector D. as system output, and the various disturbances to which the system may be subject (e.g. plant and observation noise) as system input. A given control rule is then characterised by the frequency response function G(iw) from input to output which it induces. One wishes to choose the rule to make G small in some norm, and there are many norms which could be chosen. So, suppose that the noise inputs could be regarded as a collective system input ( which is white with covariance (power) matrix 91, say. (A pre-normalising filter which would achieve this could be incorporated in the total filter ~ Expression (47) then becomes

7(8) =

!j

4 9

logj/ + 99tG(iw)91G(( -iw?lciw

!j

=4

9

logl/ + 99tG91GI dw (48)

and the problem then becomes one of minimising this expression with respect to G. Expression (48) is indeed a norm on G, generated by the notion of LEQG optimisation (in the stationary regime). To phrase the optimisation problem in this way helps in discussion of certain issues, as we shall see in the next chapter. However, it also glosses over an essential issue: how does one cope with the fact that the class of response functions G generated as policy is varied is quite restricted? It is a virtue of the dynamic programming approach that it provides an automatic recognition of the constraints implied by specification of the system; some equivalent insight must be provided in a direct optimisation.

316

RISK-SENSITIVITY: THE LEQC MODEL

For reasons which will emerge in the next chapter, one sometimes normalises the matrices 9t and 9l to identity matrices by regarding G as the transfer function from a normalised input to a normalised output (so absorbing 9t and 9l into the definition of G). In such a case (48) reduces to

-y(O) =

1 11" 4 8

J

-

logj/ + OGGI dw.

(49)

The criterion function (48) or (49) is sometimes termed an entropy criterion, in view of its integrated logarithmic character. However, we should see it for what it is: an average cost under LEQG assumptions. In the risk-neutral case (49) reduces simply to the mean-square norm (47r)- 1 ftr[GGJJdw, also proportional to a mean-square norm for the transient response. 11 TIME-INTEGRAL METHODS FOR THE LEQG MODEL The time-integral methods of Sections 6.3 and 6.5 are equally powerful in the risksensitive case, and equally well cut through detailed calculations to reveal the essential structure. Furthermore, the modification introduced by risk-sensitivity is most interesting. We shall consider this approach more generally in Chapters 21 and 23, and so shall for the moment just consider the perfect observation version of the standard state-structured regulation model (12.1)-(12.4). The stress has the form §

= 2:)c +!eT(eNr 1elr +terminal cost. T

This is to be extremised with respect to u and e subject to the constraint of the plant equation (12.1). If we take account of the plant equation at time T by a Lagrange multiplier >.r and extremise out e then we are left with an expression 0 = L[c(xr, Ur) + x;(xr- Axr-1 - Bur-d- !O>.JN>.r] +terminal cost (50) T

to be extrem.ised with respect tox, u and>.. This is the analogue of the Lagrangian expression (6.19) for the deterministic case which we found so useful, with stochastic effects taken care of by the quadratic term in >.. If we have reached timet then stationarity conditions will apply only over the time-interval T ~ t. The stationarity condition with respect to Er implies the relation

ON>.(I) T

= f(l) T

(T

~

t)

between the multiplier and the estimate of process noise. In the risk-neutral case fJ = 0 this latter estimate will be zero, which is indeed the message of Section 10.1. Stationarity of the time-integral (50) with respect to remaining variables at time T leads to the simple generalisation

12 WHY DISCOUNT?

sT Q -Bfl

[ _ ATff-1] -BT!/-1 -BN

317

[x] (r) =0 u .A

(T

~

t).

(51)

T

of equation (6.20) (in the special regulation case, when deterministic disturbances and command signals are set equal to zero). The matrix operator «
j [c(x, u)

+.AT (.X- a(x, u)) -

4(J.A TN.A] dT +terminal cost,

(52)

which can be treated in the same way for linear a and quadratic c. This can be regarded as a stochastic generalisation of the form (7.2) associated with the maximum principle, with the quadratic term in .A representing the stochastic effect insofar as it represents the effect of risk-sensitivity. We take up these matters in Part 5.

12 WHY DISCOUNT? Suppose we assume a model in discrete time which is time-homogeneous, and has both an infinite horizon and an infinite past. Suppose also that it is discounted, so that the cost function from time 0 is§ = 2:~;;.. 0 /3T Cn where /3 is the discount factor and the instantaneous cost C7 does not depend explicitly upon time T. The formulation then seems time-invariant and one expects the optimal policy also to be so (i.e. stationary). However, if we consider optimisation on an exponential criterion from time t ( ~ 0), then the policy 1r is chosen to extremise E,{exp[-BCJI W,} ex E,{exp[-B/31C,JI W,}

(53)

where C, = 2:~., 1 (37- 1cT is the present value of future discounted cost at timet. The proportionality factor involves costs incurred before time t, which will not affect policy from timet. We see from (53) that the optimal policy cannot possibly be time-invariant, because in shifting the starting-point from time 0 to time t we have in effect changed the risk-sensitivity parameter from B to /318, all other factors remaining statistically constant. One feels that the model is nevertheless time-invariant if correctly viewed, and wonders how the correct view can be restored. The remedy is certainly not to renormalise the cost-function to present value at each stage; this would be

318

RISK-SENSITIVITY: THE LEQC MODEL

inconsistent with the principle that an optimisation from a given time can be embedded in an optimisation from an earlier time. It seems necessary to think through the concept of discounting afresh. There seem to be three motivations or justifications for introducing discounting, and some doubt as to whether any of them are applicable in the context of process control. These are (i) mathematical convenience, (ii) the possibility of random termination, and (iii) the growth of capital by compound interest. For point (i), discounting is indeed a convenient way of reducing a total cost over an infinite horizon to something finite, mathematically attractive to handle and reminiscent of the concept of Abel summation. However, convenience is not a reason; any regularising reformulation must have an operational justification. If one expects that the controlled system will settle down to a steady state of some kind then minimisation of average cost in this steady state seems the natural criterion. The only point which should perhaps be emphasised is that already made earlier: the model should be such that transients are excited spontaneously if a policy effective in the steady state can also be guaranteed to be effective against transients. Point (ii) corresponds to the conviction that nothing goes on for ever. The investor of Section 2.2 cannot postpone consumption of his wealth for ever; he must reckon with the fact that he will die at some point No industrial plant or process will be run for ever; the uncertainties of future economics and technology are at least certain in predicting that Suppose that, if the plant has survived in its existing form to a given time t, then it has probability {3 of doing so to time t + 1. Then one can regard the discounted term {3F(·, t + 1) in the dynamic programming equation as representing the expectation of future cost over survival prospects. The expectation should also include a term ( 1 - {3) times the cost incurred if the plant does not survive to time t + 1. This term will presumably be zero if non-survival simply means termination; it will be something different if non-survival means movement into a new technology, for example. . However, while one may have to reckon with such contingencies on a planning time-scale, one scarcely needs to do so on the time-scale of routine control, for a which an average criterion then seems more appropriate. A secondary observation is that a discounting induced in this way would, under LEQG assumptions, nevertheless destroy LEQG structure: see Exercise 1. It is alternative (iii), the effect of compound interest, which is the usual justification. However, a complete model has to make the interest mechanism explicit. The introduction of interest implies the existence of two alternatives: either to use one's money to cover the costs of the enterprise (i.e. of the system being controlled) or to leave it in the bank to grow by compound interest. (More generally, one could consider the allocation of resources between several enterprises competing for support, but the comparison of one uncertain enterprise with the certainty of an interest account is sufficient to make the

12 WHY DISCOUNT?

319

point.) To the variable x, describing the process being controlled we must now add another variable, z, say, which represents the capital of the plant-owner who is running the process and paying the costs. This capital will obey the recursion Zt+l

/3- 1(z,- c,)

=

(54)

1

if unused capital grows by a factor ;3- at each stage and c, is to be regarded as the net process cost to the plant-owner. Thus Zh

=

;3-h

(

zo -

h-1 L /Y

) Cr

.

r=O

If the plant-owner wishes to maximise terminal capital zh then he will choose policy 1r to maximise E,.[U(zh)] for some utility function U If U is linear or if there is no statistical variation then this will be equivalent to minimisation of the expected discounted cost E,.("L~:bfJ'cr)· In all other cases it will not, and the current capital z1 must figure as part ofthe state variable ofthe problem. Of course, there is still the point that one must have an infinite-horizon formulation if one is to derive a stationary optimal policy, and some care is needed in finding such a formulation, because the surplus of capital above that needed to cover net process costs is growing exponentially. However, one could, for example, consider the minimisation of the probability of ruin: that z will at some future time fall below zero. This is in a sense a side-issue. The point is that it is the non-linearity of the exponential criterion which exposes the conventional discounted-cost model as essentially incomplete, and incomplete in its state description. Exercises and comments (1) Suppose that the criterion is that one wishes to minimise total undiscounted cost incurred over a lifetime, and that, conditional on current process history, one has constant probability j3 of surviving the the next unit time step. (Maximisation of reward would be more natural, but this can be translated into cost minimisation.) Then, given that the optimiser has survived to age t and is using an exponential criterion, the expression which he must extremise thereafter is

Here t is the time of his death, a random variable, and (1 - j3)j3i = P(t = t + jlt ~ t). This problem is invariant in t, and its optimal solution is stationary. However, even if one makes LQG assumptions on costs and dynamics (apart from those of death), the form of this last expression implies that LEQG structure has been lost.

320

RISK-SENSITIVITY: THE LEQC MODEL

Notes on the literature The LEQG criterion was first investigated by Jacobson (1973, 1977), who solved the perfect-observation state-structured case and saw the role of the noise variable as an independent auxiliary control. This treatment was extended by Speyer (1976) and Speyer, Deyst and Jacobson (1974) and Kumar and van Schuppen (1981). However, the general state-structured case with imperfect observation resisted solution until Whittle (1981) deduced the risk-sensitive certainty equivalence principle; the assumption of state structure was dropped in Whittle (1986). Whittle did not observe the point that the extremal operations in equation (8) do not commute in general; this point was corrected by Fragopoulos (1994). However, it has no consequences if the appropriate saddlepoint exists, and is located by stationarity conditions. The entropy evaluation (47) of the average cost under LEQG assumptions is due to Glover and Doyle (1988). Chang and Sobel (1987) observed that the combination of an LEQG criterion and discounting implied non-stationarity of the optimal policy.

CHAPTER17

The H 00 Formulation 1 THE Hoo LIMIT In Section 16.10 we quoted the expression

'Ya(B)

= 4 ~(}

J

log[II

+ BG(iw)G( -iw)Tj] dw = 4 ~ 0

J

log[ji + BGGIJ dw,

(1)

derived in Section 13.5, for the average rate of cost incurred in the stationary regime under the LEQG criterion with risk-sensitivity parameter B. This relation, first derived by Glover and Doyle (1988), expresses the average cost very explicitly in terms of the system transfer function G induced by a stationary, linear stabilising policy. By 'system response' we mean the response of a standardised deviation output to a standardised white-noise input. We are assuming that the equivalent filter has action G(~) and so transfer function G(s). The limits of integration are ±7l' or ±oo in the discrete-and continuous-time cases respectively. The class of response functions induced by some linear stationary stabilising policy constitutes the class of feasible G. The aim of steady-state optimisation is to choose a G in this class which is minimal in an appropriate norm, and expression (1) can be regarded as the norm on G which is implied by choice of the LEQG criterion. Let us summarise the immediately relevant properties of this norm.

Theorem 17.1.1 Assume that GG is integrable over the relevant frequency range. Then: (i) In the range(} > Ba for which the matrix I + BGG is positive definite for all real w, expression (1) is positive, finite, and non-increasing in e. (ii) The critical value Ba, necessarily negative, below which 'YG( B) is no longer defined is determined by -8(; 1 = cl(G)

(2)

where al( G) is the maximal value (over w) ofthe maximal eigenvalue of GG. Proof The normalisation of expression (16.48) to expression (1) implies that 9t ~ 0, and so that the cost function is non-negative. It follows then that the criterion x,.(O) of (16.1.3) is non-negative for all real (} for which it is defined; we see

322

THE H""' FORMULATION

from Exercise 16.1.3 that it is also non-increasing in 0. These properties then transfer to the limit evaluation 'YG (B). Integrability of GG (which implies that G is proper, in the continuous-time case) implies integrability oflogj/ + OGGI as long as the matrix I + BGG remains positive definite. As 0 is decreased this condition fails first at 0 = OG. As we shall see by example, the average cost 'YG(B) may or may not become infinite at that value, but its definition certainly fails from that point, because the derivation of evaluation (1) in Section 13.5 required positive-definiteness. D Since the class of control rules corresponding to the feasible G includes the optimal stationary rules at any given value of 0 we have

Theorem 17.1.2 The breakdown value Bof() in the stationary case can be identified with infGOG, where the infmum is over feasible values of G. Equivalently,

-&- 1 = inf al(G). G

(3)

The quantity a( G), the non-negative square root of al( G), is also a norm on G. It is known as the Hoo norm of G, otherwise written

IIGIIoo =

a(G).

The conclusion of Theorem 17.1.2 can then be expressed: an optimal stationary LEQG control rule at the breakdown point Bis also H 00 -optimal in that it minimises the Hoo norm ofthe effective transfor function. This is the essential implication of the Glover-Doyle paper, and was found particularly striking because the Hoo criterion had attracted intense interest since about 1981, for reasons we shall explain in the next two sections. The realisation that it is so closely related to the LEQG criterion, whose investigation had proceeded completely independently, marked a real breakthrough in understanding. We can make the immediate assertion that the H 00 problem is solved for the state-structured problem of the last chapter by the LEQG-optimal policies deduced in Sections 16.4-16.7, applied at the breakdown point B =B. However, we should make the connection more explicit. If the actual white-noise input has covariance matrix 91 and if the cost attach~d to an output deviation ~ is ~ ~T9t~ then the form GG is replaced by 9iG91G, to within a similarity transformation which does not affect evaluation of eigenvalues, and so of the Hoo norm. Indeed, both 91 and 9i could be wdependent, corresponding to allowance of non-white noise and inclusion of lagged terms in the quadratic cost function. Consider the system dx+~u= E

y+C€X=TJ

u = Jf"y

(4)

1 THE Hoc LIMIT

323

where the noise inputs are white with covariance matrix (12.3) and the the instantaneous cost-function is given by (12.4) (with rate-interpretations in the continuous-time case). We are thus considering a pure regulation problem for which the three relations of (4) constitute respectively plant equation, observation relation and control rule. Suppose system (5) has the solution

Then the optimal Hoo rule is that which chooses the operator :Y{ to minimise the maximal eigenvalue (maximal over eigenvalues and over w) of the matrix

[~ ~] G(iw) [~

.; ] G(-iw?.

This problem is solved in the state-structured case by the rules deduced in Sections 16.4--16.7, taken in the stationary limit and at the breakdown value (} = iJ. It follows then that the H 00 -optimal control rules are linear Markov for the case of perfect state observation, and are certainty-equivalent versions of these latter in the case of imperfect state observation. That is, the minimal stress estimate 1 at iJ (a linear estimate) is substituted for Xr in the rule. However, the 'canonical model' is often given in input-output form rather than state-structured form, see Section 3, which means that the dynamical structure which we have exploited is largely lost. One may wonder whether the concept of H 00-optimality is a useful one, since it corresponds to LEQG-optimality at the very point () = iJ at which LEQGoptimisation fails. As we shall see by example in Exercise 1 and know already from Theorem 16.9.1, if the plant is unstable then both II and K become infinite at this point. If the plant is stable then finite solutions exist for these quantities, but the determination of II is in fact unstable to positive perturbations. However, the point of the Hoo norm has rather been its usefulness in the treatment of robustness, a property which we develop in the next two sections. The optimal stationary LQG rule would be said to be based on an H 2 criterion, in that it minimises the quadratic norm Jtr( GG) dw.

x

Exercises and comments (I) Consider the direct determination of the H 00 -opti.mal rule in the simplest case: that of perfect-observation regulation to zero in the scalar continuous-time case. We assume the usual state-structured model with S normalised to zero. If N = WWT and one adopts a stabilising policy u Kx then the transfer function from the normalised plant noise to the deviation pair (x, u) is

=

G(s)

= [iL~r

324 where r = A

THE Hoo FORMULATION

+ BK. It follows that (with s = / 1

+

BHill = 1

+

iw)

BN(R+ QK2) w2 + r2

and so that

+ QK (}- = - [·m f sup BN(R ..? r2 K

w

2

)] -I

w-+

The maximising value of w is zero, so the Hoc-optimal value of K minimises (R + QK2 )1 (A + BK) 2 subject to stability, i.e. to A + BK < 0. The freely minimising value is at K = BRI A Q and is acceptable if A < 0; we have then -BN = B 21Q + A 2 I R. If A ~ 0 then the minimising value of K (subject to stability) is K infinite and of opposite sign to B, corresponding to -BN = B2 I Q. We thus confirm the identity of the Hoc-optimal rule with the LEQG-optimal rule at the breakdown point, set out in Theorem 16.9.1.

2 CHARACTERISTICS OF THE Hoo NORM The Hoc norm was originally introduced for considerations which are not at all stochastic in nature, let alone LEQG. In order to understand these we should establish some properties of the norm.

Theorem 17.2.1 Consider the case in which G is a constant matrix, so that II Gil~ is the maximal eigenvalue ofGGT. Ui? then have the characterisations

(5) where the suprema are respectively over non-zero vectors 8 and non-zero non-negative definite matrices M ofappropriate dimension. Proof The second expression in (5) is sup6[(8TGTG8)1(8ro)] which is indeed the· maximal eigenvalue of GT G and so also of GGr. If M has spectral representation M = Lj >..AoJ, where the eigenvalues ~ are necessarily non-negative, then tr(GrMG) _ I:~IGo/ tr(M) - L >..iloi which plainly has the same sharp upper bound as does the second expression ~~

0

One might express the first characterisation verbally as: II Gil~ is the maximal 'energy' amplification that G can achieve when applied to a vector. Consider now

2 CHARACTERISTICS OF THE Hao NORM

325

the dynamic case, when G(iw) is the frequency response function of a filter with action G( ~).

Theorem 17.2.2 In the case ofafilter with transfor function G(s)

E!G(~)612

IIGII2 = su p

oc

El6! 2

where the supremum is over the class of stationary vector processes {6(t)} of appropriatedimensionforwhichO < El6! 2 < oo. Proof Suppose that the 6-process has spectral distribution matrix F(w). Then we can write

E!G(~)6! 2 Jtr(G dF G) El61 2 = Jtr(dF) But the increment dF = dF(w) is a non-negative definite matrix. We thus see from the second characterisation of the previous theorem that the sharp upper bound to this last expression (under variations of F) is the supremum over w of the maximal eigenvalue of GG, which is just II Gil~· The bound is attained when 6'( t) is a pure sinusoid of appropriate frequency and vector amplitude. 0 We thus have the dynamic extension of the previous theorem: II Gil~ can be characterised as the maximal 'power' amplification that the filter G can achieve for a stationary input. Finally, there is a deterministic version of this last theorem which we shall not prove. We quote only the continuous-time case, then give a finite-dimensional proof in Exercise 1 and a counterexample in Exercise 2. Suppose that the response function G(s) belongs to the Hardy class, in that is is causal as well as stable. This implies that G(s) is analytic in the closed right half of the complex plane and (not trivially) that the maximal eigenvalue of G(s)G(s)T (over eigenvalues and over complex s with non-negative real part) is attained for a value s = iw on the imaginary axis. One can assert:

Theorem 17.2.3 Ifthe filter G belongs to the Hardy class then

II

GII2 = su OO

p

f IG(~)6!2dt f l6i 2dt

I

where the supremum is over non-zero vector signals 6( t) ofappropriate dimension. This differs from the situation of Theorem 17.2.2 in that 6 is a deterministic signal of finite energy rather than a stationary stochastic signal of finite power. An integration over time replaces the statistical expectation. The assertion is that

326

THE Hoo FORMULATION

IIGII~ is the maximal amplification of 'total energy' that G can achieve when applied to a signal. Since we wish G to be 'small' in the control context, it is apparent from the above that adoption of the H 00 criterion amounts to design for protection against the 'worst case~ This is consistent With the fact that LEQG design is increasingly pessimistic as 0 decreases, and reaches blackest pessimism at 0 = 0. The contrast is with the Hz or risk-neutral case, 0 = 0, when one designs with the average case in mind. However, the importance of the H 00 criterion over the past few years has derived, not from its minimax character as such, but rather from its suitability for the analysis of robustness of design. This suitability stems from the property, easily established from the characterisations of the last two theorems, that

(6) Exercises and comments

(1) The output from a discrete-time fllter with input 61 and transient response g1 is y, = 'L.rKrOt-r· We wish to find a sequence 1} whose 'energy' is amplified 2 maximally by the filter, in that ('L,, 1Ytl / 'L,, 18,1 is maximal. Consider the SISO case, and suppose time periodic in that Cr has period m and all the sums over time are restricted to m consecutive values. Show (by appeal to Theorem 17.2.1 and Exercise 13.2.1, if desired) that the energy ratio is maximised for 61 = eiw1 with 2 w some multiple of 2rrfm, and that the value of the ratio is then IG(iw)l =

16

IE. cre-iwrl 2 • (2) Consider the realisable SISO continuous-time filter with transient response e"' and so transfer function G(s} = (sThen II Gil~ is just the maximal value of IG(iw}f, which is a- 2. Consider a signal 8(t) = e-f3t for t ~ 0, zero otherwise, where /3 is positive. Show that the ratio of integrated squared output to integrated squared input is

or'.

2/3(o + /3}-21oo (e2at- 2e(a-f3)t + e-2f3t) dt.

If o < 0 (so that the filter is causal) then this reduces to (o2 - a/3}- 1 which is indeed less than o- 2 • If a > 0 (so that the filter is not causal) then the expression is infinite.

3 THE Roo CRITERION AND ROBUSTNESS Suppose that a control rule is designed for a particular plant, and so presumably behaves well for that plant (in that, primarily, it stabilises adherence to set points or command signals). The rule is robust if it continues to behave well even if the actual plant deviates somewhat from that specified. The concept of robustness

3 THE H,., CRITERION AND ROBUSTNESS

327

G

u

K

w +

Figure 1 A block diagram ofthe system ofequation (7). Ifa pole ofG(s) corresponding to a plant instability is cancelled by a zero of K(s), then the controlled system will not be completely stable.

thus takes account of the fact that the plant structure may never be known exactly, and may indeed vary in time. In control theory, as in statistics and other subjects, the conviction has grown that a concern for optimality must be complemented by a concern for robustness. Furthermore, both qualities must be quantified if one is to reach the right compromise between goals which are necessarily somewhat conflicting. For an example, consider the system of Figure 1, designed to make plant output v follow a command signal w. Here G and K are the transfer functions of plant and controller, and actual plant output v is distinguished from observed plant output y = v + 'TJ, where 'TJ is observation noise. The block diagram is equivalent to the relations

u = K(w- v- ry)

(7)

v- w =(I+ GKr 1 (w- GKry)

(8)

v =Gu, whence we deduce the expression

for the tracking error v - w in terms of the inputs to the system. For stable operation we require that the operator I + GK should have a stable causal inverse. Let us assume that stability holds; the so-called small gain theorem asserts that a sufficient condition ensuring this is that IIGKIIoo < l. We see from (8) that the response functions of tracking error to command signal and (negative) observation noise are respectively

One would like both of these to be small, in some appropriate norm. They cannot both be small, however, because S1 + S2 =I. S1 is known as the sensitivity of the

328

THE Haa FORMULATION

system; its norm measures performance in that it measures the relative tracking error. S2 is known as the complementary sensitivity, and actually provides a measure of robustness (or, rather, of lack of robustness) in that it measures the sensitivity of performance to a change in plant specification. This is plausible, in that noise-corruption of plant output is a kind of plant perturbation. However, for an explicit demonstration, suppose the plant operator perturbed from G to G + 8G. We see from the correspondingly perturbed version of equations (7) that

v =(I+ GK)- 1GK(w -11) +(I+ GK)- 1 (8G)K(w- v -1]). The perturbed system will remain stable if the operator I+ (I+ GK) -I ( 8G)K acting on v has a stable causal inverse. It follows from another application of the small gain theorem that this continued stability will hold if

i.e. if the relative perturbation in plant is less than the reciprocal of the complementary sensitivity, in the Hoo norm. Actually, one should take account of the expected scale and dynamics of the inputs to the system. This is achieved by setting w = w, wand 77 = W2 ry, say, where W 1 and W2 are prescribed filters. In the statistical LEQG approach one would regard wand ij as standard vector white noise variables. In the worst-case deterministic approach one would generate the class oftypical inputs by letting w and 1i vary in the class of signals of unit total energy. Performance and robustness are then measured by the smallness of IIS1 W,ILXl and IIS2 W2lloo respectively. Specifically, the upper bound on IIG- 16GIIoo which ensures continued stability is

IIS2W211~'·

Of course, in a simple minimisation of quadratic costs a compromise will be struck between 'minimisation' of the two operators S 1 and S2 in some norm. There will be a quadratic term in the tracking error v - w in the cost function, and this will lead to an expression in H 2 norms of S1 and S2. The more observation noise there is, the greater the consideration given to decreasing s2. so assuring increased robustness. The advantages of an H 00 formulation were demonstrated first in Zames (1981), who began an analysis which has since been brought to a high technical level by Doyle, Francis and Glover, among others; see Francis (1987) and and Doyle, Francis and Thnnenbaum (1992). The standard formulation of the problem is the system formulation of equations (6.44)/(6.45~ expressed in the block diagram of Figure 4.4. The design problem is phrased as the choice of K to minimise the response of D. to ( in the H 00 norm, subject to the condition that 'K should stabilise the system', i.e. that all system outputs should be stabilised against all system inputs. The analysis of this formulation has generated a large specialist literature. We shall simply list a number of observations.

3 THE H.., CRITERION AND ROBUSTNESS

329

(i) The relation of the Hoo criterion to the LEQG criterion means that many problems already have a solution in the now well-established LEQG extension of classical LQG ideas. The need to take account of observation noise ensures a degree of robustness in rules derived on either LQG or LEQG criteria, although this is an insight which might not have been apparent before Zames. (ii) The two problems formulated in this section are stated in input-output form, in that the plant is specified simply by its response function G rather than by a state-structured model, for example. There have been a number of attempts over the years to attack such problems directly, usually in the LQG framework, by, for example, simply seeking the filter K in (7) which minimises a quadratic criterion such as E(.6.T9i.6.). (See Newton, Gould and Kaiser (1957), Holt eta/. (1960), Whittle (1963),Youla, Bongiorno and Jabr (1976)). One can make a certain amount of progress, but, as we noted in Section 6.7, a head-on approach can prove perplexing and tends to miss two insights which are revealed almost automatically in a state-space analysis of the non-stationary problem. These are (i) certainty-equivalence, with its separation of the aspects of estimation and control, and (ii) the usefulness of the introduction of conjugate variables; Lagrange multipliers associated with the constraints constituted by the prescribed dynamic relations. Exploitation of these properties simplifies the analysis radically, especially in the vector case. We consider the determination of optimal stationary controls without reduction to state structure in Chapters 18-21, but using time-integral methods which incorporate these insights by their nature and yield precisely what is needed. (iii) The operator factorisation techniques associated with these time-integral methods are distinct from both the Wiener-Hopf methods used in a direct attack on the input-output model and the matrix J-factorisation techniques expounded by e.g. Vidyasagar (1985) and Mustafa and Glover (1990) for solution of the Hoo problem. (iv) Simple minimisation of .6. in some norm will in fact assure overall stability of the system (provided this is physically achievable) if 6. includes measures of relevant signals in the system. For example, the inclusion of u itself in the criteria used through the whole of this text avoids the possibility that good stability or tracking could be achieved at the expense of infinite control forces. Further, and as we have seen, such an optimisation will automatically achieve some degree of robustness if observation noise is assumed in the model Finally, one might say that the ultimate guarantee of robustness is use of an adaptive control rule; an observation which opens wider vistas than we can explore. n~

PART 4

Time-integral Methods and Optimal Stationary Policies This Part is devoted to the direct determination of the optimal stationary policy for a model which is supposed LQG or LEQG and time-homogeneous, but not necessarily state-structured. The theme is an interesting one in that the subthemes of time-integrals, certainty equivalence, the maximum principle, transform methods, canonical factorisation and policy improvement coalesce naturally. The methods are a development of those already applied to the deterministic state-structured case in Section 6.3 and generalised in Section 6.5. The reader content to consider only the state-structured case could omit this Part without prejudke to what follows (although passing up a chance for enlightenment, in the author's view!). The problem first to be considered is that of deriving a stationary optimal control policy for a time-homogeneous LQG model, in general neither statestructured nor perfectly observable. Our methods intrinsically assume the existence of infinite-horizon limits, so that the optimal policy is derived as the infinite-horizon limit (in fact stationary) of an optimal finite-horizon policy. Such a policy will certainly have the property of being average-optimal: i.e. of minimising the average cost incurred per unit time in the stationary regime. It will in fact also have the stronger property of optimising reaction to transients, which an average-optimal policy may not have unless plant noise is such that it can stimulate all modes of the system. We have an expression for the average cost in formula (13.24), and policy is specified by the time-invariant realisable linear operator :1{ which one uses to express the control u in terms of current observables. One might regard the optimisation problem simply as a matter of direct minimisation of this

332

TIME-INTEGRAL METHODS

expression with respect to .Y(. This was the approach taken by a number of authors in the early days of control optimisation (see Newton, Gould and Kaiser (1957), Holt et a/. (1960), Whittle (1963)), when it seemed a natural development of the techniques employed by Wiener for optimal prediction. However, while the approach yields results rather easily in the case of scalar variables, in the vector case one is led to equations which seem neither tractable nor transparent. In fact, by attacking the problem in this bull-like fashion one is forgoing all the insights of certainty equivalence, the Kalman filter etc. One could could argue that, if these insights had not already been gained, they should be revealed in any natural approach. If so, then this is not one. In fact, the seeming simplification of narrowing the problem down to one of average-optimisation blinds one to an even more direct approach. This is an approach which is familiar in the deterministic case and which turns out to be available even in the stochastic case: the extremisation of a time-integral. We use this term in a rather specific technical sense; by a time-integral we mean a sum or integral over time of a function of current variables of the model in which expectations are absent and which is such that the optimal values of decisions and estimates can be obtained by a free and unconstrained extremisation of the integral. In earlier publications on this topic the author has referred to these as 'pathintegrals', but this is a usage inconsistent with the quantum-theoretic use of the term. Strictly speaking, a path-integral is an integral over paths (i.e. an expectation over the many paths which are possible) whereas a time-integral is an integral along a path. The fact which makes substantial progress possible is that a path-integral can often be expressed as an extremum over time-integrals. For example, we we saw in Chapter 16 that the expectation (i.e. the path-integral) E[exp( -OC)] could be expressed as an extremum of the stress§= C + o- 1[}. If one clears matrix inverses from the stress by Legendre transformations (i.e. by introducing Lagrange multipliers to take account of the contraints of plant and observation equations) then one has the expectation exactly in the form of the extremum of a time-integral. , It is this reduction which we have exploited in Chapters 6 and 16, shall exploit for a general class of LQG and LEQG models in this Part, and shall extend (under scaling assumptions) to the non-LQG case in Part 5. We saw in Section 6.3 that the state-structured LQ problem could be converted to a time-integral formulation by the introduction of Lagrange multipliers, and that the powerful technique of canonical factorisation then determined the optimal stationary policy almost immediately. We saw in Sections 6.5 and 6.6 that these techniques extended directly to models which were not state-structured. These solutions extend to the stochastic and imperfectly observed case by the simple substitution of estimates for unobservables, justified by the certaintyequivalence principle. We shall see in Chapter 20 that time-integral techniques also take care of the estimation problem (the familiar control/estimation duality

TIME-INTEGRAL METHODS

333

finding perfect expression) and, in Chapter 21, that these methods extend to the LEQG model. All these results are exact, but the extension to the non-LQG case of Part 5 is approximate in the same sense as is the approximation of the pathintegrals of quantum mechanics by the time-integrals (action integrals) of classical mechanics, with refmement to a higher-order approximation at sensitive parts of the trajectory. There is a case, then, for developing the general time- integral formalism first, which we do in Chapter 18. In this way one sees the general pattern, uncluttered by the special features which necessarily arise in the application to control and estimation. There is one point which should be made. Although our models are no longer state-structured, they are not so general that they are given in input/output form. Rather, we assume plant and observation equations, as in the state-structured case, but allow variables in them to occur to any lag up to some value p. The loss of an explicit dynamic relationship which occurs when one reduces the model to input/output form has severe consequences, as we saw in Section 6.7. This Part gives a somewhat streamlined version of the author's previous work on time-integral methods as set out in Whittle (1990a). However, the material of Sections 20.3-20.6 and 21.3 is, to the best of the author's knowledge, new. There must be points of contact in the literature, however, and the author would be grateful for notice of these.

CHAPTER 18

The Time-integral Formalism 1 QUADRATIC INTEGRALS IN DISCRETE TIME For uniformity we shall continue to use the term 'time-integral' even in discrete time, despite the fact that the 'integral' is then a sum. Consider the integral in the variable~

(1) with prescribed coefficients G and (. This is then a sum over time of a quadratic function of the vector sequence {~T}, the function being time-invariant in its second-degree part. So, for the models of Sections 6.3 and 6.5 ~is the vector with sub-vectors x, u and .X, and the time-varying term (T would arise from known disturbances d or known command signals xc, UC. The matrix coefficients G1 and the vector coefficients (T are specified, but no generality is lost by imposition of the normalisation

(2) If the sum in (1) runs over h1 < r < h2, then the 'end terms' arise arise from contributions at times r ~ h1 and r ;?: h2 respectively. The final term can be regarded as arising from a terminal cost and the initial term as arising from a probabilistic specification of initial conditions. Suppose the optimisation problem is such that we wish to extremise the integral with respect to the course of the variable {~T}. We cannot at the moment be more specific than 'extremise', as we see by considering again the models of Chapter 6, for which one minimised with respect to x and u and maximised with respect to >.. In any case, the stationarity condition with respect to ~T is easily seen to be

(3) if r is sufficiently remote from both h 1 and h2 that neither end term involves where

Cf>(ff) :=

L Gjffi. j

~n

(4)

THE TIME-INTEGRAL FORMALISM

336

The normalisation (2) has the implication

(5)

4>=CI>

indicating a kind of past/future symmetry in relations (3). Here, as ever, the conjugation operation bas the effect CI>(z) = CI>(z- 1 onz-transforms. Suppose that we regard our optimisation as a 'forward' optimisation in that, if t is the current instant of time, then is already determined for T < t and we can optimise only for r ;;a: t. We shall then write equation (3) rather as

Y

eT

I

(r ;;a: t)

(6)

to emphasise this dependence on t. That is, relation (6) determines the optimal course of the process from timet onwards for a prescribed past (at timet). This was exactly the situation which prevailed for the control optimisation of Sections 6.3 and 6.5. If the horizon h2 becomes infinite then we may indeed demand that (6) should hold for all r ;;a: t and, if the sequence {(T} and the terminal term are sufficiently well-behaved, expect that the semi-inversion of the system (6) analogous to equation (6.27) should be valid. For the estimation problems of Chapter 20 we shall see that the same ideas apply, but with the optimisation applied to the past rather than the future (cf. Section 12.9). To consider the general formulation (1) is then already a simplification, in that we see that these two cases can be unified, and that the operator CI>(S') is immediately related to the matrix coefficients Gj of the path integral Indeed, partial summation brings this integral to the neater form

e

0=

L (.r)eT - (~ ~T) + end terms. T

If CI>(z) possesses a canonical factorisation then, in virtue of (5), we may suppose it symmetric:

(7) 1

where both 0 (necessarily symmetric) for generality. It could be normalised to the identity by redefinition of ¢J (see Exercise 1), but we shall not find this to be the natural normalisation. If we assume 'dynamics of order p' in that G1 = 0 for ljl > p then it will turn out that
Theorem 18.1.1 Suppose that CI>(z) has the canonical factorisation (7). The the infinite-horizon system (6) has the semi-inversion

¢J(ff)e~'>

1

= CI>01¢J(S'f (T

(r ;;a: t)

(8)

2 FACTORISATION AND RECURSION IN THE 'MARKOV' CASE

337

---1

(with the understandings that :!1 operates on the subscript T and that¢(:!/) is to be expanded in non-positive powers of:!/) if the right-hand member of (8) is defined. This latter condition will be satisfied if ~r decays as pr for increasing T, where p is less than the radius ofconvergence of¢(z)- . The passage from (6) to (8) converts the recursion (6), symmetric in past and future in the Sense (5), to a Stable forward recursion, expressing e~) for T ~ t in terms of past values e~l(a < r); the right-hand side of (8) constitutes a known driving term for this recursion. One could complete the inversion to obtain a complete determination, but to do so is positively undesirable. Relation (8) determines explicitly (in terms of the known fort- p ~ T < t) and, aS We saw for the control application of Chapter 6, this determination yields the optimal control rule in the form which is both closed-loop and natural for realtime realisation. Of course, if (8) is to hold in the infinite-horizon limit h2 --> oc for system (6) then certain conditions must be satisfied. The analogue of the controllability/ sensitivity conditions of Theorem 6.1.3, which were the basic ones for the existence of infinite-horizon limits, is simply that a canonical factorisation (7) should exist. Regularity conditions will also be required of the terminal contributions implied in the specification of the path integral (1). If such conditions are granted, then the infinite-horizon solution (8) is formally immediate. However, to make it workable we must find a workable way of deriving the canonical factorisation (7). We shall achieve this by returning to the recursive ideas of dynamic programming and relating the canonical factor ¢ to the value function. This value function need never be determined; it is enough that the dynamic programming technique of policy improvement implies a rapidly convergent algorithm for determination of the canonical factorisation. Let us begin with consideration of the 'Markov' case, for which dynamics are of order one and recursions are simple.

e;t)

er

Exercises and comments

(1) A necessary condition that (z) should be invertible in stable form is that it should be non-singular for z on the unit circle, which implies that all three factors in (7) should be non-singular there. A sufficient condition for canonical factorisability is that (z) should be strictly positive definite on the unit circle. This latter condition will in general not hold in our analysis, however, because the quadratic form constituted by Dhas in general a mixed convex/concave character. 2 FACTORISATION AND RECURSION IN THE 'MARKOV' CASE If G1 = 0 for ljl > p, so that interactions (cross-terms) appear in the time-integral only at time lags up to p, we shall say that we are dealing with pth-order

338

TIIE TIME-INTEGRAL FORMALISM

dynamics. We shall then refer to the case p = 1 as the Markov or state-structured case. In the Markov case the time-integral can be written D=

L

Cr

+ end terms,

T

where

(9) Consider now a recursive extremisation of the integral, with Cr regarded as an 'instantaneous cost', and with the closing cost at the horizon point h2 assumed to be a quadratic function of ~h2 alone. If we are interested in a forward optimisation then we can define a 'value function' F(~~o t), this being the extremised 'cost' from time t onwards for given ~1 • This will obey the dynamic programming equation

(10) where 'stat' signifies evaluation of the bracket at a value of ~t+I which renders it stationary. As a dynamic programming equation this has the simplifying feature that one optimises with respect to the whole vector argument ~ rather than just some part ofit. In the case under consideration the value function will have the quadratic form

(11) for t =::;; h2 if it does so at h2, with the coefficients (respectively matrix, vector and scalar) obeying the backward recursions

II, =Go- G-1II~ 1 G1 a,=(,-

G_Irr;;1at+I

(12)

61 = 61+1- aJ+ 1 II~ 1 at+l· The first of these is just the Riccati recursion; strikingly simpler in this general formulation than it was when written out for the state-structured case of the control problem in equations (225)/(2.26). The second relation likewise corresponds to the equation before (2.65). The third gives the recursion for the ~­ independent component of cost incurred which we thought too troublesome to derive in Section 2.9. The extremising value of ~t+ 1 in (10) is determined by

(13) Suppose that we are in the infinite horizon limit h2 ---) +oo, and that the problem is well-behaved in that II, has, for fixed t, a limit II. It is under these conditions that one expects the forward recursion (13) for ~~ to be stable in that its solution

3 VALUE FUNCTIONS AND RECURSIONS

339

should tend to zero with increasing t. In other words, that II + G1ff is a stable operator, or that r = - II- 1G1 is a stability matrix. This leads one to the conjecture that the right canonical factor ¢>(z) of ili(z) might be proportional to

II+ G1z. Indeed, we find from the equilibrium form of the Riccati equation for II above that (14) is a factorisation of iii, and we see from (13) that this is indeed the canonical factorisation we seek if one exists. The condition for factorisability is then that the Riccati equation (the first equation of (12)) should have a solution II for which n- 1 G1 is a stability matrix. Equation (14) thus relates the canonical factorisation to the recursive calculations familiar to us from dynamic programming. We should now like the version of this relation in higher-order cases. 3 VALUE FUNCfiONS AND RECURSIONS In the last section we related the canonical factorisation of ili(z) in the Markov case to the evaluation of the matrix II of the value function. We aim now to extend that result to the case of pth-order dynamics. In doing so we may suppose that (r 0, since iii and its properties are determined entirely by the second-degree part of the path integral (1). In control language, we specialise to the case of regulation to zero. One can group the terms in Dto define an 'instantaneous cost' Cr in various ways; we shall define it as

=

p-1

Cr = H:-p+ 1Go€r-p+1

+ L{:_JGp-J{r-p·

(15)

j=O

If we define the value function

F({, €1-1, ... , {t-p+1; t) =stat

{,;r>l

[I>r] T

(16)

;;.t

then this will obey the optimality (dynamic programming) equation

We suppose that the terminal cost is purely quadratic with a limited dependence on the past. The assumption of pth-order dynamics then implies that F will indeed be a function of just the variables indicated in (16), and it will in fact be homogeneous quadratic (i.e. of degree two) in the {-variables:

THE TIME-INTEGRAL FORMALISM

340

sa)' The 'optimal policy' will then be of a p-lag linear form in that the extremal criterion in (17) will determine 1 linearly in terms of the p previous values of

e:

et+

p

I:ak(t+ l)e,-k+t

=o

(19)

k=O

say, with a 0 necessarily symmetric. Our expectation is that the coefficients ak(t + 1) will become independent oft in the infinite-horizon limit (if this exists) and that (19) will then be a stable forward recursion which, written ¢(ff)et+l = 0, defines the canonical factor ¢(z). The argument follows. Ifwe substitute the form (19) into the optimality equation (17) we obtain a 'superRiccati' equation for the p(p + 1) /2 matrices ITjk which is most forbidding. The relations become much more compact in terms of the generating functions

F(w, z; t)

=I: L ITjk(t)wlzk j

l)(w, z) =Go+

(20)

k p

p

j=l

}=I

L Gjzj + L G_jwl

¢(z;t+ I) =F(O,z;t+ I)+GpzP

(21) (22)

p

a(z;t+ I)= L:ak(t+ I)zk

(23)

k=O

in the complex scalars z and w. We shall refer to F( w, z : t) as the value function

transform. We shall now find it convenient to suppose that F is not necessarily the value function corresponding to an optimal policy, but that corresponding to an arbitrary p-lag linear policy of the form (19). It will then still have the homogeneous quadratic form (18).

Theorem 18.3.1 (i) The value function transform under the policy (19) obeys the backward recursion (dynamic programming equation)

F(w, z; t) = (wz)- 1[(wz)P ~(z- 1 , w- 1) + F(w, z)- ¢(w)Ta01a(z) - a(w)Ta 1¢(z) + a(w?aij 1¢oaij 1a(z)jt+l

(24)

0

where the final subscript indicates that F, a and¢ all carry the additional argument t + 1.

4 A CENTRAL RESULT: THE RICCATI/FACTORlSATION RELATIONSHIP

341

( ii) The optimal choice ofo is

o(z; t + 1} = ¢(z; t + 1) = F(O,z; t + 1)

+ GP:zP

(25)

With this choice (24) reduces to the optimality equation F(w, z; t)

= (wz)- 1[(wz)PCJ>(z- 1, w- 1) + F(w, z)- rt>(wl rj>01rt>(z)) + 1. 1

(26)

The proof is by heavy verification; the detail is given in Whittle (1990a) p.164. One can see that the terms independent of o in the right-hand member of (24) originate from the presence of the cost term c1 in (18) and the translation of the tsubscripts of the ~-arguments. The terms in o originate from the fact that ~t+l is expressed in terms of earlier ~values by (19). Although derivation may be heavy, we see that in (26) we have an elegant expression of the optimality equation, and in (25) an elegant expression of what will soon be identified as a canonical factor ¢ in terms of the value function transform F. Note the need to introduce the double generating furnction (21), related to CI>(z) by

CI>(z) = CI>(z- 1,z).

(27)

4 A CENTRAL RESULT: THE RICCATI/FACTORISATION RELATIONSHIP If we assume the existence of infinite-horizon limits under an optimal policy then the assertions of Theorem 18.3.1 become even simpler.

Theorem 18.4.1 (i) In the infinite-hnrizon limit the optimal value function transform has the evaluation

in terms of

rj>(z)

= F(O, z) + G1 :zP.

(29)

t)

(30)

(ii) Theequation (r

;;J!:

holds along the optimal path. The assertions also begin to have a very clear significance. If we set w = z- 1 in (28) we obtain

(31) This is nothing but a canonical factorisation of 4>(z), since the forward recursion (30) is stable. We thus deduce

342

THE TIME-INTEGRAL FORMALISM

Theorem 18.4.2 (i) If infinite-horizon limits exist then~ has a canonical factorisation (7) with
= Gp

(32)


~o =
1

(33) (34)

•

These two theorems express the close and elegant relation between the infinitehorizon value function transform F(w, z) and the canonical factor
cP(;)(z)

=L


J=O

denote the approximation to
([J(i+l)cP"i:/
+ ([J(r!
(35)

5 POLICY IMPROVEMENT AND SUCCESSIVE APPROXIMATION

343

Theorem 18.5.1 Recursion (35) has the following properties. (i) It is the recursion generated by application ofthe Newton-Raphson algorithm to the factorisation (31 ), regarded as an equation for ¢. (ii) It is also the recursion which would be derived by application of the policyimprovement algorithm. (iii) It conserves the normalisation properties (32) and (33) in that, if ¢u) (z) is a polynomial in z of degree p for which the absolute term ¢;o is symmetric and the coefficient ¢;p ofzP equals GP' then the same is true of ¢(i+I) (z). Proof Consider first assertion (i). For notational simplicity, denote the consecutive trial solutions ¢(1) and ¢{i+I) by¢ and 'lj; = ¢ + .6.. Replacing¢ by¢+~ in (31) and expanding as far as first-order terms in .6. we obtain ~ = ¢¢"01 ¢

+ 3.¢"01 ¢ +

¢¢"01.6. + ¢¢r;l.6.o¢(jl ¢.

Setting .6. = 'lj; - ¢ in this relation we deduce (35). Assertion (iii) is easily confirmed from relation (35). Assertion (ii) can be confirmed by appeal to relations (24) and (25). More economically, the proof of equivalence of the two algorithms established for the state-structured LQ control 0 problem in Section 3.5 will transfer to this more general case. Property (34) is implicitly conserved under the algorithm, in that we define ~;o -~,-1 as 'l';o . Equation (35) constitutes a linear equation for ¢(i+l)• but we can be more specific. The solution of this equation can in fact be expressed in a form which is explicit to within application of the operation of truncating the series expansion of a function. Suppose we have a functionf(z) with a power series expansion 00

f(z) =

L

jjz 1

J=-oo

valid on the unit circle. Then we shall define [f(z)]+ as the function obtained by the retention only of terms for non-negativej in this expansion: 00

[f(z)]+ = Ljjzl. }=0

Theorem 18.5.2

The solution ofequation (35) can be expressed ¢i+l

= ¢;o[¢(i)~¢un+¢(i)·

Proof We can write relation (35) as ¢"01'1j;¢-l + cjj-lifi¢"01 = ¢-~~r~ + ¢"0l'lj;o¢ol·

(36)

344

THE TIME-INTEGRAL FORMALISM

in the ¢, 'ljJ notation introduced earlier. Applying the truncation operator 0 throughout, we deduce (36). It is worth while noting a point which is not immediately evident from (36): that the right-hand member of this equation is indeed a polynomial of degree pin z. We shall refer to (35) and its more specific form (36) as the PI/NR algorithman abbreviation for 'policy improvement/Newton-Raphson'. We shall apply the algorithm to particular control examples in the next chapter and shall see then that it is useful, not merely for numerical conclusions, but also for structural ones. Explicitly: Theorem 18.5.3 Suppose that a property of the normalised trial factor ¢(i) is conserved under the PIINR algorithm, so that it is shared by ¢(i+I} Then the normalised canonicalfactor ¢has this property. Proof If the policy at stage i is non-optimal, so that ¢(i) is not a canonical factor, then ¢(i) will be modified by the algorithm. The conservation ofa property under the algorithm thus implies that it is shared by some canonical factor. But the normalised canonical factorisation is unique, so the property must be shared by ¢. 0

6 THE INTEGRAL FORMALISM IN CONTINUOUS TIME The continuous-time analogue of the path-integral (1) will now indeed be an integral over time

0(~) =

j c(~)

dT +end effects

(37)

where c( ~) is a quadratic function of the vector variable ~( t). We shall assume the specific form

c(() =42:2:2:I>J~·l~J'l~fl- (T~, j

(38)

k

where ~J'l = ~, ~J is the rth time differential of the jth com~onent of~ and ( is a known vector function of time. The scalar coefficients c1~s) are supposed constant, so that the integral is time-homogeneous as far as the second-degree terms are concerned. As a matter of normalisation we can demand the symmetry (sr)

(rs)

(

c~ = ~k

) 39

However, this normalisation is not the only possible recasting of the integral (37). Partial integration yields the effective equality ,dr],ds] _ _ ,dr-l],ds+l]

"'J "'k -

"J

"'k

(r > 0),

(40)

I

345

6 THE INTEGRAL FORMALISM IN CONTINUOUS TIME

~ere we mean by 'effective equality' t~at t~e substi.t~tion wi.ll cha~ge ~nly ~he ~~d-terms and so does not affect the optimality conditions at time pmnts mtenor ':io the range of integration. · We can carry the reduction (40) so far as to write the path-integral (37) as

~ =!

JI: I: I: L cXs) (-)'{j~f+s] J dT-

(T ( dT +end effects,

or, more compactly, u=

I[~ e'P(!?))~- (T ~] dT +end effects.

(41)

Here ip (!?)) is an n x n matrix of operators with jkth element

(42)

Theorem 18.6.1 The condition that the (37) stationary can be wn'tten

~-path

for

(T

T

~

~

t should render the integral

t)

at time points interior to the interval of integration. The matrix operator Hermitian in that

(43) 'P(.~)

is

(44) The second assertion is an immediate consequence of (39) and (42); the first then follows from (41). We can write the symmetry condition (44) as fP = ~. consistently with previous continuous-time conventions. If, as implicitly indicated, the optimisation is a forward one then, as ever, we seek a canonical factorisation

'P(s) = fP_(s)'Po'P+(s). The Hermitian character of fP means that we can indeed look for a symmetric factorisation

'P(s) = ¢(s)'Po¢(s)

(45)

1

say, where both ¢(s) and ¢(sr have all their singularities strictly in the left half of the complex s-plane. If infinite-horizon limits exist then the infinite-horizon solution is again effectively given by the semi-inversion of (43) to the stable forward equation

(46) for the optimising~· As in the discrete-time case, there is a natural normalisation of the factorisation (45), determined by deduction of this factorisation from the

346

THE TIME-INTEGRAL FORMALISM

optimality equation of a recursive approach. We cover these matters in the next section, which reveals both the analogies to and some differences from the discrete-time material of Sections 4 and 5. 7 RECURSIONS AND FACTORISATIONS IN CONTINUOUS TIME

In considering the factorisation question we can again normalise to the homogeneous ('regulation to zero') case ( 0. There is a difference between the discrete- and continuous-time cases which does not seem more than notational, but is annoyingly persistent. It refers to degree of dynamics, and has already arisen in Section 4.7. When we said that dynamics were of order p in the discrete-time case we really meant that they were of order p at most. If they were in fact of lesser order in some relations then this did not matter; the only requirement was that a recursion such as (19) should really be a determining forward recursion, in that the coefficient a 0 of ~ at the current time-argument should be non-singular. The corresponding requirement in the continuous-time case is that the matrix coefficient of the differentials of highest order should be non-singular, and this order may well be different for different components of ~. Suppose that, under the normalisation (39), the differentials of ~j occur inc(~) up to order rj exactly (j = 1, 2, ... , n). Then what one might term a minimal-order linear policy would determine the forward evolution of~ by a set oflinear relations

=

(j= 1,2, ... ,n),

(47)

say, where the coefficients K. may be time-dependent. The point is that the optimal policy will lie within this class, and that determination of the optimal relation (47) in the infinite-horizon limit implies a determination of the canonical factor ¢(s). Let F(~, t) denote the value function under policy (47), starting from a known~­ history at time t. Although we have loosely written this as ~-dependent, in fact it will be dependent on the function ~(T) through the differentials ~j'l(r < rj; j = 1, 2, ... , n) at the current instant t. Under appropriate restrictions on the form of the terminal cost it follows then that the value function under policy (47) is of the homogeneous quadratic form

F(~,r)

=

1LLLLnXs)(t)~j'~~r1 j

(48)

k

and that Fwill obey the backward equation

c + /JF + " " ~[r+IJ /JF = O

ot

~~ 1

r
1

adrJ

(49)

'>j

with ~}'1 ) expressed in terms oflower-order derivatives by (47). Under an optimal policy (49) is replaced by the optimality equation

7 RECURSIONS AND FACTORISATIONS IN CONTINUOUS TIME

stat [c +

C::: + L LEJ'+ 11 J

r

8

~1 ] = 0

a~j

347

(50)

where the stationarity is with respect to the highest-order derivatives ~[r;J(j

= 1, 2, ... , n).

~ Again we introduce transforms, and define the matrices of generating functions

(:L:Ln}~1 ''2 Js; 1 s;2 )

F(s1,s2) =

'I

'2

the t-dependence of Fand II being understood. The bracketed expression is the jkth element of the matrix in question, j and k taking values 1, 2, ... , n if~ has dimension n. We again refer to F(s 1, s2) as the value function transform. We shall have occasion to distinguish the parts of these generating functions associated with the highest-degree terms, and so define

H

= (h1k) = (c)i''k)) d(s)

=

diag(s'J).

c1 (s)

=

(2:: c)!t •) s•)

(51)

s
F 1 (s)

Theorem 18.7.1

•J s·) . = ( "~ nU-1, jk

(i) The optimality equation for the value function transform

F = F(s1, s2) is c(s1, s2)

8F T 1 +at+ (s1 + s2)F- cP(si) n- cP(s2) = 0,

(52)

where cP(s) = Hd(s) + c1 (s) + F1 (s). (ii) The optima/future course of~ is determined by the forward equation cP(~)~ = 0.

(53)

(54)

This is the analogue of Theorem 18.3.1 (ii), and proved analogously; for further details see Whittle (1990a) pp.l94-5. Note that, as previously, the form (53) of the canonical factor is determined simply from the stationarity condition in the dynamic programming equation. This gives a forward differential equation for E.

348

THE TIME-INTEGRAL FORMALISM

necessarily stable if infinite-horizon limits exist, and we write it as¢(£!))~ = 0 to deduce (52). The way is now open for our principal conclusion; the analogue of Theorem 18.4.2.

Theorem 18.7.2 Suppose infinite-horizon limits exist for the optimal policy. Then the canonicalfactorisation (55)

holds with H equal to the known matrix (51) and ¢(s) equal to the infinite-horizon limit of expression (53). This factorisation then satisfies the normalisation that both ci> 01 and the matrix coefficient of the highest-degree terms in ¢(s) equal the known matrix H. This follows simply by the substitution ofs 1 = -s2 = -sin the equilibrium fonn of (52), and appeal to the fact that (54) must be a stable forward recursion. The continuous-time case thus shows one simplification: the matrix H defined in (51) is known before the factorisation is attempted, while the matrix ¢o of (31) is not.

8 THE PIINR ALGORITHM IN CONTINUOUS TIME The same simplification persists in the policy-improvement algorithm which is, as ever, identical with the Newton-Raphson algorithm applied to relation (45) as an equation in¢. If ¢(i) is the determination of the canonical factor at stage i then the recursion analogous to (35) is -1 -1 -1 ¢U+I)H ¢(i) + ¢(i)H ¢(1+1) =+ ¢uJH ¢uJ (56) and the normalisation indicated in Theorem 18.7.2 is conserved under the algorithm. The more explicit recursion (36) also has an analogue. Suppose a functionf(s) is representable for purely imaginary s by the integral

f(s)

=

1:

51

e dg(t).

1:

We then define the truncation operator [ ]+ in continuous time by [f(s)]+

=

31

e dg(t).

Then, just as in Section 5, the solution of (56) can be expressed 1 ¢(i+1) = n[¢(1) ci>¢(in+¢(i).

(57)

The derivation of these results is very much as for the discrete-time case; the argument is amplified in Whittle (1990a), pp. 197-8.

CHAPTER 19

Optimal Stationary LQG Policies: Perfect Observation 1 DERIVATION OF THE SOLUTION FROM THE TIME-INTEGRAL The essential steps in the treatment of the perfect-observation case by pathintegral methods are summed up in the short Section 6.5. In this section we simply recapitulate and expand that discussion. In the following two sections we see how the results of the previous chapter can be used to sharpen conclusions. ln the discrete-time case we assume a plant equation

(1) where d and !!J are time-invariant causal linear operators, {dr} is a known deterministic sequence of disturbances and { fr} is Gaussian white noise with zero mean and covariance matrix N. If we assume pth-order dynamics than the operators will have the forms d =A(~)= I:~=O Ar~r and !!J B(fl") E~=I Br~r. We can make the normalising assumption Ao = I, and have set Bo = 0, corresponding to the assumption that the control u1 determined at time t has an effect first at time t + 1. This reflects the inevitable fact that, if u1 may depend upon data which includes Xr. then x 1 cannot be causally dependent

=

=

onUt·

We shall suppose, as in Section 2.4, that the instantaneous cost function has the time-homogeneous quadratic form

Cr=c(x,,u,)=![~r[~ ~][~l

(2)

The command signals xc and ut make no appearance here. We have assumed that, if they are known in advance, then they have been normalised to zero, by the adoption of x - r and u - ut as redefined process and control variables and of d- dxc- !!Juc as a redefined disturbance variable (cf. Section 2.9). If they are not known in advance, then we presume them generated in the model, and so subsumed as part of the process variable x. We could generalise c1 by the incorporation of higher-order lags in the variables, as we have in the plant equation. However, it is easier to start from the familiar assumption (2); the natural generalisation becomes obvious at a later point.

350

OPTIMAL STATIONARY LQG POLICIES: PERFECT OBSERVATION

The development from now on is exactly as in Section 6.3. By the certainty equivalence principle the white-noise term t: will have no effect on control optimisation, and we can for the moment delete it (although it will certainly play a role in the next two chapters). Regard the consequent deterministic process equation as a constraint and associate with it a vector Lagrange multiplier >.,. The consequent Lagrangian form h-l

u

=L

r=O

h

c(xr. u,., 7") +

L >.;(dx + gju- d)T + ch.

(3)

r=l

constitutes our time-integraL Here C11 is the closing cost at time h; we could, consistently with other assumptions, suppose it a quadratic function of ~ (h- p < 7" ~h). We again make the distinction between 7", a general running time variable, and t, which labels the moment 'now: In other words, we assume that Ur has already been determined for T < t, not necessarily optimally, and that u,. is to be determined forT~ t. Extremisation of I with respect to (x, u, >.)r gives a linear system of equations which we can write as

(t

~ 7"

< h- p).

(4)

The corresponding equations for h - p < T ~ h will be modified in that they will include contributions from Ch. As in Section 6.5, iJI and 91 are the operators conjugate to .s;l and t'J in that, for example, 91 = A(S"'- 1) T = I;,=O A"[!!T-r. So .s;l operates into the past and .il into the future. Note that, effectively, >.r = 0 for 7"

>h.

The superscript (t) indicates that the optimisation is one holding from timet onwards. The values of .x}'l and u}1l for T ~ t determined by (4) and subsequent equations constitute the prediction at time t of the future course of the optimally controlled process. This will coincide with the actual course only if there is no plant noise t:. However, the determination of ui'l thus obtained is optimal under all circumstances. If we write the equation system (4) as

(5) then this can be identified with the equation (18.3) of the general path-integral treatment. The matrix function ~(z) thus implicitly defined certainly satisfies the symmetry requirement~=~ of the general treatment. Suppose that ~(z) has the canonical factorisation

(6)

2 EXPRESSION OF THE OPTIMAL CONTROL RULE IN DISCRETE TIME

351

where we have supposed the normalisation of (18.31), the natural discrete-time normalisation. Then. the semi-inversion

(r

~

t)

(7)

of (5), if valid, provides the optimal stationary solution in the infinite-horizon

limit. It provides the solution in that relation (7) constitutes a stable forward relation determining the predicted course of the optimally-controlled process. However, more importantly, relation (7) for r = t gives an explicit expression for the optimal value of Ut in closed-loop form. We shall make this determination more explicit in the following section; one may well claim it as the most direct, economic and natural determination of the optimal stationary policy. The condition that a canonical factorisation (6) should exist is just the condition that infinite-horizon limits should exist in the case ( 0; the case of 'regulation to zero'. The condition that the right-hand member of (7) should be finite (with the operator acting into the future) is just the condition that the optimally controlled system should be able to cope with the input (; see the discussion after Theorem 6.3.1. The continuous-time analogue is fairly immediate. The operators in the plant equation are now polynomials in the differential operator~. so that, for example, .91 =A(~)= A,~'. Sums are replaced by integrals in (3), and relation (4) holds with the definition i1 =A( -~)T of conjugacy. In analogue to (5) one writes this system as

=

Lr

(8) where a time argument r is understood and !!) is understood as d/ dr. If ~(s) has the canonical factorisation

(9) where we have supposed the normalisation of Theorem 18.7.2, then the analogue ofthe semi-inversion (7) is

(r

~

t).

(10)

2 EXPRESSION OF THE OPTIMAL CONTROL RULE IN DISCRETE TIME

l

For the LQG control problem the matrix generating function cl>{z) has the form cl>(z)

R(z)

S(z)

A(z)

A(z)

B(z)

0

= S(z) Q(z) B(z) . [

(11)

352

OPTIMAL STATIONARY LQG POLICIES: PERFECT OBSERVATION

Here have in fact generalised the cost function somewhat by replacing the matrix R by a generating function R(z), etc. This corresponds to the inclusion of lagged variables in the cost function c1; the generalisation can be made painlessly at this point and the conclusions we now reach remain valid under it. Recall that B0 = 0, and suppose the normalisation Ao = I. This form is special enough that we can say something about the form of the canonical factor cp(z) in the normalised factorisation (9).

Theorem 19.2.1 The canonicalfactor cp ofthe matrix generating function (11), under the normalisation of Theorem 18.4.2, has the form

c/J =

c/Jxx c/Jux

[ A

c/Jxu /] c/Juu 0 , B

(12)

0

the partitioning being the (x, u, .X) partitioning of , and an argument z being understood throughout. If the cost function involves lags ofsize p- l at most then the sub matrices with the cp-labels are polynomials in z ofdegree p - 1 at most. Proof Validity of the form (12) follows by appeal to the recursive algorithm (18.35). We leave the reader to verify that, if ¢J(i) has the form (12~ then so does ¢J(i+l)· So then does cp, byTheorern 18.5.3. The final assertion follows from the fact that, under the assumption stated,

c/Jp see (18.32).

= Gp = [

~ Ap

00 0]0 ; Bp

0

0

Theorem 19.2.2 Suppose that infinite-horizon limits exist. Then the optimal value ofu, is given explicitly and in closed-loop form by

(13) where the u.X subscript on the bracket indicates the extraction of the corresponding submatrix in the (x, u, .X)-partitioning of the bracketed matrix. If the cost function involves lags ofsize p - 1 at most then relation (1 3) expresses the optimal u1 in terms of(x,, Xt-l, ... , Xt-p+l), (ut-1 1 Ut-2 1 • • • , ut-p+l)and (d, dt+l, .. .). Proof Relation (13) follows if we extract the u-subvector relationship from rela0 tion (7) at T = t, taking account of the form asserted for cp in Theorem 9.2.1. The closed-loop determination (13) of the optimal control is explicit to within achievement of the canonical factorisation, and is both as neat and as explicit a solution as the general case will allow.

3 OPTIMAL CONTROL RULE IN CONTINUOUS TIME

353

We gain both some econo~y and some insight if we introduce the s:ystem notation of Section 6.6, and write the system (4) in the generalised case (11) as

[9t(5)

~(5)]

~(ff)

[x](r)= [0]

0

).

d

r

r

(14)

1

where then

X=[~], and the plant equation reduces to ~"'=d. The assertion (12) is then that the normalised canonical factor ifJ takes the form

ifJ(z) _ -

[ifln(z)

~o]

~(z)

0

(15)

'

where 91o is the absolute term in the matrix polynomial ~(z). We can follow through the proof of Theorem 19.2.1, but in the more economical system notation throughout, to deduce (15) directly. However, we have to return to the more explicit notation of relation (13) if we wish to distinguish the particular role of u, and deduce the control rule explicitly. Exercises and comments (1) For the Markov case .s
=

+ ATru R S + BTIU [ 1-Az

+ ATITB Q + BTIIB

sT

-Bz

1]0 [9t + ~ I19ll T

=

0

where II is the matrix of the infinite-horizon value function F( x). 3 EXPRESSION OF THE OPTIMAL CONTROL RULE IN CONTINUOUS TIME We again have the feature, discussed already in Sections 4.7 and 18.7, that, for instance, the analogue of the matrix Ao of the discrete-time case is the matrix coefficient, A. say, of the highest-order derivatives of components of x occurring in the plant equation. That is, suppose that 'i is the highest order of derivative of the jth component Xj of x that occurs in the plant equation. If A, has jkth component a)2 then (16)

354

OPTIMAL STATIONARY LQG POLICIES: PERFECT OBSERVATION

This matrix must be non-singular if the plant equation is to constitute a dynamic relation which determines the forward path of the process. We correspondingly define B. as the matrix coefficient in !II of the highest-order derivatives of u occurring in the plant equation or the cost function c, and Q. as the matrix in c of these highest-order derivatives. For the control problem the matrix <JI(s) again ha~ the form qt), with argument s replacing z, and of course the modified definition A = A ( -s) of conjugacy. Theorem 19.3.1 For the control problem in continuous time the factors Hand ¢(s) ofthe normalisedfactorisation (9) have the forms

H=

0 0

[ A.

0 Q. B.

AT] BI '

{17)

0

The form of H follows from its general definition in (18.51~ and the form of¢ by the argumentofTheorem 18.2.1. The fact that B. is in general non-zero makes the solution for the optimal value of u(t) a little more complicated than the corresponding expression (13) for the discrete-time case. From relation (10) at T = t we deduce that

¢xx(!'l)x + ¢x,(21)u + AY >. = Px>.(!'l)d ¢ux(~)x + ¢uu(~)u +BY>.= P.,>.(21)d

whereP(!'I} = H¢(~) -I and Px,x(~) etc. indicates the correspondingsubmatrix. All quantities are evaluated at time t. Eliminating >. we deduce the relation [¢ux(~)- B.A; 1¢xx(!'l)]x + [¢.,.,(~)- B.A; 1¢xu(!'l)]u = [P.,;,.(~)- B.A; 1Px>.(!'l)]d.

(18)

This must be regarded as a forward differential equation for the optimal u, driven by terms in differentials of known x. It can, of course, be solved in transform terms, but would be generated in real time by a mechanism realising the differential equation. The relation simplifies considerably under certain conditions. Suppose, for example, that u does not appear in differentiated form in either cost function or plant equation and that the cross-terms between u and x in the cost function have been normalised to zero. Then <J!(s) takes the form <J!(s) =

R(s) 0 [ A(s)

0 A(s) Q B;f Bo 0

l

(19)

355

3 OPTIMAL CONTROL RULE IN CONTINUOUS TIME

where Qis now a simple matrix, as is Bo. Let us suppose, for simplicity, that A. has been normalised to the identity. Then one, finds, by the arguments of Theorem 18.7.2, that the expressions (17) reduce further: to

H

=

[

0 0

0 Q

I

Bo

l

I

BJ ,

¢(s)

=

0

rPxx(s) 0

[ A(s)

l

0 BJ I . Q Bo 0

It follows then that (18) simplifies to

.u = Q- 1 {BJ"¢xx(~)x+ [PuA(~)- BJPxA(~)]d}.

Exercises and comments (1) The return difference equation

Consider again the classic feedback loop of Figure 4.2. The operator -:1{<:§ is the loop operator, giving the effect on a signal which passes successively through the plant and the controller, and J == .I + :1(<:§ is the return difference operator, giving the difference in effect between no traverse and one traverse of the loop. If a disturbance d is superimposed on the control u as it is fed into the plant then u satisfies j u = d. Let J(s) be the continuous-time transfer function of J. The return difference equation is a version of the inequality ) QJ > Q, which holds for LQ-optimal control, at least under state-assumptions, if Q is the penalty matrix for control. It expresses the fact that the operation J- 1 attenuates at all frequencies. The relation is very easily proved from the canonical factorisation of cJ> with factors of the known form (17). If the column vector with subvectors x, u and .X is written!:::., let the solution of¢!:::. = 0 in terms ofu be written!:::. = Du. Then the return difference equation amounts simply to

D(f;H- 1¢D = Dcl>D.

(20)

Show that in the special case (19) relation (20) implies )QJ = Q + lJA.- 1RA- 1B, where J =I- Q- 1BJ rPxxA- 1B. Show that in the case cJ>(s) =

R(s) [

0

A(s)

0 Q(s) B(s)

~]

B(s) , 0

special only in that S(s) has been normalised to zero, relation (20) amounts to

)Q.J= Q+BA-1RA-1B, where J = Q:;- 1 {[rPuu(.~)- B.A:;- 1¢xu]- [rPux- B.A:;- 1¢xx]A- 1B}. In both cases J is indeed the return difference transfer function for the optimal control.

CHAPTER 20

Optimal Stationary LQG Policies: Imperfect Observation 1 THE PROCESS/OBSERVATION MODEL: APPEAL TO CERTAINTY EQUIVALENCE

We assume the linear model of equation (19.1) together with an observation relation: dx

+ P.lu = d + f

(1)

y+~x = 11·

(2)

This specification covers both the discrete- and continuous-time formulations; the time variable has not been explicitly indicated. The operator ~ is, like .rJ and dl, a causal translation-invariant linear operator. Let us initially discuss the discrete-time case withpth-order dynamics, with~ having the form C(ff) = 2:::=! C,ff'. As ever, dis a deterministic disturbance term and f and 11 are plant and observation noise respectively. We suppose that these noise terms jointly constitute Gaussian white noise with zero mean and covariance matrix

In this case of imperfect observation the information W, available at time consists of the observation and control histories Y1 and U1-1, plus the complete course of the deterministic component of disturbance {d1}. We shall ultimately be passing to the stationary regime, in which the past as well as the future is infinite, so that observation and control histories extend into the infinite past. Since the model is totally LQG we can appeal to the certainty equivalence principle of Section U.3 to deduce the optimal control in this imperfectly observed case from that for perfect observation. We know from the analysis of Section 19.2 that, in the case of perfect observation, the optimal stationary determination of u1 is given in closed-loop form by

rPux(ff)x,

+ rPuu(fl)ut =

---1

[¢or/J(f/)

LA·

(3)

358

OPTIMAL STATIONARY LQG POLICIES

The left-hand member of this relation gives the feedback component of optimal control, expressing u, in terms of x., (t- p < r::::;; t) and u.,. (t- p < r < t). The right-hand member gives the feedforward term, in terms of d., (r ~ t). In the case of imperfect observation recursion (3) for the optimal control still holds, except that x 7 must be replaced, where it occurs, by the current projection estimate x~l. We are then led to the inference problem; the determination of these estimates. The duality of estimation and control has already been demonstrated for the Markov case in Section 12.9; we shall see how this extends to dynamics of general order.

2 PROCESS ESI1MATION IN TIME-INTEGRAL FORM (DISCRETE TIME) The characterisation that we shall take of projection estimates is not the linear least-square property asserted in (ii) of Theorem 12.6.3, but rather the dual probability-maximising (or discrepancy-minimising) property asserted in (iii) of the same theorem. It is this characterisation which yields the natural timeintegral formulation. A related point is that canonical factorisations are then factorisations of something like the reciprocal of an AGF rather than (as in the LLS approach associated with the Wiener filter) a factorisation of an AGF. This means that, in the case of pth-order dynamics, the factors are polynomials of degreep. We can set up the problem rigorously under the supposition that observation began at a finite time h1. and can then pass to the infinite-history limit h 1 - -oo, just as we passed to the infinite-horizon limit for control optimisation. Histories will then be restricted histories, so that X, is {xr; h1 - p::::;; T::::;; t}, etc. Then the negative exponent in the Gaussian density of X, and Y, for prescribed U,_ 1 is the discrepancy

10 1 =priorterms+&t[E]T[~ r=ht

17

T

Z]-I[E], 17 T

Where E and 17 are expressed in terms of x, y and u by appeal to the plant and observation relations (1) and (2). The 'prior terms' reflect the distribution assumed for relevant system history before time h1. In the case ofpth-order dynamics they will constitute a quadratic function of {x..,yr, u.,.; h1 - p ::::;; r < h1 }. The projection estimates x~l are just the values of x 7 minimising [) 1. Estimation thus amounts to a backward rather than a forward optimisation problem; a timeintegral (or a precursor to one) is to be extremised over its course before the current moment t rather than after. Actually, we would not regard extremisation of 10 1 as extremisation of a timeintegral, because it is subject to the constraints implied by the plant and observation relations (1) and (2). Let us eliminate these by the introduction of

2 PROCESS ESTIMATION IN TIME-INTEGRAL FORM

359

Lagrangian multiplier vectors 1,. and m,. for the constraints constituted by the relations at time T and so extremise a Lagrangian form !Dr+ LW(dx + P-lu- d- t:) + mT(y + ~x -17)],. T

with Dr expressed in terms ofthe noise variables as above. Minimisation of this form with respect to the noise variables reduces the problem to extremisation of the past path integral r

llp(l,m,x) =(prior terms)+ L[v(l,m) -IT(dx + P-lu- d)- mT(y + ~x)]..,., -r=ht

(4) Here

v(l,m)

=

![ ~ r[~

~ ][~]

is the informational analogue of c(x, u). Strictly, we should should give the multipliers I and m superscripts (t), to indicate that they are the multipliers associated with optimisation on the basis of observables at time t. The relation between the multipliers and the noise estimates is

The direct interpretation of the multipliers is the familiar one of sensitivities: of /~r) and m~l as respectively the rates of change of the minimal value of lOr (subject to constraints (1) and (2)) with changes in the actual values of E,. and 7],.. Note one effect of the transformation from lOr to Up; the expression for the timeintegral has been cleared of all inverses of covariance matrices. Thus one puzzling and seemingly perverse feature associated with a stochastic formulation is removed: that it actually seems to become anomalous if some noise components are zero, and so the noise covariance matrix is singular. One cannot directly relate the multipliers l, and .AT associated with the plant equation at past and future times. This is because the past optimisation takes precedence: one first minimises !D with respect to unobservables and then C with respect to undetermined controls. This is again a reflection of the degenerate character of the risk-neutral model. When we come to the risk-sensitive formulation in the next chapter we shall see that effectively there is a single timeintegral which spans both past and future, with the consequence that past and future multipliers can be related. The estimates x~r) of process history at time t are obtained by minimising lOr with respect to the corresponding x,.. The path integral llp must then be

360

OPTIMAL STATIONARY LQG POLICIES

maximised with respect to these variables and minimised with respect to corresponding/, m. We deduce then the stationarity conditions

[ ~.il ~ S:] [~~] ~

0

X

(tJ =

[d -_:u] 0

T

(rJ

(hi
(5)

T

the estimation analogue of the set of control optimisation relations (19.4). Just as we rewrote (19.4) in the condensed form (19.5), so we rewrite (5) as

(6) say. The terminal conditions for this equation set are effectively l(r) T

=0

1

m(tl T

=0

(T

> t)

(7)

1

since terms for T > t do not appear in Dp. The initial conditions are provided by the stationarity conditions for r ~ h1. which deviate from the pattern (5) in that they will involve the prior distribution. Note that relations (7) imply that (6) has a solution independent of the values of x~r) for r > t, because x~l simply does not occur in the equation system (5) for r > t. We shall refer to this feature as the lack of forward coupling. It reflects the fact that the estimates of present and past variables do not depend upon the estimates of future variables. One can contrast it with the essential presence of backward coupling in the corresponding control relation (19.5); present control is undoubtedly expressed in terms of present and past values of the process variable. Relation (6) plus its modified version at earlier time points determines in principle all past estimates x~tl. We wish to solve these equations efficiently, but we really only need to solve them for the estimates x~l ( t - p < T ~ t) which are needed for implementation of the control rule. In the infinite-history limit this point is again met by appeal to a canonical factorisation; this time of w(z). In the infinite-history limit h 1 ---. -oo equations (5) and (6) hold for all T ~ t. If we assume that w(z) has a canonical factorisation w(z) = 1f;(z)'f/J011/J(z),

(8)

which we have taken in the normalised form analogous to (19.6), then the system (6) can be semi-inverted to

(r

~

t).

(9)

Here the operator '1/J( §') acts into the future, with an effective boundary condition x~l = 0 (T > t) implied by the property of lack of forward coupling. The oEerator '1/J( §') -t acts into the past. Relation (9) for r = t determines x~r) (and so x/l) explicitly. The relations forT= t- 1, t- 2 ... then determine the values of (t) {t) . X1_ 11 X1_ 2 , ... recurs1vely.

3 THE PARALLEL KALMAN FILTER (DISCRETE TIME)

361

With this one would seem to have estimates of the process variable at the relevant times in the form one would wish. This is not quite true, however. Rather than an expression for x~'l in the form of an infmite sum involving all past observations (which is what (9) yields if we set r = t) we would wish to generate this estimate by some simple updating recursion, as was achieved by the Kalman fllter in the state-structured case. The question is, then, whether the backward recursion with respect to r implied by (9) can be converted into a natural forward recursion with respect tot: the pth-order equivalent of the Kalman ftlter. We shall see in succeeding sections that this conversion is almost immediate. Note one necessary difference between factorisations (19.6) and (8): the order of factors is now reversed, in that the factor 1/J(z) = Ef=o 1/J1zl, for which both 1/J(z) and its matrix inverse have expansions in non-negative powers of z valid in izl ~ 1, is now the initial factor. Otherwise the factorisation is the complete analogue of (19.6) (e.g., in that 1/Jo is the constant term in 1/J(z) and is symmetric) and can be achieved by the same policy-improvement algorithm. Of course, the improvement is in inference rule rather than control rule. The existence of a canonical factorisation (8) is again the essential condition that infinite-history limits should exist for the estimation rule and the distribution of estimation errors. Exercises and comments (1) For the Markov cased= I - A:!i, f!l = -B:!i and~= -C:!ithenormalised canonical factor 1/J(z) of\ll(z) has the form 1/J(z) =

N+AVAT LT + CVAT [ I

L+AVCT M + VCVT 0

I-Azl -Cz 0

where Vis the limit value of the covariance matrix of the estimation error .X - x (cf. Exercise 19.2.1).

3 THE PARALLEL KALMAN FILTER (DISCRETE TIME) The innovation in the observations now has the form I" .,, - y 1 - y 1(1-1)

-

y, + fix,(t-1) .

( 10)

Here the time-translation operator acts, as ever, only on the subscript, so that

fix~•- I) = E~=l C,x~~~l). We assume the normalised form (8) of the canonical factorisation. Theorem 20.3.1 (The parallel Kalman filter) dated by the relations

The estimates x~l (r ~ t) are up-

362

OPTIMAL STATIONARY LQG POLICIES u ( + ( .9/- I) XI(1-1) + ::.t~Ut ~ dI+ ilQ l1

(t)

(II)

1111

X1

(t) _ x(t-1) , XT T T

H 1-T ( I

(T < t),

(12)

where ( 1 is the innovation (10) and the matrix coefficients Hj are determined by 00

H(z)

:=

LHJz-i = -[1P(z) -llxm·

(13)

j=O

Equation (ll) is recognisable as a generalised form of the Kalman filter; it must now be supplemented by the relations (12) which update also the estimates of the lagged x-values. The real novelty of the theorem lies in the compact evaluation (13) of the coefficients H1 in terms of the canonical factor 1P- The xm factor indicates that we extract the corresponding submatrix from the (j, m, x)partitioned matrix. We term relation (ll) plus relations (12) for 0 < j < p the parallel filter because it simultaneously updates estimates of the all the components of what would in fact be a state vector. It contrasts with another possible version of the Kalman filter, the serial filter, which emerges in the next section.

Proof Note the crucial difference between relations (19.5) and (6) on which we have already commented Relations (19.5) couple back into the past in that they involve Xr forT < t. However, relations (6) do not couple forward into the future; future values of x are not involved and future values of I and m are zero, as asserted in (7). This absence of forward coupling implies an effective boundary condition

(r > t).

(14)

We deduce from (6) that w(.:t)(x~lx~-ll)

= P:-1 _ P~-~

(r

~

t).

( 15)

But the right-hand member of (15) is equal to zero for T < t and to the column vector 1 1 with partition (0, -(1, 0) for T = t. Premultiplying relation (15) by the operator 1P(.:7) - 11Po we thus deduce that

(r

~

t).

This, together with the effective boundary condition (14), implies that

Xr(1)- Xr(1-1)- -h I-TTI -- h1-T

[~] ~

(r

~

t),

(16)

where h1 is the coefficient of z-1 in the expansion of -1P(z) -I in non-positive powers of z. This demonstrates the validity of (12) for T ~ t with the evaluation

4 THE SERIAL KALMAN FILTER (DISCRETE TIME)

3()3

(13) of the coefficients /lj. For the particular case T = t we deduce an expression for x~t-I) from the relation .91x1(t-1)

+ fflu = d1, 1

so deducing (ll) from (12) for the case T = t.

D

Exercises and comments (1) Consider again the Markov case, for which the canonical factor 1jJ is given in 1 Exercise 1 of Section 2. The xm submatrix of -1/J(zf is

H(z) = H +z- 1 V(l.- nTz- 1)- 1cT(M + cvcT)- 1

(17)

whereH = (L+AVCT)(M + CVCTf 1 and S1 =A- HC. ThenHo indeed has the standard evaluation H. The relations (12) for T < t are of only academic interest in this case, but we see from (17) that llj = V[nTy- 1 cT(M + CVCT)- 1 for j > 0. Part of the reason why this formula does not hold at j = 0 is th.at observation of y, gives some information on the value of f 1 if L =/:- 0 (i.e. if plant and observation noise are correlated~ This helps in the estimation of x, but not of earlier x-values. 4 THE SERIAL KALMAN FILTER (DISCRETE TIME) Direct operations on the relation (6) yield an interesting alternative form of the higher-order Kalman filter. Let us define Xt as '¢(ff) -I p, , i.e as the solution of (18)

1/J(ff)x = P·

Theorem 20.4.1 The vector (19)

can be identified with x~'l. In particular, x1 can be identified with x~ 1 l, the estimate of current process variable. Proof It follows from (6) and (18) that '¢"0 1 1/J(ff)x~'l

=

Xr

(T

~

t)

(20)

But, because of the lack of forward couplin~ and the fact that 1/Jo is the absolute term in 1/J, relation (20) forT = t reduces to x/l = Xt· D

364

OPTIMAL STATIONARY LQG POUCIES

One might define the serial estimate of process history at timet as {xr; T ~ t},, and the updated or revised estimate as {x~lT ~ t}. Correspondingly, the best 1) =- L::= C,x\~~ 1 l, whereas the serial predictor of y, at time t- 1 is 1 predictor is L::=l C,x,_,. Corresponding to the notion of the innovation (, = y 1 - y\ - 1) is then that of the serial innovation

l-

f_,= -

s

(,=

8

y,- y,= y, + ~x,.

(21)

By the same argument as that which proved the special form (19.12) for the canonical factor¢ of 41 we deduce that the canonical factor 1/J has the form

1/J11 1/Jtm 1/J = [ 1/Jml 1/Jmm I 0

(22)

where an argument§' or z is understood.

Theorem 20.4.2 (The serial Kalman filter) forward pair ofrecursions

The variables x and mobey the stable

dx + P.lu = d + 1/Jim(ff)m 1/Jmm(ff)m = y + ~x

(23) (24)

so that xis determined recursively by the serial Kalmanfdter

dx + fJiu = d + 1/J~m(ff)'r/Jmm(ff)- 1 (,

(25)

itselfa stableforward recursion. Proof Written in full, relation (18) becomes

[t:, t:o I

~] [~] o

x'T

0

=[

-_:u] o 'T

(26)

From this it follows that 1= 0 and that the equation system reduces to (23), (24). s This reduced system implies the determination m= 1/J;;.~ ( and the Kalman filter recursion (25) for X. Stability of all relations as forward recursions is guaranteed by the canonical character of 1/J. 0 Relation (25) has indeed the character of the classical Kalman filter, in that it is equivalent to the driving of a plant model by the serial innovations, or of a plant/ observation model by the observations. The parallel fJ..lter of the last section has rather the character of the driving of a state-reduced plant model by the innovations; the serial filter avoids such a reduction. However, the fact that the s serial filter works on serial innovations means that the driving term 1/Jtm..P~ (is

365

5 INVALIDITY OF THE SERIAL FILTER s

s

:in general not a function of current ( alone, but also of past (. Some ;compensation is required to take account of the fact that history has been 'estimated serially. The best way to view relation (25) is in fact to revert to the ·equation-pair (23), (24) and regard this as a coupled system of plant/observation .!Ilodel and compensator driven by the observations. Of course, the reason why l1 = 1Jtl is zero for all t is that the plant equation at t constitutes no essential constraint; the variable x 1 appears in this relation alone and can be given the value which satisfies it best, without affecting the estimates of earlier history or their fit. The following result clarifies the character of m1 and relates the two innovations. Theorem 20.4.3

The serial and parallel innovations are related by s

1/JQ".!v.,( = m= 1/J;;,~ ( .

(27)

Proof Relation (16) at r = t plus the lack of forward coupling implies that

1/Jo(x~'l - x~t-l)) =It· Because 1/J has the form (22) this last relation implies that lf'l -1(1-l) = 0, which we know, and also that 1/Jornm(m~ 1 ) - m~t-l)) = (1•

Since m~r-l) from(24).

=0

the first equality of (27) thus follows; the second we know

D

Finally, summation of the equation x~") - x~a-l) over the ranger < a ~ t leads to the conclusion

= Ha-r(a with respect to

a

Theorem 20.4.4 The updated estimates of past process values are obtained from the serial estimates .X by the formula 1-T

x~1 )

= x.,. + L

j=l

l-1"

Hj(r+J

= X.,. + L

Hj1/!0mmmr+J

(r

~

t).

(28)

j=l

5 THE CONTINUOUS-TIME CASE; INVALIDITY OF THE SERIAL FILTER Interesting points arise in the continuous-time case. The transfer from the discrete-time case can not be taken mechanically, and some points are delicate enough to affect implementation. When it comes to estimation then relation (5), written again as (6~ still holds, with the boundary condition of lack of forward coupling. We appeal to a canonical factorisation (29) However, it is now sometimes advantageous to vary the normalisation from that indicated in Theorem 18.7.2. In the next section we shall demonstrate that a

366

OPTIMAL STATIONARY LQG POLICIES

factorisation can be found of the form (22); see (35). It will then also follow that, if we again define x(t) as the solution of(l8), then x(t) can again be identified with the estimate of current process value x('l(t). Furthermore, it follows from the form of 1jJ that and obey the analogues of (23), (24)

x

m

Sllx + f!lu = d + 1/Jtm(51J)fn 1/Jmm(51J)fn = Y +~X,

(30)

so that x again obeys the serial Kalman filter relation, analogous to (25),

dx + ~u = d + 1/Jtm(fJ)t/Jmm(!!Jr' (

s

.

(31)

Here (is again the serial innovation. The true and serial innovations now have the definitions s

(=y+~x

(32)

respective!~

Here we have used xr( t) to denote the rth differential fJr x( t) of x at t and Xr (t) to denote its projection estimate on information at t: (33)

Note that 5lJ acts, as ever, on the running time argument r in (33). However, relations (31) and (32) have only a formal validity. The innovation ( is not differentiable with respect to time ~since it has a white-noise component). Neither in fact is the serial innovation (, and the fact that equation (32) relates s differentials of to differentials of ( is an indication that some of the differentials of xdo not exist either. Equations (30) are proper in that they define a stable filter with input and well-defined outputs and in. However, the equations themselves constitute an improper realisation of this filter, in that they express relations between signals (variables) which are so ill-defined mathematically as to be hopelessly ill-conditioned physicall~ · In order to obtain a set of updating relations in which all variables are well defined we have to revert to the parallel filter and so, effectively, to a state reduction of the model. This goes rather against our programme, one of whose elements was the refusal to resort to state reductions. However, the reduction is for purposes of proof only, and we shall see that the parallel filter, generating all the relevant estimates Xr, can be derived directly from the form of the canonical factor 1/J.

x

y

x

6 THE PARALLEL KALMAN FILTER (CONTINUOUS TIME) The continuous-time case has shown itselfto differ from the discrete-time case in that the serial Kalman filter cannot be implemented as it stands. Turning then to

6 THE PARALLEL KALMAN FILTER (CONTINUOUS TIME)

367

the parallel filter, we see that there is necessarily another difference. Whereas one X at any lag (either in discrete or COntinUOUS tiJne) one cannot consider the estimate of differentials of x of any order, because differentials of components of x will in general exist only up to order one less than the maximal order occurring in the plant equation. (Recall that the highestorder derivative has in general a white-noise component) This is of course acceptable, in that these are the only estimates of differentials that one needs for purposes of control. However, if one restricts oneself to estimating just these differentials, then one has effectively reverted to a state-reduced treatment, which is rather against the spirit of our programme. We shall in fact appeal to a state-reduction for purposes of proof, but the final conclusions do not appeal to such a reduction. That is, the continuous-time analogue of the matrices Hi occurring in the parallel filter analogous to (11), (12) will be deduced directly from the canonical factor 1/J of 'It. Let us for the moment omit the input terms d and f!Ju in the plant equation; these can be left out and later restored without affecting estimation. In the Markov case (with d(s) = si- A, ~(s) = -C for constant matrices A and C) we find canonical factors

can well consider the estimation of

1/J(s) =

v [ I0

L

+veT si-

A] ,

M

-C

0

0

(34)

where Vis the stationary covariance matrix of the estimation error x- x. We shall see that this generalises to

A]

[0 0 I]

1/Jii 1/Jim 1/J = 1/Jmi 1/Jmm C , 1/J. = 0 M 0 [ I 0 0 I 0 0

(35)

where 1/J and its elements have arguments s or 9J as appropriate. Let us initially suppose that the dynamics are of order p exactly, so that the matrix coefficient Ap of f!)P in .s;l is non-singular, and can be normalised to the identity. The Kalman filter for the state-reduced model, which gives the parallel updating for the unreduced model, then has the form (analogous to (11), (12)),

(0::;,. r
(36)

p-1

Eexp-i + LA,x, =Hp-J(

(37)

r=O

where ( is the innovation, expressed more explicitly in (32), (33). This constitutes the parallel filter. It is convenient to define the matrices G, = MH, and the generating function

368

OPTIMAL STATIONARY LQG POLICIES

p-1

G(s) =

L G,s-'- 1.

(38).

r=O

The problem is then to determine the coefficients H, in (36), (37) directly from the canonical factor 1/J of (35), without reversion to a state-reduced treatment. This is resolved as follows.

Theorem 20.6.1 The coefficients Hj = GjM- 1 in the parallel Kalman filter (36), (3 7) are determined in terms ofthe factor 1/J ofthe canonicalfactorisation (35) by the relation (39) 1/Jim(s) = [A(s)G(s)]+ where the truncation operator [ ]+ retains only the terms in non-negative powers ofs. This implies an equation system p

'1/Jrlm =

L

(r=O,l, ... ,p-1)

AkGk-r-1

(40)

k=r+1 which determines Gp_ 1, Gp_ 2 , ••• recursively. Suppose dynamics are of variable order, in that rj is the order of the highest derivative of the jth component of x which occurs in the plant equation. Then the jkth element of '1/Jrlm is zero for r ;?: rj and we may assume the same of G,. Relations (40) then again determine the components of the matrices G, recursively from the highest order downwards. Proof The essential conclusion, expressed in (39), is as simple as one could wish. However, the proof is somewhat more extended. Consider first the Markov case and define the vector as the solution of '1/J(ff)x = p, with the canonical factors given by (34). That is,

x

[~

L+fT

~-t][~l] ~

[7]

It follows then immediately, as in Section 4, that l = 0, that filter relation

xobeys the Kalman

~x

=Ax +H(y- Cx), and that m= M- 1(y- Cx) = M- 1(. Further, the relation if;x('l(r) = '1/J.xr now amounts to

v LT +CV

[ -~/-A T

J][-l](t) [X']

0 M 0 -CT 0

-m X

T

-Mm 0

(r :E; t),

(41)

6 THE PARALLEL KALMAN FILTER (CONTINUOUS TIME)

36~

'~ence, since /(ll(t) =

0, it indeed follows that we can make the identification i(t) = x('l (t) and also m(ll (t) = m(t). Now consider the model withpth-order dynamics exactly, and consider its state reduction, using the notation of the Markov model with a distinguishing tilde on variables and coefficients. The state vector x will be a column vector with rth subcompo_!lent x, = _q}'x(O ~ r < p). The analogue of relation (41) then implies again that I = 0 and that mand the x, satisfy

-G,m + _qJ.X,- .X,+ I = 0

(0

~ r

< p- 1)

(42)

p-1

-Gp-1m + _q}xp-1

+L

A,x,

=o

(43)

1=0

p-1

-Mm+ L c,x, = -y

(44)

r=O

where the vertically partitioned matrix G = (G,) is the analogue of L + VCT. These relations imply the parallel filter relations (36), (37) with H, = G,M- 1 , and we know from consideration of the state-structured case that .X, indeed has the interpretation (33); it is the projection estimate of the current value of !!J'x. It remains then to determine G, but from the canonical factorisation rather than by actual calculation of the analogue of L + veT. If we reduce equations (42)-(44) to (30) by elimination of .X, (0 < r < p) and identification of .X0 with x then we can identify the resulting operator coefficients of with 1/Jim and 7/Jmm· Solution of (42) yields the expression

m

.x, =

r-1

_qJ'x-

L Gk_qJr-k-l,n = _qJ'x- [_qJ'G(_qJ)]+m

(45)

k=O

where the operator [ ]+ annihilates all terms in negative powers of !!J (or s, as appropriate) in the bracket. Substitution of this expression into (43) and (44) yields relations (30) with the identifications

1/Jmm(s)

= M + [~(S)G(s)k

In particular, we have the key assertion (39), which implies the linear determining relations (40). A continuation ofthis argument implies the asserted form (35) of the canonical factorisation. The final assertion of the theorem follows by adaptation of the argument given to the case of variable order. 0

CHAPTER 21

The Risk-sensitive (LEQG) Version The time-integral characterisation of control optimisation under LQG assumptions seems to reach its completion first when one incorporates risk-sensitivity by generalising to an LEQG characterisation. This is because the optimisations of control and estimation are then combined in the extremisation of a single timeintegral, representing the stress. Furthermore, the matrices of operators iP and \II generalise in so pleasing a fashion that they make the LQG model appear indeed as a degenerate case which finds its place first when it is imbedded in the class of LEQG models. With so pleasing a completion it is then unexpected that there is one step whose natural generalisation proves elusive: the final recoupling step of Section 16.7. We discuss the point in Sections 2 and 3.

1 DEDUCTION OF THE TIME-INTEGRAL Assume again the process and observation relations

.Iiix + 11u = d + f

(1)

y+~x = 1J

with jointly white plant and observation noise {f, ry). In discrete time we assume the noise covariance matrix (12.3) and instantaneous cost function (124). In continuous time we assume these same expression with a rate interpretation. The structural points which arise with the introduction of risk-sensitivity are common to both versions, so we shall keep to discrete time for definiteness. We assume the risk-sensitive criterion E.,.(e-9C), and know then from the risksensitive certainty equivalence principle of Section 16.2 that the optimal value of the control u1 is determined by extremisation of the stress

§=C+0- 10 with respect to all current unobservables and all currently undetermined controls (at time t). Theorems 16.2.1 and 16.2.2 state more explicitly the sense in which this extremisation is to be understood. Under the assumptions above we can then write the stress as §

=

L(c.,. + o- 1D.,.) +end terms 1'

372

THE RISK-SENSITIVE (LEQG) VERSION

where

D1"

=l["]T[NLT ML]2 TJ

1"

1 ["]

TJ

1"

and t:, TJ are to be expressed in terms of x, y and u by (1). As emphasised in Chapters 12 and 16, the effect of the certainty-equivalence principle is to reduce constrained minimisations to unconstrained minimisations. We still have the constraints of the plant and observation relations, however, which we reduce, as in Sections 6.5 and 12.9, by the introduction of Lagrange multipliers. We thus consider the Lagrangian form

{2) 1"

If we extremise this with respect to the noise variables then we are left with the integral U= L:!c(x, u) + ).T(Jilx + rJiu- d)+ J.lT(y + Cx)- 8v()., J.l)J.,. +end terms 1"

(3) where the cost and 'dual cost' terms c{x, u) and v()., J.l) have the expressions

-l[X]T [Rs ST] [X]u ./ u Q

c.,.-

2

1"

-1[/]T[N L][/]m .,. m .,. LT M

v.,. -

2

(4)

Expression (3) constitutes our time-integral, in that it can extremised freely with respect to every variable except those whose values are currently known. So, if one is determining the optimal value of u, then the control and observation histories u,_ 1 and Yr are known, as is also the whole course {d,} of the deterministic disturbance. All other x, y, u, ). and J.l values are then to be extremised out. Whatever the nature of the extreme, it will be attained at a stationary point if the optimisation problem is properly posed. Extremising values calculated subject to specification of observables at time will be given the superscript (t); they are to be regarded as 'minimal stress' determinations ofthe quantity in question conditional on information at time t. (A 'determination' can then have aspects of both estimation and optimisation, and 'minimal stress' is really 'extremal stress', with the nature of the extremum depending on the sign of 9.) The dual variables ). and /-1. have the physical meaning of 'sensitivities': rates of change of minimal stress with respect to changes in the value of plant and observation noise. Their extremising values are related to effective estimates of these noise inputs by

(5) as we see from the values oft: and TJWhich extremise expression (2).

2 THE STATIONARITY CONDITIONS

373

Note that the time-integral (3) is a linear combination of the time-integrals (19.3) and (20.4) arising in the separate considerations of control and estimation for the LQG case. In the LEQG treatment these combine naturally. Note also that the passage from (2) to (3) clears all matrix inverses from the integral.

2 THE STATIONARITY CONDITIONS Extremising expression (3) with respect to all disposable variables, subject to specification of the observables at timet, we deduce the two sets of equations

(6)

l[-0>..]

,s:;1 ~

-01-L

-OR

X

(r)

(7)

T

To these can be added the stationarity conditions with respect to Yr and f..L-r for T > t. The first of these yields the important relation f..L-r = 0 (T > t), saying essentially that the observation relation is longer a constraint for T > t (since the estimate of a future observation will be chosen to satisfy the relation exactly). The second gives a predictive relation for y., (i.e. for what the values of future observations would be) which is usually of little interest. The two equation systems (6) and (7) plainly generalise equations (19.4) and (20.5). They indeed seem to constitute the natural completion of these previous versions in that they are now linked by common variables, and the empty entry in the bottom right-hand corner of the two matrices of operators has now been filled; in the one case by -ON and in the other by -OR. These entries bring about the effect which we have already remarked in Chapter 16: that in the risk-sensitive case control is affected by noise properties and estimation is affected by costpressures. The systems are complete for given t if we add to them the conditions

f..L~)

= 0 (T > t);

y., specified

(T

~

t);

u., specified (T < t).

(8)

As in Sections 18.1 and 19.2 we write the two equation systems (6) and (7) as 4>(.r)~~r) = (~')

w(.r)x~l = p~l with the revised definitions of these quantities implied in (6), (7). Use of the canonical factorisations again reduces these equations almost to the required degree. For example, 4>(z) will again have a canonical factorisation of

374

THE RISK-SENSITIVE (LEQG) VERSION

1

the form (19.6) with ¢o symmetric, and the expression (19.12) of the canonical' factor 4> generalises to :

(z) =

xx ux

[ .91

xu 4>uu PJ

I 0

l

-ON

where the submatrices are also functions of z. The optimisation of control in the case of perfect observation differs only from that of the risk-neutral case by the substitution of this revised form of the canonical factor, and the optimal control rule is then again of the form (19.13). However, when observation is imperfect then solution of the equation pair (6), (7) presents a problem which was not encountered in the risk-neutral case. The two sets of equations are coupled in both directions, forward as well as back. The 1 system (6) is now linked to the past by the occurrence of J.L~ ) in the right-hand member at T = taswell as the occurrence of terms in x~1 ) forT< t. However, the real difference lies in the system (7), which was not previously linked to the future, but now is so by the fact that >.~l is non-zero forT ~ t and also by the occurrence of u1 in the right-hand member at T = t. Risk-sensitivity has the effect that estimates based on information acquired from the past are also affected by costs to be incurred in the future. In Whittle (1990a) this point was dealt with in a distinctly unsatisfactory fashion. Appeal to the canonical factorisations of 4>(z) and 'IT(z) reduced the two infinite equation systems (6), (7) to a single system of 2p vector equations, these equations being related to the determination of the estimates of xT(t- p < T:::;:; t) as the values minimising the sum of past and future stress. This reduced system is readily solved in the state-structured case p = 1, solution corresponding simply to the recoupling step of Theorem 16.7.1. However, the form of solution in higher-order cases was not evident, and there the matter was left. One might think that, since plant noise E appears as an effective auxiliary control, one might simply revert to a risk-neutral formulation in which the control u is replaced by the pair (u,E). This view fails in the case of imperfect observation, however, because the estimates fVl for T < t are formed retrospectively. That is, if we regard E as the control wielded by Nature, then Nature has the unfair advantage that she can revise past controls in her own favour. We indicate an alternative approach in the next section which yields an explicit solution in terms of a canonical factorisation, but at the cost that the function being factorised is rational rather than polynomial (for dynamics of finite order). The analysis is a pleasing and interesting one, in that it demonstrates the efficacy of the innovation concept. However, we are left, for the moment, with the conclusion that one might as well revert to a state-reduction when it comes to implementation.

375

3 A GENERAL FORMALISM

~es and comments

"(p We revert to the point raised in Section 16.10:

the non-obviousness of the :·equivalence of expressions (16.45) and (16.47) for the risk-sensitive average cost in the state-structured case, under a stabilising policy u = Kx. We may as well then normalise the model to an uncontrolled one with a stable plant equati()n x 1 = Axr-l + ft and an instantaneous cost function xT Rx. The dynamicprogramming evaluation of average cost is 'Y = ( 1/29) log II + 9NTII where II is the solution of

!

(9) (see Theorem 16.10.1). The general evaluation f_!'om Theorem 13.5.2 is 1 = (1/29) Abs log IP(z)l where P(z) = I+ 9Rsr 1N .r;~- 1 and.r;l =I- Az. If we take the time-integral approach to evaluation of the predicted path for the process then the value of ci> corresponding to (18.4) is ci> = [

~

_;N] .

We find, with some matrix manipulation, that lei>!= I.J1i'lldiiPI so that Abs log IPI = Abs log! ci> 1. But we know from the treatment of the Markov case in Section 18.2 that ci>(z) = G_,z- 1 +Go+ G,z has the canonical factorisation (18.14) where ll has a value Q satisfying Q =Go- G_,J;!- 1G1. This equation implies in the present case that ·

Q=

[~ -~N]

where n satisfies (9). We thus have Abs log!PI = Abs loglci>I log! I+ 9NIII, whence identity of the two evaluations follows.

= logi1JI =

3 A GENERAL FORMALISM In order to distinguish the wood from the trees it is better to revert to a more abstractly posed problem, which in fact generalises that of Chapter 18 in some respects. Suppose that we are extremising a quadratic time-integral 0, i.e. a sum over time T of quadratic functions of a vector 'system variable' Xr· Suppose that the column vector Xr can be partitioned (en TJr ), where the two components are distinguished by the fact that is never observed, but that at time t the second component '1r has been observed for r :::::; t. Extremisation of the integral at timet is then conditioned by knowledge of past 'T/ values. In the control context above, we think of ~r as having the components (A, JL, x)r and T'fr as having the components (Yn Ur-1)· Suppose that the time-integral has the form

er

0 = Lbr(~,'T/)- a;~r- !3;Tir] +(end terms) T

THE RISK-SENSITIVE (LEQG) VERSION

376

where the sequences {a.,.} and {.B.,.} are known and 1 has the time-homogeneous quadratic form

The entries in the matrix are operators, so that r {{,for example, should be written more explicitly as r {{(ff), a power series in the backward translation operator ff. The integral 0 thus involves cross-terms between values of the system variable at different times. We suppose symmetry of the matrix r in that f' = r. As ever, we denote the values of ~r and 1Jr on the extremal path constrained by knowledge of {ry.,.; T ~ t} by ~~t) and ry~l. If x is any function of the path let us define 6_x(tl = x(tl - x(t-l); the change in the value of x on the extremising path as the information gained at time t is added to that already available at time t - 1. Theorem 21.3.1 Suppose that infinite-horizon limits exist. Then (i) If x is any linear function of the path then 6_x(t) is a matrix multiple of the 'innovation' -

(t-1)

it - Tlt - Tlt

( 11)

.

Specifically,

(12) where Ko =I and Ki = Ofor j < 0. (ii) Suppose that the canonicalfactorisation

r '7'/(z)- r 'l{(z)r{{(z)- 1r{71(z) = v(z)vo- 1v(z)

(13)

holds. Then the generatingjimctions 00

H(z)

= LH1zi, -oo

have the evaluations

(14) Proof Extremising nsubject to information at time t we deduce the linear equations

(all T)

(15)

(-r > t)

(16)

3 A GENERAL FORMALISM

377

which then imply that

+ r{17 ~ 17~l = o

(all T)

(1 7)

r 'l{~~~~l + r '7'7~ 17~l = o

(T > t)

(1 8)

r~~~~tJ

But ~77~1 ) is zero for r < t, and for r = t equals the 'innovation' (11). Assertion (i) then follows from this statement and equations (17), (18). If we form the generating functions H(z) and K(z) then it follows from equations (17) and (18) that r~{(z)H(z) r17~(zJH(z)

+ r~11 K(z)

=

0

+ r '1'1(z)K(z) = G(z)

(19)

(20)

where G(z) is a function whose expansion on the unit circle contains only nonpositive powers of z. Suppressing the z-argument for simplicity we then have

n=

-rz~'r~17 K

(21)

(r '7'7- r '7{r«'r~17 )K =G. From this last equation it follows that vo-1 v K = v--IG . But since one side of this last equation has an expansion on the unit circle in nonnegative powers and the other in non-positive powers they must both be constant, and the constant must be v01voKo = /. Thus v01vK =I, which D together with (21) implies the determinations (14) of K and H. The conclusions are attractive. However, if we suppose 'pth-order dynamics' in that the matrix r(Y) of the cost function (10) involves powers of ff only in the range [-p,p], then the expression factorised in (13) is not similarly restricted; it is a matrix of functions rational in z.

PART 5

Near-determinism and Large Deviation Theory Large deviation theory is enjoying an enormous vogue, both because of its mathematical content and the many ways in which this can be viewed, and because of the large range of applications for which it proves a natural tool. Chapter 22 gives an introduction to the topic, by an approach perhaps closer to that of the physicist than of the probabilist. However, some justification before even that introduction would not come amiss. The theory has a clear relevance to control theory, as we explain in Chapters 23 and 25. However, in some respects one needs a more refined treatment to capture some of the essential stochastic effects. We cover such refmements in Chapter 24, which could indeed be read directly, as it demands scarcely more of large deviation theory than we now sketch. Large deviation theory is a shifted version (in quite a literal sense, as we shall see) of the two basic limit assertions of elementary probability theory: the law of large numbers and the central limit theorem. The second can be regarded as a refined version of the first (in a certain sense; that of weak convergence). The two assertions have process versions: the convergence of a stochastic process to a deterministic process or to a diffusion process if it is subject to an ever-increasing internal averaging in a sense which we make explicit in Section 22.3. To see how large deviation theory extends these concepts, consider a process {x(t)} over a time-interval (0, h), and suppose the values of x(h) and x(O) prescribed. Suppose further that x( h) does not lie on the limit-deterministic path starting from x(O) (and we shall soon be more specific on the magnitude of the deviation assumed). Then one can still determine a most probable path between the prescribed end-points. Large-deviation theory demonstrates that, under

380

NEAR-DETERMINISM AND LARGE DEVIATION THEORY

appropriate assumptions, this most probable path is just the limit-deterministic path for a 'tilted' version of the process. It also provides a first approximation to the probability that x(h) takes the prescribed value, conditional on the value of x(O). This estimate is sufficient for many purposes, but can be improved if one approximates the tilted process by a diffusion rather than a deterministic process. Let us be somewhat more specific. Consider a random scalar x which is the arithmetic average of K independently and identically distributed scalar random variables ~j ( j = 1, 2, ... , K) of mean J1. and variance a2. (A symbol such as Nor n would be more conventional than K, but these are already in full use.) Then x has mean J1. and variance a2 JK, and converges to J1. with increasing K in almost any stochastic sense one cares to name (the 'law of large numbers', in its various versions). In particular, for sufficiently regular functions C(x) one has

E"[C(x)]

=

C(Jl.)

+ o(l).

(1)

for large K. Here we have given the expectation operator a subscript K to indicate that the distribution of x depends upon this parameter. A strengthening of (1) would be

(2) which makes clear that the remainder term in (1) may be in fact O(K- 1) rather than anything weaker. One obtains stronger conclusions if one allows the function under the expectation to depend upon K as well as upon x. For example, the central limit theorem amounts to the assertion (again for sufficiently regular C) that E~< { C[(x- Jl.)/ u-./K.]} = E[ C(77)]

where 1] is a standard normal variable. The large deviation assertion is that the D(x), known as the rate function, such that

+ o( 1)

~-distribution

(3)

determines a function

(4) (The precise result is Cramer's theorem, which also evaluates the rate function; see Section 22.2.) The interest is that the function under the expectation is exponential in K, and it is this which forces the value of x contributing most to the expectation away from the central value Jl.· The point of the assertion is that there is indeed such a dominant value, and that it is the value minimising C(x) + D(x). This last observation partly explains the reason for the term 'large deviation', which perhaps becomes even clearer if we consider distributions. If it is proper to allow e-c in (4) to be the indicator function of a set .91 then (4) becomes

P"(x E .91) = exp[-K inf D(x) xEJll

+ o(K)].

(5)

NEAR-DETERMINISM AND LARGE DEVIATION THEORY

3.81

In considering the event x E .!II one is considering deviations of x from J.L of order one (if J.L does not itself lie in d), whereas the probable deviations are of order 1j fo. One is then indeed considering deviations which are large relative to what is expected, and D(x) expresses the behaviour of the tails of the x-distribution well beyond the point at which the normal approximation is generally valid. Results of the type of (5) are extremely valuable for the evaluation of quantities such as the probability of transmission error in communication contexts, or the probability of system failure in reliability contexts. If we regard C(x) as a cost function then evaluation (4) seems perfectly tailored to treatment of the risk-sensitive case. For fixed (} and large K. we would have

{6) If we wrote the left-hand member as exp( -K.8F), so defining Fas a an effective cost under the risk-sensitive criterion, then we would have

F = ext[C(x) + o- 1D(x)] + o(l),

{7)

X

where 'ext' indicates an infimum or a supremum according as (J is positive or negative. This opens the intriguing possibility that the risk-sensitive treatment of control in Chapter 16 has a natural non-LQG version, at least for processes which are near-deterministic in the particular sense that large deviation theory requires. The rate function D(x) is then an asymptotic version of the discrepancy function of Section 12.1. However, the hope that all the theory based on LQG assumptions has more general validity should be qualified If we revert to the risk-neutral case (J -+ 0 then it will transpire that relation (7) becomes simply F = C(J.L) + o( 1), which is in most cases too crude to be useful-a warning that large deviation results will often be too crude. Nevertheless, they give a valuable first indication, which can be refined Further, the rate function does in a sense encapsulate the essential stochastic features of the 1]-distribution, and results such as those of the nonlinear filtering section (Section 25.5) are not at all too crude, but again encapsulate the essentials. In considering a single random variable x we have considered a static problem. As indicated above, these ideas can be generalised to the dynamic case, when one is concerned with a stochastic process {x(t)}. In this case one should see the variable K. as representing the physical (perhaps spatial) scale of a system, and x(t) as representing a physical average over the system at timet. We consider the process version in Section 224, a necessary preliminary to control applications. As we have emphasised and shall demonstrate, a large deviation evaluation such as (5) follows from the law of large numbers, and has nothing to do with normal approximation. However, in the Gaussian case, when the ~-variables are normally distributed, then the central limit theorem is exact in that x is normally

382

NEAR-DETERMINISM AND LARGE DEVIATION THEORY

distributed over its whole range. The probability evaluation (5) must then necessarily be virtually coincident with the evaluation of the normal integral (after appropriate scaling of the variable) over d. Indeed, it turns out that the rate function D(x) has evaluation (xj2a2; the familiar negative exponent in the normal density. In other words, if variables are normally distributed then large deviation theory is 'exact', in that the effective asymptotic density const. exp[-D(x)J coincides with the actual density. As a further indication of the same point: if the ry-variables are Gaussian and the function C<x") in (4) is quadratic then the remainder term in the right-hand member is constant and 0(1), in that it is equal to log[D" / ( C" + D") ], where the double-primed quantities are the second derivatives of C and D. It is for these reasons that results which we obtain as large deviation approximations coincide, under LQG assumptions, with results which we know from Chapter 16 to be exact.

Jil

!

CHAPTER 22

The Essentials ofLarge Deviation Theory 1 THE LARGE DEVIATION PROPERTY There is a gain in clarity and speed if we define the large deviation property flrst and then motivate it, although of course this inverts the order of both history and understanding. There are many ways of approaching the subject; we follow a route which is economical and which is natural for the control context. A treatment as brief as this necessarily lacks rigour at many points, but the fact that the argument is so natural will perhaps reassure the reader that it is also rigorisable. We proceed through a succession of cases: first the 'static' case of a single random variable, then the dynamic case of a stochastic process and finally, in Chapters 23-25, the case of a controlled stochastic process. We have already attempted a first orientation in the introductory section above. It is perhaps then appropriate to treat that introduction as the zeroth section of this chapter, and to refer to its equations as equations of this chapter. Consider a random vector variable x whose distribution depends upon a nonnegative parameter K. One is thus considering a family of distributions indexed by "" The corresponding probability measures and expectation operators can then be written P,. and E,. respectively, if one wishes to emphasise the dependence. The large deviation property then states, roughly, that there exists a function D(x), the rate function, such that (5) (in the preamble to Part 5) holds forlarge K. and for sufficiently regular sets d. Equivalently, at this level of rigour, (4) holds for large K. and sufficiently regular scalar functions C(x). It is relation (5) which is usually taken as the characterising property, stated more carefully as an asymptotic inequality in one direction for closed sets .!II and in the other direction for open sets. It is the expectation characterisation (4) which is more natural in our context, however, regarded as valid at least for continuous functions C obeying appropriate growth conditions. Neither property implies the other without supplementary conditions, but relation (5) would of course follow formally from (4) if one took exp[-C(x)] as the indicator function of the set d. However, such a discontinuous choice may be unacceptable, and it is because of the possible trouble associated with such discontinuities that rigour requires a more cautious formulation of assertion (5). Why one should expect versions of either of these relations to hold and under what conditions has yet to be explained. However, note the implications. The

384

THE ESSENTIALS OF LARGE DEVIATION THEORY

expression for both the probability and the expectation is exponential in,.., to first order, with the coefficient of ,.., in the exponent determined in terms of the rate function as indicated. Further, it is a single extremising value of x which contributes dominantly to both probability and expectation. In the case of P"" (x E .91) this value of x is that value in .JJI which is most probable on the basis of a kind of unnormalised density exp[ -K.D(x)J In the case of the expectation the value of x which contributes dominantly has to compromise between achieving a high value of exp(- C) and high probability. In either case, the operation of integration (with respect to a probability measure) has been replaced by that of extremisation. In fact, one seems to find the large deviation property in just one class of cases. This class is characterised by three properties. (i) The parameter,.., measures the scale of the stochastic system being considered. For example, both the capacity of and demand on a telephone network might be of order ,.., or ,.., might measure the size of an insurance company. (ii) The random variable x is an average over the system of a vector-valued random variable. Thus, it might represent the instantaneous traffic being carried per unit of capacity on the various links of the telephone network, or the current dividend which the insurance company announces. (iii) The system has the stochastic homogeneity and ergodicity properties which imply that x behaves as the average of ,.., independently and identically distributed random variables, in that

(8) for some 1/J and for all values of the row vector a: for which the left-hand member is defined. Let us note a few points of definition and notation. If ' is a vector-valued random variable then

M(o:) = E(eek{) is its moment generating function (abbreviated to MGF). Here~ is asswned to be a column vector and a: then a row vector. M(o:) certainly exists for purely imaginary a:; and will exist for other values if the tails ofthe ry-distribution decay at least exponentially fast We shall have occasion to work with the function 1/J(o:) =log M(o:), the cumulant generating function of~ (abbreviated to CGF). These functions have a number of important properties which we summarise in Appendix 3. We use the abbreviation liD for 'independently and identically distributed'. 2 THE STATIC CASE: CRAMER'S THEOREM Cramer's theorem concerns the case when relation (8) holds exactly; i.e.

E""(e"ax)

= e"1/J(o)

(9)

2 THE STATIC CASE: CRAMER'S THEOREM

385

for some 'r/J for all real o for which the left-hand side exists. This will be the case (at least for /'1, integral, a case to which we can restrict ourselves) when x is the arithmetic mean of /'1, liD vector random variables ~J with CGF ,P( o).

Theorem 22.2.1 (Cramer's theorem) Suppose that relation (9) holds for all o: in some set of real values with non-empty interior. Then x has the large deviation property and the rate function has the evaluation D(x)

= sup[ox -1/l(o)].

( 10)

(k

Cramer's original theorem asserts the large deviation property in that (5) is shown to hold if d is an interval, or indeed any countable union of intervals. We shall rather give a proof of the expectation relation (4) which makes the line of proof for case (5) clear. Note the interesting evaluation of the rate function (10): D(x) is seen to be the negative of the minimum transform (or Legendre transform) of the CGF ,P(o). This in itself raises a number of fascinating issues, which we briefly mention in Exercise 2. We shall need to appeal to some differentiability and convexity properies of the CGF, listed in the theorems of Appendix 3. The proofs, although brief, are deferred to Appendix 3 so as not to break the argument. Consider now the modification of the distribution of by tilting, so that the expectation of a function¢(~) of~ becomes rather

e

E(a) [¢(~)] = E[¢(~)ea~].

(II)

M(o)

That is, one weights the original ~-distribution by the exponential factor ecrE.. and then renormalises the distribution. This apparently arbitrary concept is in fact motivated naturally, both by the proofs of the theorems of Appendix 3 and by the use to which we shall shortly put it. We refer to o in (11) as the tilt parameter.

Theorem 22.2.2 (i) The mean of the o-tilted distribution is £(a)(~)= 1/la, the column vector offirst differentials of1/1 at c. (ii) Define the convex function ofthe column vector a (12)

D(a) = sup[oa -1/l(o)]. a

If o is chosen as the value at which the supremum is achieved in (12) then £(a) (0

= a.

(iii) Furthermore, at points where the derivatives exist, Da = o,

Daa = [1/laar

1

(13)

where a and o are the corresponding values determined by (12).

=

Proof It follows from the definition (11) that £(a)(~)= [8M(o)j8o]jM(o) 'rPa· The function D(a) defined in (12) is certainly convex, being a supremum of

386

THE ESSENTIALS OF LARGE DEVIATION THEORY

linear function of a. The supremum will be attained at the value of a determined by '1/Ja = ~ so that the tilted distribution whose parameter has this value indeed has mean a. To prove assertion (iii), write the definition (12) as

D(a) = sup G(a, a). Q

It follows then by standard arguments that Da = Ga and Daa = Gaa- GaaG-;,~Gaa, where a is given its extremising value. These relations reduce to those asserted in (13) in the particular case (12). 0 Suppose that the liD variables ~j are given the tilted distribution (11). The correspondingly tilted expectation of a function <jJ(x) of the arithmetic mean x = x:- 1 1 E,j can then be written in terms of the until ted expectation as

L;J=

E~al[¢(x)] = M(ar,.E,.[¢(x)e,.ax].

We can reverse this relationship to obtain E,.[e-1tC(x)J = E,.{e-~<[C(x)+ax-ax]} = M(a)"E~a){e-~<[C(x)+axJ} = e,.f,.,(a)-a aJ E~a) {e -~<[C(xJ+a(x-aJJ}

=

e-~
(l4)

if a is taken as the ext remising value in (12). This is the key relation for the proof of Cramer's theorem, which we now deduce in an expectation version under almost minimal conditions on C.

Theorem 22.2.3 (An expectation version of Cramer's theorem) Suppose the relation (9) holdsfor all real a for which the left-hand member is defined, these values constituting a set with non-empty interior. Suppose C(x) continuous and define D(x) by (12). Then the large deviation relation (4) holds. Proof Choose a equal to the value of x which maximises the last curly-bracketed expression in (14), i.e. which minimises C(x) +ax. Since a and a are corresponding values this then implies a corresponding variation of a, but one ends up in any case with the relation C(a)

~

C(x) + a(x- a).

It follows from the definition (12) of D that

D(a)

~

D(x) - a(x- a).

Adding these two inequalities, we see that a is indeed the value of x which minimises C(x) + D(x).

2 THE STATIC CASE: CRAMER'S THEOREM

387

Now, since the tilted expectation of the last curly bracket in (14) cannot exceed its maximal value, we see that ( 15) for this value of a. On the other hand, since C is continuous, for any prescribed positive E we can find a neighbourhood .#"(E) of a such that C(x) + o(x- a)::::; C(a) +E. We then deduce from (14) that

(16) But it follows from the law of large numbers that, under the tilted distribution, x converges weakly to its expectation value a as "' increases, and so will ultimately lie in .#"(E) with probability one for any prescribed E. That is, the probability factor in (16) tends to unity with increasing K, however small E is chosen. Relations (15) and (16) then together imply (4). D Establishment of the upper bound (15) requires very little argument; even less than we gave it (see Exercise 2). It is the lower bound (16) which appeals to deeper properties: the law of large numbers for the tilted distribution with an appropriately cunning choice of tilting parameter. If one is willing to make stronger assumptions on the behaviour of C then stronger results can be deduced; see Exercise 4. Now that the large deviation property and the rate function evaluation (10) have been established under the hypothesis (9) it is plausible that they hold under its weakened version (8). This has indeed been demonstrated and exploited by Gartner (1977) and Ellis (1984). See Exercises 5 and 6. Exercises and comments (1) Confirm the following evaluations of CGF and rate function. For a normal vector with mean J.L and covariance matrix V, 1/J( a) = aJ.L +!a V a r and D( x) = 1(x- J.Ll v-I (x- J.L); just the normal exponent. For a Poisson variable with expectation J.L,'I/;(o) = J.L(e"- 1) and D(x) = x log(x/J.L)- x+ J.L; one recognises Stirling's approximation to the factorial. For an exponential variable withmeanJ.L,'Ij;(o) = -log(l- J.La) andD(x) = (x/J.L)- 1 -log(x/J.L). (2) We have

P"(x

E

.91)::::; E"(e"a(x-a)) = e"!.P(a)-a:a].

if a and a are such that a (x - a) ~ 0 for x in .91. If we minimise the last expression above with respect to a and a subject to this constraint then we obtain Chernoff's inequality and a probability version of (15). In particular, the commonest version of Chernoff's inequality takes the form

388

THE ESSENTIALS OF LARGE DEVIATION THEORY

P,.,(x

~c)~

inf er.[,P(a)-acJ.

(17)

o;;;.O

for scalar x. Cramer's theorem demonstrates that this simple bound is surprisingly good, in that the leading term in the exponent (for large K) is correct. The distribution of a sum of l'i. liD random variables could formally be obtained by Fourier inversion of the Kth power of the characteristic function M(iO) of a summand. Cramer's theorem demonstrates that the Fourier transform can be replaced by a minimum transform, for certain purposes and for large K.. It achieves this analytic conclusion by probabilistic arguments. A feature of large deviation theory is indeed that there is often both an analytic route and a probabilistic route to conclusions--circumstances incline the investigator to one or the other, although habit or taste may in the end prevail. (3) Even the simplest form (17) of Chernoff's inequality is of great practical value. Suppose that one has an assembly of K components which each have independent probability p of failure; the system will fail if the proportion of components which fail exceeds c. Show that the probability of system failure is bounded above by [z-c(pz +I - p)t for any z ~ 1 Show that, if c ~ p, then

)1-c]"

c{lP( system failure) ~ p p [ cC{l - c) 1-c

,

an expression with classic information-theoretic significance. (4) Suppose that the vector value a which minimises C + D is unique and that C and D possess continuous second-order derivatives at a. An appeal to the central limit theorem rather than to the law of large numbers in the final expression of (14) then leads to the evaluation

E,.{ e-I
IDaal ] !Caa + Daol

112 e-r.[C(a)+D(a)]+o(l).

(18)

where all evaluations are at a. This improves evaluation (4) in that it leaves only an o( I) remainder in the exponent. (5) Occupation times for Markov chains Let x1 be the proportion of time spent in state j over a time interval of length K. for a Markov chain with transition matrix P = (pJk). Let x be the column vector of the x1. Then (8) will hold with exp["l/;(o:)] = A(o:), the maximal eigenvalue of the matrix (p1keo:J). The GartnerEllis extension of Cramer's theorem then demonstrates the large deviation property for x with rate function

D(x) = sup[o:x -log>.(o:)]. a

A very early result of this type was deduced by Miller (1961), who remarked on the surprising fact that .A(o:) need not itself be an MGF, although it shares

3 OPERATORS AND SCALING FOR MARKOV PROCESSES

389

many of the properties. An alternative expression of the rate function, also surprising, is

Note that this application corresponds to an averaging over time, and is quite distinct from our later consideration of temporal stochastic processes which show an averaging over the system. (6) Ruin ofan insurance company (Martin-Loj 1986) Suppose that an insurance company begins with capital K and that capital then develops as a process of independent increments whose increment in unit time has CGF 1/J(a). Let T be the time at which ruin occurs (i.e. when capital first runs negative). Then a classic result due to Lundberg states that E( ell.,.) =

e~
for large K. Here, if B is real, then a( B) is the lesser of the two real a-roots of B+ 1j;(a) = 0. The function 1j;(a) is convex (see Appendix 3), so that if 1/J'(O) is positive) (corresponding to a positive drift of capital) then the smaller root of 'lj;( a) = 0 is negative, at - {3, say. Cases for which ruin does not occur are understood not to contribute to the expectation, which then just reduces to the probability of ruin e-~
inf [a+ y'lj;(a)].

a~

-P

The bound e-~
390

THE ESSENTIALS OF LARGE DEVIATION THEORY

(19) has been defined. Here c and I< are respectively instantaneous and terminal costs and tis the time at which the variable ~ = ( x, t) first enters a prescribed stopping set 9', when it has terminal value If we defme the value function F(x, t) :::::: E[C(t)lx(t) = x], then it follows, as a special case of equation (8.16), that F(x, t) obeys the equation

e.

8F c+-+AF=O 8t

(~

rf- 9'),

(20)

a form of K.olmogorov's backward equation. It is subject to the terminal condition F =!Kin!/.

*Theorem 22.3. 1 f.* can formally set (21)

A=H(x.:x)

where the differential operator acts only on the argument ofthefunction ofx to which A is applied, and not on the x-argument ofH. With this understanding the backward equation (20) can be written c(x)+

8F(x t)

a)

tu' . . +H ( x, 8x F(x,t) =0

(~

rf- 9').

(22)

Proof The definition (9.5) of H states that Aecu = H(x, a)eax. Formula (21) thus holds if A is applied to an exponential function of x, and so to any finite linear combination of exponentials. Just as in Section 4.6 we then make the formal identification (21) for the action of A on all functions in a sufficiently 0 general class. Relation (22) then follows from (20). Note that if H(x, a) has a partial power series expansion

H(x,a) = aa(x) +!aN(x)aT + · .. then a(x) ct and N(x) 6t can be identified with E[oxlx(t) = x] and E[(8x)(ox)TI x(t) = x] respectively, to within terms of smaller order in Ct. Here ex is the incrementx(t + ct)- x(t). In particular, then,

x = a(x) = H(x, 0)

(23)

is the deterministic approximation to the process, in that a(x) is the expected rate &fEhange conditional on the current value of state.

3 OPERATORS AND SCALING FOR MARKOV PROCESSES

391

So, if the process is deterministic then the DCF has exactly the form aa(x).If the process is a diffusion, in that is conditionally normally distributed, then

ox

H(x,a) = aa(x) +!aN(x)aT

(24)

e}(actly. For another example, suppose the process such that x shows the jump transition x ....... x + d;(x.) with probability intensity A;(x) (i = I, 2, ... ). Then

H(x,a)

= LA;(x)(eQd;(x)- I).

(25)

i

Just as in the static case, one requires a concept of indefinitely increasing scale if large deviation concepts are to be applicable. The natural way of achieving this is to suppose that the DCF of the process is in fact KH(x, ,..,- 1a), where K is a large positive parameter and H(x, a) is a 'standardised' or 'unsealed' DCF. For integral K this is equivalent to the assumption that the increment 6x(t) = x(t + 6t) - x(t) for the scaled process is the average of K realisations of the increment for the unsealed process, these realisations being independent conditional on the value x of x( t). This would correspond again to the idea that x is an average over a system of size K, the system having the homogeneity and ergodicity properties which ensure that time-increments of this average are effectively the average of K conditionally independent increments. However, even if the system is Markov, the requirement that the process {x(t)} constituted by the system average x should also be Markov is a very strong one. Nevertheless, there are natural examples. If the unsealed process is a diffusion then we see from (24) that the scaled process is also a diffusion with DCF aa(x) + !~t- 1 aN(x)aT. That is, the effect of scaling is simply to suppose that the white-noise component in the plant equation is scaled by a factor ,..,-t/2. For a non-diffusion example, consider the case

H(x,a) = u(eQ- I)+ px(e-Q- I)

(26)

with scalar x and a; a special case of (25). One can interpret x as the number of particles in a chamber which particles enter in a Poisson stream of rate u (later to be taken as a control variable) and leave independently with individual probability intensity p. The scaled process represents the same process scaled up by a factor of K, in that it amounts to K replicas of the original process with x now representing an 'average' in that it is the number of particles per replica. In fact, the set of replicas pool to a Markov process in that the pooled version is equivalent to a single chamber with immigration rate KU. The backward equation (22) becomes

(( ~ !/).

if the process is scaled by a factor then this becomes

K.

(27)

If we retain only terms of zeroth order in "'

392

THE ESSENTIALS OF LARGE DEVIATION THEORY

8F c+-+Fa=O 8t ;(

(~ ¢.

9').

If we retain terms up to order ~-l then it becomes

8F

c + 7ii + Fxa

1

+ 2 ~ tr(NFu) = 0

(~

¢. 9').

(29)

Predictably, these are just the forms of the dynamic programming equation which will hold if the process is approximated by a deterministic or a diffusion process respectively. The deterministic approximation to a process is often referred to as the 'fluid approximation'; see Exercise 2. In deducing (28) or (29) we have supposed the cost function C itself independent of ~- It is when we allow an exponential dependence of cost on K. (natural under a variety of circumstances) that we obtain the generalised forms of these equations which imply large deviation effects. Exercises and comments

(1) We saw in Section 10.4 how scaling could come about in a superficially somewhat different fashion. We considered a birth-death process on the integers with birth and death intensities >v and /-Lj which were slowly varying withj. This slow variation was best expressed by representing these intensities as functions ~>.( x) and ~f..L( x) of a variable x = j / ~. where ~ is a large parameter. The xprocess is then just the ~-scaled version of a process with DCF

H(x, a)= .X(x)(ea- 1) + f..L(x)(e-a- I). In the fisheries context} would represent the number offish in the population (for a given species and region) and x a measure of 'stock' in units natural for a region of extent~(2) The term 'fluid approximation' is used for the deterministic approximation, just because it represents the flow in state space of the 'fluid' constituted by points representing the individual states of independent replicas of the process. This corresponds to a simple averaging of trajectories for these replicas, and so is a cruder idea than scaling, which constructs a single trajectory by averaging replica trajectory increments. The effect is the same in the deterministic limit, but in general only in this limit.

4 THE RATE FUNCTION FOR A MARKOV PROCESS Consider the conditional expectation

Glt(x, t) = Elt(e-~tC(tllx(t)

= x)

(30)

where C(t) is the future cost defined in (19). This is obviously a process analogue of the corresponding expectation in (4). That one should consider the expectation

1':-

4 THE RATE FUNCTION FOR A MARKOV PROCESS

3 93

itif an exponential of cost should seem reasonable after our discussion of risk-

~Cp,sitivity and th~ exponential-of-cost criterion in Chapter 16; another m~tiva­

::.'tion will emerge m Chapter 24. That the scale parameter "' should occur m the 'exponent is also reasonable, because the K--independent quantity C must be seen ·as a system-average cost, and so ,cas the total system cost. - .· G,. will obey a backward equation appropriate to the scaled process; we find this to be (~ ~ 9'),

with terminal condition G,.

= e-"K

(31)

in 9'. Assume now that G,. shows the

exponential K.-dependence

G"(x, t) =

e-J(x,t)+o(~<),

(32)

so that F(x, t) can be regarded as a value function on the original cost scale.

*Theorem 22.4.1 Relation (32) holds and the value function F(x, t) obeys the equation

c + 8F - H 8t

(x' -8F) ax

= 0

(~f. f/),

(33)

with terminal condition F = Kin f/. Proof If x were scalar then the relation

(j = 0, 1,2, ... ),

(34)

would hold, for large K. There is an obvious vector analogue. Inserting expression (32) into (31), appealing to the vector analogue of (34) and retaining only leading terms in "' we obtain (33). The fact that we obtain a valid K.-independent equation for F supplies the basis of an inductive proof (working backwards from D termination) that (32) holds. The argument is obviously not rigorous. In particular, there would be a Fourier version which would avoid appeal to possibly indefinite differentiability of F. However, the argument and the conclusion are perhaps simple enough to carry conviction. It is the passage from the linear but in general higher-order differential equation (31) to the first-order but in general non-linear differential equation (33) which marks the passage from exact evaluation to large-deviation evaluation. This is virtually identical with the passage from the equations of wave optics to the eikonal equation of geometric optics as wavelength decreases, or from the

394

THE ESSENTIALS OF LARGE DEVIATION TIIEORY

equations of wave mechanics to the Hamilton-Jacobi equation of classical mechanics as Planck's constant is assumed ever smaller. This is an indication that large deviation theory is indeed well rooted in physics; in the asymptotics of passage from diffuse wave propagation to a deterministic trajectory as wavelength is decreased. We come now to the large deviation assertion.

*Theorem 22.4.2 (i) Equation (#) and its terminal condition F = the unique solution F(x, t) = sup{inf x(·)

a:(-)

II( in f/ have

j' [c(x) +ax- H(x, a)] dr +II((~)}.

(35)

t

Here the extremisations with respect to the paths of the functions x( r) and a( r) are subject to the prescription of initial value x (t) = x, and t is the value of r at which {(r) = (x(r), r)firstentersf/. (ii) The scaled Markov process has the large deviation property, and its rate function for realisations over the time interval (0, h), conditional on the value ofx (0), is Doh[x(·)] =sup fh [ax- H(x, a)] dr. a(·)

Jo

(36)

• Proof Assertion (i) certainly implies a limited version of assertion (ii). If we choose t = 0 and specify stopping at time h with zero terminal cost, then (31~ (33) imply assertion (4) with C of the particular form c(x) dr. This form is general enough for our purposes, but further generalisation is plainly possible. To establish (35), consider the expression

J;

¢J(x, t) = sup{inf x(·J

a:(·)

j' [ax- Q(x, a)] dr +II((~)}, t

where the extremisation is subject to x(t) = x. This satisfies the terminal condition ¢J = K in f/. In virtue of its construction it also satisfies the recursion

ifJ(x, t)

= supinf[ifJ(x + 6x, t + 61) + a6x- Q(x, a) 6t] + o(6t) 6x

(37)

a

outside !/'.The extremal conditions yield

6x

= Qa 6!,

a= -¢Jx

and substitution of these back into (37) shows that ¢J obeys the differential equation

¢Jr - Q(x, -¢Jx) = 0

(~ rf_

!/').

(38)

But if the integral expression for ¢J satisfies (38) and its terminal condition then the right-hand member of (35) satisfies (33) and its terminal condition. Since these

5 HAMILTONIAN ASPECTS

395

'~latter are uniquely determining, we see that expression (35) indeed provides the

_determination of Fthat we seek.

0

Actually, we have appealed only to stationarity conditions with respect to the xand a-paths, in which case there is no difficulty with the commutation of operations implicit in our 'proof: However, these conditions can be strengthened to the sup/inf characterisation if c(x) - H(x, a) is convex in x and concave in a. We know indeed that there can be failures: the 'neurotic' and 'euphoric' breakdowns of Section 16.3 marked just such points of failure, and the breakdown had significance. We see in (36) the Markov-process analogue of the static evaluation (10), with the DCF H(x, a) replacing the CGF 1/J(a). The single variable a which tilted the distribution is now replaced by the function of time a(T) which performs a timedependent tilt, sometimes referred to as a 'twist~ We return to this point in the next section. In (35) we see a time-integral analogous to the time-integral (7.2) of the Pontryagin principle and the time-integral (16.52) of the LEQG treatment. This again raises the hope that there is a version of the Pontryagin maximum principle which transfers to a stochastic case much more general than that of the LEQG model. We shall investigate the matter in the next two chapters. Note that the reduction of the expectation (30)/ (32) to the evaluation (35) on a single extremising trajectory has essentially reduced evaluation of a path-integral to the evaluation of an extremal time-integral. This is exactly what largedeviation theory achieves, at least at its crudest level. Note that the occurrence of x in the expression (36) for the rate function does not imply that x is assumed to exist for the stochastic process itself. Large deviation approximations make assertions only on the grosser aspects of the path (which is why the expectation characterisation (4) is preferable to the probability characterisation (5)) and the fact that x makes changes of number 0(~'>) and of size O(K- 1) in a given interval of time means that it indeed has something like an effective rate of change for large K.

5 HAMILTONIAN ASPECTS The extremal path x( ·)determined by the evaluation ofexpression (35) is 'optimal' in that it compromises best between high probability and low cost (or, equivalently, low discrepancy and low cost). We shall term it limit-optimal since we shall wish to consider stochastic deviations from it in a more refined treatment. A special case is that in which the process is considered over the time interval (0, h] and there is no cost at all except that which effectively prescribes the terminal value x(h). The initial value x(O) is of course prescribed. The limit-optimal path in this case is then the (asymptotically) most probable path between the prescribed end-values. It minimises expression (36)

396

THE ESSENTIALS OF LARGE DEVIATION THEORY

for the rate function with respect to this path, and so is subject to the pair of equations

.

8H

x = 8a '

. 8H a = - ox

(

t


T)

(39)

with x(O) and x(h) prescribed. This conclusion can be expressed as Theorem 22.5.1 The asymptotically most probable path obeys Hamiltonian dynamics in the variable pair (x, a) with Hamiltonian just the DCF H(x, a~

This would in fact already be clear from the fact that the (x, a) path extremises the integral (36). Indeed, 'Hamiltonian dynamics' can be defined just as those which arise by extremisation of an integral of form (36), with H(x, a) defined as the Hamiltonian and equations (39) necessarily holding on the extremal path. The conjugate variable a supplies the time-dependent 'tilting' of the process which guides it by the most probable route to the prescribed terminal value x( h). It acts then as a kind of force on the stochastic dynamics which is equivalent to the conditioning that the path should terminate at a prescribed value. Indeed, if x(h) is chosen as the most probable value for the prescribed initial value x(O) then a is identically zero on the path; see Exercise 1. We began with a Markov process of no particular structure apart from the fact that it could be scaled, and yet Hamiltonian structure has now emerged. One naturally asks why this should be. The answer is that Hamiltonian structure will always emerge if one specifies dynamics only partially, and completes the specification by the imposition of an extremal principle. Thus, Hamiltonian structure emerges in the Pontryagin treatment of optimal deterministic control because the partial dynamic specification constituted by the plant equation is complemented by the optimality condition which determines the control. In the present case, one may say that specification of a stochastic process specifies only 'slack' dynamics; if one then adds the self-generated extremal principle that one seeks the most probable path (or the path which compromises best between high probability and low cost), then one has exactly the situation which generates Hamiltonian structure. Note, however, that the structure emerges only asymptotically, in the large- scale limit. If we actually perform the a-extremisation in (36) then we obtain

Doh[x(·)] =

1h

D(x, .X) dr

(40)

where D(x, x) = sup[ax- H(x, a)]

"'

(41)

can be regarded as the rate of increase of discrepancy with time along the path. In classical-mechanical contexts one would regard expression (40) as the action integral and D(x, .X) as the Lagrangian.

6 REFINEMENTS OF THE LARGE DEVIATION PRINCIPLE

397

In the costed situation the equation-pair (39) determining the limit-optimal path is modified by modification of the Hamiltonian to H(x, n) - c(x) : .

oH

X=

OQ'

. Q

o(c- H)

= --'--:-----'ox

(t
(42)

The termination coordinates (x, t) will be determined by the extremal conditions implied in (35). Since we can identify n with -Fx the path obeys the 'twisted' plant equation (43) X= Ha(X, -Fx). Exercises and comments (1) The requirement that x(/i) should also be chosen to minimise the rate function (36) yields the condition n(h) = 0. This then implies that oH j ox= 0 and so 0: = 0 at time h. The second equation of (39) is to be regarded as a backward equation in time; continuation of this argument implies that n is identically zero at all points on the path. The most probable path is then given by x = (oH jon)o.=o· This is the equation which we have already determined in (23) as governing the path in the deterministic limit. (2) Show that for the diffusion process specified by (24) D(x, .X)

![x- a(x)] V(x)- 1 [x- a(x)].

=

(3) Weiss (see Vanderbei and Weiss, 1988) has shown that for the jump process specified by (25)

D(x, X)

~ i~f { ~[u;log(u,fA;) + A; -

u;];

~ u,d, ~ X},

with the possible x-dependence of the A; and d; understood. Prove this by taking a Lagrangian multiplier n for the constraint indicated. This relates interestingly to an approximation proposed many years ago by Bartlett (1955, 1960). Bartlett regarded the process as one of superimposed Poisson streams, the ith having instantaneous rate A; and contributing d; at each event. If one regards n; = u;8t as the number of such events in a time interval of length 8t and regards n 1, nz, ... as jointly independent and Poisson, then the expression above is the logarithm of the probability of the most probable set of n; which would yield 8x = x 8t for prescribed x.

6 REFINEMENTS OF THE LARGE DEVIATION PRINCIPLE The large deviation treatment certainly takes account of the stochastic character of the model, in that it is based on the rate function which expresses the asymptotic essence of this character. On the other hand, it is quite crude in that it

398

THE ESSENTIALS OF LARGE DEVIATION THEORY

reduces consideration of events to consideration of a single path: the limit~ optimal path. The form of the path is dependent on both the stochastics and the costs assumed for the model, but stochastic variation about the path is neglected. We shall see in the next chapter, when we bring in the idea of control, that there are circumstances under which the optimisation of stochastic control is adequately treated by large deviation methods. Indeed, the treatment is a beautiful and inevitable one, in that it exhibits the time-integral methods of Part 4 as applicable to a very much more general class of processes than the LQG or LEQG processes of that Part (although under the assumptions oflarge scale etc.). One can say that this then also provides a natural extension of the Pontryagin maximum principle to a general class of stochastic problems, since the timeintegral methods amount to just such a principle. However, the treatment can prove inadequate at sensitive points on the path. Consider, for example, a model we shall treat in Chapter 24: a stochastic version of the landing problem of Section 7.11. We saw in that section that if the 'aircraft' began in so severe a dive that a crash (i.e. 'premature landing') could only be avoided by extreme measures, then the optimal path broke into two analytically distinct sections. In the first section one brought the plane out of the dive in a ground-grazing save; after that one could manoeuvre towards the desired terminal configuration in a more relaxed manner. In the large deviation treatment of the stochastic version of this model one will have the same phenomenon, although the limit-optimal path will now compromise between high probability and low cost. However, the notion that this path should graze the ground is unacceptable. At the grazing point stochastic deviation from the path will become important, and will force one to in fact ensure a clearance between plane and ground. The size of clearance one should aim for can only be determined by a more refined consideration of stochastic effects. The situation is really exactly that of Section 10.7. We saw there that one could well use the optimal deterministic rule for a stochastic model as long as the second differentials of the stochastic value function were fairly constant in the J?eighbourhood. It is when one approaches critical points on the path that this condition is violated. We saw in Section 10.7 that an undesired termination point is just such a critical point; the grazing-point in the example just considered has the same character. To derive a more refined result, consider again the backward equation (31), which is exact. Assume however the refined version G,.(x, t) = e-ltF(x,r)-.fO(x,t)+o(tl

{44)

of (31). Then, by the including the term of next order ( K.- 1) in relation (34) one can extend the argument of Theorem 22.4.1 to conclude that F satisfies the same equation

c + F 1 - H(x, -Fx)

=

0,

{33)

7 EQUILIBRIUM DISTRIBUTIONS AND EXPECTED EXIT TIMES

399

as previously, and that £0 satisfies the equation

£? + f'2Ha(X, -Fx) + ~ tr[FxxHaa(X, Fx)] = 0.

(45)

That is, an extra cost is incurred from the point ¢c, t): the time-integral along the limit-optimal path starting from (x, t) of~ tr[F.:uN(x, t)], where N(x, t) is the effective noise covariance matrix Haa(x, Fx). See Exercise 1. This conclusion is analogous to that obtained in (29). It says essentially that the process is to be regarded as a diffusion process, for which to the deterministic drift (43) that would take it along the limit-optimal path is added a Gaussian white noise component with covariance matrix N(x, t). We have thus returned to a fully stochastic model, and one indeed which is close to the original model if that was itself a diffusion process. Exercises and comments

(1) We saw in equation (43) that the limit-optimal orbit satisfied the equation .X= Ha(x, -Fx) = a(x, t), say. (At least this holds so long as the orbit is 'free'; i.e. does not encounter constraints or additional penalties). Show that the solution ¢ of the equation ¢ 1 + t/Jxa(x, t) + '1/J(x, t) = Oisjust the time-integral of'ljJ along the limit-optimal orbit from the point (x, t) to the termination point

7 EQUILIBRIUM DISTRIBUTIONS AND EXPECTED EXIT TIMES There is a class of ideas which lies somewhat off our principal line of argument, but which nevertheless provides one of the truly interesting applications of large deviation concepts in control theory, and of which we have already seen an instance in Chapter 10. It involves notions of stationarity, quasi-stationarity, recurrence and first passage for which a brief treatment is necessarily cavalier and brief 'proofs' necessarily no more than plausibility arguments. Consider again a scaled Markov process with unsealed DCF H(x, a) and suppose that x( t) has an effective 'density'

1r(x, t)

= e-KU(x,t)+o(~<)

(46)

conditional upon initial conditions.

• Theorem 22.7.1

U (x, t) as defzned above obeys the forward equation

88~ = -n(x,- ~~).

(47)

*Proof The following argument exploits earlier analysis; a better argument is sketched in Exercise l. Suppose initial conditions specify the value of x(O). Then we know from the evaluation (36) of the rate function that (46) holds with

400

THE ESSENTIALS OF LARGE DEVIATION THEORY

·-~

_, _.,

f'

U(x, t) = inf sup [ax- H(x, o:)] dT. xO a() Jo

'l

(48)'

where the infimum with respect to x(·) is subject to prescription of x(O), t and x(t) = x. Just as we found that the function (35) of initial state satisfied equation (33), so we fmd that the function (48) of state at the upper integration limit satisfies (4n whatever x(O). 0 Suppose now that there is a value x. at which the equilibrium density rr(x) is maximal; this would be the preferred equilibrium point in a deterministic treatment. Define the quasipotential

U(x) = infsup rt[o:x- H(x, o:)] dT. x<·l a(-) Jo where the set of paths x( ·)admitted prescribe x(O) ofT for which x( T) = x on the path.

*Theorem 22.7.2

(49)

= x. and 1is the smallest value

The normalised equilibrium density for the scaled process is rr(x) =

e-~
(50)

Proof One finds that expression (49) satisfies the equilibrium form of (47). The property U(x.) = 0 implies normalisation to the order of magnitude indicated, since then ·

0 Consider, as an example, the scaled birth-death process defined in Exercise 3.1. For this we find that

U(x)

= 1~ log[p{y)/-X(y)] dy =

1x.log[-X(y)/J.L(y)] dy;

{51)

see Exercise 2. Relations (50) and (51) are consistent with the known equilibrium distribution given in equation (10.12). Suppose now that x. is only a local maximum of rr(x); in the deterministic version of the process it then constitutes a locally stable equilibrium point with basin of attraction f:JI, say. In the stochastic version of the problem escape from 11 is possible and, in the long run, certain for an irreducible process. However, one would like to assert that the escape time T, a random variable, is large in some sense. For example, for the harvesting example of Sections 10.4 and 10.5 the threshold point c was the most probable value and was locally stable in the deterministic limit. However, if extinction was possible it was also certain, but one

7 EQUILIBRIUM DISTRIBUTIONS AND EXPECTED EXIT TIMES

401

,wished for a demonstration that the expected time to extinction would increase :indefinitely as increasing scale factor"' implied approach to determinism.

•Theorem 22.7.3

If Tis the escape time from the basin ofattraction f!J ojx. then E(T)

=

exp[K inf U(x) xE8iJI

+ o(K)]

(52)

where oPJ is the boundary ofPl. The expected escape time is thus exponentially large in "'· One might imagine that, if one defined F(x) = E(Tix(O) = x) for x in PJ, then one could set up a backward equation for F (x) and solve its large-deviation version. If one does so one finds that F(x) is essentially constant in PJ, which is why there is no mention of initial value in the theorem. The explanation is that, for any starting point which is not too close to the boundary of PJ, the probability tends to unity with increasing "' that the path rapidly makes its way to a prescribed neighbourhood % of x. before it ultimately reaches the boundary of f!J. The escape problem thus 'standardises' itself to x(O) = x •. The alternative argument which we offer is extremely rough, but mirrors the actual course of events. Vary the neighbourhood AI' with"' so as to make the integral of 11'(x) over AI' approximately independent of"'· Every excursion of the process from AI' can be regarded as an attempt to reach the boundary of PJ which is successful with probability p

= exp[-K inf U(x) + o(K)]. xE8!JI

Consecutive attempts are statistically independent, so the probability of success at thejth attempt is (1 - p'j- 1p(j = 1, 2, ... ) and the expected number of attempts needed until escape is achieved is 1 + p- 1 "'p- 1. The expected time needed to escape is thus not less than ap- 1, where a is the expected duration of an excursion from %; this will be approximately independent of "'· We thus confirm 0 expression (52) as at least a lower bound on E(T). The argument is obviously extremely loose, but conveys a picture of actual events. Expression (52) in fact gives the expected number of 'escape attempts' needed, to the order indicated, and most of the escape time Tis spent relatively close to the deterministic equilibrium value x •. The asymptotic evaluation (10.18)/(10.20) of expected extinction time for the population model, derived by direct arguments, is consistent with (52) and determination (51) of the quasipotential in this case. The value x = 0, at which extinction occurs, constitutes the boundary of 81. The asymptotic evaluation (10.20) holds also for the two other stochastic population models considered in Chapter 10, with the determinations of quasi potential implied by formulae (10.39) and (10.51) for R(x).

402

THE ESSENTIALS OF LARGE DEVIATION THEORY

Exercises and comments

1 ::1

,,-l

(1) Forward equations are more naturally expressed in terms of transforms, as we~ have emphasised in Section 12.9. Define M(o:, t) as the MGF of x(t) for: prescribed initial conditions. Then the forward analogue of (22) is oM I ot ::: H(ofoo:,o:)M, an operator form of the Kolmogorov equation derived by Bartlett (1949). If the process is now scaled by a factor K and we assume M(o:, t) = exp[K'¢(K- 1o:, t + o(K), then we find, as in Section 3, that the eikonal form of this last equation is 81/J/ot = H(o'I/J/oo:, o:). This and the relation U(x, t) = supa[o:x -1/J(a., t)] imply the Kolmogorov equation in the form (47). (2) Since the boundary conditions are time-independent in (49) then H(x, o:) is zero on the extremal path and the integral reduces to I~ o:x d-r. For the case of o:(y) dy where a.(y) is the function of y determined by scalar x this reduces to H(y, a.) = 0. For the birth-death process of Exercise 3.1 we have then o:(y) = log[JL(y)/ -\(y)], whence (51) follows. Note that the value of .X determined for the extremal path is J.L(x) - ,\(x), which is the reverse of the velocity of the time-reversed deterministic process. lb.is cannot be the actual velocity, and the time-argument of the integral (49) does not represent actual time, although the fact that J.L - ,\ is zero at x. does reflect the difficulty the process has in leaving the neighbourhood of this point.

I:.

Notes on the literature For a long time the standard texts were those due to the very significant early figures in the development of the theory: Varadhan (1984), Stroock (1984), Friedlin and Wentzell (19841 Ellis (1985); Donsker is a key figure in the journal literature. To a large extent the only stochastic processes considered were diffusions, and the Hamiltonian formalism of Section 4 not really to be found. Bucklew (1990) reviewed applications. The unpublished notes circulated by Vanderbei and Weiss (1988) were widely valued, they considered jump processes occurring in telecommunication applications. A developed form of these notes ·appears as Shwarz and Weiss (1995). Other recent texts are those of Deuschel and Strook (1989) and Dembo and Zeitani (1986). The operator formalism of Section 3 is due to Bartlett (1949). It provides the natural underpinning for the formalism of Section 4. This was developed in Whittle (1990b), although it must be contained in the physical literature in the general discussion of the Hamilton-Jacobi equation and its quantum equivalents. The approach contrasts with that of the probabilists, who employ martingale arguments and the like. Probabilistic arguments can be powerful and insightful; the technique of 'tilting' a distribution or 'twisting' a process provides an example. Friedlin and Wentzell (1984) gave a very complete treatment of the expected exit time from a basin of attraction (and oftransition between such basins) for the

~.

L

7 EQUILIBRIUM DISTRIBUTIONS AND EXPECTED EXIT TIMES

403

of a diffusion process. Tegeder (1993) elucidated the whole matter very ~egantly for the general Markov case. ~·· ManY authors have considered refinements of the basic large deviation assertions, usually by direct probabilistic arguments; see e.g. Azencott (19&2, 1984) and Ben Arous (1988). The modified backward equations (33), (45) are deduced in Whittle (1990b, 19951 where the treatment is continued to derive formulae which are close to, but interestingly different from, formulae due to van Vleck (1928) for the quantum-mechanical context.

CHAPTER 23

Control Optimisation in the Large Deviation Limit The material of the last chapter has an immediate formal generalisation to the controlled (Markov) case, with the dynamic programming equation replacing the various backward equations. The time-integral solution in the risk-sensitive ('exponential-of-cost') formulation persists, with the addition of a u-extremisation. This form of solution implies validity of an asymptotic maximum principle, but one which is in general more fairly interpreted as a risk-sensitive maximum principle than as a stochastic maximum principle.

1 SPECIFICATION AND THE RISK-NEUTRAL CASE We now generalise the material of the last chapter to the controlled case. We assume the usual cost function, so that the future cost at timet is

C(t)

li

=

c(x, u) dT +II<(().

(I)

Here tis the time at which the process terminates, specified as the value ofT for which ~(T) = ( x(T), T) first enters the prescribed stopping set !/', and ( is the terminal value of~. Markov dynamics are economically specified in terms of the DCF:

E[e126x(t) IX(t), U(t)]

= 1 + H(x(t), u(t), a) 6t + o(6t).

(2)

That is, the current value u of control now simply appears as an additional argument in the DCF H(x, u, a), and the dynamic programming equation for the value function

F(x, t)

= inf ,.. E[C(t)lx(t) = xJ

can bewritten

i~[c(x, u) + aF~~· t) +

n(x, !)F(x, t)] u,

=

0

This is of course just the optimally controlled version of (22.22).

(3)

406

CONTROL OPTIMISATION IN THE LARGE DEVIATION LIMIT

Suppose now that H has the series development

H(x, u, a) = aa(x, u)

+! a.N(x, t)aT + · · · .

(4)

so that a(x, u) 8t and N(x, u) 6t are the mean and covariance matrix of 6:x conditional on the immediately previous values of x and u. Suppose furthermore the process scaled by a factor "'• so that H(x, u, a) is replaced by KH(x, u, K.- 1a) and K is assumed large. If we retain only leading terms in the scaled version of (3) then we obtain inf[c(x, u) u

+ F + Fxa(x, u)] = 1

0;

(5)

just the dynamic programming equation for the deterministic approximation to the process (cf. (22.28)). If we retain terms of relative order ~>.- 1 then we obtain

i~ { c(x, u) + F + Fxa(x, u) + 1

L

tr[N(x, u)Fxx]} = 0;

(6)

just the dynamic programming equation for the diffusion approximation to the process (cf (22.29)). We make these relatively trite points to emphasise again that the large deviation approximation, for all the interest and structure that it reveals, is only a 'tilted' form of the crude deterministic approximation associated with (5). 2 LARGE DEVIATION CONCLUSIONS IN THE RISK-SENSITIVE CASE Suppose that we adopt the risk-sensitive criterion E,.(e-"'ec), where (J is the risksensitivity parameter and "' the scale parameter of the controlled process. As in Section 22.3 it seems reasonable to include the factor K in the exponent, since if C is the cost calculated on system averages then KC is a proper assessment of the cost for the whole system. The policy 1r will be chosen to minimise or maximise ,this criterion according as fJ is negative or positive. The expectation is exactly of the form for which large deviation methods are applicable. We shall essentially just transfer the analysis of Section 22.4 to the controlled case, so that the various backward equations for the expectations now become versions of the dynamic programming equation, incorporating the taking of an extremum with respect to the current control value u. We suppose now that the relation

G"'(x, t) := extEw[e-lt8C(tlJx(t) = x] =

e-lt8F(x,t)+o(~<)

(7)

7r

holds; i.e. that the middle expression shows the exponential dependence on K indicated, for large "'·We then interpret F(x, t) as a limiting normalised value function (i.e. 'limiting' for large "'and 'normalised' to the original cost scale).

2 LARGE DEVIATION CONCLUSIONS IN THE RISK-SENSITIVE CASE

407

We can then transfer the analysis of Section 22.4 bodily, the only differences being the inclusion of the risk-sensitivity parameter () and the addition of an extremisation with respect to current u (which yields the optimal, or approximately optimal, value of u). The analogue of (22.31) is the dynamic programming equation for G:

e~t [ -K9c(x, u)GI< +a~I<+ KH

(x,

u, ,- 1

:x) Gl<]

= 0

(~f. 9')

(8)

with terminal condition Gl< = e-t
with terminal condition F = IK in 9'. The relation is only approximate because we have neglected terms of relative order ,- 1. It is the approximation made here which is the essential large-deviation approximation. The minimising value of u is optimal at (x, t) to the same level of approximation. We have now the analogue of Theorem 22.4.2, giving the path integral solution of equation (9). *Theorem 23.2.1

tion F

Suppose that ()

> 0. Then equation (9) and its boundary condi-

= IK in 9' have the unique solution

F(x, t) = inf inf sup u(·) .x(-)

.\(·)

[1' I

[c(x, u)

+).T.X-

o- 1H(x, u, O>.T)] dr + IK(~)J.

(10)

where the path x( ·) is constrained by x(t) = x. The value of u(t) thus determined is optimal in the limit of large"-· These assertions also hold in the case 9 < 0 with the modification that the extremal operations with respect to x(·) and>.(·) become sup and infrespectively.

The argument of Theorem 22.4.2 carries over formally, the presence of the extremisation with respect to u presenting no difficulty. However, to the reasons for the lack of rigour of that argument we have now added those consequent upon the presence of the extra variable u in H and its extremisation. We plead in mitigation only the extreme disproportion between the efforts needed to present formal and rigorous arguments, plus the inclination to defer these latter until they can themselves be made more insightful. Note that in the passage from (22.35) to (10) above we have replaced the row vector a by B>.T, so that>. is a column vector. The occurrence ofthe 9-factor is a consequence of its appearance in the exponent of the criterion function. We have

408

CONTROL OPTIMISATION IN THE LARGE DEVIATION LIMIT

taken>. as a column vector so that it can be identified with the familiar Lagrange. multiplier of earlier trajectory optimisations, including that of the maximum principle; see integral (7.2). Our reason for wanting to take >. as a column vector in those cases was that the extremising conditions with respect to x, u and >.could then be written down as a single equation system, as in (6.20). Note the implication of solution (10): that the optimal control can be derived by application of a maximum principle, with the Hamiltonian>.Ta(x, u) - c(x, u) of the Pontryagin treatment (Chapter 7) replaced simply by o-I H(x, u, o>.T)c(x, u). This seems to constitute what has long been sought: a stochastic maximum principle. Such a principle has been known to exist (determining the optimal control exactly in a stochastic case) in a few very special cases, but existence of a general principle has been a matter of extreme doubt. In many cases where such a principle has been announced it has amounted to little more than a statement of the dynamic programming principle, in that the present value of the dual variable >. is expressed as a conditional expectation (generally not calculable) over the future of the optimally controlled process. The principle determined from the path integral (10) requires no such conditional expectations: extremisation with respect to the (x, u, >.)-path determines the predicted future course of the optimally controlled process, just as the conventional principle determines the actual course. However, our conclusions may be too crude that one would be happy to claim that they fulfil the hopes for a stochastic maximum principle. If we consider the risk-neutral case f ) - 0 then the Hamiltonian becomes simply >.Ta(x, u)- c(x, u~ where a(x, u) is the conditional expectation of rate of change defined in (4). That is, the principle reduces just to the conventional maximum principle for the deterministic version = a(x, u) of the process; no stochastic aspect is retained at all. The reduction implied in (10), of stochastics to consideration of a single 'limitoptimal' path, reflects the same point, already made in Section 22.6. In the risk-sensitive case the stochastics of the process are of course reflected, even in the large deviation limit, in that the full DCF H(x, u, a) occurs in the !Jamiltonian. However, as we shall see in the next section, what one obtains is rather a deterministic principle 'coloured' by optimism or pessimism. This is not sufficient to enforce a positive clearance of dangerous contingencies: for that we shall need the more refined treatment of the next chapter. Nevertheless, the principle undoubtedly embodies valid and valuable conclusions. We shall refer to it as the risk-sensitive maximum principle (abbreviated to RSMP), to avoid confusion with any stronger version of a stochastic maximum principle which may be developed.

x

3 AN EXAMPLE: THE INERTIALESS LANDING PROBLEM Consider the inertialess landing problem of Section 7.9, but now with a stochastic plant equation x = u + f., where f is white noise of power N Recall that x is the

3 AN EXAMPLE: THE INERTIALESS LANDING PROBLEM

409

current height of a particle above the ground, moving horizontally with unit velocity, and that one wishes to land it (i.e. bring x to zero) in such a way as to nUn.imise the sum of the integral of control costs Q1l /2 and terminal cost IK(t), where tis the moment oflanding. This is a simple but illuminating example: too simple to be of direct practical interest in itself, but of practical interest for the insight it generates. The Hamiltonian for the RSMP is then AU+ N8A 2 /2- Qt} /2, whence we derive the equations u=

Q- 1A,

x = u + NBA,

( 11)

on a free orbit. The second equation is a plant equation with predicted noise; NOA is the predicted value of noise ~: on a minimal stress criterion. We see then that A and u are constant on the path, and the value of A is then determined by x = -(Q- 1 + NB)>.s, where s = t- tis the further time taken to reduce height to zero. The actual value of control is then X

u =- s(1

+ QNB)

= -'"'(xjs,

(12)

say. Evaluation of the integral in (10) is then immediate, and we determine the total cost as

Qx2 2(t- t)(1

+ QNB) + IK(t).

(13)

The intended landing time t = t + s must be the value exceeding t which minimises this expression. We see from (11) and (12) that the predicted path of the optimally controlled process satisfies x = - xj s, and so is a straight line from the initial point (x, t) to the terminal point (0, t ), just as for the deterministic version of Section 7.9. However, this is the minimal stress path; the actual plant equation under the control rule (12) will be .X= -'"'(x/ s + f. If we assume t prescribed and neglect the effects of noise then this has solution ·

x(t)

= c(t- t)'"',

(14)

where cis a constant, to be determined from initial conditions. Now, 'Y is less than, equal to or greater than unity according as (} is positive, zero or negative. We see then from (14) that the optimally controlled path will (apart from noise disturbances) approach a prescribed terminal point as in the graph of Figure 1: concavely, linearly or convexly according as the controller is risk-seeking, riskneutral or risk-averse. This pattern is in a way the opposite of what one might have expected. Consider the case of a pessimistic controller, when (} < 0 and 'Y > I. One might have expected that pessimism would have made him conscious of the hazards of an

410

CONTROL OPTIMISATION IN THE LARGE DEVIATION LIMIT

X

Figure 1 Effective terminal approach paths for the inertialess particle in the risk-neutral (0 = 0), optimistic (0 > 0) and pessimistic (8 < 0) cases.

early landing (when a penalty would be incurred) so that he would be more concerned than in the risk-neutral case to maintain height. That is, one would have expected the concave approach of Figure 1 rather than the convex one. The explanation is that a pessimistic controller fears that all his actions will be opposed by a contrary Nature; he therefore makes his actions all the stronger, which is why 'Y > 1 in (14). That is, pessimism may lead to convulsive and premature action rather than feeble and postponed action. Since he wishes to descend, he then tends to descend prematurely, implying the convex approach of Figure 1. The final part of his approach is a ground-skimming one, which is not what one would expect if there were a consciousness of the possibility of being driven by random disturbances to a premature landing. To this extent stochastic effects are lacking; the penalty which may be incurred on x = 0 does not 'propagate' into x > 0 as an anticipation of danger. It is in the lack of this anticipation that the RSMP shows itself inadequate, particularly when the danger is close. We shall refine the treatment to deal with this point in the next chapter, and shall see that stochastic effects do then indeed induce an incentive to gain height when danger threatens. The intended termination point t is of course not prescribed, but determined by minimisation of expression (13). If IK( t) is convex with a minimum at I opt then the minimising twill be somewhat greater than lopt. The intended termination point tis constant along the predicted path, but will vary on the actual path. In the pessimistic case the controller will set a linear predicted course to the point (0, t), but will in fact immediately fall below it, as we saw above. A new determination oft will then yield a smaller value. That is, the intended landing point of a pessimistic pilot will tend to creep backwards in time, as he tries to escape from uncertainty. That of an optimistic pilot will tend to creep forwards, since he believes that uncertainty is on his side. To take an explicit example, suppose that IK(t) = k/(d- t), so that penalty increases monotonically to an infinite value as t Td. One finds then that the

4 THE OPTIMISATION OF CONSUMPTION OVER A LIFETIME ~ue of

1 minimising expression (13) is 1 = (t + {3xd)/(1

ft == JQ:y f2k. The actual noiseless plant equation is then .

X

+ {3x),

411 where

+ {3x) = - -/X- = - 1(1 --'-.,....-----'--c-'-1-t

{3(d-t)

with solution

x(t)

= /3- 1 [c(d- t)'Y -1],

where the constant c is determined by initial conditions. The intended terminal point is then given as a function of timet by

1 = d- c- 1 (d- t) 1 -'"~. This varies with t as predicted above.

4 THE OPTIMISATION OF CONSUMPTION OVER A LIFETIME Consider a continuous-time version of the problem of Section 2.2: the classic problem of choosing consumption u so as to maximise the total utility

lot g(u) dr + G(x) over a lifetime (0, 1), where 1 is initially assumed known. Here x(r) is the optimiser's capital at time r, assumed to obey the plant equation

.X= ax-u,

(15)

where a is the rate of interest. The variables x and u are intrinsically positive, so a totally LQG formulation is impossible. The deterministic maximum principle is associated with the path integral

1 1

0=

[g(u) + >.x- >.(ax- u)] dr + G(x).

(16)

If one chooses g(u) =log u and G(x) = k log x then one finds by conventional analysis that the optimal rule, in closed loop form, is X

U=--

k+s

(17)

where s = 1 :-tis time to go, the residual lifetime of the individual. Note that the residual lifetime is effectively augmented by an amount k, which reflects the utility to the individual of being able to leave an inheritance. Remarkably, the rule is independent of a. One could introduce a stochastic element into the model by introducing noise into the plant equation (15), so introducing some uncertainty into the growth of

412

CONTROL OPTIMISATION IN THE LARGE DEVIATION LIMIT

capital (see Exercise 1). However, a more interesting alternative is to introduce some uncertainty into length of life. So, suppose that capital obeys the deterministic equation (15), but that there is a second component of the state variable, 'apparent residual lifetime' y, which obeys the equation

y=

-1

+E.

Here E is white noise of power N, and we suppose that the moment of death tis the time at which y first equals zero. The occurrence of the noise term reflects the assumption that, because of changing health or circumstance, residual lifetime is not perfectly predictable from y. The path integral equivalent to the evaluation (10) of Fis now

1 1

0=

[g(u) +

)qx

+ >.2y-

)q (ax-

u) + >. 2 +! ()N>.~]

dT

+ G(x).

(18)

Recall that the assumption of large scale amounts to the assumption that N is small. Application of the maximum principle leads one again to the closed-loop optimal control rule (17), but with the effective residual lifetimes determined in terms of current x andy by

/ - s2 = 2()Nl{l - a(k + s) +log [(k + s)jx]}.

(19)

In the risk-neutral case this leads to the relatively crude conclusion s = y. However, we can observe the effects of risk-sensitivity. As in the last section, these may seem counterintuitive at first glance, but further thought confirms them as reasonable. The graph in Figure 2 illustrates how consumption changes (as a function of capital x and apparent augmented residual lifetime k + y) if () increases from

Decreased consumption

Assets

x

Figure 2 The effect ofan increased degree ofoptimism upon consumption, as a.fimction of current assets and perceived residua/lifetime.

5 A NON-DIFFUSION EXAMPLE

413

zero; i.e. if the individual becomes mildly optimistic. The interest rate a is assumed positive. If assets x are large enough then the optimal rate of consumption decreases. Life is good at this level, and so optimism leads the individual to expect that he will Jive longer, and so must then husband his resources more carefully. However, if x is small enough then there is a band of y within which optimal consumpti<>n increases with increasing optimism. Here the individual is living at such a low level of consumption that life does not seem attractive. His life expectation is large enough that his resources appe~r inadequate to make his remaining days comfortable, but not large enough that it would be worth his while to build up his capital. The optimistic course is then to assume that his residual life span is in fact shorter than y would indicate, and so to consume at a greater rate than in the risk-neutral case. Exercises and comments (I) Consider the case where t is fixed and known but white noise of power N is introduced into equation (15). The effect is then to add a term ON >..2 to the integrand in (161 and the optimal value of u is given in terms of x and s = t - t by

!

(k + s)u- x

ON = 2au (1 -

e-2as).

5 A NON-DIFFUSION EXAMPLE: THE REGULATION OF PARTICLE NUMBERS The examples of the last two sections were all diffusion examples, in that the plant equation was driven by white noise. For a natural example of another type, consider the process with standardised DCF (22.26). For this x represents the number of particles in a chamber with controlled admission rate u and independent departure rate p. (Note that, after scaling, the actual number of particles is l'i.X, the actual admission and total departure rates are l'i.U and pl'i.x, and actual costs must also be scaled by a factor l'i..) If the instantaneous cost function is u + c(x), say, and the minimal average cost (in the risk-sensitive sense) is"'/, then, on the optimal path,

"'/ = u + c(x)- o-l [u(r/>- 1) + px(r/>-l- 1)], where r/> = e9>. and we can identify the term in the right-hand member as the negative Hamiltonian -ff (see Exercise 7.2.2). We suppose that 0 ~ u ~ M, say; the condition that -.Jf{ be minimal with respect tq u then implies that the optimal u must be zero or M according as r/> is greater or less than 1 + 0. The value of x = .Yf>. = ur/>- px¢- 1 must be positive or negative according as to which of these holds. The equilibrium condition ..\ = -.Yfx = 0 yields

414

CONTROL OPTIMISATION IN THE LARGE DEVIATION LIMIT

..i

c'(x)- o-• p(¢- 1 - 1) = 0. The equilibrium value of x obviously coincides with.; the threshold value Xthr at which u changes discontinuously, and so at whichl ¢ = 1 + 0. Eliminating¢ (and so>.) from these last two equations we obtain th~' equation

c'(x) = -p(1

+ or•

determiningxthr· If c(x) is convex with a minimum atxopt then xthr will be smaller than Xopt. corresponding to a displacement cost which is balanced out by the decreased u-cost. The value of Xthr will increase with increasing B, representing increased optimism that a deficit in input u will be made good by random variation. The value of(} at which neurotic breakdown occurs is plainly -1. Notes on the literature The use of large deviation methods for the optimisation of stochastic control was suggested passingly in Friedlin and Wentzell (1984) (section 8.2) and applied to the minimisation of exit time probabilities by Fleming (1971, 1978, 1985) and Fleming and Tsai (1981). Its association with a risk-sensitive criterion and so the deduction of the Hamilton-Jacobi equation (9) with solution (10) and its implied maximum principle are due to Whittle (1990b). This was also the first paper to break away from diffusion models and consider a general type of Markov process.

CHAPTER24

Controlled First Passage .we have already emphasised in the last two chapters that large deviation methods need refinement in cases for which the limit-optimal path grazes forbidden or high-penalty regions of state space, as in the landing problem. Refinement will .take the form that one replaces the 'deterministic' process which generates the limit-optimal path by a diffusion. There is a class of cases for which this diffusion process is readily generated and treated. This class also has the property that the problem of minimising expected cost for a controlled process can be transformed to that of calculating expected _exponential-of-cost for the corresponding uncontrolled process. That is, 'risksensitivity' is not assumed, but is generated by control optimisation itself. An example of this class has already been seen in the first-passage problem associated with the 'fly-paper' effect (Section 10.7). The example generalises to cover plausible forms of the landing problem. In particular, we shall deduce the optimal margin of clearance (of hazards) for some models which are far too simple for realism, but which are archetypal. It is the author's preoccupation with the landing problem which largely motivates this work. Attempts to optimise the landing approach have often simply optimised the control of deviations (on an LQ model) from a prescribed ideal approach path. But the ideal approach path should itself be derived from a full stochastic model. If air-space were physically homogeneous and free of other traffic (a legitimate first simplification!) then one would hold a particular height, not because there was virtue in that height, but because the hazards of the ground compelled one to keep a respectful distance from it. Similar considerations would dictate a landing path: the unpredictability induced by stochastic effects would cause costs defmed on the ground to 'propagate' into air-space to influence actions there.

1 REDUCTION OF THE CONTROL PROBLEM Consider the controlled diffusion process with plant equation

x=

a(x, t)

+ Bu + f,

!uT

(1)

instantaneous cost function c(u) = Qu and tenninal cost function K(€) incurred at first entry of the state/time variable € = (x, t) to a stopping set 51. The white noise component f is assumed to have power matrix N.

416

CONTROLLED FIRST PASSAGE

The specialising assumptions are thus (i) that the problem is LQ in its dependence on the control variable and (ii) that no state-dependent instantaneous costs are incurred. The second assumption is less restrictive than it seems, in that one can often redefine the state variable to achieve this (see Exercise 1). Also, as emphasised above, it is the essence of some problems that sensitivity to state value is induced, not by an explicit state cost, but by the propagation of terminal costs into state space. One could allow the matrices B, Q and N to depend upon time, but we shall suppose them constant, for simplicity. We must, however, add the key assumption (iii) that (2) for some scalar "'·The assumption is then that control and noise are comparable in their driving effect on the plant equation, in that the control power matrix N is proportional to the noise power matrix 1 The assumption may seem a very restrictive one, but it certainly holds in a number of cases of interest. Note that if 1 K is large then N is of order K- , for fixed Band Q. Theorem 24.1.1 Under the assumptions above, the value function trolled process is given by

F(~)

of the con(3)

if the expectation is well-defined. Here E{ is an expectation over the coordinates (of first entry ofthe uncontrolled process .X= a(x, t)

+E

(4)

into!/, conditional on a start from a point~ = (x, t). Proof The optimality equation for the controlled process is

inf(! uT Qu + F, u

+ Fx(a + Bu) + ~ tr(NFxx)]

outside!/, with the boundary condition F(~) =

II<(~)

= 0

(5)

in!/. It follows from (5) that

u = -Q-IBTFJ

(6)

with the consequent reduction of the optimality equation

F,

+ Fxa- !FxBQ- 1BTFI +!

tr(NFxx)] = 0.

(7)

Now, if (2) holds, then the transformation '1/J(~)

= e-"FW

(8)

linearises (7) to (9)

1 REDUCTION OF THE CONTROL PROBLEM

417

This is subject to the boundary condition '¢(~) = e-d<W

(~ E

9').

(10)

No~ if the right-hand member of (3) were denoted '¢( ~) then (9), (10) are just the equation and boundary condition that the characterisation of the theorem would require of it. The solution of (9), (10) is unique, however, and must then be identifiable with the right-hand member of (3), if this is well-defined. Good definition will, for example, imply that the uncontrolled path from ~ will terminate in Y with probability one. D

The theorem has two striking implications: (i) that the optimisation of control has been reduced to calculation of an expectation for the uncontrolled process, and (ii) that the risk-neutral ambition of minimising the expectation of cost C has been transformed into the 'risk-sensitive' evaluation of the expectation of exp( -~~:K). This latter calculation is then perfectly set up for the application of large-deviation methods, if~~: is large. One still has the first-passage problem of determining the distribution of terminal coordinate for the uncontrolled process. We follow this up in Section 3. Some cases can be solved exactly; for others, we assume N small (and so tt large) and deduce what amounts to a refmed version of a large deviation evaluation. Note that the rule (6) for the optimal control becomes

e

( 11) in terms of 1/J. The key conclusion (3) holds only for a diffusion process, and one of a rather particular form. One has to admit that, even though we have managed to break away from the limitation of attention to the diffusion case which has so marked the work of the last three decades, the diffusion processes do have a particular role and character. Exercises and comments (1) Suppose that the instantaneous cost has the form c(x) (9) becomes

ttc'¢ + '¢1 + 1/Jxa + 1 tr(N'I/Jxx)

+! uT Qu. Then relation

= 0.

(12)

with solution (3) now modified to

This expression of the solution of (12) is just what is often referred to as the 'Feynman-Kac formula'. In the dynamic programming context it would be

CONTROLLED FIRST PASSAGE

418

regarded as a rather straightforward assertion, at least under circumstances for which the expectation is well defined. (2) Risk-sensitivity can be worked in very naturally. Consider the normalised risksensitive value function F(~) defined by e-BF = ext,.E,.(e-11C(tllx(t) = x). Show that, under the same assumptions as above, 'lj; = e-(~<+B)F satisfies (9), so that (3) holds with It replaced by K + 0 throughout. 2 THE LARGE DEVIATION EVALUATION Theorem 23.1.1 casts the solution in a form (3) which seems made for large deviation evaluation in the limit oflarge K. However, it turns out, as we shall see, that in its unrefined form this yields nothing but the deterministic version of the original stochastic control problem. Nevertheless, the establishment of this connection gives some insight in itself, and an indication of the path to a more refined approximation. Let us denote the control power matrix BQ- 1 BT by J, so that the basic assumption (2) can be written simply N = K- 1J. Then the unsealed DCF of the uncontrolled process (3) is aa(x, t) + !aJaT and the large deviation evaluation of the value function F(x, t) implied by (3) is 1

F(x,t)

"'infsup(1 (a.X-aa-~aJaT) dr+IK(~)). x(·) a{)

(13)

1

Here the infimum over paths x(·) is constrained by the initial condition x(t) = x and the fact that the path must not enter Y before the terminal moment t. The path is thus not necessarily a free one before this point: it may consist of free sections separated by grazing encounters with Y. Theorem 24.2.1 The approximate evaluation (J3) is just the value function ofthe deterministic version ofthe problem, and the path x( ·) determined by it is just the path ofthe optimally controlled deterministic process. Proof Take (13) as a defining equality, for simplicity. Then we have Fx = -a (at the initial time) so that the optimal control on this basis would be

u = Q-IBTaT.

(14)

Performing the a(· )-maximisation in (13) we obtain

x-a=JaT =Bu

( 15)

and are left with the expression

!j

aJaT dr + IK =!juT Qu dr + IK

(16)

3 THE CASE OF LINEAR DYNAMICS

419

which the path x( ·) must minimise, subject to the constraints indicated above. But relation (15) and expression (16) are nothing but the plant equation and the cost function of the deterministic version of the control problem, and the minimising path nothing but the optimally controlled path for this version. 0 This may seem like a disappointing collapse. It is indeed an indication that one must refme the large-deviation approach if one is to capture stochastic effects (or, rather, tl:tose stochastic effects which would manifest themselves for a process which was not completely LQG). However, it also indicates interesting connections. The path which minimises control costs between prescribed initial and terminal points in the deterministic version is the most probable path between these points in the uncontrolled stochastic version (3~ The most probable first passage path in this second version corresponds to a constrained path in the first: that which does not enter f/ before termination. 3 THE CASE OF LINEAR DYNAMICS Suppose that the plant equation takes the linear form

x= Ax+Bu+c::,

(17)

with, as ever, N = ,..- 1J. Then the minimal value of the rate function for passage of the uncontrolled process from~ to {takes the form 1

D(~,{) = infsupj [oT x- oT Ax- !oTJo] dT. a t x

It follows then from the control-equivalence proved in the last section and the calculations of Section 7.8, that, in the case when the free path does not meet f/ prematurely, this has the evaluation

(18) Here s = 7 - t and

(19) In expression (18) we recognise just the ,..- 1 times the expression in the exponent of the distn'bution of i = x(t) conditional on the value x of x(t); a Gaussian distribution with expectation eAsx and covariance matrix ,..- 1 V(s). In the controlled/deterministic version we would see V(s) as the control Gramian. If K. is large then it is shown in Whittle (1995) that the first-passage density (of { conditional upon ~) for the uncontrolled process x = Ax + e is to the first approximation given by

(20)

420

CONTROLLED FIRST PASSAGE

and to the second approximation given by /(~1~)

=

p(K/27rr/21 V(s)l-l/2e-"'n(.;l)+o(l).

(21)

where pis a Jacobian term related to the transformation from x(t) for given t to a set of coordinates for~ on the surface of!/. That is, to the normal density of x conditional on the value of x for fixed s, corrected by the factor p. For example, if the stopping time is prescribed and we take (21) as giving the distribution of x(t) for prescribed t then p = I, trivially. If the process stops at a prescribed value of x 1 and one takes (21) as the density of (x 2 , x 3 , ... , Xn, t) at termination then p = ld.X1/dtl evaluated at the termination point, where i(t) is the most probable path. However, these assertions are subject to the proviso that the most probable path from~ to~ (i.e. the Hamiltonian path determined by (22.39)) should not enter Y before termination at { Should it do so, then for the coarser approximation (20) one must replace the path by that most probable path which avoids Y before the final encounter at ~- This will consist of free segments separated by grazing encounters withY, just as for the deterministic control problems of Sections 7.87.10. In such a caseD(~,~) must be replaced in (20) by the evaluation for the!/avoiding path. In the case of assertion (21) the concept of 'the most probable path which avoids Y before termination at~, may well be too simple; certainly such a path will clear Y by some margin rather than graze it. We shall in fact consider a number of cases which are simple enough that the first passage distribution is known, and for which simple recommendations on the margin of clearance can be deduced. We shall use the standard notation for the normal density and integral:

~(x) =

1:

¢(y) dy.

The asymptotic expression 1 - ~(x) = ¢(x)[x- 1 + o(x- 1)] for large positive x will be found useful. In Exercise 1 we derive the more refined version ¢(x) 1 _ ~(x)

=X

+ X -1 + 0 (X -1) .

(22)

Exercises and comments

(1) We can write

I

~(x)

~(x)

=

= whence (22) follows.

1oo exp[!(~-~)]dy= Joroo exp(-xu-!u )du 2

x

roo e-xu(l- !u

lo

2

+ .. ·)du = [x- 1 -

x- 3 + o(x- 3 )],

4 LANDING THE INERTIALESS PARTICLE

421

4 LANDING THE INERTIALF.SS PARTICLE Consider the stochastic version of the 'landing problem' for the inertialess particle discussed in Section 7.9. The height x of the particle is governed by the plant equation x = u + e. It starts from a height x at time t and the aim is then to bring it to zero height in such a way as to minimise the expected value of the cost Qtrdu + IK(t + s), where t +sis the moment oflanding. The hypotheses of the last section are satisfied, with 1t = 1/ N Q, and we have

! J/+s

D = D(e, ()

= x2j2NQs.

The only random coordinate of the termination point is its time, t + s. We require then to know the distribution of s, the time taken for first passage from height x to zero height for the uncontrolled process x = €.. But this distribution is well known; it has density f(s) =

~e-:ilf2Nss-3f2

(s

Vfffii

;;?;

0).

(23)

Note that this is precisely what would be given by formula (21), which is exact in this case. The most probable path from (x, t) to (0, t + s) for prescribed sis the straight-line path x(r) = (t+s- r)xjs, so that the Jacobian p of formula (21) equals xis. Multiplying this by the probability density of x = x(t + s) for prescribed s at x = 0 we obtain exactly expression (23). Relation (3) for the value function of the controlled process now becomes

1/J{e)

:= e-F(x,t)/NQ =

~

{

Yfffii lo

00

e-C(x,t,.J)jNQ ds

(24)

where C(x, t, s) is the'cost'expression

C(x, t, s) = ; :

+ K(t + s)

(25)

Recall now the conclusions of Section 7.9 for the deterministic version. The value function was exactly the minimal value of C(x, t, s) with respect to s. If the minimising value is s (a function of x and t) then t + s is the optimal landing time and the optimal control value at (x, t) is

u = -xjs.

(26)

Note that t = t + s, the optimal landing time, is constant on the optimal path. Returning to the stochastic case, let us define the integral

I[g(s)]

=

1oo

g(s)e-C(x,t,s)JNQ ds,

for a functiong(s). We see then from expression (24) that the optimal control rule at (x, t) is

422

CONTROLLED FIRST PASSAGE

(27) This is close to the deterministic rule (26), but differs from it in two interesting respects. Firstly, there is the term Nix, which impels the particle to gain height whenever it is dangerously low. This is exactly the cautionary effect we were seeking: the appreciation of the danger of a premature landing if height is low. The effect is of course absent in the deterministic case x = 0. The other point is the replacement of the factor s- 1 in (26) by I (s- 1) I I ( 1), an average of s- 1 for the distribution (23) weighted by the factor exp( -IKI NQ). This average will indeed be close to s -I if sis not too small and N is small. However, in the final moments of landing the two evaluations will differ, as they must if the cautionary heightgaining term Nix is to be nullified. One can reasonably define the 'ideal approach path' as that for which the height x(t) at timet is the value of x minimising F(x, t). Since u is proportional to Fx then this is just the (x, t) locus on which u is zero (although control action is needed to keep it on this locus). Suppose that topt is the truly optimal landing point, in that it is the value oft minimising IK(t). Then topt is certainly the ideal time at which to terminate, so that the control (27) approximates (NIx) - xI (to - t) which is zero on the locus

x(t) = JN(topt- t).

(28)

This then describes the ideal approach path (necessarily for t < t0 p1). For N = 0 the ideal approach is a ground-skimming one at infinitesmal height-to bring the particle from this height to zero height requires only infinitesmal control cost. If N > 0 then the appreciation of hazards induces the square-root approach of (28). The approach is steepest at the end of the path, as one tries to drop on to the optimal point. Once we introduce inertia then this dropping will of course be modified by a levelling-out at the very end.

5 OBSTACLE AVOIDANCE FOR THE INERTIALESS PARTICLE Suppose that it is not a question of landing the inertialess particle, but oflifting it over an obstacle which will be encountered at time t 1• (We assume, as ever, constant horizontal velocity.) This could be regarded as a mountain peak which is so sharp and high relative to the surrounding terrain that, if one surmounts the peak itself, then one certainly clears all other hazards in the neighbourhood. We can then regard t 1 as a kind of 'termination time', in that flight truly terminates if one hits the mountain, and continues with a known future cost if one clears it. Since t 1 is known then the only first-passage coordinate which is unpredictable is the height y of the particle at time t 1• Conditional on a current position (x, t) the random variable y is normally distributed with mean x and variance Ns, wheres is, as ever, the time to go, t 1 - t.

5 OBSTACLE AVOIDANCE FOR THE INERTIALESS PARTICLE

423

Formula (3) for the value function F(x, t) of the controlled process then becomes

'1/J(x, t) = e-F(x,t)jNQ = ___ 1 J2nNs

1

00

e-C(x,t,y)jNQ

dy

(29)

-oo

where

C(x,t,y)=

Q(y- x)2 +IK(y) 25

(30)

and IK(y) is the cost incurred if the particle is at heighty at time t 1• The optimal control is

u = N'I/JxN = y(x,s)- x s

(31)

Jye-c dy y(x,s) = J e-C dy .

(32)

where

That is, y(x, s) is an average ofy based on its distribution weighted by the factor exp( -!K/NQ). If one were aiming for a valuey(t 1) = Y1 in the deterministic case then the optimal control would be u = (Yl - x)js. The value y(x,s) can thus be seen as a provisional aiming point. This may seem a simpler example than that of the landing problem of the previous section, in that the hazard is localised rather than distributed (over the surface of the ground), and the first-passage distribution is trivial. However, the fact that IK may be discontinuous (reflecting the very different contingencies of clearing or not clearing the peak) means that more care is needed in the evaluation of the integrals of(32) for small N If the mountain is of height h then the simplest terminal cost prescription would be

(x:::; h) (x > h)

(33)

where K is a large positive constant. That is, all collisions are equally disastrous, all clearings equally happy. The optimal control rule in the deterministic case is then that u should equal (h - x) j s for x :::; h and should be zero otherwise. The interest lies in seeing how rule (31) expresses proper stochastic caution in differing from this. In this special case evaluations (29) and (31) become e-F(x,t)/NQ

u=

= 1 _ 4>(()

+ 7]4>(()

(1- 7J)JNTS¢(() 1 + (7J- 1)4>(()

-'-=---:~--::-:---::-'-:~

(34)

(35)

424

CONTROLLED FIRST PASSAGE

where (=(h-x)j,;Ns and TJ=exp(-KjNQ). We look for approximate evaluations of these expressions for small N, and, almost independently of this, for large K. For small Nand fixed K we recover the same feature as that observed in Section 7.9: that one will choose to crash on the mountain rather than make the exertions necessary for escape if condition (7.58) holds. We indicate the argument in Exercise 1. However, we see the significant results more clearly if we let K become indefinitely large and so TJ indefinitely small. If we simply set TJ = 0 in the expressions above then we obtain non-degenerate conclusions. This indicates, as for the example of Section 10.7 (see equation (10.36)), that control in the neighbourhood of the peak is vigorous enough that collision and the infinite penalty it would bring are avoided. (More specifically, the probability of collision is zero in the strong sense that this contingency of infinite cost contributes nothing to the expectation of cost.) Suppose ( large and negative. This means that (x- h)j,;Ns is large and positive, so that one is well above the peak relative to the variation one can expect in the time remaining. Then formula (35) yields (in the case TJ = 0)

u"' JNTS¢(()

=

~exp[-(h- x) 2 j2Ns]

(36)

which is indeed very small. Suppose ( large and positive, so that one is well below the peak relative to the variation one can expect in the time remaining. Appealing to relation (22) we then have I1:Ti: h- X N urv VHJ->((+Cl) = - - + - - . s a-x

(37)

That is, the optimal control is exactly the optimal deterministic control plus a stochastic 'lift' term reminiscent of that which we saw in the control (27) for the landing problem. It may seem strange that this lift term should become stronger as h- x becomes smaller, i.e. as one begins to come up level with the peak. However, it can be argued that the first (deterministic) term in the control rule (37) provides adequate lift for x well below h, and it is when x begins to come up level with h that it needs to be supplemented. It is moreover just then that the assumption oflarge (begins to fail. However, there is a version of the control rule which, although equivalent to (37) as far as terms of order N, carries real significance.

Theorem 24.5.2

At values ofxfor which (is large and positive the optimal control is h+Nd-x u = -+ o(N). s

for small N, where d = sj(h- x) is invariant on the optimal deterministic path.

(38)

5 OBSTACLE AVOIDANCE FOR THE INERTIALESS PARTICLE

425

That is, the effect of plant noise on the optimal control is that one behaves as though the height of the mountain had been increased by an amount Nd, this intended clearance being essentially constant until one is within distance o(N) of the peak. The steeper the gradient g =a- I ofthe deterministic path, the smaller will be the safety margin N d thus allowed.

Proof If we equate the expressions (37) and (38) for the control then we obtain the evaluation asserted for d. The point of the theorem is that this correction is constant on the deterministic path, and so subject only to slow variation on the stochastic path. D Exercises and comments (I) For ( = (h- x)/ ffs large and positive expression (34)yields e-F(x,t)fNQ

"'maX

2

[e-(h-x) /2Ns,e-KfNQJ

if we retain only the dominant factors: those exponential in 1/N. The second term 2 in the square bracket will dominate if Q(h- x) /2s > K. This is exactly the condition deduced in (7.58) for the option of crashing to be cheaper than that of striving to clear the peak. (2) Suppose that the specification (33) of the 'terminal' cost is modified to K(y) = Qy 2 /2s 1 for y > h. This would represent the further control cost involved if one were required to bring the the particle to zero height a time s 1 after having cleared the peak. There is then an incentive to clear the peak by no more than is necessary Show that the formula analogous to (34) is e-F(x,t)/NQ

=

~exp[!((?- (f)][l- !)((2)] + exp(-K/NQ)!)((l)

Vs~

where (t = (h- x)j../Ns

(2

=

(h -~) /ffl s+ s1

and

>. =

~. s+s1

That is, ( 1 and (2 are respectively proportional to the current distance below the peak and the distance that one would be below the peak at time t1 if one continued on the direct flight path to the final landing point at time t 1 + s1. Note that

the difference in control costs between the straight-line paths to the final landing point from the peak and from the starting point respectively. Show that in the case K = +oo the optimal control is

426

CONTROLLED FIRST PASSAGE

x s+s 1

N ..f>..

( (2)

s

1-4?((2)'

U=---+---:--'-~"':"'"7"

The two terms represent respectively the control that would take the particle to the final destination by the straight-line path and the correction needed to lift it over the peak. Show that in the case of ( 2 large and positive (when the straightline path would meet the mountain) the formula analogous to (37) is

u = _h_~_x + N[h(-s+_SI_sl)- xrl+o(N).

{39)

The first term sets a course for the peak. The second introduces a lift inversely proportional to the amount by which the particle is currently below the straightline path passing through peak and destination. We can recast the control rule (39) in the more significant form (38), but with the constant d now modified to d = (g + hlsi)- 1 where g = (h- x)ls is the constant gradient of the optimal deterministic approach path. The term hI s1 is correspondingly the descent rate required on the other side of the mountain. The effect is then again that one aims to clear the peak by an amount Nd, this intended clearance showing only slow variation on the optimal approach path, but decreasing as the steepness of either required approach or subsequent descent is increased 6 CRASH AVOIDANCE FOR THE INERTIAL PARTICLE Consider the stochastic version of the inertial particle model of Section 7.10 in which the applied forces are the sum of control forces and process noise, so that the stochastic plant equation is

x= v,

v=

u+t:.

Then the condition (2) is again trivially satisfied, with K. = 1I NQ. It would be interesting to consider both landing and obstacle-avoidance for this model. However, the most natural first problem to consider is that analysed in Section 7.10; that of pulling out of a dive with minimal expenditure of control energy. That is, one begins with x > 0, v < 0 and wishes to minimise expected total control cost up to the moment when velocity v is first zero under the condition that height x should be non-negative at this this point (and so at earlier points~ The moment when v becomes zero is to be regarded as the moment when one has pulled out of the dive. In a first treatment, costs incurred after this moment are not considered Recall the results of the deterministic case. If no control is exerted then the particle of course crashes after a time= xlv. The optimal control is u = 2V2 l3x which brings the particle out of its dive (and grazing the ground) after a time

-3xjv.

6 CRASH AVOIDANCE FOR THE INERTIAL PARTICLE

427

Relation (4) now becomes e-F(x,•)/NQ = 1 _ p

+ Pe-KfNQ

where P = P(x, v) is the probability of a crash (i.e. the probability for the uncontrolled process that height is negative when velocity is first zero) and K is the penalty incurred at a crash (assumed constant). If we assume K infinite then this reduces simply to 1 - P, and the optimal control is

u = -NP,/(1- P).

(40)

Let us use s to denote elapsed time, with x(O) = x, v(O) = v. Then x(s) is normally distributed with mean x + vs and variance N; /3, and the probability that x( s) is negative is ci>( (), where

( = -(x+ vs)J;;.

(41)

Now, the point at which stochastic variation from the limit- optimal path is likely to take the path into !/is, by the discussion of Section 2, just the grazing point of the optimal deterministic path, which we know to be at s = -3xjv. We see from (41) that this is also the value of s which maximises (, and so maximises the probability that x(s) ~ 0. One is then led to a conjecture: that the grazing points of the optimally controlled deterministic path are just the points which maximise the probability that the uncontrolled stochastic path has strayed into !/. The conjecture is true in a class of cases which includes this one, as we shall see from the next section, but not generally. The value of (at this grazing point is

(=~ [vii 3Ylix' and we shall have P(x, v) ""P(( ~ 0) = ci>((). (Strictly, it is the logarithms of these quantities whose ratio is asymptotically unity.) To within terms of smaller order in N we thus have

u=

~ ¢(() •.

v~ 1- ci>(()

(42)

If (is large then we can appeal to the approximation (22) and deduce that

2y2

3N 2v

u"'---.

3x

(43)

This again follows the pattern: optimal deterministic control plus a cautionary height-gaining correction. In this case the correction is inversely proportional to current velocity rather than current height. As for the obstacle-avoidance

CONTROLLED FIRST PASSAGE

428

problem of the last section, the form of the correction term is somewhat counter. intuitive, but is explained in the same way. It can also be given the more' illuminating form of preservation of a near-constant clearance.

Theorem 24.6.2 At values ofx and vfor which (is large and positive the optimal control is

2v2 u = 3(x- Nd)

+ o(N)

(44)

for small N, where d = -9.x2 / 4vl is invariant on the deterministic path. That is, the effect of plant noise on the optimal control is that one behaves as though one had to miss the ground by a margin of Nd, this margin being essentially constant until one is within distance o(N) of the ground. The stronger the control needed to pull out of the dive, the smaller will be the safety margin Ndthus allowed.

Proof If we equate the expressions (43) and (44) for the control then we obtain the evaluation asserted for d. It follows from the analysis of Section 7.10 that vx- 213 is indeed constant on the deterministic path. The point of the theorem is then again that this correction is constant on the deterministic path, and so sub0 ject only to slow variation on the stochastic path. 7 THE AVOIDANCE OF ABSORPTION GENERALLY The conclusions of the last section generalise nicely. Suppose that we consider the stochastic version of the general avoidance problem of Section 7.11. That is, the plant equation has the linear form (17) with control cost! JuT Qu dr, and the stopping set sis the half-space aT x ~ b. As emphasised in Section 7.11, this last assumption is not as special as it may appear. Let us normalise a so that lal = 1; a modification of b to b + d then shifts the plane a distance d in the direction normal to it. As in Section 7.11, we shall define the variable z = aT X. Since the process is time-homogeneous we may as well take the starting and termination points as t = 0 and t = s, so that € = (x, 0), €= (.X, s), and sis the time remaining to termination at x. We take the termination point s as the grazing point; the first time at which, if no further control were exerted, the uncontrolled deterministic path would henceforth avoid s. We may as well then assume that initial conditions are such that there is such an s; i.e. that x is such that some control is necessary to avoid s. We again make the assumption that the relation N = K- 1J holds, and assume"' large and J fixed. We have then

1/;(x) := e-~
= E[e-~
= P[z(t)

> b; 0 < t ~ slx(O) = x]. (45)

7 THE AVOIDANCE OF ABSORPTION GENERALLY

429

Here s is again the first time at which the path would no longer enter s subsequently if it continued in an uncontrolled deterministic fashion, and F(x) is the minimal cost of a path starting from x which avoids !I' up to that point. Note that s is a random variable, now determined for the uncontrolled process. The last equality holds in (45) if we assume that K = +oo, so that entry to !I' is to be avoided at all costs. One might now conjecture that

1/;(x) = e-~
,....,

inf P[z(s) s>O

> bix(O)

= x].

(46)

As we shall see in a moment, the conjecture is not valid unconditionally, but let us assume it valid for the moment. The random variable b- z(s) for given s is 2 distributed normally with mean tl and variance a(s) I"'· where

a= a(s) =

JaTV(s)a.

where V has the evaluation 14. Thus, (46) becomes

(47) where ( = .,(K,tlla, and s maximises(. That is, the extremising value of sis exactly the values determining the optimal grazing point in the deterministic case. Now, the conjecture (46) is equivalent to the conjecture that the asymptotically (large K-) most probable !I' -avoiding path grazes !I' just at the time s determined by the minimising condition in (46). Since, by the argument of Section 2, this path is just the optimal deterministic !/'-avoiding path, the conjecture is effectively that the grazing point of this path is that which maximises the cost of passage to !I' with respect to time taken. But we know from Theorem 7.11.2 that this assertion is valid under the conditions that the process is controllable and aT B = 0. This last condition amounts to the assertion that control does not affect the statecomponent aT x directly. For the uncontrolled stochastic process it amounts to the assertion that aT Na = 0, or that aT ~:is zero in mean square. We deduce

Theorem 24.7.1 Suppose that N = K-BQ- 1BT, that aT B = 0 and that the deterministic system [A, B, ·]is controllable. Then.forlarge K.: (i) The optimal !I'-avoiding control at x is

(48) where ( = .,(K,tl I a, and sis given the values which maximises(. (ii) This can be further approximated

(49)

430

CONTROLLED FIRST PASSAGE

where sis given the values and

(50) The quantity dis then to be regarded as the clearance which is a imedfor at the grazing point; it is constant along the optimal deterministic path. Proof Assertion (i) follows from the discussion before the theorem and its conclusion (47). Assertion (ii) follows, as previously, by using the approximation (22) in formula (48) and then recasting the O(x:- 1) correction term as a perturbation dof b. Since d depends upon current coordinates only through 6./ which is the terminal value of ,\ for the optimal deterministic path, it is indeed invariant on

cr,

~~

0

In the control rule (49) we indeed recognise the deterministic rule (7.78) with b modified to b +d. Notes on the literature The material of sections 1, 3 and 4 appeared in Whittle and Gait (1970), the essential result being of course the reduction expressed in Theorem 24.1.1. The usefulness of the logarithmic transformation of F in the control context has subsequently and independently become apparent to other authors; e.g. Holland (1977), Fleming (1978) and Benes, Shepp and Witsenhausen (1980). The material of the other sections is new, as far as the author knows.

CHAPTER 25

Imperfect Observation; Non-linear Filtering The conclusions of Section 23.2 transfer to the case of imperfect observation, with the consequence that we can obtain a risk- sensitive maximum principle for this case. The conclusions are quite illuminating. It remains true, as we have noted repeatedly, that the large-deviation approximation is in a sense a crude one, in that it gives inadequate expression to the stochastic character of the model at points where the limit-optimal path bifurcates or becomes subject to special constraints or penalties. However, the large deviation approach gives what is in fact rather a good treatment of state estimation, perhaps just because these phenomena do not then arise. We consequently give some space to the question of pure estimation in this general context: non-linear filtering. 1 SPECIFICATION OF THE PLANT/OBSERVATION PROCESS

We suppose that the observation y is, like the process variable x, a vector. It turns out to be natural to work to some degree in terms of the integrated observation z(t) = y(r) dr. In these terms the familiar linear white-noise model (12.26) would then become

J'

x=Ax+Bu+E,

z= Cx+ry.

(1)

so that both plant and observation relations are cast as flrst-order differential equations. It also has the effect that z is a regular stochastic process, whereas y, with its white noise component, can only be understood in a generalised sense. Of course, model (1) is a very special case of the Markov models we now consider. We assume that the plant/observation process has a time-homogeneous Markov character, and that the observation process is wholly subsidiary to the plant process in that the distribution of the increments ox and oz of x and z in the time interval (t, t + ot) conditional on plant, control and observation histories at time t is in fact conditioned only by x( t) and u( t). The plant/observation process is then specifled by the joint DCF

H(x, u, a, ,B)= ~i~m(ot)- 1 {E(e0 6x+tHzlx(t) = x, u(t) = u]- 1}.

(2)

Note that H(x, u, a, 0) = H(x, u, a), the DCF of the plant process already introduced in Chapter 23. The joint DCF for the linear specification (1) is

432

IMPERFECT OBSERVATION; NON-LINEAR FILTERING

H(x, u, a, ,B) = a(Ax + Bu)

+ ,BCx + i [a ,B] [ ~

t Hfi~]'

(3)

under our standard assumptions on plant and observation noise. When it comes to scaling of the joint process we shall assume that the joint DCF has the form ~tH(x, u, aj~t, ,8/~t) in terms of a unsealed DCF H Here K. is the scale parameter, which will be assumed large. We shall consider the joint process from time t = 0; a matter of convention rather than a real constraint. We must have an initial condition, however, in that we must have a distribution for x(O) (understood to be conditional on the information state at time t = 0). We shall assume that this is subject to the same scaling, in that there exists a rate function Do(x) such that P(x(O) E .PI)= exp{-K. inf Do(x) xE.Jil

+ o(~t)}

(4)

for large K. and appropriate .Pl. 2 'CERTAINTY EQUIVALENCE' AND THE SEPARATION PRINCIPLE We consider the risk-sensitive criterion E'lf( e-..SC) with cost function (23.1) and the corresponding (unnormalised) total value function G(W(t)) = extE'lf[e-..sciW(t)].

(5)

'If

Here the operation 'ext' indicates the taking of a maximum or a minimum according as fJ is positive or negative, and W(t) is the information available at time t. G obeys the dynamic programming equation G( W(t)) =ext E[G( W(t + 6t)l W(t), u(t)J u(t)

+ o(t5t).

(6)

However, as in Chapter 12, it is more convenient to work in terms of the function

J( W(t))

=/( W(t))G( W(t)),

(7)

where /( W) is the density function of the information W relative to a Itindependent measure m on histories W. In terms of this the dynamic programming equation (6) becomes

J( W(t)) =ext

u(t)

f J( W(t + 6t)) + o(6t),

}y

where Jh indicates an integration over {y(T); t that the extremising u( t) is optimal. Define also the associated quantity

(8)

< T ~ t + c5t} and we can assert

J* ( W( t), x( t)) = f( W( t), x(t) )E[e-..ec'1 W( t), x( t)]

(9)

2 CERTAINTY EQUIVALENCE' AND THE SEPARATION PRINCIPLE

433

where C' is the cost up to time t. The expectation is independent of 1r, because all control values occurring are in the past and have been determined Let us provisionally assume that an asymptotic expression

J*( W(t), x(t))

= e-I
(10)

holds, where P is independent of K.. Both the validity of the assumption and the evaluation of P will follow from large-deviation theory, as we shall see in the next section. Finally, rec3.Il the definition (23.7) of F(x, t); the scaled and normalised future value function for the same control problem, but with perfect state observation. The quantities P and Fare in fact asymptotic equivalents of the past and future stresses defmed for the LEQG problem in Chapter 16. We shall prove what looks like a certainty equivalence principle in terms of these; for the state-structured case only, but greatly relaxed from the LEQG case in that it holds asymptotically for any model for which large-deviation assertions are valid. However, the assertion is so much cruder than a true certainty equivalence principle (in a sense to be explained) that we shall not use the term, but shall speak rather of 'stress-extremising:

• Theorem 25.2.1 The stress-extremising principle. Assume the exponential-ofcost criterion E"'( e-..ec), the scaling conditions listed above and the provisional validity of(JO). Then, for large K.." (i) The function J has the asymptotic evaluation J( W(t)) ,....,

J

e-l
exp{~~:i~f O[P( W(t), x) + F(x, t)]}.

(11)

(ii) Suppose that the optimal closed-loop form of u(t) is u(x(t), t) in the case of perfect state observation. Then the optimal value in the case of imperfect state observation is u(x(t), t) to within a term o(J) in~~:, where x(t) is the minimising value ofxin (11). Ifwe write the final expression for Jin (11) as exp{ -~~:OS(t)} then one can see

S(t) as a 'stress' extremised with respect to all quantities not observable and all control decisions as yet unmade at time f. The components P and F then indeed appear as past and future components of stress and the minimising value x of x as the 'minimal-stress' estimate of the current state value x( t). The significant fact is that the future stress Fis nothing but the normalised value function for the case of perfect observation, and it is this which implies assertion (ii), which looks so much like an assertion of certainty equivalence.

*Proof The proof is inductive. Relation (11) certainly holds at the horizon point h; suppose it true at a timet + 6t ~ h. If u( t) = u then the relation

434

IMPERFECT OBSERVATION; NON-LINEAR FILTERING

1

J*(W(t+bt},x(t+6t})

=

j J*(W(t},x)e-KIIc(.:o:,u)btf(x(t+bt}ix(t} =x,u(t) = u) dx

follows from the definition (10) of J*. In virtue of this we can write the version (8) of the dynamic programming equation (6) as

Je-~~:9P,E[e-KIIc(.:o:,u)6~-t~:9F(.:o:(t+6t),t+6t)ix(t) = = e~t J ~tOA(x, + o(~t6t)

J(W(t)),....., e~t

x,u(t)

e-KII(P,+F,)[1-

= u]dx (12)

u, t) 6t] dx

say, where we have written P( W(t), x) and F(x, t) as P1 and F1• The extremising value of u in (12) is the optimal value of u(t}, to within a term o(1) for large""· For given"" we must assume 6t so small that ~t6t is small. However, for large "" the dominating contribution to the integral in (12) will come from the value i ofxminimising O(P1 + F 1}.We then have

l(W(t}, t),....., [1-

~tOi~f A(x, u, t) 6t]

Je-~~:B(P,+F,)dx+ o(~t

,. . ., [1- ~tOinf A(i, u, t) 6t]e-~<11 inf,(P,+F,) u

6t}.

+ o(~t 6t) + o(1).

( 13)

where the final remainder term is one that is small for large ""· Here again the infimising value of u is asymptotically optimal. But the recursion which the dynamic programming equation (23.8) implies for F implies that the minimal value of A~, u, t) with respect to u is zero, attained at u = u(x, t). Both assertions of the theorem then follow from (13). D It is assertion (ii) which has the appearance of a certainty-equivalence statement: that the optimal control is (in the limit of large ~t) that for the case of perfect observation with an estimate i(t) substituted for the unobserved value of x(t). In the risk-neutral case i(t) is just the value of x minimising P( W(t), x); the large-deviation form of the most probable value. In the risk-sensitive case the estimate i(t) depends also upon both past and future costs in a manner familiar from analysis of the LEQG case in Chapter 16. However, a true certainty-equivalence interpretation does not hold in general. To see this, consider the risk-neutral discrete-time case and let i 1 be the projection estimate of x 1 based upon W 1 = ( Y1, U1_ 1 )0. If Q(x) is a quadratic function then

where Qu is the matrix of second-degree terms in Q(x) and V1 is the covariance matrix, conditional on W11 of the estimation error i 1 - x 1• The proof of certainty equivalence in the LQG (risk-neutral) case relied on the fact that the term

435

3 THE RISK-SENSITIVE MAXIMUM PRINCIPLE

tr(Oxx V 1) could be dropped from consideration because it was historyindependent. That is, it showed no dependence upon W, and so none upon past policy. It is dropped from consideration in the risk-neutral version of the present case because it is small, of order K.- 1• It will in fact be historydependent in general, because of plant/observation non-linearity or because control actions affect the quality'of observations. The approximation made is virtually the brute-force substitution of for x on the grounds that the difference is small. This does not mean that the theorem is without content. For one thing, an appropriate estimate is deduced. For another, the effects of risk-sensitivity are discerned. As in the LEQG case, the separation principle manifests itself in that the provisional specification of the value x of current state x(t) allows one to decouple the evaluations of past stress P 1 and future stress F,. These evaluations may be said to be concerned with estimation and control respectively, in that the evaluation of P, sums up the effect of current information W(t) and the evaluation of F1 implies the determination of the perfect-observation optimal control rule u(x, t). However, the actual extimate and control rule u(x, t) are then derived by a recoupling: the minimisation of O(P, + F,) with respect to x = x(t).

x

x

3 THE RISK-SENSITIVE MAXIMUM PRINCIPLE Just as for the LEQG case, the stress-extremising principle derived in the last section enables us to characterise the optimal control problem in time-integral form. This form is made explicit by large-deviation evaluations of past and future stress, P and F, which leads then to a risk-sensitive maximum principle for the case of imperfect observation. We have hitherto worked in terms of a fixed horizon h, but now state the obvious generalisation to the case when termination occurs at the first entry of = (x, t) to a prescribed stopping set f/, with terminal cost 11\\(~) for terminal value~-

e

Theorem 25.3.1

U=

Define the time-integral

e- 1Do(x(O)) + li[c(x,u) + ).Tx+ p.Tz- e- 1H(x,u,O>..T,ep.T)] dT+K(~) (14)

e

Then, in the limit of large K, the optimal value of u( t) in the cases () > 0 ( < 0) is that determined by seeking the infimum of 0 with respect to controls undetermined at time t, the supremum (infimum) with respect to the conjugate variables .X(·) and J.L( ·)and the infimum (supremum) with respect to variables unobservable at timet.

436

IMPERFECT OBSERVATION; NON-LINEAR FILTERING

In the case of negative 0 the extremal operations must be applied sequentially in time, as indicated in equation (16.8).

Proof Split the path integral Rinto the integrals R' and 0, for the separate ranges r < t and r ~ t. Then we shall prove the theorem by establishing the asymptotic identifications P(W(t),x) = ext0 1,

F(x, t) = extO,,

where the extremum is that specified in the theorem, but subject to the constraint x(t) = x. 0 The extremal conditions with respect to future y(r) yield J.L(r) = 0 (r > t). But then the expression for 0, reduces exactly to the expression for F(x, t) asserted in Theorem 23.2.1. The corresponding evaluation of P( W(t), x) follows by direct appeal to the rate-function evaluation (22.36). The course of the Markov process with state variable (x, z) for given u( ·) (not necessarily optimal) over the time interval 0 ::so; r ::so; t will have density f(X(t),Z(t)) = exp(-ttD(t) + o(tt)], where X(t) denotes x-history to timet, etc. and

D(t)

= Do(x(O)) +

sup

a(·), PO

( [o:.X + (3z- H(x, u, o:, /3)] dr.

Jo

The large deviation property then implies that (9) holds, with the identification

OP( W(t), x)

1 = inf[9C + D(t)] x(·)

where C 1 is the cost up to timet and the infimum is subject to x(t) = x. But this yields exactly the evaluation (14) if we set .i = y, o: = o>..T and (3 = OJ.LT. 0 The statement that one would generally recognise as a maximum principle follows if one writes down the stationarity conditions on the path integral (14). In terms of the effective Hamiltonian

these would be

X = 8Jff ).T = - 8Jff 8)...T' 8x

y = 8Jft'T 8J.L

(0 < T < t'

'J

J.L = 0 (t < r < t), Jff is maximal in u (t ::so; r < t) u is prescribed (0 ::so; r < t), y is prescribed (0 < r ::so; t) plus the end conditions implied by the theorem.

5 PURE ESTIMATION: NON-LINEAR FILTERING

437

4 EXAMPLES AND SPECIAL CASES Under the LQG assumptions of the usual quadratic cost function and joint DCF (3) one readily finds that the assertions of the last section agree with those derived for the state-structured LEQG case in Chapter 16, known in that case to be exact. If we go to the risk-neutral limit we find that the asymptotically optimal control is just u(x(t), t), where u(x, t) is the optimal risk-neutral perfect information control and x(t) the value of x minimising D( W(t), x)-the large deviation version of the conditionally most probable value. As indicated above, this conclusion rests on nothing more sophisticated than the facts that x(t)- x(t) has zero expectation and a covariance matrix of order ~-I. If one formally lets etend to zero in the integral of (14) one obtains what seems like an excessive collapse, in that e-l H(x, u, e>.. T' Oj.LT) reduces to >..Ta(x, u)+ J.L Tc(x, u), where c(x, u) is the expected value ofy(t) conditional on the past. That is, the process seems to reduce to determinism in observation as well as plant, which is surely too much! However, one will then be left with a term J.L T(y - c(x, u)) in the integral, whose extreme value with respect to J.L will be infinite unless y is exactly equal to its expected value c(x, u). This simply reflects the fact that the rate function D for the process over a given time interval is nonzero unless the process lies on its expected (deterministic) path, so that in other cases D/8 will have an infinite limit as e tends to zero. Consideration of the behaviour of D on its own is the theme of our final section: the consideration of process statistics unskewed by costs.

5 PURE ESTIMATION: NON-LINEAR FILTERING An interesting special case is that for which there is no aspect of cost or choice of control policy, and one is simply trying to infer the value of x(t) from knowledge of current observations W(t), possibly in the presence of past (and so known) controls. In the LQG case this inference is supplied by the Kalman filter, supplemented by the recursive calculation of associated matrices such as the covariance matrix V, as we saw in Chapter 12. The general problem is referred to as that of 'nonlinear filtering' by analogy with this case. If we specialise the material of this chapter we have then a large deviation treatment of the nonlinear filtering problem which is exact under LQG assumptions. Since the controls are known and have no cost significance we can write W({) simply as Y(t), the observation history. Consider the expression

D1( Y(t), x) = inf [Do(x(O)) x(·)

+

sup a(·),/J(·)

['[ax+ {3y- H(x, u, a, {3)] dr]

Jo

(15)

where the infimum is subject to x(t) = x. Then exp[-~D 1 ] is the large deviation approximation to the joint probability density of x(t) and Y(t) and so provides, to

438

IMPERFECT OBSERVATION; NON-LINEAR FILTERING

within a Y(t)-dependent normalising factor, the large deviation approximation to the probability density of the current state value x( t) conditional on observation history W(t). However, as we have seen in Chapters 12 and 20 and Exercise 22.7.1, the updating of such conditional distributions is more naturally carried out in transform space. We shall in fact deduce a large deviation form of a forward updating equation for the corresponding MGF M(a, t)

= E[eax(r)l Y(t)].

( 16)

*Theorem 25.5.1 Suppose that the Markov process {x(t), z(t)}, where i = y, has DCF K,H(x, u, /'i,- 1a, K,- 1/3) with known past controls u. Then for large /'i, the conditional characteristic function (16) has the form

( 17) fort

> Oifit has this form fort= 0. The unsealed CGF'IjJobeys the updating equation &rj;(a, t) =a( a)- a(O)

( 18)

8t

where

(19) Proof Relation (17) will hold at t = 0 under the assumption (4), when we shall have

(20)

'1/J(a, 0) = sup[ax- Do(x)]. X

The function '1/J(a, t) will differ from ll!(a,t) = sup[ax-D 1 (Y(t),x)] X

=sup [ax(t)- D 0 (x(O))x(-)

sup a(·), {3(-)

([ax+

Jo

f3y- H(x, u, a, /3)] d-r]

(21) exp[K,\li(K,- 1a,

only by an additive term, independent of a. The point is that t)] is the large deviation evaluation of the conditional MGF of x(t) times the probability density of the observation history Y(t). We must divide out this latter to evaluate the MGF; equivalently (22)

5 PURE ESTIMATION: NON-LINEAR FILTERING

439

Note the distinction in (21) between the vector function a{r)(r ~ t) and the vector a; we shall see that there is an effective identification a(t) =a. The extrema with respect to x( ·) and {3( ·) are unconstrained. A partial integration of (21) and an appeal to (20) allows us to rewrite 1¥ as

1¥(a, t) =sup x(-)

inf ['1/J(a(O), 0) 130

a(·),

+

Jo['[ax- {3y + H(x, u, a, {3)] dr]

(23)

where the extremisation with respect to x(t) has indeed induced the constraint a( t) = a. Now, by the same methods by which the expression (23.10) for the value function was shown to satisfy the approximate dynamic programming equation (23.9), we deduce that expression (23) obeys the forward equation in time

~~ = i~f[n(~: ,u,a,{3)- {3y]

= a(a).

Relations (22) and (24) imply that '1/J satisfies (18)/ (19).

(24)

0

Equation (18) is indeed the precise analogue of the dynamic programming equation (23.9). However, the exact version of (18) is not, in general, the precise analogue of the the exact version (23.8) of (23.9). Relations (18) and (19) provide the updating equation for the conditioned xq) distribution and so can be regarded as providing the large-deviation equivalent of the Kalman filter for the general case. In the LQG case, when H has the evaluation (1), then (18) is exact. We find that (18) is solved by

'1/J(a,t) = ax(t) +!aV(t)aT if x( t) and V( t) obey the familiar updating equations for the state estimate and the covariance matrix of its error: yet another derivation of the Kalman filter and Riccati equation! If we regard the state estimate x( t) as again the conditionally most probable value, then, in the large deviation approximation, it will be the value maximising infa[~~:'I/J(~~:- 1 a, t)- ax]. The two extremisations yield the conditions a= 0 and x(t) = [&tp(a, t)/ooJa=O• SO that x(t) is nothing but the large deviation evaluation of the expected value 6f x(t) conditional on Y(t). The fact that the mode and the mean of the conditional distribution agree is an indication of the implied supposition of a high degree of regularity in the distribution. Relations (18) and (19) are fascinating in that they supply the natural updating in the most general case to which large deviation methods are applicable. However, they do not in general supply a finite-parameter updating (i.e. '1/J(a, t) does not remain within a family of functions of a specified by a finite number of tdependent parameters). One is updating a whole distribution and will have such a reduction only in fortunate cases; in others it can only be forced by crude approximation. The LQG case of course supplies the standard fortunate

440

IMPERFECT OBSERVATION; NON-LINEAR FILTERING

example: the conditional distribution remains Gaussian, and is parametrised by its time-dependent mean and covariance matrix. However, for an interesting non-Gaussian example, consider again the problem of regulating particle numbers in a chamber, associated with the unsealed DCF (22.26). Suppose that one cannot observe the actual number n of particles present in the chamber, but can only register particles which collide with a sensor, each particle in the chamber having probability intensity v of registering. One then has essentially a running count m( t) of the number of registrations. Defining the normalised cumulative observation z(t) = m(t)/ K-, we then have the normalised DCF H(x, u, a, /3) = u(ea- 1) + px(e-a- 1) + vx(e13 - 1). for x

= nj K- and z. Thus cr(a), as defined by (19~ has the evaluation cr(a)

= u(ea-

1)

+ p'I/J'(e-a- 1)- v'I/J' + y- y logy+ y log (v'I/J')

and the updating relation (18) becomes

0:: = u(ea -

1)

+ p'I/J

1 (

e-a - 1) - v( 1/J' -

1/J~) + y log (1/J' 11/J~)

(25)

where 1/J' = 81/J / 8a and 'lj/0 = [1/J'la=O· Of course, y = z does not exist in any naive sense (as it does not even in the conventional case (1), of observation corrupted by white noise) and we must interpret this last relation incrementally:

61/J = [u(ea -1) + p'I/J'(e-a- 1) -v(1/J'

-1j;~)]6t

+ [log(1j;'/1/J~)]8z

(26)

where 6z is K-- 1 times the number of particles registered in the time-increment 6t. Remarkably, relation (26) is exact; see Exercise 1 below. However, it is still not finite-dimensional in character. Exercises and comments (1) Since we aim to prove that the updating equation (26) is exact, we may as well set K- = 1. Suppose that M(a, t) = exp(ax)p(dx, t). If a particle is registered (i.e. 6z = 1), then, by Bayes' theorem, M changes instantaneously to

J

J

vxeaxp(dx, t)/

J

vxp(dx, t) = Ma/MaO = (1/Ja/1/JaO)M.

Thus 1/J suffers an instaneous increment of log(1/JaNao) which is just what equation (26) asserts. If no particle is registered in (t, t + 6t), so that 8z = 0, then over the time increment 6t the characteristic function M changes to const.

J

(1 +A 8t)eax(l - vx 8t)p(dx, t)

= const.

J

eax[1

+ H(x, a) 6t- vx &] p(dx, t).

5 PURE ESTIMATION: NON-LINEAR FILTERING

441

Here A is the infinitesimal generator of the x-process alone, H(x, a) = u( e0 - 1) + px( e-a - 1) the corresponding DCF and the constant is such as to retain the normalisation M(a, t + 8t) = 1. This indeed implies relation (26) with 8z =0. Notes on tbe literature

The material of this chapter is largely taken from Whittle (1991a). There is a very large literature on non-linear filtering; an appeal to large deviation ideas is explicit in the papers by Hijab (1984) and James and Baras (1988). These present what is essentially a recursive updating of the rate function D ( Y (t), x) in the case when the stochastic element of both plant and observation dynamics is a diffusion, the updating equation being characterised with some justification as 'the wave equation of non-linear filtering'. However, this description could much more fittingly be applied to the general updating relation (18) I (19). That this is set in transform space is, as we have emphasised, natural in the inference context.

APPENDIX 1

Notation and Conventions For discrete-time models the time variable tis assumed to take signed integer values; for continuous-time models it may take any value. A variable x which is time-dependent is a signal. In discrete time the value of x at time tis denoted x 1• More generally, if ( ... ) is a bracketed expression then (... )1 denotes that expression with all quantities in the bracketed evaluated at time t, unless otherwise indicated. In continuous time the time-dependence of x is either indicated x(t) or understood. In either discrete or continuous time the expression x(tl denotes the optimal (minimal stress) estimate of x based on 1 information at time t. The special estimate x~ ) is denoted x1• A circumflex is also used to denote a Laplace transform or z-transform in Chapter 4 alone. If {x 1} is a sequen.ce ofvariables in discrete time then X 1 denotes the history {x ... ;r ~ t} of this sequence up to time t. The starting point of this history may depend upon circumstances; it is usually either T = 0 or T = -oo. The simple capital X denotes a complete realisation-the course of the x-variable over the whole relevant time interval. There are other standard modifiers for signals. The command value of x is denoted r, and the deviation X- XC denoted x•. The notation X denotes either the equilibrium value or the terminal value of x, in different contexts. Do not confuse with the effect of the overbar on complex quantities and operators, where it amounts to the operation of conjugation; see below. The bold symbol xis used in a system notation, when it denotes all internal variables of the system collectively. A matrix is denoted by an italic capital: A. Its transpose is denoted by AT. If Q is a matrix then Q > 0 and Q ~ 0 indicate respectively that Q is positive definite and positive semi-definite (and so understood to be symmetric). If A is a matrix with complex elements then A denotes the transpose of its complex conjugate. The overbar thus combines the operations of transposition and complex conjugation, which we regard as the conjugation operation for matrices. If xis said to be a vector then it is a column vector unless otherwise indicated. If F(x) is a scalar function of X then 8F f8x is the row vector of first differentials of F with respect to the elements of x. We sometimes denote this by the convenient subscript notation Fx. and use Fxx to denote the square matrix of second differentials. If a(x) is itself a vector function of x then ax denotes the matrix whosejkth element is the differential ofthejth element ofa with respect to the kth element of x. If H (p) is a scalar function of a row vector p then Hp is the column vector of differentials of H with respect to the elements ofp.

444

APPENDIX 1

The subscript notation is also sometimes used to distinguish sub-matrices in a partitioned matrix; see, for example, Sections 19.2 and 20.3. Operators are generally denoted by a script capital. Important special operators are the identity operator .Y, the backward shift operator f7 with effect f7x 1 = x 1_ 1 and the differential operator~ with effect ~x = dxjdt. A symbold will denote a distributed lag operator A(f/) L_j Ajf/j in discrete time_ and a differential operator A(~) = L_j A1 ~ is continuous time. The conjugate .91 of .91 is defined as A (f/- 1) T in discrete time and A (- ~) T in continuous time. We shall often consider the corresponding generating functions A(z) and A(s), the complex scalars z and s corresponding to f7 and ~ respectively. These will also be denoted by .91, the interpretation (operator or generating function) being clear from the context. In this case the conjugate .91 is to be identified with A(z- 1 )T and A( -s) Tin discrete- and continuous-time contexts respectively. The shell or Gill characters §, [D and [D are used to denote stress and cost and discrepancy components of stress respectively. The large-deviation evaluation of [D is proportional to the rate function, which we denote by D. The cost from time t is denoted C 1, so if the process is obliged to terminate at a given horizon point h then the closing cost is denoted Cn. This does not necessarily coincide with the terminal cost IK, which is the cost incurred when the process is obliged to stop (possibly before h) because it has entered a stopping set Y. The notations max or min before an expression denote the operation of taking a maximum or minimum, with respect to a variable which is indicated if this is not obvious. Correspondingly, sup and inf denote the taking of a supremum or infimum; stat denotes evaluation at a stationary point; ext denotes the taking of an extremum (of a nature specified in the context). The expectation operator is denoted by E, and Err denotes the expectation under a policy 1r. Probability measure is indicated by P( ·) and probability density sometimes by f(·). Conditional versions of these are denoted by E(·l·), P(·l·) and f (·I·) respectively. The covariance matrix E{[x- E(x)][y- E(y)]T} between two random vectors x andy is written Vxy, or cov(x, y). We write cov(x, x) simply as cov(x). If Vxy = 0 then x andy are said to be orthogonal, expressed symbolically as x.ly. The modified equality symbol:= (resp. =:)in an equation indicates that the left- (right-)hand member is defined by the expression in the right- (left-)hand member. Notations adopted as standard throughout the text are listed below, but some symbols perform multiple duty. Abbreviations Abs AGF CEP

Absolute term in a power series expansion on the unit circle Autocovariance generating function Certainty equivalence principle

APPENDIX I

CGF DCF MGF Tr

445

Cumulant generating function Derivate characteristic function Moment generating function Trace (of a matrix)

Standard symbols

A,B,C

Coefficient matrices in the plant and observation equations d,I?J,((i Operator versions of these for higher-order models The system operator (.91!11) 21 Cost function Instantaneous cost function or rate c(x,u) !?) The time differential operator Rate function D [} The discrepancy component of stress Plant disturbance; various d Expectation operator E Expectation operator under policy 1r E"' Future value function; future stress F Transient cost; probability density; the operator in the Riccati equation f Total value function; the coefficient of the quadratic terms in a timeG integral; a transfer function; controllability or observability Gramian g A function defining a policy u1 = g(x 1) Hamiltonian; the innovation coefficient in the Kalman filter; the H constant factor in a canonical factorisation; the matrix coefficient of highest-order derivatives Horizon; various h The identity operator ..1 The identity matrix I Time-integral 0 J The control-power matrix The matrix coefficient in the optimal feedback control K A terminal cost function IK Forward operator; covariance matrix of plant and observation noise L The forward operator of the dynamic programming equation; Laplace !l' transform operator Forward operator in continuous time; the covariance matrix of M observation noise; a moment generating function; a retirement reward: an upper bound on the magnitude of control m The dimension of the control variable u The covariance matrix of plant noise (termed the noise power matrix in N continuous time) The dimension of the process variablex n

c

446 p p

Q R 9l r

s §

y s f/ t

u

u,

u

v

v(,\, J.l)

w,

w X X, X

y

Y, y

z a (3

r I ~

6 f

( 1'J

e >.

APPENDIX I

Past stress; probability measure The order of dynamics; the conjugate variable of the maximum principle. The matrix of control costs The matrix of process-variable costs The cost matrix in a system formulation The dimension ofthe observationy The cross-matrix of process- and control-variable costs Stress The stopping set Tune to go; the complex scalar argument of a transfer function corresponding to !!) The backward shift operator Tune; the present moment The complete control realisation Control history at time t The control (action, decision) variable A covariance matrix, or its risk-sensitive analogue; a value function under a prescribed policy. The information analogue of c(x, u) Information available at time t; a noise-scaling matrix The command signal The complete process realisation Process history at time t ·. The process or state variable The complete observation realisation Observation history at time t Observation The complex scalar corresponding to !F Discount rate; the row vector argument in transform contexts; various Discount factor; the row vector argument in transform contexts; various Gain matrix Average cost, either direct or in a risk-sensitive sense System error; estimation error Increment (as time increment 6t); Kronecker ~function Plant noise Innovation; primitive system input; coefficient of the linear term in a time integral Observation noise Risk-sensitivity parameter The Lagrange multiplier for the plant constraint; birth rate

APPENDIX 1 J-L

v ~

IT 1r

a T

.P d;

w

'1/J

x !1 w

447

The Lagrange multiplier for the observation constraint; death rate A Gittins index; various A sufficient variable; the combination ~. t) of state variable and time; the argument of a quadratic time-integral The matrix of a quadratic value function Policy The coefficient of the linear term in a quadratic value function; a function occuring in the in the updating of posterior distributions A running time variable A matrix of operators (or generating functions) occurring in forward optimisation; the normal integral A canonical factor of .P; the normal density; various A matrix of operators (or generating functions) occurring in backward optimisation; a function ocurring in the updating of posterior distributions A canonical factor of w; a cumulant-generating function; an expectation over first passage A cost-renormalisation of a risk-sensitive criterion; various The information gain matrix Frequency

Modifiers, superscripts, etc. .X

d B xc .X x(r)

x1 .X,

Equilibrium value or terminal value of the variable x Conjugate of the operator or generating function .sd The critical value ofthe risk-sensitivity parameter B The command value of x The deviation x- xc; limit-optimal path The best (minimum stress) estimate of x based on information at timet The best estimate of x 1 based on current information: x~r) The best estimate of x 1 based on current information and past costs alone

APPENDIX 2

The Structural Basis of Temporal Optimisation A stochastic model of the system will specify the joint distribution, in some sense, of all relevant random variables (e.g. process variables and observations) for prescribed values of the control variables. These latter variables are then parametrising rather than conditioning, since they affect the distribution of the former variables but are not themselves initially defined as random variables. The information at a given time consists of all variable-values which are known at that time, both observations and control values. The process model describes not merely the plant which is to be controlled, but also variables such as command signals. Either these signals have their wh.ole course specified from the beginning (which is then part of the model specification) or they are generated by a model which is then part of the process model. Consider optimisation in discrete time over the time interval t ;;.: 0. As in the text, we shall use Xt, U1 and Y1 to denote process, control and observation histories up to timet, and W 1 to denote information available at the time t. More explicitly, Wt denotes the information available at the time the value of u1 is to be chosen, and so consists of (W0 , Y1, U1_I). That is, it includes recollection of previous controls and Wo, the prior information available when optimisation begins. We take for granted that it also implies knowledge oft itself: of clock time. Let us for simplicity take W0 for granted, so that all expectations and probabilities are calculated for the prescribed Wo. Our aim is to show that the total value function

G( W1) = inf E(CI Wt)

(1)

1f

satisfies the optimality equation

G( Wt) = inf E[G( Wt+t)[ Wt, ut] u,

(2)

and that the infimising value of u 1 in (2) is the optimal value of control at timet. These assertions were taken almost as self-evident in Chapter 8, but they require proof at two levels. One is at the level of rigour; there may be technical problems due to the facts that the horizon is infinite or that that the control may take values in an infinite set. We shall not concern ourselves with these issues, but rather with the much

450

APPENDIX 2

more fundamental one of structure. The optimality equation is only 'self-evident' because one unconsciously makes structural assumptions. These assumptions and their consequences must be made explicit. That there is need for closer thought is evident from the fact that the conditional expectations in (1) and (2) are not well-defined as they stand, because the control history U,_I or U, is not defined as a random variable. The following discussion is an improved version of the first analysis of these matters, which was given in Whittle (1982), pp. 150-2. Complete realisations Xoc, U00 andY00 will be denoted simply by X, Uand Ylt can be assumed without Joss of generality that w = gives complete information on the course of all variables. We shall use naive probability notations P(x) and P(xJy) for distributions and conditional distributions of random variables x and y, as though these variables were discrete-valued. All such formalism has an evident version or interpretation in more general cases. A subscript 1r, as in P,(x), indicates the distribution induced under policy 1r; correspondingly for expectations. The policy 1r is subject to the condition of realisability: that the value of the current control u1 may depend only on current observables W,. We shall express this by saying that u, must be W,-measurable. By this we do not mean measurability in the technical sense of measure theory, but in the naive structural sense, that u, can depend on no other variable than W,. We shall in general allow randomised policies, in that the policy 1r is determined by specification of a conditional probability distribution P,(u1 W 1 ) for each t. This is convenient, even though the optimal policy may be taken as deterministic, in that it expresses u1 as a function of W, for each t. One must now distinguish between conditioning variables and parametrising variables. A model for the stochastic dynamics of the process would imply a specification of the probability distribution of X for any given U However, this is not a distribution of X conditional on U, because U is not even defined as a random variable until a control policy has been specified. Rather, U is a parametrising variable; a variable on which the distribution of X depends. We shall write the distribution of X conditioned by Y and parametrised by Z as P(XJ Y; Z). The specification of a stochastic model for the controlled process thus corresponds to the specification of the parametrised distribution P( X!; U). The full stochastic specification of both plant equation and observation structure corresponds to specification of P( X, YJ; U). We see then that we should more properly write P,(u1 W,) as P,(ur!; W 1 ), because we are simply defining a u, distribution which allows an arbitrary dependence upon W1 • Calculation of expectations for prescribed Wo may also be a mixture of conditioning and parametrising. These distinctions turn out not to matter, but thanks only to the ordering imposed by a temporal structure. The following are now the basic assumptions of a temporal optimisation problem.

w)()

1

1

APPENDIX 2

451

(i) Separateness ofmodel and policy. ()()

P1r(X, Y, U) = P(X, Yl; U)

IJ P1r(u1l; W1).

(3)

1=0

(ii) Causality. P(X~, Y1l; U)

= P(X~, Y1i;

U1-1).

(4)

The assumption W1 = (Wo, Y1, U1-1) implies the further properties (iii) Non-anticipation. W 1 is (Y1, U1_1)-measurable. (iv) Retentionofinformation. W1_1 and U1-1 are W1-measurable. Conditions such as these are often taken for granted and not even listed, but the dynamic programming principle is not valid without them. The fact that information at time t includes that at time t - 1 would be expressed in some of the literature by saying that in (1), for example, one is conditioning on an increasing sequence of u-fields constituting a filtration. We mention the point only to help the reader make the connection; this is not a language that we shall need. Condition (i) factors the joint distribution of X, Yand U under policy 1r into terms dependent on the model and the policy respectively. Note that it is by specification of the policy that one completes the stochastic specification which allows U to be regarded as a random variable jointly with X and Y. Note that relation (3) also expresses realisability of the policy. Condition (ii) does indeed express the vital causality condition: that the course of the process up to a given time cannot be affected by actions after that time. Here the distinction between parametrising and conditioning variables is cruciaL In the sense expressed by (4) the variable x 1 (for example) cannot be affected by uT(T ~ t). However, once a policy is specified, then x 1 will in general show a stochastic dependence on these future control variables, simply because these future control variables will in general show a stochastic dependence upon x 1 . Condition (iv) expresses the fact that information is in principle never lost or discarded. ('In principle', because assumptions of the Markov type make it unnecessary to retain all information.) In particular, past decisions are recalled. The aspect of optimisation is introduced by requiring 1r to be such as to minimise E1r(C) for a suitably specified cost function C = q W). Usually the observations yare regarded as subsidiary to the process variable x, in that they are observations on the process variable. This dependent role would be expressed by some property such as P(x1,Y1IX1-1, Y1-1i U1-1) = P(x~,yt!X1-1; U1-1)

and the assumption that the cost function C may depend on X and U but not on Y. However, there is no need to make such assumptions; one can simply regard {x 1, y 1} as a stochastic process describing system evolution (and parametrised by U) of which only the component {y 1} is observable.

452

APPENDIX 2

We shall now show that the dynamic programming principle follows from these assumptions. For notational simplicity we shall abbreviate P,.(u1!; W1) to P11"1· An unrestricted summation 2: will be a summation over all W. A symbol ( W1) under the summation sign will indicate that the summation is to be extended over all W consistent with a given W1• Let us define

V(1r, W1) = E,.(q W1), the total value function for a specified policy 1r. The conditional-expectation notation is legitimate, since W1 is defined as a random variable under the policy. Then V obeys the recursion

(5)

V(1r, W1) = E,.[V(1r, WI+I)! Wt] in virtue of (iv) and the properties of a conditional expectation.

Lemma A2.1

V (1r, W1) is independent ofpolicy before time t.

Proof We have V(1r, W1) =

2:(W,) C( W)P,.(X, Y, U) 2: p (X y U) . (W,)

,.

'

(6)

'

Substituting expression (3) for P,.(X, Y, U) into this equation we find that the P1rj for j < t cancel out, since they depend on W only through W1• 0

Lemma A2.2 For any function ¢ of process history the expectation E,.[¢(X1+I, Yt+ 1) IW1, u1] is independent ofpolicy and can be written

Proof Relation (7) certainly holds if P(Xt+I, Yt+tli U1) is replaced by P,.(XI+I, Yt+ 1 , U1 ). But, by the separateness and causality assumptions (i) and (ii), I

P,.(Xt+t. yl+l, Ul)

= P(Xt+l! yl+t!i Ul) IT P,.j. j=O

The P,.j terms cancel, leaving relation (7), which certainly has the implication asserted. 0

Lemma A2.3

If P,.1 is chosen optimally, for given t, then recursion ( 7) becomes

APPENDIX 2

453

V(rr, W,) = inf E[V(rr, Wt+I)jW,,u,] u,

where the expectation operator is independent of 1r and the the minimising u, is optimal, under prescription of1r at other time points. Proof We have

E'lr[V(rr, Wt+I)jW,] = I,:P,.(u,jW,)E[V(rr, Wt+I)jW,,u,]

(8)

where the last expectation operator is independent of rr, by Lemma A2.2. Indeed, a further appeal to Lemma A2.1 indicates that the final expectation is independent of P,, , so that P,, occurs in expression (8) only where it appears explicitly. The assertions of the lemma then follow. 0 The desired conclusions are now immediate. Theorem A2.4 The optimal total value function G obeys the dynamic programming equation (2), the expectation operator being independent ofpolicy. The minimising valueofu, in (2) is theoptimalvalueofcontrol at timet.

This follows simply by application of Lemma A2.3 under the assumption that policy has been optimised after timet. The explicit form of the expectation in (2) follows from (7). Essentially, P'lr( W,+d U,) = P( Wt+d; U,).

APPENDIX 3

Moment Generating Functions; Basic Properties If x is a vector random variable then its moment generating function (MGF) is defined as (1) where o is then a row vector. This certainly exists for purely imaginary o; and the characteristic function M(ie) is studied for real every much in its own right. Its relation to the MGF is exactly that of the Fourier transform to the Laplace transform. M(o) will exist for a range of real o if the tails of the x-distribution decay at least exponentially fast. In such cases the distribution of x and associated random variables has stronger properties, which are characterised immediately and simply in terms of the MGF, as we saw particularly in Chapter 22.

Theorem A3.1 The moment generating function M (o) is convex for real o. The set JJI ofreal o for which M (o) isfinite is convex and contains the value zero. Proof M(o) is an average of functions eax which are convex in o and so is itself convex. We know already that 0 E d. Jensen's inequality

M(po + q{J)

~

pM(o) + qM({J)

(o, {3

E

d;p, q ~ O;p + q = 1)

for convex functions then implies that all elements of the convex hull of d belong to d. That is, d is convex. D This convexity explains why the equation M(o) = 1 has at most two real roots in the scalar case, one at o = 0 and the other of sign opposite to that of

M'(O)

= E(x).

Theorem A3.2 M(o) possesses derivatives of all orders in the interior of d, obtained by differentiatingexpression (1) under the expectation sign. Proof This follows from the absolute convergence of the expectation thus de~d

D

Existence of derivatives is of course closely related to the existence of moments, which are proportional to the derivatives of M{o) at o = 0. We see from the

456

APPENDIX 3

theorem that, if 0 is an interior point of .91, then moments of all orders exist. The classic case for which no moments of integral order exist is the Cauchy distribution; a distribution of scalar x with probability density proportional to (1 + and characteristic function exp( -101). We could have proved convexity of M(a) by appealing to the fact that the matrix of second differentials

x2r'

(where a1 is thejth element of a) is plainly non-negative definite. This is clumsy compared with the proof above, and makes an unnecessary appeal to the existence of differentials. However, we shall indeed use it in a moment If we write the second differential in the left-hand member as M1k (a) then M1k (a) I M (a) can be identified as £(al(x1xk), where £(o) is the expectation for the tilted distribution defined in Section 22.2. That is, the a-tilted expectation of a function f/J(x) is

£(o}[f/J(x)] =

E[~~~~axJ.

The MGFofa sum of independent random variables is the product ofMGFs. The cumulant generating function (abbreviated to CGF) t/J(a) =log M(a) is then a natural quantity to consider, since the CGF of a sum of independent random variables is the sum ofCGFs.

Theorem A3.3 The function t/J( a) is also convex in .91. Proof If we use the subscript notations M1 = 8M I aa1 and M1k = Efl M I aajaak, with argument a understood, then we have • 1.'/c '~'1

= MJk _ M1Mk M

M2 .

But the matrix of these second differentials is just the covariance matrix of x on the tilted distribution, and so is non-negative definite. This proves convexity 0 oft/J.

References Azencott, R. (1982) Formule de Thylor stochastique et developpement asymptotique d'integrales de Feynman. Seminaire de Probabilites XVI Lecture Notes in Mathematics 921, 237-284. Springer, Berlin. Azencott, R. (1984) Densite des diffusions en temps petits; developpements asymptotiques. Semina ire de Probabi/ites XV/ll Lecture Notes in Mathematics 1059, 402-498. Springer, Berlin. Bartlett, M.S. (1949) Some evolutionary stochastic processes. J Roy. Statist. Soc. B, 11,211229. Barlett, M.S. (1955) Deterministic and stochastic models for recurrent epidemics. Proc. Third Berkeley Symposium, (Ed. J. Neyman), IV, 81-109. Bartlett, M.S. (1960) Stochastic Population Models in Ecology and Epidemiology. London. Ben Arous, G. (1988) Methodes de Laplace et de Ia phase stationnaire sur l'espace de Wiener. Stochastics, 25,125-153. Benes, V, Shepp, L.A. and Witsenhausen, H.S. (1980) Some soluble stochastic control processes. Stochastics, 4. Brockett, R.W (1970) Finite-dimensional Linear Systems. Wiley, New York. Bucklew, J.A. (1990) Large Deviation Techniques in Decision, Simulation and Estimation. Wiley, New York. Clark, CW (1976) Mathematical Bioeconomics. Wiley, New York. Cox, D.R. and Smith, WL. (1961) Queues. Methuen, London. Davis, M.HA. (1984) Piecewise-deterministic Markov processes: a general class of nondiffusion stochastic models. J Roy. Statist. Soc. B, 46, 353-388. Davis, M.H.A. (1986) Control of piecewise-deterministic processes via discrete-time dynamic programming. In Stochastic Differential Systems (Ed M. Kohlmann). Lecture Notes in Control and Information Sciences, 78, Springer-Verlag, Berlin. Davis, M.H.A. (1993) Markov Mode/sand Optimization. Chapman and Hall, London. Dembo, A. and Zeitouni, 0. (1991) Large Deviations and Applications. A.K. Peters, Wellesley, USA. Deuschel, 1-D. and Strook, D. (1989) Large Deviations. Academic Press, New York. Doyle, lC, Francis, B.A. and 'Th.nnenbaum, A.R. (1992) Feedback Control Theory. Macmillan, New York. Ellis, R. (1985) Entropy. Large Deviations and Statistical Mechanics. Springer, New York. Fleming,WH. (1971) Stochastic control for small noise intensities. SIAM J Control Optim., 9,473-5U. Fleming, WH. (1978) Exit probabilities and optimal stochastic control. Applied Math. Optim., 4, 329-346. Fleming,WH. (1985) A stochastic control approach to some large deviations problems. In Recent Mathematical Methods in Dynamic Programming (Eds. Capuzzo Dolcetta eta/.). Springer, Berlin. Fleming,WH. and Soner, H.M. (1992) Controlled Markov Processes and Viscosity Solutions. Springer, Berlin.

458

REFERENCES

Fleming, WH. and Tsai, C.P. (1981) Optimal exit probabilities and differential games. Applied Math. Optim., 7, 253-282. Fragopoulos, D. (1994) H 00 Synthesis Theory using Polynomial System Representations. Ph.D. Thesis, Department of Electrical and Electronic Engineering, University of Strathclyde, Scotland. Francis, B.A. (1987) A Course in H 00 Control Theory. Springer, Berlin. Friedlin, M.I. and Wentzell, A.D. (1984) Random Perturbations of Dynamical Systems. Springer-Verlag, New York. (Russian original published in 1979 by Nauka, Moscow.) Gale, D. (1960) The Theory of Linear Economic Models. McGraw-Hill, New York. Gale, D. (1967) On optimal development in amulti-sectoreconomy. Rev. Econ. Stud., 34, 1-18. Gale, D. (1968) A mathematical theory of optimal economic development. Bull Amer. Math. Soc., 74,207-223. Gartner, J. (1977) On the large deviations from the invariant measure. Th. Prob. Appl., 22, 24-39. Gibbens, R.J., Kelly, F.P. and Key, P.B. (1988) Dynamic alternative routing-modelling and behaviour. In Proceedings of the Twelfth International Teletraffic Congress. North Holland, Amsterdam. Gibbens, R.J., Kelly, F.P. and Key, P.B. (1995) Dynamic alternative routing. In Routing in Communications Networks (Ed. M.A. Steenstrup), Prentice Hall, Englewood Cliffs, New Jersey. Gihman, I. I. and Skorohod, AY. (1972) Stochastic Differential Equations. Springer, Berlin. Gihman, 1.1. and Skorohod, AY. (1979) The Theory of Stochastic Processes,Vol. III. Springer, Berlin. Gittins, J.C. (1989) Multi-armed Bandit Allocation Indices. Wiley, Chichester. Gittins, J.C. and Jones, D.M. (1974) A dynamic allocation index for the sequential design of experiments. In Progress in Statistics (Ed. J. Gani), pp. 241-266. North Holland, Amsterdam. Glover, K. and Doyle, J.C. (1988) State-space formulae for all stabilizing controllers that satisfy an H 00 -norm bound and relations to risk sensitivity. System & Control Letters, 11, 167-172. Hagander, P. (1973) The use of operator factorisation for linear control and estimation. Automatica, 9, 623-631. Hijab, 0. (1984) Asymptotic Bayesian estimation of a first order equation with small diffusion. Ann. Prob., 12,809-902. Holland, C.J. (1977) A new energy characterisation of the smallest eigenvalue of the Schrodinger equation. Comm. Pure Appl Math., 30, 755-765. fJolt, C., Modigliani, F., Muth, J.F. and Simon, H.A. (1960) Planning, Production, Inventories and Work-force. Prentice-Hall, Englewood Cliffs, New Jersey. Howard, R.A. (1960) Dynamic Programming and Markov Processes. MIT Press and Wiley, New York. Jacobson, D.H. (1973) Optimal stochastic linear systems with exponential performance criteria and their relation to deterministic differential games. IEEE Trans. Autom. Control, AC-18, 124-131. Jacobson, D.H. (1977) Extensions of Linear-quadratic Control, Optimization and Matrix Theory. Academic Press, New York. James, M.R. and Baras, J.S. (1988) Nonlinear filtering and large deviations: a PDE-control theoretic approach. Stochastics, 23,391-412. Kalaba, R. (1959) On nonlinear differential equations, the maximum operation and monotone convergence. 1 Math. Mech., 8, 519-574. Krishnan, A.R. and Ott, T.J. (1986) State-dependent routing for telephone traffic; theory and results. Proc. 25th IEEE Controland Decision Conference, 2124-2128.

REFERENCES

459

Kumar, P.R. and van Schuppen, J.H. (1981) On the optimal control of stochastic processes with an exponential-of-integral performance index.J Math Anal. Appl., 80,312-332. Lande, R., Engen, S. and Saether, B.-E. (1994) Optimal harvesting, economic discounting and extinction risk in fluctuating populations. Nature, 372, 88-90. Lande, R. Engen, S. & Saether, B.-E. (1995). Optimal harvesting of fluctuating populations with a risk of extinction. Am. Nat.145, 728-745. Martin-Lof, A. (1986) Entropy, a useful concept in risk theory. Scand. Actuarial J,??, 223235. Miller, H.D. (1961) A convexity property in the theory of random variables on a finite Markov chain. Ann. Math Statist., 32, 1260-1270. Mustafa, D. and Glover, K. (1990) Minimum Entropy Hoo Control. Springer, Berlin. Newton, G.C, Gould, L.A. and Kaiser, J.F (1957) Analytical Design of Linear Feedback Controls. Wiley, New York. Pollatschek, M. and Avi-Itzhak, B. (1969) Algorithms for stochastic games with geometrical interpretation. Man. Sci., 15, 399-413. Pontryagin, L.S., Boltyansk.ii, V.G., Gamkrelidze, RY. and Mischenko, E.F (1962) The Mathematical Theory of Optimal Processes. Interscience, New York. Puterman, M. L. (1994) Markov Decision Processes. Wiley, New York. Shwartz, A. and Weiss, A. (1995) Large Deviations for Performance Analysis. Chapman and Hall, London. Speyer, J.L. (1976) An adaptive terminal guidance scheme based on an exponential cost criterion with applications to homing missile guidance. IEEE Trans. Autom. Control, AC-21, 371-375. Speyer, J.L., Deyst, J. and Jacobson, D.H. (1974) Optimisation of stochastic linear systems with additive measurement and process noise using exponential performance criteria. IEEE Trans. Autom. Control, AC-19, 358-366. Stroock, D. (1984) An Introduction to the Theory of Large Deviations. Springer, Berlin. Stroock, D. and Varadhan, S. (1979) Multimensional Diffusion Processes. Springer, Berlin. Tegeder, R.W: (1993) Large Deviations, Hamiltonian 'R!chniques and Applications in Biology. Ph. D. Thesis, University of Cambridge. Tsitsiklis, J.N. (1986) A lemma on the MAB problem. IEEE Trans. Automat. Control, AC-31, 576-577. Van Vleck, J.H. (1928) Proc. Nat!. Acad. Sci. USA, 14, 178. Vanderbei, R.J. and Weiss, A. (1988) Large Deviations and Their Application to Computer and Communications Systems. Circulated unpublished notes, AT&T Bell Laboratories. Varadhan, S. (1984) Large Deviations and Applications. SIAM, Philadelphia. Vidyasagar, M. (1985) Control System Synthesis; a Factorization Approach. MIT Press, Cambridge, Mass. Weber, R. and Weiss, G. (1990) On an index policy for restless bandits. 1 Appl. Prob., 27, 647-648. Whittle, P. (1963) Prediction and Regulation. English Universities Press, London Whittle, P. (1980) Multi-armed bandits and the Gittins index.J Roy. Statist. Soc., B 42, 143149. Whittle, P. (1981) Risk-sensitive linear/quadratic/Gaussian control. Adv. Appl. Prob., 13, 764-777. Whittle, P. (1982) Optimisation over Time, Vol. 1. Wiley, Chichester. Whittle, P. (1983a) Optimisation over Time, Vol. 2. Wiley, Chichester. Whittle, P. (1983b) Prediction and Regulation. Second and revised edition of Whittle (1963) University of Minnesota Press and Blackwell, Oxford. Whittle, P. (1986) The risk-sensitive certainty equivalence principle. In Essays in Time Series Analysis and Allied Processes, Ed. J. Gani, 383-388. Applied Probability Trust, Sheffield.

460

REFERENCES

Whittle, P. (1988) Restless bandits; activity allocation in a changing world. In A Celebration ofApplied Probability (Ed. J. Gani), l Appl. Prob.,lSA, 287-298. Whittle, P. (1990a) Risk-sensitive Optimal Control. Wiley, Chichester and New York. Whittle, P. (1990b) A risk-sensitive maximum principle. Syst. Contr. Lett.,15, 183-192. Whittle, P. (1991a) A risk-sensitive maximum principle: the case of imperfect state observation. IEEE '/rai'IS. Auto. Control, AC-36, 793-801. Whittle, P. (1991b) Likelihood and cost as path integrals. I Roy. Statist. Soc., 8 53,505-529. Whittle, P (1995) Large-deviation expressions for the distribution of first-passage coordinates. Adv. Appl. Prob., 27,692-710. Whittle, P. and Gait, P. (1970) Reduction of a class of stochastic control problems. l Inst.

Math. Appl.,6,131-140. Whittle, P. and Horwood, JYI. (1995) Population extinction and optimal resource management. To appear in PhiL Trans. Roy. Soc. B. Whittle, P and Komarova, N. (1988) Policy improvement and the Newton-Raphson algorithm. Prob. Eng. Inf Sci., 2, 249-255. Whittle, P and Kuhn, J. (1986) A Hamiltonian formulation of risk-sensitive, linear/ quadratic/gaussian control. Int. J. Control. Willems, J. C (1991) Paradigms and puzzles in the theory of dynamical systems. IEEE Trans. Autom. Control, AC- 36, 259-294. Willems, J.C (1992) Feedback in a behavioural setting. In Models and Feedback: Theory and Applications (Eds. lsidori, A. and Tarn, T.J.), pp. 179-191. Birkhiiuser. Willems, J.C. (1993) LQ-control: a behavioural approach. In Proc.J993 IEEE Conference on

Decision and Control. Wold, H.O.A. (1938) The Analysis of Stationary Time Series. Almquist and Wicksell, Uppsala. Youla, D.C, Bongiorno, J.J. and Jabr, H.A. (1976) Modern Wiener-Hopf design of optimal controllers. Part I: The single-input- output case. IEEE Trans. Auto. Control, AC-21, 313. Part II. The multivariable case. IEEE Trans. Auto. Control, AC-21, 319-338. Zam.es, G. (1981) Feedback and optimal sensitivity: model reference transformations, multiplicative seminorms, and approximate inverses. IEEE Trai'IS. Auto. Control, AC- 26, 301-320.

Index Allocation, of activity 269-283 Average-cost optimisation 53-56, 138, 217-221,314--316 Autocovariance 256-257 Autocovariance generating function (AGF) 257 Autoregressive processes and representations 262 Avoidance of hazards 160-163, 428-430 Back-logging 20 Backwards translation operator See Translation operator Bang-bang control 31, 146 Bayesian statistics 287 Behavioural formulation 129 Blackmail 216, 220 Brownian motion See Wiener process Bush problem 145-147 Calculus of variations 17 Call routing 226-228 Cart and pendulum model 35-Jl, 99 Canonical factorisation of operators 118, 129, 262, 263, 336, 339, 342-344, 345, 348, 352, 354, 361, 3(iJ Causality 68. 451 Cayley-Hamilton theorem 99-100 Certainty equivalence 191, 230, 234--239, 298-302,432--435 Change-point detection 289-291 Chernoff's inequality 387-388 . Circulant process 261_ . Classic fonrili.hition'63=66 "='~,, · · Clearance, optimal 424; 426, 428, 430 Closed-loopproperty·J4; 25,176,191 Closing cost 12,49 · -::. : ' ·:: .. Command signal 64 Companion matrix 101 . ' · · Compound Poisson process::;t84 ' ·.

Conditionally most probable estimate 237 Conjugate variable 135, 136 Consumption, optimal 17-19,55-56,143, 411--413 Control-power matrix 34, 306 Controllability 101-106 Cost, closing 12, 49 Cost, instantaneous 12 Cost, terminal 49 Cost, transient 54 Cost function 12-13 Covariance matrix 239 Cramer's theorem 380, 384--387 Crash avoidance 157-160 Cumulant generating function (CGF) 384 Derivate charcteristic function (DCF) 181, 390 Detectability 109 Diffusion coefficient 187 Diffusion processes 186-187,206-208 Direct trajectory optimisation 42--46, 131-166 Direct trajectory optimisation with LEQG structure 316-317,371-377 Direct trajectory optimisation with LQ structure 116-121,122-123,126-129 Direct trajectory optimisation with LQG structure 331-369 Discounting 15, 29, 44, 49, 143, 178, 317-319 Discrepancy function 230-231 Domain of attraction 3, 93 Dosage, optimal 134--135 Drift coefficient 187 Dual control 285-286 Dual variable See Conjugate variable Duality, ofestimation and control248-2:53 Dynamic lags 88-89 Dynamic programming equation 13-14, 28, 174, 176, 177, 288, 289, 339, 340, 347, 4()7

462

INDEX

Eikonal equation 31 Entropy criterion 316 Equilibrium 92-93 Equilibrium point, optimal 42--46 Erlang function 228 Estimates, conditionally most probable 237 linear least square 243 minimal discrepancy 243 projection 242-246 Euler condition 17 Euphoria 304 Excessive functions 51 Extinction 196--7, 207, 210 Factorisation See Canonical factorisation of operators Feedback 14, 25,63-66, 85-89 Feedback/feedforward rules 40, 119 Filters 67-85 Filter, action on stochastic processes 257-261 Filter inversion 72-74, 80-82 Filter, proper 84, 87 Filter, Kalman See Kalman filter First-passage problems 30-31, 140-142, 201-205,415--430 Final value theorem 83 Fluid approximation 392 Flypaper effect 204 Forward operator 48 Free form 299 Frequency response function 69 Future stress 305-307, 310,311 'Gain, effective 88 Gain matrix 24, 26 Gittins index 270-275 Gramian 104, 105, 108 Grazing of stopping set 157, 162 Growth, optimal 55-56,133-135

Hoo criterion 321-329 Hoo norm 324-326 Hamiltonian 45, 136, 137, 396 Hamiltonian structure 395-397 Hardy class 325 Harvesting, optimal 2-5,31-33,61-62, 194-201, 206--213

Hedging 173 Homogeneous processes of independent increments (HPII) 182-184 Horizon 12 Imperfect observation 96--98,229-253, 285-291,308-312,357-369,431--441 Index, Gittins 270-275 Indexability 280 Infinite-horizon behaviour 26--28,47-56, lll-115, 307, 312-313 Infinitesimal generator 177, 179, 390 Infinitely divisible distributions 183 Inertial particle 37, 155-164, 426--428 Inertialess particle 37, 153-154,408--411, 421--426 Information 173, 449 Information state 229, 285, 288 Innovation 246--248, 361, 365, 376 Input-<>utput formulation 63-89 Instability, of optimisation 52, 216 Instantaneous cost 12 Insurance 389 Jump process 179, 391, 397 Kalman filter 109, 230, 239-242, 248, 252, 309,312,361-369 Lagrange's equations 99 Laplace transform 81-83 Large deviation theory 198,379--441 and control 405--414 and equilibrium distributions 399--400 and expected exit times 401 and first passage 415--430 and imperfect observation 431--441 and nonlinear filtering 437--441 refinements 397-399 Linear least square estimate 243 Linearisation 93, 97-98 Loop operator 66 LEQG models and optimisation 234, 295-320,316--317,371-377 LQ models and optimisation 22-28, 33-38,38--42,59-61, lll-129, 147, 150-163 LQG models and optimisation 189-191, 201-202,202-205,331-369

INDEX

LQG models with imperfect observation 229--253 Machine maintenance 22I-222, 28I-282, 289-29I Maximum principle (MP) l3l-166 Maximum principle, risk-sensitive (RSMP) 407-408,435-436 Minimal discrepancy estimate 243 Minimaxcriterion 169 Miss-distance 17 Modes, of a system 95,106 Moment generating function (MGF) 182, 384, 455-456 Monotonicity, of operators 48 Moving average processes and representations 261-262 Multi-armed bandits 269--283 Neurotic breakdown 303 Newton-Raphson algorithm 57-59 Negative programming 50 Neighbouring optimal control 42-46 Noise power matrix 186 Nonlinear filtering 437-441 Notation 443-447 Observable 173, 449 Observer 109 Observability lOfr 109 Observation, imperfect 9fr98, 229-253, 2~291, 308-3I2, 357-369, 431-44I Occupation times 388-389 Offset 89,116, 123-I26 Open-loop control 25, 19I Operator, forward 48 Operator, loop 66 Operator, translation 27 Operators, factorisation of See Canonical factorisation of operators Optimal stopping 205-206 Optimality equation See Dynamic programming equation Optimality conditions See Direct trajectory optimisation Optimisation criteria, expressions for 265-268 Optimism 302 Orthogonal random variables 239

463

Parametrising variables 171, 174, 450 Paststress 308-310, 311-3I2 Pendulum models 34-36,84-85,94,95 Pessimism 303 PID controllers 99 PI! NR algorithm 344, 348 Piecewise deterministic processes 180-I81, 209-213 Plant II, 64, 66, 85 Plant equation II Plant instability 128 Poisson process 183 Poisson stream 183 Pole cancellation 88 Policy 16, 174 Policy improvement 5fr61, 215-228 for call routing 226-228 for machine maintenance 22I-222 for queueing models 222-226 Policy improvement and canonical factorisation 342-344, 348 Pontryagin maximum principle See Maximum principle Positive programming 51 Posterior distribution 287 Prediction 264-265, 305, 350 Primer 149 Process variable II Production scheduling 20-21 Projection estimate 242-246 Proper filter 84-87 Queueing models 193-194, 222-226, 282 Rate function 380, 385, 394 Rational transfer function 71 Realisation 109 Recoupling 310, 374 Recurrence 219 Reference signal See Command signal Regulation 23 Replica Markov processes 391 Reservoir optimisation 40-42, I65-166 Resonance 96 Restless bandits 277-283 Return difference 355 Riccati equation 23-24, ll3, 240, 252, 305, 309,311,338, 34(}-341

464

INDEX

Riccati equation, alternative form of 121-122,253 Risk-sensitivity 172-173, 295-320, 406--414, 432-437 Robustness 326-328 Routh-Hurwicz criterion 109 Satellite model 98, 106, 109 Scaling of processes 195, 384, 391 Sensitivity 327-328 Separation principle 299,432-435 Setpoint 23 Shot noise 184 Small gain theorem 327 Spectral density {function and matrix) 259-261 Stability 3, 92 Stability, internal 87 Stability, local 93 Stability matrix 40, 92 Stability, of filters 69, 71, 83-85 Stabilisability 103 State structure ll-12, 91-110,175-176 Stationary policies 16, 115-121 Stationary processes 255-257 Stopping set 136, 139, 151, 394, 415 Stress 298 future 301,305-307,311 past 301,308-310,311-312 Submodularity 274 Switching locus 148 System formulation 125 Tangency condition 205-206 Temporal optimisation, bases of 449-453

Terminal cost 49 Terminal conditions 138-140, 205-206 Tilting, of a distribution 385 Time-homogeneity 16, 68 Time-integral methods 331-377,407, 435-436 See also Direct trajectory optimisation Time-integral methods, a generalised formulation 375-377 Time-invariance 16 Time-to-go 16 Tracking 38-42, 65, 115--121, 308 Transfer function 69, 75, 79 Transient cost 54 Transient response 68, 80 Transition intensity 179 Translation invariance 68 Translation operator 27 Transversality conditions 138-140, 163-165 Turnpike 56, 133 Twisting, of a process 395 Type number 88 Utility function 295 Value function 13, 174, 339, 347 von Neumann-Gale model 56, 134 White noise 185, 189 Wiener filter 264--265 Wiener process 184--187 Wold representation 263 z-transfonn 69, 77, 79 Zermelo's problem 144