Optimal Control and Viscosity Solutions of Hamilton-Jacobi-Bellman Equations

Job #: 111369

Author Name: Bardi

Title of Book: Optimal Control

ISBN #: 9780817647544

Modern Birkh~user Classics

Many of the original research a n d survey m o n o g r a p h s in pure a n d applied m a t h e m a t i c s published by Birkh~user in r e c e n t d e c a d e s have b e e n g r o u n d b r e a k i n g a n d have c o m e to be r e g a r d e d as foundational to the subject. Through the MBC Series, a select n u m b e r of t h e s e m o d e r n classics, entirely u n c o r r e c t e d , are being re-released in p a p e r b a c k (and as eBooks) to e n s u r e that t h e s e treasures remain accessible to new g e n e r a t i o n s of s t u d e n t s , scholars, a n d researchers.

Optimal Control and Viscosity Solutions of Hamilton-Jacobi-Bellman Equations

Martino Bardi

Italo Capuzzo-Dolcetta

R e p r i n t o f the 1997 E d i t i o n

Birkh~iuser Boston 9 Basel ~ Berlin

Italo Capuzzo-Dolcetta Dipartimento di Matematica Universi~ di R o m a "La Sapienza" 00185 R o m a Italy

Martino Bardi Dipartimento di Matematica P. ed. A. Universith di Padova 35131 Padova Italy

Originally published in the series Systems & Control: Foundations & Applications

ISBN-13:978-0-8176-4754-4 DOI: 10.1007/978-0-8176-4755-1

e-ISBN-13:978-0-8176-4755-1

Library of Congress Control Number: 2007940263 Mathematics Subject Classification (2000): 49L20, 49L25, 35F20, 90D25 @2008 Birkh~iuser Boston All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Birkh~iuser Boston, c/o Springer Science+Business Media LLC, 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. Cover design by Alex Gerasev. Printed on acid-free paper. 987654321

www.birkhausercom

Martino Bardi

Italo Capuzzo-Dolcetta

Optimal Control and Viscosity Solutions of Hamilton-Jacobi-B ellman Equations

Birkh~iuser Boston

9 Basel ~ Berlin

Martino Bardi Dipartimento di Matematica P. ed A. Universit~ di Padova 35131 Padova Italy [email protected]

Italo Capuzzo-Dolcetta Dipartimento di Matematica Universit~ di Roma "La Sapienza" 00185 Roma Italy capuzzo @mat.uniromal.it

Library of Congress Cataloging-in-Publication Data Bardi, M. (Martino) Optimal control and viscosity solutions of Hamilton-Jacobi-Bellman equations / Martino Bardi, Italo Capuzzo Dolcetta. p. cm. -- (Sytems and control) Includes bibliographical references. ISBN 0-8176-3640-4 (hardcover : alk. paper) 1. Viscosity solutions. 2. Control theory. 3. Differential games. I. Capuzzo Dolcetta, I. (Italo), 1948II. Title. III. Series: Systems & control. QA316.B 33 1997 515' .64--dc21 97-23275 CIP AMS Classifications:

49L20, 49L25, 35F'20, 90D25

Printed on acid-free paper 9 1997 Birkh~iuser Boston

Birkh~user

Copyright is not claimed for works of U.S. Government employees. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without prior permission of the copyright owner. Permission to photocopy for internal or personal use of specific clients is granted by Birkh~iuser Boston for libraries and other users registered with the Copyright Clearance Center (CCC), provided that the base fee of $6.00 per copy, plus $0.20 per page is paid directly to CCC, 222 Rosewood Drive, Danvers, MA 01923, U.S.A. Special requests should be addressed directly to Birkh~user Boston, 675 Massachusetts Avenue, Cambridge, MA 02139, U.S.A. ISBN 0-8176-3640-4 ISBN 3-7643-3640-4 Camera-ready text provided by the authors in E~TEX. Printed and bound by Edwards Brothers, Ann Arbor, MI. Printed in the U.S.A. 987654321

Contents

Preface Basic notations

xi xv

C h a p t e r I. O u t l i n e o f t h e m a i n i d e a s o n a m o d e l p r o b l e m 1. T h e infinite horizon discounted regulator . . . . . . . . . . . . . . . 2. T h e D y n a m i c P r o g r a m m i n g Principle . . . . . . . . . . . . . . . . . 3. T h e H a m i l t o n - J a c o b i - B e l l m a n e q u a t i o n in the viscosity sense . . . . 4. C o m p a r i s o n , uniqueness and stability of viscosity solutions . . . . . 5. Synthesis of o p t i m a l controls and verification t h e o r e m s . . . . . . . 6. P o n t r y a g i n M a x i m u m Principle as a necessary and sufficient c o n d i t i o n of o p t i m a l i t y . . . . . . . . . . . . . . . . . . . . . . . . . 7. Discrete t i m e D y n a m i c P r o g r a m m i n g and convergence of approximations .............................. 8. T h e viscosity a p p r o x i m a t i o n and stochastic control . . . . . . . . . . 9. Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . .

1 1 2 3 6 10

17 20 21

Chapter II. Continuous viscosity solutions of Hamilton-Jacobi equations 1. Definitions and basic properties . . . . . . . . . . . . . . . . . . . . 2. Some calculus and further properties of viscosity solutions . . . . . 3. S o m e c o m p a r i s o n and uniqueness results . . . . . . . . . . . . . . . 4. Lipschitz continuity and semiconcavity . . . . . . . . . . . . . . . . 4.1. Lipschitz continuity . . . . . . . . . . . . . . . . . . . . . . . . 4.2. S e m i c o n c a v i t y . . . . . . . . . . . . . . . . . . . . . . . . . . . 5. Special results for convex H a m i l t o n i a n s . . . . . . . . . . . . . . . . 5.1. Semiconcave generalized solutions and bilateral supersolutions 5.2. Differentiability of solutions . . . . . . . . . . . . . . . . . . . . 5.3. A c o m p a r i s o n t h e o r e m . . . . . . . . . . . . . . . . . . . . . . . 5.4. Solutions in t h e e x t e n d e d sense . . . . . . . . . . . . . . . . . .

25 25 34 50 61 62 65 77 77 80 82 84

13

vi

CONTENTS 5.5. Differential inequalities in the viscosity sense . . . . . . . . . . 5.6. Monotonicity of value functions along trajectories . . . . . . . 6. Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . .

Chapter I I I . O p t i m a l c o n t r o l p r o b l e m s w i t h c o n t i n u o u s v a l u e functions: u n r e s t r i c t e d s t a t e s p a c e 1. The controlled dynamical system . . . . . . . . . . . . . . . . . . . . 2. The infinite horizon problem . . . . . . . . . . . . . . . . . . . . . . 2.1. Dynamic Programming and the Hamilton-Jacobi-Bellman equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2. Some simple applications: verification theorems, relaxation, stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3. Backward Dynamic Programming, sub- and superoptimality principles, bilateral solutions . . . . . . . . . . 2.4. Generalized directional derivatives and equivalent notions of solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5. Necessary and sufficient conditions of optimality, minimum principles, and multivalued optimal feedbacks . . . . . . . . . . 3. The finite horizon problem . . . . . . . . . . . . . . . . . . . . . . . 3.1. The HJB equation . . . . . . . . . . . . . . . . . . . . . . . . . 3.2. Local comparison and unbounded value functions . . . . . . . 3.3. Equivalent notions of solution . . . . . . . . . . . . . . . . . . 3.4. Necessary and sufficient conditions of optimality and the Pontryagin Maximum Principle . . . . . . . . . . . . . . . 4. Problems whose H J B equation is a variational or quasivariational inequality . . . . . . . . . . . . . . . . . . . . . . . 4.1. The monotone control problem . . . . . . . . . . . . . . . . . . 4.2. Optimal stopping . . . . . . . . . . . . . . . . . . . . . . . . . 4.3. Impulse control . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4. Optimal switching . . . . . . . . . . . . . . . . . . . . . . . . . 5. Appendix: Some results on ordinary differential equations . . . . . . 6. Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . .

86 90 95

97 97 99 99 110 119 125 133 147 147 156 160 170 183 184 193 200 210 218 223

Chapter I V . O p t i m a l c o n t r o l p r o b l e m s with continuous value functions: restricted state space 1. Small-time controllability and minimal time functions . . . . . . . . 2. HJ B equations and boundary value problems for the minimal time function: basic theory . . . . . . . . . . . . . . . . . . . . . . . 3. Problems with exit times and non-zero terminal cost . . . . . . . . . 3.1. Compatible terminal cost and continuity of the value function 3.2. The H J B equation and a superoptimality principle . . . . . . . 4. Free boundaries and local comparison results for undiscounted problems with exit times . . . . . . . . . . . . . . . . . . . . . . . . 5. Problems with state constraints . . . . . . . . . . . . . . . . . . . . 6. Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . .

227 227 239 247 248 254 261 271 282

CONTENTS

vii

Chapter V. Discontinuous viscosity solutions and applications 285 1. S e m i c o n t i n u o u s sub- and supersolutions, weak limits, and stability . 286 2. N o n - c o n t i n u o u s solutions . . . . . . . . . . . . . . . . . . . . . . . . 295 2.1. Definitions~ basic properties, and e x a m p l e s . . . . . . . . . . . 295 2.2. E x i s t e n c e of solutions by P e r r o n ' s m e t h o d . . . . . . . . . . . . 302 3. E n v e l o p e solutions of Dirichlet problems . . . . . . . . . . . . . . . . 308 3.1. E x i s t e n c e and uniqueness of e-solutions . . . . . . . . . . . . . 309 3.2. T i m e - o p t i m a l problems lacking controllability . . . . . . . . . . 313 4. B o u n d a r y conditions in the viscosity sense . . . . . . . . . . . . . . 316 4.1. M o t i v a t i o n s and basic properties . . . . . . . . . . . . . . . . . 317 4.2. C o m p a r i s o n results and applications to e x i t - t i m e p r o b l e m s and stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326 4.3. Uniqueness and c o m p l e t e solution for t i m e - o p t i m a l control . . 333 5. Bilateral supersolutions . . . . . . . . . . . . . . . . . . . . . . . . . 342 5.1. P r o b l e m s w i t h exit times and general targets . . . . . . . . . . 342 5.2. F i n i t e horizon problems w i t h constraints on the e n d p o i n t of the trajectories . . . . . . . . . . . . . . . . . . . . . . . . . 348 6. Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . 357 Chapter VI. Approximation and perturbation problems 359 1. Semidiscrete a p p r o x i m a t i o n and s - o p t i m a l feedbacks . . . . . . . . . 359 1.1. A p p r o x i m a t i o n of the value function and c o n s t r u c t i o n of o p t i m a l controls . . . . . . . . . . . . . . . . . . . . . . . . . 360 1.2. A first result on t h e rate of convergence . . . . . . . . . . . . . 369 1.3. I m p r o v i n g the r a t e of convergence . . . . . . . . . . . . . . . . 372 2. R e g u l a r p e r t u r b a t i o n s . . . . . . . . . . . . . . . . . . . . . . . . . . 376 3. Stochastic control w i t h small noise and vanishing viscosity . . . . . 383 4. A p p e n d i x : D y n a m i c P r o g r a m m i n g for Discrete T i m e S y s t e m s . . . 388 5. Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . 395 Chapter VII. Asymptotic problems 1. Ergodic p r o b l e m s . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1. Vanishing discount in the s t a t e constrained p r o b l e m . . . . . . 1.2. Vanishing discount in the unconstrained case: optimal stopping .......................... 2. Vanishing switching costs . . . . . . . . . . . . . . . . . . . . . . . . 3. P e n a l i z a t i o n . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1. P e n a l i z a t i o n of stopping costs . . . . . . . . . . . . . . . . . . . 3.2. P e n a l i z a t i o n of s t a t e constraints . . . . . . . . . . . . . . . . . 4. Singular p e r t u r b a t i o n problems . . . . . . . . . . . . . . . . . . . . . 4.1. T h e infinite horizon p r o b l e m for systems w i t h fast components ............................. 4.2. A s y m p t o t i c s for the m o n o t o n e control p r o b l e m . . . . . . . . . 5. Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . .

397 397 397 402 411 414 414 417 420 420 426 429

viii

CONTENTS

Chapter VIII. Differential Games

431

1. D y n a m i c P r o g r a m m i n g for lower a n d u p p e r values . . . . . . . . . . 2. E x i s t e n c e of a value, r e l a x a t i o n , v e r i f i c a t i o n t h e o r e m s . . . . . . . . 3. C o m p a r i s o n w i t h o t h e r i n f o r m a t i o n p a t t e r n s a n d o t h e r n o t i o n s of v a l u e . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1. F e e d b a c k s t r a t e g i e s . . . . . . . . . . . . . . . . . . . . . . . . 3.2. A p p r o x i m a t i o n b y d i s c r e t e t i m e g a m e s . . . . . . . . . . . . . . 4. B i b l i o g r a p h i c a l n o t e s . . . . . . . . . . . . . . . . . . . . . . . . . . Appendix

A. Numerical

Solution

of Dynamic

1. T h e infinite h o r i z o n p r o b l e m . . . . . . . . . . . . . . . . . . . . . . 1.1. T h e D y n a m i c P r o g r a m m i n g e q u a t i o n . . . . . . . . . . . . . . 1.2. S y n t h e s i s of f e e d b a c k c o n t r o l s . . . . . . . . . . . . . . . . . . . 1.3. N u m e r i c a l t e s t s . . . . . . . . . . . . . . . . . . . . . . . . . . . 2. P r o b l e m s w i t h s t a t e c o n s t r a i n t s . . . . . . . . . . . . . . . . . . . . 3. M i n i m u m t i m e p r o b l e m s a n d p u r s u i t - e v a s i o n g a m e s . . . . . . . . . 3.1. T i m e - o p t i m a l c o n t r o l . . . . . . . . . . . . . . . . . . . . . . . 3.2. P u r s u i t - e v a s i o n g a m e s . . . . . . . . . . . . . . . . . . . . . . . 3.3. N u m e r i c a l t e s t s . . . . . . . . . . . . . . . . . . . . . . . . . . . 4. S o m e h i n t s for t h e c o n s t r u c t i o n of t h e a l g o r i t h m s . . . . . . . . . . 5. B i b l i o g r a p h i c a l n o t e s . . . . . . . . . . . . . . . . . . . . . . . . . .

1. 2. 3. 4. 5. 6. 7. 8.

448 448

457 468

Programming

Equations by Maurizio Falcone

7-/o~ c o n t r o l by Pierpaolo Soravia Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Linear systems .............................. 7-/0~ c o n t r o l a n d differential g a m e s . . . . . . . . . . . . . . . . . . . Dynamic Programming equation .................... On the partial information problem .................. Solving t h e p r o b l e m . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bibliographical notes ..........................

Appendix B . N o n l i n e a r

433 442

471 472 473 478 481 484 488 488 494 496 499 502 505 506 510 512 515 521 527 530 531

Bibliography

533

Index

565

To Alessandra and Patrizia

Preface

The purpose of the present book is to offer an up-to-date account of the theory of viscosity solutions of first order partial differential equations of Hamilton-Jacobi type and its applications to optimal deterministic control and differential games. The theory of viscosity solutions, initiated in the early 80's by the papers of M.G. Crandall and P.L. Lions [CL81, CL83], M.G. Crandall, L.C. Evans and P.L. Lions [CEL84] and P.L. Lions' influential monograph [L82], provides an extremely convenient PDE framework for dealing with the lack of smoothness of the value functions arising in dynamic optimization problems. The leading theme of this book is a description of the implementation of the viscosity solutions approach to a number of significant model problems in optireal deterministic control and differential games. We have tried to emphasize the advantages offered by this approach in establishing the well-posedness of the corresponding Hamilton-Jacobi equations and to point out its role (when combined with various techniques from optimal control theory and nonsmooth analysis) in the important issue of feedback synthesis. The main ideas are introduced in Chapter I where the infinite horizon discounted regulator problem is taken as a model. After the derivation of the Hamilton-JacobiBellman equation from the Dynamic Programming optimality principle we cover, in a rather informal way, such topics as uniqueness, stability and necessary and sufficient conditions for optimality. A quick review of discrete time and stochastic approximations to the value function is given in the last two sections. Chapter II is devoted to the basic theory of continuous viscosity solutions. A long section of this chapter deals with the case of Hamilton-Jacobi-Bellman equations, corresponding to Hamiltonians which are convex with respect to the gradient variable. This is the relevant case in connection with optimal control problems. In particular we discuss the connections between viscosity solutions and some different notions, such as Lipschitz continuous functions solving the equation almost everywhere, Barton and Jensen's bilateral supersolutions, and solutions in the extended sense of Clarke. In the next two chapters the basic theory of Chapter II is specialized and developed with reference to various optimal control problems with continuous value

xii

PREFACE

function. We made the choice to present in Chapter III some problems with unrestricted state space, corresponding to the simpler case of Hamilton-Jacobi equations without boundary conditions. We note that Section 4 deals with some quasivariational inequalities and systems arising in connection with stopping times, impulse control, or switching costs. On the other hand, Chapter IV is dedicated to problems involving exit times from a given domain or constraints on the state variables for systems having suitable controllability properties, leading to boundary value problems. Chapter V is dedicated to the case of discontinuous value functions, a typical motivation coming from minimum time problems lacking controllability. Various notions of discontinuous viscosity solutions, including those of Barles and Perthame, Ishii, Barron and Jensen, and Subbotin, are discussed and compared. Section 1 of this chapter is of independent interest since there we develop the so-called weak limits technique of Barles and Perthame, a very useful tool which we adopt to tackle the various limit problems in Chapters VI and VII. In particular, in Chapter VI we consider an approximation scheme for value functions based on discrete time Dynamic Programming. This is an important issue in the applications because it also provides a method to construct almost optimal feedback controls. The convergence of the scheme and some estimates on the rate of convergence are proved by viscosity solutions methods. A section is dedicated to regular perturbation problems where similar techniques can be employed. Chapter VII deals with the analysis of some asymptotic problems such as singular perturbations, penalization of state constraints, vanishing discount and vanishing switching costs. The limiting behavior of the associated Hamilton-Jacobi equations is analyzed in a simple way by the viscosity solutions approach and the weak limit technique. The last chapter is intended as an introduction to the theory of two-person zero sum differential games. Different notions of value are discussed together with the derivation of the relevant Hamilton-Jacobi-Isaacs equations. It is worth noting that the viscosity solutions method appears to be highly privileged for the treatment of this kind of nonconvex, nonlinear PDE's. Finally, two appendices deal with some additional topics which are important for the applications. The first one, by M. Falcone, describes in some detail a computational method based on the approximation theory developed in Chapter VI. The second, by P. Soravia, gives a fairly complete account of some recent results on the viscosity solutions approach to 7-/~ control. Our main goal in planning this work was to provide a self-contained but, at the same time, rather comprehensive presentation of the topic to scientists in the areas of optimal control, system theory, and partial differential equations. To this end, each chapter is enriched with a section of bibliographical and historical notes. As is often the case, the present book originated from lecture notes of courses taught by the authors. We believe that the style of presentation (in particular, the set of exercises proposed at the end of each section) reflects this pedagogically oriented origin, and so selected parts could easily be used for a graduate course in

PREFACE

xiii

optimal control. In this regard, a possible path for a one semester course, which we have tested in lectures at Paris-Dauphine, Rome, Padua, and Naples, might include the first three sections of Chapter II, a choice of a few sections from Chapters III and IV (for example, selected parts of sections 1-3 of Chapter III and the first two sections of Chapter IV) and the initial sections of Chapters V and VI. The prerequisites for reading most of the book are just advanced calculus, some very basic functional analysis (uniform convergence and the Ascoli-Arzel~ theorem) as well as, of course, the fundamental facts about nonlinear ordinary differential equations which are recalled in the appendix to Chapter III. These are also the only prerequisites for the course outlined above. More sophisticated mathematical tools such as relaxed controls, 2nd order PDE's, stochastic control and the geometric theory of controllability appear in the book but are not essential for understanding its core. We are happy to thank L.C. Evans and P.L. Lions who introduced us to viscosity solutions in the very early stages of the theory and also the other colleagues and friends with whom we have collaborated~ in particular H. Ishii, B. Perthame, M. Falcone, P. Soravia. We had the occasion to discuss the overall project of the book with W.H. Fleming, M.H. Soner and G. Barles. Additional thanks are due to M. Falcone and P. Soravia for contributing the Appendices. Several people read parts of the manuscript. We are particularly grateful to O. Alvarez, S. Bortoletto and A. Cutr~ for reading the book from cover to cover and for stimulating remarks, and to F. Da Lio, P. Goatin, F. Gozzi, S. Mirica, M. Motta, F. Rampazzo, C. Sartori, F. Sullivan. We also thank A.I. Subbotin, N.N. Subbotina, and A.M. Tarasyev for translating some of their papers in Russian into English for us. Finally, thanks are due to G. Bertin for skilled and sensitive typing.

April 1997

Martino Bardi,

Italo Capuzzo-Dolcetta

Basic notations

RN

the euclidean N-dimensional space

x.y

the Y= the the

scalar product ~ xiyi of vectors x = ( x I , . . . , XN) and (YI,...,-YN) i=1 vector space generated by the vectors {... } euclidean norm of x c ~N, ix I = ( x . x) 1/2

the the the the the the the

open ball {x 9 R N : IX -- X01 "~ r } closed ball {x 9 ]RN : Ix - x0] _< r} b o u n d a r y of the set E interior of the set E closure of the set E complement of the set E convex hull of the set E

N

span{... }

Ixl B(xo, r) /~(x0, r) OE int E E Ec co E ~6E ~(X) ~1 CC

the closed convex hull of the set E the set of all subsets of X means fY C_ f~

IEI = m e a s E

the Lebesgue

d( x, E) = dist(x, E ) diam E

the diameter of the set E (i.e., diam E -- sup{Ix - Yl : x, y 9

dH(E, S)

the Hausdorff

n(x)

the outward

~Ao ",/Va

min{%

a} for V, cr 9

max{%

a} for % cr 9 R

,7+., ,7-

the positive and negative part of '7 9 R (i.e., V + = "7 v 0,

measure

of the set E

the distance from x to E (i.e., d(x, E) = infyeE Ix -- Yl)

E}) distance between normal

unit vector to a set E at x 9 0E

~- =-7 A O) [r]

the sets E and S (w

the integer part of r 9

xvi

BASIC NOTATIONS

sgn r

the sign of r E l~ (1 if r > 0, - 1 if r < 0, 0 if r = 0)

arg m i n E u

the set of m i n i m u m points of u : E ---* 1~

IluH~

the s u p r e m u m n o r m SUPxeE lu(x)[ of a function u : E --~ IR

ttn "-~ U

the sequence of functions u~ is nonincreasing a n d tends to u

Un / / ~ U

the same as the preceding, b u t un is nondecreasing

Un ==~ U

uniform convergence of the functions u~ to u

supp u

the support of the function u, i.e., the closure of the set

{x: u(x) # 0} Xx

the characteristic (or indicator) function of the set X (1 in X , 0 outside)

Od

a modulus, i.e., a function aJ: [0, +oc[ ~ [0, +co[ continuous, nondecreasing, a n d such that w(0) = 0; or, more generally, w : [0, +ec[ 2 ~ [0, +ec[ such t h a t for all R > 0 w( 9 R) has the preceding properties

o(t) as t ---* a Du(x)

a function u such t h a t l i m t ~ a u ( t ) / t = 0 the gradient of the function u at

OU (x ,

x,

i.e.,

Du(x) =

Ou (x))

-5-~Xl ) .... D+u(x), D-~(x) O~u(x) O~(x)

the super- a n d subdifferential of u at x (w

D*u(x) Ou(x; q)

{p e ]t~N: p ----lim,~--,+oc Du(xn), x,~ ~ x} (w

the subdifferential of the convex function of u at x (w the Clarke's gradient of u at x (w the (one-sided) directional derivative of u at x in the direction q (w

O+u(x; q), o-~,(x; q)

the (generalized) Dini directional derivatives (w w

uo(x; q), u~

the regularized directional derivatives (w

q)

and

N 02u

Au

the laplacian of the function u, i.e., A u = i=1 ~ 0X/2

u_(x) = lim i n f . u z ( x )

the lower weak limit of uc at x as s --~ 0 + (w

g(x) = lira sup*ue(x)

the upper weak limit of uz at x as z -+ 0 + (w

c~O +

e~O+ U., U*

A Ar

vx(t,~) tx(~)

the (w the a : the

lower a n d the upper semicontinuous envelopes of u set of controls, i.e., (Lebesgue) measurable functions [0, +c~[ ~ A set of relaxed controls (w

the state at time t of a control system (w the first entry time of the state in some given closed set T (w w and w the first entry time of the state in i n t T

(w

BASIC NOTATIONS

f(x,A) Dxf B(E)

{q 9

]I~N :

xvii

q = f(x, a) for some a 9 A}

the Jacobian matrix of f with respect to the x variable

C(E)

the space of functions u : E --* R with Ilullo~ < + o c the space of continuous functions u : E --*

Be(E)

the space B(E) A C(E)

UC(E)

the space of uniformly continuous functions u : E --* R

BUC(E)

the space B(E) n UC(E) the space of Lipschitz continuous functions u : E --* R, i.e., such t h a t for some L > 0 lu(x) - u(y)l < L I x - Yl for all

Lip(E)

x, y E E Liplor

C~

the space of locally Lipschitz continuous functions u : E -* ~ , i.e., functions whose restriction to any compact subset of E is Lipschitz continuous the space of "~-Hhlder continuous functions u : E ---*R, "~ 9 ]0, 1[, i.e., sup lu(x) - u(Y)l < +oo

x,yeE Iz- yP

Ck(~) Ck(E) USC(E), LSC(E) BUSC(E), BLSC(E) BCo(f~)

for k >_ 1 and f~ open subset of R N, the subspace of C(f~) of functions with continuous partial derivatives in f~ up to order k for E c_ ~lN, the space of functions that are the restrictions to E of some u E Ck(f~) for some open set f~ _D E the spaces of lower and upper semicontinuous functions u : E --* I~ the spaces uSe(E) N B(E) and LSC(E) A B(E) the subspace of B(f~) of the functions continuous at all points of 0f~

Ll([to, t:])

the Lebesgue space of integrable function [to, t:] ---* IR

L~([t0, tl])

the

L 1(0, T; E)

the space of integrable function [0, T] --* E the space of essentially bounded function [0, T] --* E small-time controllability on T (w

Lebesgue

space

of essentially bounded

function

[to, tl] --* L~162 T; E) STCT STLC

small-time local controllability (w

STCT-

small-time controllability on int T (w

F, A

nonanticipating strategies for the first and the second player

<:]

feedback strategies for the first and the second player (w end of a proof end of a definition, end of a remark, end of an example

(w

CHAPTERI

Outline of the main ideas on a model problem

The purpose of this introductory chapter is to motivate the relevance of the notion of viscosity solution of partial differential equations of the form

F(x, u(x), Du(x)) = 0 in a Dynamic Programming approach to deterministic optimal control theory. The important role played in this respect by the theory of viscosity solutions is exhibited with reference to the classical infinite horizon discounted regulator problem. The presentation is rather informal in its aim of pointing out the main ideas. For detailed treatment of all topics discussed here the reader is referred to subsequent chapters. 1. T h e i n f i n i t e h o r i z o n d i s c o u n t e d

regulator

Consider a control system governed by the state equation

(i.i)

y'(t) = f(y(t), a(t)), y(0)

t>O,

= x.

Here, the control a is any measurable function of t E [0, +oc[ with values in A, the control space (typically, A is a closed bounded set in R M or, more generally, A is a topological space). Assume that the dynamics f : ~N • A ~ ~ N is such that, for any choice of the control a and of the initial position x E ~ N the state equation (1.1) has a unique solution defined for all t E [0, +oz[, denoted by yx(t, a). The model also includes a running cost, associated with this controlled evolution, described by a given function g : ~N X A --* It~. The cost functional to be minimized is

(i.2)

J(x, ~) =

~0 +~

e(yx(t, ~), ~(t)) e - ~ dr,

2

I. OUTLINE OF THE MAIN IDEAS

where A > 0 represents a (fixed) discount factor. The first step in the standard dynamic programming approach to the optimal control problem described above is to introduce the value function v, defined by v ( x ) : = inf J ( x , a ) ,

(1.3)

aEA

where ,4 denotes the set of all measurable functions a : [0, +cc[ ~ A. The fundamental idea of dynamic programming is that v satisfies a functional equation, often called the Dynamic Programming Principle, and when v is smooth enough, its infinitesimal version, the Hamilton-Jacobi-Bellman (HJB) equation. These equations contain all the relevant information needed to design, through the so-called synthesis procedure, an optimal feedback map for the problem. In the subsequent Sections 2-5 we outline this program in the framework of the theory of viscosity solutions. 2. T h e D y n a m i c

Programming

Principle

Assume temporarily that an optimal control a~ exists for each x, so that v(x) = J ( x , ~ ; ) =

e(yx(t, ~ ; ) , ~ ; ( t ) ) e -At at.

Observe that, for any fixed T > O,

J(x, ax) =

/o

e(yx(t, a;), a;(t)) e -At dt +

/j

f(yx(t, a*), a*(t)) e -At dt

A simple argument based on the semigroup property (2.1)

yx(t + s, ~ ) =

yy~(t,o;)(s, ~ ( "

+t))

Vt, s > 0,

leads to the following identity (2.2)

v(x) =

g(yx(t, a*), a;(t)) e -At dt + v(yx(T, a *)) e -AT

which holds for all T > 0 and x E R g. In the general case (i.e., when the existence of optimal controls is not assumed), (2.2) is replaced by (2.3)

v(x) = inf

g(yx(t, a), a(t)) e -At dt + v(yx(T, a)) e -AT} .

~E.A ~ Jo

The functional equation (2.3) comprises the statement of the Dynamic Programming Principle for our problem. An important fact to be stressed at this point is that (2.3) holds for all x E ] ~ g and T > 0 under very general conditions on the data. When e and, consequently, v is bounded, (2.3) characterizes the value function v in the sense that if u is a bounded function satisfying (2.3) for all x E ]I~N and T > 0, then u ~- v.

3. THE HJB EQUATION IN THE VISCOSITY SENSE

3

Let us observe, finally, that the semigroup property (2.1) has a counterpart at the level of the dynamic optimization problem that leads to a more abstract viewpoint on the Dynamic Programming Principle. More precisely, consider the one parameter family of nonlinear operators {Gt} defined by t

(Gt~)(x) = inf { ~0 g(yx(s, a), a ( s ) ) e -As ds + ~(yz(t, ~))e -~t } (2.4) =: inf(Gp~)(x). c~

It is not hard to check that {Gt} has the following properties (2.5)

Gt7~ 9 B(][(N) Go~ = c2,

Gt+s~ = Gt(Gs~)

Vt, s > o

for all W 9 B(RN), the space of bounded functions o n ]I~N. Moreover, from the inequality - J(x, ~)1 ~- (/~-1 s u p lel + s u p I~1)

[(G~)(x)

e-;~t,

which holds for any fixed a, we get (2.6)

lim IIC,~ - vl[~ = o,

t~+~

v~ 9

B(]I(N),

where ]]. IIor denotes the sup-norm. On the other hand, the semigroup property in (2.5) yields lim Gt+TT = GT tfiInoo Gt~ . t~+oc Hence, by (2.6), v satisfies (2.7)

v(x) = (GTv)(x)

V x 9 ~Y., T > O.

The preceding says that the value function v is invariant for the dynamical system defined on B ( R N) by the semigroup {Gt}. Of course, (2.7) is an equivalent formulation of the Dynamic Programming Principle (2.3). 3. T h e H a m i l t o n - J a c o b i - B e l l m a n

e q u a t i o n in t h e v i s c o s i t y s e n s e

In this section we derive an infinitesimal version of the Dynamic Programming Principle (2.3) or (2.7) in the form of a first order nonlinear partial differential equation satisfied by the value function v, the Hamilton-Yacobi-Bellman equation: (HJB)

Av(x) + s u p { - f ( x , a ) . aEA

Dv(x) - g(x,a) } = O .

Here, Dr(x) denotes the gradient of v at the point x.

4

I. OUTLINE OF THE MAIN IDEAS

The derivation of (HJB) requires, of course, some smoothness of v. In this event, (HJB) holds at all points where v is differentiable. This follows from (2.3) by s t a n d a r d arguments (divide both sides of (2.3) by T > 0 and let T --* 0 ... ). This classical result can be found in many books (see the notes at the end of the chapter for references). A complete proof appears in Chapter III. It is well known, however, that everywhere differentiability is a too restrictive assumption on v. A simple explicit example is the following. EXAMPLE 3.1. Consider the infinite horizon problem with N = 1, A = { - 1 , 1}, f ( x , a) = a. Suppose t h a t ~(x, a) = ~(x) is a smooth function such that g(x)=i(-x),

f---0

maxg=~(0)>0,

forlx]>R,

x#(x)
for I x l < R .

It is easy to realize that an optimal control is a*(t) = sgnx for x 7~ 0, whereas, for x = 0, both a*(t) - 1 and a*(t) = - 1 are optimal since ~ is even. A simple computation then leads to the following expression for the value function, fo +~ g(x - t) v(x) =

~o+~

e-At dt,

e ( - t ) e -At dt =

x < O,

~o+cr

f(t) e -At dt,

, fo +~ t ( x + t) e -At dt,

x = O, x > 0.

Hence v~_ (0)

lira v(z) - v(0) = [ + ~ X J0

e_Atdt

x~0 +

lira v ( x ) - v(O) _ [ + ~ g ( _ t ) e_At dt " X Jo Since g ' ( - x ) = - e ' ( x ) , v is not differentiable at x = 0. The (HJB) equation for this example is

Vt_._(O)

x-*O-

~v(x) +

I v ' ( x ) l - g ( x ) = 0;

of course the above has no classical meaning for x = 0 since v is not differentiable at x - - 0 . <3 The preceding discussion shows that the derivation of (HJB) should be performed in some weak sense. Observe that under rather general conditions on the data, including the boundedness of g, the value function is continuous and bounded in IRg . Let us describe then how to interpret (HJB) in the viscosity sense assuming that v is continuous in ]RN. For this purpose, we consider the sets D + v ( x ) = {p E ll@r: limsup v ( y ) - v ( x ) - p . ~x

ly-

(y - x) < 0}

~l

D-v(x) = {q E I~N l i m i n f V ( y ) - v ( x ) - q ' ( y - x ) : y--z

l Y - x[

-

>0}, -

3. THE HJB EQUATION IN THE VISCOSITY SENSE

5

called, respectively, the superdifferential and the subdifferential of v at x. Note t h a t if b o t h D+v(x) and D - v ( x ) are n o n e m p t y at some x, then D+v(x) = D - v ( x ) = {Dv(x)} and v is differentiable at x. Following [CL83] and [CEL84], a continuous function u is a viscosity solution of the nonlinear p a r t i a l differential equation

F(x, u(x), Du(x)) = 0,

(3.1)

x E ll~N.9

if the following conditions are satisfied: (i)

F(x,u(x),p) <_0

VxE~N

VpED+u(x)

(ii)

F(x, u(z), q) > 0

Vx E ]~N, Vq E D - u ( x ) .

A n y u satisfying (i) will be called a viscosity subsolution of (3.1) whereas, if (ii) holds, then u is a viscosity supersolution. A n equivalent way to formulate conditions (i) and (ii) is in t e r m s of test functions. Namely, (i) holds if and only if (j)

for any ~2 E C 1, if x is a local m a x i m u m point for u - ~, then

F(x, u(x), D~(x)) <_ 0; similarly, (ii) holds if and only if (jj) for any qz E C 1 , if x is a local m i n i m u m point for u - p, then

F(x, u(x), D~(x)) > O. We refer to C h a p t e r II for the p r o o f of the equivalence. Let us now sketch the proof t h a t if the value function v is continuous, then it satisfies (HJB) in the viscosity sense. We first check the subsolution condition (i) which, in the present case where

F(x, u,p) = ~u + sup{ - f ( x , .) .p - t(x, a)

},

aEA

reads as (3.2)

Av(x)+sup{-f(x,a).p-g(x:a)}

<_O,

VpED+v(x), Vxe]~ N .

aEA

To see this, choose (~(t) -- a with a r b i t r a r y fixed a E A in the D y n a m i c P r o g r a m ming Principle (2.3) to o b t a i n (3.3)

v(yx(T,a)) - v(x) + ~0 T t(yx(t,a),a)e-;~t dt + V(yx(T,a))(e -;~T - 1) > O.

For any p E D+v(x) we have, for small enough T > 0,

v(yx(T, a)) - v(x) < p. (yx(T, a) - x) + o(lyz(T , a - xl) ,

6

I. O U T L I N E OF T H E MAIN IDEAS

since y~(T, a) --~ x as T --* 0 +. Hence, from (3.3), after dividing by T > 0, (3.4)

1/oT

p yx(T,a)T - x + o([yx(T,a)T - xt) + T

g(yx(t'a)'a)e-:~t dt e -AT - 1

+ v(y~(T, a)) - - > 0 . Observe now that if f and ~ are continuous, then

fT

yx(T,a) - x 1 T - Tjo

f(yx(t,a),a)dt~f(x,a),

T

e(yx(t, a), a) e -At dt ~

e(x, a),

as T --* 0 +. Therefore, if v is continuous, letting T --~ 0 + in (3.4) yields

p . f ( x , a ) + g(x,a) - Av(x) > O . Since a was arbitrary in A, this proves (3.2). To check the supersolution condition, note that for any z > 0 and T > 0 there exists a control ~ (depending on z, T) such that T

(3.5)

v(x) >

g(yx(t,-~),-~(t))e-~t dt + v ( y x ( T , - ~ ) ) e - ~ T - T z .

This follows easily from (2.3). A similar argument as above, with an additional technical difficulty due to the fact that ~ is not necessarily constant, allows us to use (3.5) to conclude t h a t Av(x) + s u p { - f ( x , a ) ,

q-

s

}> 0

V x C R N, V q E D - v ( x ) .

aEA

Therefore, v is a viscosity solution of equation (HJB). Note finally that the above proof shows that (HJB) holds in the classical sense at all points where v is differentiable. For the reader who knows the classical derivation of the HJB equation at points of differentiability of v, we point out that proving t h a t v is a viscosity subsolution is very easy by means of the second definition based on test functions. In fact, at a point x where v - p has a maximum we can replace the term v(yz(T, a)) - v(x) in (3.3) by ~(y~(T, a)) - p(x), preserving the inequality. Then, the passage to the limit can be performed because ~ is C 1. 4. C o m p a r i s o n ,

uniqueness

and stability of viscosity solutions

The topics we discuss in this section form, in a sense, the core of the theory of viscosity solutions, specifically in the direction of applications to optimal control.

4. COMPARISON, UNIQUENESS AND STABILITY

7

On the other hand, from a PDE point of view, they relate to the notion of wellposedness. We have seen in Section 3 that the value function of the infinite horizon problem satisfies, when it is continuous, the HJB equation (HJB)

Av(x) + sup{ - f ( x , a). Dv(x) - g(x, a) } = O,

x E ~N,

aEA

in the viscosity sense, that is (4.1)

Av(x) + s u p { - f ( x , a) .p - g(x, a) } <_ 0

YX

E

]~N p E D+v(x)

aEA

(4.2)

>_0

Av(x)+sup{-f(x,a).q-g(x,a)}

VxE]~N

qED-v(x).

aEA

It is of major importance to point out that v is in fact characterized by conditions (4.1), (4.2) under rather general assumptions on the data f, g, A (see w and w This is a key point since it provides the basis for rigorous justification of the synthesis procedure in the nonsmooth case (see Sections 5-7). Let us present here a prototype uniqueness result for (HJB). Assume that A = 1, f ( x , a) - f(a), g(x, a) - g(x) is uniformly continuous and, in order to avoid technicalities, that v satisfies the "boundary condition" (4.3)

lim v(x) = O. Izl~+oc

The claim is that if u is any bounded continuous function in ~N satisfying (HJB) in the viscosity sense and (4.3), then (4.4)

u --= v .

In the present simplified setting (HJB) becomes (4.5)

v(x) + H(Dv(x)) - g(x) = 0

in ]~N,

with (4.6)

H(p) = s u p { - f ( a ) . p } . aEA

To prove the claim, we exploit the information contained in the subsolution condition (i.e., (4.1)) for u and in the supersolution condition (i.e., (4.2)) for v, to obtain the comparison principle (4.7)

u(x) ~ v(x)

Vx ~]~Y.

The technical tool to this end is to consider the auxiliary function ~ ( x , ~) : = u ( x ) - v ( v ) -

Ix - yl 2

2~

x, y E ]~N

8

I. OUTLINE OF THE MAIN IDEAS

where ~ > 0 is a small parameter. Assume by contradiction that (4.7) is false so t h a t (u - v)(E) > 0 for some 5. Then, by virtue of (4.3) there exists a global maximum point (x~, Yc) for (b. Then x~ is a maximum for ~(x)

:= ~(x) - v(~)

-

Ix -- yzl 2

while ys is a minimum for Ix~ -- yl 2 ~(y)

:= ~(y) - u(x~) + -

-

2~

Consequently, as easily seen from the definition of D +, D - , (4.8)

0 E D+~a(xc),

0 e D-r

Now it is not hard to realize (note that the terms Ix - y~12/2~ and [xe - y12/2E are differentiable) that D+p(x~ ) = D+u(xE )

x~ -

E,

D-~p(y~) = D - v ( y E )

xr - yr

E

(see w

E

for these calculus properties). Therefore, using (4.8), we see that x~ - y..._.~Ec D + u ( x ~ ) A D - v ( y ~ ) . E

Since u and v are, respectively, viscosity sub- and supersolution of (4.5), we then obtain -e(xE)
u(xz)§ \

E

v(y~)+H(xE-Y~)--g(y~)>_O.

/

\

E

/

By addition the above yields u(x~) - v(yz) <_ f(x~) - g(y~). Since, for any x E ~Y, r YE) >_ Cb(x, x) = u ( x ) -- v ( x ) , we deduce the inequality (4.9)

~ ( x ) - v ( x ) < e(x~) - e(y~)

Vx e

~.

From the inequality (I)(xz, y~) > (I)(xc, x~) we get Ix~ - y~l ~ < v ( x ~ ) - v ( y ~ ) .

2E Since v is bounded, the preceding yields the estimate Ixs - yE] <_ C s 1/2 9

Therefore, from (4.9) and the uniform continuity of g, (4.10)

~ t ( X ) -- V ( X ) ~

o2(C~ 1/2)

Vx E ~ N ,

where aJ is a modulus of continuity for g. The above inequality holds for any E > 0, and the this contradicts (u - v)(E) > 0 for small ~. Hence (4.7) is proved. Since u

4. COMPARISON, UNIQUENESS AND STABILITY

9

and v are, respectively, also a viscosity super and subsolution of (4.5), we can exchange the roles of u and v in the previous proof to obtain the inequality u >__v. This completes the proof of (4.4). Let us observe explicitly that the preceding proof shows that the following comparison principle holds: (4.11)

if ul, us are, respectively, viscosity sub- and supersolution of (4.5), then Ul _< us 9

Essentially the same proof works when one adopts the test functions definition. The test functions one uses are x ~ - ~ v ( y ~ ) + Ix y~l_____~ ~ .- ~l(X)

2~ and Ix~ - yl s : = ~ s ( y ) ;

y,

~ U(Xs)

2c

note that

Dczl(x~) - x~ - y~ = D~2(yE) . The second issue that we touch upon here is that of stability of viscosity solutions. In order to illustrate this point, consider viscosity solutions v,~ of

(HJB)~

,Xvn(x) + H n ( x , D v , , ( x ) ) = 0

in ][~N

with

H n ( x , p ) := sup{ - f n ( x , a) . p - gn(X, a) } . aEA

Assume we know that, as n --~ +co, (4.12)

vn ---* u,

H~---*H

with

H ( x , p ) = sup{ - f ( x , a) . p - s

a) } .

aEA

The natural question is: does u satisfy (HJB) in the viscosity sense? The answer is yes, provided the convergence in (4.12) is uniform on compact sets. Of course, if uniqueness holds for the limit equation (HJB), this is a stability property of the value function v with respect to suitable perturbations of the data f,/?. The proof is particularly simple with the test function definition. Let x0 be a local maximum, which we can always assume to be strict, for u - ~, ~ C C 1. By uniform convergence, there exists a sequence {x~,} such that (4.13)

xn is a local maximum for vn - ~, xn --~ xo,

vn(xn) ~

u(xo) .

10

I. OUTLINE OF THE MAIN IDEAS

By definition of viscosity solution we have

Avn(xn) + Hn(x~,Dp(xn)) <_ 0; letting n -~ + o c in the above we obtain since D p is continuous

(4.14)

Au(x0) + H(xo, D~(xo)) <_ 0

that is, u is a viscosity subsolution of (HJB). The proof that u is also a supersolution is completely similar. Let us point out that the so-called generalized solutions of (HJB), i.e., locally Lipschitz continuous functions satisfying the equation almost everywhere, do not share these uniqueness and stability properties (see w In some situations it is not easy to use the stability property as previously formulated since equicontinuity estimates ensuring the uniform convergence of v~ may be hard to get. In fact, a stronger stability property holds. Consider the following weak limits in the viscosity

sense: ~(x) := lim inf{v,~(y) : I x - Yl <- 6, n > 1/6} 5---~0+

~(x) := lim s u p { v n ( y ) : I x - Yl <- 6, n > 1 / 6 } . 6--~0+

It is not hard to prove that if x0 is a strict local maximum for ~ - ~, then there exists {xn} satisfying (4.13). Hence, the argument above shows t h a t ~ satisfies the subsolution inequality (4.14). Note, however, that ~ is, in general, just upper semicontinuous. Similarly, the lower semicontinuous function _u_satisfies the supersolution inequality. This leads naturally to extending the notion of subsolution to upper semicontinuous functions and t h a t of supersolution to lower semicontinuous functions. A remarkable fact is that the comparison principle (4.11) still holds for these extended definitions. Therefore, ~ < u and since the reverse inequality is obvious, we obtain u -- ~. This implies the continuity of the weak limits u, ~, their coincidence with the value function v and the local uniform convergence of vn to v. This topic is discussed in Chapter V, and several applications are given in Chapters VI and VII. 5. S y n t h e s i s

of optimal

controls and verification theorems

Let us recall the classical synthesis procedure when the value function v is smooth. In the derivation of the Dynamic Programming Principle we observed t h a t the function

h(t) := v(y *(t))e - ~ + f 0 t e ( y * ( s ) , ~ * ( s ) ) e -~s ds

5. S Y N T H E S I S A N D V E R I F I C A T I O N T H E O R E M S

11

is constant for all t > 0 if and only if a*, y* is an optimal control-trajectory pair for the initial position x; see (2.2). Therefore, if v is smooth, the optimality condition is h t -- 0, t h a t is

e-~t[Av(y*(t)) - f(y*(t), a*(t)) . Dv(y*(t)) - e(y*(t), a*(t))] -= 0 . Since in this case v is a classical solution of the HJB equation

Av + H ( x , D r ) = 0

in ~ Y

where

H ( x , p ) := sup{ - f ( x , a) .p - e(x, a) }, aEA

we conclude t h a t the control a* is optimal for the initial state x if and only if

a*(t) = S(y*(t))

(5.1)

for a.e. t > 0,

for any choice of S(z) such t h a t (5.2)

S(z) e arg max{ - f ( z , a). D v ( z ) - g(z, a) }, aEA

t h a t is, if and only if

Y ( y *(t), D v ( y *(t))) = - f(y*(t), a *(t)) . Ov(y*(t)) - e(y*(t), a*(t)), for a.e. t > 0. This characterization of optimal open loop controls provides a method for constructing an optimal pair control-trajectory for every initial condition. The first step is to find a m a p S : ]1~N ~ A with the property (5.2); if v is known this is a static, finite dimensional, m a t h e m a t i c a l programming problem. Such a map S is called an optimal feedback map. The second step is solving

r (5.3)

y(0) =

t >

0,

,

and a solution y*(t) generates a control a*(t) : = S(y*(t)), which is optimal for the initial state x. The applicability of this method requires the regularity of the value function for the characterization of optimal controls and some regularity of the feedback m a p S for the solvability of (5.3). A typical example when this favorable situation occurs is linear systems with quadratic costs. In Chapter III we present several necessary and sufficient conditions of optimality in the spirit of this method holding for nondifferentiable value function. We consider, for instance, the case when the value function v is semiconcave; t h a t is, for some C > 0 (5.4)

v ( z - h) - 2v(z) + v(x + h) < C rhl 2

12

I. O U T L I N E O F T H E M A I N I D E A S

for all x and h. A simple situation where this is true occurs for f and t~ smooth and interest rate A large enough. In this case a necessary and sufficient condition of optimality is that (5.1) holds for a.e. t > 0, where S satisfies (5.5)

S(z) E a r g m a x { - f ( z , a ) . p - g ( z , a ) }

forallpED+v(z) UD-v(z)

aEA

instead of (5.2). We remark that a key property of semiconcave functions in this respect is that D+v(z) ~ 0 for all z. For merely Lipsehitz value functions even D+v(z) U D - v ( z ) may be empty at some point, so that condition (5.5) does not make sense. However, it can be replaced by a condition involving generalized directional derivatives in the sense of Dini (see w If the value function is merely continuous, (5.1) and (5.5) are still necessary conditions of optimality and the problem of finding sufficient conditions in this form is open to our knowledge. A slightly different approach to sufficient conditions of optimality for merely continuous value functions is a nonsmooth version of the classical Hamilton-JacobiCarathdodory verification technique, also named "the Basic Algorithm" by L.C. Young [You69]. This is based on the notion of verification function. Classically, such a function u is assumed to be bounded and C 1 and satisfy (5.6)

Au(x)+sup{-f(x,a).Du(x)-g(x,a)}


VxE]~ N .

aEA

Let a* be an admissible control with corresponding trajectory y* starting at x. A simple verification result is that a* is optimal if there exists a verification function u such that (5.7)

)~u(y*(t)) - f(y*(t), a*(t)), nu(y*(t), a*(t)) - g(y*(t), a*(t)) = 0

for t E [0,+oc[. In fact, for a r b i t r a r y a E A and corresponding trajectory y(t) starting at x, we have actually

Au(y(t) ) - f(y(t), a(t) ) . Du(y(t) ) <_g(y(t): a(t) ) by the very definition of verification function. member of the above inequality is precisely

Observe now that the left-hand

- e ~ ~t(e-~u(y(tl)). Hence, integration of both sides of the inequality on [0, +oz[ yields

u(z) <_

fo

e(y(t),a(t))e-~t dt = J(x,a),

For the particular control a* the same argument gives

u(x) = J(x, ~*),

V(~ E A .

6. PONTRYAGIN MAXIMUMPRINCIPLE

13

so that a* is optimal. Conversely, it is immediate to check that if the control law x ~ ax E ,4 is optimal for all x and u(x) := J(X, ax) is differentiable, then u is a verification function. Indeed, by its very definition, u coincides with the value function v, which in this case is a classical solution of (HJB) and, a fortiori, of (5.6). Viscosity subsolutions provide a quite natural tool to extend the verification technique to a more general setting. Let us stipulate that u E C(• N) is a generalized verification function if inequality (5.6) is satisfied in the viscosity sense. The verification result in this framework is as follows: a control a* E .4 is optimal for the initial state x if and only if there exists a generalized verification function u such that u(x) > J(x, a*). The proof is very simple: the comparison principle yields u(x) < v(x), so t h a t

J(x,a*) ~ u(x) < v(x) ~ J(x,a*). To prove the reverse implication, just observe that the optimality of a* implies v(x) = J(x, c~*) and recall t h a t v itself is a viscosity subsolution of (HJB). A computationally oriented approach to the synthesis problem is reviewed in w 6. P o n t r y a g i n M a x i m u m Principle as a necessary and sufficient condition of optimality The Pontryagin Maximum Principle is the most classical and useful necessary condition of optimality. It is well-known that if the value function is smooth, there is a formula linking its gradient with the costate vector. Next we show how to interpret this relation in terms of generalized gradients when the value is not differentiable. A relevant by-product is a formulation of the Principle as a necessary and sufficient condition. To illustrate these ideas we choose the simplest problem, that is, the finite horizon Mayer problem whose cost functional is

(6.1)

d(x, t, ~) := g(yz(t, ~)),

g e c ~(~N),

where t is the fixed horizon. Later in this section we outline the extension of the result to the infinite horizon problem. The value function is now

v(x,t):= inf J ( x , t , a ) . aEA We fix x0 E ]~N, T > 0 and c~ E A, and investigate the optimality of the corresponding t r a j e c t o r y

y(t) := y~o(t,a),

0 < t < T.

Assume that the dynamics f is differentiable with respect to the state variable and D x f is bounded. The costate (or adjoint vector) associated with x0, T, a is the

14

I. O U T L I N E O F T H E M A I N I D E A S

unique solution of the adjoint system (6.2)

SP'(t ) = - p ( t ) D x f ( y ( t ) , [p(T) Dg(y(T)) .

a(t))

0 < t < T,

The classical Pontryagin Maximum Principle (shortly, PMP) states that the following condition is necessary for optimality:

(6.3)

-p(t) 9S(y(t), ~(t)) = m ~ { -p(t) 9

a) }.

Moreover, if v is smooth, the following relations between the costate and the value function are well known (see also w (6.4) (6.5)

p(t) = Dxv(y(t), T - t), ~Ov( y ( t ), T - t ) .

p(t). f(y(t),a(t))=

The Hamilton-Jacobi-Bellman equation for this problem is Ov O--t + H ( x , D~v) = 0

in ]~N

X

]0.-]-O(~[,

where H ( x , q ) := s u p { - q , f ( x , a ) } . aEA

With this notation (6.5) can be written as (6.6)

Ov - ~ ( y ( t ) , T - t) = - H ( y ( t ) , p ( t ) )

.

In the case where v is not differentiable we consider the following nonsmooth counterpart of (6.4) and (6.6): (6.7)

(p(t), - H ( y ( t ) , p ( t ) ) )

E D+v(y(t), T - t ) .

Here, D + denotes the superdifferential of v with respect to the N + 1 dimensional vector (x, t). Now we can formulate the P M P as follows: the control (~ is optimal if and only if (6.3) and (6.7) are verified for a.e. t E ]0, T[. To prove the necessity it is enough to show that, for a.e. t, (6.8)

(p(t),p(t). f ( y ( t ) , ~(t))) E D+v(y(t), T - t ) .

In fact, v being a subsolution of the HJB equation, this implies p(t) . f ( y ( t ) , a(t)) + sup{ - p ( t ) . f ( y ( t ) , a) } <_ 0, aEA

6. PONTRYAGIN MAXIMUM PRINCIPLE

15

which immediately gives (6.3) and (6.7). Next we sketch the proof of (6.8) (see Proof 3 of Theorem III.3.42 for more details). First note that, if a is optimal, the Dynamic Programming Principle yields v(y(t),T

- t) = J ( y ( t ) ; T

- t,a(t)),

for all t C [0, T], where

~(~)(s) := , ( t + s). Therefore, (6.9)

v ( x , s) - v ( y ( t ) , T - t) <_ J ( x § h ( x , s); T - t, (~(t)) _ J ( y ( t ) , T - t, ~(t))

where (6.10)

h ( x , s) --

--S

[ f ( y ( 7 ) , c~(T)) + o(1)] dr

as s --* T - t, x -~ y(t). By the differentiability of g and of solutions of ordinary differential equations with respect to the initial data, with some computations one can see that J is differentiable with respect to the state variable with D x J (y( t ), T - t, a (t)) = p( t ) .

Hence, by (6.9) and (6.10), v ( x , s ) - v ( y ( t ) , T - t) <_ p ( t ) . (5 + h ( x , s) - y ( t ) ) + o(Jx - y ( t ) l + fs - T + tl) .

Now, the validity of (6.8) for a.e. t follows easily from (6.10) and Lebesgue derivation theorem. For the sufficiency, we set po(t) := - H ( y ( t ) ,

p(t))

p(t). y'(t) -po(t)

= 0

and note that, by (6.3),

for all t where y~ exists. Then, by (6.7), S-I[v(y(~:

§ ~),T

-

t -

c) -

y(y(t),T

-

(6.11)

t)] _~ p . y'(t) - P 0 § o(1)

= o(1)

for ~ ~ 0 .

Now we recall that h(t) := v ( y ( t ) , T - t)

is nonincreasing if and only if the trajectory is optimal. Since v is locally Lipschitz because the data f and g are differentiable, h is locally Lipschitz as well, therefore

16

I. OUTLINE OF THE MAIN IDEAS

differentiable almost everywhere. Thus for almost every t (6.11) gives h'(t) <_ O, which completes the proof. It is useful to point out that the previous formulation of the P M P remains valid even if g is not differentiable, provided D+g(y(t)) is nonempty. Indeed, it is easy to check that everything works if the terminal condition in the definition of the costate (6.2) is replaced by

p(T) E D+ g(y(T) ) . Next we show how to formulate the P M P as a necessary and sufficient condition of optimality for the model problem of this chapter, that is, the infinite horizon problem. To this end we introduce (6.12)

V(x. , t) := inf / f T t(y(s), a(s)) e -as ds + e - ~ T v ( y ( T ) ) } , aEA ~ Jt

where v is the value function of the infinite horizon problem defined in w and y is the trajectory of the system such that y(t) = x. Note that

v(x, o) = v(x) by the Dynamic Programming Principle (2.3), and it is not hard to show that (6.13)

V(x, t) = e-~tv(x)

for all t E [0, T ] .

Observe also that, by its very definition, V is the value function of a finite horizon Bolza problem where v represents the terminal cost. By standard tricks based on the introduction of new state variables and by reversing time this problem can be reformulated as a Mayer problem such as the one previously discussed in this section. Then we can apply the PMP, provided ~ is C 1 with respect to the state variables, and D+v(z) is nonempty for all z E/R N. This is the case, for example, if v is semiconcave; that is, it satisfies (5.4). For problems with running cost, the costate equation is (6.14)

p'(t) = - p ( t ) D x f ( y ( t ) , a(t)) -

e-atDxg(y(t), a(t))

.

For the Bolza problem in (6.12) one defines the costate pT(t), t E [0, T], as a solution of (6.14) with the terminal condition (6.15)

pT(T) E e-aT D+v(y(T) ) .

For this problem, the control a is optimal if and only if

(6.16)

--pT(t)" f(y(t), a(t)) -- e-~tf(y(t), a(t)) = ~I(t, y(t),pT(t))

where ~r(t, z, q) := sup{ - q . f(z, a) -- e-'Xte(z, a) },

aEA

7. CONVERGENCE OF APPROXIMATIONS

17

which replaces (6.3), and

(pT(t), ~I(t, y(t),pT(t))) E D+Y(y(t), t), which replaces (6.7). Observe that, by (6.13), this inclusion can be split as (6.17)

pT(t) e e-~tD+v(y(t))

and (6.18)

9(t, ~(t), pT(t))

=

-~e-~,(y(t)).

Suppose now t h a t a is an optimal control for x in the infinite horizon problem. Then by the Dynamic Programming Principle in the form (2.2), it is also optimal for the finite horizon T > 0. The necessary condition of optimality can therefore be obtained by letting T --~ +~c in (6.16), (6.17), (6.18) and it is the following: there exists a solution p(t); t E [0, +~c[ of the costate equation (6.14) such that for a.e. t :> 0

-p(t). f(y(t), ~(t)) - e-~tg(y(t), c~(t)) = ~I(t, y(t),p(t)), p(t) E e-~tD+v(y(t)), 9 ( t , y(t), p(t) ) = -:~e-~tv(y(t) ) . This result holds as soon as one knows that pT(t) has a limit p(t) as T --+ + o c which satisfies the costate equation. This is true, for instance, if v is Lipschitz continuous and, for all t, A > R e # ( t ) for all eigenvalues #(t) of the symmetric part of the matrix Dxf(y(t), a(t)). The assumption on v is used to get pT(T) ---*0 from (6.15), and t h a t on A allows us to pass to the limit in the variation of constants formula for pT(t). The proof of the sufficiency of these optimality conditions is essentially the same as that for Mayer's problem. 7. D i s c r e t e t i m e D y n a m i c P r o g r a m m i n g and convergence of approximations In w we discussed the synthesis algorithm in the classical as well as in some "semiclassical" setting (i.e., for semiconcave or Lipschitz continuous value functions). As pointed out, sufficient conditions in the form described there fail for merely continuous value functions. A natural way to overcome the difficulties related to the lack of regularity is to look at approximate value functions Vh satisfying pointwise a finite increment approximate version of the HJB equation. This can be done in several ways. The approach outlined below, which relies on discrete time Dynamic Programming, produces closed loop controls which are suboptimal for the original problem by virtue of a convergence result of Vh to v. Let us also point out that the convergence of vh to v, supplemented by quantitative error estimates, is the basis for computational methods of solution of the

18

I. O U T L I N E OF T H E MAIN IDEAS

HJB equation. We refer to Chapter VI and Appendixes A and B for developments of the themes touched here. In order to illustrate the approach sketched above, assume that the evolution described by the control system (1.1) is observed only at a sequence of times tj = jh (j = 0, 1 , . . . ) , where h is a fixed positive real number. Assume as well that the dynamics f and the running cost e remain unchanged between two subsequent observations. More precisely, suppose that, for t E [tj, tj+l[,

f(y(t), a(t)) = f(yj, aj),

e(y(t), a(t)) = e(yj, aj),

where aj = a(tj) and yj = y(tj) is given by the recursion (7.1)

Yo =

x,

Y j + I = Yj + h f ( y j ,

aj),

j = O, 1 . . . .

The infinite horizon cost functional associated with initial position x and control a is chosen in this discretized model as the series +oc

(7.2)

h~-'3Je(yj,a3),

ajEAforj=0,1,...

j=0

with 13 = 1 - Ah. The corresponding value function is +vz

(7.3)

Vh(X) := inf h Z /3Je(yj,aj) , aEAh

j=0

where Ah is the subset of A consisting of those controls that take on a constant value on each time interval, namely, a(t) = aj for t E [tj, tj+l [. It is not hard to check that Vh satisfies a discrete time version of the Dynamic Programming Principle, namely k-1

(7.4)

Vh(X) = inf ~h ~ t3Je(yj, aj) +/3kvh(yk) } aE'Ah ~" j~=O

for all x E R N and integer k. For the choice k = 1 the above reads as (HJB)h

vh(x) + sup{ --~Vh(X + hf(x, a)) - he(z, a)

} = o.

aEA

This equation for Vh can be thought as a discrete version of (HJB). Let us examine now the limiting behavior of Vh as h --* 0 +. Assume we know that Vh converges locally uniformly in ]RN as h --, 0 + to some continuous function u. As shown below, it turns out that u is a viscosity solution of (HJB). Hence u coincides with v, the value function of the infinite horizon problem, provided uniqueness holds for (HJB) (see w The fact that u = limh~0+ Vh is a viscosity subsolution of (HJB) is easily verified. For this purpose, let ~ E C I ( R N) and x be a strict local maximum point for u - 4- By uniform convergence, there exist

7. C O N V E R G E N C E

OF APPROXIMATIONS

19

Xh ~ x as h ~ 0 + such that Xh is a local maximum for Vh -- 9~. Therefore, for small h,

Vh(Xh)--Vh(Xh+hf(xh,a))>_c2(xh)--~(xh+hf(xh,a))

,

VaEA.

From (HJB)h we then obtain

0 > c2(Xh ) -- qO(Xh + h f ( x h , a)) + .~hvh(Xh -4- h f ( x h , a)) -- ht(xh, a) for all a 9 A. Since c2 9 C 1 this yields, after dividing by h > 0,

0 :> --Dcp(~h)" f ( x h , a) -t- /~Yh(Xh -Jr-h f ( x h , a)) -- g(Xh, a), for s o m e ~h ~ 0 as h ~ 0 + By uniform convergence we can pass to the limit as h ~ 0 + in the preceding and take the supremum with respect to a to get 0 > sup{ - D ~ ( x ) .

f ( x , a) + .ku(x) - t~(x, a) } .

aEA

Hence, u is a viscosity subsolution of (HJB); the proof that u is a supersolution as well is similar. Let us observe that the previous argument shows that the weak notion of viscosity solution allows us to pass to the limit as h --~ 0 + in the nonlinear equations (HJB)h under uniform convergence and just continuity assumptions on f and g. Note also that the same result could have been obtained by the weak limits technique (see w and Chapter VI). Let us come back now to the synthesis problem. It is quite natural in the present setting to consider the mapping

(7.5)

Sh(z) 9 arg max{ --/3Vh(Z + h f ( z , a)) - hi(z, a) } . aEA

This is easily seen to be an optimal feedback map for the discrete time problem (7.3), that is

(7.6)

Vh(X) = h ~ Z

j

* * t(y~, Sh%))

Vx 9 S~,

j=O with

(7.7)

y~ = x,

Y~+I = Y~ + hf(y~, Sh(y~)).

Of course, no regularity of Vh or Sh is required for defining the feedback map Sh and the optimal discrete trajectory {y~}. By the foregoing, the optimal cost v(x) for the original continuous time problem can be approximately computed by (7.5), (7.6), (7.7). We refer to Chapter VI for more complete convergence results.

20

I. OUTLINE OF THE MAIN IDEAS

8. T h e v i s c o s i t y a p p r o x i m a t i o n

and stochastic control

A different insight to viscosity solutions is provided by the so-called viscosity approximation of equation (HJB), namely, (HJB) ~

- s A u ~ + hu t + sup{ - f ( x , a). Du e - ~(x, a) } = 0,

x E R N.

aEA

Here, c is a positive real parameter and N

02

i=1

is the Laplace operator. Equation (HJB) ~ can be interpreted in terms of stochastic optimal control theory. A rigorous explanation of this assertion is outside the scope of this book. Therefore we just outline it and refer to [FR75, FS93] for details. Consider the function (8.1)

v~(x) := inf Ex

f0

e(y~(t), ~(t)) e -~t at.

In the above formula, yE is the solution of the stochastic differential equation

(8.2)

dy = f(y(t), c~(t)) dt + ~-~dw(t), y(O) = x,

where w is a N-dimensional standard Brownian motion, Ex denotes expectation and the infimum is taken on the class of progressively measurable functions a with values in A. Function vE appears then as the value function of a stochastic version of the infinite horizon problem where one tries to control in an optimal way the trajectories of (8.2), a random perturbation of system (1.1). Under suitable conditions on the data, v ~ happens to be a smooth, say C 2, solution of (HJB)~; this follows from the Dynamic Programming Principle and Ito's stochastic calculus. It is therefore natural to guess that when the randomness parameter ~ is sent to zero, the value functions v~ should converge to the value function v, the viscosity solution of (HJB). Let us next sketch a PDE proof of this guess. Observe that passing to the limit in (HJB) E as c ---* 0 + in order to recover equation (HJB) is not an easy task. This is due to the nonlinearity of the equation and the fact that the required estimates on u 6, which can be assumed to be smooth, explode as c --* 0 + since the regularizing effect of the additive term --cA becomes weaker and weaker. On the other hand, one does not expect (HJB) to have smooth solutions. Assume therefore that u ~ E C2(~ N) converge locally uniformly as e ~ 0 + to some u E C(]~N). Now let ~ be a C 2 function and x be a strict maximum point for u - ~. By uniform convergence, u ~ - T attains a local maximum at some point x ~ and x E ---* x as c --~ 0+. Hence, by elementary calculus, D(~ ~ - ~;)(~)

= o,

-/',(~

-

~;)(~) _> O .

9. BIBLIOGRAPHICALNOTES

21

By (HJB) ~, then

-r

E) + )~uE(x ~) - f ( x E, a) . D ~ ( x ~) - t ( S , a) <_ 0

for all a E A. Since x ~ ~ x as e ~ 0 +, we can easily pass to the limit in the preceding using the continuity of A~, D~, f, g and the uniform convergence of u E to u. The conclusion is that

;~u(x) + sup{ - f ( x , a) . D r ( x ) - g(x, a) } < O, aEA therefore u is a viscosity subsolution of (HJB). In a similar way one can show that u is also a supersolution in the viscosity sense. The limiting procedure described above to deal with the singular perturbation problem (HJB) E --~ (HJB) can be thought as a way to define weak solutions of (HJB). This is actually the motivation for the terminology viscosity solutions in the original paper of M.G. Crandall and P.L. Lions [CL81]. 9. B i b l i o g r a p h i c a l

notes

The Dynamic Programming method in optimal control theory was developed in the '50s by Richard Bellman and his school [Be157, Ho60, BelD62, Be171], mainly for discrete time dynamics; for systems governed by differential equations it appears in a paper of Kalman [Ka163] and in Isaacs' monograph on differential games [Is65]. However, the central role played by value functions and Hamilton-Jacobi equations in the classical setting of the Calculus of Variations was recognized as early as in 1935 by C. Carath~odory [Ca35], see [PB94] for a recent survey. In the '60s, Dynamic Programming became a standard topic in deterministic optimal control theory; see, for example, the books [Hes66, LM67, You69] and the more recent [FR75, Ber76]. However, the lack of smoothness of the value function, even for simple optimal control problems of continuous time processes, caused a considerable restriction to the range of applicability of the HamiltonJaeobi theory. The main reason for this limitation was the very basic difficulty to give an appropriate sense to the Hamilton-Jacobi equation, a fully nonlinear first order partial differential equation, satisfied at points of differentiability by the value function, and to characterize the value as the unique solution of that equation. Several non-classical notions of solution of Hamilton-Jacobi type equations have been therefore proposed in the literature. Krugkov built a rather complete theory for semiconvex functions solving the equation almost everywhere in the ease of convex (and sufficiently smooth) Hamiltonians [Kr60, Kr67, Kr75] (see also [Dou61]), including the vanishing viscosity approximation [Kr64, Kr66a], and the convergence of finite difference schemes [Kr66b]. More recent references are [Bt77, Hr78, Cla83]; see also [Go76] and, for second order equations, [Me80a, Me80b, BL82]. All these papers dealt with equations convex in the derivatives of the unknown function, whereas Subbotin [Su80, Su84] gave some results for the nonconvex Hamiltonians arising in differential games. His ideas evolved at the end of the '80s into the theory of minimax solutions [Su91b, Su95] which has several deep connections

22

I. OUTLINEOF THE MAINIDEAS

with viscosity solutions (at least for first order equations); see w of Chapter III, as well as Chapter VIII for some more remarks in connection with the theory of differential games. A new impulse to the rigorous mathematical justification of Dynamic Programming was originated by the introduction in the early '80s of the notion of viscosity solution by hI.G. Crandall and P.L. Lions [CL81, CL83] for general Hamiltonians. Their original definition is related to Kru~kov's theory of entropy solutions for scalar conservation laws [Kr70] (see [Cas92] for a direct proof of the connection between the two theories and [CDP96] for related results). In [CL83] some equivalent ways to define viscosity solutions were indicated; the one based on test functions is intimately connected with earlier work of Evans on weak passages to the limit in fully nonlinear PDEs [E78, ES0]; see also [Eg0]. This property was already noted in [Kr75] in the case of convex Hamiltonians. Using this equivalent notion of solution Crandall, Evans and Lions gave rather simple proofs of the good comparison and stability properties of viscosity solutions [CEL84]. The theory was extended to second order degenerate elliptic HJB equations by P.L. Lions [L83a, L83b] with a proof of uniqueness that relied on stochastic control arguments. A major breakthrough was achieved by R. Jensen [J88] who proved a comparison principle for nonconvex degenerate elliptic equations by purely PDE methods; see also [J89]. A comprehensive account of the theory of viscosity solutions at this level of generality can be found in [CIL92, C95], where the reader can find more information and references on its history and developments. For some more recent advances of the theory of second order equations and several applications to various fields we refer to the lecture notes [CDL97], to [AGLM93, ESp95, CafCa95, CCKS96, CKSS96], and the references therein. Starting from P.L. Lions' remark in the book [L82] that value functions do in fact satisfy Hamilton-Jacobi equations in the viscosity sense, a considerable amount of work has been subsequently devoted to applications of viscosity solutions to optimal control and differential games. Detailed references are given at the end of each chapter of the present book (see also the bibliographies of [L92, Ba94, FS93]). Let us just mention here that applications to switching and impulse control problems were considered, respectively, by Capuzzo Dolcetta and Evans [CDE84] and Barles [Ba85a], differential games were treated first by Evans and Souganidis [ES84] and Barron, Evans and Jensen [BEJ84], that boundary value problems arising in time-optimal control were studied by Bardi [B89] and Evans and James [EJ89], and those arising in state-constrained problems by Soner [$86] and Capuzzo Dolcetta and Lions [CDL90]. Relevant extensions of the theory, motivated by exit time, constrained and differential games problems, to include discontinuous solutions, were proposed by Barles and Perthame [BaP87], Ishii [I87a, I89], Barron and Jensen [BJ90], Bardi and Soravia [BS91a], Subbotin [Su93a]. The book of Elliott [E187] is devoted to some applications of viscosity solutions to deterministic optimal control, whereas Barles' book [Ba94] gives a wider and more recent account.

9. B I B L I O G R A P H I C A L NOTES

23

The problem of the synthesis of optimal feedbacks received important contributions from Boltyanskii [Bo166], Brunovsky [Bru80] and Sussman (see [FR75, Sus89a, Sus90, Mi90a, PSus96] for some recent presentations) but it remains largely open. The approach we present here develops some results of Berkovitz [Be89] and Frankowska [Fr89b], whereas the use of viscosity subsolutions as verification functions gives verification theorems as a very easy consequence of the comparison principle. For the connections of viscosity solutions with the Pontryagin Maximum Principle we refer to the papers [BJ86, Sua89, Z90, CF91], while the result for the infinite horizon problem presented in w is adapted from [Te.95]. The relevance of comparison and stability properties of viscosity solutions in connection with the convergence of approximations and computational schemes for value functions and optimal feedbacks has been recognized since the early stages of the theory by Capuzzo Dolcetta [CD83] and Crandall and Lions [CL84], see also [CDI84, Sou85a, Fa87] and Appendix A of the present book. Among the more recent applications of viscosity solutions in optimal control theory we mention those concerning risk-sensitive and H ~ control (see [BJ89, FM92a, Jam92, Sor96a]) and Appendix B of this book. The generalization of the theory of viscosity solutions to optimal control problems for diffusion processes, initiated by P.L. Lions [L83a, L83b, L83c], goes beyond the scope of the present work. We refer to the recent book by Fleming and Soner [FS93] for a comprehensive presentation of this important topic, to [$95] for its applications to mathematical finance and to [FSo89] for stochastic differential games. Another issue that is not touched in this book is the viscosity solutions approach to optimal control problems for distributed parameter systems and the corresponding theory of infinite dimensional Hamilton-Jacobi equations; see, for example, [CL85, CL94a, CL94b, L88, L89, CDaP89, CDaP90, CGS93, CT96, I92a, T94, LOS93, KSw95, KSS97], the books [BDaP83, BDDM93, Bar93a, LiY95] and the references therein.

CHAPTERII

Continuous viscosity solutions of Hamilton-Jacobi equations

This chapter is devoted to the basic theory of continuous viscosity solutions of the Hamilton-Jacobi equation (H J)

F(x, u(x), Du(x)) = 0

x E ~,

where 12 is an open domain of ]RN and the Hamiltonian F = F(x, r, p) is a continuous real valued function on 12 • R • R N. Special attention will be dedicated in w to the case where p ~-~ F(x,r,p) is convex and, more particularly, of the form (0.1)

F(x,r,p)=r+H(x,p)=r+sup{-f(x.a).p-e(x,a)}. aEA

Hamiltonian functions of this form arise naturally in connection with optimal control problems as indicated in Chapter I. 1. D e f i n i t i o n s

and basic properties

In this section we recall the two equivalent definitions of viscosity solutions of (H J) introduced in Chapter I and discuss their relations with a comparison principle as well as some connections with classical notions of solutions of (H J). DEFINITION 1.1. A function u E C(ft) is a viscosity subsolution of (H J) if, for any

~ c1(~), (1.1)

F(xo, u(xo), D~(xo)) <_0

at any local maximum point x0 E f l of u - ~. Similarly, u E C(f~) is a viscosity supersolution of (H J) if, for any ~a E C 1(~), (1.2)

F ( x l , u ( x l ) , D ~ ( x l ) ) ~_ 0

at any local minimum point xl E ~ of u - ~. Finally, u is a viscosity solution of (H J) if it is simultaneously a viscosity sub- and supersolution. <~

26

II. C O N T I N U O U S V I S C O S I T Y S O L U T I O N S O F H-J E Q U A T I O N S

Let us mention explicitly t h a t the definition applies to evolutionary HamiltonJacobi equation of the form ut(t,y) +F(t,y,u(t,y),Duu(t,y))

= O,

( t , y ) E ]0, T[ • D .

Indeed, the equation above is reduced to the form (H J) by the positions x = (t, y) E f t = ]0, T[ • D _C R N+I,

F(X, r, q) = qN+l + F ( x , r, ql . . . . , qN)

with q

=

(ql ....

, qN, q N + l )

E ]1~.N + I

.

REMARK 1.2. In the definition of subsolution we can always assume that x0 is a local strict maximum point for u - p (otherwise, replace ~(x) by T(x) + Ix - x012). Moreover, since (1.1) depends only on the value of D ~ at x0, it is not restrictive to assume that u(xo) = ~(x0). Similar remarks apply of course to the definition of supersolution. Geometrically, this means that the validity of the subsolution condition (1.1) for u is tested on smooth functions "touching from above" the graph of u at x0. We note also that the space C1(9/) of test functions in Definition 1.1 can be replaced by C~c(~), see Exercise 2.1. <~ The following proposition explains the local character of the notion of viscosity solution and its consistency with the classical pointwise definition. PROPOSITION 1.3. (a) I f u E C(f~) is a viscosity solution of (H J) in f~, then u is a viscosity solution of (H J) in fY, for any open set f~ C f~; (b) if u E C(9/) is a classical solution of (H J), that is, u is differentiable at any x E f~ and (1.3)

F(x,u(x),Du(x))

= 0

V x E f~,

then u is a viscosity solution of (HJ); (c) if u E Cl(f~) is a viscosity solution of (HJ), then u is a classical solution of (H J). PROOF. (a) If x0 is a local maximum (on f~) for u - ~, r E Cl(f~'), then x0 is a local maximum (on f~) for u - ~, for any ~ c c l ( f ~ ) such that c~ =- ~ on B ( x o , r ) for some r > 0. By (1.1) 0 >_ F(xo, u(xo), D ~ ( x o ) ) = F ( x o , u(xo), D p ( x o ) ) showing that u is a viscosity subsolution of (H J) on ~t~. The same argument applies to prove that u is also a supersolution on fl~. (b) Take any ~ E C l ( f l ) . By the differentiability of u, at any local maximum or minimum x E ft of u -- c2 we have D u ( x ) = D ~ ( x ) . Hence (1.3) yields 0 = F(x0, u(z0), D~(x0)) _< 0

I, DEFINITIONSAND BASIC PROPERTIES

27

if x0 is a local maximum for u - ~ and

0 = F(xl,U(Xl), D ~ ( x , ) ) >_ 0 if Xl is a local minimum for (c) If u E C I ( f t ) , then ~ solution. W i t h this choice, minimum for u - ~. Hence,

u - ~. u is a feasible choice in the definition of viscosity any x E f t is simultaneously a local maximum and by (1.1) and (1.2),

F(x,u(x),Du(x)) = 0

Y x E f~.

4

Statement (a) says that the notion of viscosity solution is a local one. Consequently, one can take the test functions in (1.1) and (1.2) in CI(]R N) or in any sufficiently small ball B(x, r) centered at x E f~. The definition of viscosity solution is closely related to two properties t h a t are typical in the theory of elliptic and parabolic equations, namely the maximum principle (MP) and the comparison principle (CP). For equation (H J) these properties can be respectively formulated as follows. DEFINITION 1.4. A function u E C(ft) satisfies the comparison principle with smooth strict supersolutions, briefly (CP), if for any ~ E C1(~) and O open subset of D,

F(x,~y(x),D~(x))>O

inO,

u<~

on00

implies u < ~ in O. We say t h a t u E C(f~) satisfies the maximum principle (MP) if for any c2 E Cl(f~) and O open subset of f~ the inequality

F(x, ~(x), D~(x)) > 0

in O,

implies that u - ~ cannot have a nonnegative maximum in O.

<1

It is quite clear t h a t (MP) implies (CP). The connections with the notion of viscosity subsolution of (H J) are expressed by the next result. PROPOSITION 1.5. If u E C(fl) satisfies (CP), then u is a viscosity subsolution of (HJ). Conversely, if u is a viscosity subsolution of (HJ) and r ~ F ( x , r , p ) is nondecreasing for all x,p, then u satisfies (MP) and (CP). PROOF. Assume that u E C(ft) satisfies (CP). If, by contradiction, u is not a subsolution of (HJ) there exist x0 E ft, ~ E Cl(12), such that x0 is a strict maximum point for u - ~, (u - ~)(xo) = O, and

F(xo, u(xo), D~(xo)) > o. For n large enough we have an:=

sup (u-~)<0. OB(xo,1/n)

28

II. CONTINUOUSVISCOSITY SOLUTIONS OF H-J EQUATIONS

Observe also that u - (~ + an) _< 0 u(x0) - ~(x0)

on OB(xo, 1/n),

- a~ > 0.

By (CP) for a n y n there exists x,~ E O,~ : = B(x0, 1/n) such that F ( x ~ , ~(x~) + a~, D~(x,~)) <_O. Since an --~ 0 and x,~ --+ x0 we o b t a i n the contradiction F(x0, u(x0), D~(xo)) < O. Conversely, let u be a viscosity subsolution of (H J) and take any ~ E CI(Q) such that

F(x, ~(x), D~(x)) > 0

for all x E O .

If u - ~ a t t a i n s a local m a x i m u m at some x0 E (9 with u(xo) - ~(xo) >_ O, t h e n the m o n o t o n i c i t y a s s u m p t i o n on F implies the contradiction

0 < F(xo, ~(x0), D~(xo)) <_F ( / 0 , u(xo), n~(xo)) <_O. Therefore, u satisfies (MP) and, afortiori, (CP).

4,

A similar result holds for viscosity supersolutions, provided all inequalities are reversed in (CP), (MP) a n d nonnegative m a x i m u m is replaced by nonpositive minimum. A perhaps striking fact to be stressed here is t h a t viscosity solutions are not preserved by change of sign in the equation. Indeed, since a n y local m a x i m u m of u - ~2 is a local m i n i m u m of - u - ( - p ) , u is a viscosity subsolution of (H J) if a n d only if v = - u is a viscosity supersolution of - F ( x , - v ( x ) , - D r ( x ) ) = 0 in 12; similarly, u is a viscosity supersolution of (H J) if and only v = - u is a viscosity subsolution of - F ( x , - v ( x ) , - D v ( x ) ) = 0. A n explicit example is as follows. EXAMPLE 1.6. The function u(x) = Ix I is a viscosity solution of the 1-dimensional equation -Iu'(x)] + 1 = 0

x E ]--1,1[ .

To check this, notice first that if x ~ 0 is a local e x t r e m u m for u - ~, t h e n ut(x) -= ~(x). Therefore, at those points b o t h the supersolution a n d the subsolution conditions are trivially satisfied. Also, if 0 is a local m i n i m u m for u - ~, a simple calculation shows t h a t ly;'(0)l _ 1 and the supersolution condition holds. To conclude it is enough to observe t h a t 0 c a n n o t be a local m a x i m u m for u with ~2 E C 1 ( ] - 1 , 1[) (this would imply - 1 > ~'(0) _ 1). On the other hand, u(x) = Ix[ is not a viscosity solution of

l u ' ( x ) l - 1-- o,

xE]-l,l[

.

Actually, the supersolution condition is not fulfilled at x0 = 0 which is a local m i n i m u m for IxI - ( - x 2 ) . <~

1. DEFINITIONS AND BASIC PROPERTIES

We describe tion (H J) and previously (see with a function

29

now an alternative way of defining viscosity solutions of equaprove the equivalence of the new definition with the one given Exercise 1.11 for another equivalent definition). Let us associate u E C(f~) and x E f~ the sets

D+u(x) : =

{ p E R N : l i m s u p ~ ( y ) - u ( z ) - p . (y - x) < 0 } tx - Yl y ~ x , yEl~

D - u ( x ) := { p E ]~N : l i m i n f ~ ( y ) - u ( x ) - p . (y - x) >_ 0 } u~x, ye~ [x -- Y[ These sets are called, respectively, the super- and the subdifferential (or semidifferentials) of u at x. The next lemma provides a description of D+u(x), D - u ( x ) in terms of test functions. LEMMA 1.7. Let u E C ( ~ ) . Then, (a) p E D+u(x) if and only if there exists ~ E C 1 ( ~ ) such that D p ( x ) = p and u - ~ has a local m a x i m u m at x; (b) p E D - u ( x ) if and only if there exists ~ E CI(f~) such that D ~ ( x ) = p and u - ~ has a local m i n i m u m at x. PROOF. Let p E D+u(x). Then, for some 5 > 0, u(y)~u(x)+p.(y-x)+a(]y-xI)[y-x

[

VyEB(x,5),

where a is a continuous increasing function on [0, +oc[ such t h a t a(0) = 0. Now define a C 1 function ~ by ~(r) =

~(t) dt.

The following properties of Q ~(o) = d(o)

= o,

~(2r) > ~(r)r

imply, as it is easy to check, that the function ~ defined by ~(y) = u(x) + p - ( y -

x) + ~ ( 2 1 y - x[)

belongs to C1(I~ N) and D ~ ( x ) = p. Moreover, for y E B ( x , 5), (u - ~)(y)

< ~ ( l y - x l ) l y - xl - ~(21y - x l ) < 0 = (u - ~ ) ( x ) .

For the opposite implication it is enough to observe that u(y) - u(x) - D ~ ( x ) . (y - x) <_ ~(y) - ~(x) - D ~ ( x ) . (y - x) for y E B ( x , 5) and the proof of (a) is complete. Since D - u ( x ) = - ( D + ( - u ) ( x ) ) , the proof of (b) follows from the above argument when applied to - u .

30

II. CONTINUOUSVISCOSITY SOLUTIONS OF H-J EQUATIONS Some properties of the sub- a n d superdifferential are collected in L e m m a 1.8.

LEMMA 1.8. Let u E C(f~) and x E f~. Then, (a)

D+u(x) and D - u ( x ) are closed convex (possibly empty) subsets of R Y ;

(b)

i f u is differentiabIe at x, then { O n ( x ) } = D+u(x) = D - u ( x ) ;

(c)

if for some x both D+u(x) and D - u ( x ) are nonempty, then D+u(x) = D - u ( x ) = {Du(x)};

(d)

the sets A + = {x C ft : D+u(x) r 0}, A - = {x E f~ : D - u ( x ) r O} are dense.

PROOF. The convexity of D+u(x) and D - u ( x ) is a straightforward consequence of the definition of lim sup a n d lim inf. To prove t h a t D+u(x) is closed we take a sequence Pn --* P such t h a t pn E D+u(x) for all n, and assume by contradiction that lim u(y,~) - u(x) - p . (Yn - x)

(1.4)

n

lyn_xl

=c~>0

for a sequence y,~ --* x. For k large enough we have IPk --Pl <-- a/2. a d d i n g a n d s u b t r a c t i n g Pk" (Y,~ -- x)/lYn -- xl to (1.4) we get limsup

Then, by

u ( y n ) - u ( x ) -- Pk " (Yn -- x ) > a_ lY~ -- xl -- 2 '

a contradiction to Pk E D+u(x). To proceed in the proof observe that for any x, y C 12 and p, q c •N we have

y--x (1.5)

_ u(y) - u(x) - q.(y-

(P-q)'ly-xl

~)

u(v)

- u(~)

- p. (y - x)

ly-xl

by-xl

For any n E N, set Yn : = X + (1/n)(p -- q) and take y = Yn in (1.5) to o b t a i n Ip - ql = u ( y n )

- u(x)

- q. (yn - x) _ u(y~)

ly~ - xl

- u(~)

- p. (y~ - x).

ly~ - xl

'

by definition of lim sup a n d lira inf this yields

(1.6) Ip-ql-< limsup y-~x, y ~

u(y)-u(x)-q.(y-x)_ lY - xl

liminf y--x, y~a

u(y)-u(x)-p.(y-x) lY - xl

If u is differentiable at z, t h e n D+u(x) A D - u ( x ) ~ 0 since it contains Du(x). In this case D+u(x) a n d D - u ( x ) reduce to singletons as a consequence of (1.6). Conversely, if for some x one has D+u(x) 7~ 0, D - u ( x ) 7~ 0, t h e n by (1.6) D+u(x) = D - u ( x ) is a singleton. This means that u is differentiable at x with { D u ( x ) } =

D+u(x) = D - u ( x ) .

1. DEFINITIONS AND BASIC P R O P E R T I E S

31

In order to prove (d), let Y E fl and consider the smooth function ~s(x) = Ix - y[2/r For any s > 0, u - ~s attains its maximum over/3 = / ? ( y , R) at some point xs. From the inequality (u - ~E)(x~) ~ (u - ~ ) ( ~ )

= u(~)

we get, for all ~ > O, Ix~ - ~12 _< 2s sup l u ( x ) l . xEB

Thus xe is not on the boundary o f / ? for e small enough, and by Lemma 1.7 (a), Dc2e(xe) = 2(x~ - ~)/r belongs to D+u(xe). This proves that A + is dense, and similar arguments show that A - is dense too. 9 As a direct consequence of Lemma 1.7 the following new definition of viscosity solution turns out to be equivalent to the initial one: a function u E C(f~) is a viscosity subsolution of (H J) in fl if (1.7)

F(x, u(x),p) <_ 0

V x E 12, Vp E D+u(x);

a viscosity supersolution of (H J) in 9t if (1.8)

F(x,u(x),p) > 0

V x E f~, Vp E n - u ( x ) .

Of course, u will be called a viscosity solution of (HJ) in f~ if (1.7) and (1.8) hold simultaneously. The above definition, which is more in the spirit of nonsmooth analysis, is sometimes easier to handle than the previous one. We employ it in the proofs of some important properties of viscosity solutions. As a first example we present a consistency result that improves Proposition 1.3. PROPOSITION 1.9. (a) If u E C(f~) is a viscosity solution of (H J), then F(x, u(x), Du(x)) = 0 at any point x E fl where u is differentiable; (b) if u is locally Lipschitz continuous and it is a viscosity solution of (H J), then F(x,u(x), Du(x)) = 0

almost everywhere in ~ .

PROOF. If x is a point of differentiability for u then by Lemma 1.8 (b) { D u ( x ) } = D+u(x) = D - u ( x ) . Hence, by definitions (1.7), (1.8) 0 >_ F(x, u(x), Du(x)) > O, which proves (a). Statement (b) follows immediately from (a) and the Rademacher's theorem on the almost everywhere differentiability of Lipschitz continuous functions (see [EG92]). 9

32

II. CONTINUOUSVISCOSITY SOLUTIONS OF H-J EQUATIONS

REMARK 1.10. P a r t (b) of Proposition 1.9 says t h a t any viscosity solution of (H J) is also a generalized solution, i.e., a locally Lipschitz continuous function u such t h a t

F(x~ u(x), Du(x)) = 0

a.e. in f t .

T h e converse is false in general: there are m a n y generalized solutions which are not viscosity solutions. As an example, observe t h a t u(x) = Ixt satisfies

lu'(x)l

- 1 = 0

in

]-1,1[

\ {0},

b u t it is not a viscosity supersolution of the same equation in ] - 1 , 1[ (see E x a m ple 1.6 or, alternatively, just observe that p = 0 belongs to D-u(O) a n d (1.8) is not satisfied at x = 0). In R e m a r k 2.3 we define infinitely m a n y generalized solutions of this e q u a t i o n which are not viscosity solutions. We shall come back to this point in w <3 Exercises

1.1. Check t h a t ~(x) = ~X'_

[1

x,

0 < x < 1/2 1/2<x<1

is a viscosity solution of l u ' ( x ) l - 1 = 0, x 9 (0, 1). Is u a viscosity solution of e q u a t i o n - ] u ' ( x ) l + 1 = 0 in ]0, 1[? 1.2. Let u(x) = {01,

x<0

bx 2 -t- ax,

x > O.

C o m p u t e D+u(O). 1.3. If u : l~N ~ ]~ is convex (i.e., u()~x + (1 - A)y) < $u(x) + (1 - $)u(y), for any x, y in ]~N, ~ E [0, 1]), t h e n its subdifferential at x in the sense of convex analysis is the set

acu(x) := {p 9 RN: u(y) > u(x) + p . ( y - x),Vy 9

}.

Show t h a t if u is convex t h e n Ocu(x) = D - u ( x ) at any x. 1.4. Let u 9 C(f~). Prove that D - u ( x o ) ~ 0 if x0 9 ~ is a local m i n i m u m o f u and t h a t D+u(xo) ~ 0 if x0 is a local m a x i m u m . 1.5. Show t h a t D+u(O) = D-u(O) = 0 where u is given by ~(x) =

Ixl~/~ sin 1Ix 2,

x # O,

u(O) = 0

while D+v(O) = 0, D-v(O) -- { 0 } for

v(x) = Ix sin 1/x],

x • 0,

v(0) -- 0 .

1. DEFINITIONS AND BASIC P R O P E R T I E S

33

1.6. Let u E C([a, b]). Prove the m e a n value property: there exists ~ E (a, b) such that

u(b) - u(a) = p(b - a) for some p E D - u ( ~ ) U D+u(~). 1.7. Check t h a t b o t h u l ( t , x ) =- 0 a n d u2(t,x) = ( t - ]xl) + are viscosity supersolutions of

ut-lu'(x)l=O

in [O,+ec[ x ]R

u(0, z) = 0,

x E R.

Is u2 a subsolution? [Hint: look at D + u 2 ( 1 , 0).] 1.8. Consider for x E IR the function u(x) : = Ce -Elxl with C, E > 0. Check t h a t

D-u(x)

S-CEe-EIxlx/Ixl

i f x -- 0 .

1.9. Let F ( x , r , p ) : = r + H(p) - g ( x ) L,A,B,

[H(p)l <_ n IPl,

if x r 0

with H , g such that, for some c o n s t a n t s

Vx, p E R y .

Ig(x)l _< Ae -BIzl,

Show t h a t if E < rain{ B; 1 / L } a n d C = A / ( 1 - L E ) , t h e n u given in Exercise 1.8 is a viscosity supersolution of

u(x) + H ( D u ( x ) ) - g(x) = O,

x E RN .

1.10. Assume t h a t un E C(f~), u,~ ---* u as n ~ + e c locally uniformly in f}. Show t h a t for a n y x0 E f} the following holds

D + u(xo) C lira sup D+u~ (x) X~X

0

(i.e., for any p E D+u(xo) there exist xn E f}, Pn E D+u~(x,~) such t h a t xn ---+xo, pn ---*p as n ---~ +ec). 1.11. Show t h a t u E C(f~) is a viscosity subsolution of (HJ) if a n d only if the following holds: for all ~ E C~(f~), ~ > 0 and k E ]R, if m a x n c2(u - k) > 0 t h e n there exists x0 E f} satisfying ~(u - k)(xo) = m a x n ~ ( u - k) a n d

~[xo) Also find the corresponding p r o p e r t y of supersolutions. 1.12. Let u be a Lipschitz continuous function in ~ with Lipschitz c o n s t a n t L. Prove t h a t D + u ( x ) a n d D - u ( x ) are contained in B(0, L) for all x E ~, a n d t h a t u is a (viscosity) s u b s o l u t i o n of IDuI - L = 0 a n d s u p e r s o l u t i o n of - I D u l + L = 0 in ~.

34

II. CONTINUOUS VISCOSITYSOLUTIONSOF H-J EQUATIONS 2. S o m e c a l c u l u s a n d f u r t h e r

properties

of viscosity solutions

In the first part of this section we collect some important stability properties of viscosity solutions and basic rules of calculus (change of unknown in (H J), chain rule . . . . ). In the second part we establish some useful formulas for the semidifferentials of continuous functions of the form u(x) -- infbe B g(x, b). This is an important class of nonsmooth functions, sometimes called marginal functions which includes the distance function and is closely related to Hamilton-Jacobi equations. The first result is on the stability with respect to the lattice operations in C ( ~ ) :

(u V v)(x) = max{ u(x), v(x) }

(u A v)(x) = min{ u(x),v(x) }. PROPOSITION 2.1. (a)

Let u,v E C(E~) be viscosity subsolutions of (HJ); then u v v is a viscosity subsolution of (H J).

(b)

Let u, v E C(~) be viscosity supersolutions of (H J); then u A v is a viscosity supersolution of (H J).

(c)

Let u E C(fl) be a viscosity subsolution of (H]) such that u > v for any viscosity subsolution v E C ( ~ ) of (H J); then u is a viscosity supersolution and therefore a viscosity solution of (H]).

PROOF. Let x0 be a local maximum for u v v - ~ with ~ E C 1(f~) and assume without loss of generality that (u v v)(xo) = u(xo). Then x0 is a local maximum for u - ~; so

F(xo, u(xo), D~(x0)) < 0 and (a) is proved. An analogous argument can be used for (b). In order to prove (c), let us suppose by contradiction that h := F(xo, u(x0), Vp(xo)) < 0 for some p E C1(~) and x0 E ~ such that

V x E ~(~0, 60) c ~,

u(xo) - ~(x0) < u(x) - ~(x)

for some 60 > 0. Consider next the function w E Cl(f~) defined by

w(~) = ~(~)

-

Ix

- x012 + u(x0)

-

~(~0) + 89

for 0 < 6 < 60. It is immediate to check that (2.1)

(u - w)(xo) < (u - w)(x),

Vx such that Ix - xol = 6 .

2. FURTHER

PROPERTIES

O F VISCOSITY

SOLUTIONS

35

Let us prove now that, for sufficiently small 6, (2.2)

F ( x , w ( x ) , D w ( x ) ) <_ 0

Vx 9 B(xo,5).

For this purpose, a local uniform continuity argument shows that, for 0 < 5 < 60,

fl~(x)

(2.3)

-

~(xo)l < ~1(5),

( I D a ( x ) - 2(x - xo) - D~(xo)l <_~2(5) + 25,

for any x 9 ]3(xo, 5), where oJi (i = 1, 2) are the moduli of continuity of ~ and D~. Hence

I~(~) - ~(~o)t < ~ ( 5 ) + 6 2

v x 9 ~(x0, 5).

Now,

(2.4)

F(x,w(x),Dw(x)) = h + F(x, w(x), D ~ ( x ) - 2(x - x0)) - F(xo, u(xo), D p ( x o ) ) .

If w is a modulus of continuity for F, then F(x, w(x), Dw(x)) < h + ~(5, wl (5) + 52, w2(5) + 25), for all x E B(xo, 5). Since h < 0, the preceding proves the validity of (2.2) for small enough 5 > 0. Fix any such 5 and set on

~(x)

(xo,5)

on f~ \ B(xo, 5) .

It is easy to check that ~ E C(~) (see (2.1)) and, by Propositions 1.3 (a) and 2.1 (a), is a subsolution of (HJ) in ~. Since ~(x0) > u(xo), the statement is proved. The next result is a stability result in the uniform topology of C(f~). PROPOSITION 2.2. Let un E C(s

(n E N) be a viscosity solution of

Fn(x, us(x), Dun(x)) = 0

(HJ)n

in ~.

Assume that Un

_______+,11,

Fn---* F

locally uniformly in f~ locally uniformly in ~t • ]~ • ]~:.

Then u is a viscosity solution of (H J) in f~.

36

II. CONTINUOUS VISCOSITYSOLUTIONSOF H-3 EQUATIONS

PROOF. Let 9~ C Cl(f~) and x0 be a local maximum point of u - p. As observed before, it is not restrictive to assume that u(x0) - ~ ( x 0 ) > u ( x ) - ~(~)

for x ~ x0 in a neighborhood of x0. By uniform convergence, un - 9~ attains, for large n, a local maximum at a point xn close to x0 (see Lemma 2.4). Then, Fn(xn, un(x,~), D~(x,~)) <_ O .

Since xn ~ xo, passing to the limit as n --* +oc in the above yields F ( x o , u(xo), D ~ ( x o ) ) < O.

A similar argument proves that u is also a viscosity supersolution.

9

REMARK 2.3. Proposition 2.2 does not hold for generalized solutions of (HJ)n. As an example, consider the saw-tooth like functions us defined by u l ( x ) = 1 - x and for n > 2 by x - 2--~ u,~(x)=

2j+2 2----U - - x

x 9 ]2j/2 n, (2j + 1)/2~[

j=0,1,...,2

n-l-1

x 9 ](2j + 1)/2 n, (2j + 2)/2n[

for x 9 ]0, 1[. It is evident that ]u~(z)] - 1 = 0 almost everywhere in ]0, 1[, but although ul is a classical (and therefore a viscosity) solution, un is not a viscosity solution for n > 2. The uniform limit of the sequence {u,,} is identically zero and does not satisfy the equation at any point. <~ In the proof of Proposition 2.2 we used the following elementary fact which is useful in many situations. LEMMA 2.4. Let v 9 C(f~) and suppose that xo 9 f~ is a strict m a x i m u m point for v in B ( x o , 6) c f~. I f vn 9 C(f~) converges locally uniformly to v in f~, then there exists a sequence { x n } such that (2.5)

zn

, x0,

v n ( z ~ ) ___ v ~ ( x )

vx 9 ~(~0, 6).

PROOF. Let Xn be a maximum point for vn o n / 3 ( x 0 , 5 ) and let {xnk}, k 9 1%1,be any converging subsequence of {x,~}, n 9 N. By uniform convergence, v,~(x~)

--~ v(~)

as k -* + o ~ ,

where ~ = limx,~ k as k --* +oc. The choice of {xn} yields

~(~) > v(~)

v~ 9 ~(x0,6),

so that, in particular, v ( ~ ) > ~(~o) 9

This implies ~ = xo and the convergence of the whole sequence.

9

2. FURTHER PROPERTIES OF VISCOSITY SOLUTIONS

37

The next result is on the change of unknown in (H J). PROPOSITION 2.5. Let u C C(f2) be a viscosity solution of (HJ) and 9

C C1(]~)

be such that @'(t) > O. Then v = cb(u) is a viscosity solution of

(2.6)

F ( x , ~(v(x)), ~ ' ( v ( x ) ) D v ( x ) ) = 0

x e ~,

where 9 = ~p-1.

PROOF. Since G ( x , r , p ) : = F ( x , ~ ( r ) , q ~ ' ( r ) p ) is defined only for r e (I)(~), here by viscosity solution of (2.6) we mean a function taking its values in (I)(I~) and satisfying the properties of Definition 1.1. Let x E l~ and p E D + v ( x ) . Then

v(y) _ < v ( x ) + p . ( y - x ) + o ( l y - x l )

asy-~x.

Since if2 is increasing, 9 (v(y)) < ~(v(x) + p - ( y -

x) + o ( l y - xl) )

= ~(v(x)) + ~'(~(x))p 9 (y - x) + o(ly - xt). By definition of v, this amounts to saying that 9 '(v(x))p 9 D + u ( x ) .

Therefore, F(x, u(x), ~ ' ( v ( x ) ) p ) <_ O, showing that v is a viscosity subsolution of (2.6). In a completely similar way one can prove that v is also a viscosity supersolution of (2.6). A slight generalization which is useful when dealing with evolution equations (see Exercise 2.3) is as follows. PROPOSITION 2.6. Let u 9 C ( ~ ) be a viscosity solution of (H J) and 9 : ~ • ]~ --~ ]~ a C 1 function such that

9 r(~, r) > o

v (x, r) 9 ~ • R .

Then the function v E C ( ~ ) defined implicitly by

9 (x, v(~)) = u(~), is a viscosity solution of

(2.7)

F(x, v(x), D v ( x ) ) = 0

in l~,

where F ( x , r,p) = F ( x , oF(x, r), D S F ( x , r) + (~r(x, r)p) .

38

II. CONTINUOUS VISCOSITY SOLUTIONS OF H-J EQUATIONS

PROOF. Let us detail only the subsolution part. Let x E ~ a n d p E D+v(x). Then, v ( y ) _< v ( x ) + p . ( y - x ) + o ( l y - x]),

as y --* x. Since ~ is nondecreasing with respect to r. we have ~(Y,v(Y)) < ~(Y,V(X) + P " ( Y - x) + ~

xl)),

and consequently 9 (y, v(y)) <_ 4)(x, v(x)) + O z ' ( X , v ( x ) ) . (y - x)

+ ~r(x, ~(~))p- (~ - ~) + o(1~ - x l ) . B y definition of v this means t h a t Dx~(X, v(x)) + qbr(X , v(x))p E D+u(x). Therefore

F ( x , u ( x ) , D x ~ ( x , v ( x ) ) + ~r(x~v(x))p) <_ 0, t h a t is, v is a viscosity subsolution of (2.7).

9

The next proposition collects the semidifferential versions of some useful facts in elementary calculus. PROPOSITION 2.7. Let u E C(~). Then

for v(x, r) = ~(r)u(x) (x E ~, r E 1~) we have

(i)

D + v ( x , r ) = { (q,o) E ~N+~ : q E ~(r)D+u(x)i ~ = ~'(r)u(x) }, provided ~ E C 1(]~), ~(r) > 0 for all r E JR; (ii)

for u(x) = v(T(x)), with v E C(~), we have p E D+v(yo)

if and only if

(DT(xo))tp E D+u(x)

(where t denotes transposition) provided T : f~ ~ ~ is a Cl-diffeomorphism and Yo = T(xo); (iii)

for ~?(r) = u(y(r)) we have D+v(r) ~_ D+u(y(r)) " O(r), provided y E C 1(]~, f~).

PROOF. Let us give some details only for p a r t s (i) and (ii) and leave the proof of (iii) as an exercise. Since ~ E C 1(R) we have ~ ( y , s) - ~ ( x , r ) = ~ ' ( r ) ~ ( ~ ) ( s

-- ~) + ~ ( ~ ) ( ~ ( ~ )

-- ~ ( x ) )

+ o(}s -- rl);

hence (q, ~) E D+v(x, r) if and only if (~'(r)~(~)

- o ) ( ~ - r) + V ( s ) ( ~ ( y )

- ~(x))

- q ' (Y - x ) < o ( l y - ~l + I~ - rl)

2. F U R T H E R PROPERTIES OF VISCOSITY SOLUTIONS

39

for all (y, s) in a neighborhood of (x, r). This easily implies a = ~'(r)u(x). Assume now ~ ( r ) ~ 0 and that q/~(r) ~ D+u(x). This implies ~ ( r ) ( ~ ( y ) - ~ ( x ) ) - q. (y - x) > ~ ( r ) o ( l y - xJ),

a contradiction to the above inequality with s = r. If p ( r ) = 0 the choice s = r gives - q . (y - x) < o(ly - xI) , thus q = 0. Hence (i) is proved. In order to prove (ii), let Y0 C 5 and p e D+v(yo). Then v(y) < v(~0) + p .

(y - y0) + o(ly - y01)

as y ~ Y0. Set xo = T-1(yo) and x = T-~(y). Then ~(x) < u(x0) + p . (T(x) - T(x0)) + o(Ix - ~01) = u(xo) +p.

- ~o) + o(Ix - x01),

DT(~o)(~

which proves (ii).

4,

REMARK 2.8. Similar results hold for D - . The sign condition on ~ is essential in (i) (see Exercise 2.4). The equality in the "chain rule" (iii) does not hold in general. The "change of variables" formula (ii) implies that u is a viscosity subsolution of (H J) in F/if and only if v(~) = u(T -1 (~)) is a viscosity subsolution of

F ( T - I ( ~ ) , v ( ' ~ ) , D T ( T - I ( ~ ) ) D v ( ~ ) ) = O,

~ e 5,

see Exercise 2.6.

<~

Let us now discuss the following quite natural question: when is a piecewise C 1 function satisfying (H J) at all points of differentiability a viscosity solution of the same equation? The answer is very simple in the one-dimensional case. Assume u c C ( ] - a , a D N C 1(]-a~ 0[ U ]0, a D is a classical and therefore a viscosity solution of

F ( x , u ( x ) , u ' ( x ) ) -- 0

in l - a , 0[ U]0, a [ .

It is immediate, by Taylor~'s expansion, to check that for small enough lyI,

o(lYl)

if y > O,

~(Y) = [ ~ ( 0 ) + ~'~(0)y + o(lyl)

if y < 0,

~u(O) + u'+(O)y +

where u~_(0), u'_(0) denote, respectively, the right and the left derivative of u at 0. Hence, p E D+u(O) if and only if ~(0)

< p _< ~'_(0).

Similarly, p E D-u(O) is equivalent to p E [u~_(O), u~(O)]. Therefore u will be a viscosity solution in l - a , a[ provided F(0, u(O),p) < 0

P(O,u(O),p) >_ 0

Vp e I+ := [u~(0), u'_(0)] = D+u(O) Vp 9 I_ := [u' (O),u~_(O)] = D - u ( O ) .

40

If. CONTINUOUS VISCOSITY SOLUTIONS OF H-J EQUATIONS

Of course, in the case where u~_(0) # u~(0), only one of the intervals I+, I_ is nonempty. The preceding simple remark provides a different explanation of the phenomenon described in Example 1.6. In order to extend these considerations to higher dimensions, let us assume t h a t

where f~i (i = 1, 2) are open subsets of ~ and F is a smooth surface. Let us denote by n(x) and T ( x ) (x E F) the unit normal vector to F pointing inward f~l and the tangent space to F at x, respectively. Denote finally by PT and PN the orthogonal projections of ]~N onto T ( x ) and, respectively, the space spanned by n(x). It is an immediate consequence of the definition t h a t if u E C ( ~ i) is a viscosity solution of (H J) in fli (i = 1, 2), then u is a viscosity solution of (H J) in ~1 u s 2 = gt \ F. As expected, such a u will be a viscosity solution in fl if some interface condition on F is satisfied. PROPOSITION 2.9. Let u E C ( ~ ) and assume that its restrictions u i to f~i U F belong to C1(~ i U F) (i = 1, 2). Then u is a viscosity solution of (HJ) in ~ if and only if the following conditions hold (a) (bl)

(b2)

u i is a classical solution of (H J) in Fti (i = 1, 2) F ( x , u(x), PrDu~(x) + ~n(x)) < 0 V~ 9 [Dul(x) 9n(x), Du2(x) . n(x)],

Vx 9 r ,

F ( x , u(x), P T D u i ( x ) + ~n(x) ) >_ 0 V~ E [ D u 2 ( x ) . n ( x ) , D u l ( x ) . n ( x ) ] ,

Vx 9

PROOF. By Proposition 1.3, u is a viscosity solution of (H J) in fl \ F if and only if (a) holds. So we are left to prove that (bl), (b2) are equivalent to the fulfillment of (1.7), (1.8) at any x 9 F. Since ul -= u2 on F we have (ul - u2)(x(t)) = 0 for all t 9 [0, 1], where x ( . ) is a parametrization of a curve lying on F such t h a t x(0) = x. Hence, by the smoothness of F, 0 = D ( u l - u 2 ) ( x ( t ) ) , b(t),

kit 9 [0, 1].

In particular for t = 0 this gives D ( u l - u2)(x) 9b(O) = 0 so

D(ul-u2(x).r

=0

Vr E T(z).

whence it follows t h a t P T D u l ( x ) = PTDu2(x),

x E F.

Let x E F and observe that u(y) = u(x) + ( P r D u i ( x ) + P N D u i ( x ) ) 9 (y - x) +

o(ly -

~1)

2. FURTHER

PROPERTIES

OF

VISCOSITY

SOLUTIONS

41

for any y E B ( x , (i) n D{, (i = 1, 2). Hence, p E D + u ( x ) if and only if (2.8)

( P T D u i ( x ) + P g D u i ( x ) ) . (y - x) <_ ( P T P + PNP)" (Y - x) + o(ly -- x[),

for any y E B ( x , 5). If we choose now y = x + tr, with ~- E T(x), we get P r p = PTDul(x) = PTDu2(x) .

On the other hand, the choice y = x + t n ( x ) with t > 0 in (2.8) gives P N D u l ( x ) 9 n ( x ) <_ PNP" n(x) + o(t)/t,

since y E ~1. Similarly, y = x - t n ( x ) E f~2 for t > 0 and therefore PNP" n ( x ) <_ P g D u 2 ( x ) . n ( x ) + o ( t ) / t .

Now, PNq " n ( x ) = q . n(x), so that D+u(x) c { p ~ ~N : p = P r D u i ( x ) + ~n(x); E [ D u l ( x ) . n ( x ) , D u 2 ( x ) 9n(x)] }

for any x E F. It is not hard to show that this inclusion is actually an equality by taking an element of the set on the right-hand side and proving by contradiction that it satisfies (2.8). In a completely analogous way one shows that D - u ( x ) = { q E ]RN : q = P T D u i ( x ) + {n(x); { E [Du2(x) 9n(x), D u l ( x ) 9n(x)] }

and the proof is complete.

9

The following technical lemma is useful in the study of evolution equations. LEMMA 2.10. Let us assume that (2.9)

Pl ~

F(~, P, Pl, P2 . . . . , PN)

is nondecreasing f o r any fixed x, r, P2 . . . . PN" A s s u m e also that f~ = ]a, b[ x ~', with ~' an open subset in ]~N-1. I f u E C(-~) is a viscosity subsolution (respectively, a supersolution) of (H J), then

(2.10)

F(5, u(5), Dp(~)) < 0,

(respectively, >_ O)

at any local m a x i m u m (respectively, m i n i m u m ) ~ of u - ~ over ]a, b] • ~', f o r any E Cl(]a, b] • ~').

PROOF. The only thing to check is that (2.10) holds true at strict local maxima of the form ~ = (b, x2,- 9 ~Y). Let xn be a maximum point of ~;n(X) = u(x) -

~ ( x ) + n(b

Xl---------~

42

II. CONTINUOUS VISCOSITY SOLUTIONS OF H-J EQUATIONS

in [b - r, b[ x / ~ ( 3 , r) C ~t. We claim t h a t xn ~ 3. To prove the claim we assume that

after taking a subsequence if necessary. Since ~bn(xn) < (u - ~)(x~), t h e n ~ < (u - ~o)(}). On the o t h e r hand, we take x n = (b - 1 / v ~ , 3 2 ~ . . . , 3 N ) to get e~(x~)

>__ r

= (u - ~ ) ( x ~) - - n

therefore > (u - s ) ( 3 ) .

Since 3 was a strict m a x i m u m point for u - c2 we conclude t h a t ~ = 3. B y the definition of subsolution,

F(xn,u(x,),p~) < 0 where

p~ =

G,~(x~)

+ ~(b-

;&,,~(xn)

By t h e m o n o t o n i c i t y a s s u m p t i o n on F this implies F ( x n , u(x~), D ~ ( x ~ ) ) < 0 and t h e claim follows by letting n --* + o c .

*1

In the final part of this section we consider functions t h a t are expressible as pointwise infimum of a family of functions: (2.11)

u(x)

= inf

g(x, b),

bcB

where g : f~ • B --~ JR, ~ c_ ]~:~g is open and B is a topological space. Functions of this kind are often called marginal functions. In the context of this b o o k the m a i n examples are, of course, value functions of o p t i m a l control problems,

V(x)

= inf

J(x, a)

aGA

(see C h a p t e r I and, for a detailed investigation, C h a p t e r III, w example is the distance function from a set S c ]RN,

d(x, S)

: = inf Ix - s] . sES

A m o r e basic

2. F U R T H E R P R O P E R T I E S O F VISCOSITY SOLUTIONS

43

Let us also mention that in w we consider the inf-convolution of a function u, defined as

uE(x) := inf [u(y) + Ix - yl2/2z],

z > 0,

yEfl

another relevant example of marginal function. We denote by M(x) the (possibly empty) set

M(x) = arg m i n g ( z , b) :--= { b E B : u(x) = g(x, b) } bEB

and assume that g(x, B) is bounded for all x E f~ as well as that x ~-* g(x, b) is continuous in x uniformly with respect to b E B; that is (2.12)

Ig(x, b) - g(y, b)l < w(Ix -

Yl, R),

for all [x[, lYl -< R, b E B, for some modulus w. A first elementary result is on the relations between the semidifferentials of u and the semidifferentials of g with respect to the variable x, denoted by D~g. LEMMA 2.11. Let us assume (2.12). Then u E C(12) and

D+~(~) ~ D~+g(~,b) D;g(x, b) ~_ D-u(x):

for all b E M(x) .

PROOF. The continuity of u is a simple consequence of definition 2.11 and assumption (2.12). Now let p E D+g(x, b); then

g(x+h,b)
aslh[--~O.

For b E M(x), we deduce

u(x+h)<_u(x)+p.h+o(]h[)

as Ih[ ~ 0 ,

and thus p E D+u(x). I f p E D-u(x), then

u(x+h)>u(x)+p.h+o(Ih[)

as [hi --* 0 .

Hence, for b E M(x) we deduce

g(x + h,b) > g(x,b) + p. h + o(lhl) showing that p E D;g(x, b), for all b E M(x).

as [hi--*0,

9

REMARK 2.12. The result shows in particular that D+u(x) ~ 0 at those points x where g is differentiable with respect to x and M(x) ~ 9. ,~

44

II. CONTINUOUS VISCOSITY SOLUTIONS OF H-J EQUATIONS

To proceed further we assume g( -, b) differentiable at x uniformly in b; that is, for some modulus a~l, (2.13)

Ig(x + h, b) - g(x, b) - Dxg(x, b) . h i <_ Ihl Wl(Ihl)

for all b E B and h small. We assume also that (2.14)

b,

(2.15)

b~

, Dzg(x, b) is continuous, g(x, b) is lower semicontinuons.

We will use the following notations:

Y(x) := { D x g ( x , b ) : b E M ( x ) } and, for the (one-sided) directional derivative of u in the direction q, Ou ( x u(x + tq) - u(x) ~qq ) : = lira t-~O+

t

(in w and Chapter III this directional derivative will be also denoted with cOu(x; q)). PROPOSITION 2.13. Let us assume (2.12), (2.13), (2.14), (2.15) and that B is

compact. Then

(2.17)

Y(x) r O, D+u(x) = -~-6Y(x),

(2.18)

D_u(x)={{oy }

(2.16)

ifY(x)={y}, if Y ( x ) is not a singleton.

In particular, u is differentiable at x if and only if Y ( x ) is a singleton. Moreover, u has the (one-sided) directional derivative in any direction q, given by ~qq (X)

min y . q

yeY(x)

min

p . q.

pED+u(x)

PROOF. By the assumptions made, M ( x ) and Y ( x ) are nonempty. The inclusion ~6Y(x) C_ D+u(x) follows from Lemma 2.11 and the fact that D+u(x) is closed and convex (see Lemma 1.8). To prove the opposite inclusion, let us assume by contradiction the existence of some p E D+u(x) such that p ~ K6Y(x). Then there exists a unit vector v such that (2.19)

p-v_<0,

q.v>0

for a l l q E K 6 Y ( x ) .

Let xn = x + v / n and bn E B be such that (2.20)

g(x., b.) < u(x.) + 1/n 2 .

2. FURTHER PROPERTIES OF VISCOSITYSOLUTIONS

45

From the definition of D+u(x) and (2.13) we get v

1

v

n

~t

n

g(x, b,~) + Dxg(x, b~)- - + - W l ( l / n ) <_ u(x) + p. - + o(1/n), as n --* +oc, and then, by (2.19) and the inequality g(x, b,~) > u(x),

o(1/~)

Dxg(x, b n ) . v < ~ / n

(2.21)

+Wl(1/n)

asn~+~z.

We extract now from bn a subsequence converging to some b E B and use (2.14) to obtain

Dxg(X, b) . v < O, a contradiction to (2.19) if b E M ( x ) . implies

This is indeed the case; actually (2.13)

g(x, bn) <__g(xn, bn) - Dxg(x, bn) " v / n + o(1/n) . Thus, using (2.15), (2.20), the continuity of u and assuming without loss of generality t h a t b~ --* b,

g(x,b) < liminf g(x, bn) < u(x) . This completes the proof of (2.17). In order to prove (2.18), let us assume first that Y ( x ) is not a singleton and, by contradiction, that D - u ( x ) # r Since we know by (2.16), (2.17) that D+u(x) # O, then u is differentiable at x with D+u(x) = {Du(x)} (see Lemma 1.8). Since Y ( x ) and, a f o r t i o r i , u6Y(x) is not a singleton, we obtain a contradiction. Consider now the case Y ( x ) = {y}. By Lemma 2.11, D - u ( x ) C {y}. To prove the opposite inclusion, assume by contradiction t h a t

(2.22)

u(x + h, 0 - u(x) - y . h,, < -E Ihnl

for some E > 0 and hn ~ 0. Choose b~ E B such that

9(x + ha, bn) <_ u(x + ha) + ]hal 2,

(2.23) and use

(2.13)

and (2.22) to get

g(x, b,~) + Dzg(x, b,~) . hn - u(x) - y . h,~ < - c Ih,~l + o(lh~l), and, consequently, h.

(Dzg(x, bn) - y) . ~

< - ~ + o(1)

as n ---* + o c .

46

II. CONTINUOUS VISCOSITY SOLUTIONS OF H-J EQUATIONS

Now we extract subsequences such that, after renaming, b~ --~ b E B and hn/ih,~l ---+ q ~ 0. Then

(Dxg(x, b) - y ) . q <_ - ~ , which gives a contradiction if b E M ( x ) . This is indeed the case. Actually, from

g(x, b.) <_ g(x + h., b.) - D~g(x, b . ) - h . + o(Ihnl), (2.23), the continuity of u and the lower semicontinuity of g with respect to b, it follows that g(x,b) < l i m i n f g ( x , bn) < u(x) . n----*~r162

To prove the statements about the directional derivatives, we first observe that for all p E D + u ( x ) l i m s u p u(x + tq) - u ( x ) < P ' q . t~O

t

Then we pick tn --* 0 such t h a t lira u(x + t~q) - u(x) = lim inf u(x + tq) - u(x)

n

tn

t~o

t

and choose bn E B such that g(x + tnq, bn) <_ u(x + tuq) + t 2. Next we extract a subsequence converging to b E B and deduce, by the same arguments as before, that lim u(x + t~q) - u(x) >_ Dxg(X, b). q n

tn

and that Dzg(x, b) E Y(x), since b E M ( x ) . Hence, for all p E D + u ( x ) ,

p . q >_ limsup u(x + tq) - u(x) >_ liminf u(x + tq) - u(x)

(2.24)

t ~O +

t

t ~O +

t

>_ Dzg(x,-b) . q . W i t h the choice p = Dxg(x,-b), the above proves the existence of OU(x) bq " From (2.24) it also follows that inf

pcD+u(~)

P" q > O U ( x .

-- Oq

) > yeY(:c)inf y . q .

By (2.14) and (2.17), "C6Y(x) = D + u ( x ) is a convex compact set and the proof is complete. 4

2. FURTHER PROPERTIES OF VISCOSITY SOLUTIONS

47

Next we apply P r o p o s i t i o n 2.13 to c o m p u t e the semidifferentials of the distance function from a n a r b i t r a r y set S C_ ]~Y S ~ ~, t h a t is

(2.25)

d(x) = d(x,S) : = inf Ix - z I = m i n i x - z I . zES zES

T h e last expression for d(x) is more convenient a n d leads to consider the set of projections of x o n t o S, t h a t is

P(x) : = { z E OS : d(x) = I x - z I } # ~ . PROPOSITION 2.14. For any S 7s ~, d E C(N N) and for all x ~ S and unit vector q,

D+d(x) = ~

lz:zl

{~-P(X)

:z ~

P(x)

ifP(x)={P(X)}

D-d(x) =

-P(X)l

i f P ( x ) is not a singleton

~(x)=

min x - z zEP(x) Ix -- z I "q"

PROOF. T h e c o n t i n u i t y of d is easy to prove. If we restrict ourselves to those x such t h a t e < d(x) _< z -1, Ixl _< c for some e, C > o, t h e n the search for the m i n i m u m in (2.25) can be restricted to z E S such t h a t Izl _< C + z -1. For x and z satisfying these restrictions, x ~ Ix - z I has b o u n d e d second derivatives, thus g(x, z) : = Ix - z[ fulfills (2.13) as well as the other a s s u m p t i o n s of Proposition 2.13, a n d the conclusions follow from the arbitrariness of e. 9 REMARK 2.15. Note that, by P r o p o s i t i o n 2.14 a n d L e m m a 1.7, d is differentiable at x if a n d only if the projection of x onto S is unique. A well-known result in convex analysis (Motzkin's theorem, see [Val64 D asserts that the set of projections P(x) is a singleton for all x if a n d only if 3 is convex. T h e n 3 is convex if a n d only if d is differentiable in ]~g \ 3. In this case the projection p(x) d e p e n d s continuously on x, so t h a t d E C I ( ~ N \ ~). <3 It is well k n o w n t h a t for s m o o t h OS, say C 2, the distance function is s m o o t h near OS a n d satisfies the eikonal equation (2.26)

IDul = 1

in ]~N \ 3,

locally a r o u n d OS. If this e q u a t i o n is interpreted in the viscosity sense t h e n it holds globally for a n y S. COROLLARY 2.16. The distance function d from S is a viscosity solution of (2.26). It is also a viscosity subsolution, but not a supersolution, in all 1~N. PROOF. T h e first s t a t e m e n t follows i m m e d i a t e l y from Proposition 2.14.

Since

d(x) > O, we have 0 E D - d ( x ) for all x E S, so d is n o t a supersolution at a n y such point. Moreover, for x E 3 either D+d(z) = 0 or D+d(x) = {0}, so d is a subsolution.

9

48

II. C O N T I N U O U S VISCOSITY SOLUTIONS OF H-J EQUATIONS

In w we prove that d is actually the unique viscosity solution of (2.26), continuous up to OS and null there, provided ]RN -. S is bounded (see Remark 5.10). A uniqueness result for the case when this set is unbounded can be found in Chapter IV. Another notion which is strongly related to the properties of the distance function is t h a t of exterior normal n(x) to the set S at the point x E OS. In the case of smooth OS, the function n is the unique unique extension of Dd to OS, and for x ~ S sufficiently close to OS we have (2.27)

x

= p(x) + d(x) n(p(x)),

Dd(x) = n(p(x)),

where p(x) indicates the (unique) projection of x onto S. following definition.

This motivates the

DEFINITION 2.17. Let S C ~ N be a general nonempty set. A unit vector v is a (generalized) exterior normal to S at z E OS (and we denote z~ c N(z)) if there exists x ~ S such t h a t

x = z 4- d ( x ) ,

and

{z} = P ( x ) .

<]

Equivalently, z is the projection onto S of some point off S in the direction of ~, i.e., 3t > 0 such that

z + t , r S,

P ( z + tv) = {z}.

Equivalently, there is a ball outside S, centered at z + tu for some t > 0, touching precisely at z, t h a t i s / ~ ( z 4- tv, t) A S = {z}. Clearly N ( z ) = {n(z)} if OS is C 2, but in general N(z) may be empty or may not be a singleton, see Exercise 2.11. Note that, by Proposition 2.14, the second formula in (2.27) becomes (2.28)

Dd(x) e N(p(x)),

P(x) = {p(x)},

at every point x where d is differentiable. For other characterization of the exterior normal and the connection with the notion of proximal normal see Exercises 2.10 and 2.12. Exercises 2.1. Show by a density argument that an equivalent definition of viscosity solution of (H J) can be given by using C~r instead of C1(~) as the "test function space". 2.2. Show by exhibiting an example that is false in general that if u, v are viscosity solutions of (H J) the same is true for u v v, u A v. 2.3. Let u be a viscosity solution of ut

+ r(t, x, u(t, x), Dxu(t, x)) = O.

Use Proposition 2.6 to show that v(t, x) : = e-Atu(t, x)~ A E JR, is a viscosity solution of

vt 4- Av 4- e-~tF(t, x, e~tv(t, x), e~tDxv(t, x)) = O .

2. FURTHER PROPERTIES OF VISCOSITYSOLUTIONS

49

2.4. Let u E C ( ~ ) . Prove the following statements: (a) (b) (c) (d)

D+ (au)(x) D+(au)(x) D+(u + c;)(x) D+ua(x)

= = = =

ifa>O, aD+u(x) if a < O, aD-u(x) D+u(x) + D~(x) if ~ E C~(f~), aD+u(x) + (1 - c~)D~o(x),

where ~o E C1(12) and ua : = c~u + (1 - a)~,o, a E [0, 1]. 2.5 (ON THE CHAIN RULE). For u E C ( ~ ) and y E CI(]R, ~) let ~(r) := u(y(r)). Prove that

{ p E ] R : p = ~ l ( r ) . p , p E D + u ( y ( r ) ) } C D+r?(r),

Vr E]R.

The two-dimensional example

u(x) = u(xl, x2) = Ixll 1/2,

y(r) = (yl(r),y2(~))

= (0,~)

shows t h a t a strict inclusion may occur in the above formula. 2.6. Let T : 12 ~ T ( ~ ) := ~ be a C 1 diffeomorphism. Prove that u E C ( ~ ) is a viscosity solution of

F(x, u(x), Du(x)) = O,

x E

if and only if v(~) := u ( T - l ( ~ ) ) is a viscosity solution of

F(T-I(~); v(~), DT(T-I(2))Dv(2)) = O,

~ E ~.

Here D T denotes the jacobian matrix of T. Work out in detail the linear case f(x,r,p) = Ar-f(x).p-g(x), A > 0. 2.7. Suppose t h a t equation Fn(x, u,~(x), Dun(x)) = 0 has a classical solution Un E C1(~) for n = 1 , 2 ; . . . . Show that, under the assumptions of Proposition 2.2, u = l i m ~ + ~ r u~ is a viscosity solution of

- F ( x , u(x), Du(x)) = O . 2.8. Let u E C(~ N) (N >_ 1), and e i be the ith vector of the standard basis in ~N. The set

D+u(x)

:-~

{aE]~:limsupt_0 u(x § te i)l ]t - u ( x ) - a t

~_ 0 }

is called the i th partial superdifferential of u at x. Prove that (a) p = (Pl...PN) E D+u(x) if and only if pi E D+u(x); (b) D+q(xl) = D+u(xl,x2 . . . . ,XN), for any fixed ( x 2 , - . . ,XN) E ~:~N-1, where T/(Xl) :-~ U(ZI~.~2,-.. ,~N);

50

(c)

If. CONTINUOUS VISCOSITY SOLUTIONS OF H-J EQUATIONS

deduce from the above that u is a viscosity subsolution of (Ou)/(Oxl)(x) < 0, x E ~N, if and only if

~'(xl) < 0,

xl c

in the viscosity sense, for any (x2,. 9 XN) E R g-1. 2.9. Consider the distance function d from a set S. Under which conditions does u := - d solve ]Du[ = 1 in the viscosity sense? 2.10. Prove that v is a (generalized) exterior normal to S at z if and only if there exists E > 0 such that

B(z +ev, E) n ~ = O. 2.11. Compute the (generalized) exterior normal vectors to S in the following cases:

(a)

s={0};

(b)

S = { X E ]~N : X ~- t x 0-{-

(c)

s = { x c R N :g(x) < 0 } n { x ~ R

(d)

s = { x E ~ N : g(x) < 0 } u { x E ~ N : f ( x ) < 0 }

(1 --t)Xl,

t e

~:fr

[0,1] }; <0};

Here, f and g are C 1 functions. Finally, let

s = {x = (Xl,X2) c ~2: ]x~l ~ Ix113/2 } Show that N(0) = 0 although the exterior normal at x = 0 exists (namely, n(0) = (0, 1)). Observe that OS is C 1 but not C 2. 2.12. Define a proximal normal to a closed set S C R g at x E 9 S as any vector p E it(N such that d(x + rp, S) = r [Pl, for some r > 0. Prove the following characterization of generalized exterior normals to S:

N ( x ) = {~ = P/IPl : P proximal normal to S at x } . 3. S o m e c o m p a r i s o n

and uniqueness results

In this section we address the problem of comparison and uniqueness of viscosity solutions. This is a major issue in the theory, also in view of its relevance in connection with sufficient conditions in optimal control problems. The results presented here are not the most general; in fact they are selected to show the main ideas involved in the proofs with as few technicalities as possible. In Chapters III, IV and V we prove many other comparison theorems especially designed for the Hamilton-Jacobi-Bellman equations arising in optimal control theory, and we refer also to the literature quoted in the bibliographical notes for more general results.

3. SOME COMPARISONAND UNIQUENESS RESULTS

51

As an introduction to the subject, suppose that Ul, u2 9 C ( ~ ) r c l ( f l ) , satisfy the inequalities (3.1)

ul(x) + H(x, DUl(X)) < 0 u2(x) + H(x, Du2(x)) k 0

for x 9 ~ and (3.2)

ul _< u2

on 0 f l .

Suppose ~ bounded and let xo be a maximum point for Ul - u2 on ~ . If x0 E f~, then Dul(xo) = Du2(xo) and (3.1) gives by subtraction u l ( x ) - u2(x) < u l ( x 0 ) - u2(x0) < 0

vx e

~.

vz e

~,

If, on the other hand, x0 E 0f~, then u l ( x ) - u 2 ( x ) < u l ( x 0 ) - u2(~0) < 0

as a consequence of (3.2). Hence, ul < u2 in ~ . Reversing the role of Ul and u2 in (3.1), (3.2), we get, of course, the uniqueness of a classical solution of the Dirichlet problem

u(x) + H(x, Du(x)) = 0,

x 9 fl,

u = 4,

on 0f~.

The preceding elementary proof fails if ul, u2 are continuous functions satisfying the inequalities (3.1) in the viscosity sense since Dui may not exist at x0. However, the information contained in the notion of viscosity sub- and supersolution is strong enough to allow the extension of comparison and uniqueness results to continuous viscosity solutions of equation (H J) with rather general F . In the following we present some comparison theorems between viscosity suband supersolutions in the cases f~ bounded, ~ = R N, and for the Cauchy problem. As a simple corollary, each comparison result produces a uniqueness theorem, as indicated in the remarks. For simplicity we restrict our attention to the case F ( x , r , p ) = r + H(x,p); the results hold, however, for a general F provided r F(x, r,p) is strictly increasing (see Exercises 3.6 and 3.7), and for some special F independent of r, see Theorem 3.7, w and Chapter IV, w THEOREM 3.1. Let ~ be a bounded open subset of]~ y . Assume that Ul, u2 E C ( ~ )

are, respectively, viscosity sub- and supersolution of (3.3)

u(x) + g ( x , Du(x)) = 0,

and (3.4)

ul <_ u2

on O~ .

x 9 f~

52

II. CONTINUOUSVISCOSITY SOLUTIONSOF H-J EQUATIONS

Assume also that H satisfies IH(x,p) - H(y,p)] <_ wl(Jx - yl(1 + ]p])),

(H1)

for x, y C f~, p E ]~N, where Wl : [0, +oc[ ---* [0, +cxD[ is continuous nondecreasing with wl(O) = 0 . Then ul <_ u2 in-~. PROOF. Define, for r > 0, a continuous function (I)E on f~ x D by setting

9 ~(x,y) = ul(z) - u2(y)

I x - yl 2 2r

a n d let (xr y~) be a m a x i m u m point for ~ (3.5)

on i~ x f). Then, for any r > 0

m a _ x ( u l - u 2 ) ( x ) = m a x ( I ) ~ ( x , x ) __% m a x xef~

xEfl

(x,y)e~xl2

{E(x,y)=(I)~(xE, y~).

We claim t h a t (3.6)

l i m i n f (I)~(x~, y~) _< 0

as r --* 0 .

This, together with (3.5), proves the theorem. In order to prove (3.6), let us observe first t h a t the inequality 9 ~(x~, x~) <_ ,~(x~, y~) a m o u n t s to Ix~

y~21 2E

<

u2(x~)

-

u2(y~).

--

This implies

Ix~- y~l < (Cz) 1/2, where C depends only on the m a x i m u m of ]u21 in f/. Therefore, (3.7)

]xE - Y~I - - ~ 0

as ~ --+ O;

and, by c o n t i n u i t y of u2, (3.8)

[z~ - yA 2

2~

~0

as c -+ 0.

Now there are two possible cases:

(i)

(Xen,ye,) C 0(ft X fl) for some sequence an ---* 0+;

(ii) (xE, yc) E ~ X ~ for all E E ]0,~[.

3. SOME COMPARISON AND UNIQUENESS RESULTS

53

In case (i) either xE, E 0~, and then, by (3.4), ?Al(Xe. ) -- u2(yen ) ~-~ U2(Xen) -- ?As(yen),

or ye, E O ~ and then

Note that the right-hand sides of both these inequalities tend to 0 as n --* oc by (3.7) and the uniform continuity of Ul and us. Therefore

9 .(x.,y~o) < ~l(X~o)- ~s(y~o)---~0

as ~ - ~ ~ ,

and the claim (3.6) is proved in this case. Assume now that (xe, y~) E ~ x ~ and set ~ s ( x ) = ~s(y~) + y~l______~ Ix ~ 2c '

~l(y) = ~1(~)

I ~ - yl s 2E

It is immediate to check that ~i e CI(E~) (i -- 1, 2) and x~ is a local maximum point for ul - ~s, whereas yc is a local minimum point for u2 - ~1. Moreover, D~j(y~) - x~ - YE _ D ~ s ( x E ) 9 By the definition of viscosity sub- and supersolution, then, _

By (H1), this implies

C

and, afortiori

E

Taking (3.7) and (3.8) into account, (3.6) follows and the proof is complete.

9

REMARK 3.2. If Ul, uS are both viscosity solutions of (3.3) with Ul = us on c9~, from Theorem 3.1 it follows that ul = us in f~. <~ REMARK 3.3. The statement of Theorem 3.1 is true also for the equation Au(x) + H(x, Du(x))

= 0

x E fl

with )~ > 0. On the other hand, for the equation H ( x , D u ( x ) ) = 0 non-uniqueness phenomena may appear. An extreme case is H ( x , p ) = 0 for all x and p: in this case any u e C(f~) is a viscosity solution. <~

54

II. CONTINUOUS VISCOSITYSOLUTIONSOF H-J EQUATIONS

REMARK 3.4. W h e n the Hamiltonian has the form (0.1), the assumption (H1) of the theorem is satisfied, for example, if f is continuous, A is compact, x ~ f(x, a) is Lipschitz continuous uniformly with respect to a C A and x ~ e(x, a) is continuous in ~, with a modulus of continuity independent of a E A (see Exercise 3.2).
BC(~ N) of bounded continuous functions on ]Rg. The scheme of the proof is similar to that of Theorem 3.1, with more technicalities due to the lack of compactness. We assume the following condition on H: (H2)

H(y, )~(x - y) +p) - H(x, s

- y) + q) <_w2([x - y[ + A Ix - y[2, R) + wa([P - ql) for all ~ _> 1, p,q E/~(0, 1), x,y E B(O,R), V R > O,

where w2, w3 are moduli. It is easy to realize that (HI) and (I-I3)

I H ( x , p ) - H(x,q)l <_w(Ip-q[)

Vx,p, q e R N,

imply (H2) with w3 = ~ and ~2(r,R) = wl(2r) for all r,R > O. THEOREM 3.5. Assume that Ul, u2 E BC(]~ N) are, respectively, viscosity sub- and

supersolutions of (3.9)

u(x) + H(x, Du(x)) = O,

x e ]~N

with H satisfying (H2). Then, ul <_u2 in ]~N. PROOF. Set g(x) ---- 89log(1 + Ixl 2) and

9 (x,y)=ul(x)-u2(y)

Ix-Yl2

Z(g(x)+g(y)),

2z where z, ~ are positive parameters to be chosen later. Observe that for ~ = 0, coincides with the auxiliary function employed in the proof of Theorem 3.1. Suppose by contradiction that 5 := u l ( x ) - u2(x) > 0 for some 5 and choose /3 such that ~g(~) < 5/4 and w3(2~) < (~/6. Hence, (3.10)

(~/2 < 5 - 2~g(~) = ~ ( ~ , 7 ) <

sup ~ ( x , y ) . RN x Ry

Since 9 is continuous and limlzl+M~+~z ~ ( x , y) = - o o , there exists (xe, ys) such that

9 (x~, y~) =

sup 4~(z, y). ~:v x

~N

The claim is that ~(x~, Yc) < (~/2 for small enough c > 0, leading to a contradiction to (3.10). In order to prove the claim, let us observe first that ~(xc, y~) _> 5/2 > 0 implies

~(g(x~) + g(y~)) < sups1 + sup(-u2).

3. S O M E C O M P A R I S O N

AND UNIQUENESS

RESULTS

55

Therefore there exists R > 0 such t h a t x~ and ye belong t o / ~ ( 0 , R) for all ~ > 0. In fact, if this were not true, the above inequality would contradict the b o u n d e d n e s s of ul, u2 since l i m l x l ~ + ~ g(x) = +ec and ,2 > 0. Next we observe t h a t the inequality ~(xE,. xE) + ~(YE, Yr _< 2~(xz, y~) a m o u n t s to

]xr - yet 2 <_ ul(xz) - ul(yE) + u2(xr

(3.11)

- u2(y~)

E

and this gives

Ixe - Y ~ I < (zC) 1/~,

(3.12)

where C is a constant d e p e n d i n g on sUp~N lull and supNN lu21. By plugging (3.12) into (3.11) we o b t a i n (3.13)

[xr - y~l 2

~0

as z --~ 0 +

because of the uniform continuity of ul, u2 i n / ~ ( 0 , R). Let us consider now the C 1 functions

~l(y) = ~ ( ~ ) _ [x~ - yl 2

~(g(~) +g(y)),

2r Ix - yr 2 ~2(x) = u2(yE) +

2~

+ ~(g(x) + g(Ye));

observe t h a t x~ is a m a x i m u m for Ul - ~2, y~ is a m i n i m u m for u2 - ~1 and t h a t D~I(y~) - ~ - y~

ZDg(y~),

E

Dc22(xe) = xs - Y______! + ZDg(xe). By the definition of viscosity sub- and supersolution, ul(xr

+ H ( x~,

r

~2(y~) + H ( y~, x e - ~ ye

+

_< 0 3Dg(ye)) > 0

so t h a t

ul(x~) - u2(y~) _< H(y~,A(x~ - y~) + p) - H(xE, A(x~ -- YE) + q) with A = l / e , p = --SDg(yE), q =/3Dg(x~). Since IDgl < 1, choosing/~ < 1 allows us to a p p l y (H2). This gives

~l(x~)

--

u2(~)_<~2(l~-~t+

IxE

-Y~f2 E ' R~ /

T h a n k s to (3.12) and (3.13), the preceding gives Ul(XE) - u2(YE) ~ 5 / 3 + ~ 3 ( 2 ~ )

-f- ~

"

56

II. CONTINUOUS VISCOSITY SOLUTIONS O F H-J EQUATIONS

for sufficiently small E > 0; a fortiori, 9 (x~, y~) < 5/3 + ~3(29) 9 Since/3 has been chosen such that w3(2/3) < 5/6 the desired contradiction is obtained. It is easy to generalize Theorem 3.5 to cover the case of a general unbounded open set f~ C ]Ry . Indeed, one can prove that if (H2) holds and ul, us E BC(-~) are, respectively, a viscosity sub- and supersolution of

u(x) + H(x, Du(x)) = O,

x E f~,

with ul _< us on 0fL then Ul ~< u2 in f~. The proof is similar to that of Theorem 3.5; the case where the maximum point (xe, yE) of the auxiliary function belongs to cg(fi x f~) is dealt with as in the proof of Theorem 3.1 (see also Exercise 3.7). REMARK 3.6. A useful variant of Theorem 3.5 is obtained when the boundedness assumption on ul and u2 is replaced by their uniform continuity (which implies that they grow at most linearly at infinity). One can prove indeed t h a t if ul, u2 E UC(R N) are, respectively, viscosity sub- and supersolution of (3.9) with H satisfying (H1) and (H3), then ul <_ us in ]RN. <~ The next comparison result concerns an evolutionary case. As usual it gives a uniqueness result for the Cauchy problem

ut(t,x) + H(t, D u ( t , x ) ) = 0 u(O, X) = UO(X) with initial condition uo E Uc(RN). nians.

(t,x) e ]0, T[ x •N X C R N,

See Remark 3.8 for more general Hamilto-

THEOREM 3.7. Assume H E C([0, T] x ]I~.N). Let ut, u2 E UC([0, T] x I~N) be,

respectively, viscosity sub- and supersolution of ut(t, x) + H(t, Dxu(t, x)) = 0

in ]0, T[ x R N .

TheTt~ sup (Ul -- u2) _< sup(ul(0, 9 ) -- u2(O, 9)). [O,T]xI~N ]~N PROOF. We may assume that A := sup(u1(0, -) - u2(0, -)) < + o c . ~N

Set, for r _> 0 and i = l, 2, wi(r) = s u p { l u i ( t , x ) - ui(s,y)] : I t - sl + I x - Yl < r};

3. SOME COMPARISON AND UNIQUENESS RESULTS

57

by uniform continuity we have

~i(r)

(3.14)

sup < +oz. r_>0 1 + r

A straightforward computation shows t h a t U l(t, X) -- U 2(8, y) ~-- U l ( t , X) -- U 1(0, X) -{- U 1(0, X) -- ?/2(0, X) + ?/2(0, X) -- U2(8, y)

< wl(t) + A + w2(s + Ix - y]), for all (t, x, s, y) E Q2 where Q :-- [0, T] • R N. Hence (3.14) yields

lul(t,x)-uz(s,y)]
(3.15)

v ( t , x , s , y ) e Q 2,

for some C > 0. The proof proceeds by contradiction; we assume therefore that (3.16)

sup(u1 - us) > A + a0 Q

for some a0 > 0. Consider for (~, z > 0 the auxiliary function

62(t,x,s,y) = u l ( t , x ) - - u s ( s , y ) Thanks to (3.15), supQ2 9 < + ~ (to, xo, so, Yo) C QS with

Ix-y?

{t-sl s

so that, for each 0 < 6 < ~o/2, there exists

(I)(to, xo, so, Yo) + 6 > sup (I). Q2 Choose now ~ E C~r

N) such that

<(zo, Yo) = l,

0_<(<1,

ID((x,y)[ < 1.

For 0 < a < ~ro/2T we set

r

~, s, y) = ~(t, z, s, ~) + ~<(x, u) - ~,t.

It is easy to realize t h a t there exists (t, 5, ~, ~) E Q2 such that (t, 5, $, ~) = sup 9 . Q2 The inequality

2~,(~,5,~,~) > ~(~,5,7,5) + r and (3.15) yield after some computations

15-~12 - - +

It-~]~

< c ( i + 15- ~l) + 46 + ~ ( ~ - ~) < c ( 1 + 15 - ~l) + 2~0,

58

II. CONTINUOUS VISCOSITY SOLUTIONS O F H-J EQUATIONS

for 0 < 5 < ao/4 and 0 < a < ao/2T. The above implies that for each ~, a 9 ]0, 1[ there exists constants Ce, C~ > 0 such that (3.17)

]x - Y] _< Ce,

I~ - ~l < Ca.

lim Co = 0 .

If (7, 5), (a, y) 9 ]0, T] x IRN, using the definition of viscosity solution and L e m m a 2.10 we obtain 1 a + 1(7a - -~) + g ( t , -~(~ - ~) - aD~:((&~)) <_ 0 and 1 (7 - a) + H(a, 7 ( ~ - y) + 5Dyr

y)) > 0 .

Therefore, by addition,

o _< H(~,

(5 - ~) + 5Dy~(~, ~)) - H(~, 7(~ - ~) + aDx~(~, ~))

Hence, by (3.17),

(3.18)

-<

+ 5,

+ o0)

where the local modulus OaH is given by

w H ( r , R ) : = s u p { [ H ( t , p ) - H(s,q)[ : [ t - s[ + [ p - q[ < r, v,q 9 B(0, R ) } . Now we choose a and 5 small enough to make the right-hand side of (3.18) strictly less than er and get the desired contradiction. In order to complete the proof we must examine the case where t -- 0 or ~ -- 0. We show t h a t this cannot occur if a is small enough. Assume by contradiction there is a sequence an -+ 0 + such that the corresponding k0 attains the maximum at (tn, xn, y,,,sn) with tn = 0 or sn = 0 for all n. I f t n = 0 , by (3.17) and the definition of A, "al(Xn, 0) -- u2(Yn, Sn) ~_ A + u2(yn, O) - u2(yn,s~) _< A + w 2 ( C a , ) . Similarly, if sn = 0

u l ( x ~ , t n ) - u2(y~,0) _< A + oal(Can) . Therefore sup kO(t,x,t,x) <_ s u p ~ < A + o ( 1 ) + 5

(t,x)EQ

Q2

as n --+ oc. On the other hand, by (3.16), sup ~ ( t , x, t, x) > sup(ul - u2) - a T > A + ao - a T . (t,x)eQ

Q

Then ~*0 < 5 + a T + o(1) as n -+ or which contradicts the choice of 5 and a for n large enough. 9

3. SOME COMPARISONAND UNIQUENESS RESULTS

59

REMARK 3.8. The comparison Theorem 3.7 can be extended to the equation

ut + H(t, x, Du) = 0 if the Hamiltonian H E UC([0, T] x ]~Y • B(0, R)) for every R > 0 and H satisfies (H1) with a modulus wl independent of t E [0, T]. Further extension can be found in Exercises 3.9 and 3.10, also to Hamiltonians depending on u. <~ REMARK 3.9. The comparison principles can be used to obtain bounds in the supremum norm for viscosity solutions of (H J). To show this in a simple setting, let un E B c ( ~ N ) , n E N, be a viscosity solution of (3.19)

u~(x) + Hn(x, Dun(x)) = O,

x E ~N,

where Hn satisfies (H1), (H3) for each n E N. Assume also t h a t sup IH,~(x, 0)1 < C < + ~

zER N

for some constant C, uniformly in n. It is obvious t h a t C and - C are, respectively, a super- and a subsolution of (3.19) for any n 9 1~. Hence, by Theorem 3.5

- C <_ u~(z) < C, for any n 9 N and x 9 ]~N. Several applications of this idea will be considered in subsequent chapters. <J Exercises

3.1. Observe that, if (3.4) does not hold, the proof of Theorem 3.1 shows actimlly that ~x(ul

- u2) + < % ~ x ( u l - us) +

Compare this with Proposition 1.5. Observe also that the result of Theorem 3.1 remains valid if one replaces assumption (H1) with the requirement t h a t either Ul or us is Lipschitz continuous. 3.2. Let H ( x , p ) = SUPaeA { --f(x, a ) . p --g(x, a) }, with A compact f and g continuous. Assume also that, for all x, y,

I f ( x , a ) - f ( y , a ) l <_ LIx - y I,

Ig(x,a) - g(y,a)l <_w(Ix - yl)

where the constant L and the modulus w are independent of a E A. Show t h a t H satisfies assumption (H1) in Theorem 3.1. 3.3. Prove t h a t the result of Theorem 3.1 remains true if assumption (H1) is replaced by the coercivity condition r + I4(x,p)

~

+~

a s Ipl - ~ + ~ ,

60

II. CONTINUOUSVISCOSITY SOLUTIONS OF H-J EQUATIONS

uniformly with respect to x E fL IrI --< R, (VR > 0). inequality

ul(x~) + H

[Hint: observe t h a t the

x~,

in the proof of Theorem 3.1 a n d coercivity imply that ](x~ - y~)/c I is uniformly b o u n d e d as ~ --+ 0+.] 3.4. Take H ( x , p ) = IA(z)pl s - g(x), where A ( x ) = (aij(x)) is a M x N matrix, M < N . Assume t h a t g, aij are b o u n d e d a n d Lipschitz continuous. Observe t h a t ]H(x,p) - H ( y , p ) l " C i x - y] ]pl 2 and consequently (H1) is not satisfied. Observe also that if M = N a n d A ( x ) is invertible at all x E ~, t h e n H satisfies the coercivity condition in Exercise 3.3. Prove t h a t if ul, u2 E C ( ~ ) are viscosity solutions of

u(x) + I A ( x ) D u ( x ) I 2 - g(x) = 0

in fl,

with ~ C_ ]~g open and b o u n d e d , t h e n u l = u2 provided u l = u2 on 0El. 3.5. Let ~ be a b o u n d e d open set with s m o o t h b o u n d a r y 0f~. Assume t h a t

t ~

F(x, r,p + tn(x))

is nondecreasing for each fixed r, p a n d for all x E F C OFt. E x t e n d L e m m a 2.10 to this setting. Generalize t h e n T h e o r e m 3.1 to show t h a t if u a n d v are viscosity sub- and supersolution of (3.3) with u _< v on 0 ~ \ F, t h e n u _< v in ~ . 3.6. Prove the following generalization of T h e o r e m 3.1. Assume t h a t for a n y R > 0 there exists "YR > 0 such t h a t

F ( x , r , p ) - F ( x , s,p) > ~/R(r -- s) for all x E ~, p E ]~N and - R < s < r < R. Assume also t h a t

I F ( x , r , p ) - V ( y , r , p ) l <_ wR(ix - yl(1 + ]p])), with wR(t) ---* 0 as t --* 0 +, for all x, y E ~ a n d Irl < R. Show t h a t if ul, u2 E C ( ~ ) are, respectively, viscosity sub- a n d supersolution of

V(x,u(x),Du(x))

= O,

x E ~2,

then, max(u1 - us) + < n ~ x ( u l - us) + . D

3.7. Assume Ul, u2 E B C ( ~ ) are, respectively, a sub- a n d a supersolution of

k(x)u(x) + H(x, Du(x)) = 0

in f~,

such that ul < u2 on Of~, where Yt _C ]RN is open, H satisfies (H2), and k E C ( ~ ) , A(x) > 0 for all x E 9L Prove t h a t Ul < u2 in f~.

4. LIPSCHITZ CONTINUITY AND SEMICONCAVITY

61

3.8. The one-dimensional equation 1 l u , ( x ) l 2 = 0,

x ~

~,

has the classical solutions Ul = 0 and u2(x) = - 8 9 xo) 2 (for any fixed x0). Prove t h a t ul is the unique viscosity solution in B C ( R ) and in UC(~). 3.9. Under the assumptions of Remark 3.8, namely, H E UC([0, T] x R g • B(0, R)) V R > 0 and, for a modulus wl,

IH(t, x , p ) - H(t, Y,P)I <- Wl(lX - y l ( 1 + ]Pl)) for all t, x, y,p, consider solutions ui E UC([0, T] x ~ N ) of

ut + H ( t , x, D u ) = f~(t, x)

in ]0, T[ x ~N,

where fi E UC([O,T] x ]~N), i = 1,2. Prove that sup(ul(t, .) -- u2(t, ")) < sup(ul(O, .) -- u2(O, ")) + RN

]~N

/o

s u p ( f l ( s , ") -- f2(s, "))ds NN

for all t E [0, T]. 3.10. Prove comparison results for solutions in B U C and in UC of It t + H ( t , u, D~t) = 0

in ]0, r [ x ]~N,

with H continuous such t h a t r ~ H ( t , r,p) is nondecreasing. Then extend it to the case where r H H ( t , r , p ) + Ar is nondecreasing for some A > 0. [Hint: make the change of variables v --= e-;~tu.] 3.11. Under the assumptions of Theorem 3.7 let ul, u2 be, respectively, solutions of

ut + A u + H ( t , D u ) = e - t f i ( t , x )

in]O,+oc[•

N,

for i = 1,2, where A > 0 and f~ E UC([O,T] x ~ N ) VT > 0. Prove that, if ul(0, 9 ) - u2(0, 9 ) and f l - ]'2 are bounded, then supRN lul(t, 9 ) -- u2(t, 9 )I decays exponentially to 0 as t --* +~c. [Hint: make the change of variables v --= e-)'tu and use Exercise 3.9.]

4. Lipschitz continuity and semiconcavity In the first part of this section we present two results (Propositions 4.1 and 4.2) which show that, under suitable assumptions on H, viscosity solutions of (HJ)

Au(x) + H ( x , D u ( x ) ) = O,

x E RN

are actually Lipschitz continuous. The section includes a quick overview of some differentiability properties of Lipschitz functions. Section 4.2 is devoted to the basic theory of semiconcave functions and the related topic of inf-convolution. The main results in this direction are Theorem 4.9 which asserts that, under suitable conditions on H, Lipschitz continuous viscosity solutions of" (H J) are semiconcave and Proposition 4.13, showing that a viscosity subsolution of (H J) can be uniformly approximated from below by a semiconcave subsolution uE of an approximate equation.

62

II. CONTINUOUS VISCOSITY SOLUTIONS OF H-J EQUATIONS

4.1. L i p s c h i t z c o n t i n u i t y We assume for the next result that H satisfies the coercivity condition (H4)

H ( x , p ) ----+ +~c

as IPl-+ +oc,

uniformly with respect to x. When H is given by (4.1)

H ( x , p ) = sup{ - f ( x , a ) . p

- g(x,a) },

aEA

sufficient conditions for (H4) to hold are the boundedness of g together with the controllability assumption (4.2)

3r > 0 : B(0, r) c_ e-6f(x,A),

g x e RY,

(see Exercise 4.1). PROPOSITION 4.1. Let us assume condition (H4). Then any viscosity subsolution u E BC(]R N) of (H J) is Lipschitz continuous. PROOF. For fixed x E ]]~N consider the function ~(y) = u(y) - C l y - xl where C > 0 is a constant to be chosen later. The boundedness of u implies the existence of ~ E R g such that

~(~) = ~

~(~)

The claim is that ~ = x for C large. If not we would have -< 0,

+

since u is a viscosity subsolution of (HJ) and y H C lY - x[ is differentiable at y = ~ ~ x. For sufficiently large C, independent of x, (4.3) is in contradiction to (H4). Therefore, for such C, u ( y ) - c ly - xl < u ( ~ ) - c I~ - xl = ~ ( x ) ,

vy e R N

By interchanging the roles of x, y, the proof is complete. A different condition on H guaranteeing the Lipschitz continuity of viscosity solutions of (H J) is

3c>0: (x,C For H given by (4.1), condition (Hs) holds with C > M / ( 1 - L), provided that f, g satisfy ( f ( x , a ) - f ( y , a ) ) . ( x - y) <_ n i x [e(x,a)-e(y,a)l

(see Exercise 4.2).

y[2,

< MIx-yl,

L < 1,

Vx, y,a

4. LIPSCHITZ CONTINUITY AND SEMICONCAVITY

PROPOSITION 4.2. Let us assume (HI), (H3), a viscosity solution of (H J). Then lu(x) - u(y)l

(H5), ;~ _> 1

< cq~-

63

and let u E UC(]~N) be

yl 9

PROOF. It is easy to check that v(x, y) = u(x) - u(y) is a viscosity solution of

Av(x, y) + ~I(x, y, Dxv(X, y), Dyv(X, y)) = 0 where H(x, y,p, q) := H ( x , p ) - H ( y , - q ) . satisfies

in ~ 2 N

On the other hand, w(x, y) := C Ix - Y l

),w(x, y) + .~(x, y, D~w(x, y), Duw(x, y)) - g(x, y) = 0

in ]R2N,

where

x-y

x-y

Since g > 0 by (H5), v, w E UC(]~2N) and H satisfies (H1), (Ha), as it is easy to check, the comparison result in Remark 3.6 gives v < w in R 2N, which proves the statement. Let us recall briefly now some relevant differential properties of locally Lipschitz continuous functions. By the classical Rademacher theorem (see [EG92]) such functions are differentiable almost everywhere with locally bounded gradient. Hence, if u E Liploc(~ ), the set

n * u ( x ) = { p E ~N : p = n~+or lim nu(x,~), x~ ---* x } is nonempty and closed for any x E f~. Denote by co D*u(x) its convex hull. It is a well-known result in nonsmooth analysis (see [Cla83]) that

coD*u(x) = cgu(x),

(4.4)

Vx efL

where Ou(x) is the generalized gradient or Clarke's gradient of u at x, defined by CgU(X) : :

{ p E ~Y

: uO(x; q) > p . q, V q E ~ Y }

={pERN:uo(x;q)<_p.q,

VqE~N}.

Here u~ q) and u0(x; q) axe the generalized (or regularized) directional derivatives defined by u~

q) := limsup

y~x

u(y + tq) - u(y) t

t~O +

uo(x; q) := liminf u(y + tq) - u(y)

y~x

t~O +

t

64

II. CONTINUOUS VISCOSITY SOLUTIONS OF H-J EQUATIONS

A related notion is that of Dini directional derivatives, namely,

O+u(x; q) := lira sup t~0 §

u(x + tq) - u(x) t

O-u(x; q) := liminf u(x + tq) - u(x) t ~o+ t From the definitions it follows easily that (4.5)

uo(x;q)<_O-u(x;q)<_O+u(x;q)
q E N N,

VxEQ,

and this implies that for u E Liploc(f~ ),

D - u ( x ) U D+u(x) c_ Ou(x),

(4.6)

Vx Eft

(see Exercise 4.4). Observe also that D+u(x), D - u ( x ) are bounded sets (see Exercise 1.12). The next result is about the existence of classical (one-sided) directional derivatives of locally Lipschitz continuous flmctions, namely ~(x)

Ou(x; q)

:. . ~

lim u(x + tq) - u(x) t~O

+

t

Iql '

1 "

PROPOSITION 4.3. Let u E Liploc(ft). Then, for all q with Iq[ = 1, there exists

OU(x) = min p . q = u0(x;q) Oq pEO+u(x)

(4.7)

at any x E f t where D+u(x) = Ou(x). PROOF. Let p E D+u(x) and Iql = 1. Then

u(x + tq) - u(x) - tp . q < o(Itl), for sufficiently small t. Hence P'q>

u(x + tq) - u(x) t

o(t) t '

for s m a l l t > O .

Consequently, inf p . q >_ cO+u(x; q) . peD+u(x) Combining this with (4.5) we get u0(x; q) <_ O-u(x; q) <_ O+u(x; q) <

inf p ' q . pED+u(x)

It is not hard to show (see Exercise 4.4) that u0(x;q) =

rain p . q ; p~Ou(x)

hence (4.7) holds if x is such that D+u(x) = Ou(x).

4

4. LIPSCHITZ C O N T I N U I T Y AND SEMICONCAVITY

65

The preceding proposition allows us to prove a useful variant to Proposition 2.13 on semidifferential and directional derivatives of a marginal function u defined by

u(x) :-- inf g(x, b) . bEB

PROPOSITION 4.4. Let us assume B compact, g continuous on f i x B, differentiable with respect to x with Dxg continuous on ~ x B. Then u E LiPloc(fl ), D+u(x) = On(x) for all x and the same conclusions of Proposition 2.13 hold. PROOF. By the assumptions made, x ~ g(x, b) is locally Lipschitz, uniformly in b E B. Then it is easy to conclude that u E Liploc(gl). Next we show that D+u(x) = K6Y(x), where

Y ( x ) = { Dxg(x,b) : b E M ( x ) }

and

M ( x ) = {b E B : u(x) = g(x,b) } .

By Lemma 2.11, it is enough to prove that D+u(x) C K6Y(x); in view of (4.4) and (4.6) this amounts to show that D*u(x) C Y ( x ) . Let p E D*u(x) and take xn --~ x such that Du(xn) -~ p. Then pick b~ E M ( x ~ ) and suppose, without loss of generality, that bn --~ b E B. Since g(xn, bn) <_ g(xn, b) for all n E N and b E B, we conclude by continuity that b E M ( x ) . By differentiability of u at x~ and Lemma 2.11, we have Du(x,~) = D~g(xn, b,). Hence, letting n --+ +oc, we get p = D~g(x,b); that is, p E Y ( x ) . The existence of directional derivatives and the formula (4.8)

On(x) = min y . q , Oq yEY(z)

Vq with Iql = 1,

follow directly from Proposition 4.3. It remains to prove that D - u ( x ) = {y}, whenever Y ( x ) reduces to the singleton {y}. Observe that in this case (4.8) implies On(x; q) = y . q for all q, and this ensures y E D - u ( x ) (this is not hard to check, see Lemma III.2.37 for a more general result). The opposite inclusion follows from Lemma 2.1 I. 9 4.2. S e m i c o n c a v i t y We say that u : ~ --* ~ is semiconcave on the open convex set ~ if there exists a constant C > 0 such that (4.9)

#u(x) + (1 - #)u(y) < u(l~x + (1 - p)y) + ~C#(1 -

~)lx -

y[2

for all x, y E i~ and # E [0, 1]. This amounts to the concavity of x ~ u(x) - 1C[x[2, as it is easy to check. If u is continuous, an equivalent way to express condition (4.9) is to require that (4.10)

u(x + h) - 2u(x) + u ( x - h) < Clhl ~,

for any x E l-I and h E R N with sufficiently small Ihl (see Exercise 4.5). Concave functions are, of course, semiconcave. A less trivial class of semiconcave functions is that of C 1 functions with locally Lipschitz gradient.

66

If. CONTINUOUS VISCOSITY SOLUTIONS OF H-J EQUATIONS

A relevant class of nondifferentiable semiconcave functions is that of marginal functions u(x) = infbeB g(x, b), provided x H g(x, b) satisfies (4.9) uniformly in b (see Exercise 4.6). Consequently, value functions of optimal control problems are semiconcave under suitable conditions on the data (see later chapters for developments in this direction). An immediate application of this remark gives the following examples. EXAMPLE 4.5. Let S C ]~g, S ~ 0,

d(x) := dist(x, S) = inf Ix - s I . sES

Then d 2 is semiconcave in ]~N because x H Ix - sl 2 is C ~ with constant second derivatives. On the other hand, d itself is semiconcave on any compact set having positive distance from S, because x ~-~ Ix - s] has bounded second derivatives in such a set, uniformly for s in a bounded set.
u(x + h) - u(x) = ~ ( x + h) - ~(x) + C x . h + ~ Ihl2; where r = u(x) - ~1 C i x I is concave and, consequently, locally Lipschitz continuous in ~. Hence the statement. 9 We know already from w that D + u ( x ) c_ Ou(x) = c o D * u ( x ) for any u e Liploc(~ ). If in addition u is semiconcave, it turns out that D + u ( x ) = Ou(x). This and other differential properties of semiconcave functions are collected in the next statement. PROPOSITION 4.7. Let u be semiconcave in ~. Then for all x E (a)

D + u ( x ) = Ou(x) = c o D * u ( x ) ;

(b)

either D - u ( x ) = 0 or u is differentiable at x;

(c)

if D + u ( x ) is a singleton, then u is differentiable at x;

(d)

Ou ( x ) = peD+~(x)min p ' q , f o r all unit vectors q. -~q

PROOF. Let us start by showing that

(4.11)

C u ( z + h) - u(x) - p . h _< ~ Ihl 2,

for all p E co D * u ( x ) and sufficiently small ]hi.

4. L I P S C H I T Z C O N T I N U I T Y AND SEMICONCAVITY

67

A s s u m e first t h a t u is differentiable at x. Since

u(x+h)=

lim u ( x + ( 1 - p ) h ) #~0 +

and

- D u ( x ) . h = D u ( x ) . ( - h ) = lim u ( x - #h) - u(x) tt-.-.O +

=

#

lim (1 - #) u ( x - tth) - u ( x ) t~--~0+

]~

we have (4.12)

u(x + h) - u(x) - D u ( x ) . h I

=

lim l [ p u ( x + ( 1 - #)h) + (1 - # ) u ( x - #h) - u ( x ) ] . p~0 + #

Now let us apply (4.9) with x + (1 - # ) h a n d x - # h playing the role of x a n d y, respectively, to o b t a i n C

# u ( x + (1 - # ) h ) + (1 - tt)u(x - #h) - u(x) <_ -~ #(1 - #)]hi 2 . T a k i n g (4.12) into account we get C

u(x + h) - u(x) - D u ( x ) . h <_ ~ [hi2,

(4.13)

which proves (4.11) in the present case. Let us now take a n a r b i t r a r y x E fl a n d p E D*u(x). Next pick xn --+ x such t h a t Du(x,~) ~ p. By (4.13), C

u(xn + h) - u(xn) - Du(xn) . h <_ F Ih[~ a n d (4.11) follows after letting n --~ +oo. The validity of (4.11) for a r b i t r a r y p E co D*u(x) follows b y a convexity a r g u m e n t . I n view of (4.4) a n d (4.6), the proof of (a) is reduced to show t h a t co D*u(x) c D+u(x). T h i s is a n i m m e d i a t e consequence of (4.11). Since D*u(x) • 0, (a) implies D+u(x) 7~ ~. Hence, if D - u ( x ) ~ 0, t h e n u is differentiable at x (see L e m m a 1.8) a n d (b) is proved. Observe now that, in view of (a), the one-sided e s t i m a t e (4.11) holds t r u e for all p E D+u(x). I n order to prove (c), let D+u(x) = {p0} a n d choose x~ --~ x such that (4.14)

lim

u(x'~)-u(x)-P~176

68

If. CONTINUOUS VISCOSITY SOLUTIONS OF H-J EQUATIONS

On the other hand, using (4.11) at x =

(4.15)

Xn

with

h = x - Xn,

C

u(x~)-~(x)-po.(x,~-:~)>--~lx,~-xl~-Ip,-pollx,~-~:l,

for all Pn E D+u(x,~). Since D + u ( x n ) is bounded uniformly in n (see Exercise 1.12), we can assume (passing to a subsequenee if necessary) that p~ -* ~ as n --~ +r Estimate (4.11) gives C ~(xn + h) - ~(x~) - p ~ . h < ~ Ihl 2 and therefore, by continuity, u(x+ h)- u(x)-~,

C h < ~ Ihl: .

This shows that l i m , ~ + ~ p n = ~ E D + u ( x ) . Since we assumed D + u ( x ) = {P0}, we conclude that ~ = P0 and the whole sequence {Pn} converges to p0. From (4.14), (4.15) it then follows lim inf u(y) - u(x) - Po " (Y - z) >_ 0;

~x

ly- xl

thus P0 E D - u ( x ) . Therefore (c) is proved, taking Lemma 1.8 into account. Finally, statement (d) is an immediate consequence of Proposition 4.3 in view of (a). 4 As a simple consequence of the previous Proposition 4.7 we get a partial converse to statement (b) in Proposition 1.9 (see also w for further results in this direction). PROPOSITION 4.8. Let u be semiconcave and such that (4.16)

F ( x , u(x), D u ( x ) ) > 0

a.e. in a ,

where F is continuous. Then u is a viscosity supersolution of

(4.17)

F ( x , u(x), D u ( x ) ) = 0

in a .

PROOF. By Proposition 4.7 (b), at any x E D either D-u(x) = 0

or

D-u(x) = D+u(x) = {Du(x)} .

In the first case the supersolution condition is automatically satisfied. Assume therefore that x is a point of differentiability of u. Then, there exists a sequence xn ---* x such t h a t u is differentiable at xn and (4.18)

F ( x , ~ , u ( x n ) , D u ( x n ) ) >_ 0

Vn

4. LIPSCHITZ CONTINUITY AND SEMICONCAVITY

69

(one can take x~ = x, provided (4.16) holds at x). Since u is locally Lipschitz continuous, by definition of D*u(x) we have

Du(xn)

~p E D*u(x)

as n ~ +ec,

at least for a subsequence. In the present case,

D*u(x) = D+u(x) = D - u ( x ) = {Du(x)}, so that, letting n --* + o c in (4.18), we get

F ( x , u(x),p) >_ 0

Vp E D - u ( x )

and the statement is proved. The preceding proof satisfies F ( x , u ( x ) , p ) = spirit is Theorem 5.5 in The next result is on

4

shows that a semiconcave generalized solution of (4.17) 0 for all x E fl and p E D - u ( x ) . A deeper result in this the following section. the semiconcavity of viscosity solutions of (H J),

THEOREM 4.9. Let u E BC(N N) A Lip(N N) be a viscosity solution of

u(x) + H(x, D u ( x ) ) = O,

(H J)

x E R N,

with Lipschitz constant Lu. Assume that H satisfies IH(x,p) - H ( x , q ) I <_w(Ip-ql),

(Ha)

Vx, p,q E R N,

and that, for some C > 0 and L ~ > 2Lu, (H6)

H(x+h,p+Ch)-2H(x,p)+H(x-h,p-Ch)>_-CIhl

2

holds for all x, h E NN, p E B(O, L'). Then u is semiconcave on R N. PROOF. Consider the function v : II~3N ---*N defined by v(x, y, z) = u(x) - 2u(y) + u(z). It is easy to check t h a t

(p, q, r) E D+v(x, y, z) if and only if

p E D+u(x),

q --~ E D - u ( y ) ,

r ~ D+u(z).

Since u is a viscosity solution of (H J), then

~(x) + H(x, p) < 0

V p ~ D+u(x),

u(z) + H ( z , r ) < 0

V q E D-u(y) .

Hence, v is a viscosity solution of (4.19)

v(x, y, z) + ~I(x, y, z, Dxv, Dyv, Dzv) = O,

(x, y, z) E

~3N

70

II. CONTINUOUS VISCOSITY SOLUTIONS OF H-J EQUATIONS

where ~r(x, y, z, p, q, r) = H ( x , p) - 2 H ( y : - q / 2 ) Consider now the auxiliary function

+ H ( z , r).

C Ix - yl 2 - ~C Iv - zl 2 - 5(1 + Iy12)l/z ~ ( x , y, z ) = v( x, y, z) - ;I }x - 2y + z[ 2 - F

where e, 6 are positive p a r a m e t e r s and C is any fixed constant satisfying (H6). Prom the boundedness of u it follows t h a t 9 a t t a i n s a m a x i m u m o v e r ]~3N at some point (xo, Yo, zo) depending on e, 5. Now a c o m p u t a t i o n shows t h a t D x ( v - ~ ) ( x o , Yo, zo) = Po + C ( x o - Yo) D y ( v - ~ ) ( x o , yo, zo) = - 2 p 0 + 6 ( y o ) - l y o - C ( x o - yo) + C ( y o - zo) D ~ ( v - 4))(xo, yo, zo) = po - C ( y o - zo),

where we set (y) = (1 + ]y]2)a/2 and Po = 2(xo - 2yo + z0). Since v is a subsolution of (4.19) we o b t a i n (4.20)

v(xo, yo, zo) < - g ( x o , P o

+ C ( x o - yo)) 16 -1 1 + 2 H ( y o , P o - ~ (Yo> Yo + -~C(xo - 2yo + zo)) - H ( z o , P o + C ( z o - Y o ) ) .

The inequality O(y + h, y, y - h) _< ~5(xo, yo, zo), which holds for any y, h E IRN, gives (4.21)

u ( y § h ) - 2u(y) § u ( y - h ) - C Ih] ~ - 6(y> < ~,

where we set :

v(xo, yo~. zo) -

C

C

~- Ixo - yol 2 - ~

lyo - zol 2 - 6.

By assumption (H3), H(yo,po

-

89

+ lC(xo

- 2vo + zo))

<_ H ( y o , P o + 8 9

- 2yo + zo)) + cv(6/2)

since (yo)-l[yo} < 1. Therefore, taking (4.20) into account, ~ is e s t i m a t e d from above as (4.22)

~ _< - H ( x o , P o

+ C ( x o - Yo)) + 2H(yo,Po + 89 + 2~(6/2)

- 2yo + zo)) - H ( z o , P o + C ( z o - Yo))

C C - 6 - F Ixo - ~012 - -5 Ivo - zol 2 9

Let us proceed now to e s t i m a t e various quantities using the Lipschitz continuity of u. The inequality #P(0: 0, 0) < ~(xo, yo, zo) yields C

C

- 5 < L u [ x o - yo] + r u l y o - zol - -~ )xo - yo} z - -~ ]Yo - zol 2 - 5(yo)

4. L I P S C H I T Z C O N T I N U I T Y A N D S E M I C O N C A V I T Y

71

and this implies that (4.23)

Ixo - YO] + lYo - zol + 3 _< R

for some R > 0. On the other hand, @(xo, 89

+ zo), zo) <_ 6P(xo, yo, zo) gives

1 Ix0 - 2y0 + zol 2 <_ 2[u(l(xo + z0)) - ~(y0)] + ~ - ~, E

so that (4.24)

~lxo-2yo+

zol2 < _ L ~ l x o - 2 y o +

zo]+51 89

zo-2yo)l .

Here we applied the mean value theorem to ~;(t) = (1 + t2) 1/2 and the fact that ]~b'[ _< 1. Consequently, (4.25)

IpoI < 2L~ + 5 < L'

for 0 < 5 < L' - 2 L ~ .

We send now ~ ~ 0 + for fixed 6 as above; by (4.23), (4.24), (4.25), at least for a subsequenee,

(4.26)

x0-Yo ~h, YO-- zo

~

h t,

Yo

'Y,

Po

~ P,

for some h, h', ~ E/3(0, R), p E/~(0, L'). Also, from (4.24), x 0 - 2y0 + z0

~0

so that h = h / = h. Hence, from (4.22), _< - H ( y + h,~ + Ch) + 2H(y,p) - H ( y - fi, p - Cfi) - C f ] 2 + 2~(5/2) - 5(~). By (H6) then, limsup~ < 2~(5/2) - 6(y); ~-~0 +

using this in (4.21), the claim follows after letting 5 --* 0 +. REMARK 4.10. If H is convex with respect to p, then (H6) is satisfied under suitable assumptions on the x-dependence. Consider the simplest case H ( x , p ) = H(p) - g(x), with H convex. Then, as it is easy to check, (H6) holds if g semiconcave (see Exercise 4.8). A more general case is H of the form H(x,p) = sup{-f(x,a) acA

. p-

g(x,a) } .

Assume that, for some L, M, F _> 0 the following hold ( f ( x + h , a ) - f ( x , a ) ) , h < LIh] 2 I f ( x + h,a) - 2 / ( x , a ) + f ( x - h,a)] <_ FIhl 2 g(x + h,a) - 2g(x,a) + g(x - h,a) < MIhl ~,

for all x, h E R N, a E A. An easy computation shows that if L < 1/2, then (H6) is satisfied with C > (21I + 2FL~,)/(1 - 2L) (see Exercise 4.9). <1

72

II. C O N T I N U O U S VISCOSITY SOLUTIONS OF H-J EQUATIONS

A convenient way to build semiconcave approximations of a given function is provided by the method of inf-convolution, a standard tool in convex and nonsmooth analysis. Let fl be a subset of R N and u a bounded flmction in ~. Define, for any ~ > 0,

1

(4.27)

uE(x) := inf{ u ( y ) + ~

]x - yl2: y E f~ } .

The function us is called the e inf-convolution of u. Similarly, 1

(4.28)

uS(x) := sup{ u(y) - K Ix - yl2: y e

~}

is the z sup-convolution of u. LEMMA 4.11. Let u be c o n t i n u o u s and bounded in f L T h e n (a)

us and - u s are s e m i c o n c a v e in f~;

(b)

us/2

(c)

the inf and the sup in (4.27) and (4.28) are attained if~ < d2(x, Of})/(411ulloc ).

u, u s ~ u, as ~ --+ 0 +, locally u n i f o r m l y in f~;

PROOF. Let us observe that -- - x " y-t-

~

ly12 : Y E f~

.

Hence us(x)-

[xl 2 = inf

{ u ( y ) - -1 x . y + 1 tyl2 : y E f} }

.

Since

1

x,

, u(y)-

- x .y +

1,

E

y[2

is affine for any fixed y E ft, it turns out that u s ( x ) - ~71x] 1 2 is concave, so that us is semiconcave. d2(x, 0 ~ ) In order to prove (c) observe t h a t if ~ < 411un~ then, B=B(x,2vGnul}~2)

Cf},

Vxef~.

Observe also that

u(y) + ~1 Ix - yl 2 _> -Ilurl~ + 211uI1~r

Vy ~ ~ -., ~

Since u~(x) < Ilull~, the above implies 1

u e ( x ) = m i n { u ( y ) + ~-~elx- y[2 y e B } .

4. L I P S C H I T Z C O N T I N U I T Y A N D S E h I I C O N C A V I T Y

73

In order to prove (b), let us observe t h a t 0 < E~ < z implies

By (c), there exists y~ 9 f~ such t h a t

~(z)

1

= ~(y~) + ~ l x - ~12

This implies YE --* x as E ---* 0. Since u is continuous,

]im u~(x) > lira u(y~) = ~(x); E~0 +

r

+

therefore

u~(x) /

u(x),

as ~ -~ 0,

for any fixed x E Ft. T h e local u n i f o r m convergence now follows from Dini's Lemma. T h e proofs for u E are completely similar. 4 T h e next l e m m a collects some useful facts a b o u t u~ a n d u ~. These are simple consequences of results on m a r g i n a l functions. By (c) of L e m m a 4.11 we can set, for ~ small,

M~(x) : = arg rain{ u(y) + Ix - yl2/2r }, yEfl

M~(z) := argma~{ ~(~) - Ix - y12/2~ yEft

}.

LEMMA 4.12. Let u E C(fi) be bounded, x EFt and ~ < d2(x, Of~)/(411ul[r ). Then, either D - u r = 0 or D - u E ( x ) = {(x -- y~)/z}, where {YE} = ]tI~(x) (respectively,

either D+u~(x) = 0 or D+u~(x) = { - ( x - y~)/~} where {y~} = ~I~(x)) Moreover, for any y~ 9 M~(x) (respectively, M~(x)),

(ii)

Ix - yE[2/~ ---* 0 as r ---* 0 +, uniformly on compact subsets of Ft;

(iii)

(x - y~)/r 9 D - u ( y E ) (respectively, --(x - YE)/~ 9 D+u(Y~)) .

PROOF. Let us detail only the proof for u~, the a r g u m e n t for u ~ being completely similar. By L a m i n a 4.11 (c)

u~(x)=min{u(y)+ yc B

1 ix_yl2 } =u(yE)+

~E

1/2

where J~ : t3(x, 2,/'/}lullo~ ). The function -

-

g(x, V) = u(v) + 1 I x - yl ~

74

II. CONTINUOUS VISCOSITYSOLUTIONSOF H-J EQUATIONS

satisfies on /J the assumptions of Proposition 4.4. Since D x g ( x , y,) = (x - y , ) / r the first part of the statement follows. Since ]lfE(x) C_ JB, we get immediately (i). If y, 9 _Al~(x), from the inequality !

2s

Ix - y~l ~ = u , ( ~ )

- ~(y~) < u(x) - u(y,),

and the continuity of u, we get (ii). To prove (iii) it is enough to observe t h a t 1 0 9 D ~ g ( x , y,) = D - u ( y ~ ) - -~(x - y,), because g(x, 9 ) attains a minimum at y~.

9

Lemmas 4.11 and 4.12 suggest that a continuous viscosity solution of (HJ) has a two-sided uniform approximation by locally Lipschitz continuous viscosity solutions of approximated equations. More precisely, we have PROPOSITION 4.13. A s s u m e that H satisfies (HI)

[ H ( x , p ) - H ( y , p ) [ <_ o31([x

-

-

yl(1 + [Pl)),

f o r x , y 9 ~, p 9 ~N, where Wl is a modulus. I f u 9 C ( ~ ) is a viscosity subsolution of (H J) in ~, then u C 9 Lipzoc(gt) is a viscosity subsolution of

(HJ~)

AuE(x) + H ( x , D u e ( x ) ) = #e(x)

in s

where

and #E(x) --* 0 as ~ --, O, u n i f o r m l y on compact subsets o i l .

PROOF. By Lemma 4.12 it is enough to check that :,u~(x) + H ( x , - ( x

- y~)/r

< ~(x)

in Q~,

where ys is such that uS(x) = u(ys) - Ix - ysl2/2s. Now by Lemma 4.12 (i), ys 9 ~ for x 9 s s. Hence (H J) gives Au(yE) + H ( y E , - - ( x -- y~)/E) <_ 0 by Lemma 4.12 (iii). Therefore we use u~(x) < u(ys) and (H1) to get .ku~(x) + H ( x , - ( x

- yE)/E) <_ Wl(IX -- ys[(1 + IX -- Y~I/~)),

and the statement follows on the account of Lemma 4.12 (i), (ii).

9

REMARK 4.14. A similar result holds of course for a supersolution u and its infconvolution. <1

4. LIPSCHITZ C O N T I N U I T Y AND SEMICONCAVITY

75

Exercises 4.1. Take H as in (4.1) with f continuous holds. Show t h a t (i)

sup(-f(x,a).p}= aEA

on

sup

]I~N X A. Assume also that (4.2)

(-~.p}:>rlp

I.

~eU6f(x,A)

Prove then t h a t H satisfies (H4) provided ~ is bounded. 4.2. Take H as in (4.1) with f , g such t h a t ( f ( x , a) - f ( y , a)) . ( z - y) < L I x - ~l 2

l e ( x , a ) - ~(y,a)] _< ~ i l x - Yt,

v~;y

with constants L, M independent of a E A. Show that for any C > 0:

4.3. Let u : ~t ~ R be locally Lipschitz continuous in the open set ~. Prove the following statements: (a) D+u(x) and D - u ( x ) are bounded for all x C ~; (b) u0(x; q) = - u ~

for all x e ~, q e RN;

(c) for any x E ~, q --* u~ q) is finite, positively homogeneous, subadditive, convex (and locally Lipschitz continuous); (d) (x, q) --~ u~

q) is upper semicontinuous.

4.4. For u as in Exercise 4.3 show t h a t

D+u(x) c_ Ou(x),

D - u ( x ) C Ou(z) .

[Hint: use inequality (4.5) and choose y = x § tq with t > 0 in the definition of Prove also t h a t

D•

u0(x; q) = peo~(x)minp - q;

u~

max p 9q . q) =- peo~(x)

4.5. Prove t h a t the two definitions (4.9) and (4.10) of semiconcavity are equivalent for a continuous function. 4.6. Show t h a t the marginal function u(x) = infbe B g(x, b) is semiconcave provided t h a t for each b E B, x ~-* g(x, b) satisfies (4.9) with a constant independent of b E B. [Hint: for any E > 0 choose bE E B such that u ( , x + (1 - p)y) > g(~x + (1 - ~)y, bE) -and observe that

#u(x) + (1 - #)u(y) < #g(x, bE) + (1 -- #)g(y, b~) .]

76

II. CONTINUOUSVISCOSITY SOLUTIONS OF H-J EQUATIONS

4.7. Let u be locally Lipschitz continuous in ~t. Show t h a t for all x, y E f~ with Ix - Yl sufficiently small t h e n ]P - ql -< L + diam D + u ( y ) ,

Vp E D+u(y), q E D+u(x)

where L is a (local) Lipschitz c o n s t a n t for u. [Hint: use inequality (1.6) in the proof of L e m m a 1.8 to get Ip-ql+liminf yxE~ -x

u(y)-u(x)-p'(y-x) ly -

51


VqED+u(x).]

-

4.8. Let H : ] ~ N _.~ ]I~ be convex. Show t h a t H ( p + Ch) - 2H(p) + H ( p - Ch) > 0 for all p, h E ]~g a n d C > 0. Deduce from this that F ( x , p ) : = H ( p ) - g(x), with g semiconcave, satisfies (H6). 4.9. Let H be as in R e m a r k 4.10 a n d set X(C)=H(x+h,p+Ch)-2H(x,p)+H(x-h,p-Ch).

C>_O.

Prove t h a t for any E > 0 there exists ae E A such t h a t X ( C ) > -X1]Pl - C X ~ 9 h - X3 where X l = I f ( x + h, a~) - 2 f ( x , hE) + f ( x - h, aE)l X2 = f ( x + h, hE) - f ( x - h, a~) X3 = [g(x + h, a~) - 2g(x, at) + e(x - h, at)] 9 Deduce t h a t if L < 1/2 t h e n (H6) holds for any C > (2FLu + M ) / ( 1 - 2L). 4.10. Let f ( x , a ) = E x + Ba, g(x,a) = 8 9 x + 89 a, where E , B , M , Q are matrices of appropriate dimensions. Prove t h a t f , g satisfy the conditions in R e m a r k 4.10 with L = IIE]I, F = 0, M = IIMI[. 4.11. Let u be semiconcave in ~, p E C 1 ( ~ ) a n d x0 E ~ a local m i n i m u m for u - ~. Show t h a t u is differentiable at x0 with D u ( x o ) = D ~ ( x o ) . [Hint: semiconcavity of u yields u(xo § h) - u(xo) < u(xo) - u(xo - h) + CihI2.] 4.12. Let u be Lipschitz continuous on ~ N . Show t h a t sup Iu - uEI ~ Cz, where u~ is the inf-convolution of u.

5. SPECIAL RESULTS FOR CONVEX HAMILTONIANS

77

5. S p e c i a l r e s u l t s f o r c o n v e x H a m i l t o n i a n s The convexity of the function p ~ F(x, r,p) allows us to prove some additional results in the theory of viscosity solutions of Hamilton-Jacobi equations. On the other hand, the Hamiltonians associated with optimal control problems, always satisfy the convexity property (see the various examples in Chapter III and IV). For the sake of simplicity we shall consider only equations of the form

Au(x) + H(x, Du(x)) = 0

(H J) where A > 0 and H satisfies (HT)

, H(x,p) is convex for any fixed x .

p~

In the first subsection and in w we study the relations among viscosity solutions and other generalized notions of solution, namely semiconcave generalized solutions, bilateral supersolutions, and solutions in the extended sense of Clarke. The main result is Theorem 5.6 asserting that, when (HT) holds, viscosity solutions of (HJ) are characterized by the single condition Au(x) + H(x,p) = 0 for any p E D-u(x); this is the starting point of the theory of lower semicontinuous viscosity solutions that we develop in w Different issues where (HT) plays a role are regularity, treated in w and comparison for the "degenerate" case A = 0, see w The last two subsections are devoted to monotonicity properties along trajectories of viscosity subsolutions of the Hamilton-Jacobi-Bellman equation (HJB)

Au(x) + sup{ - f ( x , a). Du(x) - g(x, a)} = 0

in

aEA

with A _> 0 (see Theorem 5.21). This is related to the method of characteristics and to some different characterizations of the value function. 5.1. S e m i c o n c a v e g e n e r a l i z e d s o l u t i o n s a n d b i l a t e r a l s u p e r s o l u t i o n s

Let us go back first to the question about the relationship between generalized and viscosity solutions of (H J). PROPOSITION 5.1. Let u E Liploc(12 ). If

Au(x) + H(x, Du(x)) <_0

a.e. in f~

and (HT) holds, then u is a viscosity subsolution of (H J) in f~. PROOF. Let Q E C~(~N), Q >_ O, with supp ~ C B(0, 1) and f~N O(Y) dy = 1 be a standard regularizing kernel and set

~(x) =

~-;e

,

~ > o.

78

II. CONTINUOUS VISCOSITY SOLUTIONS OF H-J EQUATIONS

It is well-known that for each f~J CC f~,

u~(x) := (u 9 Qs)(x) := f u ( y ) ~ ( x - ~) dy, JR N

x e

~',

is well defined (for small (~) and C 1, with Du~(x) = (Du * g~)(x). Moreover~ as 5 --~ 0 +, (5.1)

u~

>u

locally uniformly in f~.

Since

)~u(y) + H(y, Du(y)) <_ 0

a.e. in f~,

we have

s

(5.2)

+ / R N H(y, Du(y))~5(x - y) dy <_ 0

a.e. in f~.

Now, if H depends only on p, the above gives

(5.3)

/~u~(x) + H ( D u s ( x ) ) <_ 0

V x E ~t',

as a consequence of (H7) and the Jensen's inequality. Since U 5 E C l ( ~ ' ) , b y Proposition 1.3 (b), it satisfies (5.3) in the viscosity sense. Thanks to (5.1) we can apply Proposition 2.2 to obtain that u is a viscosity subsolution of (H J) in fit~, for any ft ~ CC ft and the claim is proved. In the general case, from (5.2) and Jensen's inequality it follows that Aua(x) + H(x, Dua(x)) < ha(x)

V x E f~',

where

f h~(x) = / R N [H(x, Du(y)) - H(y, Du(y))]g~(x - y) d y . By continuity of H, h5 converge uniformly to zero as 5 ~ 0 + and the proof is completed as above. 9 By putting together Propositions 4.8 and 5.1, and taking into account Theorem 4.9, we obtain COROLLARY 5.2. Assume (HT). (i)

If u is a semiconcave generalized solution of (H J) in ~, then it is also a viscosity solution.

(ii)

If, in addition, u E BC(]~ N) A Lip(II~N) and H satisfies the assumptions (H3) and (H6) of Theorem 4.9, then u is a viscosity solution of (H J) in K S if and only if it is a semiconcave generalized solution.

5. SPECIAL RESULTSFOR CONVEX HAMILTONIANS REMARK equations sumption the sense

79

5.3. Corollary 5.2 has various extensions, e.g., to solutions of evolutive which are semiconcave in the space variables. The semiconcavity asof part (i) can be weakened to semi-superharmonicity, i.e., A u _< C in of distributions, and even to N

(u(x + te ) - 2u(x) + u(x - tei)) < Ct 2-~

C [0, 1[,

i=1 for all t >_ 0 and x E IRN, where ei are the vectors of the canonical basis of IRN, but the proof t h a t u is a viscosity supersolution requires a different argument. <~ PROPOSITION 5.4. Let u E LiPloc(Q ) be a viscosity subsolution of (H J) and assume that (HT) holds. Then u is a viscosity supersolution of (H J)_

- ) , u ( x ) - H ( x , Du(x)) = O,

x E d.

PROOF. By Proposition 1.9 (b), (5.4)

)~u(y) + U(y, Du(y)) < 0

a.e. in ~ .

W i t h the same notations as in the proof of Proposition 5.1,

)~u5(x) + H(x, Du~(x)) <_ h5(x)

Vx E ~'.

Since u5 E Cl(f~'), the foregoing implies

-Au~(x) - H(x, Du~(x)) >_ - h ~ ( x ) at any x E 12' and a f o r t i o r i , by Proposition 1.3 (b), in the viscosity sense. Since u~ converges locally uniformly to u as 5 ~ 0 + we obtain

- A u ( x ) - H(x, D u ( x ) ) > 0

in 12

in the viscosity sense (see Proposition 2.2). REMARK 5.5. The convexity of H is necessary for Proposition 5.4 to hold (see the Example 1.6). <~ Theorem 5.6 shows that if H is convex and satisfies some technical assumption, then any viscosity solution of (H J) satisfies (5.5)

;~u(x) + H ( x , p ) = 0

V x E l~, Vp E D - u ( x ) .

and, conversely, the fulfillment of the preceding implies automatically the subsolution condition. Any u E C ( ~ ) satisfying (5.5) will be called a bilateral supersolution (see w for further study of this notion and w for its extension to lower semicontinuous functions). This rather surprising result can indeed be proved immediately for H smooth enough, because viscosity solutions coincide with semiconcave generalized solutions, by Corollary 5.2, and then the result follows at once from the property of semiconcave functions that either D - u ( x ) -- ~ or u is differentiable at x (see Proposition 4.7 (b)). This observation can be used in the general case via regularization by inf and sup convolution and stability properties, yielding:

80

II. CONTINUOUS VISCOSITY SOLUTIONS OF H-J EQUATIONS

THEOREM 5.6. A s s u m e that (H1), (H7) hold. Then u e C ( ~ ) is a viscosity solution of (H J) if and only if (5.6)

)~u(x) + H ( x , p ) = O,

V x 9 fL V p 9 D - u ( x ) .

PROOF. Let us assume that u is a viscosity subsolution of (H J). By Propositions 4.13 and 5.4 its sup-convolution u E satisfies, in the viscosity sense, (HJ~-)

in fY,

-)~u~(x) - g ( x , D u e ( x ) ) >_ - e e ( x )

where ee(x) ~ 0 as e ~ 0 + and f~e is as in Proposition 4.13. Hence, by Proposition 2.2 and Lemma 4.11 (b), -Au(x)-

H(x,p) > 0

Y x 9 f~, Y p 9 D - u ( x ) .

If u is also a viscosity supersolution of (H J), the reverse inequality is also true and (5.6) is proved. Let us assume now that (5.6) holds. We can take, in particular, x = Ye where ue(x) = u(y~) + ]x - ye12/2r and p = (x - Ye)/~ 9 D - u ( y ~ ) (see Lemma 4.12). Hence, (5.6) yields Au(ye) + H(y~, (x - ye)/~) = 0

and, by (HI),

(5.7)

)~ue(x) + H ( x , (x - V~)/~) < e~(x)

in me,

where e,(x) := ()~/2r x - ye] 2 + O~l(IX - yd(1 + Ix - y~l/E)). Since ue is semiconcave, it is differentiable almost everywhere by Proposition 4.6, and D u e ( x ) = (x - YE)/r by Lemma 4.12. Therefore (5.7) reads, for almost every x E fie, )~ue(x) + H ( x , D u e ( x ) ) < ee(x),

and by Proposition 5.1 the same inequality holds in the viscosity sense. By Lemm a t a 4.11 and 4.12, u~ and Pe converge, respectively, to u and 0 locally uniformly as E -~ 0 +. Therefore, using Proposition 2.2 we can conclude t h a t u is a viscosity subsolution of (H J) in f~. Since (5.6) implies trivially that u is a viscosity supersolution, the proof is complete. 9

5.2. Differentiability of solutions A different issue where the convexity of H plays a role is regularity. The next proposition is a (local) regularity result. PROPOSITION 5.7. Let us assume that u E C(f~) is a viscosity solution of )~u(x) + H ( x , D u ( x ) ) = 0

in f~,

with )~ > O. A s s u m e also that p ~ H ( x , p ) is strictly convex for any fixed x E and that - u is semiconcave. Then u E C 1 (~).

5. S P E C I A L RESULTS F O R C O N V E X H A M I L T O N I A N S

81

PROOF. Let us first show that u is differentiable at any x E f l . In view of Proposition 4.7 (c) we just have to prove t h a t D + ( - u ) ( x ) is a singleton for any x E f~; by Proposition 4.7 (a) this will be true, provided D * ( - u ) ( x ) is a singleton. Suppose by contradiction t h a t pl ~ p2 9 D * ( - u ) ( x ) . Then there exist sequences {xn}, {ym} in f~ where u is differentiable and such that

z=

lim x~ =

n--++oo

lira Ym,

rn~+oc

pl=

lim D ( - u ) ( X n ) ,

n~+ar

p2=

lim

m~+oc

D(-u)(ym).

By Proposition 1.9 (a),

AU(Zn) + H(xn, Du(xn)) = Au(ym) + H(Yra, Du(ym)) = 0; by continuity this yields (5.8)

,ku(x) + H(x, _ p l ) = zku(x) + H(x, _p2) = O.

Now let 1 1

1

by strict convexity, (5.8) implies (5.9)

)~u(x) + H ( x , - 3 )

< Au(x) + 1 H ( x , _pl) + ~ H ( x , _p2) = O.

On the other hand, by Proposition 4.7(a), 3 9 c o D * ( - u ) ( x ) = D + ( - u ) ( x ) = - D - u ( x ) . Since u is a viscosity solution of (H J), then

.~u(x) + H ( x , - 3 ) >- 0, contradicting (5.9). Hence u is differentiable at all points. The continuity of Du is a consequence of the upper semicontinuity of the multifunction D+u for semiconcave u, that is the following property:

Xn ~ X,

Pn 9 D+U(Xn),

Pn -'-+P ~

P 9 D+u(x)

(the proof of this fact is implicit in that of Proposition 4.7 (c)).

4

REMARK 5.8. The strict convexity assumption is essential. To realize this, observe that u(x) = Ix[ is a viscosity solution of a(x)(iDu(x)l 2 - 1) = 0, where a is a continuous function such that a(x) > a(O) = 0, for any x. In this example - u is semiconcave but p ~ H ( x , p ) = a(x)(lpI 2 - 1) is not strictly convex for z = 0. Another remark is that strict convexity of H is a serious restriction for Hamiltonians of optimal control problem. In fact it is not satisfied in the very simple case

H ( x , p ) = sup{ - f ( a ) p - g(x) }

x , p 9 IR,

aEA

with A compact, A c IR, because H takes the explicit piecewise affine form

H(x,p) =

-pminf(a)-g(x),

aeA -pma~f(a)-g(x),

p>O, p < O.

<~

82

II. CONTINUOUS VISCOSITY SOLUTIONS OF H-J EQUATIONS

5.3. A comparison theorem In this subsection we present a comparison and uniqueness result where the convexity of H with respect to the p variable plays a key role. As already pointed out in Remark 3.3, equations of the form (5.10)

H(x, Du(x)) = 0

may have more than one viscosity (and even classical) solution satisfying prescribed boundary data. The classical model problem for (5.10) is the eikonal equation of geometric optics (5.11)

IDu(x)l = e ( x ) .

We recall that in w we proved that for ~ - 1 a viscosity solution of this equation in the open set ~ is the distance function from OFt. THEOREM 5.9. Let ~ be a bounded open subset o f R N. Assume ul, u2 9 C ( ~ ) are, respectively viscosity sub- and supersolution of (5.10) in D with ul < u2 on Ol-t. Assume that H satisfies

(H1)

IH(x.,p) - H(y,p)I

< ~o~(Ix -

yl(1 + IP]))

for x, y 9 l~ and p 9 ]~N. Assume also

(H7) p ~ H ( x , p ) is convex on ]~N f o r each x 9 ~; (H8)

there exists ~ 9 C ( ~ ) n C1(~) such that ~ <_ Ul in -~ and supH(x,D~o(x)) <0,

V~' CC ~ .

xE~'

Then Ul < u2 in ~.

PROOF. Set, for t E ]0, 1[, ut(x) = tul(x)

+ (1 - t ) ~ ( x ) ,

x 9

~.

The following properties of u t are easy to cheek (see also Exercise 2.4): ut 9

ut
in~;

D+ut(x) = {q 9 ]RN : q = t p + (1 - t ) D ~ ( x ) , p 9 D+u1(x)}

and

(5.12)

u t -"--+ Ul

ast~l.

By (H7), for any q E D+ut(x), we have H ( x , q) = H ( x , tp + (1 - t ) D ~ ( x ) ) < t H ( x , p ) + (1 - t ) H ( x , D~o(x)),

5. SPECIAL RESULTS FOR CONVEX HAMILTONIANS

83

for some p e D+ul(x). Since ul is a s u b s o l u t i o n of (5.10), (5.13)

H ( x , q) < (1 - t ) g ( x , D p ( x ) ) ,

Vq e D+ut(x);

this m e a n s t h a t for a n y t E [0, 1], u t is a viscosity subsolution of (5.14)

H(x, Out(x)) - (1 - t)g(x) = O,

x e f~,

where g(x) : = g ( x , D r ( x ) ) 9 C(ft). We claim t h a t this implies (5.15)

u t _< u2

on s

V t 9 ]0, 1[.

From (5.12) the thesis then follows. I n order to prove (5.15), let us suppose by contradiction that, for some t 9 ]0, 1[, (5.16)

s u p ( u { - u2)(x) = 5 > 0 . xEl~

Since ul _< u2 on 0 n , the open set

~' = {x 9 ~ : ( ~

-

~2)(x) > ~/2}

is such t h a t ~ C ft. Consider now the auxiliary function

Ix - yl 2 a n d let (xe, ye) be a m a x i m u m point for (I)~ on 12~ x ~ . rem 3.1 one obtains

(5.17)

Ix~ - y~l

yE,21 2S

,0,

As in the proof of Theo-

* 0

as s ---* 0 +. Let us observe also that ~E(x~, y~) _> sup ~c(x,x) : sup (u{(x) - u2(x)) : 6. xEft'

xEi-t'

This implies t h a t (x~, YE) E ~t x s ~ for sufficiently small s. Indeed, if this were not the case, t h e n either xE or yc would belong to Of~t, yielding

where ~2 is a m o d u l u s of c o n t i n u i t y for u2. Taking (5.17) into account we o b t a i n a contradiction. Since x~ is a local m a x i m u m for x ,-+ J ( x ) - (u2(yz) + Ix - yE]2/2s) a n d u ~ is a viscosity s u b s o l u t i o n of (5.14), we have (5.18)

H(xE, x~ - y~ ) _ ( 1 - i)g(xe) <_ 0; E

84

II. CONTINUOUS VISCOSITY SOLUTIONS OF H-J EQUATIONS

on the other hand, Yc is a local minimum for y ~-~ u2(y) - (ut(xc) - [ x c - y12/2e), yielding (5.19)

H ( y ~ , xE

~Y~) _>0.

since u2 is a viscosity supersolution of (5.10). By (Hs), there exists o > 0 such that, for all ~ small, (1 - ~)g(xe) = (1 - t ) H ( x ~ , D p ( x , ) )

<_ -c~ .

Hence, taking (H1) into account, from (5.18), (5.19), and the preceding, it follows that

a contradiction for g small enough, in view of (5.17). REMARK 5.10. Theorem 5.9 applies to the eikonal equation (5.11) if ~ is uniformly continuous in ~ and strictly positive. In fact (H8) is satisfied by taking ~2(x) - min ul 9 Therefore the Dirichlet problem for this equation has at most one viscosity solution in C ( ~ ) for any bounded open set D. In particular u(x) = d(x, 0 ~ ) is the unique solution of

IDul = 1 in t2,

u = 0

on cOgt.

The next remark shows that this uniqueness result is sharp because there can be many solutions if ~ vanish at a single point in ~. <~ REMARK 5.11. Assumption (Hs) is not satisfied by H ( x , p ) = IPl - 21xl with ~ = ] - 1 , 1[. Indeed, the supremum of H ( x , ~a'(x)) in any neighborhood of x = 0 is at least I~J(0)l for any T c C1(~). Uniqueness is not true for the associated Hamilton-Jacobi equation. Actually Ul(X) = x 2 - 1, u2(x) = 1 - x 2 are both classical solutions of lu'l = 2Ix I in ] - 1 , 1[ and ul = u2 = 0 at x = -t-1. A more careful investigation reveals the existence of a continuum of viscosity solutions connecting U 1 with u2 (see Exercise 5.6). <~ REMARK 5.12. An alternative approach to comparison and uniqueness for equation (5.10) under different assumptions is via the Kru2kov transform: see Chapter IV for this approach (see also Exercise 5.4). <~ 5.4. S o l u t i o n s i n t h e e x t e n d e d s e n s e We briefly discuss here the relation between viscosity solutions and another concept of solution which is known in the Lipschitz continuous case. Let us consider the model Hamiltonian

H(x,p) = sup{-f(x,a).paEA

~(x,a) }

5. SPECIAL RESULTS FOR CONVEX HAMILTONIANS

85

with the usual continuity assumption. The following notion of solution of the Hamilton-Jacobi-Bellman equation is given in terms of the Clarke's gradient Ou of u (see w for the definition). A function u E Lip(l}) is said to be a solution in the extended sense of the equation (H J)

(5.20)

Au(x) + H(x, Du(x)) = O,

Au(x) +

x E 12~

max H ( x , p ) = 0 pcOu(z)

V x E gl.

PROPOSITION 5.13. Let u E Lip(i}). If u solves (HJ) in the extended sense, then u is a viscosity subsolution of (H J) and a viscosity supersolution of - ~ u ( x ) - H(x, Du(x))

= 0,

x e

~.

Conversely, any viscosity solution of (H J) is a solution in the extended sense. PROOF. Since, by (4.6), D + u ( x ) U D - u ( x ) C_ On(x), from (5.20) we obtain directly Au(x) § H ( x , p ) <_ O,

V x E 12, Vp E D+u(x)

~u(x)+H(x,q)<_O,

Vxe~,

VqeD-u(x).

From the above inequalities the first part of the statement follows. Now let u E Lip(~) be a viscosity solution of (H J). We know already that Au(x) § g ( x . D u ( x ) ) = 0

almost everywhere in i}

(see Proposition 1.9). By definition of D*, it follows that )~u(x)+H(x,p)=O

Vx E 12, V p E D * u ( x ) .

Hence (5.21)

Au(x) §

max

pED*u(x)

H ( x , p ) = O,

Vx E ~.

Now the convexity of H with respect to p yields max

pEco D*u(x)

H(x,p)=

max

pED*u(x)

H(x,p)

and the thesis follows from (5.21) and the identity co D*u(x) = Ou(x) (see (4.4)). REMARK 5.14. Lipschitz continuous viscosity solutions form in general a proper subset of solutions in the extended sense. To realize this, consider the onedimensional equation [ u ' [ - 1 = 0 in ] - 1 , 1[ and the functions ui(x) = 1 - I x I , u2(x) = Ix] - 1. Since C~ul(O) ---- Ou2(O ) : [-1, 1], it is easy to check that both u I

86

II. CONTINUOUS VISCOSITY SOLUTIONS OF H-J EQUATIONS

and u2 solve the equation in the extended sense. However, Ul is a viscosity solution while u2 is not. Observe also that the boundary value problem ]u~l - 1 = 0 in ]-1, 1[, u ( - 1 ) = u(1) = 0, has a unique viscosity solution by Remark 5.10, namely ul. A final remark is that solutions in the extended sense are not stable in the uniform topology. A simple example is provided by the saw-tooth function in Remark 2.3. <~ 5.5. D i f f e r e n t i a l i n e q u a l i t i e s in t h e v i s c o s i t y sense The classical method of characteristics for first order partial differential equations starts from the observation that smooth subsolutions of the linear equation (5.22)

-f(x)

. Du(x) = 0

with f E C I ( R N) are nondecreasing along the trajectories of the characteristic system (5.23)

$(t) = f ( y ( t ) ) y(0) = x.

Proposition 5.18 shows that the same result holds true for viscosity subsolutions of (5.22). We then consider the nonlinear model case (HJB)

s

+ sup{ - f ( x , a ) . D u ( x ) - g(x, a) } = 0 aEA

and prove that viscosity subsolutions of (HJB) have the same behavior on the trajectories of the controlled system (5.24)

y(t) = f ( y ( t ) , a(t))

y(O) =

where (~ E .4 = {(~: [0~ +oc[ --~ A, ~ measurable }. The first step is the following one-dimensional monotonicity result. LEMMA 5.15. Let u E C(]0, TD, T :> 0. Then the following statements are equivalent:

(i)

u is nondecreasing on ]0, T[;

(ii)

u' > 0 in ]0. T[ in the viscosity sense;

(iii)

- u ' ~ 0 in ]0, T[ in the viscosity sense.

PROOF. Let p E CI(]0, T[) and to be a local minimum for u - ~;. If u is nondecreasing, then ~(t) - ~(t0) < ~(t) - u(t0) < 0

5. SPECIAL RESULTS FOR CONVEX HAMILTONIANS

87

for t < to close e n o u g h to to. This implies ~'(t0) _> 0, showing t h a t (ii) holds. Let us a s s u m e now t h a t (ii) holds a n d suppose t h a t u ( t l ) > u(t2)

(5.25)

for some 0 < tl < t2 < T .

By continuity, there exists t E [tl,t2] such t h a t u(t2) < u ( t ) < u(Q). Choose now E c l ( ] t l , t2D (see Exercise 5.7) such t h a t

~'(t) < O,

Vt E ]tl,t2[,

~(t) > u(t),

~(ti) = u(ti),

i = 1,2.

Since (u - ~;)(ti) = 0, (u - F ) ( t ) < 0, u - ~ a t t a i n s its m i n i m u m over [tl,t2] at some interior point to. Hence ~J(t0) < 0, a contradiction to (ii). T h e equivalence between (i) a n d (iii) is proved by similar arguments. ~, REMARK 5.16. It is easy to deduce from L e m m a 5.15 t h a t if g E C(]0, T D, t h e n

t ~ u(t) + fo ~(s) ds is nondecreasing if a n d only if u' + ~ > 0 or - u ~ - g < 0 in the viscosity sense. Also, it can be proved similarly t h a t u is n o n i n c r e a s i n g if a n d only if u ~ < 0 in the viscosity sense. Hence, u ~ = 0 in the viscosity sense is equivalent to u b e i n g a constant. <1 In order to generalize this result to a higher dimension, let ~ be a n open subset of ]~N and u E C ( ~ ) . For fixed z = (x2 . . . . , XN) E ]~N-1 we set ~ z := { xl c ~ : x = ( ~ , z) E ~ } and ~ z ( X l ) : = U(Xl, Z),

Uz : ~ z ' - - - ~ I ~ .

LEMMA 5.17. Let u, ~ E C(~). Then the following statements are equivalent: (i)

for each z E ]~N-1 uz is a viscosity supersolution of U~(Xl) ~ ~z(Xl)

(ii)

in ~ z ;

u is a viscosity supersolution of O~zI(Xl,X2,...., XN) ~ ~(Xl,X2,...,XN)

in ~ .

PROOF. We first prove that (ii) implies (i). Let z ~ E ~ N - 1 be such t h a t flz o r 0 a n d assume t h a t x ~ is a strict local m i n i m u m for u~o - ~? with ~ E C 1. It is n o t restrictive to a s s u m e t h a t ~ < - 1 in B ( x ~ 5) for some (f > 0. Consider now ~E(Xl~..

XN):= 7](Xl)(1 + [ - - - ~ )

88

II. CONTINUOUS VISCOSITY SOLUTIONS OF H-J EQUATIONS

If x C = (x~, z ~) is a m i n i m u m p o i n t for u - p~ i n / 3 ( x ~ 6) (x ~ = (Xl~ z~

~(~) - ;~(x ~)

(5.26)

u(x ~)

=

,(x~)

-

,(~)

-

Iz~

-

then

~~

E"

< u(x ~

~ ( x o) = ~o(~1 ~

-

-

,(x~

Since r / < - 1 in B ( x ~ ~), it follows t h a t

Iz~ - z~

< ~zO(~l~ - , ( x ~ -w ,(~I) - u(x~) 9

E Therefore

Iz~

-

z~

_<

C,

Z~ ~

ZO.

a8 ~ ' ~ 0 + .

,a>_O,

asz~O

T h e n , at least for a s u b s e q u e n c e , X~

Iz E _ zOl 2

~ Xl

E

+ .

L e t t i n g z --~ 0 + in (5.26) we o b t a i n ~zo(xO)

_

,(xo) _>~(~1,

z ~

-

,(~1)

-

,(~)~

UzO(Xl) -- T](~I) . Since x ~ w a s a l o c a l s t r i c t m i n i m u m

f o r uzo - rl, t h e a b o v e i m p l i e s 71 -= x ~ a n d

a = 0. Now, a s s u m i n g t h e v a l i d i t y of (ii),

o~Eo~I (s) = v'(x~)(i+ ~-~-E lz~ ~~ >_ e(s) If we let a ~ 0 + in the above inequality we conclude V'(x ~ >_ g(x ~ z ~ = gzo (x~ this shows that (i) holds true. The proof of the reverse implication is straightforward. It is enough to observe that if ~ = (~i, ~) is a local m i n i m u m for u - ~, E C l ( f ~ ) , t h e n ~1 is a local m i n i m u m for u-~(xl) - ~ ( X l , ~ ) . In t h e following we use t h e n o t a t i o n t~ : = i n f { t > 0 : yx(t) ~ ft },

x E f~,

w h e r e Yx(" ) is t h e s o l u t i o n of (5.23). PROPOSITION 5.18. Let us assume f E C I ( ~ , ] I ~ N ) , g E C(['~), /~ E ]~ and u E C(f~). Then the following statements are equivalent: (i)

e-A~u(yx(s)) - e-Atu(y~(t)) <_ f o r all O < s < t < tx, x E fl;

Z

g(yx(T)) e -A" dr,

5. SPECIAL RESULTS FOR CONVEX HAMILTONIANS (ii)

89

-)~u(x) + f ( x ) . Du(x) + t(x) >_ O, x E ~, in the viscosity sense;

( i i i ) ) ~ u ( x ) - f ( x ) . Du(x) - t(x) <_ O, x E ~, in the viscosity sense. PROOF. Let us observe first t h a t it is n o t restrictive to assume A = 0. Indeed, u satisfies (ii) if a n d only if ~(x, XN+l) : = XN+lU(X) is a viscosity supersolution of f'- D ~ + ~">_ 0

in f~ • JR+,

where f(x, xi+l) := if(x),--;~XN+l) and "[(X,XN+I) := xi+l~(x) (see Proposition 2.7). O n the other hand, it is easy to check t h a t (i) is equivalent to ~(~(s)) - ~(~(t))

<

~(~-)) d~-,

s < t,

where ~ is the solution of ~(t) = ]'(~(t)) = (f(y(t)), --)WN+I (t)) ~(0) = (x, 1 ) . Assume t h e n t h a t (i) holds with s = 0. T h e n it is not hard to prove, for x E a n d s < 0, Is I small enough, t h a t (5.27)

e(yx(~-)) d~-.

~(y~(s)) - ~(x) <

Now, if x E ~ is a local m i n i m u m for u - 4, P E C1(~2), t h e n (5.28)

~(~(s))

- ~ ( x ) _< u ( y x ( s ) ) -

~(~)

for s < 0 small enough. C o m b i n i n g (5.27) with (5.28) we o b t a i n ~(y~(s)) - ~(x) <

/o

e(y~(~)) d ~ .

Dividing this by - s > 0 a n d letting s ~ 0 we conclude t h a t - f ( x ) . Dp(x) < g(x) a n d (ii) is proved. To prove the reverse implication, let us assume first t h a t x0 is a local m i n i m u m for u - ~ with ~ E C ~(fl). T h e n (ii) gives (5.29)

f(xo) " Du(xo) + t(xo) > O.

Now if f(xo) = 0, t h e n yxo(~-) =- x0. I n e q u a l i t y (i) reduces in this case to 0 _< g(x0)

d~- = (t - s ) t ( x 0 )

for s _< t which is clearly implied by (5.29).

90

II. CONTINUOUS VISCOSITYSOLUTIONS OF H-J EQUATIONS

Consider now the case f(xo) # O. Since f E CI(]I(N), by classical results on ordinary differential equations, there exists a local diffeomorphism 4) such t h a t in the new coordinates ~ = (I)(x) system (5.23) becomes

~(t) = el = (1, 0 , . . . , 0 )

(5.30)

{(0) = x 0 .

The change of coordinates in (ii) implies t h a t u((I)-l(~)) satisfies f ( ~ - l ( { ) ) . D,~(zp-l({))Du(~-l({)) + e((i)-l({)) >_ 0 in the viscosity sense (see Exercise 2.6). Thanks to (5.30) this amounts to

o~ (~_~({)) + t(~_~({) ) > o.

0& Using now

Lemmas 5.17 and 5.15 (see also Remark 5.16) we conclude t h a t

u(e-l(s,0,...,0))-u(~-l(t,0,...,0))

<

s

e ( ~ - ~ ( T , 0 . . . . . 0))dT

for 0 < s < t, with s, t sufficiently small. By definition of (I) this is the same as

u(yxo(S)) - u(yxo(t)) <

//

e(~xo(~)) d~,

for 0 < s < t, s, t small. A simple continuation argument shows the validity of (i) for a n y 0 < s < t
s is constant for any fixed x E ~.

<

5.6. M o n o t o n i c i t y o f v a l u e f u n c t i o n s a l o n g t r a j e c t o r i e s Let us now consider the controlled system (5.24). We denote by Yx(" , a ) the solution of (5.24) corresponding to a given control a E .4 and set, for x E f~,

t~(a) := inf{t >__0 : yx(t,a) ~ ~t }.

5. SPECIAL RESULTS FOR CONVEX HAMILTONIANS

91

If we use constant controls a(t) = a in (5.24), then the equivalence between

(5.31)

e-a%(yx(s,a)) <_e-atu(yx(t,a)) + Vxcf~,

O<s
e(yx(r,a),a)e-a" dr

VaEA,

(5.32)

Au + H(x, Du) <_ 0

in f~ (viscosity sense),

(5.33)

- A u - I-I(x, Du) >_ 0

in f~ (viscosity sense),

with H ( x ; p ) = sup{ - f ( x , a) . p - g(x, a) }~ acA

is a straightforward consequence of Proposition 5.18. A repeated application of Proposition 5.18 allows us to show that (5.32) and (5.33) are equivalent to (5.34)

e-~Su(yx(s,a)) <_ e-;~tu(yz(t,a)) +

g(yz(T,a),a(T))e-;~r d7

V x E ~, 0 < s < t < tx(a), V a C P, where P C A is the class of piecewise constant controls. To establish the equivalence between (5.32), (5.33), (5.34) for general controls a E .4 we need the following approximation lemma.

LEMMA 5.20. Let a E A, x E ~, and y(t) = yx(t, a) be the corresponding solution of (5.24). Assume (5.35)

f is continuous and bounded on l-I • A,

(5.36)

SL>O:lf(x,a)-f(y,a)[
Vx, y ~ f l ,

VaEA.

Then for every T > 0 there exists a sequence {an} C A such that

(5.37) an is pieeewise constant on [0, T], lan(t) - a(t)l < 1/~, (5.38)

Vt e E~ C [0, T], E~ c o m p a c t , I[0, T] \ E~I < Yn

1/n

uniformly in [0, T],

, y

where yn(t) := y z ( t , a , ) and [El denotes the Lebesgue measure o r E .

PROOF. Assertion (5.37) is a consequence of Lusin's theorem (see Exercise 5.8). To prove (5.38) observe that lye(t) - y(t)l < L

/0'

lye(s) - y(s)l ds +

/0'

IB~(s)l

where L is as in (5.36) and Bn(s)=f(y(s),an(s))

-

f(y(s),a(s)).

ds,

92

II. C O N T I N U O U S VISCOSITY SOLUTIONS OF H-J EQUATIONS

Hence, by Gronwall's inequality, (5.39)

lye(t) - y(t)l <

/0

IB~(s)l ds + L e LT

IBn(v)l dTds, JO

JO

for t 9 [0, T]. From (5.37), an(t) --~ a(t) a.e. in [0, T] (at least for a subsequence). Hence, by continuity a.e. in [0,T] .

B , ( T ) ----+ 0

Moreover, by assumption (5.35),

IB,,(~)I _< c

v-,- e [0,Tl, n 9 N.

Assertion (5.38) now follows from (5.39) and the dominated convergence theorem. THEOREM 5.21. Let us assume that f satisfies (5.35), (5.36), and (5.40)

f ( . ,a) E CZ(~)

for every a E A .

Assume also (5.41)

E C ( ~ x A),

~ bounded, A >_ 0 .

Then for u E C ( ~ ) the following statements are equivalent:

(J) (jj)

lv~

9 A, z 9 ~, 0 < s < t < tx(~); Au(x) + sup{ - f ( x , a ) . D u ( x ) - g(x, a) } < 0

in ~ ;

aEA

(jjj)

- A u ( x ) - sup{ - f ( x , a ) . D u ( x ) - ~(x~ a) } > 0

in ~ ;

aEA

where (jj) and (jjj) are understood in the viscosity sense.

PROOF. The discussion before Lemma 5.20 shows that in order to prove the statement it is enough to show that (5.34) implies (j) for any u 9 C(gt) (the reverse implication being trivial). For this purpose let a be arbitrary in ,4 and take a,~ and y~ as in Lemma 5.20. By (5.34), e-~(~(~))

- e-~u(y~(t)) <

//

e(y~(~),~(~))e -~d~

for 0 < s < t < tz(an). Since a,~ ~ a almost everywhere and yn ---* y ( ' , a) uniformly on compact intervals by Lemma 5.20, a passage to the limit in the previous inequality as n --~ +oc gives the desired result (see also Exercise 5.10). 4 REMARK 5.22. All the results of this section hold under more general assumptions (see Chapter III, w Let us mention here that the differentiability assumption (5.40) on f can be relaxed to ( f ( x , a ) - f ( y , a ) ) . (x - y) <_ LIx - yl 2. <~

5. SPECIAL RESULTS FOR CONVEX HAMILTONIANS

93

REMARK 5.23. T h e o r e m 5.21 shows in p a r t i c u l a r t h a t if u is a viscosity subsolution of (HJB) t h e n

g(s) := e-~S~(yx(S, ~)) +

/0

e(yx(~-, ~)) e -h" d~-

is n o n d e c r e a s i n g for a n y choice of c~ E .4. Of course one c a n n o t expect g to be n o n i n c r e a s i n g for all cr E .4 when u is a viscosity (or even classical) supersolution of (HJB) (see Exercise 5.9). Indeed we will see in C h a p t e r III, w t h a t g n o n i n c r e a s i n g is a sufficient condition of o p t i m a l i t y for a.
5.24. T h e n l a x i m u m b o u n d e d function uM satisfying (j) has been propoliterature as a good n o t i o n of weak solution of (HJB) (see [BL82, MeS0a]). 5.21 a n d Proposition 2.1 show t h a t uAf is actually a viscosity solution provided uM E C(Y~). <~ Exercises

5.1. Let H ( x , p ) = supaeA { - - f ( x , a ) . p -- *(x,a) } where A is a topological space, f : ]~N • A --* ]I~N~ ~ : ]~N X A --* ]R are given functions. (i)

Show t h a t p ~ H(x, p) is convex for each x.

(ii)

Assume t h a t x ~ f ( x , a), z ~-~ g(x, a) are c o n t i n u o u s for each fixed a e A a n d show t h a t H ( x , p ) is lower semicontinuous.

(iii) Assume t h a t for some moduli a~'l, cv2 a n d M > 0 i n d e p e n d e n t of a E A

If(x, a) - I(y,a)I <- Wl(I x - Yl), It.(x,a)-f.(y,a)[ <_o32(}x-yI),

If(x,a)I < M, 'v'x, y e ~ N

a n d show t h a t H is jointly continuous with respect to (x, p). 5.2. Suppose t h a t (HT) holds. Use Propositions 1.9 a n d 5.1 to show t h a t the set S : = { u e Lip(Q) : u(x) + H(x, Du(x)) <_ 0 in ~, in the viscosity sense } is convex. 5.3. Take H ( x , p ) as in Exercise 5.1 with f ( x , a ) = a C A, a closed convex set in ]RN. Assume also t h a t a ~-~ g(x, a) is C 1 strictly convex a n d (in case A is u n b o u n d e d ) t h a t g(x,a)/la ] ---* +ec as laI ~ +oc, for each fixed x. Show t h a t

H ( x , p ) = - a * . p - g(x,a*) where a* = a* (x, p) E A is the u n i q u e solution of the variational inequality

(Da~.(x,a*) + p ) . (a - a*) ~ 0

Va e A .

C o m p u t e explicitly a* in the special case A = R N, g(x, a) = Go(x) + ~1 a12 .

94

II. CONTINUOUS VISCOSITY SOLUTIONS OF H-J EQUATIONS

5.4 (KRU2KOV'S CHANGE OF VARIABLES). Show t h a t u is a viscosity solution of IDu(x)l -- t(x) if and only if v(x) = - e -u(z) is a viscosity solution of [Dv(x)l + g(x)v(x) = o .

5.5. Let u(x, b), b E B compact, be a family of C 1 solutions of

H(x, u(x, b), Dxu(x, b)) = 0 in ft. Suppose t h a t b ~-~ u(x, b) is lower semicontinuous and t h a t p ~-* H(x, r,p) is convex for all fixed x, r. Show t h a t u(x) := minbeB u(x, b) is a viscosity solution of the same equation. [Hint: use P r o p o s i t i o n 2.13.] 5.6. Show t h a t for each t E [0, 1] the function

vt(x)=

is a viscosity solution of

1 - - x 2.

--1 < x < - - t

x 2-1+2(1-t2), 1 - x 2,

t<x<<_l

lu'(x)l-

2txl = o, x

Ix I < t

e ( - 1 , 1 ) such t h a t v t ( - 1 ) = vt(1) = 0.

5.7. Let u E C([tl,t2]), t E ]tl,t2[ with u(t2) < u(~) < u(tl). Fix 5 > 0 such t h a t + 5 < t2, and set

~mt+q, p(t)= [at2+bt+c,

tl < _ t < t + 5 t+5
F i n d constants m,q,a,b,c and 5 such t h a t r E Cl(tl,t2), p ' < 0, p ( t ) > u ( t ) , ~(ti) = u(ti) (i = 1, 2). 5.8. Let (~ : [0, T] --~ A be measurable. Recall t h a t (Lusin's Theorem) for any E > 0 there exist a compact set E C [0, T] and 5(c) > 0 such t h a t

I~(t)-~(s)l
Vs, t E E w i t h i s - t ] < 5 ( ~

)

Use this and the partition of [0, T] into intervals [k& (k + 1)5[ (k -- 0,..., [T/b]) to prove assertion (5.37) of Lemma 5.20. 5.9. Observe that u(x) = x is a classical solution of Iu'(x)I- I = o, x E JR, or (equivalently) of sup

{-au'(x)

- 1} = O.

ae[-1,1]

For a r b i t r a r i l y fixed T > 0 set

and show t h a t g(s) < g(t) for s < T < t, where g is as in R e m a r k 5.23.

6. BIBLIOGRAPHICALNOTES

95

5.10. Give alternative proofs of Propositions 5.1 and 5.4 by means of (4.4) and (4.6) without using regularization procedures. 5.11. Use Theorem 5.21 to show that the value function of the infinite horizon problem in Chapter I is the maximum viscosity subsolution of the corresponding (HJB) equation. 5.12. With the assumptions and notations of Lemma 5.20 prove that liminf tx(an) > tx(C~). n

6. B i b l i o g r a p h i c a l n o t e s The presentation of the theory of viscosity solutions in Sections 1 to 3 is basically inspired by the papers [CEL84, CL83, I84] and the lecture notes [I88b]. We adopted the common habit, initiated by Crandall, Evans and Lions [CEL84], of working with either the "test functions" or the "semidifferentials" definitions of viscosity solutions rather than with the original one in [CL81, L82, CL83]. The connections with the Maximum Principle in elliptic theory are inspired by [Caf89, CafCa95]. The subdifferential D - u defined here coincides with the classical subdifferential of convex analysis [R70] provided that function u is convex (see Exercise 1.3). The comparison result for unbounded solutions Theorem 3.7 as well as Proposition 4.2 on Lipschitz continuity and Theorem 4.9 on semiconcavity are due to Ishii [I84]. Among the many papers touching the important topic of comparison and uniqueness of continuous viscosity solutions let us mention here also [CDE84, CN85, Sou86, Ba85b, L85b, $86, I89, BaP87, BaPg0, CDL90, DI90, BaL91, I91, CIL92, IK96] for bounded solutions, and [185a, I86, CL86, CL87, CIL87, B89, CDLg0, Ba90a, Bag0b, ME95, BDL96, Alv97a] for unbounded solutions, the book [Ba94], and the references therein. Our treatment of the basic theory of marginal functions in w seems to be new, as well as some results; the formula for directional derivatives in Proposition 2.13 is known in Operations Research as Danskin's theorem (under the assumptions of Proposition 4.4), see [DeM71, Cla75] and the references therein. The notion of (generalized) exterior normal is due to Bony [Bon69]; proximal normals are a standard tool in nonsmooth analysis JR81, Cla89]. A proof of the classical Rademacher theorem on differentiability of Lipschitz functions can be found for example in [EG92]. The generalized differential calculus for Lipschitz continuous functions and the related notion of solution of HamiltonJacobi equations were developed mainly by F.H. Clarke [Cia75, Cla83]. Some relations between Clarke's generalized gradient and the semidifferentials of the theory of viscosity solutions have been investigated in [CS87, CS89, Fr89a]. See also the recent survey [CLSW95]. The role of semiconcavity in Hamilton-Jacobi theory and optimal control was pointed out by Kru2kov [Kr60, Kr67, Kr75] (see also [Bt77, Hr78]) and in connections with viscosity solutions in [L82, CDI84, CS87, CS89, CF91, CSi95a]. Proposition 4.7 follows [CS89]; the semiconcavity of viscosity solutions (Theorem 4.9)

96

II. CONTINUOUSVISCOSITYSOLUTIONSOF H-J EQUATIONS

is taken from [I84] and Proposition 4.13 is from [Ba93]. For the related topic of inf-convolutiom we refer to [Mo66] and [LL86]. Other results on the regularity of solutions and the regularizing effect in the Cauchy problem can be found in [L85d, Ba90b, Ba90c, ART96]. The equivalence of semiconcave generalized solutions and viscosity solutions for convex and smooth Hamiltonians (Corollary 5.2) is adapted from [CL83]; it gives a further explanation of the uniqueness theory for semiconcave generalized solutions in [Kr60, Kr67, Kr75, Dou61, L82]. The proof of Remark 5.3 is in [LS95]. The characterization of viscosity solutions as bilateral supersolutions in the case of convex Hamiltonian (Theorem 5.6) is due to Barron and Jensen [BJg0], but our proof follows [Ba93]. Proposition 5.7 is a variant of a result in [CS89]. Theorem 5.9 is essentially a result of [Kr75], presented here following [I87b]. Most of the results in w are from [CL83], our presentation and w take advantage of some unpublished work by P. Soravia. The connection between the classical method of characteristics and viscosity solutions is studied in more detail in [Sua91, Ko93a, Ko93b]. Some important topics in the general theory of viscosity solutions of first order Hamilton-Jacobi equations are not covered in this chapter. First of all, we did not give any existence result. The reason is that in the next two chapters we always prove directly that the value function of a given control problem is a viscosity solution of the corresponding HJB equation. The best approach to the existence theory is the so-called Perron's method, settled by Ishii [I87a], which needs the notion of discontinuous solution and is presented in Chapter V, w for earlier results see also [L83d, I83, Ba84, EI84, I86] and the references in [CIL92, Ba94]. Representation formulas are a classical issue since the work of Hopf [H65] and Kru~kov [Kr67]. The connection between Hopf's formulas and viscosity solutions was studied in [L82, BE84, LR86], see also [BOs91, Su95, BJL96, BFag96] for some recent extensions of these formulas, and [Car93] for their use in linking viscosity solutions with geometric solutions in the sense of symplectic mechanics. Of course any characterization of the value function in optimal control or differential games as the unique solution of a HJ equation gives a representation result for solutions of a class of equations whose Hamiltonian can be written in the Bellman or Isaacs form. This point of view was pursued, e.g., in [L82, ES84, I85b, I88a]. Among the other topics that we do not touch in this book we also mention the analysis of singularities of solutions in [CS87, CS89, ACS93, Ko93a, Ko93b, IzK95, Bad94, Wi96], geometric properties of solutions in [B85, BadB90, BadB92, Bad94, ALL97], propagation of fronts with curvature-dependent speed [OSe88, ESp91, ESp95, CGG91, Sor94a], the problem of shape-from-shading in vision theory [RT92, Tou92, LRT93], equations with singular coefficients [Si95, IR95], the connections with noneonvex problems in the Calculus of Variations [Ba90a, Cut93, DAM96].

CHAPTER

III

O p t i m a l control p r o b l e m s w i t h c o n t i n u o u s value functions: u n r e s t r i c t e d s t a t e space

In this Chapter we consider several optimal control problems whose value function is defined and continuous on the whole space R N. This setting is suitable for those problems where no a priori constraint is imposed on the state of the control system. For all the problems considered we establish the Dynamic Programming Principle and derive from it the appropriate Hamilton-Jacobi-Bellman equation for the value function. This allows us to apply the theory of Chapter II, and some extensions of it, to prove that the value function can in fact be characterized as the unique viscosity solution of the corresponding Hamilton-Jacobi-Bellman equation. This has some direct consequences such as verification theorems. More involved applications such as those to necessary and sufficient conditions of optimality require some additional tools from nonsmooth analysis like generalized directional derivatives. These themes are developed in Sections 2.3 to 2.5 with reference to the infinite horizon problem. The important issue of connections with the Pontryagin Maximum Principle (in a necessary and sufficient form) is covered in detail in Section 3.4 for the finite horizon problem. 1. T h e

controlled dynamical s y s t e m

We list here some basic assumptions on the nonlinear control system that we consider. We will assume:

(A~

A is a topological space, f : R g • A ~ R N is continuous;

(A1)

f is bounded on B(0, R) • A for all R > 0;

(the local boundedness of f , uniformly with respect to the control variable a)

(A2)

there is a modulus ~v/such t h a t ] f ( y , a ) - f ( x , a ) l < ~ f ( I x - yl, R), for all x, y E B(0, R) and R > 0,

98

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

(the local uniform continuity of f , uniformly in a) where a modulus is a function w : ]E+ x R+ ~ JR+ such that, for all R > 0, w( 9 R) is continuous, nondecreasing, and w(0, R) = 0. The reader can assume for simplicity that A is a subset of R M. In most applications A is compact and in this case (A1) and (A2) are consequences of (A0). We will also assume (A3)

(f(x, a) - f(y, a)). (x - y) <_ L [x - yl 2

for all x, y E ~N, a E A;

that is, there is L E lt~ such that f ( 9 a) - LI, where I is the identity, is a monotone "nonincreasing" map for all a. A more familiar condition which implies (A3) (and (A2)) is the global Lipschitz continuity of f in the state variable, uniformly in the control variable, that is, (1.1)

If(x,a)-f(y,a)i<_nlx-yI

forallx, yE]RN, a e A .

We look at trajectories of the nonlinear system (S)

~ y'(t) = f(y(t),a(t)),

t > O,

( y(o) = x, for control functions a ( 9 ) ("open loop" controls) in the set 1 A := {measurable functions [0, +oc[ --* A } . This means that y( 9 ) solves the integral equation (1.2)

y(t) = x +

S(y(s),

t > 0,

so that, in particular, y( - ) is absolutely continuous on compact intervals of [0, +oo[ and it solves (S) almost everywhere. Under the assumptions (A0), (A1), (A3) the existence and uniqueness of a solution of (S) (i.e., of (1.2)), defined for all t E [0, +oc[, follows from the standard theory of ordinary differential equations and we recall the proof in the appendix of this chapter, Section 5. For given a E `4 and x E ~N, the solution will be denoted by yx(t, a), or briefly by y~(t) if no confusion may arise. The basic estimates on yx are the following (for the proofs see w (1.3)

{yx(t, a) - x I < Mxt

for all a E .4 and t E [0, 1/Mz],

where -~lx := sup{ ]f(z,a)l: Iz - x I < 1, a E A}; (1.4)

lYx(t, a)l ~ (IxI + 2x~-~)e Kt

for all a E .4 and t > 0•

where K := L + sup{ If(0, a)l : a E A }; 1We recall that a : [0, +co[ --~ A is (Lebesgue) measurable if for any open set B C A the set {t : a(t) E B} is Lebesgue measurable.

2. T H E I N F I N I T E HORIZON PROBLEM

(1.5)

[yx(t,a)-y~(t,a)l<_eLtlx-z

I

99

for all a E A and t > 0 .

We will also be interested in some subsets of A, such as piecewise constant controls 7~ := { a E .4 : there is an increasing sequence t,~ such that l i m t n = +oc and a is constant on ]tn,t,~+l[ for all n }, n

and m o n o t o n e controls, for A _ N, Am := { a E .4 : a is nondecreasing } .

2. T h e i n f i n i t e h o r i z o n p r o b l e m 2.1. D y n a m i c P r o g r a m m i n g and the Hamilton-Jacobi-Bellman equation In this section the cost to minimize is

J ( x , a) :=

//

g(yx(t), a ( t ) ) e -;~t dt,

where the r u n n i n g cost f : N N • A --~ R and the constant interest rate A satisfy t is continuous; there are a modulus a3t and a constant M such that (A4)

It(x, a) - f(y, a)[ <_ wt(Ix - Yl) and ]f(x,a)l < M , for all x, y E IRN and a E A; A>O.

The value function for this problem is

v(x)

:= inf J ( x , a ) . aEA

It is also called Bellman's function, the optimal result function, the guaranteed outcome. The name "value" comes from an economic interpretation of the equivalent problem of maximizing the payoff - J : in this problem - v ( x ) is the best we can get from the initial state x, thus it is the economic value of x. We first look at the regularity of v. PROPOSITION 2.1. A s s u m e (A0), (A1), (As), and (A4). Then v E BUC(NN). I f moreover a~t(r) = L t r (i.e., f is Lipschitz in y, uniformly in a), then v is Hhlder continuous with the followin 9 exponent ~/:

7 =

1

irA>L,

any 7 < 1

if A = L,

)~/L

if A < L .

100

III. PROBLEMS

WITH

UNRESTRICTED

STATE

SPACE

PROOF. The estimate Iv(x)I < M/)~ for all x E l~ N is an immediate of (A4). Next we fix x,z E R N, ~ > 0 and ~ E ,4 such that

v(z) >

consequence

g(yz(t), 5 ( t ) ) e -At dt - e .

Then (2.1)

v(x) - v(z)

<

f0 ~ e-~tle(~x(t), ~(t)) - e(yz(t), ~(t))l at + f ( 2 M e -A~ at + ~ .

For T large enough we can make the second integral in the right-hand side less then z and get v(x)-v(z)

<

(2.2) <

~e(lyx(t)-y~(t)l)e-atdt+2e

/( w~(eLrlx

- zl) e -At dt + 26,

where we have used the estimate (1.5). Now the right-hand side of (2.2) can be made less than 3e for Ix - z] small enough, which proves the uniform continuity of v. Next we assume w~(r) = Let and get from (2.1) and (1.5) v(x) - v(z) <_ L d x - z I ~ooT eLte -~t d t + -2-M f(2.3)

e (L-A)T - 1 = L ~ l x - zl

L - ,~

e_AT +

2 M e_AT

+ ~

s

+ 6,

where the last inequality holds only for L # $. If ~ > L, it is easy to get from (2.3), letting T --* +oc and s ~ 0,

Iv(z)- v(z)l <

Le

i x _ zl '

which proves the statement in this case. If ~ < L, we restrict ourselves (without loss of generality) to Ix - z[ < 2 M / L ~ and choose 1 2M T = Z log L~lx--------ff~zl> 0, which minimizes the last term of (2.3) as a function of T. We get v ( x ) - ~(~) < C ~ l x - z] Ix - zl ( A - L ) / L + C21x - zl AlL

+~,

where Ci depend on ~ , L and L~, and then Iv(x) - v(z)l < C3]x - zl A/L, which proves the statement in this case as well.

2. THE INFINITE HORIZON PROBLEM

101

Finally if A = L, we use the first inequality in (2.3) with the choice T = _A-1 log Ix - z I (and we restrict ourselves to Ix - z I < 1) and get

v ( x ) - v(z) < - 5n e-

2M Ix - z I + E I x - z I l o g ] x - z] + T

Then the modulus of continuity of v in this case is wv(r) = C4r]logr], and the proof is complete. 9 REMARK 2.2. In the study of ergodic problems in Chapter VII we need the following estimate. Under the assumptions of Proposition 2.1 with we(r) = Ltr, if L < 0 then for all A > 0 Lt x - z I I v ( x ) - v(z)l < --3

for all x, z E ]I~N

<3

Lipschitz and HSlder continuity of v can be obtained by assuming some controllability of the system instead of a large interest rate. In the next result we assume, for instance, that in a given set/C the following holds: (2.4)

there exist C, y > 0 and, for each x, z E /(:, a control ~1 E M such that yx(tl, ~1) = z for some tl _< C Ix - zl~.

Roughly speaking (2.4) means that the system can reach any point of K: in a time proportional to a power of the distance from the initial point. In the terminology of Chapter IV, (2.4) is equivalent to saying that, for any z E/C, the minimum time function with target set {z} is finite and 7-H51der continuous in/C. In Section IV.1 we give some results on the HSlder continuity of minimal time functions for linear and nonlinear systems. Note that a trivial condition for (2.4) to hold with "y = 1 in K: convex is that f(x, A) D/~(0, 1/C). This condition for Lipschitz continuity can be weakened to ~-6f(x, A) D B(O, r) for some r > 0, as we show in Corollary 2.10. PROPOSITION 2.3. Assume (A0), (A1), (A3), (A4), and (2.4) in a set IC C R N.

Then, for all x, z E 1C,

Iv(x) - v(z)l < M C l x - zl ~ PROOF. Since ~ is bounded it is not restrictive to assume v > 0 because the value function of the problem with r u n n i n g cost t~ - inf t~ > 0 is v - inf ~. For given x, z E/C~ we choose H as in the previous proof and define O~l(t) =

t-

t ~ tl, tl)

t >

tl.

Then y~(s + t l , a 2 ) = y~(s,H) for s _> 0, and

v(x) < J(x, ~2)

-~ .~atl e-At~(yx(t , o~1) , o~1 (t)) dt -~- e -Atl .~a~ e-AS~(yx(S Jr tl, <_ Mtl + e-~t~(v(z) + z) < v(z) + M C I x -

o~2) , ~ ( s ) )

ds

zl ~ + s,

where in the last inequMity we used (2.4) and v > 0. The proof of the claim is easily completed by exchanging the roles of x and z. 9

102

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

EXAMPLE 2.4. The following example shows that the regularity proved in Proposition 2.1 is optimal if we do not assume controllability conditions, and t h a t v(x) is not differentiable in general, even if N = 1, A is a singleton, and t~ and f are C ~c. Let f E C~c(R) be such that f(x) = x for [x I < 1, ]f(x)] < 2 and 0 _< f'(x) < 1 for all x. The system is y' = f(y) = f(y, a). Let A be the discount rate and take g(x,a) = *(x) := A If(x)l~(1

-

f'(x)).

Note t h a t ~ C C ~ (JR) for all A > 0 because g =- 0 in [ - 1 , 1]. Now we compute, for x > 0 (so t h a t f(yx(t)) > 0 for all t) v(z) =

/0

e(yx(t))e-at dt =

--~

I~(yx(t))e -~t

dt = I ~ ( z ) .

A similar calculation for x < 0 shows t h a t v(x) = If(x)l "x. Then v(x) = Ixl ~ in [ - 1 , 1]. Note t h a t L = 1 in this example. <3 The next result is fundamental. As is common nowadays, we use Bellman's terminology and call it the Dynamic Programming Principle, briefly DPP, even if our formulation is more akin to what Isaacs called the tenet of transition. See R e m a r k 2.6 for Bellman's formulation. The D P P expresses as an equation the intuitive belief t h a t to achieve the minimum cost function v(x) it is necessary and sufficient to behave as follows: (a) let the system evolve for a time t choosing an arbitrary control function a on the interval [0, t]; (b) pay the corresponding cost, that is, (c)

/0

f,(yx(S), c~(s))e -~s ds;

pay what remains to pay with the best possible controls, that is, v(yz(t))e-)'t;

(d) finally minimize over all possible controls a. PROPOSITION 2.5 (DYNAMIC PROGRAMMING PRINCIPLE). Assume (A0), (A1), (A3), and (A4). Then for all x E ~ N and t > 0

(2.5)

v(x) = i n f ~ f t ~ ( y ~ ( s , a ) , ~ ( s ) ) e - ~ S d s + v ( y x ( t , a ) ) e - ~ t } . aE.A ~.Jo

P R o o F . Let us name w(x) the right-hand side of (2.5). We first show t h a t v(x) > w(x). For all a E .A we have

(2.6)

J(x, ~) =

e(y~(s), ~(s))e - ~ as +

= I1 +

e(yx(s), ~(s))e -xs as

e(y~(s + t), a(s + t ) ) e - a S e - a t ds

= I~ + e-~tj(yx(t), 8) > h + e-~tv(y~(t)) > w(z), where I~ is the first integral in the right-hand side of (2.6) and 8(s) := c~(s + t). Taking the infimum over A we get v(x) >_ w(x).

2. THE INFINITE HORIZON PROBLEM

103

To prove the opposite inequality, we fix c~ E .A, set z := yx(t, a), and fix E > O and a 1 E .A such that

v(z) > J(z, a l ) _ z . Define the control ~'(8) ---- [0~1(8 -- t)

8 > t,

and let ~ and yl be the trajectories corresponding to ~ and a 1, respectively. Then

(2.7)

v(x) <_ J(x,-~) =

e(~(~),-~(s))e-X~ds+

/o

= I1 + e -~t

e(Yx(S),-~(~))e-aSas

e(y~O-), ~lO-))e-~" a~-

= I1 + e -'~t J(z, c~1) < h + e-'~tv(y~( t, c~)) + e, where /1 is the first integral in the right-hand side of (2.7). Since e and a are a r b i t r a r y we get v(x) <_ w(x). REMARK 2.6. If there is an optimal control c~* for x, that is, v(x) = J ( x , a*), then the infimum in (2.5) is attained at c~*:

(2.8)

/:

v(~) =

e(y~(~, ~*), ~*(~))~-~ ds + v ( ~ ( t , ~*))~-~

for all t > 0. In fact the calculation after (2.6) gives

v(x) =

/o

e(yz(s),a*(s))e-XSds+e-XtJ(yx(t,a*),a*(.

+t))

which shows the inequality ":>" in (2.8). Therefore we also have

vOw(t, ~*)) = JOx(t, ~*), ~*(. + t)), that is, an optimal control for x is optimal for each point of the corresponding trajectory, provided it is appropriately shifted in time. This is the way Bellman formulated the "Principle of Optimality" [Bel71]: "An optimal policy has the property t h a t whatever the initial state and the initial decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision" (Belhnan referred to discrete time processes. In our setting the "initial decision" is the choice of the control in the interval [0, t]). Note also t h a t (2.8) implies that the function

t~---*

/o'

e ( y x ( S , a ) , a ( s ) ) e - X ~ d s + v ( y z ( t , a ) ) e -~t

is constant if and only if a is optimal. It is also easy to show, using the DPP, that this function is nondecreasing for any a E A: we leave the proof as an exercise for the reader. <~

104

III. PROBLEMS W I T H U N R E S T R I C T E D STATE SPACE

REMARK 2.7. Proposition 2.1 holds for w(x) = infaet3 J(x, a) for any B C A. Proposition 2.5 holds for any 13 with the following properties: (i)

if~cB,

r>0,

thentHa(t+T)

isinB;

(ii) i f a l , c ~ 2 C / 3 , s > 0 ,

~(t):=

al(t)

t ~ s,

~2(t-~)

t>~,

then a G B. Hence, for instance, v~(x) = infae p 3(x, a) satisfies (2.5) with A replaced by P.

<~

The Hamiltonian in the Hamilton-Jacobi-Bellman equation for the infinite horizon problem is H : ]~Y X ] ~ g ..._+ ]~ defined by

H ( x , p ) := sup{ - f ( x , a) . p - g(x, a) } .

(2.9)

aEA

Note that H ( x , p ) < +oc for all x and p by (A1) and (A4). The main consequence of the Dynamic Programming Principle is the following. PROPOSITION 2.8. Assume (A0)-(A4).

Then the value function v is a viscosity

solution of )~v + H(z, Dv) = 0

(2.10)

in ]~N.

PROOF. Let r E C 1 ( ~ N) and x be a local maximum point of v - ~, that is, for some r > 0 v(x) - v(z) > r - r for all z 9 B ( x , r ) . Fix an arbitrary a 9 A and let y~(t) be the solution corresponding to the constant control a ( t ) = a for all t. For t small enough y~(t) 9 B(x, r) (by (1.3)) and then r

- r

< v(x) - v(y,(t))

for all 0 < t < t o .

By using the inequality "<" in the D P P (2.5) we get

r162

_<

e(y~(s),a)e-XS ds + v(yx(t))(e - ~ - 1 ) ;

therefore, dividing by t > 0 and letting t ---, O, we obtain, by the differentiability of r and the continuity of v, Yx, f, and ~, - D e ( x ) . y•(0) = - D e ( x ) . f ( x , a ) < e(x,a) - Av(x). Since a 9 A is arbitrary we have proved that

)w(x) + sup { - f ( x , a). D e ( x ) - g(x, a) } < 0, aEA

that is, v is a viscosity subsolution of (2.10). Next assume x is a local minimum point of v - 6, that is, for some r > 0 (2.11)

v(x) - v(z) < r

- r

for all z 9 B(x, r ) .

2. T H E I N F I N I T E HORIZON PROBLEM

105

For each z > 0 and t > 0, by the inequality ">_" in the D P P (2.5), there exists c ,4 (depending on z and t) such that

v(x) >_

(2.12)

e(~x(s),-6(s))e -~s ds + V(~x(t))e -~t - tz,

where ~ ( s ) = yx(s,~), the trajectory of (S) corresponding to ~. Now observe that (1.3) and (Aa) imply

le(~=(s),~(s))-

(2.13)

e(x,~(s))l < w~(Mxs)

for 0 < s < t0,

and (1.3) and (A2) imply

If(~x(s),-6(s)) - f ( x , - 6 ( s ) ) I < w f ( M ~ s , Ixl + Mxto)

(2.14)

for 0 < s < to, where to does not depend on ~, e and t. By (2.13) the integral on the right-hand side of (2.12) can be written as

ote(x,~(s))e-~Sds+o(t)

as t - , O,

where o(t) indicates a function g(t) such that limt_~o+ Ig(t)I/t = O, and in this case Ig(t)[ < ta~e(]tlxt). Then (2.11) with z = ~x(t) and (2.12) give

-

(2.15) ~(x) - r

e(x, ~ ( s ) ) e - ~ ds + v(y~(t))(1 - e -At) >_ - t s + o(t).

Moreover, r

= - f0 t ~dr

- r

(2.16)

= -

/:

Dr

f(~x(s),-6(s)) ds

= - jr0t De(x) 9f(x,-6(s)) ds + o(t), where we used (1.3), (2.14) and r e C 1 in the last equality to estimate the difference between D e . f computed at ~x(s) and at x, respectively. Plugging (2.16) into (2.15) and adding 4- fo ~(x,-6(s)) ds we get (2.17)

I'

{-De(x)

+

9 f(z,~(s)) - e ( x , ~ ( s ) ) }

as

/o'

e(x,~(s))(1 - e - a ~ ) d s + v ( y ~ ( t ) ) ( 1

- e -xt) > - t s + o ( t ) .

The term in brackets in the first integral is estimated from above by sup{ -Dcp(x) . f ( x , a) - g(x, a) } aEA

106

III. PROBLEMS W I T H U N R E S T R I C T E D STATE SPACE

and the second integral is o(t) because t is bounded by (A4), so we can divide (2.17) by t > 0 and pass to the limit to get sup{ - D r

f ( x , a) - g(x, a) } + A v(x) _> - z ,

aEA

where we have also used the continuity of v at x and of ~, at 0. Since r is arbitrary, the proof that v is a supersolution is complete. 9 REMARK 2.9. The proof of Proposition 2.8 shows that any function satisfying the D P P (2.5) with ,4 replaced by any B ___,4, B containing the constant functions, is a viscosity solution of (2.10). Then, for instance, v~ satisfies (2.10) by Remark 2.7. 0 such that B(O, r) g -c-6f(x, A). Then the value f u n c t i o n v is Lipschitz continuous in IRN. PROOF. Since v is a bounded solution of (2.10) we can apply Proposition II.4.1. Indeed, under the present assumptions we have sup

H(x,p)>_snp{-f(x,a).p}-M= aE A

>_ s u p { - q . p } - M = r l p Iql_
{-q.p}-M

qEUdf (x,a )

I- M .

9

We know from the section on differential inequalities in the viscosity sense in Chapter II that the value function v satisfies v(x) > u(x), at any x, for all subso. lutions u of (2.10). Therefore, by Proposition 2.8, v is the maximal subsolution of (2.10), so it is characterized by the Hamilton-Jacobi-Bellman equation. Since v is also a supersolution of (2.10), we are now going to characterize v as the unique solution of (2.10). We do this via a comparison principle which also implies that v is the minimal supersolution and gives a completely different proof of the "maximal subsolution" characterization previously recalled. We premise a technical lemma on the regularity of the Hamiltonian H defined by (2.9). LEMMA 2.11. A s s u m e (A0)-(A4). Then H is continuous and it satisfies (2.18)

H ( y , # ( x - y) - r y ) - H ( x , # ( x - y) + ~fx) < # n Ix - y[2 + r K ( 1 + ly[ 2) + ~/g(1 + Ix[~) + w~([x - Yl)

for all tt, r, 7 > O, x, y E R N, where K := n + sup{ if(O,a)] : a E A }. Moreover, if (1.1) holds, then H satisfies

(2.19)

[H(x,p) - H(y,p)] <_ IPl n Ix - Yl + w~(ix - Yl)

f o r all x, y, p E ~ g .

(2.20)

If~ in addition,

5M' :

If(x,a)[ <_ M '

for all x E IRN, a E A,

then

(2.21)

]H(x,p) - H ( x , q ) [ <_ M'Ip - q[ .

2. THE INFINITE HORIZON PROBLEM

107

PROOF. Observe first that (A3) implies

< L Iz[ 2 + suplf(0, a)l Izl <_ K(1 + Iz]2) aEA for all z E ~Y, a E A. Now fix s > 0 and ~ such that (2.22)

f(z,a).z

H(y,p(x-

y) - 7y) < - f ( y , ~ ) . ( # ( x - y) - Ty) -- e(y,~) + S .

Then the left-hand side of (2.18) is majorized by ,(f(x,

~) - f ( y , ~ ) )

9 (x - y) + ~ f ( y , ~ )

9y + ~f(x,~)

9 x - ~(y, ~) + ~(x, ~) + ~,

which can be estimated by the right-hand side of (2.18) plus E by (A3), (A4), and (2.22). Since z is arbitrary the proof of (2.18) is complete. The proofs of the other statements are similar and are left to the reader. Lemma 2.11 implies that if f satisfies (A0), (2.20), and (1.1), then the Hamiltonian H satisfies the assumptions of Theorem II.3.5, which states that a subsolution of (2.10) is nowhere larger than a supersolution, provided both belong to BUC(]~N). Since, by Proposition 2.8, the value function v is both a viscosity subsolution and a supersolution of (2.10), we conclude that v is below any supersolution in BUC(]~ N) and above any subsolution in BUC(~N). Hence v is the unique viscosity solution of (2.10) in BUC(]~ N) and, more precisely, it is at the same time the maximal subsolution and the minimal supersolution, a property that we summarize by saying that v is the complete solution of (2.10) in BUC(RN). All this holds under the more general assumptions (A0)-(A4) as well, as the next theorem states. THEOREM 2.12. Assume H : ]~2N ~ ]~ is continuous and satisfies (2.18). If Ul, U2 E BUC(]I~ N) are, respectively, a viscosity sub- and supersolution of (2.23)

u + g ( x , Du) = 0

in ]t~N,

then ul <_ u2 in ~ g . In particular, if H is given by (2.9) and (A0)-(A4) hold, then the value function v is the complete solution of (2.10) in B U C ( ~ N) and thus it is also the unique viscosity solution in the same class. PROOF. The second part of the theorem follows from Proposition 2.8 and Lemma 2.11: it is enough to divide (2.10) by A > 0 and observe that H/)~ satisfies the structural assumption (2.18) (with the constants and the modulus ~;~ divided by A, which does not affect the following proof). The proof of the comparison assertion is a variant of the proofs of Theorems II.3.1 and II.3.5; we consider 9 : ~2N _._. ]~, (~(X, y) : : Ul(X ) -- u2(y )

IX -- yl2 2s

~((x)rn .~_ (y)rn),

where (x) :-- (1 + Ix[2) 1/2, and ~, ~, m are positive parameters to be chosen conveniently. Note that for ~ -- 0 we recover the auxiliary function 4) of Theorem II.3.1.

108

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

Let us assume by contradiction that there is 5 > 0 and 5 such that u1(5) u2(5) = 5. We choose j3 > 0 such that 2/3(5) _< 5 / 2 , so that, for all 0 < m < 1, (2.24)

5 ~ < 5 - 2r

rn = (I)(5, 5) _< sup (I)(x, y ) .

Since 4) is continuous and lim[xd+ly]~or (I)(x, y) = - o c , there exist 5, ~ such that (2.25)

0(5, ~) = sup (I)(x, y ) .

From the inequality 6#(5, 5) + 4)(~, ~) < 24)(5, ~) we easily get

(2.26)

- -

E

__~ U l ( X ) -- u l ( Y ) -}- U2(5) -- lt2(Y) 9

Then the boundedness of ul and us implies 15 - Y l - < c v ~

(2.27)

for a suitable constant c. By plugging (2.27) into (2.26) and using the uniform continuity of Ul and us we get (2.28)

1 5 - ~1~ _< w ( ~ ) , E

for some modulus w. Next define the C 1 "test functions" ~(x) := ~2(~) +

~(y) := ~,(5)

rx _ ~[2 + Z(<x> m + (~)r~), 2----7--

15- yl 2 2z

~(<5>,n+ ~),

and observe that, by definition of 5, ~, Ul - ~2 attains its maximum at 5 and us - ~b attains its minimum at ~. It is easy to compute Dcfi(2) -- --x - y + "72,

"7 = /3rn(5) rn-2,

C

DO(~) -

x - y

-c~,

v = ~ m ( ~ ) rn-2 .

By definition of viscosity sub- and supersolution, (2.29)

uI(5) + H(5, D~(5)) _< 0 < u2(~) + H(y, Dr

and by using (2.18) with # = l/e, ~-, ~ as above, and the definition of (x), we get L

~t1(5) -- u2(Y) ~ 7 CX -- y l 2 + W / l " < y ) 2 %- " / g ( x ) 2 + Wg(15 -- y[)

_< L ~ ( v ~ ) + ~mK((5) m + (~)m) + ~ ( c v ~ ) , where in the last inequality we have used (2.27) and (2.28). Therefore, by choosing 0 < m < 1 / K , we obtain (5, y) _< u 1(5) - u2 (Y) - / 3 ( {5>m +
2. THE INFINITE HORIZON PROBLEM

109

REMARK 2.13. The uniform continuity assumption on ul and u2 in Theorem 2.12 can be easily replaced by mere continuity, so that v is the complete solution of (2.10) in B C ( R N ) (bounded and continuous functions ~ N __, R) as well. To prove this observe that, once/3 is chosen, we have the estimate

(~)"

+

(~)~

<

sup u] - inf u2 fl

= : co,

for all r > 0, m E ]0, 1], because the violation of this inequality leads to 9 (5, ~) ~ 0, a contradiction to (2.24) and (2.25). Then, for all r > 0, 5 , ~ are in the compact set B(0, c oUrn"), where m = rain{l, l / K } , and so the uniform continuity of ul, u2 in such a set is enough to deduce (2.28) from (2.26) and (2.27), which is the only step where we used the uniform continuity of Ul, u 2. <3 REMARK 2.14. Theorem 2.12 and Remark 2.13 are easily extended to a comparison result for a Dirichlet b o u n d a r y value problem that is useful in Section 3 and Chapter IV. If H is continuous and satisfies (2.18), ~ C_ ] ~ g is an open set, u l , u2 E B C ( - ~ ) are, respectively, a viscosity sub- and supersolution of u + H ( x , D u ) in ~ and Ul _< u2 on O f , then ul _< u2 in ~ . The only difference in the proof is that the m a x i m u m point of 4), (~,~), may not belong to f x ft. If 9 E 0ff, then the b o u n d a r y inequality holds (I)(~,~) ~ Ul(~ ) -- u2(x) Jr" u2(~) -- u2(y) ~ ~ ( C v ~ ) , where ~ is the modulus of continuity of u2 in /J(0, c 1 / m ) (see the proof of Remark 2.13), and we have used (2.27). By choosing r small enough we get a contradiction to (2.24) and (2.25). If 5 c f and ~ C 0 f we add and subtract ul(~) and use the continuity of Ul to conclude in the same way (see also the proof of Theorem I1.3.1). <3 REMARK 2.15. An immediate consequence of Theorem 2.12 and Remarks 2 . 7 and 2.9 is the equality v = v ~, that is, measurable control functions and piecewise constant control functions generate the same value function. In w we prove in a rather different way that also Lipschitz continuous controls generate the same value function. <3 We end this section with some comments on the optimality of the assumptions of the comparison Theorem 2.12 for Hamiltonians of the form (2.9). First note that we cannot drop the assumption t h a t Ul and u2 are both bounded because the linear 1-dimensional equation u - x Ux = 0 in ]R has the classical, uniformly continuous solutions ul ~ 0 and u2 = x. It has also the viscosity solutions u3(x) = Ix] and u 4 ( x ) = - I x ] which are, respectively, bounded below and bounded above and remain solutions in a r b i t r a r y dimension for u - x 9Du = 0

in ]I~N .

We remark that there are comparison and uniqueness results in classes of unbounded solutions, typically under suitable restrictions on the rate of growth at infinity (see, e.g., Remark II.3.6).

110

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

T h e a s s u m p t i o n of uniform c o n t i n u i t y of ~ in (A4) can be relaxed to mere continuity, see Exercise 2.7. Now let us focus on (An) by looking at d i m e n s i o n N = 1, A a singleton, so t h a t the H a m i l t o n - J a c o b i - B e l l m a n e q u a t i o n is linear, and f(x) a power of x. T h e "decreasing case," f(x) : = - s g n x Ix[ b, x E R, satisfies (A3) for any b > 0 (with L = 0), so T h e o r e m 2.12 applies to

u+sgnxlxlbuz = 0

inI~

a n d u -- 0 is the u n i q u e b o u n d e d solution. T h e "increasing case", f(x) := sgn x Ix[b, b > 0, satisfies (A3) if a n d only if b -- 1, a n d t h e n u -- 0 is the only b o u n d e d solution of u - sgn x [xlbux = 0

in

if b = 1. Here is a second classical solution in BUC(]~) when b > 1:

u(x) : = exp ( ~ )

if x ~ 0, u(0) = 0,

as it is easy to check since ux(x) = sgnxlxl-bu(x) i f x r 0 a n d ux(0) = 0. Note t h a t for vector fields C 1 in the x variable the inequality in (A3) always holds if x a n d y are restricted to a compact set ]C, for a suitable constant LK:. Thus, for s m o o t h f , (An) is essentially a growth condition at infinity, saying, roughly speaking, t h a t there is no restriction if the fields are m o n o t o n e nonincreasing, whereas the growth has to be at most linear in the opposite case. The preceding example says that this a s s u m p t i o n is essentially o p t i m a l for the comparison Theorem 2.12, at least for fields having a polynomial growth at infinity. 2.2. S o m e s i m p l e a p p l i c a t i o n s : verification theorems, relaxation, stability In this subsection we draw some simple consequences on the infinite horizon optimal problem from the results on the associated H a m i l t o n - J a c o b i - B e l l m a n equation of the previous section. We begin with some necessary and sufficient conditions for optimality of a control law or presynthesis which is defined as follows. DEFINITION 2.16. A control law or presynthesis on a set ~'/___]~N is a m a p A : l'~ --~ .4, t h a t is, it associates with each point x E ~ a control function A(x) = : az. It is optimal (on ~'/) if the cost associated with it, t h a t is, JA(x) : = J(x, c~x), satisfies

J•(x) = m i n J(x, a) = v(x) ~EA

for all x E ~ .

<~

The most i m p o r t a n t examples of control laws are generated by feedback maps (I) : f~ - , A, provided the feedback is admissible in the following sense.

2. T H E I N F I N I T E HORIZON PROBLEM

111

DEFINITION 2.17. A feedback map on a set ~ C ~ N ~ : ~ ._~ A, is admissible if for all x E Ct there exists a unique solution Yx(', ~) on [0, + ~ [ of y, = f ( y , ~ ( y ) ) y(0) = x

such that t ~ 4)(yx(t, ~)) is measurable and y~(t, 4~) e i2 for all t > 0.

<~

It is natural to associate the following control law with an admissible feedback map ~x(. ) : = r

;r

e ,4,

and call it a feedback law; ax in this case is called a closed-loop control. A feedback law has the property of being memoryless, that is, for any z = yz(t, ax), a~(. ) = az ( 9 + t). A memoryless presynthesis defined on all ~N is called a synthesis. Before stating the conditions of optimality of a control law we recall the Hamilton-Jacobi-Bellman equation (2.30)

Au + sup{ - f ( x , a) . D u - g(x, a) } = O . aEA

In the next result by subsolution we mean a bounded and uniformly continuous viscosity subsolution, and we use the same convention for supersolutions and solutions. By Remark 2.13 "uniformly continuous" can be replaced by "continuous." COROLLARY 2.18. A s s u m e (A0)-(A4) and let A be a control law on a set f~. Then (i)

(Necessary condition for optimality) I f A is optimal and f~ is open, then the associated cost JA is a viscosity solution of (2.30) in ~.

(ii)

(Sufficient condition for optimality) I f there exists a subsolution u of (2.30) on ]~g, such that JA <- u on l-l, then A is optimal.

(iii)

(Sufficient conditions for non-optimality) I f there exists a supersolution w of (2.30) on l~ N, such that JA > w at some point o f f l , then A is not optimal.

(iv)

In particular, f o r fl = m y , the optimality of A is equivalent to each of the following facts: JA is a subsolution of (2.30) in l~Y ; JA is a solution of (2.30) in ]~g ; JA <_ w f o r any supersolution w of (2.30) in ]~g .

PROOF. All the statements follow from the equivalence of the optimality of A and the equality JA = v, where v is the value function. Thus (i) follows from Proposition 2.8 and the fact that being a viscosity solution is a local property; (ii) follows from Theorem 2.12 because v <_ JA < u < v

inft;

(iii) follows from the same theorem because v _ < w < JA; and (iv) is also a trivial consequence of Theorem 2.12.

112

III. P R O B L E M S W I T H U N R E S T R I C T E D STATE SPACE

Corollary 2.18 and the classical verification technique recalled in Chapter I suggest the name (generalized) verification function for any viscosity subsolution u E BUC(]~N): to verify that a control law A is optimal in ]t~N it is enough to find such a function satisfying u > JA. By analogy we call a falsification function a viscosity supersolution w E BUC([(N), because the existence of such a function satisfying w < JA implies the non-optimality of A. It is easy to give a similar verification theorem valid for open loop controls a E A: we say that a0 E .4 is optimal for x0 E B y if J(xo, (~0) = min~eA J(x0, a) = v(xo). COROLLARY 2.19. Assume (A0)-(A4). (i)

A control ao is optimal for xo if and only if there exists a verification function u such that J(xo, ao) < u(xo);

(ii)

cro is not optimal for xo if and only if there exists a falsification function w such that J(xo, ao) > w(xo).

PROOF. The very easy proof is left to the reader. -k -k -k

Next we want to show that the value functions of two different "relaxed" problems coincide with v. These results are consequences of the following simple observation. Consider a control system f : R N x .4 --~ ] ~ N and running cost ~satisfying the structural conditions (A0)-(A4) (maybe with different constants), and the corresponding Hamiltonian

Yt(x,p) := s u p { - ] ( x , a).p - ?(x, a)}. aE,4

If H ( x , p ) = H ( x , p ) for all x , p E ~ N then the corresponding value function satisfies ~(x) = v(x) for all x E l~g. In fact v and ~ satisfy the same HamiltonJacobi-Bellman equation (2.10) and then they must coincide by Theorem 2.12. The first relaxation result concerns the convexified problem. Let us begin with a simple case. Assume the system is aJfine, that is,

f ( x , a ) = f l ( x ) + f2(x)a

for all x E ]RN, a E A,

where A C R M and f2(x) is an N • M matrix, and the running cost is concave in the control variable, that is, -/~ can be extended to all of R N • ]l~M as a convex function of a for each fixed x. We consider the same system and cost, but for the larger control set .4 = coA (i.e., the convex hull of A), and we call ~ the corresponding value function. We claim that v = ~, and to prove the claim we have only to check that H > H because the inequality H < H is trivial. Fix x , p E ]~N C > O, and b E .4 such that _~(x,p) - E < - f l ( x ) "p - f 2 ( x ) b . p - g(x, b).

2. T H E I N F I N I T E H O R I Z O N P R O B L E M

r~

a.

113

n

We write b = Y~i=l ,~i ~ with ai E A, ,~i >- O, ~-]~i=l,~i = 1, and obtain, by the concavity of g in the control variable, n

~I(x,p) - s < - f l ( x ) "p - Z

n

,~if2(x)ai "p - Z

i=1

,kig(x, ai)

i=1

n

<_ ~--~Ai s u p { - f l ( x ) " p i=l

f2(x)a.p-

g(x,a) } = H ( x , p ) ,

a6A

which completes the proof of the claim after letting e ---, 0 +. The meaning of the result is clear: the value function does not change if we allow the controls to take values just on the vertices of a polyhedron rather than on the full polyhedron. This is a weak (nonlinear) version of the classical bang-bang principle. For general system and cost, the convexified problem is the following. Let N+2

A := ( A E ]~N+2 : )~i > OVi, Z

)~i = I },

AC = A x A N+2,

i=1

and define fc and gc on ]RN x A c by N+2

fc(x,

~, a l . . . . , aN+2)

= Z

N+2

Air(x, ai),

gO(x,A, al . . . . , aN+2) = Z

i=1

,kig(x, ai).

i=1

It is easy to check that f e U satisfy (Ao)-(A4). Their crucial property is that (2.31)

e o ( f ( x , A ) x g ( x , A ) ) = f ~ ( x , A c) x gO(x, AC),

for all x E N N. In fact, by a classical theorem of Carath@odory (see [Ce83]), if X C ]R~, then every element of the convex hull c o X can be written as a convex combination of at most n + 1 elements of X. Using (2.31) it is easy to show, as previously, that the Hamiltonian corresponding to f c gc A c coincides with H, so the value functions of the original and of the convexified problem are the same. We recall that convex problems are interesting because they have some special properties; for example, if A is compact, the set of their trajectories is closed with respect to the uniform convergence over a bounded time-interval.

The second relaxation result concerns the enlargement of the set of measurable control functions .4 to the class of relaxed or chattering controls, that is, ..4r

:~-

{ measurable functions [0, +oc[ --~ A r },

where A r := '{ Radon probability measures on A }

114

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

and we are assuming that A is a compact subset of ]~M.2 To give sense to the system (S) and the cost J for controls in A r we extend f and e to RN x A r by setting

fr(x,m):=/Af(x,a)dm, ~r(x,

V r a E A r,

m) := JA ~(x, a)dm,

V m E A r,

and observing that i f ( x , ha) -'- f(x, a), gr(x, ha) -- g(x, a), where 5~ is the Dirac measure concentrated at a. If we identify a E A with 5a E A r we can view f r and gr as extensions of f and g, respectively. We also endow A r with a suitable topology by regarding it as a subset of C(A)*, the dual space of the space C(A) of all continuous functions A ~ ~ , with the weak star topology. It can be proved t h a t A r with this topology is metrizable and compact: see [Wa72]. We recall that a sequence mn converges weak star to m if

limfA~(a)dmn=fA~(a)dm LEMMA 2.20. If f, satisfy (A0)-(ha).

2, A satisfy

(A0)-(Aa)

for all ~ E C(A) .

and A C_~ i is compact, then fr, ~r, A T

PROOF. It is enough to check the continuity of f r and ~r on sequences (xn, rnn) --* (x, ra) in lt~N • A r. By definition of weak star convergence

fAf(x,a)dmn

' f A f ( x , a ) d m = fr(x,m),

whereas, for R such that Ixn] < R for all n,

fr(xn'rar')--

fA f(x,a)drnn

fA[f(xn'a)- f(x'a)}dmn

=

<- ./A wy(Ixn - xl' R) dmn =

~s(Ix~

-

xl, R) ~

o,

because fA dm,~ = 1, thus fr(xn, m,~) ---.+i f ( x , ra). The same proof works for gr. The other proofs are straightforward; for instance, (A3) holds because (fr(x, Ti2)

fr(y,m)). ( x - y) = ~(f(x,a) - f(y,a)). ( x - y) dm

<-fA

L Ix - y[2 d r n = L Ix - yl 2 .

,11

2A p r o b a b i l i t y m e a s u r e on A is a R a d o n measure if it is defined on all Borel sets of A (i.e., on the smallest a - a l g e b r a of subsets of A t h a t contains the open sets).

2. T H E I N F I N I T E H O R I Z O N P R O B L E M

115

We keep the notation yz(t) or yx(t, #) for the trajectory of the relaxed system, that is, (S) with f replaced by fr and a by # E .Ar, and note that, by definition, it is a solution of

y(t) = x +

f(y(s), a) dp(s) ds;

similarly the cost and the value function of the relaxed problem are

Jr(x, it) :=

S/

gr(y:(t), #(t))e -;~t dt =

v r ( x ) : = inf J r ( x , # ) .

io'l.

g(yx(t), a) dp(t)e -•t dt,

#EA r

Note that the relaxed trajectory and cost associated with #(t) := ~(t) for all t, where a E A, coincide with Yz( 9 c~) and J(x, a), respectively. COROLLARY 2.21. If f, ~, A satisfy (A0)-(A4) and A C_ R M is compact, then v = vr; that is, the value functions of the original and the relaxed problem coincide. PROOF. Let

Hr(x,p) := sup { - f r ( x , m ) . p - gr(x,m) }. rnEA r

By Lemma 2.20 and Propositions 2.1 and 2.8, v r E BUC(~ N) and it is a viscosity solution of Av + Hr(x, Dv) = 0 in R N. We prove that H -- H r so that the conclusion follows immediately from Theorem 2.12. For all x,p E R N we have

Hr(x,p) >_ sup{ - i f ( x , 3a) " P - gr(x, 5a) } = H ( x , p ) aEA

by the definitions of f r ~?r and H, see (2.9). On the other hand, for all x,p E ]~N,

mEAr. - i f ( x , m) .p - gr(x, m) = ~A(--f(x ~a) .p -- g(x, a)) dm -< fA sup{ --f(x, b) .p - ~(x, b) }dm = H(z,p), bEA

because fA dm ----1. Now we take sup of both sides and get Hr(x,p) < H(x,p). mEA r 4

We point out that relaxed controls are interesting because the relaxed problem usually has an optimal control whereas the original one does not in general. In Chapter VI we construct an optimal relaxed control for the problem discussed here as the limit of piecewise constant controls computable via Dynamic Programming methods.

116

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

Finally we remark t h a t considering relaxed controls is another way of convexilying the problem. In fact, it is not hard to prove that

~-6(f(x, A) • g(x, A)) = i f ( x , A r) • ~r(x, Ar),

(2.32) see Exercise 2.10.

Next we prove a simple stability property of the value function v which is an immediate consequence of the stability of viscosity solutions with respect to the uniform convergence of the Hamiltonians. We consider a family of systems Y'=fh(Y,a), y(o)

a(t) E Ah, t > O,

= x

and costs

Jh(x, a) =

e-~hteh(yx(t), a(t)) dt, ~0 ~176

where the parameter h varies in [0, 1]. We want to show that if the d a t a fh, Ah, Ah, th are continuous in a suitable sense for h --* 0 +, then the value function Vh(X) : = inf~EA h Jh(x, a) is also continuous as h ~ 0 +. We assume that for all h E [0, 1] fh satisfy (Ao)-(A3) with the same compact A C_ ]~M and the same constant L in (A3), and that gh, Ah satisfy (A4) with the same constant M and modulus wt. We also assume that fh --* fo, th --* tO, as h --~ 0 +, uniformly on K~ x A for each compact K: _ ]RN, and that limh--,o Ah = )~0. To define the convergence of the control sets Ah to Ao we need the notion of Hausdorff distance of two compact sets S, U _ ]~M:

ds(S, U) := max{ maxdist(s,ses U), umea~dist(u, S) }, where dist(s, U) = rain{ Is - u[ : u E U }. Note that

dH(S,U)
ifandonlyif

SC_JB(U,r)

and

UC_B(S,r),

w h e r e / ~ ( X , r) -= { x E R M : dist(x, X ) < r }, so in particular dH(S, U) = 0

if and only if

S = U.

We assume that:

Ah C_ A are compact for all h E [0, 1], lim dH(Ah, Ao) = 0. h~0

COROLLARY 2.22. Under the preceding assumptions Vh converge to vo as h --~ 0 +

uniformly on each compact set of ]~N.

2. THE INFINITE HORIZON PROBLEM

117

PROOF. Fix h > 0 such that Ah > A0/2 for all h E [0,hi. By Proposition 2.1 Vh E BUC(IR N) for all h, and it is easy to see from the proof of Proposition 2.1 that for h E [0, h] there is a uniform bound and a common modulus of continuity for all Vh, depending only on M, A0, L, we. By the Ascoli-Arzel~ theorem, for every compact K: C ]I~N there is h~ --+ 0 such that Vh,, converges uniformly on ~. We construct a sequence converging on all ]~N by the following standard "diagonal procedure": let h (1) be such that Vh~lI

converge uniformly to u (1) on/?(0, 1), and define by induction an h (m) subsequence of h (m-l) such that vh~m) converge uniformly to u (m) on /~(0, m). By construction u (m) -- u (m-l) on/~(0, m - 1), thus u ( x ) := u(m)(x) if IxI < m is the uniform limit of Vhr on every compactum of ]Ry. We claim that u is a viscosity solution of

A0u + H0(x, D u ) = 0

in l~Y~

where H h ( x , p ) = maxaeAh{ --fh(x, a) . p -- eh(X, a) }, h e [0, 1]. This follows from Proposition II.2.2, once we have checked that Hh converge uniformly to H0 on compacts of IR2N. For this purpose we pick ah E Ah such that H h ( x , p ) = - - f h ( x , ah) " p -- eh(X, ah)

and a0 C A0 such that ]ah -- aol = dH(Ah, Ao). Then H h ( x , p ) - H o ( x , p ) <_ --fh(x, ah) " p -- eh(X, ah) + fo(x, ao) " p + t0(x, a0) <_ lpl(ffo(x, ah) -- fo(x, ao)l + [fh(x, ah) -- fo(x, ah)l)

+ le0(x, ah) - e0(~, a0)l + lea(x, ah) - Co(x, ah)l. The right-hand side of this inequality tends to 0 as h goes to 0, uniformly for Ixl, IPl -< R, because Ifo(x, ah) -- f0(x, a0)l _< aJo(lah -- aoJ) = ~zo(dH(Ah, A0)),

where aJ0 is the modulus of continuity of f0 on/7(0, R) x A, whereas Ifh(x, ah) -f0(x, ah)l goes to 0 by the uniform convergence of fh to f0, and the same arguments hold for the terms involving e. A similar argument on Ho - Hh proves the desired convergence Hh --, Ho. Now the uniqueness Theorem 2.12 implies that u -- v0, so we have the compactness of Vh and the uniqueness of the limit of subsequences, and this implies that Vh ~ Vo by the following standard argument. Assume by contradiction there exists/C compact such that limsup IlVh -- v0ll = ~ > O, h~O

where II" II is the sup-norm in/C. Then there exists h,~ --* 0 such that (2.33)

l~l II~ho -- v011 = 5 .

But the previous arguments imply the existence of a subsequence of Vh, converging to v0 uniformly on each compactum of N N, in particular/C, which contradicts (2.3a). ~1

118

III. PROBLEMS W I T H U N R E S T R I C T E D STATE SPACE

Corollary 2.22 is essentially a perturbation result: if we are aware of the possibility of errors in the knowledge of our d a t a f, A, A, s but the errors are small and do not affect the structure of the problem or the regularity of the data, then the value function is affected only by a small error. In Chapter V we also allow more general perturbations, and in Chapter VI we give estimates of the errors. Corollary 2.22 can also be used to give approximation results. In the proof of the next corollary we smooth the d a t a and "fatten" the control set A and get an approximation by more regular value functions. COROLLARY 2.23. Assume f,A,~, A satisfy (A0)-(A4), and that for all x E IRN

there exists a~ E A such that f(x, ax) = O. Then there exist sequences fn, ~n, An, and a control set A, such that the corresponding value functions vn satisfy: (i)

v~ is locally Lipschitz continuous;

(ii)

v~ is semiconcave (provided that f is bounded and, for L as in (A3), ,~ _> 2L ) ;

(iii)

vn ---+v as n ~ +oc locally uniformly in l~g.

PROOF. To get the Lipschitz approximation we set An =/~,/~n = ~ for all n, ,4 := Ax{bE]R N:lbi_
fn(x,a,b) = f ( x , a ) + l b . n

By the assumption on f we have fn(x,A) D B(O, 1/n), therefore lipschitzean by Corollary 2.10. The semiconcave approximation is setting A~ = A + 1/n and by smoothing f and g with respect to x the usual mollifiers, that is, nonnegative t)~ E C~ N) such that fNN and supp Q~ = / ? ( 0 , 1/n). Then

fn(x,a,b) := s

v~ is locally obtained by by means of

Qn(x) dx = 1

Q~(X-- y) f ( y , a ) d y + l

g,~(x,a,b) := /ll~N p,~(X -- y)e(y,a)dy, are C~ N) for a and b fixed and it is easy to check that they satisfy (A0)-(A4) with the same constants and moduli. It is well known (and not hard to check) that f,~, gn converge uniformly to f, g, respectively, on K: x A for each compactum ]C. The Hamiltonians

Hn(x,p) := s u p {

-f,~(x,c)

. p - fn(x,c)

}

cEA,~

satisfy the assumptions of Theorem II.4.9 (with constants C depending on n), as it is easy to check (see also Remark II.4.10), thus Vn are semiconcave. The convergence of v,~ to v follows from the uniform convergence on compacta of Hn to H, as in the previous corollary. 9

2. THE INFINITE HORIZON PROBLEM

119

2.3. Backward Dynamic Programming, p r i n c i p l e s , bilateral solutions

sub- and superoptimality

In this subsection we discuss some further results about the Dynamic Programming optimality principle. First we derive from it a similar inequality concerning the backward trajectories of the system and prove t h a t it implies that the value function v is also a viscosity supersolution of minus the Hamilton-Jacobi-Bellman equation (recall t h a t in general it is not allowed to change the sign of a PDE in the viscosity sense, see Example II.1.6). Then we show t h a t certain local versions of the DPP, named (local) sub- and superoptimality principles, are equivalent to the definition of viscosity solutions. This is the basic step to proving, in the next section, the equivalence of viscosity solutions with other concepts of weak solution of the Hamilton-Jacobi-Bellman equation. Let us first define . 4 - = { a : ] - o c , 0] --~ A. measurable } and recall that under the assumptions (A0), (A1), (Aa) there exists for all negative times the backward trajectory of the system with initial point x, that is, the unique solution of

S y'(s) = f ( y ( s ) , a ( s ) ) y(0) = x,

(S-)

s < 0,

or, more precisely, the unique solution of

y(s) = x -

f(y(T), a(T)) dT,

s < 0.

The proof is the same as t h a t for forward trajectories recalled in the appendix of this chapter, w We keep the notation yx(s, a) = yz(s) for the solution of ( S - ) . DEFINITION 2.24. We say t h a t x is an optimal point, and write x E O, if there exist Z E ]~N O:* C .4 such that v(z) = J ( z , a*), and t > 0 such that x = yz(t, a*). <~ PROPOSITION 2.25 (BACKWARD DYNAMIC PROGRAMMING PRINCIPLE). A s s u m e (A0), (A1), (A3), and (A4). Then, for all x E I~N, t > 0 and a E . 4 - ,

(2.34)

v(x) >_ v ( y ~ ( - t , c~)) e at -

e ( y ~ ( - s , c~), a ( - s ) ) e as ds;

moreover, for x E (.9 and for all t > 0 small enough, (2.3a)

v(x) = max ( v ( y x ( - t , ol)) e At ~EA-

e As ds

I

PROOF. Fix any x, t and a E , 4 - , let a l ( s ) := a ( s -- t) for 0 < s < t and extend it arbitrarily for s > t so t h a t a l E A. If z = y z ( - t , a ) , then x -- yz(t, al). Then the D P P Proposition 2.5 gives

(2.36)

v ( z ) <_

e ( y ~ ( s ) , C ~ l ( s ) ) e - a ~ d s + v ( y z ( t , a l ) ) e -~t .

120

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

We m u l t i p l y b o t h sides by e ~t, make the change of variable s = t - T in the integral, a n d use yz(S, al) = yx(s - t,a) to get (2.34). To prove the second part we write x -- yz(t, a*) with a* optimal for z, define a2(s) = a*(s + t) for - t < s < 0 a n d e x t e n d it arbitrarily for s < - t so t h a t a2 E . 4 - . By construction z = y x ( - t , a2). Now we write the D P P in the form (2.8) (see R e m a r k 2.6) a n d get the conclusion by the same calculations as in the first part of the proof. ~, REMARK 2.26. It is easy to see t h a t (2.34) is actually equivalent to the suboptireality principle (2.36), and b o t h are equivalent to the fact that

t~-*

e(y~(s,a),a(s))e-~S ds + v ( y x ( t , a ) ) e -~t

is nondecreasing for any ~ E .4. We recall a definition from w

<1 of C h a p t e r II.

DEFINITION 2.27. Let f~ C_ ]~Y be open, u E C ( ~ ) , x E f~. We say t h a t u is a bilateral (viscosity) subsolution (respectively, supersolution) at x of

F ( x , u, Du) = 0

(2.37)

if it is a subsolution (respectively, supersolution) of (2.37) as well of

- F ( x , u, Du) = 0; t h a t is, F ( z , u(x),p) = 0 for all p E D+u(x) (respectively, F ( z , u(x),p) -- 0 for all p ~ D-u(~)). If u is simultaneously a bilateral sub- a n d supersolution at x, we say t h a t u is a bilateral solution at x. <~ Of course, equivalent definitions can b e given in terms of test functions. Note also t h a t a viscosity solution is a bilateral solution at all points where it is differentiable, by L e m m a II.1.8. As a consequence of the Backward D P P we o b t a i n COROLLARY 2.28. Assume (A0)-(A4).

Then the value function v is a bilateral

supersolution of

in ]~N,

Av + H(x, Dv) = 0

where H is given by (2.9), and it is a bilateral subsolution of the same equation at any x E (9. PROOF. T h e a r g u m e n t is essentially the same as in the proof of Proposition 2.8. If ~ E C1(~ N) a n d v - ~2 has a local m i n i m u m point at 5, then, by (2.34), for any aEAand t > 0 ~(x) - ~ ( y ~ ( - t ) ) > v(x) - v ( y ~ ( - t ) )

_> (e ~ - 1) v ( ~ ( - t ) )

-

I'

e(y~(-s), ~(-~)) e~ d~.

2. THE INFINITE HORIZON PROBLEM

121

We choose a(s) =- a, divide by t, and pass to the limit as t --* 0 + to get

D ~ ( x ) . y~(O) = D ~ ( x ) . f(x, a) > Av(x) - *(x, a). Then, by the arbitrariness of a E A,

-Av(x) + inf { f ( x , a ) . D~(x) + ~(x,a) } > 0 LEA

and the left-hand side is precisely -Av(x) - H(x, D~(x)). The second statement is proved by the argument of the second part of the proof of Proposition 2.8, and it is actually easier because there exists an optimal control: we leave it as an exercise for the reader. 9 Here is an application to the regularity of the value function. COROLLARY 2.29. Assume (A0)-(A4) and let x be a point where H(x, 9) is strictly

convex. Then D - v ( x ) has at most one element, and if x E O, D+v(x) also has at most one element. PROOF. The proof is easily obtained by contradiction using the convexity of 9

D+v(x) and D - v ( x ) and Corollary 2.28.

Next we give a simple example showing that the set of optimal points O, though large, is not all of ~ g . EXAMPLE 2.30. Let N = 1, f(x, a) = a, A = [-1, 1], g(x, a) = max{ 1 - Ix], 0 }, = 1. It is easy to guess that an optimal control for x > 0 is a(t) -= 1 and, for x _< 0, a -- - 1 . By computing the cost corresponding to this choice we get

v(x) =

e xl-l-lxl 0

for Ix I < 1, for Ix I _> 1.

The Hamilton-Jacobi-Bellman equation is (2.38)

v+lv'l-max{1-1xl,O}=O

in~

and v is a classical, and therefore bilateral, solution for all x r 0. At x -- 0 we have D-v(O) = 0 and D+v(O) = [e-1 - 1, 1 - e -1] and it is immediate to check that v is a subsolution of (2.38) but not of "minus (2.38)." Indeed, 0 is the only point that does not belong to the relative interior of an optimal trajectory, thus o = ~t-.. { 0 } .

<

The Dynamic Programming Principle has been the crucial step in the proof that the value function v solves the Hamilton-Jacobi-Bellman equation; conversely, a solution of this equation in all of ]~N has to be the value function, by the uniqueness Theorem 2.12, and therefore satisfy the DPP. Next we show that this equivalence is a local fact and that it can be split into two parts: subsolution is equivalent to a local suboptimality principle and supersolution to a local superoptimality principle.

122

III. PROBLEMS W I T H U N R E S T R I C T E D STATE SPACE

DEFINITION 2.31. Let ~ C ~N be open and bounded and u E C(~). We say that u satisfies a (local) suboptimality (respectively, superoptimality) principle, if for every x E fl there exists ~- > 0 such that (2.39)

u(x) <_ inf ( f t ~ ( y x ( s , a ) , a ( s ) ) e - ~ S d s + u ( y x ( t , a ) ) e - A t ) aEA Jo

for all 0 < t < T, (respectively,

u(x) > inf ( i t ((yx(s, a), a(s)) e -As ds + u(y=(t, a)) e -At) aEAxJO

for all 0 < t _< T).

<~

We recall that the Hamilton-Jacobi-Bellman equation is Au + sup{ - f ( x , a) . Du - g(x, a) } = O .

(2.40)

a6A

THEOREM 2.32. Assume (A0)-(A4), ~ C ~N open and bounded, and u

E C(-~).

Then (i)

u is a viscosity subsolution of (2.40) in ~ if and only if it satisfies a suboptimality principle in ~;

(ii)

u is a viscosity supersolution of (2.40) in ~ if and only if it satisfies a superoptimality principle in f~.

PROOF. By the proof of Proposition 2.8, if u satisfies a suboptimality (respectively, a superoptimality) principle in fl, then u is a viscosity subsolution (respectively, supersolution) of (2.40) in Q. Let us assume now that u is a viscosity supersolution of (2.40) and warn the reader that we are going to use a simple result from Section 3. For z > 0 set f~E := {x E ~ : dist(x,0~) > ~}, and take hE E cl(]l~ N) such that hE = 1 on f~E and h E - - 0 o n ~ C 0_
in ll~N,

aEA

where fE := hEf and s := her + (1 -- hE))~u. Then it is also a (time-independent) supersolution of the initial value problem (2.41)

wt + )~w + sup{ -rE(x, a). Dw - tE(x, a) } = 0 in ]~N X ]0, 2v~[, aEA W(X, O) = U(X) in R y .

It is easy to check that fe and tE satisfy the structure assumptions (A0)-(A3) and (2.91), with different constants and moduli (depending on z). Then by Corollary 3.6 and Remark 3.8 in the next section, the unique solution of (2.41) is (2.42)

wE(x,t)

=

inf ~ FtE(y(~)(s),a(s))e -A~ ds +u(y(E)(t))e -A< }, c*EA " JO

2. THE INFINITE HORIZON PROBLEM

123

where y(~)( 9 ) is the t r a j e c t o r y of the system y' = f~(y, c~), y(0) -- x, and

u(x) >_ w~(x, t)

(2.43)

for all x E ~ g , t > 0 and ~ > 0 .

Now we fix x 9 ft and choose E = 1 dist(x, Oft). Next we pick T such that yx(t) E

B ( x , z ) for all t _ ~-, so that, for all such t, h~(y~(t)) = 1, y(~)(t) = yx(t) and g~(y(z~)(t), &(t)) = g(y~(t), ~(t)). Then. combining (2.42) and (2.43), we obtain u(x) >_ inf ~ f t ~(yx(s), ~(s)) e -As ds + u(yx(t)) e -At } wEA ~'JO for all 0 < t _< ~-. Therefore u satisfies a superoptimality principle. The proof of the "only if" part in (i) can be done in a completely analogous way. However, a more direct proof of a more general result can be obtained by extending to the present assumptions Theorem II.5.21 on differential inequalities. We state it separately as Theorem 2.33. 9 The next result is the extension to the current hypotheses of the main theorem on differential inequalities in the viscosity sense, Theorem II.5.21. We recall the definition of exit time from f}:

tx(C~) :=

+co, min{t : yx(t,a) C Oft},

if {t : y~(t,a) E Oft} = O, otherwise.

THEOREM 2.33. Assume (A0)-(A4), ft open, u 9 C(-~). Then the following state-

ments are equivalent: (i)

u is a subsolution of (2.40) in ft;

(ii)

for all a E .4, x 9 ft, and O < s < t < tz(a) e-~u(yx(s,a))

(2.44) (iii)

- e - ~ t u ( y x ( t , a ) ) < ~s t g(y~(~-,a),a(~-))e-~TdT;

u is a supersolution of

(2.45)

-Au-

sup{-f(x,a). aEA

Du-

*(x,a) } = 0

in Q .

PROOF. We only explain how to relax the assumptions of Theorem II.5.21. In L e m m a II.5.20 the proof of the uniform convergence of a trajectory Yn corresponding to a piecewise constant control ~n to the trajectory y corresponding to &, where an --+ a a.e. in [0, T], makes use of the uniform Lipschitz continuity of f with respect to x, i.e., (1.1). This assumption can be replaced by (A3) by writing

ld

t

-~-~fYn( ) -- Y(t)] 2 = (yn(t) -- y ( t ) ) " [ f ( y n ( t ) , & n ( t ) )


-- f ( y ( t ) , oL(t))] a.e. t E [0, T],

124

llI. PROBLEIxIS W I T H U N R E S T R I C T E D STATE SPACE

where Bn := (Yn - Y)" If(Y, c~) - f(y, a)], integrating on [0, s] and applying Gronwall's inequality as before. Moreover, the continuous differentiability of x ~-* f(x, a) can be removed in Proposition II.5.18 by the following approximation argument. Let t~, be a sequence of mollifiers, i.e., gn are smooth and nonnegative~ f r y g~ dx = 1, supp g~ =/~(0~ 1/n), and define f,~(x, a) := f~N f(y, a) ~,~(x - y) dy. Then fn satisfy (A3) with the same constant L for all n, and f~( .~ a) converge uniformly to f ( - ~a) on compact subsets of I~g. If y~ is the solution of y' = f,~(y,a), y(O) = x~ for a E A fixed, y(t) = yx(t,a), then lyn(t)l. ]y(t)l < C for all t E [0, T] and all n. Moreover, l d i y n ( t ) - y(t)] 2 -= (y,~ - y)" [f~(yn, a) - f(y, a)]


on [0, T],

and by Gronwall"s inequality we get the uniform convergence of y~( 9) to y( 9 ) on [0~ T]. Since (2.44) is valid for y~(- ) and c~ = a we can pass to the limit as n --~ cc and get it for y( 9 ). REIVIARK 2.34. It follows from Theorem 2.33 that any subsolution u E C(Q) of (2.40) satisfies for all x E 12 and t > 0

u(x) < inf ~ ftAt~ ~(y~(s)~ ~(s) ) e - ~ ds + e-~(tAt~)u(yx(t A t~))}~ c~EA ~ JO

which is a global principle of suboptimality. The local suboptimality principle is immediately obtained by observing that T(x) := inf~e A t~((~) > 0 if x E f l . A global principle of superoptimality is proved in w of Chapter IV by means of the inequality t

for all x E 12 and 0 < t < T~(x) for some s > 0, where Te(x) :=- inf{ t : dist(yx(t, c~), 0~) <_ ~, c~ 6 A } . This inequality follows immediately from the proof of Theorem 2.32. Note that the assumption A > 0 can be replaced by A _> 0 in Theorem 2.32 and Theorem 2.33: the only change in the proof of Theorem 2.32 is the use of Theorem 3.7 instead of Corollary 3.6. Also the boundedness of Q can be dropped and replaced by u E BC(-~). In the proof we redefine f/~ as { x 6 f t : dist(x, 0~) > ~ and ]x I < 1/r the other minor changes are obvious.

<~

We end this subsection with the extension to the current weaker assumptions of one of the main results of w in Chapter II, which extends also the first statement of Corollary 2.28.

2. THE INFINITE HORIZON PROBLEM

125

COROLLARY 2.35. Assume (A0)-(A4), f~ C_ ]~N open. Then u C C(~) is a viscosity solution of (2.40) in f~ if and only if it is a bilateral supersolution. PROOF. It is a straightforward consequence of Theorem 2.33: it is enough to apply it to open subsets of f~ where u is continuous up to the boundary. 9 2.4. G e n e r a l i z e d d i r e c t i o n a l d e r i v a t i v e s a n d e q u i v a l e n t n o t i o n s of s o l u t i o n In this subsection we recall some definitions of generalized directional derivatives (Dini derivatives, contingent derivatives) and use them to give an alternative definition of weak sub- and supersolutions of the Hamilton-Jacobi-Bellman equation Au + sup{ - f ( x , a). Du - f(x, a) } = 0, aEA

the so-called Dini solutions, or minimax solutions, and prove the equivalence with viscosity solutions. This characterization of the value function is used in w to give necessary and sufficient conditions for optimality and to construct optimal multivalued feedback maps. DEFINITION 2.36. Let u : l) -~ ~, f~ _C ]I~N an open set. The lower (generalized) Dini derivative, or upper contingent derivative, of u at the point x E f~ in the direction q E ]~N is

O-U(X; q) = l i m i n f u(x + tp) -- u(x) t~O + p--~q

t

:= s u p i n f S1 u(x + tq + tz) - u(x) : 0 < t < 5 . 6>0

~

t

.

I z f < S S I> - O c ;

.

.

.

similarly the upper Dini derivative, or lower3contingent derivative, of u at x in the direction q is

O+u(x;q) := limsup u(x + tp) - u(x) <_ +oe . t~O § p--~q

<~

t

Note that

O-u(x; q) < lira inf u(x + tq) - u(x) (2.47)

t--0+

t

'

O+u(x; q) >_ limsup u(x + tq) - u(x) t~O +

t

'

where the right-hand sides of these inequalities are the classical lower and upper Dini derivatives. It is straightforward to check that the inequalities (2.47) are both 3The s w i t c h i n g of the a d j e c t i v e s u p p e r a n d lower for c o n t i n g e n t derivatives m a y seem a bit weird. However, t h i s t e r m i n o l o g y was i n t r o d u c e d in the field of set-valued analysis and it is n a t u r a l in t h a t context; see the notes a t the end of the c h a p t e r for some references on the subject.

126

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

equalities if u is locally Lipschitz. This is no longer true if we drop the Lipschitz assumption, and u may even have the classical (one-sided) directional derivative Ou(x;q) :=

__c)U(x) u ( x + tq) - u(x) := lim Oq t ~o+ t '

despite O-u(x; q) < O+u(x; q) (for instance, u(xl, x2) = Ixll 1/~ has 0u(0; (0, 1)) = 0 - u ( 0 ; (0, 1)) = 0 and 0+u(0; (0, 1)) = +oc). The link between the sub- and superdifferentials and generalized directional derivatives is given by the following lemma. LEMMA 2.37. Let ft be an open set, u : f~ -~ R. Then D-u(x) = {p:p.q

< O - u ( x ; q ) for all q E ~N },

D + u ( x ) = { p : p . q > O+u(x; q) for all q

E ]~N } .

PROOF. We prove only the first equality, the other being analogous. We begin with the inclusion "C". Fix p E D - u ( x ) , q E N g q 7~' and tn --, 0 +, q,~ --~ q such that O-u(x; q) = lim u ( x + t~qn) - u(x) n

tn

Then O - u ( x ; q) - p . q = linm Iql u(x + tnq~) - u(x) - p . tnqn

t~ Iqnl > lql l i m i n f U(X + Z) - U(x) - P" Z > 0 . z~O Izl -

To prove the inclusion "D", we fix p in the set on the right-hand side and choose zn -* 0 such that

(2.48)

liminfU(X+Z)

z--0

-u(x)

-p.z

Izl

=limU(X+Zn)

~

-u(x)

-p.

zn

Iznl

and limn zn/lznl = q. Since zn = ]znl q~ with qn -~ q, the right-hand side of (2.48) is larger than or equal to liminf u ( x + t~) - u(x) - p . q > O t~O + (~q

t

by the definition of 0 - and the choice of p. Let us introduce the notation (FL)(x) := ~-6{ ( f ( x , a), g(x, a ) ) : a e A },

2. T H E INFINITE H O R I Z O N P R O B L E M

127

and note t h a t ( F L ) ( x ) is c o m p a c t by (A2) and (A4). W i t h a little abuse of n o t a t i o n we call (f, g) the generic element of ( F L ) ( x ) . The H a m i l t o n i a n of the HamiltonJ a c o b i - B e l l m a n equation

)~u + g ( x , Du) = 0

(2.49) can be r e w r i t t e n as (2.50)

n ( x , p ) := sup{ - f ( x , a ) .p

-

aEA

max { - f . p - g}, g(x,a) } = (S,*)e(FL)(x)

as it is straightforward to check. M o t i v a t e d by this, and by the classical formula Du(x). q = Ou(x; q) for differentiable functions we introduce the following definition. DEFINITION 2.38. Let ~ be an open set and u E C(f~). T h e n u is a (generalized) Dini subsolution of (2.49) in gt, with H given by (2.50), if for all x E f~ (2.51)

~u(x) +

sup

{-O+u(x; f ) - ~} < 0;

(f,/)E(FL)(x)

similarly it is a Dini supersolution if (2.52)

~u(x) +

sup

{ - 0 - u ( x ; f ) - ~} > 0,

(f,~)E(FL)(x)

and it is a (generalized) Dini solution (or m i n i m a x solution) if it is a sub- and supersolution. <3 Let us make some c o m m e n t s on this definition. The first comment is t h a t the choice of the p a r t i c u l a r representation formula for the Hamiltonian, i.e., the last one in (2.50), is crucial in Definition 2.38, whereas this choice is irrelevant for viscosity solutions. The reason is that, for instance, f ~ O-u(x; f) is not concave in general, so t h a t SUPaEA { --O-U(X; f(x, a)) --/~(x, a) } can be strictly smaller t h a n the convexified c o u n t e r p a r t in (2.52). This occurs in the next example. EXAMPLE 2.39. Take A = 1, t~ -- 0, f(x, a) = a, A = {a : [a I = 1}, so t h a t the H a m i l t o n i a n is

H(x,p) = m a x { - a . p }

= IP] = m a x { - a . p } .

lab=l-

lal_
is easy to check t h a t u(x) = IxI is a Dini supersolution of u + IDuI = 0 in ~ N , since O-u(O, f ) = If I, but It

u(0) + ~ _ _ ~ { - 0 - u ( 0 ; f ( x , a ) ) }

= - 1 < O . <~

The second c o m m e n t is t h a t if u is locally lipschitzean, if it has directional derivatives in the directions f for all (f, g) E (FL)(x), and if it is a Dini solution of (2.49), then u(x) +

sup (/,~)E(FL)(x) ~

Ou c,j

-

= o

128

III. P R O B L E M S W I T H U N R E S T R I C T E D STATE SPACE

In this case we say that u is an intrinsically classical solution of the HJB equation (2.49), since no generalized derivatives are necessary in this formulation, even if u may not be differentiable and then it may not be a truly classical solution of (2.49). All this occurs, for instance, if u is semiconcave, see Chapter II, w We give two explicit applications of this remark at the end of the subsection. The third comment is that some results in this section look nicer in the special case

(FL)(x) = f ( x , A) x g(x, A)

(2.53)

for all x E N N .

In this case the definition of Dini solution of (2.49) can be written as

Au(x) + sup { - 0 + u ( x ; f ( x , a)) - t(x, a) } <_ 0 aEA

Au(x) + sup { - O - u ( x ; f ( x , a)) - s

a) } > 0 .

aEA

This is a very natural generalization of the equation

)~u(x) + s u p f aEA"

Ou(x) Of(x, a)

e(x, a) } = 0

which is equivalent to the HJB equation (2.49) when the solution u is smooth. The property (2.53) is obviously satisfied if f ( x , A) and g(x, A) are closed convex sets for all x. Two more examples are the convexified problem i f , t~c, A c, and the relaxed problem fr, gr A r introduced in w provided A is compact, see (2.31) and (2.32). The main result of this subsection is the following. THEOREM 2.40. Assume (A0)-(A4), ~ is a bounded open set, and u E C(-~). For the equation (2.49) in ~, with Hamiltonian given by (2.50), the following state-

ments are equivalent: (i)

u is a viscosity subsolution (respectively, supersolution);

(ii)

u satisfies a suboptimality (respectively, superoptimality) principle;

(iii)

u is a Dini subsolution (respectively, supersolution).

PROOF. The equivalence of (i) and (ii) has already been proved in Theorem 2.32. We prove first that (iii) implies (i). By Lemma 2.3"/we have

p. f <_ O-u(x; f) <_ c9+u(x; f) <_ q. f for all p E D - u ( x ) , q E D+u(x), f E R y . (2.49), Au(x) +

sup (f,t)E(FL)(x)

Thus, if u is a Dini supersolution of

{ - f . p - e } > Au(x) +

sup

{ - O - u ( x ; f) - g } > O,

(f,t) E(FL)(x)

for all p E D - u ( x ) . From (2.50) it follows that u is a viscosity supersolution of (2.49). The proof for subsolutions is just the same. In order to complete the proof we need to show t h a t (ii) implies (iii). For this purpose we make use of the following result in convex analysis.

2. THE INFINITE HORIZONPROBLEM

129

LEMMA 2.41. For all x E N N we have (2.54)

(FL)(x) = {g = (f, 2) E

Ii~Y+l :

g = n--+oct"-~ lira 1 j~otn (f(x, an(s)),2(x, Ctn(S)))dsforsometn---*O + a n d a n E M } ; . PROOF. By the m e a n value theorem, for any tn --* 0 + a n d a n E .4,

1 j~0t~ (f(x,a~(s)),2(x,c~n(s)))ds E ~-6(f(x,A) x 2(x,A)) = ( F L ) ( x ) . ~n Hence, the set on the r i g h t - h a n d side of (2.54) is contained in (FL)(x). For the opposite inclusion, set G ( x , a ) : = (f(x,a),2(x,a)); if g E coG(x,A), t h a t is,

J

J

g = E / k i G ( x , ai),

ai E A,

Ai > O,

i=1

E

Ai = I'

i=1

then (2.55)

1/0'

g = -~

G(x, o~t(8)) d8

for any t > 0,

where

at(s) = ak

for s E

I k-1

k

i=l

i=l

tE,~i,tZ,ki

I

, k = 1,2 . . . . . j .

Take now g E ~-6G(x, A) a n d choose g,, E co G(x, A) such t h a t g~ --, g and tn --* 0 +. Since gn can be w r i t t e n in the form (2.55) with t = tn, the conclusion follows easily. 9 END OF THE PROOF OF THEOREM 2.40. Fix x E ~, a n d (f, 2) E (FL)(x). L e m m a 2.41 to pick tn --~ 0 + a n d a n E .4 such t h a t

Use

l _ _ f t" tn Jo f(x, a~(s)) ds --~ f, as Tl --+ ~0 . 1 j~otn t-~ 2(x, ~,,(s)) ds ---~ 2, If u satisfies the s u b o p t i m a l i t y principle (2.39), then, for 0 < t _< r a n d all ct E A,

(2.58)

u(x) - u(yx(t)) + u(yx(t)) 1 -- e -:~t t ~

1 it t 2(yx(s), a(s)) e - ~ ds <_ O. dO

By (Ao)-(A4) (see (2.13) a n d (2.14)) we have yx(tn,C~n) = x +

~0 tn f ( x , a ~ ( s ) ) d s + o ( t ~ ) = x + t n Z + o ( t ~ )

1 fO tn 2 ( y x ( s , a ~ ) , ~ ( s ) ) e - ~ d s t--2

= 2+o(1)

a s n --* o z .

130

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

Then, using a,~ and t,~ in (2.56), we get

u(x) - u(x + t , f + o(t,~)) + Au(x) - ~ < o(1) tn

as n --+ oc

and then

Au(x) - c9+u(x; f ) - g = Au(x) +

liminf

u(x) - u(x + tg) _ g <_ 0,

t

(t,g)--(O+,f)

which shows that (ii) implies (iii) for subsolutions. To prove that (ii) implies (iii) for supersolutions, fix x E l). By the superoptimality principle, for every n C N we can find an E A and the corresponding trajectory y(~)(t) = yx(t, an) such that (2.57) u(x) -u(y(')(t))t

+u(y(n)(t)) 1 - te-At

tl fotg(x, a n ( s ) ) d s > - n 1_ +o(1).,

as t --+ 0 +, where we have already used (2.13) in the integral. By the compactness of (FL)(x) we can find tn ---* 0 +, a subsequence of an, still denoted a,~, and (f, 2) E (FL)(x) such that

V.I ~0 tn e(x, ~(8))

ds ---~ g,

•176 tn Jo

aS n--~

(X) .

f ( x , a,~(s)) ds ----+ f,

Moreover, by (2.14),

y(~)(t,) = x +

f(x,a~(s))ds+o(tn)=x+t~f+o(tn)

as n -~ co,

and thus, by (2.57),

0 <_ au(x) + lira inf u(x) - u(x + t n f + ~(t,d) _ [ n tn <_ ~u(x) +

limsup u(x)- u(x + tg) _ [ = )~u(x)_O_u(x;f)_[ t

(t,g)~(o+,]) _< Au(x) +

sup {-0-u(x;f) (f,g)C(Fa)(z)

- ~}.

~,

REMARK 2.42. If u satisfies a suboptimality principle, then it satisfies a local backward suboptimality principle, that is, inequality (2.34) for all 0 < t _< ~-. By the arguments of the previous proof one can get

~u(x) +

sup { O - u ( x ; - f ) - g} <_ O. (f,g)E(FL)(x)

Note that this inequality does not follow directly from either (2.51) or (2.52), and that it implies immediately, via Lemma 2.37, that u is a viscosity supersolution of - A u - H ( x , Du) = 0 (see Corollary 2.28). See also Exercise 2.11. <1

2. THE INFINITE HORIZON PROBLEM

131

REMARK 2.43. Some stronger results can be proved if u is locally lipschitzean because the generalized Dini derivatives take the simpler form of classical Dini derivatives (i.e., the right-hand sides of (2.47)). In the assumptions of Theorem 2.40. if u is locally Lipschitz and satisfies a suboptimality principle, then (2.58)

Au(x) + lim sup

sup

t~O +

( u(x) - u(x + t f ) _ e} < O.

(f,QE(FL)(x)"

t

--

uniformly for x in a compact set of ~, which easily implies

Au(x) +

(2.59)

sup

{ - O - u ( x ; f ) - e} <_ O .

(/,t)E(FL)(x)

Note that (2.59) is stronger than (2.51), and it implies t h a t u locally lipschitzean is a viscosity solution in fl of (2.49) if and only if it is a lower Dini solution, that is,

/ku(x) +

sup

{-O-u(x;

~} =

f) -

0.

(f,~)E(FL)(z)

Here is a proof of (2.58). t > O, a(t ) E A such that

f =

Choose ( f , t ) E c o ( f ( x , A ) • g(x,A)) and, for any

'/0'

7

f(x,a(t)(s))ds,

e=

if

7

t(x,a(t)(s))ds.

Then yx(t, a(t)) = x + t f + o(t) as t -~ 0 + and u(yx(t, a(t))) = u ( x + t f ) + o(t) as t --* 0 + by the local Lipschitz continuity of u. Then from (2.56) we get, as in the previous proof, (2.60)

u(x) - u(x + t f ) + Au(x) - ~ < o(1) t

as t ---* 0 + .

Since (f, ~) E c o ( f (x, A) x g(x, A)) is arbitrary, the left-hand side of (2.60) is continuous in (f, f), and the right-hand side tends to 0 independently of ( f , f ) and uniformly for x in a compact set of ~, we get

Au(x) +

~ u(x) - u ( x + t f ) _ g~ < o(1)

sup (f,t)E(FL)(x)

t.

t

)

as t -~ 0 +,

--

and (2.58) follows immediately. See Exercise 2.12 for a result on locally Lipschitzean supersolutions.

<~

REMARK 2.44. A direct proof of the equivalence of viscosity and Dini supersolutions (respectively, subsolutions) can be given by means of the following property linking Dini derivatives and subdifferentials: if u is continuous in a neighborhood of x, C is a closed convex set, and

cg-u(x; q) > 0

for all q E C,

then there exist sequences xn ---* x and Pn E D - u ( x n ) such that Pn 9q > 0 See Exercise 2.13.

for all q E C . <~

132

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

We end this subsection with two special cases where, under stronger assumptions, the value function is an intrinsically classical solution of the Hamilton-JacobiBellman equation. The first case is semiconcavity of v, see w in Chapter II for the definition. We just recall t h a t sufficient conditions for semiconcavity are A > 2L (L is the constant in (A3)) and suitable smoothness (e.g., C 2) of f and 6 in the state variable uniformly in the control variable: see Remark II.4.10. PROPOSITION 2.45. Assume (A0)-(A4) and that the value function v is semiconcave. Then v is locally lipschitzean, it has all (classical one-sided) directional derivatives Or(x; q) everywhere, and it solves (2.61)

Av(x) +

sup { - O v ( x ; f ) - 6} = 0 (/,Qe(FL)(z)

for all x e N N .

In particular, the forvnulation (2.61) is equivalent to the viscosity and the Dini formulations of the Hamilton-Jacobi-Bellman equation (2.49). PROOF. The Lipschitz continuity of v is proved in Proposition II.4.6 and the existence of the directional derivatives in Proposition II.4.7 (d). All the other statements are immediate consequences of the theory of this subsection. 9 REMARK 2.46. We proved in Corollary II.5.2 that for a semiconcave function being a viscosity solution is also equivalent to solving the Hamilton-Jacobi-Bellman equation a.e. We recall also t h a t if A _> 2L, v can be approximated uniformly on comp a c t a of ]RN by semiconcave value functions of similar infinite horizon problems: see Corollary 2.23. <1 The second case has a different proof but its assumptions are essentially of the same type: large A and smooth f and 6. PROPOSITION 2.47. In addition to (A0) (A4) suppose that f and 6 are differentiable with respect to the state variable x and [f(x + q, a) - f ( x , a) - D x f ( x , a)q[ <_ [q[ w(iq[)

le(x +

q, a) - e(x, a) - Dx6(x, a). ql <

Iql

for all x, q E R N, a C A, where a~ is a modulus and sup{ I D x f ( x , a)[: x e ]~N, a S A } < A. Then all the conclusions of Proposition 2.45 hold. SKETCH OF THE PROOF. The Lipschitz continuity of v follows from Proposition 2.1. All the statements follow easily from the theory of this section once we prove the existence of the directional derivatives. This follows from the next lemma, which is a variant of the results on derivatives of marginal functions of Chapter II, w167and 4.1. 9

2. THE INFINITE HORIZON PROBLEM

133

LEMMA 2.48. Let B be a set, W b : • N __+ ~ , b E B, be equibounded, equicontinuous functions with directional derivatives such that

wb(x + hq) - wb ( x ) _ cOwb(x; q) < w ( h ) h for all x, q E ]RN, b E B, h > O, where a3 is a modulus. If w(x) := infbe B wb(x) is continuous, then it has all directional derivatives everywhere and Ow(x; q) = inf{ lim infn Ow b€ (x; q ) : limn w bn (x) = w(x) } . The proof of this lemma is left to the reader, see Exercise 2.15. To complete the proof one has to show t h a t wb(x) := J ( x , c~), with b = a and B = .2,, satisfy the assumptions of Lemma 2.48. This is a tedious but straightforward computation t h a t we leave to the reader. 2.5. N e c e s s a r y a n d s u f f i c i e n t c o n d i t i o n s o f optimality, minimum principles, and multivalued optimal feedbacks In this Section we apply the theory of w to give several necessary and sufficient conditions for optimality of open loop controls in terms of Dini derivatives or suband superdifferentials of the value function. Some of them take the form of a Minimum Principle, and we explain the connection with the classical Pontryagin Maximum Principle. We also study the optimality of feedback maps and construct optimal multivalued feedbacks which are a counterpart for problems with nonsmooth value functions of the classical calculations of optimal feedbacks in linear-quadratic problems. Before stating the main result we introduce some notations and give some motivations. Let us fix x0 E ]~.N and for a given a C .A let us write for brevity

y(t) := yxo(t, ~ ) . We know from the D P P and, in particular, from Remark 2.6, t h a t the control is optimal for x0, i.e., v(xo) = J(xo, ~), if and only if the function (2.62)

g(t) : = v(y(t)) e -at +

/0 t e(y(s), c~(s)) e -xs ds

is constant, and t h a t in this case the control s H a ( s + t) is optimal for y(t), for all t _> O, so t h a t ~ generates, by shifting, an optimal control for each point of the trajectory. Moreover, since the function g is nondecreasing for any control, the true condition of optimality is that g be noninereasing. Thus, if everything is smooth (v differentiable and ct continuous), the condition of optimality is g'(t) _< 0 for all t, that is,

e -At [Dv(y(t)). y'(t) - )~v(y(t)) + t(y(t), a(t))] _< 0

for all t .

134

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

There are several ways to rewrite this necessary and sufficient condition of optimality in an equivalent way which makes sense in the general nonsmooth case. Here are some possibilities: for a.e. t > 0

p. y'(t) + e(y(t), ,(t)) < A v(y(t)),

(2.63)

either for all or for some p C D-v(y(t)), where we denote

D•

:= D+v(x) O D - v ( x ) ;

(2.64)

O-v(y(t); y'(t) ) + f(y(t), c,(t) ) <_ )~v(y(t) )

(2.65)

-o+v(~(t); -g(t)) + e(y(t), ~(t)) < ;~v(v(t)).

Since v is a viscosity subsolution and a bilateral supersolution of the HJB equation k v + s u p { - D v - f ( x , a ) - g(x,a) } = O, aC:A

(2.63) is equivalent to

(2.66)

p . y'(t) + e(y(t), ~(t)) = )~v(y(t))

and it implies (2.67)

p. f(y(t), a(t)) + f(y(t), c~(t)) = rain{ p. f(y(t), a) + g(y(t), a) }. aEA

Note that the right-hand side of (2.67) is - H ( y ( t ) , p), where H is the Hamiltonian given by (2.50). Similarly, if v is locally lipschitzean then it is a lower Dini solution of the HJB equation (see Remark 2.43), that is, ~v(z) +

sup

{ - O - v ( x ; f) - g } = 0,

(f,g)e(FL)(x)

so in this case (2.64) is equivalent to

(2.68)

o-v(v; y') + e(y, ~) = ~v(~)

and to (2.69)

O-v(y; f(y, a)) + g(y, a) =

rain

(f,l)c(Fa)(y)

{ O-v(y; f) + ~ } .

We first prove the necessity of these conditions. THEOREM 2.49 (NECESSARY CONDITIONS OF OPTIMALITY). Assume (A0)-(A4).

For c~ E ,4 let y(. ) be the corresponding trajectory with initial point xo C ~N. If c~ is optimal for Xo, then conditions (2.64), (2.65), and (2.66), (2.67) hold for all p E D ! v ( y ( t ) ) , for a.e. t > 6.

2. THE INFINITE HORIZON PROBLEM

135

T h e necessity of c o n d i t i o n (2.67) for all p 9 D • is called the M i n i m u m Principle (briefly M P ) or the m a x i m u m principle if one changes sign to b o t h sides of (2.67). It is a weak form of the classical P o n t r y a g i n M P because we are not a s s u m i n g a n y differentiability on the data. Note t h a t condition (2.67) can be r e w r i t t e n as (2.70)

a(t) e argmin{p-f(y(t), A

.) + g(y(t), .) } .

Also c o n d i t i o n (2.69) can be viewed as a m i n i m u m principle expressed in terms of directional derivatives. W h e n v is locally lipschitzean it implies the following further necessary condition of o p t i m a l i t y

a(t) 9 A , ( y ( t ) )

(2.71)

for a.e. t,

where (2.72)

A . ( x ) : = a r g m i n { O-v(x; f ( x , . )) + g(x, . ) } . A

In the proof of the t h e o r e m we need t h e following chain rule for Dini derivatives. LEMMA 2.50. Let y : [0, T] ~ f~ be differentiable at s and u C C(f~). Then

O - ( u o y)(s; 1) _> O-u(y(s); y'(s)),

O+ (u o y)(s; 1) _< O+u(y(s); y'(s) ) . If, moreover, u is locally lipsehitzean, then both these inequalities are equalities, PROOF. Since y(s + h) = y(s) + h y ' ( s ) + o(h) as h ~ O,

O - ( u o y)(s; 1) = l i m i n f u(y(s + h)) - u(y(s)) > O-u(y(s); y'(s)) . h~O + h If u has a Lipschitz c o n s t a n t C in a neighborhood of y(s), t h e n u ( y ( s + h)) - u(y(s)) < u(y(s) + h y'(s)) - u(y(s)) + C o ( h ) h h ' so we get the desired equality by t a k i n g the lira infh_0+ of b o t h sides, since in this case the generalized Dini derivative coincides with the usual Dini derivative. T h e proof for 0 + is analogous. 4 PROOF OF THEOREM 2.49. If c~ is optimal, the fimction g defined by (2.62) is constant. T h e integral in the definition of g is differentiable for a.e. t, a n d therefore so is v(y( 9 )) e - ~ . We fix a n y such t where y( - ) is differentiable as well, a n d choose hn ----*0 + such t h a t lim v(y(t + hn)) - v(y(t)) = O - ( v o y)(t; 1 ) .

n

hn

136

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

Then

(v(y(t))e-At) ' = lira [v(y(t + hn)) - v(y(t))] e -At + v(y(t + hn)) [e -A(t+h~) - e -At] n hn = - ~ e - A t v ( y ( t ) ) + e - A t O - ( v o y)(t; 1)

> e -At [ - ~ v(y(t)) + 0-v(y(t); y'(t))], where the inequality comes from Lemma 2.50. Thus

0 = J(t) > e -At [-:~ ~(y(t)) + 0-~(y(t); y'(t)) + e(y(t), ~(t))] for a.e. t, and so (2.64) holds. The proof of the necessity of (2.65) is similar and we omit it. Now (2.64) and Lemma 2.37 give (2.63) for all p C D - v ( y ( t ) ) . bilateral supersolution of the HJB equation, for all x and a

Av(x)<_p.f(x,a)+~(x,a)

Since v is a

forallpED-v(x),

thus the inequality in (2.63) is an equality and we get (2.66) and (2.67) for all

p E D - v ( y ( t ) ) . Similarly, (2.65) and Lemma 2.37 give (2.63) for a l l p E D+v(y(t)), and then (2.66) and (2.67) hold for such p because v is a subsolution of the HJB equation. 9 REMARK 2.51. It follows immediately from Lemma 2.50 that for locally lipschitzean u and y the classical directional derivative Ou(y(s); y'(s)) exists for all s such t h a t both y and u o y are differentiable, therefore a.e. in [0, T]. In this case the Dini derivatives in (2.64) and (2.65) are in fact classical (one-sided) derivatives. <~ Next we study the sufficiency of the necessary conditions of optimality listed previously. THEOREM 2.52 (SUFFICIENT CONDITIONS FOR OPTIMALITY). Assume (A0)-(A4) and y ( . ) = Y~o(', 5). Suppose that there exists a subsolution u E BC(R N) of the HJB equation which is locally lipschitzean in a neighborhood of { y(t) : t >_ 0 }.

Then a is optimal for xo if for a.e. t > 0 at least one of the following conditions holds:

(2.73) (2.74) (2.75) (2.76)

O-u(y(t); y' (t) ) + t~(y(t), a(t) ) <_ ~ u(y(t) );

- O§ + e(y(t), ~(t)) < ~ u(y(t)); 3p E D• : p. y'(t) + ~(y(t), a(t)) <_ A u(y(t)); u is also a supersolution of the H J B equation in a neighborhood of y(t) and there is p E D - u ( y ( t ) ) or p C D*u(y(t)) • D+u(y(t)) such that (2.67) holds.

By Remark 2.43, (2.73) is equivalent to (2.77)

,k u(y) = a-u(y; y') + g(y, a) =

min { O-u(y; f ) + t} . (f,g)E(FL)(y)

2. T H E I N F I N I T E HORIZON PROBLEM

137

PROOF. Let us consider the function

h(t) := u(y(t)) e -;~t +

e(y(s), a(s)) e -;~s ds,

t >_ O .

Our goal is to prove t h a t h is nonincreasing, because in such a case

u(xo) = h(O) >_ lim h(t) = J(xo, a) . t~4-zc

By the comparison principle (Theorem 2.12 and Remark 2.13) u(x) <_ v(x) and then J(xo, a) = v(x0), showing t h a t a is optimal for x0. Since g(y, a) and yl are locally bounded and u is locally lipschitzean, h is locally lipschitzean and h'(t) = e - ~ t [(u o y)'(t) - ~ u(y(t)) + ~(y(t), ~(t))].

for a . e . t . The right-hand side of this equality is _< 0 if, for a.e. t, either (2.73) or (2.74) holds, because at every point of differentiability of both u o y and y

(u o y)'(t) = O-(u o y)(t; 1) = O - u ( u ( t ) ;

yl(t))

and

(u o y)'(t) = - 0 + ( u o y ) ( t ; - 1 ) = - O + u ( y ( t ) ; - y ' ( t ) ) , by Lemma 2.50. Thus h is nonincreasing in this case. On the other hand, if (2.75) holds with p c D - u ( y ( t ) ) , then, by Lemma 2.37,

)~u - ~ ~ - p . ( - y ' ) > - O - u ( y ; - y ' )

>_ -O+u(y; _yl)

and thus (2.74) holds. Similarly (2.75) with p E D+u(y(t)) implies (2.73). Now let us assume (2.76) with p C D - v ( y ( t ) ) . Then (2.75) holds because v is a bilateral supersolution of the HJB equation. Finally let p E D*u(y(t)) N D+u(y(t)). By definition of D*, see w p = limn Du(xn) for a sequence xn --~ y(t) of points where u is differentiable. Since u is a viscosity solution of the HJB equation near y(t), by Proposition II.1.9 this equation is satisfied in the classical sense at points of differentiability, thus

Au(xn) + g ( x ~ , Du(xn)) = O. By letting n --~ ~c we get

Au(y(t) ) 4- H (y(t),p) = 0, and then (2.67) implies (2.75).

,,1

REMARK 2.53. From the proof of the theorem one sees that the two Dini derivatives in (2.73) and (2.74) coincide. Moreover they are classical (one-sided) derivatives by Remark 2.51, thus each of the conditions (2.73) and (2.74) is equivalent to

cgu(y; y') 4- g(y, a)
-Ou(y; - y ' ) + t?(y, a) < Au(y) .

<~

138

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

The previous results can be combined to get several characterizations of optimal controls which are listed in the next theorem. Recall that the (local) semiconcavity of v implies D+v(x) = Or(x) ~ O for all x by Proposition II.4.7, where Ov denotes Clarke's generalized gradient. THEOREM 2.54 (.-NECESSARY AND SUFFICIENT CONDITIONS OF OPTIMALITY). Assume (A0)-(A4) and y(. ) = Yxo(" ,c~). Suppose the value function v is locally lipschitzean in a neighborhood of {y(t) : t > 0}. Then (i) a is optimal for xo if and only if for a.e. t > 0 one of the following conditions

holds:

(2.78)

0v(y(t); y'(t)) + e(y(t), ~(t)) = ~v(y(t)), O-v(y; f(y, a)) + t?(y,a) = (L~)mi-n)(y) { e ( [ L O-v(y; f) + e };

(ii) if in addition D•

• 0 for a.e. t, then the optimality of ~ for xo is equivalent to each of the following conditions 9 for a.e. t there exists p 9 D• such that p - y' +/?(y~ a ) = Av(y),

(2.79)

9 for a.e. t and for all p 9 D i v ( y ( t ) ) (2.79) holds; (iii) if, moreover, D+v(y(t)) = Ov(y(t)) for a.e. t, then a is optimal for xo if and

only if for a.e. t

(2.80)

p. f(y, a) + e(y, a) = ma~n{ p. f ( y , a) + e(y, a) }

for all p 9 D• PROOF. (i) The statement about the first condition follows from Theorems 2.49 and 2.52 and Remark 2.53. To prove the statement on the second condition it is enough also to use Remark 2.43. (ii) is an obvious consequence of Theorem 2.49 and 2.52, and so is the "only if" statement in (iii). To complete the proof of (iii) we recall that D*v(z) ~ 0 for all z because v is locally lipschitzean. Since Or(z) = coD*v(z) for all z, we have D*v(y(t)) c_ D+v(y(t)) for a . e . t . Therefore for a.e. t there exists p 9 D*v(y(t)) n D+v(y(t)), and it satisfies (2.80) by assumption. Then the conclusion follows from Theorem 2.52. 9 REMARK 2.55 (THE PONTRYAGIN MAXIMUM PRINCIPLE). A second interesting case where D• ~ 0 along an optimal trajectory occurs when ~ and f are C 1 with bounded derivatives and A is large enough. In Section 1.6 we saw that there exists a solution p(t) of the adjoint system (I.6.14) such that, for a.e. t, (2.81)

e~tp(t) 9 D+v(y(t)),

e~tp(t) 9y'(t) + g(y(t), a(t)) = Av(y(t))

2. T H E I N F I N I T E HORIZON PROBLEM

139

if y is the t r a j e c t o r y associated with an optimal control c~. By Theorem 2.49 the maximum in the definition of the Hamiltonian H(y(t), e~tp(t)) is attained at ~(t); t h a t is, (2.82)

e~tp(t) 9f(y(t), a(t)) + g(y(t), c~(t)) = min{ e~tp(t) . f(y(t), a) + e(y(t), a) }, aEA

for a . e . t . This is the classical necessary condition of the PMP. If v is locally lipschitzean around a t r a j e c t o r y y ( . ) = Y~0( ' , a) we can use Theorem 2.54 to extend the P M P to a necessary and sufficient condition of optimality: c~ is optimal for x0 if and only if the costate p(t) satisfies (2.81) and (2.82) for a.e. t > 0. Note that even if the P M P is a special case of the conditions of optimality derived from Dynamic Programming, it has the advantage that the necessary condition (2.67) can be checked without knowing the value function: it is enough to know the t r a j e c t o r y y and to solve the adjoint equation. <~ REMARK 2.56. If in addition to the assumptions of Theorem 2.49 we have (FL)(x) = f(x, A) x ~(x, A), then the necessary and sufficient condition of optimality (2.78) becomes

O-v(y; f(y, c0) + g(y, c~) = m~n{ O-v(y; f(y, 9 )) + g(y, 9 ) } for a.e. t, that is, (2.83)

a(t) e A.(y(t))

for a.e. t > 0,

where A . is defined by (2.72). If the running cost does not depend explicitly on the control (i.e., g(x, a) = g(x) for all x and a) then

A.(x) = argminO-v(x; f(x, 9 )). A

In this case the optimality condition (2.83) has a rather intuitive geometric interpretation: at almost every time t the controller must choose among all the available directions f(y(t),a) at the current state y(t) any one providing the steepest descent of the value function at the point y(t), in the generalized sense of lower Dini derivatives. <~ REMARK 2.57. We saw in w

that

(FL)(x) = fr(x, Ar) x gr(x, Ar) . Then from the previous remark one gets, under the assumptions of Theorem 2.49, t h a t a relaxed control # E A r is optimal for x0, i.e., Jr(x0, #) = V(X0), if and only if the corresponding relaxed t r a j e c t o r y y(. ) = Yxo ( ' , #) satisfies

O-v(y; fr(y, p)) + gr(y, #) = n~in{ O-v(y; if(y, 9 )) + gr(y, . ) } for a . e . t .

<~

140

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

Let us make a brief comparison between the conditions of optimality in this subsection (Theorems 2.49, 2.52 and Corollary 2.54) and those in subsection 2.2, the so-called verification theorems. In both cases one needs first to know a global object, a subsolution of the HJB equation or the value function itself, but then the criteria are different. In a verification theorem (e.g., Corollary 2.19) one has to know the whole open loop control function, compute the payoff, and then compare it with the verification function. On the contrary, the conditions (2.64) or (2.66) relate the control at a given time t only with the state at the same time, so they are fit to design an optimal policy prescribing the instantaneous choice of the control according to the actual state. This leads to the problem of the optimality of feedback maps and closed loop controls, which we discuss next. The classical synthesis procedure recalled in Section 1.5 states that if the value function v is differentiable, then an admissible feedback 9 : RN ~ A (see Definition 2.17) is optimal if ~(z) E arg rain{ Dv(z) . f ( z , 9 ) + g(z, 9 ) } A

for all z E R N, and by the HJB equation this condition can also be written as

(P(z) E

a E A : Of(z,a---~ + g(z'a) < Av(z)

.

To obtain the generalization of this condition to the case of a merely continuous value function we introduce the sets SD(Z) := { a E A : O-v(z; f ( z , a)) + g(z, a) <_ Av(z) },

(7 S v ( Z ) :~-

argminA{P'f(z,')+g(z,')}

ifD+v(z)~

O,

pED-v(z)

if D •

A

= O,

where the subscripts D and v stand for Dini and viscosity, respectively. Under the assumptions (A0)-(Aa), a necessary condition for the optimality of an admissible feedback 9 is the existence of a set X c R N such that (2.84)

~(z) E SD(Z)

for all z E X,

and X is "large on the trajectories of ~" in the following sense (2.85)

V x E ]~N

yx(t, ~) E X

for a.e. t > 0 .

To prove this statement it is enough to define X := { z : O(z) E SD(Z) } and use Theorem 2.49. By the same argument we see that another necessary condition of optimality for 9 is (2.86)

~(z) E Sv(z)

for some set X satisfying (2.85).

for all z E X,

2. THE INFINITE HORIZON PROBLEM

141

Assume now v is locally lipschitzean. Then (2.84) with (2.85) is also a sufficient condition for the optimality of 9 by Theorem 2.52. If we assume in addition D + v ( z ) = Ov(z) for all z E R N, then (2.86) with (2.85) becomes also a sufficient condition of optimality for ~, by Theorem 2.54 (iii). It is natural to try to use the sufficient conditions (2.84) and (2.86) to construct optimal admissible feedback maps. A n easy example is the case when SD(x) (or Sv) is nonempty and single valued for all x, and for each initial point there is a unique solution of y' = f ( y , SD(y)) such that a * ( . ) = SD(y( 9)) is measurable, for instance when SD is a Lipschitz function. Then a* is an optimal open loop control for the initial point y(0), the unique one if we identify controls coinciding a.e., and SD is an optimal feedback map. This situation occurs, for instance, in linearquadratic problems, but it is very peculiar. Typically SD and Sv are multivalued, or empty, or discontinuous. Therefore an optimal admissible feedback may not exist, as in the following simple example. EXAMPLE 2.58. Consider the one-dimensional problem of Example 2.30. The value function v is globally lipschitzean and differentiable for x ~ 0. Since f ( x , a) = a E [-1, 1] we compute 0 - v ( 0 ; a) -- ILl (e -1 - 1). Now it is easy to check that

SD(X) :Sv(x) = { - 1i,, x>O, x < 0,

SD(0) = { --1, 1 },

Sv(0) = 0 .

If 9 were an optimal feedback map, it should be ~(x) = 1 for a.e. x > 0 and 9 (x) = - 1 for a.e. x < 0. Therefore, no matter how we define ~(0), there are two distinct solutions of y' = f ( y , ~(y)), y(0) = 0, namely, yz(t) = t and y2(t) = - t . Thus 9 is not admissible. <~ The previous discussion leads naturally to the introduction of the notion of a

multivalued feedback map. DEFINITION 2.59. A map S : R N ----*P ( A ) is a multivalued admissible feedback if for all x E ]~N there is a pair (y, a) such that a E A and

/

y(t) = x +

/:

f(y(s),a(s))ds,

((~(s) e S ( y ( s ) ) ,

t > 0 a.e. s > 0 .

Such a pair (y, ct) is called a solution of (2.87)

fy I E f(y,S(y)),

[y(0)=x.

t > O,

<

Note that any solution ( y , a ) of (2.87) is such that y ( . ) = Y x ( ' , a ) in our standard notation. Our definition of solution of (2.87) is slightly different from the usual one in the theory of differential inclusions, because we explicitly mention the "selection" a as part of the solution, and we require it to be measurable. We do this because we want the cost functional to be defined on every solution of (2.87). However, this does not restrict the class of admissible feedbacks, by a classical theorem of Filippov.

142

III. P R O B L E M S W I T H U N R E S T R I C T E D STATE SPACE

DEFINITION 2.60. A multivalued admissible feedback S is weakly optimal if for all x E N N there exists an optimal solution (y, a) of (2.87), that is, a solution such that J(x, a) = minA J(x, 9 ); S is fully optimal if for all x E IRN every solution of (2.87) is optimal. <~ The first obvious remark on this definition is that a necessary condition for the existence of a weakly optimal multivalued feedback is the existence of an optimal open loop control for every initial point x E R N, and in this ease S(x) = A for all x is a trivial weakly optimal feedback, certainly not fully optimal in general. The next result says that this condition is actually sufficient for the existence of nontrivial weakly optimal feedbacks; it gives formulas for some of them in terms of generalized derivatives of the value function, and shows that they can be fully optimal. THEOREM 2.61. Assume (A0)-(A4) and suppose that for all x E lI~N there is a

cow'responding optimal control in ~4. Then (i)

the multivalued feedbacks SD and Sv are weakly optimal;

(ii)

if in addition the value function v is locally lipschitzean, then SD is fully optimal;

(iii)

if moreover D+v(z) = Ov(z) for all z E R N, then Sv is fully optimal as well.

PROOF. Given x E R N let a* E .A be a corresponding optimal control and set y*( 9) = y ( - , a*). We claim that the pair (y*, a*) is a solution of the differential inclusion (2.87) with S = SD. In fact, by Theorem 2.49, for a.e. t > 0

O-v(y*(t); f ( y *(t), ct*(t))) + f(y*(t), a*(t)) _< ,kv(y*(t) ), thus a*(t) E SD(y*(t)) for a . e . t . Then the feedback SD is weakly optimal, and a similar argument works for Sv. To prove the full optimality of So (respectively, Sv) it is enough to observe that any solution of the differential inclusion (2.87) with S = SD (respectively, S = Sv) is optimal because it satisfies the sufficient condition (2.73) of Theorem 2.52 with u = v (respectively, condition (2.80) of Theorem 2.54 (iii)), for a . e . t . 9 REMARK 2.62. It would be nice to have a representation for SD in the form "arg max of the Hamiltonian," as we have for Sv. It is easy to deduce from Remark 2.43 t h a t a E So(z) if and only if 0 - v ( z ; f(z, a)) + e(z, a) =

rain (f,t)E(FL)(z)

{ O-v(z; f) + e },

for any z E R N. In particular, if (FL)(z) = f ( z , A ) x ~(z,A) for all z, then SD(Z) = A.(z) = argmin{ O-v(z; f ( z , 9 )) + f(z, 9 ) },

for all z .

<~

A

The crucial assumption of Theorem 2.61 is the existence of an optimal open loop control for every initial point. There is a large literature on this issue. The most

2. THE INFINITEHORIZONPROBLEM

143

classical and simple sufficient condition for this property is a convexity assumption on the problem. Next we give two explicit applications of this observation, beginning with the relaxed problem. COROLLARY 2.63. Assume (A0), A compact, (1.1), (A4) with ,3~(r) = M r .

Then

the following multivalued feedback for the relaxed problem, A~.(z) := argmin{O-v(z; i f ( z , 9)) + gr(z, 9)}, AT

is weakly optimal, and it is fully optimal if. in addition, v is locally lipschitzean. PROOF. In Section VI.1 we will construct for any initial point an optimal relaxed control #* E .AT, under the current assumptions. Moreover, by (2.32), we have ((FL))(z) = f r ( z , A r ) • er(z, At), for all z. Therefore the conclusions follow from Theorem 2.55 and Remark 2.62.

9

COROLLARY 2.64. Assume the hypotheses of Corollary 2.63 and suppose f ( x , A) x g(x, A) is a convex set for all x e ~ Y . Then (i)

the multivalued feedbacks SD = A , and Sv are weakly optimal;

(ii)

if,. in addition, v is locally lipschitzean, then SD = A , is fully optimal;

(iii) if, moreover, v is locally semiconcave, then Sv is fully optimal. PROOF. In Section VI.1 we prove the existence of an optimal control for any initial point, under the current assumptions. Therefore all the conclusions follow from Theorem 2.61, Remark 2.62, and Proposition II.4.7. 9 We end this subsection with a few remarks on the problem of finding an optimal synthesis as defined in Definition 2.17. In analogy to w we can associate with a multivalued admissible feedback S a multivalued control law or multivalued presynthesis A s : ] ~ N ~ ~D(.A) as follows AS(x) = { a C .A : (y. (~) is a solution of (2.87) } . Under all the assumptions of Theorem 2.61, the presynthesis generated by SD and Sv satisfy ASD(x) : ASv(x) : A*(x)

for all x E ]~N

where A* is the maximal optimal presynthesis A*(x) := { ct E .A : a is optimal for x }, as it is easy to see from the proof of the theorem. By Remark 2.6, A* is memoryless, in the sense that A*(z) = { a ( . + t) : t _> 0, a C A(x),z = yz(t,a) }, and we call it a multivalued synthesis. Therefore we have a method for generating the maximal optimal multivalued synthesis A*. To our knowledge there are no general results on the existence of single-valued optimal syntheses.

144

III. PROBLEMS W I T H U N R E S T R I C T E D STATE SPACE

Exercises 2.1. Prove directly that there is at most one bounded function satisfying the DPP, that is equation (2.5). 2.2. Denote by .7" the set of admissible feedback maps on R N, and set, for 4) E .7-,

J~(x, 4))

:=

/0"

e-~te(ux(t, 4)), 4)(yx(t, 4)))) at

v.~(x) = inf Jj,(x, 4)). (i) Prove that v~ = v. [Hint: one inequality is trivial, the other is easily obtained by adding time as a state variable.] (ii) Prove directly (without using (i)) that v r is continuous and satisfies the Dynamic P r o g r a m m i n g Principle. 2.3. Let H : R 2N ~ IR be continuous and satisfy (2.18), and assume Ul,U2 C B UC(R N) solve, respectively,

ul + g ( x , Dul) + g(x) <_O

i n n N,

u2 + H(x, DE2) + h(x) >_0

in R N,

with g, h c BUC(~N). Show that

sup(~l - u2) _< sup(h - g). R~v

~x

2.4 (VARIABLE INTEREST RATE). Let A : ~ N • A --* It( satisfy 0 < Ao _< A(x, a) _< 1l/r and IA(x,a) - A(y,a)] < w~(Ix - y]), where w), is a modulus, for all x,y E R N and a C A. Consider the payoff

J ( x , , ) := J 0 ~ e x p ( - fot)~(y~(s),a(s))ds)~(yx(t),(~(t))dt under the hypotheses (A0)-(h4). (i) Prove that the value function v = inf~ J satisfies the following D P P : =

g(yx(s), a(s)) exp -

A(yx(T), o~(w))dT ds

aEA ~

+v(yz(t))exp(--/OtA(Yx(T),a(T))dv)}. (ii) Prove that v is a viscosity solution of sup{ A(x, a)v - f(x, a). Dv - / ( x , a) } -- 0 aEA

(iii) Rewrite the Bellman equation in (ii) in the form (2.88)

v + H(x, v, Dv) = 0

in ]I(N

in R N .

2. THE INFINITEHORIZONPROBLEM

145

with H continuous and such that (2.89)

r ~-~ H ( x , r,p) is nondecreasing in R, for all x , p E ~ N H(y; r, tz(x - y) - vy) - H ( x , r, p ( x - y) + 7 x )

(2.90)

< pL'}x - yl 2 + ~-K(1 + lyl 2) + ~K'(1 +

Ixl 2) + ~(Ix -

yl, R)

for all p, 7, "~ > 0, x, y E I~N, r E IR such that Ir ] < R where g is a modulus, L' = L/,~o, K ' = K/Ao. 2.5. Prove the analogue of Theorem 2.12 for equation (2.88) and H satisfying the structure conditions (2.89) and (2.90). (Hint: follow closely the proof of Theorem 2.12, but treat separately the case ul(Y) <_ u2(Y) and the case ul(Y) > u2(~).] 2.6. Discuss the possibility of dropping some assumptions on A in the previous exercises. 2.7. Prove that v E B C ( R N) also when the uniform continuity of 6 in (A4) is weakened to

(2.91)

]e(x, a) - 6(y, a)l _< w~(Ix - Yl, R)

for all a E A and x , y E R N such that IxI, lyl -< R, and all R > 0, and wt is a modulus. Prove that Theorem 2.12 keeps holding if ~t is allowed to depend on R as above and with B U C replaced by BC. [Hint: see Remark 2.13.] 2.8. Prove that the value function v in Example 2.1 is semiconcave if and only if A _> 2. Note that the sufficient condition for semiconcavity in Remark II.4.10 (i.e., > 2L) is almost optimal. 2.9. Prove that "4-Ix]b and the constant 0 are viscosity solutions of u - ( 1 / b ) x . D u = 0 in RN for any b > 0. In dimension N = 1 we also have the solutions • b. 2.10. Under the assumptions of Lemma 2.20 prove (2.32). [Hint: for the inclusion "D" prove the following mean value theorem: for a probability measure m on A and an m-measurable G : A --~ R M, fA G(a) d m E ~6G(A).] 2.11. Prove the assertion in Remark 2.42. Using the backward D P P prove that the value function v satisfies, at any x E (9, ~v +

sup {0%(x;-f) (f,~)E(FL)(x)

- 6} >_ O,

and deduce from this that v is a subsolution of - ) w - H ( x , D r ) = 0 at points of (9. 2.12. In the assumptions of Theorem 2.40 prove that a locally lipschitzean supersolution satisfies Au(x) + lim inf sup [ u(x) - u ( x + t f ) _ 6} > O . t--0+ (Z,QE(FL)(x) t. t --

146

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

2.13. Extend the property of Dini derivatives given in Remark 2.44 to the following: if, for some b E IR, O-u(x; q) > b for all q E C, then there exist sequences x,~ ~ x, p, E D-u(x,~), such that Pn 9q >- b for all q E C. Note that this implies

O-u(z;q) < l i m s u p s u p { p , q : p E D - u ( z ) } Z~X

for all q E R N, which complements Lemma 2.37. 2.14. Give a direct proof of the statement (i) ~ (iii) in Theorem 2.40 by means of Exercise 2.13. [Hint: consider the function U(z, s) := u(z) + s, s E ]R, and its derivatives in (x, 0).] 2.15. Prove Lemma 2.48. [Hint: to prove "_<" choose b~ such that wb~(x) ~ w(x) and show that

w(x + hq) - w(x) < liminf{ Owb~(x;q ) + w ( h ) }. h To prove ">" for h~ ~ 0 + choose bn such that w b" (x + h,~q) <_ w(x + hnq) + h 2, show t h a t wbn(x) ~ W(X) and Owb'~(x;q) -- w(hn) _< w(x + hnq)hn - w(x) + hn.] 2.16. Fill in the details of the proof of Proposition 2.47. 2.17. Extend the chain rule Lemma 2.50 to the case y has just right or left-hand derivatives. 2.18. Prove that a necessary condition of optimality holding for all t >_ 0 is that

(2.92)

(f, e) := 1 ~ 1t,~ Jo [t~ ( f ( y ( t ) , e ( t + s)),e(y(t),a(t + s))) ds

satisfies O-v(y(t); f) + g <_ )~v(y(t)), for all t~ ~ 0 + such that the limit in (2.92) exists. [Hint: follow the last part of the proof of Theorem 2.40.] Note that, by the compactness of (FL)(x) for all x, the limit in (2.92) exists for some tn ~ 0 +. 2.19. For what values of • > 0 is the value function of Example 2.4 an intrinsically classical solution of the HJB equation in ] - 1 , 1[? 2.20. Derive a necessary condition for optimality from the Backward D P P Proposition 2.25, and discuss its sufficiency. 2.21. Consider the multivalued feedbacks S~(z) := { a e A : p. f ( z , a ) + e(~,a) < ),v(z) Vp e D % ( ~ ) }, sD.(z) := { ~ e s . ( ~ ) : - 0 % ( x ; - f ( z , ~ ) ) + e(z,~) < ~.(z) }, and prove that (i)

they are weakly optimal under the assumptions of Theorem 2.61 (i);

(ii) Sz)+(z) C SD(Z) _C S~(z) C S~(z) for all z E ]~N; (iii) Sit is a fully optimal feedback if D+v(x) 7~ ~ for all x E N g and v is locally lipschitzean.

3. THE FINITE HORIZONPROBLEM

147

3. T h e f i n i t e h o r i z o n p r o b l e m 3.1. T h e H J B e q u a t i o n In this section the cost to minimize is

J(x, t, ~) :=

e-~se(~x(s, ~),

~ ( s ) ) ds + e-~tg(y~(t,

~)),

where the r u n n i n g cost g and the constant interest rate ~ satisfy (A~)

$ satisfies the hypotheses in (Aa); A > 0;

and the terminal cost g satisfies, in this subsection,

(A5)

g E BUC(]~N); that is, there exists a modulus ~g and a constant G such that Ig(x) - g(Y)l <- wg(I x - Yl) and Ig(x)t < v for all x , y e R N .

In w we consider the case of unbounded terminal cost. Following the terminology of the calculus of variations, the problem of minimizing J ( x , t, 9 ) is called a Mayer problem if ~ - 0 and A -- 0, a Lagrange problem if g - 0, and a Bolza problem in the general case. If A = 0 and g is Lipschitz continuous with respect to the state variables uniformly in the control variables, then the Bolza problem can be converted into a Mayer problem by adding a scalar state variable YN+I to the vector y, with dynamics y~V+l(S) = t ( y ( s ) , ~ ( s ) ) ,

s > 0,

YN+I (0) ~- O, and minimizing the payoff yN+l(t) + g(yx(t)). (This new terminal cost is unbounded but we can replace it with ~(YN+I + g(Y)) where ~b is any bounded and strictly increasing function, e.g., r = a r c t a n r , without changing the optimal control problem.) Therefore in the sequel the reader may assume for simplicity that g is null, at least when A = O. There are more general formulations of finite horizon problems. It is often important to incorporate constraints on the state variables, and this can be done by the methods of Chapter IV. It is also important to consider the nonautonomous case, that is, the case of dynamics and running cost depending explicitly on time s > O: f -----f(s, y, a),

g ---=g(s, y, a) .

If the dependence of f on the variable s is Lipschitz continuous uniformly in y and a, then this case can be reduced to the standard autonomous one by treating time as an extra state variable, that is, by adding Y~v+I(s) = 1,

YN+I ~- O,

s > 0,

148

III. PROBLEMS W I T H U N R E S T R I C T E D STATE SPACE

to the system (S). When f depends on time in a less regular way we have to treat the nonautonomous problem separately, but this extension is not hard: see Remark 3.10 and Exercise 3.7. The value function of the finite horizon problem is

v(x,t) := inf J ( x , t , a ) . aE.A

We begin with a regularity result for v. As for the infinite horizon problem, the value function fails to be differentiable even for linear systems and smooth costs: the reader can easily construct some examples or see Exercises 3.3 and 3.4. PROPOSITION 3.1. Assume (A0), (A1), (A3), (A~), and (A5). Then (i)

v is bounded and continuous in R N x [0, T] (i.e., v E BC(1R N x [0, T])), for all T > O;

(ii)

,k > 0 implies v E BC([( N x [0, +ocD;

(iii)

g Lipsehitz continuous and ,:t(r) = Ltr (i.e., ~ is lipschitzean in y uniformly in a) imply that v is lipschitzean on K: x [0, T] for all compact K. C_ IRN and T>0.

PROOF. Let us fix x . z C R N, t,'r E [0, T], ~ > 0 and ~ E ,4 such that

e - ~ e ( y ~ ( s ) , ~(~))d~ + e - A ' g ( y z ( ~ , ~ ) ) - ~ .

v(z,~) > Then (3.1)

v(x,t)-v(z,T) <_

2

e - ~ e ( y x , ~) ds + e - ~ t g ( y x ( t , ~ ) ) -

< MIt-,-I+

/o"

e-~e(yz,~)ds

f

- e-a'e(y~(r,~)) + c

tAT

e - ) ' ~ t ( l y x ( s ) - yz(s)]) ds

J0

+ It - ~-I a + e-~'"~g(l~(t) - yz(T)l) + ( M 4- a ) l t - T t -[- Ta3g(eLTIx -- zl) -~ a3g(C It - TI -~- eLTIx -- zl) ~- ~,

where C is a Lipschitz constant for yz(" ,-a) on [0, T] and we have used (A~), (A5) and (1.5). This proves the continuity of v. The boundedness of v follows from

]J(x,t,a)]
forallxENN,

tE[O,T],aEA.

The estimate (3.1) proves (iii) as well, because C can be chosen independently of x in a compact set by (A1). To prove (ii) it is enough to observe that

]J(x,t,a)]<M

e-a~ds+G

for all x , t , a .

3. T H E F I N I T E HORIZON PROBLEM

149

PROPOSITION 3.2 (DYNAMIC PROGRAMMING PRINCIPLE). A s s u m e (A3), (A~), and (A5). Then for all x E R N and 0 < r <_ t (3.2)

v(x,t)=

(A0), (A1),

infl f'e-~Se(yx(s),a(s))ds+e-~'v(yx(r),t-r)}. aC,.4 ~ dO

PROOF. Since v(z, O) = g(z) by definition, (3.2) reduces to the definition of v when T = t. For T < t and a E ,4 fixed, we define ~(s) := a ( s + T) and observe that (3.3)

J ( x , t, a) =

/0

e-~Sb(yx(S), a(s)) ds +

= I1 + =

fJO

t--T

e-~Se(yz(s), a(s)) ds + e-;~tg(yx(t))

e-~e-~'t(yx(r + 7), a(r + 7)) dr + e-~re-~(t-r)g(yx(t))

I 1 -[- e-~'g(y~(T, a), t -- T, 8) >_ I1 + e-X'v(y~(~ -, a), t -- T),

w h e r e / 1 denotes the first integral in (3.3). By taking the inf over .4 we get the inequality ">" in (3.2). To prove the opposite inequality we fix a E A, set z := yz(r, a), fix z > 0 and a 1 E A such that v ( z , t - T) + s > J ( z , t - r, al). We define the control

[~(s) s< ~(s) = "[~X(s- ~) s >

T~

,,

and let ~ and yl be the trajectories corresponding to U and a 1, respectively. Then (3.4)

v ( x , t ) <_ J(x,

t, ~)

fo ~ ~-~e(~(s),

~(s))

ds + ~'e-~'e(~x(S),

~(s)) ds + e - ~ ' g ( ~ ( t ) )

= I1 + e -'x" ~0 t-T e-Are(y 1 (r), ct1 (r)) dr + e-ATe-A(t-r)g(y1 (t -- 7-)) = I1 + e - ~ T J ( z , t -- r, a 1) <_ 11 + e-~Tv(z, t - 7) + z, where I1 is the first integral in (3.4). Since a and ~ are arbitrary we have obtained the desired inequality. REMARK 3.3. It is not hard to see from the arguments of the proof of Proposition 3.2 that if c~* is optimal for (x, t) (i.e., v(x, t) = J ( x , t, a*)), then the infimum in the D P P (3.2) is attained at a* for all % and that a * ( - + ~-) is optimal for (yx(7, a*), t - "7") for all 0 < 7 < t. Moreover the function 7- ~

/0"

e-~Se(yx(S), ~ ( s ) ) as + e-~'V(Vx(T), t -

~)

is nondecreasing for all a E .A, and it is constant if and only if a is optimal for (x, t). See Remark 2.6 for the corresponding facts in the infinite horizon ease and their interpretation in terms of Bellman's Principle of Optimality.
150

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

REMARK 3.4. Proposition 3.1 and the D P P still hold if we replace the set of admissible control functions `4 in the definition of the value function with a n y 13 C .4 with the properties of being closed with respect to the operations of t r a n s l a t i o n in time a n d piecing together two different controls: see Remark 2.7. Thus for instance v~(x, t) := inf~e p J ( x , t, a), the value function associated with piecewise c o n s t a n t controls, satisfies (3.2) with .4 replaced by ~ . < As in Section 2 the D P P implies t h a t v satisfies the H a m i l t o n - J a c o b i - B e l l m a n equation. We recall the definition of the H a m i l t o n i a n H

H ( x , p ) : = sup{ - f ( x ~ a) . p - g(x, a) } .

(3.5)

a6A

PROPOSITION 3.5. A s s u m e (A0)-(A3), (A~), (As). Then the value function v(x, t) is a viscosity solution of

vt + )~v + H ( x , D v ) = 0

(3.6)

in R N

•

]0, ~-OO[,

where vt denotes the partial derivative with respect to the last variable and D v = D x v the gradient with respect to the first N variables. PROOF. Let 4) E C1(]~ N x ]0, +oo[) and (x, t) be a local m a x i m u m point of v - 4); t h a t is, for some r > 0 (3.7)

v ( x , t ) - v ( z , s ) _> 4)(x,t) - 4)(z,s),

if I x - z I < r, i t - sl < r .

Fix a n a r b i t r a r y a E A a n d let yz( - ) be the solution corresponding to the c o n s t a n t control c~(7) = a for all T. For 7 small enough lye(v) - x[ < r by (1.3), a n d t h e n

4)(z,t) - 4)(y~(T),t-- T) < v ( z , t ) -- V(yx(7-), t -- "r)

(3.8)

for 0 < v < t o .

By using the inequality "<_" in the D P P (3.2) we get (3.9)

r

t) - r

t - T)

<_

i"

e-~'e(yx(S),~(s))ds

+ v(yx(~-),t-

T)(e-~" - l) .

Next we divide by ~- > 0 a n d pass to the limit as T ~ O, to o b t a i n

et(x,t)

- D4)(x,t) . y'~(o) = 4)t - D4). f ( x , a ) < ~.(x,a) - ,~v(x,t),

where we have used the differentiability of 4) a n d the continuity of v, 9:~, f , and f. Since a E A is a r b i t r a r y we get

4)t(x, t) + Av(x, t) + sup{ - f ( x , a ) . D4)(x, t) - g(x, a) } < 0, aEA

which means t h a t v is a viscosity subsolution. To check t h a t v is a supersolution we assume now t h a t (x, t) is a m i n i m u m point of v - 4), so that (3.7) holds with the inequality reversed. For each r > 0 a n d ~" > 0,

3. T H E F I N I T E HORIZON PROBLEM

151

by the inequality "_>" in the D P P (3.2), there exists a E ,4, depending on E and r, such that

v(z,t) >_

e-~'g(yx(s),c4s))ds+e-~'v(y~(r),t-r)--TE.

Then (3.8) holds with the reversed inequality, and so does (3.9) with an extra term - r E in the right-hand side. As before we divide by r > 0 and let r ~ 0: we have reduced to the classical proof because we have replaced v by the differentiable test function 05. However the details are a bit more lengthy t h a n in the first part of the proof because c~ depends on T and it is not a constant function: we leave them as an exercise for the reader since they are analogous to those of the proof of Proposition 2.8. The conclusion is

05t(x, t) + )w(x, t) + sup{ - f ( x , a) . D05(x, t) - e(x, a) } >_ -r aEA

which completes the proof by the arbitrariness of s. Next we characterize the value function v as the unique viscosity solution of the HJB equation (3.6) with the natural initial condition (3.10)

v(x, O) = g(x)

for all x e finN.

Indeed, as in Section 2, we want to compare v with any subsolution of the initial value problem (3.11)

fut+ ku+ g(x, Du)=O ( u ( x , O) = g(x)

in]R N • in ]RN,

for T < +oc, namely, a subsolution Ul of the HJB equation such that Ul(X,0) _<

g(x) in ]RN, and any supersolution of (3.11), that is, a supersolution u2 of the HJB equation such that u2(x, 0) >_ g(x) in IRN. By analogy with Section 2 a subsolution and a supersolution of (3.11) may be called, respectively, a verification function and a falsification function. We call v a complete solution of (3.11) in a set of functions if it is the maximal subsolution and the minimal supersolution in such a set. Of course a complete solution in a set is the unique solution in the same set. A first result for the case A > 0 is an easy consequence of the theory in Section 2. COROLLARY 3.6. Assume (A0)-(A3), (A~), (As) and ~ > O. Then the value function v is the complete solution of (3.11) with T = +oc in BC(]R N • [0, oc D. PROOF. The value function v has the desired regularity by Proposition 3.1 (ii). The proof follows from Remark 2.14. It is enough to observe that H : ]~2N+2 __.+]t~, ~I(x, t,p, po) := A-l(po + H(x,p)) satisfies the structural assumption (2.18), rewrite the P D E in (3.11) as

u(z) + H ( z , D,u) = 0

in ~

: = ]~N X

]0, ~ [ ,

where z := (x, t), Dzu := (Dxu, ut), and note t h a t the b o u n d a r y condition Ul _< u2 on 0f~ (see Remark 2.14) follows from ul(x, O) < g(x) < u2(x, 0) in ]RN.

152

IIL PROBLEMS W I T H U N R E S T R I C T E D STATE SPACE

Next we drop the assumption A > 0. We recall that a comparison result for (3.11) with A = 0 was already proved in Chapter II under different assumptions on H and on the sub- and the supersolution (see Theorem II.3.7). The main result of this section is the following. THEOREM 3.7. Assume H : ~ 2 N __+ ~ is continuous and satisfies (2.18), T E ]0,+oc[, and A > O. If ul, u2 6 BC(]~ N x [0, TI) are, respectively, a viscosity sub-

and supersolution of (3.12)

ut + Au + H ( x , Du) = 0

in R N

• ]0, T [ ,

then (3.13)

sup

( u 1 - u 2 ) _<

~ N X [0,T]

sup

(Ul - *t2) -t- 9

~ N X {0}

In particular, if H is given by (2.9) and (A0)-(A3), (A~), and (As) hold, then the value function v is the complete solution of (3.11) in BC(]~ y • [0, T]). PROOF. The second part of the theorem follows easily from the first because for any subsolution u of (3.11) u - v < 0 on R N • {0}, by definition, and then u ~ v on R N x [0, T], and similarly v is below any supersolution. The proof of the comparison assertion is based on the arguments in the proofs of Theorem 2.12, and Remarks 2.13 and 2.14, and makes use of Lemma II.2.10. Define 4)(x,y,t,s)

:= Ul(X,t) -u2(y, s ) -

I~ - yl ~ + tt - sl 2 _ ~ ( ( x ) m

2~

+ (y)m) _ ~(t +

~),

where s,/3, m, r / a r e positive parameters, (x} = (1 + IxI2) 1/2. Set A:=

sup ( U l - U 2 ) + RN x {o}

and assume by contradiction there exists 5 > 0, 5 , t - s u c h that (ul - u 2 ) ( 5 , t ' ) = A+5. Choose/3>0and~>0suchthat 2fl(~) m + 2 ~ t _ < 5 / 2 for a l l m _ < 1, so that (3.14)

5

A+~<_A+5-2B{5)"-2V-{=4)(5,'s

sup

]t~2N • [O,T] 2

4).

Since 4) is continuous and tends to - o c as Ix] + ]y] --* +ec, there exist 5 , ~ , t , ~ such that (3.15)

sup

4) = 4 ) ( ~ , y , t , ~ ) .

~2N X [0,T] 2

Since 4)(~,~,t,~) >__A, we have (3.16)

/3((y}m + (~)m) __<sUpUl -- infu2 -- A =: Cl,

for all z > 0, m E ]0, 1].

3. T H E FINITE H O R I Z O N P R O B L E M

153

Now we fix rn := ( ~ / K c ~ ) A 1, where K is the constant appearing in assumption (2.18). Observe that ~ , y ~/~(0, (c~//3)Um) =: B~ for all e > 0, by (3.16), and let w be a modulus such that (3.17)

u i ( x , t ) - u i ( y , s ) = co(Ix - Yl +

It -

~1)

for i = 1, 2, x, y ~ B~, t, s ~ [0, T]. From the inequality (I)(5, 5, t, t) + 4)(~, ~, a, a) _< 2(I)(5, ~, t. ~) we easily get (3.18)

8

and the boundedness of Ul; u2 gives, for a suitable constant C, (3.19) By combining (3.17),(3.18) and (3.19) we get (3.~o)

I~ - 91 ~ + I~ - a? < ~ ( c , / 7 ) . s

Now we show that neither t nor a can be 0. In fact, in t - 0 ~(~, ~, O, ~) < u~ (X, O) - u~ (5, O) + u:(5, O) - u~(~, a) < A + w(Cv'~) and we get a contradiction to (3.14) and (3.15) by choosing E such that w ( C v ~ ) < 5/2. The proof that a > 0 is analogous. Next define the test functions ~(x,t)

:= ~ 2 ( ~ , ~ ) +

O(y,8)

:= Ul(5. t ) --

I* -

Yl 2 + It - al 2 2~ + ~((~)~

+ ( ~ ) ~ ) + ~(t + ~),

I ~ - yl2 + I ~ - st2 -- /~((~)m + (y}m) _ rl(t+ s), 2~

and observe that u~ - ~ o attains its maximum at (5,t) and u 2 - ~b attains its minimum at (~, a). Moreover

i-a

~(~,~)

= --

~s(~,~)

=

s

+ 7,

D~0(~, ~ ) _- _ _-x y + 75,

- 7,

Dygo(~,-$) - x - y

~-a

r~,

E

where 7 = rn~(5) 'n-2 and r = mfl(~} m-2. Since ~,a C ]O,T], by the definition of viscosity sub- and supersolution and Lemma II.2.10, we have (3.21)

~t(~, t) + Aul(~,t) + H(~, Dx~2(g,t)) < 0 r

+ ),u2(~,g) + H ( 9 , Dy~b(~, a)) _> 0 .

We use (3.22)

ui(~,~)

- u 2 ( y , ~ ) _> r

2 A + a / 2 >_ 0

154

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

and the structural assumption (2.18) on H to obtain from (3.21) 27 < L I~ - ~1 ~ + ~K(~> 2 + ~K<~> 2 + ~ ( 1 ~ - ~t)

(3.23)

_< 2Lw(Cx/-~) +

mK~((~) m +

_< o(1) + mKCl <_ o(1) + r/

(5)m) + ~ze(C~/~) as s ~ 0 +,

where in the second inequality we have used (3.19) and (3.20), in the third inequality we have used (3.16) (and o(1) indicates a function of z going to 0 as s ---* 0+), and in the last inequality we have used m < ~/Kcl. Now we reach the desired contradiction by taking ~ small enough. ~1 REMARK 3.8. If A = 0 in (3.12), we can drop the symbol of the positive part "+" in (3.13) and conclude that sup

(~1 - ~2) <

]t~N X [0~T]

sup

(~ -

~).

~N X {0}

In fact in the proof of the theorem we use that A > 0 only in the estimate (3.22), which is useless if A = 0. The assumption (2.18) can be weakened by replacing the term w~(]x - Yl) by "~(Ix - Yl, R), where w ( - , 9) is a modulus, and the inequality is assumed to hold for all Ix], lYl -< R and all R > 0. In fact, in the proof ~ , ~ E B(0, R), R = (Cilia) 1/m for all z > 0, and so in (3.23) ~(1~-

~1,~) < ~(c~,~)

----~ 0

as E - ~ 0 +

Therefore, if H is given by (3.5), the assumption of uniform continuity of the running cost in (A~) can be weakened to ]e(x, a) - t~(y, a)l < ~(Ix - Yl, R) for all a C A, ]x[, lyl <- R and all R > 0.

<

REMARK 3.9. As in w we can apply the theory of this section to give verification theorems, relaxation results, and to prove stability for perturbations and approximations of the problem. We encourage the reader to figure out how to state these results. < REMARK 3.10 ( B A C K W A R D H J B E Q U A T I O N AND N O N A U T O N O M O U S S Y S T E M S ) . Since we have assumed our system f and running cost ~ to be autonomous (independent of time) the problem of minimizing J(x, T - t, o) is equivalent to that of minimizing

I ( x , t , a ) :=

[

e~(t-~)s

where y(s; x, t, ~) indicates the position at time s __ t of the solution of

y' = f(y, o), y(t) = x.

3. THE FINITE HORIZON PROBLEM

155

The value function of this problem

w(x, t) = inf I(x, t, a) aCA

is related to the previous one by the formula

(3.24)

v(x, t) = w(x, T - t).

It is not hard to check the Dynamic Programming Principle for w: (3.25)

w ( x , t ) = inf

e;~(t-s)g(y(s;x,t,a),a(s))ds+e:~(t-r)w(y(T;x,t,c~),T)

c~E .A ~ J t

}

for all 7 C It, T], and to deduce from this, or from (3.24), that w is a viscosity solution of

- w t + Aw + H ( x , D w ) = 0

in A N

X

] - - o o , T[,

w(x, T) = g(x), under the same assumptions as the corresponding statements for v. If one is really interested only in minimizing the cost functional over the interval [O,T], the choice between the intermediate functionals I and J is a m a t t e r of taste. We made the first choice for simplicity of notation and because initial value problems for forward evolution equations are much more usual in the theory of PDEs t h a n terminal value problems for backward equations. However, the second choice b e t t e r fits the direct treatment of nonautonomous problems (see the comments at the beginning of this section), and that is probably the reason why it is more usual in the control theoretic literature. In fact, if the system f depends explicitly on the time variable, a notation such as y(s; x, t, a) is m a n d a t o r y because the t r a j e c t o r y depends on the initial time, and the Dynamic Programming Principle is (3.25) (with a possible explicit dependence of g on the time variable s). The proof is essentially the same, once we replace the semigroup property for autonomous systems

yx(t + s, ~ ) = yy(x,t)(s, ~(- + t ) )

t,s>O

with the unique evolution property

y(t + s; x, -z, a) = y(t + s; y(t; x, ~-, a), t, a)

s >_ O, t >_ T .

The Hamiltonian in the HJB equation depends also on the t variable in this case (see Exercise 3.7), but it is not hard to extend the theory of the present section to cover this situation. In this setting it is also possible to consider time-dependent constraints on the controls. <~ REMARK 3.11. By Remark 3.4 and the proof of Proposition 3.5, v ~ C B C ( ~ N • [0, T]) and solves (3.11). Thus, under the assumptions of Theorem 3.7, v = v~; t h a t is, measurable controls and piecewise constant controls generate the same value function. <~

156

III. PROBLEMSWITH UNRESTRICTEDSTATESPACE 3.2. L o c a l c o m p a r i s o n a n d

unbounded

value

functions

In this subsection we prove a local comparison result typical of evolutive HJB equations, saying that two viscosity solutions ul, u2 of (3.26)

ut + H ( x , Du) = 0

in the cone

c :-- { (x, t ) : 0 < t < T and Ixl < C ( T - t) }, such that ul _~ u2 at the initial time t = 0 in the ball ]x I < C T , satisfy the same inequality in the whole cone C, provided C is a Lipschitz constant for the Hamiltonian H ( x , p ) in the variable p. This is usually called a cone of dependence result, and it is a typical property of hyperbolic PDEs, whereas global comparison results are typical properties of elliptic and parabolic PDEs. Of course the result has an easy control-theoretic interpretation. Then we apply this result to prove global comparison and uniqueness of merely continuous solutions of (3.26). This allows us to extend the theory of the previous subsection to the case of unbounded running and terminal costs, paying a price in the strengthening of the structural conditions on the system. In particular, we assume the existence of a constant K > 0 such that (3.27)

]f(x,a)l <_ K ( l x l + 1)

for all x 9 ~ x

a 9 A,

which is true, for instance, for linear systems with a bounded control set A. THEOREM 3.12. Let H : B(O, C T ) x I~N -~ R be continuous and satisfy (3.28)

I H ( x , p ) - H ( x , q)l < C IP - ql

(3.29)

]H(x,p) - H ( y , p ) I < w(lx - y]) + w(lx - Yl IPl)

f o r all x , y 9 B(O, C T ) , p , q 9 ]~N where w is a modulus and C , T > O. l f ul, u2 9 C(-C) are, respectively, a viscosity sub- and supersolution of (3.26) in C and Ul(X , O) ~_ U2(X , O) for all x 9 B(O, C T ) , then u I ~_ u 2 in C. PROOF. We modify the auxiliary function in the proof of Theorem 3.7 in order to localize the problem in the cone C. Assume by contradiction there exists 0 < 5 < T and (5, t-) such that (3.30)

(U 1 - u~)(5, ~) = ~

and

151 < C ( T - t ) - 2 5 .

Take M > s u p { l u l ( x , t ) - u2(y,s)l : ( x , t , y , s ) 9 C 2} >_ 5 and h E CI(R) such that h ~ < 0, h(r) = 0 for r < - 5 , h(r) = - 3 3 I for r > 0. Now define (x)z := (ixl2 +/32)1/2 and r

: : Ul(X,t ) -- u2(y, S) -- Ix - yl 2 + It - sl 2 2~ - ~7(t + s) + h ( < x > e - C ( T - t)) + h ( < y ) ~ - C ( T - s ) ) ,

3. T H E FINITE H O R I Z O N P R O B L E M

157"

where ~ ~/,/3 are positive parameters. Let (5, ~, 7, ~) E ~2 be such that max~ = ~(~,t,~) ~2

.

We claim that either t = 0 or ~ = 0 or (~, ~, t, ~) E C2, for ~3 and r/small enough. In fact, if I~] = C(T - t ) or lY] = C ( T - g) we have r < M - 3M = - 2 M by the definition of h, because (x)~ > IxI for all x and /3. On the other hand, by (3.30) and the definition of h, for any/3 < ~ and r / < ~/4t, (3.31)

=~-2V~+2h((Y)z-c(r-t))

max~ > r

> ~,

which proves the claim. From the inequality ~(5, ~, t, t ) + ~(~, ~, ~ ~) < 2~(5, ~, t, ~) we get (3.32)

I~-u12 + It-~12 <_ u~(~,~) - ~ ( ~ , ~ ) + ~2(~,~) - ~ ( ~ , ~ ) ,

and thus (3.33)

I ~ - ~l ~ + I t - ~1~ < 2M~.

By the continuity of u~ and u2, (3.32) and (3.33), we obtain (3.34)

t~ -~1~ + I~ -~1~ _< Wl(e),

for a suitable modulus Wl. Now we show that neither t nor ~ can be zero if z is chosen appropriately. In fact i f t = 0, since ua(~,0) < ug_(~, 0),

9 (~,~, o,~) _< ~ ( ~ , o) - ~ ( ~ , o) + ~ ( ~ , o)

-

~(~,~)

_< ~ ( ~ ) ,

where w~ is the modulus of continuity of u2 in C, which gives a contradiction to (3.31) for z small enough. The proof that ~ > 0 is analogous. Next we define the test functions

~(x,t) := tb(y,s) :=

Ix - ~l ~ + It - ~l ~ 2z + ~(t + ~ ) - h ( ( x } z - C ( T - t ) )

I~- yl22~ + I~-

s[2

- ~7(~+ s) + h(~ - C(T - s)),

so that u~ - F has a maximum at (~,t) and u ~ - ~b has a minimum at (~,~). We compute the partial derivatives of p and r set X = (5)~ - C(T - t), Y = {~)~ - C(T - -~), and use the definition of viscosity sub- and supersolution to get

2~ <_C(h'(Y) + h'(X))

(--

+H ~,x-Y

+h'(Y)

-H

(2,-

~

'

h (x)

158

III. P R O B L E M S W I T H U N R E S T R I C T E D STATE SPACE

Now we add and subtract H(-2, (~ - ~)/~ + h ' ( Y ) ~ / ( y ) ~ ) to the right-hand side and use (3.28) and (3.29) to obtain (3.35)

2~] < C ( h ' ( Y ) + h ' ( X ) ) + w(l~ - ~])

Now we use (3.33), (3.34), and h' = -Ih'] to get that the right-hand side of (3.35) tends to zero as ~ --~ 0 +, a contradiction which completes the proof. REMARK 3.13. By combining the proofs of Theorems 3.7 and 3.12 the reader can easily extend Theorem 3.12 to the case where the assumption ul _< u2 at t = 0 does not hold, so that one proves the inequality (3.36)

sup(ul - u2) _< C

sup

(Ul - u2) 9

B(O~CT)x{O}

As in Theorem 3.7 one can include a term &u in the equation, provided A > 0, and obtain inequality (3.36) with ul - u2 replaced by (ul - u2) + in the right-hand side. By using (3.36) and an easy approximation argument we can relax the conditions on us and u2 by assuming their continuity on C W B(O, C T ) x (0) only. Theorem 3.12 can also be extended to the case where H depends explicitly on time t, by just adding the assumption t h a t H : C x l~ y --+ 1~ be continuous, see Exercise 3.5. <1 REMARK 3.14. It is easy to check that Theorem 3.12 holds in the cone {(x,t) : 0 < t < T and I x - x01 < C ( T - t)} for any x0, if (3.28) and (3.29) hold for all x, y e B(xo, C T ) and us _< u2 in B(xo, C T ) • {0}. <3 THEOREM 3.15. Let H : ~ 2 N ~ ~ be continuous and satisfy (3.37)

]H(x,p) - H ( x , q ) I < K(]x I + 1)lp - q]

for all x, p, q E ~ N and (3.38)

IH(x,p) - H(y,p)I <_ w(Ix - Yl, R) + or(Ix - Yl Ipl, R)

for all p E INN , x , y 9 B(O,R), R > O, where w is a modulus. I f us,u2 9 C(IK g x [0, T]) are, respectively, a viscosity sub- and supersoIution of (3.26) in ]~N X ]0, T[ and u l ( x , 0 ) _< u2(x,O) f o r all x 9 I~N, then ul <_ u2 in 11~y x [0~T]. PROOF. We m a y assume without loss of generality that T < l / K , because the proof of the general case is obtained by iterating the following argument on time intervals of fixed length smaller than 1 / K . We fix X0 9 ] ~ N and define r :-KT([xol + 1)/(1 - K T ) > 0, so that (3.39)

r = KT([xol + 1 + r) =: C T .

3. T H E F I N I T E HORIZON PROBLEM

159

We define the cone Cxo:={(x,t):0
v(x, t) : = inf ~f ftg(yx(s), a(s)) ds + g(yx(t)) } aEA "JO

as the unique solution of the associated HJB equation under the conditions t h a t g is continuous, t~ satisfies (3.40)

f g is continuous and bounded on B(0, R) x A; [ I t ( x , a ) - t?(y;a)l < w~(Ix- YI,R), Y R > O, x,y 9 B(O,R), a 9 A,

where w~ is a modulus, and the system f is locally lipschitzean in the state uniformly in the control and satisfies the growth condition (3.27). THEOREM 3.17. Assume (A0), (A1), (A2) with ~ f ( r , R ) - - L R r , (3.27), (3.40), g 9 C(]RN), and 0 < T < +oc. Then the value function v is the complete solution in C(]Ry x [0, T]) of the HdB equation (3.26) in IRy • ]0, T[ with H given by (3.5), plus the initial condition u(x, O) = g(x) in I~g. PROOF. The existence, uniqueness, and basic estimates on the trajectories under the current assumptions are given in Remark 5.7 of the appendix to this chapter. The proof of the continuity of v in Proposition 3.1 is still valid because the trajectories with initial points in a bounded set are uniformly bounded, and g and g are bounded on bounded sets of the state space. Also, v is a viscosity solution of (3.26) because this is a local property and the proofs of Propositions 3.2 and 3.5 remain valid. Moreover, the argument of the proof of Lemma 2.11 and (3.27) show that the Hamiltonian H satisfies (3.37), and (3.38) with :~(r, R) = LRr + a~(r, R). Therefore, by Theorem 3.15, v is below (respectively, above) any continuous supersolution (respectively, subsolution) u such that u _> g (respectively, u _< g) at t = 0. Thus v is the complete solution of the initial value problem (3.26) with g as initial data. REMARK 3.18. Theorem 3.17 can be extended without difficulties to cost functionals involving terms e -;~t with interest rate ~ > 0, as in w by Remarks 3.13 and 3.16, and to nonautonomous problems, see Remark 3.10 and Exercise 3.7. <~

160

III. PROBLEMS W I T H U N R E S T R I C T E D STATE SPACE

3.3. Equivalent n o t i o n s o f s o l u t i o n

In this subsection we study for the finite horizon problem the properties discussed in w167 and 2.4 for the infinite horizon problem: Backward Dynamic Programming, sub- and superoptimality principles, and the equivalence of viscosity solutions with bilateral supersolutions and with (generalized) Dini solutions, semiconcavity of the value function and existence of its directional derivatives. As a rule we leave the proofs as exercises for the reader as long as they are straightforward adaptations of those of w167 and 2.4. For simplicity we restrict ourselves to a Mayer problem, that is, to the case of null running cost and interest rate, and we assume the conditions ensuring the well-posedness of the HJB equation according to the theory of w167 and 3.2: '0
/~(x,a)=0,

A=0;

either (i): (A0) (A3) and g C B U C ( ~ n ) ; [ o r (ii): (A0), (A1), (A2) with ,~f(r,R) = LRr, (3.27) and g C C(R'~).

Since some results are of a local nature these assumptions can be weakened: the reader can search for more general conditions for each result. We recall that in this case the value function is v ( x , t ) := inf g ( y x ( t , ~ ) ) c~EA

and the HJB equation is (3.42)

ut + s u p { - f ( x , a) . D u } = O, acA

where D u = D z u denotes the gradient with respect to space variables x. We refer to w for the definitions of backward trajectories, bilateral solution, and exit time from ~, t~(a). We begin with the Backward Dynamic Programming Principle. DEFINITION 3.19. We say that x E R N is an optimal point for the horizon t > 0, and write (x, t) C O, if x belongs to an optimal trajectory for some larger horizon, that is, there exist z E ]I~N, s > t, and (~* c .4 such that v ( z , s ) = Y(z,s,a*) and x = yz(s - t, ~*). <~ PROPOSITION 3.20 (BACKWARD D P P ) . A s s u m e (3.41). t, ~- > O, and ~ E . 4 -

Then, f o r all x E m y ,

v(x, t) > V(yx(-~, ~), t + ~); for (x, t) C (9 and for all 7 > 0 small enough v(x, t) = max v(yx(-~', ~), t + r) . mE.A-

COROLLARY 3.21. A s s u m e (3.41). Then the value function v is a bilateral supersolution of (3.42) in ~N • ]0, oc[, and it is a bilateral subsolution of the same equation at any (x, t) C (9. I f in addition H ( x , 9 ) is strictly convex for some x, then D - v ( x : t) has at most one element f o r all t, and if (x, t) C (9, then D + v ( x , t) has at most one element also.

3. THE FINITE HORIZON PROBLEM

161

Next we s t u d y sub- and s u p e r o p t i m a l i t y principles. THEOREM 3.22. A s s u m e (3.41), ~ C_ R N open and u C C ( ~ x [0~ T]). following s t a t e m e n t s are equivalent:

Then the

(i)

u is a subsolution of (3.42) in ~ • ]0, T[;

(ii)

f o r all c~ E .4 and 0 < s < t < tx(C~) A T the function s ~-+ u(yx(s,c~)~t - s) is nondecreasing;

(iii)

u is a supersolution of - u t - sup{ - f ( x , a ) . D u } = 0 aEA

(iv)

u satisfies a local suboptimality principle, that is, for all x C ~, 0 < t < T, there exists r > 0 such that u ( x , t ) < inf U ( y z ( S , C ~ ) , t - s) aEA

(v)

in ~t x ]0, T[;

for all 0 < s < T;

u satisfies a global suboptimality principle, that is, u ( x , t ) < _ inf u ( y ~ ( s A t x ( a ) , a ) , t - - s A t x ( a ) ) c~cA

foraUO<s<_t.

PROOF. The proof of P r o p o s i t i o n 3.5 gives (iv) ~ (i); (iv) and (v) follow immediately from (ii). The equivalence of (i), (ii), and (iii) is a special case of T h e o r e m 2.33 because we can consider t i m e as an a d d i t i o n a l s t a t e variable by adding T' = --1, r(0) = t to the system (S), and then rewrite (3.42) in the form (2.40) with ~ -= 0 and ~ = 0 (this is allowed by R e m a r k 2.34). REMARK 3.23. F r o m T h e o r e m 3.22 we get t h a t u E C(Ft x ]0, T D is a viscosity solution of (3.42) in ~ x ]0, T[ if and only if it is a bilateral supersolution (it is enough to a p p l y T h e o r e m 3.22 to open subsets of ~ x ]0, T[ where u is continuous up to the b o u n d a r y ) . Therefore the value function v is also the unique bilateral supersolution of (3.42) in R N x ]0, ~c[ taking up the initial d a t a g continuously. <1 THEOREM 3.24. A s s u m e (3.41), ~ C_ ]~N open, and u C C ( ~ x [0, T]). Then u is a supersolution of the H J B equation (3.42) if and only if it satisfies a local superoptimality principle, that is, for all x E ~ and 0 < t < T there exists T > 0 such that u ( x , t ) > inf u ( y x ( s , ~ ) , t aE.A

s)

f o r all 0 < s _< T .

PROOF 1. As in the previous proof we add time as a state variable and we a p p l y T h e o r e m 2.32 with g ~ 0 and )~ = 0, which is allowed by R e m a r k 2.34. ~, PROOF 2. It is worthwhile presenting a simple direct proof if either ~ = ]I~N o r f is locally Lipschitzean in x uniformly in a. A s s u m e first ~ = ~ N . F i x 0 < a < t < T and set h(z) := u(z, t - ~),

w(x, s) := inf h(yx(s, a ) ) . aE.A

162

Ill. PROBLEMS WITH UNRESTRICTED STATE SPACE

By Proposition 3.5 w solves the HJB e q u a t i o n (3.42) in ]t~N X ]0, O~[, a n d the initial condition w(x, O) = h(x). O n the other hand, the function (x, s) ~-* u(x, s + t - a) satisfies the same condition at s = 0 and it is a supersolution of (3.42) in I~N x ]0, T + c r - t[. By a comparison t h e o r e m (either T h e o r e m 3.7 or T h e o r e m 3.17) w(x, s) < u(x~ s + t - a), a n d setting s = cr gives the conclusion with ~- = t. Note t h a t in this case we have proved a global s u p e r o p t i m a l i t y principle. Next let l~ be any open set. Fix x0 a n d let r be such t h a t B(xo, r) C_ f~, C be a n upper b o u n d on f i n / ~ ( x 0 , r) x A. Set ~- : = ( r / C ) A t a n d c = { (x, s ) : 0 < s < ~, Ix - x0l < c ( ~ - s) } . For (x,s) E -C we have ]y~(s,a) - xl _< Cs and t h e n y~(s,a) 9 -~. T h u s w is well defined in C and it satisfies (3.42) in C. In order to invoke the local comparison T h e o r e m 3.12 we need the condition I f ( x , a ) - f ( y , a ) l <_ Cll x - Y l for all x , y 9 B(xo, r) a n d a 9 As before we get w ( x , s ) < _ u ( x , s + t - c r ) for all (x,s) 9 and s < T + a - t, which yields the conclusion by choosing s = a. Now we consider Dini derivatives a n d Dini solutions. First we simplify a bit the n o t a t i o n for (generalized) Dini directional derivatives (see Definition 2.36):

O-u(x,t;q,s)

:= c9-u((x,t); (q,s)) :=

u(x + hp, t + hT) -- u(x, t) h '

liminf

h~O + (p,~)~(q,~)

u ( x + @ , t + h~) - u ( x , t ) h

O+u(x,t;q,s) :=cO+u((x,t);(q,s)) := l i m s u p h~O + (p,~')~(q,s)

a n d for the classical (one-sided) directional derivative CQu

Ou(x, t; q, s) . - cO(q, s) (x. t) : = lim '

~(x + @, t + hs) - ~(x, t)

h~O+

We define the compact set

F ( x ) := K6f(x, A), a n d note t h a t the HJB e q u a t i o n (3.42) c a n be rewritten (3.43)

m a x I(

feF(x)

Ut

- f " Dxu }

-~-

0

.

Definition 2.38 of (generalized) Dini subsolution becomes, for this equation, sup { - O + u ( x ; t ; f , - 1 ) } f~F(x)

< O,

a n d that of Dini supersolution, sup { - O - u ( z , t ; f , - 1 ) } fEF(x)

>_ O~

3. T H E F I N I T E HORIZON PROBLEM

163

and u is a Dini solution if it is simultaneously a sub- and a supersolution. The analogy with (3.43) comes from the formula

Ou(x,t; f , - 1 ) = f . D u ( x , t ) - ut(x,t) at points where u is differentiable. If a lipschitzean Dini solution u has all directional derivatives at a point (x, t), then it satisfies (3.44)

Ou O ( f , - 1 ) (x't) J

sup IeF(x)"

0

and we say t h a t it is an intrinsically classical solution of (3.42) (even if it may not be a classical solution!). See w for some more comments on these definitions. The connection of the two notions of weak so]ution is given by the next result. THEOREM 3.25. Assume (3.41), ~ C_]~N open, and u C C ( ~ x [0, T]). Then (i)

u is a viscosity subsolution (respectively, supersolution ) of (3.42) in ~ x ]0, T[ if and only if it is a Dini subsolution (respectively, supersolution);

(ii)

if, in addition, u is locally lipschitzean, then it is a viscosity solution of (3.42) in ~ x ]0, T[ if and only if sup { - O - u ( x , t; f, - 1 ) } = 0

fEF(x)

in f~ x ]0, T [ .

PROOF. As in the previous results of this section we can view (3.42) as an equation of the form studied in w Thus (i) is a special case of Theorem 2.40 and (ii) is a special case of R e m a r k 2.43. 9

In the second part of this subsection we give two regularityth results, one on the semiconcavity of the value function v, the other saying that v is a good marginal function, in the sense that it satisfies the assumptions of Proposition II.4.4, under some regularity assumptions on the data. In both cases, v turns out to have all classical directional derivatives, thus it is an intrinsically classical solution of the HJB equation, and to have a nonempty superdifferential at every point, a useful fact for the synthesis problem in next section. We say that a function is locally semiconcave in a set X if it is semiconcave in any bounded open set whose closure is contained in X. THEOREM 3.26. Assume (3.41) and, for some L, M > O, (3.45) (3.46)

If(x,a)

-

f ( y , a ) l <_ L I x - Yl

t f ( x - h, a) - 2 f ( x , a) + f ( x + h, a)I < M Iht 2,

for all x, y, h G ]~N, a E A, and g locally semieoncave. Then v is locally semiconcave in ~ N x [0. T].

164

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

A n example where (3.46) is satisfied with M = 0 is any system affine in the state variables for all choices of controls (see Exercise 3.9 for a stronger result in this case). We split the proof in some l e m m a t a . LEMMA 3.27. For a flxed t > 0 let T ~-* Zx(T;CY:t) be the solution of

f z ' ( r ) --- tf(z('r),c~(v)), (3.47)

T > O,

[z(0) = x.

Then yx(t, a) = zz(1; c~t, t) for c~t(T) : = c~(t~-) and v(x; t) = inf g(zx(1; c~, t)).

(3.48)

aEA PROOF. The first s t a t e m e n t is straightforward to check and the m a p G : ,4 --* ,4, G ( a ) : = ~t is a one-to-one correspondence. We say that ~ : f't --~ ]~N is semiconcave-convex in the open set f~ C ]~k if for some constant C I~(x - h) - 2~(x) + ~(x + h)l _< C l h l 2 for all x E f~ and h sufficiently small. T h e n a m e comes from the fact t h a t for N = 1, is semiconcave-convex if a n d only if b o t h c2 a n d - p are semieoneave. It is easy to see t h a t ~ is semieoneave-convex if it has a Lipschitz gradient, a n d the converse is also true (but less obvious, see Exercise 3.8). Note that a s s u m p t i o n (3.46) means t h a t f is semieoncave-eonvex in R N, uniformly for a E A. LEMMA 3.28. / n the notations of L e m m a 3.27, the map ~ ( x , t ) : = z = ( 1 ; a , t ) is semiconcave-eonvex, Lipsehitz, and bounded in any bounded set of ~ N • ]0, T[,

uniformly for ~ e ,4. PROOF. Fix a E ,4, r/E R, x, h E R N such that Ihl _< 1, and set z(s) : = Zx(S; cx, t),

z+(s) := zx+h(s; ~ , t + ~), W := z_ -- 2Z-i- Z+,

z_(s) :=

Zx_h(s;~,t

~:=z+--Z_

-

~),

.

In the sequel we do not write a a n y more in the formulas. First note t h a t ~(s) satisfies

{~

'=(t+v)f(z+)-(t-~)f(z_),

(3.49)

s>0,

((0) = 2 h .

Since the trajectories of (3.47) are b o u n d e d in [0, 1] uniformly for x in a b o u n d e d set and t E [0, T], there is C1 > 0 d e p e n d i n g only on Ixl such t h a t If(z+(s))], I f ( z - ( s ) ) l <_ C1 for all s E [0, 1]. T h e n I(t + v ) f ( z + ) - (t - r~)f(z_)l <_ t n l ( I + 2171C1,

3. THE FINITE HORIZON PROBLEM

165

and applying Gronwall's inequality to (3.49) we get

(3.50)

t~(s)l < c 2 ( b l + IhP)

for 0 < s < 1.

By a similar argument we also obtain (3.51)

[z-(s) - z(s)l <_ C a ( N + [hi)

for 0 < s < 1

(note t h a t this gives the Lipschitz continuity of ~). Next observe t h a t w(s) satisfies

w'=(t-rl)f(z_ w(O)

=

)-2tf(z)+(t+~)f(z+),

s>O,

o,

and estimate the norm of the right-hand side of the differential equation by (3.52)

t If(z_) - 2f(z) + f(z+)[ + [r/[ L [~[.

Now we write z_ = z + (z_ - z), z+ = z - (z_ - z) + w and use (3.45) and (3.46) to estimate the first term of (3.52) by T M ]z_ - z[ 2 + T L ]w[. Then we use (3.50) and (3.51) to estimate (3.52) by T L ]w] + C4(]r][ 2 + [h[2). By Gronwall's inequality this gives Iw(1)] < C4(Irll 2 + Ihl 2) e TL, which completes the proof.

9

LEMMA 3.29. Assume ~ : ~ --+ R N i8 semiconcave-convex, Lipschitz, and bounded. I f g : ~ N .._+]~ iS locally semiconcave, then g o ~ is locally semiconcave. PROOF. Set ~ := ~ ( x - h) - 2~(x) + ~ ( x + h). By assumption [~] < C~[h[ 2 and by writing ~ ( x - h) = ~(x) - (~(x + h) - ~(x)) + ~ we get (3.53)

g ( ~ ( x - h)) - 2g(~(x)) + g(~(x + h)) < Cgl~(x + h)

-

~(x)j 2 +

Lgl(I

< CgL~lh] 2 + LgC~,lhl 2,

where Cg and Lg are, respectively, the constant of semieoneavity and the Lipschitz constant of g on ~(f~). 9 PROOF OF THEOREM 3.26. By Lemma 3.27 v is the marginal function of the family g(zx(1; a , t)) with respect to the parameter a E A, and all these functions are locally semiconcave uniformly with respect to c~ by Lemmas 3.28 and 3.29 (see the explicit estimate (3.53)). Then, v is locally semiconcave by Exercise II.4.6. 9 Next we use relaxed controls (see w to represent the value function v as a good marginal function when A is compact. For a relaxed control function p E .Ar we denote by zx( 9 ; #, t) the solution of (3.54)

fZ'(T) = t f r ( z ( r ) , # ( 7 ) ) , Lz(o) x . =

r >0

166

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

It is straightforward to adapt the a r g u m e n t s of w to the finite horizon problem a n d show t h a t the value function of the relaxed problem coincides with v. T h u s L e m m a 3.27 gives v(x,t) = inf g(zx(1;#,t)) = inf G(x,t,p), where ~CA r

/~E.4~

v ( x , t , v ) := g(zx(1;v,t)),

(3.55)

.A~ := { measurable functions [0, 1] -+ A r } .

THEOREM 3.30. Assume (3.41) (ii) and A compact. Then there is a topology on ~4[ such that .4~ is a compact metric space and G : R g x [0, oc[ x M[ ---+N defined by (3.55) is continuous. In particular, for all (x, t) there exists an optimal relaxed control function. PROOF. By the theory of Warga [Wa72], A r can be identified with a compact subset of the dual space of the space /3 = { 0 : [0, 1] x A --+ R : 4)(",a) is measurable for all a, 4)(t, 9 ) is continuous and suplr

_< v>(t) for all t, for some ~ E LI([0, 1]) },

aEA

with the weak star topology. More explicitly, a sequence it,, in A r converges to tt if a n d only if for all 4) E/3 (3.56)

i.mio'l

1

Moreover, s i n c e / 3 is check the continuity a n d denote z~, z the O D E for w,~ we add [0, s] to get

4)(,, a) d # , ( 7 ) d T =

r

a) d#(T) d r .

separable, this topology is metrizable and so it is enough to of G on sequences. We take xn --+ x, t,, ~ t and #,~ ~ # corresponding solutions of (3.54) and w~ : = z , - z. In the a n d s u b t r a c t tfr(zn, #~) a n d tf(z, #n) and we integrate on

~(~) = ~ - x + ~(~)

(3.57)

io'i.

+ ~(~)

+

L"

r

&,

where s

~n(S) := (tn --t) ~n(s) : = t

io"

L

fr(zn,#n)dT

(fr(z,#n)-fr(z,#))d7

----t

io'i.

f(z:a) d ( p n - # ) d T "

~n :----t(fr(zn, #n) - if(z, #n)) 9 It is easy to check, as in L e m m a 2.20, t h a t f r satisfies the s t r u c t u r a l conditions (3.41) (ii) on f . T h u s all trajectories of (3.54) are uniformly b o u n d e d on [0, 1] for initial conditions in a b o u n d e d set. T h e n Ir

<__c

Itn

- tl

ICn(~)l -< tLR I w d ~ ) l

for all 0 _< s _< 1, for all 0 _< ~- _< 1.

3. .THE FINITE HORIZON PROBLEM

167

Moreover, by choosing 4)(7, a) null for ~- > s and equal to a component of f(z(T), a) for T _< S in the definition of weak convergence (3.56), we obtain l~(~(s)

= 0

for all s .

Therefore we apply Gronwall inequality to (3.57) and get that Iw,~(s)l ---*0 for all s c [0, 1]. Since g is continuous the proof is complete. REMARK 3.31. If, in addition to the assumptions of Theorem 3.30, f(x, A) is convex for all x, then f ( x , A) = if(x, A r) by (2.32). Then any relaxed trajectory is a standard t r a j e c t o r y because a controlled system y' = ~(y~ b), b C B, is equivalent to the differential inclusion y~ E ~(y, B) by a theorem of Filippov. Therefore Theorem 3.30 implies the existence of an optimal standard control function for all (x, t) under the present assumptions. This is a well-known result that can be proved also by convexity methods (Carath(~odory's theorem) instead of using relaxed controls. <] THEOREM 3.32. Assume the hypotheses of Theorem 3.30, g E CI(RN), and f differentiable with respect to x with D x f continuous. Then (i)

G defined by (3.55) is differentiable with respect to (x~ t) with D(x,t)G continuous;

(ii)

v is locally lipschitzean;

(iii) for all (x, t) the superdifferential of v coincides with Clarke's gradient, that

is, D+v(x, t) = Or(x, t); (iv) v is differentiable at all points where there is a unique optimal relaxed control. PROOF. Since A is compact, D x f is bounded on B(x, r) x A; thus a standard theorem on differentiability and continuity of integrals depending on parameters shows t h a t f r is differentiable with respect to x with D~fr(x, m) = fA Dxf(x; a) din, and it is easy to see t h a t this derivative is continuous with respect to (x, m). By the differentiability of solutions of an ODE with respect to the initial d a t a (Theorem 5.8), the i-th partial derivative of x ,-~ zx(1; #, t) at x0 is the solution, at time v = 1, of (3.58)

fq'(T) = tDxfr(z(7),#(v))q(v),

7- > O,

[q(0) =

where z ( r ) = zxo(v; #, t), ei has the i-th component 1 and the other components null, and D x f r is the Jacobian matrix of f r with respect to x (i.e., it has ofr/Oxj on the i-th row and j - t h column). By an argument similar to the proof of Theorem 3.30, one can show that the solution of (3.58) at v = 1 depends continuously on (xo,t,#). The derivative of t ~-* zx(1;#,t) at to can be computed either by a theorem on the differentiability of solutions of an ODE with respect to parameters, or by adding t as an extra variable to the system (3.54) with trivial dynamics

168

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

tl(T) = 0 and initial condition t(0) = to, and then using again Theorem 5.8 on derivatives with respect to initial data. The derivative turns out to be the solution, at time T -- 1, of (3.59)

f~'(~r) = toDzfr(z(T),#(T))99(3- ) + fr(z(T),p(T)),

T > O,

= o,

where Z(T) = Zx (7; #, to), and this can be shown to depend continuously on (x, p, to) as above. Since g E CI(•N), this proves the first statement of the theorem. Then v satisfies the assumptions of Proposition II.4.4 which gives D+v(x, t) = Or(x, t) and formulas for D+v(x, t) and D - v ( x , t) in terms of minimizers of G(x, t, 9 ) in .A~. In particular, the uniqueness of the minimizer implies the differentiability of v at (x, t), and such a uniqueness is equivalent to the uniqueness of the optimal relaxed control function by the proof of Lemma 3.27. 9 REMARK 3.33. From the arguments of the proof of Theorem 3.32 one can compute

D+v and D - v at a point (x, t) where all optimal controls are known. In fact, by Proposition II.4.4,

D+v(x, t) = U6{ D(x,t)G(x, t, p) : # E arg min G(x, t, . ) }, p E A~ minimizes G(x,t, 9 ) if and only if s ~ p(s/t) is an optimal control for (x, t), and we computed D(x,t)G in terms of solutions of certain linear differential equations. It is actually more useful to know explicitly some elements of D+v along an optimal trajectory: this leads to necessary conditions of optimality in the form of the Pontryagin Maximum Principle, as we see in the next section. It is not hard to prove (see Lemma 3.43) t h a t for any (relaxed) trajectory y( ) = y x ( ' , p) we have

DxG(y(t), T - t, [AT_t)

~-

p(t)

(with ttT_t(7") := # ( ( T - t)T)) and the costate or adjoint vector p(. (3.60)

satisfies

f p'(s) = - p ( s ) D x f r ( y ( s ) , p(s) ), (p(T) Dg(y(T)).

Therefore, if p is an optimal (relaxed) control, we have

(p(t),po(t)) E D + v ( y ( t ) , T - t),

po(t) := ~ t ( Y ( t ) , T -

t,#T-t) .

A formula for P0 can be deduced from (3.59) and other formulas are given in the next section in connection with the PMP. < The uniqueness of the optimal control is not necessary for the differentiability of v: it is enough to take f --- 0 to have a counterexample. A more subtle question is whether the uniqueness of the optimal trajectory for (x, t) is sufficient for the differentiability of v at (x, t). The answer is positive for some problems with smooth Hamiltonian, as happens in the calculus of variations, but in general it is negative, even if the system and the terminal cost are smooth and f(x, A) is convex, as the next example shows.

3. THE FINITE HORIZON PROBLEM

169

EXAMPLE 3.34. Consider the system in ]R2

{

y~ = a ,

y~

h(yl) + a2y2,

with A -- [-1,1], h smooth such t h a t h(0) = 0, h(s) > 0 for s # 0, and cost functional g(y2(t)) = y2(t). For the initial condition x = (yl(0), y2(0)) = (0, 0) the only optimal t r a j e c t o r y is (Yl0-), Y2(~-)) -- 0 which is obtained by just one standard control, a = 0 a.e., but infinitely many relaxed controls, all those such t h a t

~A a diz(w) = 0

for a.e. ~-

i

Thus v(0, 0, t) = 0 for all t. To prove t h a t v is not differentiable at such points we show t h a t v(0, z, t) = z and v(0, -~, t) = -Eet for all t, E > 0. If the starting point is (0, E), then c~ = 0 a.e. is again the unique optimal control, and now it is also the unique optimal relaxed control (thus v is differentiable at (0, z, t) for all t, a > 0). If the starting point is (0, - ~ ) , then ~(r)=~

15

1 -1+~51

fora.e.T

(Sa is the Dirac measure concentrated at a) is the unique optimal relaxed control whose corresponding optimal t r a j e c t o r y is ~- ~-. (0,-~e~). <~ Next we apply Theorems 3.26 and 3.32 to prove the differentiability of the value function along an optimal trajectory under a strict convexity assumption on the Hamiltonian. By an optimal trajectory for (x,T) we mean y*(s) := y~(s, ~*), s E [0, T], where c~* is optimal for (x, T), that is v(x, T) = g(yz(T, a*)). COROLLARY 3.35. Under the assumptions of either Theorem 3.26 or Theorem 3.32 let y* be an optimal trajectory for (x, T). Then v is differentiable at (y*(T - t), t) and the classical gradient Dv is continuous at this point, for all t E ]0, T[ such that H(y*(T - t), 9 ) is strictly convex. PROOF. For all t E ]0, T[ Y := ( y * ( T - t ) , t )

E O.

Then D + v ( Y ) has at most one element by Corollary 3.21. However D+v(Y) ~ 0 because D + v ( Y ) = Or(Y) (if we are under the assumptions of Theorem 3.26 this follows from Proposition II.4.7). Therefore D+v(Y) is a singleton, and this implies the differentiability of v at Y by either Proposition II.4.7 or II.4.4. Finally,

D r ( Y ) = lim Dv(Xn) rt

for any sequence Xn ---* Y of points of differentiability of v because

D*v(Y) = D + v ( Y ) = { D r ( Y ) } and Dv is locally bounded, so we get the continuity of Dv at Y.

9

170

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

The last result of this subsection is a straightforward consequence of Theoreins 3.26 and 3.32. COROLLARY 3.36. Under the assumptions of either Theorem 3.26 or Theorem 3.32 the value function is an intrinsically classical solution of the HJB equation (3.42), that is, v satisfies (3.44) in ]RN x ]0, +oc[. In particular, the formulation (3.44) of the HJB equation is equivalent to the viscosity and the Dini formulations. PROOF. Semiconcave functions have D+v = Ov everywhere by Proposition II.4.7, and this implies the existence of all directional derivatives by Proposition II.4.3. 3.4. Necessary and sufficient conditions of optimality

and the Pontryagin Maximum Principle In this subsection we apply the theory of w to prove various necessary a n d / o r sufi3cient conditions for optimality of open loop controls in terms of Dini derivatives or sub- and superdifferentials of the value function. Under some additional differentiability assumptions on the d a t a we then give an easy proof of the Pontryagin Maximum Principle where the classical necessary condition of optimality is slightly strengthened to become sufficient as well. The new formulation enlightens the connection between the adjoint variables and the gradient of the value function. Throughout the subsection we keep the main assumption of w that is (3.41). W'e fix x 0 E I[~.N, T > 0, and ct E A and study the optimality of the corresponding trajectory

y(t) := Yxo(t,a),

0
A very easy consequence of the DPP, Proposition = 0, A = 0, is that the fimction

3.2,

and of the assumptions

h(t) := v(y(t), T - t) is nondecreasing for any a and is constant if and only if c~ is optimal for (x0, T), see Remark 3.3. Therefore h ~ = 0 is a necessary condition of optimality and hP(t) < 0 for a.e. t is a sufficient condition of optimality if h is absolutely continuous (e.g., lipschitzean). If v is smooth we have the necessary and sufficient condition of optimality (3.61)

D,v(y(t), T - t). f(y(t), c~(t)) - vt(y(t), r - t) < 0

for a.e. t,

and since the HJB equation gives

vt(x, s) - Dxv(x, s). f ( x , a) < 0

for all x, s,

an equivalent condition is the Minimum Principle (briefly MP) (3.62)

Dzv(y(t), T - t). f(y(t), c~(t)) = m ~ D:~v(y(t), T - t). f(y(t), a),

3. THE FINITE HORIZON PROBLEM

171

for a.e.t. In the following we study conditions that generalize (3.61) and (3.62) to the case of nonsmooth value functions by replacing the classical derivatives either with (generalized) Dini derivatives or by elements of the sub- and superdifferentials. We denote D•

t) := D + u ( x , t) u D - u ( x , t),

and the elements of this set are written (P, P0) with p E ~ N P0 E ~. THEOREM 3.37 (NECESSARY CONDITIONS OF OPTIMALITY). A s s u m e (3.41). If the control function ~ is optimal for (xo, T), then the corresponding trajectory y ( . ) satisfies for a.e. t, 0 < t < T, (3.63)

O - v ( y ( t ) , T - t; y'(t), - 1 ) < 0,

(3.64)

- O + v ( y ( t ) , T - t; -y'(t)~ 1) ___0~

(3.65)

Po - P" y'(t) = 0

V (p, Po) E D •

(3.66)

- p . y'(t) = max{ - p . f ( y ( t ) , a) }

(3.67)

a(t) E arg m i n p . f ( y ( t ) , 9 )

LEA

T - t),

~/(P, P0) E D • V (p, 19o) E D •

T - t), T - t) .

A

PROOF. Since h(t) = v(G(t)), with G(t) := (y(t), T - t), is constant, by Lemma 2.50 we have, for each t where y( 9 ) is differentiable, 0 = h'(t) = c9(v o G)(t; 1) _> O - v ( y ( t ) , T - t; y ' ( t ) , - 1 ) which is (3.63). Similarly, we get (3.64) by writing h'(t) = - O ( v o G)(t; - 1 ) . From Lemma 2.37 we have O-v(y(t), T - t; y'(t), - 1 ) > p . y'(t) - Po for all (P, P0) E D - v ( y ( t ) , T (3.68)

t), so that (3.63) implies po - p . ~'(t) > 0

for such vectors. Similarly, (3.64) implies (3.68) for all (P, po) E D + v ( y ( t ) , T - t). Since v is a viscosity subsolution of the HJB equation (3.42), at each point where y( - ) is differentiable we have (3.69)

Po - P" y'(t) < Po + sup{ - p . f ( y ( t ) , a) } <<_0 LEA

for all (P, Po) E D+v(y(t), T - t), so (3.65), (3.66) and (3.67) hold for such vectors. Moreover, v is a bilateral supersolution of the HJB equation by Corollary 3.21, thus (3.69) holds for all (p, po) E D - v ( y ( t ) , T t) as well and so (3.65), (3.66), (3.67) are verified. 4 Next we state some sufficient conditions of optimality in terms of verification functions which are viscosity subsolutions u of the HJB equation (3.42) such that u(x, O) <_ g(x) and with the following regularity: either u E BC(]~ y • [0, T]) if we assume (3.41) (i), or u E C ( R N • [0, T]) if we assume (3.41) (ii). By the comparison theorems in w and w the value function v is the maximal verification function.

172

III. PROBLEMSWITH UNRESTRICTEDSTATESPACE

THEOREM 3.38 (SUFFICIENT CONDITION OF OPTIMALITY). Assume (3.41) and set Y(" ) = Yxo(', a). Suppose there exists a verification function u which is locally lipschitzean in a neighborhood of { y(t) : 0 < t < T } and such that u(y(T), O) = g(y(T)). Then a is optimal for (xo,T) if for a.e. t E [0, T] at least one of the following conditions holds: (3.70)

O-u(y(t), r - t; y'(t), - 1 ) _< 0,

(3.71)

-O+u(y(t), T - t; - y ' ( t ) , 1) <_ 0,

(3.72)

~(P, Po) E D •

- t) : P o - P " y'(t) >_ O.

PROOF. We consider ~(t) := u(G(t)) := u(y(t), T - t). Our goal is to prove that is nonincreasing, because in such a ease

g(y(T)) = u(y(T), O) = ~(T) <_ ~(0) = u(xo, T) <_ v(xo, T), where the last inequality follows from the comparison principle (Theorem 3.7 or Theorem 3.17), and then g(Yxo(T, a)) = v(xo, T). Since u and y are locally lipschitzean, we can use Lemma 2.50 to get

~'(t) = O-(u o G)(t; 1) = r

T - t; y'(t), - 1 ) ,

and

~'(t) = - 0 + ( u o G)(t; - 1 ) = -O+u(y(t), T - t; - y ' ( t ) , 1), for all t where ~ and y are both differentiable. Then ~t(t) ~ 0 at such points if either (3.70) or (3.71) holds. On the other hand, if (3.72) holds with (p, po) E D - u ( y ( t ) , T - t), then we use Lamina 2.37 to get

-O+u(y;T-

t ; - y ' , 1) _< - O - u ( y , T -

t ; - y ' , 1) < -p0 + p " y' _< 0

and thus (3.71) holds. Similarly, (3.72) with (P, P0) E D + u ( y ( t ) , T - t ) (3.70), and this completes the proof.

implies 9

THEOREM 3.39 (NECESSARY AND SUFFICIENT CONDITIONS OF OPTIMALITY). Assume (3.41) and set y(. ) = Yxo( ", a). Suppose the value function v is locally lipschitzean in a neighborhood of { y(t) : 0 < t < T }. Then (i) a is optimal for (x0, T) if and only if for a.e. t E [0, T] one of the following conditions hold: (3.63), (3.64), or (3.73)

O-v(y(t),T-t;y'(t),-1)=

min O - v ( y ( t ) , T - t ; f , - 1 ) , fEF(y(t))

where F(x) := ~-6f(x, A); (ii) if, moreover, D • - t) ~ 0 for a.e. t, then the optimality of a is equivalent to (3.65) for a.e. t and it is also equivalent to (3.72) for a.e. t; (iii) if, in addition, D+v(y(t), T - t) = Ov(y(t), T - t) for a.e. t, then the optimality of c~ is equivalent to (3.66) and (3.67) for a.e. t.

3. T H E F I N I T E HORIZON PROBLEM

173

Note t h a t the additional assumption in (iii) that the superdifferential D + v coincides with Clarke's gradient Ov is stronger t h a t the assumption of (ii), D • 7~ O, and it is satisfied if either v is locally semiconcave (and sufficient conditions for this are given in Theorem 3.26), or the d a t a satisfy the differentiability assumptions of Theorem 3.32. Note also that the conditions (3.66), (3.67), and (3.73) have the form of a Minimum (or Maximum) Principle. They say that for a.e. t an optimal t r a j e c t o r y takes a direction of steepest descent of the value function among the available directions F ( y ( t ) ) , in some generalized sense. PROOF. Most of the statements follow immediately from Theorems 3.37 and 3.38, for example, those concerning (3.63), (3.64), and (ii). It is easy to deduce from Theorem 3.25 (ii) that (3.73) is equivalent to (3.63). The statement about necessity in (iii) is in Theorem 3.37. If there exist (p, p0) E D - v ( y ( t ) , T - t) such t h a t (3.74)

- p . y'(t) = max{ - p . f ( y ( t ) , a) }, aEA

then (3.72) holds with u = v because v is a viscosity supersolution of the HJB equation (3.42), and so a is optimal by Theorem 3.38. To complete the proof of the sufficiency of (3.66) for optimality we show the sufficiency of (3.74) for some (P, Po) E D* v(y(t), T - t), which is weaker than (3.66) because D+v = Ov = ~-dD* v (see w We recall that

p = l i m D v ( x n , tn),

Po = l i m v t ( x n , t ~ )

for a sequence (xn, tn) --* (y(t), T - t) of points where v is differentiable, so that the HJB equation (3.42) holds in the classical sense at the points (x,~,tn). By taking the limit as n --~ ~c in the equation we get P0 + sup{ - p . f ( y ( t ) , a) } = O, aEA

so (3.74) implies (3.72) and we conclude by Theorem 3.38. REMARK 3.40. As in w for the infinite horizon problem, one can use the necessary and sufficient conditions of optimality for open loop controls to construct optimal multivalued feedbacks for the finite horizon problem as well. It is not a hard exercise for the reader to a d a p t Definition 2.60 of weak and full optimality and then prove results analogous to Theorem 2.61 and Corollaries 2.63 and 2.64. <~

Next we prove the Pontryagin Maximum Principle (PMP in the sequel) in a necessary and sufficient form, under the (usual) additional assumptions that g E C~(]R N) and f is differentiable with respect to x with D x f continuous. Let us first recall its classical derivation from the HJB equation under the additional assumption that the value function v is C 2. The HJB equation (3.42) gives (3.75)

vt (x, s) - D x v ( x , s) . f ( x , a) <_ 0

for all x, s,

174

III. P R O B L E M S W I T H U N R E S T R I C T E D

STATE SPACE

a n d we saw at the b e g i n n i n g of the section t h a t the left-hand side of (3.75) is >_ 0 at x = y(t), s = T - t, a = a ( t ) if y ( . ) = Y z o ( ' , a ) is o p t i m a l for (x0, T), see (3.61). Thus y(t) maximizes x ~ vt(x, T - t) - D z v ( x , T - t ) . f ( x , a ( t ) ) a n d we can differentiate with respect to x to get (3.76)

Dxvt - fD~xv - Dxv Dxf = 0

for x = y(t), s = T - t, a = a(t), where D2zzv is the Hessian m a t r i x of v a n d ( D z f ) o = O f i / O x j (so (3.76) is an equality between row vectors). We define the row vector (3.77)

p(t) := D ~ v ( y ( t ) , T - t)

a n d differentiate with respect to t to get, by (3.76), p' = f D ~ z v - D z v t = - p D x f

.

Thus the eostate or adjoint vector p(t) associated with (x0, T) a n d a c .A can be redefined, without m e n t i o n i n g v, as the u n i q u e solution of the linear system adjoint to the linearization of the controlled system a r o u n d the trajectory y( 9 ), t h a t is, fp'(t) = -p(t) D~f(y(t),a(t)),

(3.78)

/

(p(T) =

0 < t < T,

Oa(y(T)),

a n d necessary conditions of optimality are (3.79)

- p ( t ) . f ( y ( t ) , a(t)) = ma~{ - p ( t ) . f ( y ( t ) , a) } = H ( y ( t ) , p(t)),

(3.80)

v t ( y ( t ) , T - t) = p(t) . f ( y ( t ) , a ( t ) ) = - H ( y ( t ) , p ( t ) )

.

The necessity of (3.79) for the optimality of a, w i t h o u t the extra regularity ass u m p t i o n on v, is the s t a t e m e n t of the classical PMP, a n d it is particularly useful because one can try to solve (3.78) a n d (3.79) without knowing a priori the value function v. However, these two conditions alone are not sufficient for optimality, as the next example shows. EXAMPLE 3.41. For N = 2 consider the system I

/

(Yl,Y2) = (a, Yl2), (yl,Y2)(0)

~--- (Xl,X2)

reX,

A=

[-1,1],

a n d the problem of minimizing - y 2 ( t , a), t h a t is, g(y) = - Y 2 . It is easy to guess that an optimal control is a + -- 1 if Xl _> 0 and a _ =- - 1 if xl < 0, whose corresponding cost is - x 2 - x21t - 89 3 - Ixll t 2. It is easy to see t h a t this is indeed the value function v(x, t) because it satisfies the HJB equation V t -[- ]Vx, I -- X21Vx2 ~- 0

in R 2 • ]0, 2 [

and the initial condition v(x,O) = g(x). The control a0 - 0 is not optimal for (0, 0, t) because the t r a j e c t o r y remains in (0, 0) a n d the cost is 0 > - t 3 / 3 = v(0, t). The costate is ( p l , p 2 ) ( t ) = ( 0 , - 1 ) because the m a t r i x D x f is null along the null trajectory, and the m a x i m a l i t y condition (3.79) is satisfied because b o t h sides are null. <~

3. THE FINITE HORIZON PROBLEM

175

If we do not assume the extra regularity of the value function v, which is false in general, a natural nonsmooth counterpart of the formulas (3.77) and (3.80) connecting the costate with the derivatives of v is (3.81)

( p ( t ) , - H ( y ( t ) , p ( t ) ) ) E D+v(y(t), T - t) .

Note t h a t in the right-hand side of this formula it is equivalent to write D + or D + because D - v C D+v everywhere by Theorem 3.32. The condition that the eostate satisfy for a.e. t both (3.79) and (3.81) is sufficient for the optimality of a by Theorem 3.38, because (3.72) is satisfied and v is locally lipschitzean by Theorem 3.32. The next result states t h a t this condition is also necessary. THEOREM 3.42 (PONTRYAGIN MAXIMUM PRINCIPLE). Assume (3.41) (ii), A compact, g E C I ( R N ) , f differentiabIe with respect to x and D ~ f continuous. Set Y(" ) := Y~0( ' , a) and let the costate p( . ) be the corresponding solution of (3.78). Then a is optimal for (xo,T) if and only if for a.e. t E ]0, T[

- p ( t ) . f(y(t), a(t)) = max{ - p ( t ) . f(y(t), a) } = H(y(t), p(t)),

and

(p(t), - H ( y ( t ) , p(t))) E D+v(y(t), T - t) . We present three proofs of this important theorem; they make a decreasing use of the previous theory while their length is increasing. The first is an easy byproduct of the representation of v as a minimum over relaxed controls derived in w and of the necessary conditions of optimality of the present subsection. In particular, it gives the P M P for relaxed controls, see Remark 3.48. The second does not use relaxed controls but just the theory of necessary conditions. The third is self-contained, b u t still not too long, and it has an interesting extension, see Theorem 3.44. For all of them we need some notations and a lemma. For a given a E .A we denote by S the corresponding solution operator for the system (S), that is,

s(~-;

r,

x) := r

where ~ solves C' = f((, 0~),

((r) = x.

We also denote by ~(t) E A the control (~ E A shifted by a time t:

~(')(s) := ~(t + s). LEMMA 3.43. Under the assumptions and notations of Theorem 3.42, the cost functional

J(x, T,~) := g(yx(~-, ~)) corresponding to any/3 E .4 is differentiable with respect to x, and for all t DxJ(y(t), T - t, a(t)) = p(t) .

176

III. PROBLEMS W I T H U N R E S T R I C T E D STATE SPACE

PROOF. The first statement follows from the chain rule and the theorem on differentiability of solutions of an ODE with respect to the initial data, Theorem 5.8. Set A ( s ) := D x f ( y ( s ) , ~(s)) and denote by M ( s , t) the fundamental matrix associated with the linear system (3.82)

[ ' = A(s)G

t h a t is, the unique solution of

{

s 9 R,

~(s,t)=A(s)M(s,t),

M ( t , t) = I,

where I is the identity matrix. By Theorem 5.8 the Jacobian matrix of the m a p x ~ S ( t 4- r; t, x) at y(t) is M ( t 4- r, t). Since

J (x, ~-, ~(~)) = g( s ( t + -~; t, x) ), we obtain

D x J ( y ( t ) , T - t, a(t)) = D g ( y ( T ) ) M ( T , t) . On the other hand, by standard results on linear ODE, the solution of the system adjoint to (3.82)

.' = -rlA(t) with terminal d a t u m r/(T) = ~ is given by ~ M ( T , t), as it is easy to check by using the following well-known property of the fundamental matrix:

OM - ~ (s, t) = - M ( s , t ) A ( t ) . Therefore, the definition of costate (3.78) gives

p(t) = D g ( y ( T ) ) M ( T , t )

.

4

PROOF 1 OF THEOREM 3.42. The sufficiency follows from Theorem 3.38, as we remarked earlier. To prove the necessity we assume a is optimal and observe t h a t it is enough to show for a.e. t the existence of p0(t) C ~I such that (3.83)

(p(t), po(t) ) C D+v(y(t), T - t),

because then the necessary conditions of optimality (3.65) and (3.66) in Theorem 3.37 imply for a.e. t

po(t) = p(t) . f ( y ( t ) , (~(t)) = - H ( y ( t ) , p ( t ) ) which gives (3.79) and (3.81). In the present proof we use the results of w on the representation of v as a marginal function with respect to rescaled relaxed controls

3. T H E F I N I T E HORIZON PROBLEM

177

# E .A~ of the smooth (with respect to (x, t)) functions G ( x , t, #), see (3.55) and, in particular, Theorem 3.32 and R e m a r k 3.33. Since c~ is optimal for (y(t), T - t), at such point the minimum of G with respect to # is attained at the rescaled relaxed control associated with c~( 9 + t), t h a t is, ~T-t(T)

:-~ ~ c ~ ( ( T - t ) r + t )

where 5a indicates the Dirac measure concentrated at a. Then the gradient of G with respect to (x, t) at (y(t), T - t, rlT_t ) belongs to D + v at the same point, see Remark 3.33. Thus it is enough to show that D ~ G ( y ( t ) , T - t, ~ T - t )

-~

p(t),

and this follows immediately from Lemma 3.43 because G ( x , t, tit) = Jr(x, t, p)

for all # E A r,

where lit(T) = p(tT), and for a relaxed control associated with a standard control /3 E ,4 the relaxed trajectories coincide with the standard ones, so j r = j . The next proof of the necessary condition of optimality does not use the results of w involving relaxed controls, but it is still based on Theorem 3.37 on necessary conditions in the general case. PROOF 2 OF THEOREM 3.42. We recall from the beginning of Proof 1 that we have to show the existence of p0(t) satisfying (3.83). Since v ( x , s ) <_ J ( x , s , a

(t)) =: ~ ( x , s )

for all x, s, and v ( y ( t ) , T - t) = J ( y ( t ) , T - t, a (t)) = ~ ( y ( t ) , T - t), it is enough to prove t h a t there exists P0 such that ~(x, s) - ~ ( y ( t ) , T - t) < p(t). (x-

y(t)) +po(t)(s-

T + t) + o ( ] x -

y(t)l + I s - T + tl)

as x --* y(t), s --* T - t. By Lemma 3.43, ~(x, T - t) - p ( y ( t ) , T - t) = p(t) . (x - y(t) ) + o(Ix - y(t)l ) as x --. y(t). On the other hand, since ~(x, s) = g ( S ( t + s; t, x ) ) we can use the differentiability of g to get ~(x,s)-~(x,T-t)=

Dg(S(T;t,x)).

f

t+s

f(S(T;t,x),a(r))d~

+o(Is--

T + tl)

JT

<_ p o ( t ) ( s - T + t) + o([s - T + tl)

as s --~ T -

t,

where po(t) :=

lim sup (x,s)~(y(t),T-t)

D g ( S ( T ; t, x ) ) .

1

f t + s f ( S ( T ; t. x), a ( T ) ) dr

t + S -- T J T

'

which is finite because f is locally bounded. This completes the proof.

178

III. P R O B L E M S W I T H U N R E S T R I C T E D

STATE SPACE

The last proof of the necessary condition in Theorem 3.42 is the most direct: it does not use the general Theorem 3.37 on necessary conditions, but only Lemma 3.43. PROOF 3 OF THEOREM 3.42. For all x, s and

v ( x , s ) < J ( x , s , c ~ (T-s)) = g ( S ( T ; T - s , x ) ) , S(T;T

- s,x) = S(T;t,x

+ h(x,s)),

where h(x, s) :-- f7 t

(3.84)

T-s

f ( S ( r ; T - s, x), c~(r)) dT,

and so we get v ( x , s ) <_ J ( x + h ( x , s ) , T

- t,a(t)) .

Since . ( y ( t ) , T - t) = J ( y ( t ) , T - t, ~(~)),

Lemma 3.43 gives (3.85) v ( x , s) - v ( y ( t ) , T - t) <_ p(t) . (x + h(x, s) - y(t)) + o(Is - T + t I + Ix - y(t)l) as s --~ T - t, x --* y(t). Now observe t h a t y(T) = S ( r ; t , y ( t ) )

gives

s ( r ; r - s, z ) : S(r; t, x) + o(1) : y ( r ) + o(1)

as s ~ T -

t, x ~ y(t), and so h ( x , s) =

2

--8

f ( y ( r ) , c~(r)) d~- + o(1).

Moreover, by Lebesgue derivation theorem, for a.e. t, s - r + t

--s

f ( y ( r ) , a ( r ) ) d r = f ( y ( t ) , a ( t ) ) + o(1)

as s ~ T - t. Therefore, for such t, we get from (3.85) v(x, s) - v ( y ( t ) , r - t) <_ p(t) . (x - y(t) ) + p(t) . f ( y ( t ) , a ( t ) )(s - T + t) + o(Is - r + tl +

lx

- y(t)r)

as s --~ T - t, x --* y(t), which proves t h a t (p(t), p ( t ) . f ( y ( t ) , c~(t))) e D + v ( y ( t ) , T - t ) .

On the other hand, the definition of viscosity subsolution of the HJB equation (3.42) gives p ( t ) . f ( y ( t ) , a ( t ) ) + sup{ - p ( t ) .

aEA

so we get (3.79) and (3.81).

I ( y ( t ) , a) } < 0

9

3. T H E F I N I T E HORIZON PROBLEM

179

Now we give some extensions of the PMP, Theorem 3.42, and several remarks and comments. In the next result we allow the terminM cost g to be nondifferentiable and we replace the value function v with verification functions. It applies, in particular, if g is semiconcave. THEOREM 3.44 (EXTENDED P M P ) . A s s u m e (3.41) (ii) and x ~ f ( x , a ) continuously differentiable for all a E A. Given y( . ) = Yzo( " , (~), assume D+ g ( y ( T ) ) r 0 and define p( 9 ) as the unique solution of

(3.86)

~p'(t) = - p ( t ) D x f ( y ( t ) , a ( t ) ) ,

0 < t < T,

"[p(T) = p,

with ~ C D + g ( y ( T ) ) . I f a is optimal f o r (xo; T), then f o r almost every t E ]0, T[ the m a x i m u m condition (3.79) holds and

(3.87)

(p(t), - H ( y ( t ) , p ( t ) ) ) C D + u ( y ( t ) , T - t)

for any verification f u n c t i o n u such that u(y(t), T - t) = g ( y ( T ) )

for all t .

Vice versa, a is optimal for (x0, T ) if there exists a verification function u that is locally lipschitzean in a neighborhood of { y(t) : 0 < t < T } and such that (3.79) and (3.87) hold for a.e. t C ]0, T[.

PROOF. The sufficient condition of optimality is an immediate corollary of Theorem 3.38. To prove the necessary condition of optimality, first we replace Lemma 3.43 with the claim p(t) e D + J ( y ( t ) , T - t, a(t)),

where D + J ( y , 7,/3) denotes the superdifferential of x ~ J(x, T,/3) at y. In fact, under the current assumptions, the arguments in the proof of Lemma 3.43 give (3.88) (3.89)

p(t) = p M ( T , t), S ( T ; t, x) - y ( T ) = M ( T , t ) ( x - y ( T ) ) + o(x - y ( T ) )

as x --* y ( T ) , and J ( x , T - t , a (t)) = g ( S ( T ; t , x ) ) . following property of ~ E D + g ( y ( T ) ) :

By plugging (3.89) into the

g ( S ) - g ( y ( T ) ) < ~ . (S - y ( T ) ) + o(S - y ( T ) )

as S ~ y(T),

and using (3.88) we get the claim. Next we note that u(x, s) < v(x, s)

for all x, s,

by the comparison principle (Theorem 3.7 or 3.15), and u(y(t), T - t) = v(y(t), T - t)

by assumption. Thus we can repeat Proof 3 of Theorem 3.42 with v replaced by u and get the conclusion.

180

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

The next result improves a bit the necessary condition of optimality connecting the costate and the semidifferentials of the value function. COROLLARY 3.45. Under the assumptions of Theorem 3.42, if a is optimal for (x0, T), then for all t E [0, T]

D - v ( y ( t ) , T - t) __ { ( p ( t ) , - H ( y ( t ) , p(t))) } C_ D+v(y(t), T - t ) . Moreover, if either one of these inclusions is an equality, then v is differentiable at (y(t), T - t), and in this case both the inclusions are equalities. PROOF. The first inclusion follows from the second by the properties of the suband superdifferentials of good marginal functions (by Theorem 3.32 and Proposition II.4.4 either D - v is empty or v is differentiable at ( y ( t ) , T - t), and then D - v = D + v at such point). By Theorem 3.42 the second inclusion holds for a.e. t E ]0, T[. Then it is very easy to prove it for all t because D + v ( z ) = Ov(z) for all z = (x, t), by Theorem 3.32, the multivalued map z ~ Ov(z) is known to be upper semicontinuous (i.e., z,~ --* z, P~ ---' P, Pn E Ov(z~) imply p E Ov(z)) and p ( . ), y ( . ), and H are continuous. If the second inclusion is an equality, then v is differentiable at (y(t), T - t) by Theorem 3.32 and Proposition II.4.4. All the other statements follow from the basic properties of sub- and superdifferentials. *l COROLLARY 3.46. Assume the hypotheses of Theorem 3.44. If a is optimal for ( x o , T ) , then t ~-~ H ( y ( t ) , p ( t ) ) is constant. PROOF. By the assumption (3.41) (ii) the Hamiltonian H is locally lipschitzean, so t ~-* H ( y ( t ) , p(t)) is absolutely continuous. We are going to show that its derivative is null at almost every t where it is differentiable, therefore almost everywhere. We fix such a t. By Theorem 3.44 we can choose s arbitrarily close to t such that

H(y(s),p(s)) = -p(s). f(y(s), a(s)). By the definition of H

H(y(s),p(s)) - H(y(t),p(t)) _< [p(t) - p(s)] 9f ( y ( t ) , a(t) ) + p(s) . [f(y(t), a(t) ) - f ( y ( s ) , c~(s))]

= (t - s)~p'(t), f ( y ( t ) , c~(t)) + p ( t ) D x f ( y ( t ) , a(t))y'(t)] + o(It - sl) = o(It

-

sl)

as ~ -~ t,

where in the last equality we have used the definition of costate (3.86). If we exchange the roles of t and s we obtain a similar inequality which completes the proof. 4 REMARK 3.47. A more concise but equivalent formulation of the P M P is as follows: a is optimal for (x0, T) if and only if for a.e. t E ]0, T[

(p(t),p(t). f(y(t),a(t))) E D+v(y(t),T-

t).

<~

3. T H E FINITE H O R I Z O N P R O B L E M

181

REMARK 3.48. Proof i of Theorem 3.42 also gives a P M P for relaxed controls (we recall that the value function associated with relaxed controls coincides with v, so p E .A r is optimal for (x, t) if Jr(x, t, p) =- Y(X, t)). The formulation is exactly the same after replacing f with f f in (3.79) and in the definition of costate, which now is (3.60). <~ REMARK 3.49. If the trajectory y ( . ) is optimal and H is differentiable at (y(t),p(t)), then the differential equations satisfied by the costate p(. ) and by y( 9 ) take the form of the Hamiltonian system p'(t) = DxH(y(t),p(t)),

y'(t)

-DpH(y(t),p(t)) .

We recall that for H smooth this is the characteristic system of ODEs of the HJB equation (3.42) and the functions p(. ), y(. ) are the characteristics of this PDE. Exercises

3.1. Prove that under the assumptions (A1)-(A3), (A~), and (As) lim v ( x , t ) = inf t~+vc

g(yx(s),a(s))e-~Sds.

c~E.A

3.2. In addition to the assumptions of Proposition 3.1, suppose f is bounded. Prove that (i)

v E BUC(R N • [0, T]) for all T > 0,

(ii) if )~ > 0, then v E BUC(]R N x [0, oc[). 3.3. Consider N = 1, f ( x , a ) = a, A = [-1,1], g(x) = e -x2, g(x,a) =_ 0. Compute v(x, t) and show that cOv/Ox has a jump along the half-line x = 0, t > 0.

3.4. Consider N = 1, f ( x , a) = x a, A = [0, 1], g(x) = - x , f(x, a) =- 0. Compute v(x, t) and D+v(x, t), D - v ( x , t). 3.5. Prove the following extension of Theorem 3.12. Let ul, u2 E C(C) be, respectively, a subsolution of

ut + H(x, t, Du) - hi(x, t) = 0

in C

and a supersolution of

ut+ H(x,t, Du)-h2(x,t)=O

inC,

where hi, h2 E C(C), H E C(C x ]~N) satisfies

]H(x,t,p) - H(x,t,q)l < C i p - ql, IH(x,t,p) - H(y,t,p) I < w ( i x - Yl) + ~ ( I x - Yl IPl),

182

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

for all (x, t), (y, t) E C, p, q E IRx . T h e n sup

( U l ( ' ,t) - u2(" ,t))

B(O,C(T-t))

<

sup

(ul(-,0)-u2(.,0))+

/o t

B(O,CT)

sup

(hl(.,s)-h2(.,s))ds.

B(O.C(t-s))

3.6 (COMPARISON THEOREM FOR H J B EQUATIONS WITH T I M E - D E P E N D E N T H ) . Prove the following extension of T h e o r e m 3.15. Let ul, u2 E C(R N x [0, T]) be, respectively, a sub- and a supersolution of

ut + H(x, t, Du) = 0

in I~ N • ]0, T[,

where H E C(R N x [0, T] x R x ) satisfies

IH(x,t,p) - H(x,t,q)l <_ K ( l x I + 1)[p - q[ for all x, t, p, and

]H(x, t,p) - H(y, t,p) I <_a~(lx - Yl, R) + a~(lx - Yl Ipl, R), for all p E R N, t E [0, T], x, y ~ sup R N x [0,T]

B(0, R), R

> 0. T h e n

(it 1 -- 712) ----"s u p ( t t l ( " IRy

, 0 ) -- U2(" ) ) .

3.7 (NoNAUTONOMOUS PROBLEMS). A s s u m e A is a compact set, g E c(NN), 0 < T < -[-oo, f : ]~N X [0, T] x A --~ IRN and s : ]I~N X [0, T] x A ~ R are continuous, If(x,t,a)l <_ K ( 1 + Ixl) for all x, t, a, and for any R > 0 there is LR such t h a t

[f(x,t,a) - f ( y , t , a ) l <_ LRI x - y[ for all t E [0, T], a E A and x, y E B(0, R). Define

H(x, t,p) := max{ - f ( x , t, a) .p - f(x, t, a) },

and

aEA

f(y(s), s, a(s)) ds + g(y(T)) ~j,

:= aEA

where y( 9 ) is the unique solution of y'(s) = f ( y ( s ) , s , a ( s ) ) , y(t)

t < s < T,

= x

(i) Prove the D P P

v(x, t) = inf aEA

{H

f(y(s), s, a(s)) ds + v(y('r), T)

for all x E 1RN, 0 _< t < ~- _< T (cfr. R e m a r k 3.10).

}

,

4. HJB INEQUALITIES (ii) Prove that v is the complete solution problem

in C(]~ N

-vt+H(x,t, Dv)=O v(.,T) = g(-)

183 •

[0, T]) of the terminal value

inR N•

in ~N.

[Hint: use Exercise 3.6.] 3.8. Assume u : f~ ~ R is semiconcave-convex. Prove that

(i)

u e c1(~);

(ii)

u has bounded pure second derivatives in the sense of distributions;

(iii) all the second derivatives of u are in L~162 (iv) Du is Lipschitz continuous. 3.9. Show that if x H f ( x , a ) is affine for all a E A, then x ~ z x ( 1 ; a , t ) is affine (see Lemma 3.27 for the notations). Therefore. if in addition g is concave (respectively, convex), x ~ v(x, t) is concave (respectively, convex). 3.10. Fill in the details in the proof of Theorem 3.32, that is, prove t h a t the solution of (3.58) at a fixed time depends continuously on (x, t, g). 3.11. Prove that the conclusion of Theorem 3.39 (iii) holds if the assumption D+v = Ov is replaced by D+v(z) ~ ~ for a.e. t, where z : = ( y ( t ) , T - t ) , and O-v(z;q) = mincED+v(~)~.q for all q E F(y(t)) • { - 1 }. 3.12. Under the assumptions and notations of Theorem 3.44 prove that, if c~ is optimal for (x0, T), then (i) p(t) E D+u(y(t), T - t) for all t E [0, T]; (ii) if in addition u is semiconcave, then (p(t),H(y(t),p(t))) E D + u ( y ( t ) , T (cfr. Corollary 3.45).

t)

4. P r o b l e m s w h o s e H J B e q u a t i o n is a v a r i a t i o n a l or quasivariational inequality In the previous sections we considered optimal control problems where the admissible controls were all the measurable functions of time with values in a given set A. Here we give some examples where either some nontriviM restriction on the class of admissible controls is imposed (the monotone control problem in w or extra control variables appear in the model in the form of a sequence of stopping times and switchings or impulses (see w to w with corresponding costs. All these problems, which can be seen as variants of the infinite horizon problem of w share a common feature, namely, that the HJB equation satisfied by their value function takes the form of a variational or quasivariational inequality. We show that the viscosity solution approach is flexible enough to deal with these more complicated situations. Throughout the section we assume t h a t the dynamics and the running cost satisfy the standard assumptions of w and w namely:

184 (A~)

III. PROBLEMSWITH UNRESTRICTEDSTATESPACE A is compact, f : ~N • A --* ]~N is continuous,

which implies (A0) and (A1); (A3)

(f(x,a)- f(y,a)).(x-y)

< L l x - y l 2, V x , y E ~ N, V a 9

: ~N • A ~ ~N is continuous and (A4)

]g(x,a)- e(y,a)I < ~ve(Ix -Yl), [g(x,a)I -~ J}l for all x , y 9 ~N, a E A; A > 0,

where w~ is a modulus. The control system is

Sy'(t) = f(y(t), a(t)),

(4.1)

t_>O,

Ly(0) = x 9 R N,

a 9 A, the set of measurable functions with values in A.

4.1. The m o n o t o n e control problem In this problem the control set A is the closed interval [0, 1] and the dynamical system (4.1) is controlled by choosing (~ in the class (4.2)

Am := { a : [0, +c~[ --, [0, 1]: c~ nondecreasing }.

With the evolution defined by (4.1) and (4.2), we associate the infinite horizon payoff functional

(4.3)

J(x, ~) :=

fo

e(yx(t, ~), ~ ( t ) )

e - ~ t at

and the value function (4.4)

Vm(X) := inf J ( x , a ) . ~EAm

Let us observe that the class of admissible controls Am does not satisfy property (ii) in Remark 2.7. This causes difficulty in implementing the dynamic programming procedure for Vm. A way to overcome such a difficulty is to introduce, for each fixed a 9 [0: 1], the subset

(4.5)

A g := { ~ 9 Am: . ( o ) > a),

and look at

(4.6)

v(x,a) :=

inf J(x,~),

called the monotone value function. It is clear by definition that (4.7)

v(x,O) = inf J(x,(~) -- Vm(X). o~Ao

4. HJB INEQUALITIES

185

In the following we study the basic properties of v as a function of x E ]~g and a C [0, 1], leading to its characterization as the unique viscosity solution of the variational inequality (VI)

max{ Au(x, a) - f ( x , a). Dxu(x, a) - g(x, a);

Ou

~ ~

~x ~ a ~

jI = o

in IRN x [0, 1[, satisfying the boundary condition

u(z, 1) = a(x, 1),

vx 9 R ~

For this purpose we need to strengthen the standard conditions on f and g. We assume in addition (4.8)

VX, X' 9 N, a , a ' 9 [0,1).

~ ]f(z'a)-f(x''a')]<-L(Ix-x'l+[a-a']) lie(*, a) - e(x', a')I < ~({x

- x'l + la - a'l)

PROPOSITION 4.1. Assume (4.8). Then the value function v defined by (4.6) has the following properties: (i)

for each fixed x 9 ]RN, the function a ~-~ v(x, a) is nondecreasing;

(ii)

if v(x, a) = J(x, ~) for some ~ 9 A~n, then v(x, a) = v(x, a') for all a' 9

[a, a(O+)]; (iii)

v(x, 1)

(iv)

v e BUC(R N • [0, 1]).

=

e(y~(t), 1) e -at dt = J(x, 1), for a l l . 9 R N, where yl is the

trajectory of (4.1) corresponding to the constant control c~(t) =_ 1;

PROOF. Statement (i) follows immediately from the inclusion .A~ c_ A~, Va' >_ a. In order to prove (ii), observe first that by monotonicity 8(0 + ) : =

lira ~ ( t ) _ > 8 ( 0 ) _ > a . t~0 +

If ~(0 +) = a, there is nothing to prove. Hence consider the case ~(0 +) > a. Define c~* by setting c~*(0) = g(0 +) =: a + and c~*(t) = ~(t) for t > 0. Of course, a* 9 Aa~+ and a* ~ 5 almost everywhere. As a consequence, the corresponding trajectories and payoff coincide, J(x, c~*) = J(x, ~), so that, by assmnption and the definition of v, (4.9)

v(x,a) = J ( x , 5 ) = J(x,~*) > v(x,a+) .

On the other hand, by property (i)

v(x,a) <_ .(x,a') < v(z,a +)

Va' 9 [a,a+].

This combined with (4.9), proves (ii). Equality (iii) is immediate since .A~ contains only the constant control c~ = 1.

186

III. PROBLEMS

WITH

UNRESTRICTED

STATE SPACE

In order to prove (iv), let us first fix x E ]RN and take a, a' E [0, 1] with a' > a (the case a' < a is treated in a similar way). If ~ E .Aa is such t h a t

(4.10) we

v(x,a) = J(x,~),

set

t0:=inf{t>0:G(t)

> l+a-a'}

and to = + c o if ~(t) < 1 + a - a ~ for all t > 0. Consider the control ~ defined by ~(t)=

~+(t)+a'-a

for0
L1

for t > t o .

By definition of to (and the monotonicity of 5) we have (4.11)

IF(t) - K(t)l _< ]a' - a I

Vt > 0 .

Since ~ E .A~ we have, because of (4.10) and p r o p e r t y (i), (4.12)

0 <_ v ( x , a') - v ( x , a) < J ( x , ~) - J ( x , ~ ) .

Let us now show t h a t (4.13)

r J ( x , ~ ) - y ( x , ~ ) l < w(la - a'l) ,

for some modulus ~. For this purpose observe t h a t a s s u m p t i o n (4.8), Gronwall's L e m m a 5.1, and (4.11) yield (4.14)

I~x(t) - ~ ( t ) l

< Lt eL'la - a'l

Vt > O,

where Yx(" ), Yz(" ) are the trajectories of (4.1) corresponding to the controls ~, G. At this point observe t h a t f J ( x , 6 ) - J ( x , ~ ) l < I T + Ior

VT>0,

where I r = ~o T .~e(l~At) - ~ ( t ) l

I~ =

//+

+ IF(t) - ~ ( t ) l ) + - ~ tit,

2 M e -As dt .

Using the estimates (4.14) and (4.11) we o b t a i n IT <_

/o+~e( L t eLtl a -- a' t + la -- a'l) e -At dt .

4. HJB INEQUALITIES

187

Since I~r and IT can be made arbitrarily small by choosing, respectively, T large and la - a' I small, we have proved (4.13). The uniform continuity of v with respect to a follows from (4.12), (4.13). The proof of the uniform continuity of v as a function of x is the same as that of Proposition 2.1. Observing t h a t the modulus of continuity of a ~ v(x, a) does not depend on x (as well as t h a t of x ~ v(x, a) is independent of a) the proof of (iv) is complete. Indeed, the boundedness of v is a straightforward consequence of the inequalities

- M / , ~ <_ v(x, O) < v(x, a) <_ v(x, 1) _< M/,~. The next proposition is the statement of the Dynamic Programming Principle of Optimality. PROPOSITION 4.2. For any x E N N, a E [0, 1], and t > 0 we have (i)

v(x, a) <

/o'

e(yx(s, a), a) e -~s ds + e-~tv(yz(t, a), a).

If ~ E .Aam is optimal, then (ii)

v(x, a) =

/o

e(yx(s, ~), ~(s)) e -~s ds + e-'~tv(yx(t, ~), ~(t)).

PROOF. Fix t > 0 and a E [0,1] and set z := yx(t,a). For any r > 0 there exists ct 1 E A a such that

v(z,a) > J ( z , ~ 1) - ~ .

(4.15) Define the control

g

~(s)

]a

if 0 < s < t,

a l ( s - t)

if s > t .

It is easy to check t h a t ~ E .A~. Then a simple computation (see the proof of Proposition 2.5) shows t h a t

v(x,a) <_ J(x,~) =

It

e(yx(s,a),

a) e - x s

d s + e -~t

~0+~

g(yz(S,c~l),al(s))e-~Sds.

Hence v(x,a) < ~0 t e ( y x ( s , a ) , a ) e - : ~ S d s + e - ~ t J ( z , a l ) . Taking (4.15) into account and the arbitrariness of z, (i) follows. Now let ~ E Jl,~ be optimal, t h a t is

~(~, a) : J(~, ~).

188

III. PROBLEMS W I T H U N R E S T R I C T E D STATE SPACE

Observe that J(x, ~) =

/0'

f(yx(S,5),5(s))e-;~ds+e-~tJ(yx(t,~),~),

where ~(s) := 5(s + t). Since, as it is easy to check, K E .45(t) is optimal for the initial position y~(t, 5), (ii) is proved. Let us deduce now from the Dynamic P r o g r a m m i n g Principle the HJB equation, that takes the form of the variational inequality (VI). Of course, we do not expect v to satisfy it in the classical pointwise sense but only in the viscosity sense. In the present case, by this we mean that v E C(IR N x [0, 1]) and that the following two conditions hold for any ~ E CI(IR ~v x [0, 1]): (4.16)

max{ ),v - f . D x p - f ; - O p / O a }

< O,

at any local maximum point (xo,ao) E N g x [0, 1[ of v - ~, and (4.17)

max{ Av - f . D x ~ - g; - O p / O a } >_ 0

at any local minimum point (Xl, al) E R N x [0, 1[ of v - p. REMARK 4.3. Observe that ifv E C I ( R N x [O, 1]) and (4.16) and (4.17) hold, then v satisfies max{ Av(x, a) - f ( x , a ) . D~v(x, a) - s

Ov a); -~aa (x, a) } = 0

at all (x, a) E ]RN x [0, 1] or, equivalently, Av(x,a) - f ( x , a ) . D ~ v ( x , a ) - e ( x , a ) <_ O,

Ov a) < O, ~a (X,

~a (X a) [Av(x, a) - f ( x , a) 9 D~v(x, a) - g(x, a)] = 0 .


THEOREM 4.4. Assume (4.8). Then v defined by (4.6) is a viscosity solution of (VI). Moreover, v satisfies the boundary condition (4.18)

v(x, 1) = d(x, 1).

PROOF. By Proposition 4.1 (iii) and (iv), v E B U C ( N N x [0, 1]) and (4.18) holds. We must then check the validity of (4.16) and (4.17). In order to prove that v satisfies (4.16) we show that the inequalities (4.19)

- 0--s <- 0

and (4.20)

Av- f. Dx~-

e_< 0

4. H J B INEQUALITIES

189

hold at any local maximum point (xo,a0) E R/v x [0,1[ of v - ~, for all ~ E CI(]R g • [0,1]). Assume by contradiction that -(O~/Oa)(xo,ao) > 0 for some ~. Then

~(xo, ao + h) < ~(xo, ao) v(x0, a0 + h) - v(x0, ao) _< p(xo, a0 + h) - p(x0, a0) for sufficiently small h > 0. This is a contradiction to the fact that v is nondecreasing with respect to a (see Proposition 4.1 (i)). Hence (4.19) is proved. Let us now prove (4.20). Since (xo, ao) is a local maximum for v - p, we have for sufficiently small t > 0,

~(xo, ao) - ~(y~

ao) _< v(xo, ao) - v(y~

ao),

where yO(. ) is the trajectory of (4.1) starting at xo associated with the constant control a ( t ) = ao. Hence, inequality (i) in Proposition 4.2 yields

~(xo, ao) - ~(y~

<_

s176

ds + (e -At - 1)v(y~

.

Divide by t > 0 and let t --* 0 + to get -D~(xo,

a0)- f(xo, ao) <_ s

ao) - Av(xo, ao)

and (4.20) is established. . O~ To check the supersolution condition (4.17), observe first that if -O-aa (Xl, a l ) _> 0, then (4.17) is automatically fulfilled. Consider then the case 0p ~aa (xl, al) <: O.

(4.21) We have to prove in this case that

(4.22)

;~v-f. Dx~-~>o

at (xl.,al).

In order to do so, observe preliminarily t h a t if ~ E .AN is optimal for xl, then (4.21) implies (4.23)

lim ~(t) ----:~(0 +) = a x . t~O +

Indeed, were ~(0 +) > al, it would follow from (4.21) t h a t

~9(Xl, al + h) > ~(Xl, a l )

(4.24)

for some h > 0 such that al + h < 5(0+). Since (xl, a l ) is a local minimum for

V(Xl,al + h)

-

V(Xl,al) ~ ~(Xl, al + h)

--

~(Xl, a l ) 9

190

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

This, together with (4.24), leads to a contradiction to Proposition 4.1 (ii) and (4.23) is established. Let us consider the constant control ~ ( t ) - al and denote by ~(- ) and ~ ( . ) the trajectories of (4.1) with initial point xl associated, respectively, with ~ and ~. By Propositions 4.2 (ii) and 4.1 and the fact t h a t ( x l , a l ) is a local m i n i m u m for v - p, for t small we have ~ ( x l , a l ) -- r

(4.25)

>_ v ( x l , a l ) -- v ( y ( t ) , a l ) >_ (e -;~t -- 1 ) v ( ~ ( t ) , a l ) + I~

with

IS =

~(~(s),~(s)) e -~s d s

Now, for some constant C i n d e p e n d e n t of t, - T ( ~ ' ( t ) , a l ) _< -c2(~(t), a l ) + C I~(t) - ~(t)l . Therefore (4.25) implies (4.26)

1 ~ ( x l , a l ) - ~ ( ~ ( t ) , a l ) + C l ~ ( t ) - ~(t)l > _e -)'t _ - 1 v(~(t), ~(t)) + 7 I S . t t -

By the uniform continuity of 6, (4.27)

/ 0 tg ( ~ ( s ) , a l ) e - ~ S d s - /0t~ ( l ~ ( s )

IS >

- ~ ( s ) l + I~(s) - a l l ) e - ~ S d s .

The Gronwall's inequality implies I ~ ' ( t ) - ~(t)l < Le Lt

[5(s)-al[ds.

T h a n k s to (4.23), this gives (4.28)

I~'(t) - ~ ( t ) l

.0

a s t --* 0 + .

t Observe now that, as a consequence of (4.23) (4.29)

~'(0) = f ( x l , al)

(4.30)

v(~(t), ~(t))

) v(xl, al)

as t --* 0 .

Let us now pass to the limit in (4.26) as t --~ 0 +. T h a n k s to (4.27), (4.28), (4.29), (4.30) we o b t a i n - - f ( x l , a l ) " D x ~ ( x l , a l ) >_ - / k V ( X l , a l ) + ~ ( x l , a l )

and (4.17) is proved.

9

4. H J B I N E Q U A L I T I E S

191

REMARK 4.5. In the proof that v is a viscosity supersolution of (VI) we implicitly assumed the existence of ~ E .4~ such that v(xl, 31) = J(Xl, ~). The existence of optimal monotone controls for each (x, a) E ]I~N • [0, 1] can be proved by means of the Helly selection theorem and the Lebesgue dominated convergence theorem (see [LM67], Corollary 2, p. 281). <~ The main result of this section is the next one, which states the comparison principle between viscosity sub- and supersolutions of (VI). THEOREM 4.6. A s s u m e (4.8). If u , w E B U C ( R N x [0, I]) are, respectively, a viscosity sub- and supersolution of (VI) such that (4.31)

u(x, 1) ~ w(x, 1)

Vx E ]~N

then (4.32)

u(x,a) < w(x,a)

V(x,a) E R y x [0, 11.

PROOF. Consider the auxiliary function (I) : ]l~2N • [0, 1] 2 ~ I~ defined by (I)(x, y, a, b) := u(x, a) - w(y, b) - Ix - yl2 + la - hI2 - ~( (x> m + 0 , (u - w ) ( ~ , ~ )

= 6 > 0.

The first part of the proof follows along the lines of Theorems 2.12 and 3.12. We therefore omit the details. A suitable choice of fl and rh namely,

29(~> + 2~(1 - ~) < 5/2 implies that for all m E ]0, 1] (4.33)

sup

~5 _> ~(~,~,~,~) > 5 / 2 .

~[2N X [0,1] 2

Also, there exists some point (5,~,g,b) E R 2N x [0, 1]2 (depending on the various parameters) such that (4.34)

,I,(~, y, ~, b) =

sup ~2N

X

~.

[0,1] 2

Moreover, the following estimates hold: le - ~1 + I~ - bl -< C v ~ (4.35)

Ix - ~12 + I~ - bl 2 < ~ ( v ~ )

2E

for some C,

192

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

(see again the proof of T h e o r e m 2.12 for details). Observe at this point t h a t (4.36)

~ < 1,

b < 1

for sufficiently small E.

Indeed, if ~ = 1 (a similar a r g u m e n t applies in case b = 1), then (4.37)

sup r162 [0,1]2

1)+w(Y, 1)-w(~,-b).

R 2N x

Now, using the b o u n d a r y condition (4.31), estimate (4.35), and the uniform cont i n u i t y of w, from (4.37) it follows t h a t sup (I' _< w(Cx/~), R2Nx[0,1] a contradiction to (4.33) for small e > 0. T h a n k s to (4.36), we can use the information provided by the fact t h a t u, w are, respectively, a viscosity sub- and supersolution of (VI). Choose then the test functions

c2(x'a):=w(y'-b)+

Ix - ~l 2 + la - ~l =

W(u, b) := ~(m.~) -

Ix-

2~

yl 2 + I~ -

'

-kl~(<x)m+
bl 2 _ 3 ( ( m > ~

+ ( y > ~ ) _ rl((1 - ~ )

+ (1 - b ) ) .

2~

F r o m (4.16), (4.17) we get through an e l e m e n t a r y c o m p u t a t i o n (4.38)

max{ Au - f - (ps + "y~) - e; as + r/} _< 0

at (g, g),

(4.39)

max{ Aw - f . (p~ - f2) -- g; a~ -- ~ } > 0

at (y, b),

where x-

y

a-b

E = ~m<~> m-~,

~ = ~m<~> m-z

.

N o w the parameter • > 0 plays a role. The subsolution condition (4.38) obviously yields ae + ~7 -< 0, so that a~-~7<0. Therefore, by the supersolution condition (4.39), ;~w - f .

(p~ - ~-~) - e > 0

at

(~, ~).

Combine the preceding with Au - f . (Pc + "Tx) - g < 0

at (~, ~)

4. H J B I N E Q U A L I T I E S

193

(another immediate consequence of (4.38)) to get

),(u(~, ~) - w(~,

~))

_< (f(~, ~) - f(~, b)) .p~ + 7f(~, ~). ~ + Tf(~, b). ~ + g(y, ~) - e(~, b). At this point we use assumption (4.8) and the estimates (4.35) to deduce from the preceding that (4.40)

),(u(~,g) - w(~,b)) 5 2Lw(v~) + w t ( 2 C v ~ ) + Kflm(<~) m + (~)m),

where K := L + supae[0,1] If(0, a)]. Let us now choose m E ]0,)'/K] to conclude, thanks to (4.40) arid the definition of ~, that (4.41)

6(Y,~,~,b) _< 2L),-la:(vZ~) + )'-lwt(2CvZE) .

Since the right-hand side of the previous inequality tends to 0 as z --* 0, a contradiction arises to (4.33) and (4.34), and the proof is complete. The following statement gives the characterization of the value function of the monotone control problem. Its proof is a straightforward consequence of Theorems 4.6 and 4.4 and Proposition 4.1 (iii), (iv). COROLLARY 4.7. Assume (4.8). Then the value function v defined by (4.6) is the unique viscosity solution of (VI) such that

v(x, 1) = J(x, 1)

Vx e ]~N

.

4.2. Optimal stopping In the optimal stopping problem the controller can act on the evolution defined by (4.1) by choosing the control a and also by stopping it at some stopping time ~ 0. The cost associated with the controlled evolution is given by (4.42)

J ( x , a , f l ) :=

/o

f ( y x ( t , a ) , a ( t ) ) e - ~ t dt + ~(yx(~),a))e -A9 .

Here ), > 0 is the discount factor and ~ represents a stopping cost. The value function is (4.43)

v(x) := inf J(x, a, fl), acA

0_>0

where .4 = { a : [0, +oc[ --~ A : a measurable }. In the following we develop the Dynamic Programming approach to the optimal stopping problem, leading to the characterization of v as the unique viscosity solution of the variational inequality of obstacle type max{),v(x) + H ( x , Dr(x)); v(x) - ~p(x) } = 0

x E ~N.

194

III. PROBLEMS W I T H U N R E S T R I C T E D STATE SPACE

Here H is the same Hamiltonian as in Section 2, namely,

H ( x , p ) = s n p { - f ( x , a) . p - g(x, a ) } . aEA

For the sake of simplicity and in order to point out the typical features of the problem, we perform all proofs in the case f ( x , a ) ~- f ( x ) , g(x,a) - g(x). The general case can be treated easily using the arguments of w The first result is on the continuity of v. PROPOSITION 4.8. Assume (A~), (A3), (A4), and

~ E BUC(RN) .

(4.44)

Then v E BUC(RN). PROOF. The proof is very similar to (and actually simpler than) that of Proposition 2.1. We leave it as an exercise. .,, The next proposition exploits some consequences of the definition of v. These can be seen as the formulation of the Dynamic Programming Principle for the present problem. PROPOSITION 4.9. Assume (A~), (A3), (A4) and (4.44). Then for all x E R N the following inequalities hold:

(i)

v(x) < ~(x),

(ii)

v(x)<_

/o'

e ( y x ( s , a ) , c ~ ( s ) ) e - ~ S d s + v ( y x ( t , a ) ) e -at,

V c ~ E A , t>_O.

Furthermore, for any x where the strict inequality occurs in (i) there exists to = to(x) such that (iii)

v(x) = inf I

v t ~ [0,t0].

~EA ~ Jo

PROOF. Assume f i x , a) -- f i x ) , g(x, a) =- g(x). Inequality (i) is obvious since, in particular, v(x) < J(x, 0) -- r To prove (ii), fix any t ~ 0 and think of yx(t) as the initial position. For each E > 0 there exists, by definition of v, some ~)~ > 0 such that

J(y~(t), ~)~) <_ viyx(t)) + E .

(4.45)

Now set ~ = de + t and observe that

J(x,~) =

/:

giY~(S))

e- ~ "

ds +

[ ~+t e ( y x ( s ) ) Jt

e- ~

ds + ~(y~(~))

e -~

.

4. HJB INEQUALITIES

195

Since yx(T + t) = Yy~(t)(T) for any 7 > O, the second integral above can be written, by a change of variable, as e--At f o ~ e(Yy~(t)(7) ) e -;~ dr; JO also~

r

e -xo = e-x%(y~(~)(o~)) e-~o~.

Therefore (4.46)

J(x, ~) -- ~0t g(yz(s)) e -:~s ds + e - ~ t j ( y ~ ( t ) , ~)~)

Vt_>O.

Hence it follows, by (4.45),

v(x) _< J(x, ~) _<

e(y~(~)) e - ~ d~ + e - ~ t ( ~ ( y x ( t ) + ~))

and (ii) is proved. Assume now that x is such that

v(x) < ~(x)

(4.47) and let ~)n _> 0 be such that

lim J(x,~9~) = v(x) .

(4.48)

We claim that there exists to > 0 such that (4.49)

7)n > to > 0~

for sufficiently large n. To see this, set 5n = J(x, ~)n) - v(x). Then, if r modulus of continuity for ~ and M is as in (A4),

v(x) +~n >_- M

e-~sds+e-x~ f0~176

-~(bx(en)-

) is a

xl)) 9

If for some subsequence 0n ~ 0, the preceding would imply

v(x) >_~(x), a contradiction to (4.47). Hence (4.49) holds. Observe now that for t E [0, to]

o'(x, ~n) =

e(y~(s)) e - ~ as + e-atJ(yy~(t I, ~n - t ) .

Since 0n - t _> 0, it follows by definition of v that

J(x, zOn) >

e(yx(s))e-aSds+e-;~tv(yx(t))

Vt E [O, to] 9

196

III. PROBLEMSWITH UNRESTRICTEDSTATESPACE

Let now n --~ + ~ to obtain, by (4.48),

v(x) >

//

e(yx(s)) e -~8 ds + e - % ( y x ( t ) ) ,

v t 9 [0,t0]

The preceding and (ii) show the validity of statement (iii).

9

As for the problems previously analyzed in this chapter, it is possible to derive from the Dynamic Programming Principle the optimality condition for v in PDE form. In the present case the HJB equation turns out to be the following (VIO)

max{An(x) + H ( x , Du(x)); u(x) - ~(x)} = 0,

x 9 R N.

This PDE problem is usually called a variational inequality of obstacle type or a complementarity system. This terminology is more transparent if one observes that (VIO) is equivalent to n(x) < ~(x),

(4.50)

I

An(x) + Y ( x , D u ( x ) ) < O,

[ [u(x) - f~(x)][)~u(x) + H ( x , Du(x))] = O, x 9 R N, (see the proof of Theorem 4.12). The fact that v satisfies (VIO) (or equivalently (4.50)) can be easily deduced from Proposition 4.9 assuming v 9 C I ( N N ) . However, since in general v ~ C I ( R N ) , (VIO) will be satisfied only in the viscosity sense. By this we mean that v 9 C(]~N) and, for all 4 9 CI(]I~N), (4.51)

max{ )~v(xo) + H(xo, D4(x0)); v(xo) - r

} <_ 0

at any local maximum x0 of v - 4; (4.52)

max{ )~V(Xl) + H ( x l , D4(xl)); V(Xl) - r

} _> 0

at any local minimum xl of v - 4. THEOREIVl 4.10. A s s u m e (A~), (A3), (A4), and (4.44). Then v defined by (4.43) is a viscosity solution of (VIO). PROOF. We already proved that v 9 BUC(]RN). Assume again f ( x , a ) =- f ( x ) , /~(x, a) = ~(x), so that H ( x , p ) = - f ( x ) . p - f.(x). Let x0 be a local maximum of v - 4, 4 9 CI(I~N) 9 Using (ii) of Proposition 4.9 and the continuity of v and 4, 4 ( x o ) - 4 ( y ~ o ( t ) ) ___ v(~0) - V(yxo(t))

<-

g(Yxo (s)) e -As ds + (e -xt - 1)v(Yxo (t))

for sufficiently small t > 0. Divide now by t and let t --* 0 + to obtain, thanks to the differentiability of ~, - D 4 ( x o ) 9f ( x o ) <_ f(xo) - )~v(xo). Since v(x) < ~ ( x ) at all x by Proposition 4.9 (i), the subsolution condition (4.51) follows.

4. HJB INEQUALITIES

197

Now let xl be a local minimum for v - 4. If V(Xl) = ~P(xl), then obviously max{ V(Xl) - ~(Xl); k v ( x l ) - f ( x l ) 9 D~(Xl) - e ( X l ) } ~ V(Xl) - ~ ( X l ) = 0 and v is a supersolution of (VIO). Assume then V(Xl) < ~P(xl) (the only other possibility in view of Proposition 4.9 (i)). Thanks to identity (iii) in Proposition 4.9 we get ~(Xl)--

~(yxl(t))

> V(Xl) --

Y(yxl(t))

t

-

t 1 f t

= 7

g(Yx'(s))e-~Sds+

e -;~t - 1

t

v(Yxl(t))

for sufficiently small t > 0 (recall that Xl is a local minimum for v - 4). Now let t -~ 0 + in the preceding to obtain -f(xl)-D~(Xl)

_~ ~(Xl) - - . ~ V ( X l ) .

Hence the supersolution condition is satisfied as well.

9

Let us show now t h a t the value function v is indeed the unique viscosity solution of (VIO). The result is obtained as a particular case of a uniqueness theorem already established in w The optimal stopping problem can indeed be interpreted as an infinite horizon problem (and consequently the whole theory of Section 2 applies to it).= To see this, introduce the control set .4 = A U { ~ } and the dynamics f : R N x A ~ ~ g by

f(x,w)-O,

f(x,a)=-f(x,a),

foraEA.

Let us also consider the cost ~': ]Rg x .4 ~ I~ y defined as g-(x,w) = Ar

~-(x,a) = g(x, a),

for a 9 A .

The Hamiltonian of the infinite horizon problem associated with f , ~ is r,p) = sup_{

- ? ( x , a) . ; -

a) }

lEA as we know from w equation (4.53)

It is now simple to check t h a t (VIO) is equivalent to the HJB

H ( x , u(x), On(x)) = 0

in ]1~N

in the viscosity sense. Therefore, we have the following THEOREM 4.11. Assume (A~), (A3), (A4), and (4.44). Then v is the unique viscosity solution of (VIO) in BUC(]RN). PROOF. The statement is a corollary of Theorem 2.12. We leave the details to the reader. 9

198

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

Let us conclude this subsection with a different proof of uniqueness. Assume again f ( x , a) - f ( x ) , g(x, a) - e(x). The idea is to show that if u is a viscosity solution of (VIO), then u is represented by

u(x) = J(x, O*(x)), for some tg*(x) such t h a t J(x, v~*(x)) = inf,> 0 J(x, ~). THEOREM 4.12. Assume (A~), (A3), (A4), (4.44), f ( z , a) =- f ( x ) , g(x, a) =- ~(x), and let u C BUC(]~ N) be a viscosity solution of (VIO). Then (4.54)

u(x) : J(x,~*(x))

: v(x)

Vx e R N

where

(4.55)

~*(x) = inf{ t > 0: u(u~(t)) = r

}.

PROOF. Let u E B U C ( ~ N) be a viscosity solution of (VIO) and consider the open set C = { x E R g : u(x) < ~p(x) }. The first step is to prove that u _< ~

(4.56)

in I~N

and t h a t (4.57)

Au - f 9D u - ~ <_ 0

in If{N,

(4.58)

tu-

in C

f . D u - t? = 0

in the viscosity sense. Assume by contradiction that u(xo) > ~b(x0) at some x0 E ~ g . Hence, by continuity, (4.59)

u(x) > ~ ( x )

Vx E B(x0, 6), 6 > 0 .

It is not difficult to show the existence of ~2 E CI(]R N) such that u - p has a local maximum at some point 9 E B(xo, 6), so that by (4.51) max{ )~u(~) - f ( ~ ) . D p ( ~ ) - ~(~); u(5) - ~p(5) } < 0 . This contradicts (4.59) and (4.56) is established. In view of (4.56) the inequality (4.57) is a straightforward consequence of the fact t h a t u is a viscosity solution of (VIO). To prove (4.58) it is enough to show that (4.60)

;~u - f . Du - ~ > 0

in C

in the viscosity sense. At any local minimum xl E C of u - ~ , p E CI(I~N), (VIO) gives nlax{/~U(Xl) -- f ( x l ) " D ~ ( x l )

and (4.60) follows since Xl E C.

--

~(Xl); U(Xl)

--

•(Xl) } >__0

4. HJB INEQUALITIES

199

Now we apply Proposition II.5.18 (see also Remark II.5.22) with f~ = ~N. Because of (4.56) and (4.57) we obtain

u(x) <_ u(y~(~)))e - ~ +

_< ~(yx(O))

e -~'~ +

f(yx(t))e-;~t dt e(y~(t)) e - ~ at

for all ~) _> O. Hence, u(x) < inf J(x, o) =: v ( x ) . 0>__o

To prove the reverse inequality let us deal first with x ~ C. In this case u(x) = r and O*(x) = 0 by (4.55). Hence

u(x) = r

= J(x,O*(x)) > inf J(x,O) =: v(x) . ~>o

Assume now x E C, so that (4.58) holds, and apply Proposition II.5.18 with f~ = C to get, after some calculations,

u(~)

= u(yx(e)) e -~

+

/o

e(yx(t)) e - ~ at

v 0 _< ~ < O*(x)

(note that this equality can also be obtained by Remark 2.34). Let 0 ~ v~*(x) in the preceding to obtain

u(x) = J(x, ~)*(x) ) > inf J(x, ~) := v(x), o>o since u(y~(vg*(x))) = r

The statement is proved.

9

REMARK 4.13. It may happen that, for some x, yx(t) E C for all t > 0. This means that u(y~(t)) < r for all t > 0. From the previous arguments it follows that

u(x) < ~(yz(O))e -~~ +

g(yx(t)) e-At dt

for all vq ~ 0. Hence for those x no finite optimal stopping time exists.

<1

REMARK 4.14. Let us point out that (4.54) and (4.55) say that x ~-~ ~*(x) is an optimal feedback map. The optimal time v~* = v~*(x) is the first time when the trajectory Yx(" ) hits the set S = { x E ]~Y : U(X) : ~(X) }, usually called the free boundary of (VIO). <1

200

III. P R O B L E M S W I T H U N R E S T R I C T E D

STATE SPACE

4.3. Impulse control In this problem the controller can act on the evolution described by (4.1) by choosing stopping times 0 i >_ 0 at which the current state is abruptly modified by an additive impulse ~i. The state equation becomes then ~y'(t)=f(y(t),a(t)), (4.61)

tE]Oi,~i+l[

/ y ( 0 ) = x,

( y ( O +i ) -__ y ( O _ )i + ~ ,

i

i =0,1.2...

The notation y(~i_) stands for f

0i

y(O i-1) -~- JOi_ 1 f(y(t), a(t)) dt and y(Oi+) := limt_0~ + y(t). The triplet /3 -- (a, { 0 i }, { 4i }) is the control; we denote by B the set of all ~ such that a E .4 and

~)iE [O,+cr (4.62)

0/+1 > 0 i..

0i--~+~casi--*+oc,

4i E N N

(~N is the set {4 = (~I,...,~N) E ~N : ~k --> 0, k ----1 , . . . , N }). We associate the controlled evolution defined by (4.61) and (4.62) with the cost functional (4.63)

J(x, fl) =

f0+o c

e(yx(s, fl), a(s)) e -As ds + Eze c(~i) e-a~

i=O where Yx(' ) is the solution of (4.61) (see Exercise 4.8) and The impulse cost c is assumed to satisfy

(4.64)

e -~(+~r

:= 0.

/c(~) = k + co(~)

k > 0 and co E C(R+N) with co(0) = 0,

/co(~) _< co(~)

if ~ - 4 E R y

,

as

I ;I

The value function of the present problem is

(4.65)

v(x) := inf Y(x,/3). /~eB

We have PROPOSITION 4.15. Assume (At) , (A3), (A4), and (4.64). Then v E BUC(RN). PROOF. Choose/3 = (a, { ~)i }, { 4/}) with arbitrary c~ E .4 and ~90 = +oc. Then (A4) gives M < M < A - A " The proof of uniform continuity is essentially the same as that of Proposition 2.1. -

-

4. HJB

INEQUALITIES

201

Following the usual pattern we now exploit the DynamicProgrammingPrinciple for the present problem. PROPOSITION 4.16. A s s u m e (A~), (A3), (A4), and (4.64). Then v satisfies (4.66)

v(x)=

= inf/f aEA " 0 ~ >_0

do

e(yx(s, fl), a(s)) e -:~s ds + e -'x~176inf (v(yx(O~ fl) + ~) + c(()) } . ~176

~E]RN+

PROOF. For the sake of simplicity let us assume that f and g are independent of a. By definition of v, for 0 ~ < +~c,

~o (4.67)

j~§

v(x) <_~o e(Y~(S))~-~d~+

+~c

o e(Y~(~))e-~ds+ Y-~c(~)~-~' " i=0

Set

j~O_FOr Y1 :=

o

o~ f(yx(s)) e -:~s ds,

Y2 := E

i=0

c(~i) e - : ~ "

The change of variable ~- = s - 0 ~ and the state equation (4.61) give Y1 =

f0 ec ~(yx(T + S~ e -~(~+~176d T = e -:~~ f0 +co ~(Yz(~-)) e -:~r dr,

where

z = yx(s~ ) + d .

(4.68)

On the other hand it is obvious that

i=1 Hence Y1 + Y2 = e -:~~ (4.69)

c(~) e -~oi + c(~ ~ +

i=0 = e-:~~176

/0

g(yz (7)) e -:w d7

fl) + c(~~

where fl := ({ ~ }, { ~i }) with i = 0, 1, 2,... As a consequence of (4.67) and (4.69) we get

v(x) < /o ~~ e(yx(S))e -~s ds +

e-~~176

+ e-~~176

>

202

III. PROBLEMS W I T H U N R E S T R I C T E D STATE SPACE

Since/3 is arbitrary in/3 it follows that

v(x) <_

(000t(yx(s))e-;~Sds+ e-~~

+

e-~O~

for all 0 ~ > 0 and ~0 E R N. The preceding and (4.68) yield

v(x) <_ inf ~" f~176 0~

+ e-~~

~"J 0

inf (v(yx(O ~ + ~ o ) + c(~O))}, ~oE~~

because the inequality is obvious for 0 ~ = +c~. To prove the validity of the reverse inequality, let E > 0 and choose t3~ E B such that

J ( x , ~ ) < v(x) + ~ . The same arguments as in the first part of the proof show that

J(x, 3E) >_ fo ~~e(yx(s)) e -)'s ds +

~-J,~o(v(ux(oo ) + ,~o) + ~(,~o))

Hence, denoting by w(x) the right-hand side of (4.66), v(x) + r >_ w(x); since r is arbitrary the proof is complete. .q REMARK 4.17. The Dynamic Programming Principle (4.66) can also be expressed as (i)

v(x) <_ inf (v(x + ~) + c(~)),

(ii)

v(x) <

e(yx(s, ~), a(s)) e -as ds + e-'Xtv(yz(t,/3)),

Vt > O, x E •N, ~ E /3. Furthermore, for any x where the strict inequality occurs in (i) there exists to(x) > 0 such that

v(x) =

(iii)

e(yx (s, 3), a(s)) e -a~ ds + e-a%(y~(t, r

vt 9 [0, to(~)]. It is convenient to introduce the (nonloeal) operator A4 defined by (4.70)

(.Mu)(x) := inf (u(x + ~) + c(~)); ~ER+ x

with this notation the Dynamic Programming Principle (4.66) reads as (4.71)

v(x) = inf ~ f o e(y~(s, ~), Z(s)) ~ - ~ ds + r176 aEA " Jo 0>_0

Z))} 9

4. H J B INEQUALITIES

203

If we go back to w the previous formula shows that v can be interpreted as the value function of an optimal stopping time problem with stopping cost r = A//v. The implicit character of the stopping cost (it depends through A// on the value function itself) is due of course to the presence of impulse controls. Let us now derive the infinitesimal version of (4.71); in this case the HJB equation for v takes the form of a quasivariational inequality, namely, (QVI)

max{ Av(x) + H ( x , Dv(x)); v(x) - (Aqv)(x) } = 0

x 9 ]~Y.

From (4.71) and Theorem 4.10 it would follow directly that v is a viscosity solution of (QVI) (by this we mean that (4.51) and (4.52) hold with r = A4v) if we knew that A/Iv 9 BUC(]~N). This property of the operator .h4, as well as some others, which are useful later, is the content of the next result. LEMMA 4.18. Assume (A~)), (A3), (A4), (4.64). Then, for all u , w 9 BUC(]~N), (i)

(k - I]w[l~) <_ .Mw < (k + I]wlloc),

(ii)

A4w 9 BUC(I~N),

(iii)

Aqw :> ~4u

(iv)

;~4(/zw + (1 - #)u) > #A/lw + (1 - l~)J~Au for all # 9 [0, 11.

/f w _> u,

PROOF. Assumption (4.64) implies c(~) > k + c0(0) = k for all ~ 9 ~ g . Hence

~(~ + ~) + c(~) > -I1~11or + k . On the other hand, the choice ~ = 0 gives

(Mw)(x) _< ~(~) + c(0) = ~(~) + k _< II~ll:r + k, and the proof of (i) is complete. To prove (ii), fix arbitrary x, x I in R N. For any ~ > 0 there exists ~ 9 ~+N such that

~(x' + ~ ) + c(~) ___(M~)(x') + ~. Then (Mw)(x)

-

(Mw)(x') <__w(x + ~ ) + c(&) - w(x' + f~) - c(~) + ~;

if w is a modulus for w, we obtain

( M w ) ( x ) - ( M w ) ( x ' ) _< w(Jx - x'[) + ~. Exchanging the roles of x and x I the reverse inequality is established, showing that M w 9 BUC(RN). The monotonicity property (iii) is straightforward.

204

III. PROBLEMS W I T H U N R E S T R I C T E D STATE SPACE

Let us now prove that (iv) holds (i.e., ~4 is concave). For any z > 0 there exists (~ E R+N such that (4.72)

z(x + ~ ) + c(~s) < (3dz)(x) + z,

where z = pw + (1 - #)u. Now, by definition of M , ~ ( M w ) ( x ) < ~w(x + ~ ) + (1 -

#)(Mu)(x) <_ (1

-

~o(~),

p)u(x + (E) +

(1 - # ) c ( ~ )

for all # E [0, 1]. Hence, adding the two inequalities and using (4.72), p ( 3 d w ) ( x ) + (1 - p)(./Mu)(x) < (.Mz)(x) + z

and (iv) is proved.

4

THEOREM 4.19. Assume (A~)), (Aa), (A4), and (4.64). Then v is a viscosity solution of (QVI). PROOF. By Proposition 4.15 and Lemma 4.18 both v and Jt4v belong to BUC(RN). The statement follows from Theorem 4.10 thanks to the Dynamic Programming Principle (4.71). ~l The characterization of v in terms of (QVI) is completed by a uniqueness result. Its proof makes use of the nonlinear operator T defined as (4.73)

(Tw)(x)

:=

inf ( f 0

e(yx(s,~),~(s))e-A~d~+(M~)(y~(O,~))e-~~

oEA ~ Jo 0>_0

where Yz is the solution of y'(t) ---=f ( y ( t ) , a ( t ) ) , y(o) = x,

t > O,

for a E A. LEMMA 4.20. Assume (A~)), (A3), (A4), (4.64). Then, for all u, w E BUC(NN), (j)

min{ k - [ I w l l o r

(jj)

r w E BUC(NN);

} <_ r w <_ M / A ;

(jjj)

T w > T u if w >_ u;

(jv)

T ( # w + (1 - #)u) _> ,urw + (1 - # ) r u , Vtz e [0, 1];

(v)

ifwi e BUC(NN), wi _> O (i = 1,2) satisfy

(4.74)

Wl - w2 <_ ~/Wl

for some "~ E [0, 1],

4. HJB INEQUALITIES

205

then, if f >_ O, we have Twl - Tw2 <_ 7(1 - p ) T w l

v~ e ]0, k/ll~lloo A 1[

where

fO co $(yx(s, c~EA

~(x) : = inf

cO, ~(s)) e - ~ ds .

PROOF. By definition of T, L e m m a 4.18 (i) gives (Zw)(x) < M

/0

e - ~ ds + e - ~ ( k +

Llwl{~),

V~ > 0.

Let 0 ~ + e c to o b t a i n the right-hand side e s t i m a t e in (j). On the other hand, by L e m m a 4.18 (i) again,

f0 ~ t(yx(S, ~), ~(s))

e-,ks

ds + e-X~ (.h,,lw)(yx(O, 4)) >_ - M

10

e - ~ ds

+ e-~~

- I{wll~)

for all a E ..4, 0 _> 0. A simple c o m p u t a t i o n proves the validity of the first inequality (j). To prove (jj), fix a r b i t r a r y e > 0 and choose as E A, 0~ _> 0 such t h a t

f0~ e(y~,(s,~), ~(s)) e-)~s ds + (Mw)(yx,(~, ~))

< (Tw)(x') + ~.

Then

(Tw)(x) - (Tw)(x') <_ A + e -:~'~ ( ( M w ) ( y x ( Oc, a~) ) - (Mw)(y~,( O~, ac) ) ) + r where A =

f0

(g(y~(s, a~)) - g(Yx' (s, a~)))e -~s ds.

The s t a t e m e n t follows from L e m m a 4.18 (ii) and s t a n d a r d estimates on lYe( 9 4) Yx'(', 4)1 (see Exercise 4.8). The m o n o t o n i c i t y and concavity properties (jjj) and (jv) are easy consequences of the analogous ones for .s Let us now prove (v). If (4.74) holds, t h e n (jjj) gives Tw2 >_ T((1 - 7 ) w l ) . Now use (jv) with u = 0, w = wl, # = 1 - % to o b t a i n T((1 - 7 ) w l ) _> (1 - ? ) T w l + "~TO. Hence (4.75)

Tw2 >_ (1 - "~)TWl + 7TO .

Let us now show t h a t (4.76)

TO >_ # ~

V # e ]0, k/{[~H~ A 1[ .

206

III, P R O B L E M S W I T H U N R E S T R I C T E D STATE SPACE

Observe that AdO - k (it follows from Lemma 4.18 (i)); hence

(TO)(x)

=

inf ~ f o

e(yx(8,~),~(s))e-~ds+ke

-~o}

aEA ~ Jo 0>_0

Since, by definition, k >_ p~(yx(O, a)) for all # E ]0, k/[[~]]or A 1[ and all 0 > 0, we get (4.77)

(rO)(x) > _ inf { f o

~e(y~(s,,~),,~(s))e-~ds+,~-~~

aE,A Jo

Here we used the fact that g _> pt, since ~ _> 0 and 0 < # < 1. By definition of ~, the right-hand side in (4.77) is #~(x) (see Proposition 2.5) and (4.76) is proved. Observe at this point that for any a E .4, 0 _> 0 and w E BUC(~N),

(rw)(~) ~

i oe ( ~ ( s , ~ ) , ~ ( ~ ) ) e - ~ d ~ + ( A d w ) ( y x ( O , ~ ) ) ~

-~~ .

Since w E BUC(RN), we can let v~ --, + e c to obtain, by Lemma 4.18 (i), that

(T~)(~) _<

e(y~(~, ~1, ~(~11 ~-x~ ds

v ~ 9 .4

and, consequently, (4.78)

T w <_ ~

V w 9 BUC(]~ N) .

Use this with w = w~ to obtain from (4.75) and (4.76) that Tw2 >_ (1 - "~)Twl + ~f#Twl to complete the proof of (v). 9 We are now in position to prove the announced uniqueness result for (QVI). THEOREM 4.21. Assume (A~), (A3), (A4), and (4.64). viscosity solution of (QVI) such that

M M f f <_ u(x) < --~

If u 9 BUC(I~ N) is a

for all x 9 ~ N ,

then u = v, where v is given by (4.65). PROOF. It is not restrictive to assume that u, 5, and v are nonnegative. In fact, if we define "[ = g + M , ~ = v + M / A , ~ = u + M / A , where M is the constant in (A4), it is easy to see that ~ ~, ~ are nonnegative, ~ is the value function associated with the running cost ~, and ~, ~ are solutions of (QVI) with H replaced by H - M. Let us observe first that T u and Adu belong to BUC(I~ N) (see Lemmata 4.18 and 4.20). Also, by its very definition and the theory of w Tu satisfies max{ ATu + H(x, DTu); T u - Adu } = 0

in

]I~w

4. HJB INEQUALITIES

207

in the viscosity sense. On the other hand, we also have max{ ,~u + H ( x , Du); u - 3 d u } = 0

in ]]~N

in the viscosity sense. Therefore, by the uniqueness result for (VIO) (see Theorem 4.11), (4.79)

u = Tu.

The same argument also shows that (4.80)

v =

Tv.

Since v > 0, we have

Hence, by Lemma 4.20 (v) with 3' = 1, Tu-

T v < (1 - p ) T u

for all p 9 ]0, k/ll~[[~ A 1[. But u = Tu, v = Tv, so that u -

v <

(1 -

~)u

for all # 9 ]0, k/]l~[[~/x 1[. Now fix # in the same interval and apply Lemma 4.20 (v) withwl--U, W2=V,~=l-#. Hence T u - T v <_ "/2Tu or, which is the same thanks to (4.79) and (4.80), u - v _< q,2u. Iterating this procedure gives the inequality U - - V _~< ~ n u

~'~ ---- 1 ; 2 , . . .

Since u is bounded and 3' 9 ]0, 1[, we obtain u _< v. Exchanging the roles of u and v, the opposite inequality is proved similarly. 4 The proof of Theorem 4.21 shows t h a t the value function v of the impulse control problem is a fixed point in B U C ( ~ N) of the nonlinear operator T. This suggests that the iterative scheme (4.81)

vn+l = Tvn,

n = 1,2 . . . .

should converge to v, at least for a suitably chosen starting point Vl. This is indeed the case if we choose (4.82)

v i ( x ) = g ( x ) :=

inf'/ aEA

+oo

f(yx(s, ct),c~(s))e-~S ds~,

,~j 0

"

.,

and the interest rate A is large enough. Observe that ~ is the value function of the infinite horizon problem in Section 2. In view of the theory developed there, ~ is the unique viscosity solution of Ag(x) + H ( x , D g ( x ) ) = 0

x 9 N iv .

208

Ill.

PROBLEMSWITH UNRESTRICTEDSTATESPACE

PROPOSITION 4.22. Assume (A~), (A3), (4.64), and that (A4) holds with a d r ) = ctr and ~ > L. Then the sequence {v,~} defined by (4.81) and (4.82) converges locally uniformly in ]I~N to V given by (4.65).

PROOF. We showed in the proof of Lemma 4.20 (v) (see (4.78)) that T w <_ ~ for any w E BUC(I~N). Hence v2 := Tvl = T~ <_~ . Since T is nondecreasing, the previous inequality easily implies that {vn} is nonincreasing. This also gives the bound (4.83)

v~ < ~ < M / ~ .

On the other hand, since - M / , ~ < ~ < M/,k we have (4.84)

v2 = Tvl >_ T ( - M / A ) >_ - M / A ,

where we used Lemma 4.20 (j) and the assumption k > 0. From (4.84) it easily follows that v~ >_ - M / A . This and (4.83) show that IIvnll~ is uniformly bounded. The sequence {vn} is also equicontinuous, namely, (4.85)

Iv'~(x) - v~(z)l <- ~ -ce L I x - zl

Vx, z E ]~N

Estimate (4.85) is immediately checked for n = 1. Let us proceed then by induction. By definition of T, for any r > 0 there exists a~ E ,4 and v% _> 0 such that (4.86)

v~+l(~) -v~+l(z) < I(o~) + e - ~

((M~)(u.(O~, ~)) -

(Mvn)(Uz(O~, ~))) + ~,

where

A standard computation reveals that

I(0~) s yl---s

Ix - z I (1 - e -(~-L)O~)

Assuming that (4.85) holds for n, the term depending on .A4 in (4.86) is estimated by )~

c! I s - zle -(~-L)~ L

4. HJB INEQUALITIES

209

Hence (4.85) is true for n + 1. By the Ascoli-Arzelfl theorem and the monotonicity of {vn} we deduce the existence of some ~ e BUC(]~ N) such that Yn ----e ~ locally uniformly in ]~g. At this point, the inequality

l]Tw - Tull~ < IIw - utfor

Vw, u 9 BUC(R N)

(its easy proof is left as Exercise 4.9) yields

Tv,~ ~

T~

locally uniformly in ~N. Hence, by (4.81), (4.87)

~ = T~.

We observed already that T~ is, by the very definition of T, a viscosity solution of (QVI). Therefore, by (4.87) and Theorem 4.21, ~ = v. ~, REMARK 4.23. The above result holds more generally for ~ 9 BUC(]~N); in this case (4.85) is replaced by the equicontinuity of {v,~}. A different proof of the convergence of {vn} can be performed using stability and comparison properties of viscosity solutions (see Exercise 4.10). <~ REMARK 4.24. Using Lemma 4.20 (v) it is possible to show that

IIv~ vmll~ < ~m-%~ll~ -

for any "7 < 1 and 7 > (1 -

for m > n,

k/II~ll~) v 0, whence the geometric rate of convergence

Let us conclude this section by sketching how the previous results allows us to synthesize an optimal impulse control { ~ i ~'i } in the simplified case where f and e are independent of a (i.e., the controls a are not present in the model). Let v be the unique viscosity solution of (QVI). Since v is bounded and c(~) --~ +oc as ]~l ---* +oo by (4.64), the infimum in the definition of ?vlv (4.70) is attained. By standard selection results (see [FR75]), there exists a bounded measurable mapping ~': ]1:~N ~ ]~N such that

(Mv)(~)

=

v(~ + ~'(~)) + c(~(~))

Vx 9

~N.

210

III. PROBLEMS W I T H U N R E S T R I C T E D STATE SPACE

Define by recursion { ~) ~, ~ } as follows: ~0 = 0, d~~176

t>

~~

/ i n f { t _> ~ i : v ( ~ i ( t ) ) = ( A d v ) ( ~ i ( t ) ) }, (4.88)

[ + e c if the set above is empty, d

~

i

A

(t) = f ( ~ i ( t ) ) =

t > v~i, +

for i = 1, 2 , . . . . Using a technique similar to that employed in the proof of Theorem 4.12 one can show the optimality of {~i, ~'~ }. The details are left to the reader (see, in particular, Exercise 4.12). 4.4. O p t i m a l

switching

In this section we consider the problem of minimizing an infinite horizon discounted running cost as in w over the class 7) of piecewise constant controls, namely, (4.89)

P := { a e A : act ) is constant for t e ]~9i, 0 i+1] },

where {~)i} is a sequence in [0, +cc] such that 0=0

(4.90)

~ < 0 I < . .. < 0 i < O i + 1 < . . . ,

lim v~i = + c o . i---*+oc

Unlike in w however, we assume here that for every s w i t c h i n g from one control setting to a different one, the controller incurs a positive switching cost. We assume that the control set is finite, that is, (4.91)

A = { 1,...,m}

for some m E N .

Observe that an admissible control a E ;o can be identified with the couple of sequences {Oi}, { a i } , where a ~ -- ct(0), a i = ct(t) for t E ]oi, l,9i+1] and a i 7s a i - 1 if 0 i < +oc.

The controlled dynamical system (4.1) then becomes (4.92)

~y'(t)=f(y(t),ai), / y ( 0 ) = x e R N,

and the cost functional is

te]0~,0i+l[,

(i=0,1,...),

4. HJB INEQUALITIES

211

Here we assumed for simplicity that the switching cost k depends only on the control settings involved but not on the state where the switching occurs. Let us observe explicitly that if k - 0, then inf J(x, ~) = inf J(x, ~) aE'P

aEA

(see Remark 2.15); the case of zero switching cost is therefore covered by the theory of Section 2. Let us assume then that k(j, 2) > 0,

(4.94)

if ~e# j,

k(j, j ) = O, k ( j , f ) < k ( j , i ) + k(i,f),

if f 7~ j # i .

We consider the subsets :PJ of admissible controls defined by p J := { ~ e ~ ' : 4(0) = j};

correspondingly we define the value functions (4.95)

vJ(x) := inf J(x,c~),

j = 1. . . . , m .

The proof of the next result on the regularity of vj is completely similar to that of Proposition 2.1, so we omit it. PROPOSITION 4.25. Assume (A~), (Aa), (A4), and (4.94). Then v j E BUC(]R N) for all j = 1 . . . . , m. Following the usual pattern we derive next the Dynamic Programming optimality conditions. PROPOSITION 4.26. Assume (A~), (A3), (A4), and (4.94). Then, for each j E {1,... ,m} the value function vJ satisfies for all x E ]~Y and t > 0 (i)

vJ(x) < rain (ve(x) + k ( j , g ) )

(ii)

vJ(x) <_

~#j \

/:

e ( y x ( s ) , j ) e -as ds + V q y x ( t ) )

e -:'t .

Furthermore, for any x where the strict inequality occurs in (i) there exists to = to(x) such that

(iii)

vJ(x) =

2

g(yx(s), j ) e -as ds + v j (yz(t)) e -xt

Vt ~ [0, to].

PROOF. Fix g C { 1,... , m } , f # j, and choose any a = ({0i}, {de}) in Pt. Then define 5 = ({~i } , { ~ i }) by setting ~0=0 '

~0=j,

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

212

and, if ~o < ~ 1

0~"i -= 0i-1.

~i = ai-1.

i = 1,2 . . . ,

and in the case 0 o = 01 0~ = 0 ~.

i = 1, 2 , . . .

~ ~

By definition, 5 E 7~J; also we have, if 0 ~ < 01,

J(x, 5) = k(j, e) + J(x, a), as a simple computation shows, while in the case 0 o = 01,

J ( x , ~ ) = k ( j , a 1) + J ( x , ~ ) - k(e,a 1) < k(j,e) + J ( x , c 0 by (4.94). Therefore, taking the infimum over c~ C p t

vJ(x) <_ J(x, 8) <_ k(j,g) + vt(x)

e T~ j

and (i) follows. The statements (ii) and (iii) are proved in the same way as the corresponding ones in Proposition 4.9. In view of the previous proposition it is convenient to define operators ~ J acting on vectors u = ( u l , . . . , u TM) ~ B U C ( N N ) m by setting (4.96)

( M J u ) ( x ) = micn(ut(x) + k(j, t))

j = 1. . . . , m .

The HJB equation for the vector valued value function v = ( v l , . . . , v m) defined by (4.95) turns out to be the following system of quasivariational inequalities: (SQVI) max{ AuJ(x) - i f ( x ) . DuJ(x) - s

uJ(x) - ( M J u ) ( x ) } = 0

x C R N,

j --- 1 , . . . ,m. (We used here the notations i f ( z ) and eJ(x) to denote f ( x , j ) and e ( x , j ) . ) The coupling of the unknowns in (SQVI) appears only in the pointwise implicit obstacle condition (4.97)

uJ(x) <_ ( M J u ) ( x ) .

On the basis of theory developed in w167 it is natural to consider the following notion of viscosity solution of (SQVI): a vector function u = ( u l , . . . , u m) E BUC(IRN) m is a viscosity solution of (SQVI) if, for each j E { 1 . . . . . m } and each

~ CI(I~N), (4.98)

max{ AuJ(x) - i f ( x ) . D ~ ( x ) - gJ(x); uJ(x) - ( M J u ) ( x ) } <_ 0 at all local maxima x of u j - ~,

(4.99)

max{ AuJ(x) - i f ( x ) . D ~ ( x ) - gJ(z); uJ(x) - ( M J u ) ( x ) } > 0 at all local minima x of u j - ~.

4. HJB INEQUALITIES

213

THEOREM 4.27. A s s u m e (A~), (A3), (A4), and (4.94). Then the value f u n c t i o n v -- ( v l , . . . , v m) defined by (4.95) is a viscosity solution of (SQVI). PROOF. The proof is essentially the same as that of Theorem 4.10 and we leave it to the reader. 9 The main result in this section is on the uniqueness of viscosity solutions of (SQVI). THEOREM 4.28. A s s u m e (A~), (A3), (A4), and (4.94). I f u -- ( u l , . . . , u m ) , w = ( w l , . . . , w m) E BUC([(N) m are viscosity solutions of (SQVI), then u - w. PROOF. The scheme of the proof is similar to that of the comparison and uniqueness properties for HJB equations (see Section II.3 and w in this chapter). We therefore give details only for the points concerning the typical features of (SQVI) and just outline the standard part. As in the proof of Theorem 2.12 we consider the auxiliary functions

~J(~,y):=~Y(x)-wJ(y) I:~-Yl2

n((x>~ +(y)~)

2~

for j = 1 , . . . , m, where {x) = (1 + Ixl2) U2 and E, 3, ? are positive parameters to be chosen conveniently (see the proof of Theorem 2.12). Let ( ~ , f0) be a global maximum point for qsJ over ]~N • RN. Then there exists g E { 1 , . . . , m }, say g = 1, such that

(4.100)

(I)l(x~,y 1) = max l<j
max

r

RN x~N

Moreover, the estimates 1 4 - y~l - - - , o,

Ix1 - y~ C

-, o

as ~ ~

0

hold. Since u, w are viscosity solutions of (SQVI), by standard arguments

(4.101) (4.102)

max{ .~ul (x01) -- f l ( x l ) - ~ -- el(x01); ul(x 1) -- (~,llu)(xl) } ~ 0

max{ Aw'(yo1) - f l ( y 1 ) . 7 1 - gl(yol); wl(y~)

-

( J ~ J l w ) ( y l ) } _~ 0

with -

x 0~ _ y0~ + z ~ x l ( x 0 ~ ) ~ - 2 , E

~ = x I _ y~

Z~y0~/y~)~-2.

Consider now the two possibilities: c a s e 1:

w i ( v 0 ~) < ( M l w ) ( y ~ )

case 2:

wl(y~) = (Mlw)(ylo)

(no other case is possible because w 1 is a subsolution of (SQVI)). If case 1 occurs, then (4.101) and (4.102) imply, respectively, Aul(xo1) - f l ( x ~ ) . r - ~(xlo) < 0

214

III. PROBLEMSWITH UNRESTRICTEDSTATESPACE

and

~l(yo~) _ / ~ ( y l ) , ~ _ e(yo~) > o. Hence

A(ul(x~) _ wl(Ylo) ) < f l ( x ~ ) . ~ _ f l ( y ~ ) . r l + g(x 1) _ g(yl) and we conclude that u I = w 1 as in the proof of Theorem 2.12. B y (4.100), this implies u j = wJ for each j E { 1 . . . . , m } (see Exercise 4.14). If case 2 holds, then, by definition of M 1, there exists j ~ 1, say j = 2, such that (4.103)

wl(y 1) : w2(y1) d- ]g(a1, a2).

By (4.100), ffpl(x 1, yl) ~ ~2(xl ' yl), so that (4.104)

ul(x~) - u2(x 1) > wl(y~) - w2(yol).

It is easy to deduce from (SQVI) that (4.105)

u l ( x 1) < u2(x~) + k ( a l , a 2 ) .

Thus (4.103), (4.104), and (4.105) yield ul(x I) = u2(x 1) + k(a 1, aS). Because of (4.103) the above gives

~ ( x 1) _ ~ ( y 0 ~) = ~ ( x l ) _ ~(~0~), so that

by definition of ~hJ. Now repeat the previous considerations with the index 2 replacing 1. Should case 1 occur we are done. Otherwise there exists e~ E { 1 . . . . , m }, g2 ~ 2, such that w2(y01) = w~2(y 1) + k(a2,d2) . Observe that necessarily g2 • 1. Indeed, were g2 = 1, the preceding and (4.103) would imply

k(a 1, a 2) + k(a 2, a 1) = 0 a contradiction to assumption (4.94). Hence we may assume e2 = 3 and prove as before

9 ~(x~, y0~) = ,~(x0~, yl) = ~ 1 ( x l yl>. Repeating the preceding calculations with the index 3 replacing 2 and so on, after finitely many steps an index j < m for which case 1 holds is reached and the proof is complete. 4

4. HJB INEQUALITIES

215

REMARK 4.29. The proof of Theorem 4.28 shows in fact t h a t a comparison principle holds for (SQVI), namely, that uJ <_ wJ, j = 1 , . . . , m , if u = ( u l , . . . , u m) and w = ( w l , . . . , w m) are, respectively, a sub- and a supersolution in the viscosity sense. <3 By Theorems 4.27, 4.28, and Proposition 4.25 the value function v = (vl, 9 9 v m) is the unique viscosity solution of (SQVI). Let us proceed now to the synthesis of an optimal control. By analogy with the results of w and w for fixed x E ]1~N and j E A we set (4.106)

~0 = 0,

30= j

and for i = 1 , 2 , . . . (4.107)

(4.108)

f i n f { t > ~ i - 1 : vi-l(yx(t) ) = (Mi-lv)(yx(t)) } +~

if the preceding set is empty

[ a n y g # 3 i-1 such t h a t (Mi-lv)(yx(~i))=ve(yx(~i))+k(3i-l,g) = [~i-1 if~i---+oc.

The first thing to check is that ~ i ~ + c o as i ~ +oo. Indeed, we claim that ~ i > ~ i - 1 + a for some a > 0 and i = 2 , . . . . This is trivially true ifO i-1 = + o c for some i. Otherwise, assume that, for any a > 0, ~ i < ~ i - 1 + a for some i. Then, (4.107) and (4.108) imply the existence of some ~ i _< t < ~ i - 1 + c~ such that

vi-l(y,(t)) = vt(yx(t)) + k(~-l,e), for some g e A, g r ~ i - 1 vi-2(yx(~i-1)) = vi-l(yx(Oi-1)) + k ( ~ i - 2 , ~ i-1) < vg(yx(~i-1)) + k(~i-2,1f) . From assumption (4.94) it follows

0 < k ( ~ - 2 , ~ ~-~) + k ( ~ - ~ , ~ ) - k ( ~ - 2 , t ) < 2~o(a) where a; is a modulus of continuity for v(yx( 9 )). This gives a contradiction and proves the claim. Hence ~ = ({ ~ i }, { 3 i }) C PJ and it is not hard to show, following the lines of the proof of Theorem 4.12, that

J(x, 3) = v j(x) . Exercises 4.1. Consider the infinite horizon problem of w with control set A -= [0, 1]. Show that its value function v satisfies v(x) < v(x, a) for all a E A, where v(x, a) is defined by (4.6). 4.2. For Y and v as in (4.42) and (4.43) show t h a t for any e > 0 there exists 0~ _ 0 such t h a t v(x) - v(z) < J ( x , 0~) - J ( z , 0~) + E . Use this to prove Proposition 4.8.

216

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

4.3. Set ~(x) =

f ( y x ( t ) ) e -At d t .

Use the comparison results to prove that v ( x ) < min{~(x); ~(x)},

V x e ]RN,

where v is defined by (4.43). 4.4. Observe that for a,/3 E R the relation max{a, 3} = 0 is equivalent to a _< 0, /3 < 0, a 3 = 0. Use this to prove that (VIO) and the complementarity system (4.50) are equivalent in the viscosity sense. 4.5. Give a direct proof of Theorem 4.11. Show, in particular, that if u l , u 2 are, respectively, a sub- and a supersolution of (VIO), then ul _< u2. 4.6. Consider the optimal stopping time problem with f ( x , a ) /~(x). Prove that

= f ( x ) , g(x, a) =

Av - f . D v =

in the open set C = {x E ]I~N : U(X) ( ~ ( X ) } . Stampacchia type inequality

Discuss the validity of the Lewy-

min{A~;, - f . D e ; e} < Av - f . D v < g

in ~ g .

4.7. In the framework of Exercise 4.6 assume that infi~N ~(X) > A-1 sUpRN ~(X). Check t h a t in this case v ( x ) = f + ~ e(yr e -'xt dt and that 0*(x) = + o c for all x. Prove t h a t in the case ~ ( x ) -- ~0 with ~0 _< A-1 inf~,~- t?(x) the value function is v ( x ) -- ~o and that O*(x) = 0 for all x. 4.8. For fixed/3 = (a, {Oi}, {4/}) as in (4.62), denote by Y0 the solution of y'o(t) = f ( y o ( t ) , a ( t ) ) ,

for t > 0,

y0(0) = x .

Then recursively define yi (i = 0, 1, 2 . . . . ) by

y~(t) = f ( ~ ( t ) , ~(t))

for t > ~i,

yi(t) = y i - l ( t ) + X{e,=t}sci

for t _< v~i

(here, X{e,=t } = 1 if t9~ = t, X{o~=t } = 0 otherwise). Check that y~(t) = ~ ( t )

v t c [0,~q, v n > i

and define, consequently, for t _> 0 yx(t,/3):=

lim y i ( t ) .

i~+~

Prove t h a t y~(t, ~) is a solution of (4.61). Prove also that ]y.(t,/3) - y.(t, j3)[ <_ eLtlx - z I for any x, z E IRN.

4. H J B INEQUALITIES

217

4.9. For A/I as in (4.70) show that

] l M w - Mul[ac <_ ]]w- ull~c

Vw, u e B(RN) .

Use this and the inequality (Tw)(x) - (Tu)(x) < e-~~

~)) - (Mu)(yx(0~, Z))] +

(which holds for any ~ > 0 for suitable 0E > 0) to conclude that

4.10. Let un be viscosity solutions of max{Aun(X) + H,~(x; Dun(X)); un(x) - (A/Iu,)(x)} = 0 in ]RN. Assume that Hn --+ H, un --+ u locally uniformly and also that II(un- u)-I[~

, 0

as n -+ + ~ .

Prove that (i)

if u(xo) < (Mu)(xo) at some x0 then, for small r > 0,

),un + H,~(x, Du) = 0

in B(xo, r)

[Hint: nse the condition on (u,~ - u ) - and M ( u n + [[(u~ - u ) - I ] ~ ) = Mu,, +

II(u~ (ii)

u)-II~

>

Mu];

deduce from (i) and stability results that u is a supersolution of

(QVI).

Finally, use the stability properties to show that u is a subsolution of (QVI). 4.11. Prove that the sequence {oi} defined by (4.88) satisfies the feasibility condition limi~+or ~ i = +oc as required in (4.62). 4.12. Let u = ( u l , . . . , u m) be a viscosity solution of (SQVI). Mimic the first part of the proof of Theorem 4.12 to show that

uJ(x) < (.MJu)(x) 4.13. Generalize the theory in w

g x e IRN, g j e { 1 , . . . , m } . to the case of a state-dependent switching cost

k(X, a j, d). 4.14. For ~J as in the proof of Theorem 4.28, use the inequality

o21(xl, ya) e ,~J(x,x),

Vx E1RN, j E { 1 . . . . ,m},

to deduce that if u 1 = w 1 then ~J(~) - ~J(x)

for all j E {2 . . . . , m}.

_< o ( 1 )

as ~ - + 0 +

218

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

5. A p p e n d i x : Some results on ordinary differential equations We begin with a result that is used many times throughout the book. LEMMA 5.1 (GRONWALL INEQUALITY). I f w E Ll([to, tl]) andh E L~([to, tl]) sa-

tisfy, for some L > O,

(5.1)

w(t) <_ h(t) + L

w(s) ds,

for a.e. t E [to, t1],

then (5.2)

w(t) < h(t) + Le Lt

h(s) e -L~ ds,

for a.e. t E [to, tl] .

If, in addition, h is nondeereasing, then w(t) < h(t)e L(t-t~

(5.3)

for a.e. t E [to, tl] .

PROOF. Define t

z(t) := e -L(t-t~

w(s) ds Jto

and note that z is absolutely continuous in [t0,tl]. e -L(t-t~

After multiplying (5.1) by

we get

z'(t) <_ h(t)e -L(t-t~

for a.e. t E [to, t l ] .

We integrate both sides and multiply by e L(t-t~ to obtain

/to'

w(s) ds <_ e Lt

f

h(s) e -L~ ds,

for a.e. t E [to, tl].

,]to

This inequality and (5.1) give (5.2). If h is nondecreasing the integral in (5.2) is estimated by h(t) ftto e -L" ds, and then (5.3) follows from a simple computation. 4 Next we prove the existence of solutions of the integral equation (5.4) for short time of Section 1. recall that a is a constant

y(t) = x +

f(y(s), a(s)) ds

and for a given measurable control function a under the assumptions To do this we need two classical results of functionM analysis. We set Y of functions I ~ ~ g ~- C ]I~, is uniformly bounded if there C such that ]G(t)l < C for all t E I and G E Y, and it is called equicontinuous if for all z > 0 there exists (~ > 0 such that IG(t) - G(s)l _< z for all t,s E I, I t - s I < 5, and for all G E Y.

5. APPENDIX:SOME RESULTSON ODES

219

THEOREM 5.2 (ASCOLI-ARZEL~). I f I C_ I~ is compact and Y is a uniformly bounded and equicontinuous set of functions I --~ ]RN, then any sequence in Y has a uniformly convergent subscquence. For the proof see, for instance, [RUB6]. THEOREM 5.3 (SCHAUDER FIXED POINT THEOREM). Let X be a closed and convex subset of a Banach space. I f F : X --* X is continuous and f ( X ) has compact closure, then there exists x C X such that F ( x ) = x. For the proof see, for instance, [GT83]. THEOREM 5.4 (LOCAL EXISTENCE OF TRAJECTORIES). Assume (A0) and (A1), fix x e ~ and set M = M~ := s u p { I f ( z , a ) l : Iz - xt <_ 1, a ~ A } . Then f o r any to C N and ct ~ ,4 there exists a lipschitzean solution y of (5.4) defined on [to, to + 1/3I], and it satisfies (5.5)

ly(t) - x I < M ( t - to)

f o r all t .

PROOF. Set tl = to + 1 / M and X := { g : [to, tl] ~ B(x, 1) continuous, g(to) = x } . For any g E X we can define F g ( t ) := x +

y(g(s); c~(s)) ds,

to < t < tl,

because the function to be integrated is bounded by M and it is measurable. Moreover, IFg(t) - x I < M ( t l - to) = 1

for all t,

thus F : X ~ X. We claim that F is continuous with respect to the uniform convergence. In fact, if g m g ~ X, then for all t E [to, tl] IFe~(t) - Fg(t)l <

/:

If(g,~(s), ~(s)) - f(g(s),

~(s))l

ds,

and the right-hand side tends to 0 as n ~ oe by Lebesgue dominated convergence theorem if 9n --* g. Thus Fgn -* F g uniformly. Next we observe that the image F ( X ) is equieontinuous because (5.6)

IFg(t) - Fg(7-)[ = f t f ( g ( s ) , a(s)) ds <_ M It - 7-I

for all g E X, t, T E [to, tl]. Then F ( X ) has compact closure by the Ascoli-Arzelh Theorem 5.2. Since F satisfies the assumptions of Schauder's Theorem 5.3, it has a fixed point y which is a solution of (5.4) in [to, tl]. The estimate (5.6) gives the Lipschitz continuity of g with Lipschitz constant M, so (5.5) holds as well. 4

220

III. PROBLEMS WITH UNRESTRICTED STATE SPACE

Now we can prove the existence of solutions of the integral equation (5.4) for all times, their uniqueness and continuous dependence upon the initial state x. THEOREM 5.5 (GLOBAL EXISTENCE OF TRAJECTORIES). A s s u m e (A0), (A1), and

(A3). Then for any to E ]~, x E ]~g and a E A there is a unique solution Yz : [to, +oc[ ---* •N of (5.4), and it satisfies (5.7)

[yx(t)[ <_ ([x] + v / 2 K ( t - to)) e K(t-t~

where K := L + supa~A If(0, a)[. initial condition yz(to) = z, then

(5.8)

for all t > to,

Moreover, if Yz is the solution satisfying the

[yx(t) - yz(t)l <_ eL(t-t~

-- z[,

for all t >_ t o .

PROOF. We first prove (5.8) which implies the uniqueness of the solution of (5.4) by taking x = z. We compute d

d-~[yx(t) - yz(t)l 2 = 2[yx(t) - yz(t)]. [ f ( y z ( t ) , a(t)) - f ( y , ( t ) , a(t))] <_ 2L ]yx(t) - y~(t)] 2

for a.e. t >_ to,

where the inequality follows from (A3). Since ~z(t) = [yx(t) - yz(t)[ 2 is absolutely continuous, by integrating both sides from to to t we get ~(t) < ~(t0) + 2L f~i ~(s) ds, and thus by Gronwall inequality

~(t) _< Is - zl2d L(t-~~ which gives (5.8) for all t such that both solutions y~ and Yz exist. Next we show the estimate (5.7). Note that (A3) implies [f(y,a) - f(0, a)]. y < L]y[2 and therefore

(5.9)

f ( y , a ) , y <_ sup[f(0, a)[ lYl + L [y[2 _< K ( I + ]y[2). aEA

Then for any control a E ,4 the solution of (5.4) satisfies d l y ( t ) ] ~ = 2 y ( t ) - f ( y ( t ) , a ( t ) ) < 2K(1 + ly(t)12),

thus integrating from to to s

ly(s)t 2 <

Izl2+2K(s-to)+2K

[y(t)12dt.

5. APPENDIX:SOME RESULTSON ODES

221

Now Gronwall inequality gives for s < t [y(s)l 2 ~ (Ixl 2 + 2 K ( s - to)) e 2K(s-t~ which implies (5.7) for all t such that the solution exists. To show the global existence, let us consider the supremum t of the times tl such that there exists a solution of (5.4) defined on [t0,tl]. Let us assume by contradiction that t < +c~. If a solution exists in [t0,t] we have a contradiction because the solution can be continued on the right of t by the local existence Theorem 5.4. So we are left with the case that the maximal interval of existence is [t0,t[. We claim that limt_ 4- y(t) = ~ E ]t~g. In fact we have shown that ly(t)] _< (Ixl + 4 2 K ( t

- to)) e 2K(~-t~ =: C

for t e [t0,t[. Then we set M ' : = s u p { f ( y , a ) : IYi <- C, a c A } (A1)), and get from the equation (5.4), for s,t < t,

(M'<_ +~c by

t

ly(t)-y(s)[

<_

If(y(Q,a(T))ld'r<

M'lt-s],

which implies the existence of the desired limit. Now it is easy to see that the extension of y ( - ) obtained by setting y(t) = ~ is a solution of (5.4) in [t0,t], which completes the proof. 9 REMARK 5.6. Sharper estimates than (5.7) can be given. We can write, for all t such that y(t) ~ O, lY(t)l d ] y ( t ) ] = ~

ly(t)l 2 = y ( t ) . f ( y ( t ) , a ( t ) )

_ Cly(t)[ + Lly(t)[ 2 by (5.9), where C = supaeA If(0, a)l. We divide by ly(t)], integrate in time and apply Gronwall inequality (5.2) to get, with some easy computations, (5.10)

lyx(t)l _< Ixle L~ + 2C( e Lt - 1) 9

Similarly, from d M r ) - xf ~-/ly(t) - xl = (y(t) - x ) . f ( y ( t ) , a ) <_ L ly(t) - x] 2 + (y(t) - x ) . f ( x , a ) , we get (5.11)

1 (eLt lye(t) - ~1 ~ Z sup If(x,~)l -1). aEA

222

III. PROBLEMSWITH UNRESTRICTEDSTATESPACE

REMARK 5.7. Most conclusions of Theorem 5.5 remain true if we weaken (A3) to its local version (5.12)

( f ( x , a) - f ( y , a)) . (x - y) <_ L R I z - yl 2

for all x, y E B(0, R), R > 0, a E A, plus the global bound

f(y, a). y < K(1 + ly] 2)

(5.13)

for all y E 1RN, a E A. In fact (A0), (A1), and (5.12) are enough to get local existence and uniqueness of the trajectory by the same proof, and (5.13) is all we used to get the estimate (5.7) and the global existence. Of course (5.8) does not hold any more, but it is easy to show that for all T > to there is CT such that (5.14)

l y x ( t ) - y z ( t ) l <_eCT(t--t~

I,

f o r a l l t E [t0, T ] .

<~

We end this appendix with the classical theorem of differentiability of solutions of ordinary differential equations with respect to the initial data. We need a version that allows the vector field to be merely measurable with respect to the time variable. Its proof can be found, for instance, in [Ku86] and [Bre93]. We recall that the fundamental matrix M(s, t) associated with the linear system (5.15)

~'(t) = A(t) ~(t),

t e [to, tl]

is the unique solution of theequation M ( s , t) = I +

f,

A ( r ) M ( r , t) dr,

s, t E [to, tl],

where t ~ A(t) is a bounded and measurable map from [to, tl] into the set of N x N matrices, and I is the identity N x N matrix. We recall also that the i-th column mi of M ( . ,to) (i.e., mi(s) = M(s, to)ei where ei has the i-th component equal to 1 and the other components null) is the solution of (5.15) with initial condition {(t0) = ei, that is, it solves m i ( s ) = ei +

A(r)mi(~-)dr,

s E [t0,tt] .

We consider the ordinary differential equation (5.16)

SY'(t) = F(y(t),t), [y(t0)

t e ]to, h [ ,

= x.

We assume it has a solution in the usual integral sense and denote it by S(t, to, x) = y(t). We assume F : ]~N x [to, tl] ---*]~N is bounded on any compact set and 9 for any x the function t ~-* F(x, t) is measurable; 9 for any t the function x ~-~ F(x, t) is continuously differentiable; moreover, its Jacobian matrix D x F is bounded on ~ x [to, tl] for every compact K: C_]~N.

6. BIBLIOGRAPHICALNOTES

223

THEOREM 5.8. Under the previous assumptions let ~(. ) = S ( . , to, xo) be the solution of (5.16) with initial point x = xo. Then, for any t E [t0, tl], the map x ~ S(t, t0, x) is continuously differentiable in a neighborhood of xo. Moreover its Jacobian matrix at the point xo is DxS(t, to, xo) = M(t, to), where M ( 9 9 ) is the fundamental matrix of the linear equation ('(t) = DzF(~(t), t ) ( ( t ) . Note that this result gives the differentiability of the trajectories of the system (S) (i.e., the solutions of (5.4)) with respect to the initial position for a fixed control function a E ,4, that is, the differentiability of x ~ yz(t, a), under the assumptions (A0)-(A3) if in addition x ~-* f ( x , a) is continuously differentiable for all a E A with bounded Jacobian matrix over compact sets (i.e., wf(r, R) = LRr in (A2)). 6. B i b l i o g r a p h i c a l

notes

The classical theory and the history of Dynamic Programming were outlined in Chapter I. The results of Sections 2.1 and 3.1 are essentially due to P.L. Lions [L82], see also [L83b, L85a, E83]. We have weakened some assumptions, in particular we have replaced the usual Lipschitz continuity of the dynamics (1.1) with condition (A3). Under these assumptions the comparison Theorems 2.12 and 3.7 are proved by the techniques of Ishii [I84, I86, I88b], see also [BS94]. In connection with Remark 2.15 we recall that in some control problems a change in the set of admissible controls may lead to unexpected changes of the value, the so-called Lavrentiev phenomenon, see [Ce83, Zo92] and the references therein. Verification theorems such as those of w are trivial applications of comparison results for viscosity sub- and supersolutions; they were explicitly observed in [B92, BSt93], with verification functions not necessarily continuous, see Chapter V. Some results based on different tools of nonsmooth analysis for locally Lipschitz verification functions can be found in Clarke's books [Cla83, Cla89] and in [VW90a]; for the classical result in the case of piecewise smooth value function we refer to [FR75]. The main result of w Theorem 2.32, is due to P.L. Lions and Souganidis [LS85]. Corollary 2.28 was proved by Mirica [Mi88, Mi90b] (see also [Mi92b]), and the Backward Dynamic Programming Principle was first explicitly observed by Soravia [Sor93b]. The name bilateral solution comes from [Ha911. For other approaches to sub- and superoptimality principles see [Sw96a] and [Sor96b]. Subsection 2.4 mostly follows P.L. Lions and Souganidis [LS85, LS86], but they deal with the case of Lipschitz sub- and supersolutions. To treat merely continuous functions we need the generalized Dini (contingent) derivatives of Definition 2.36, which can be used even for semicontinuous functions. The classical Dini derivatives

224

III. PROBLEMSWITH UNRESTRICTEDSTATESPACE

were introduced in [D1877], and the oldest paper that we know where the generalized version appears is [Yo68]. J.P. Aubin [AuS1] derived the upper (respectively, lower) contingent derivative as the natural concept of derivative associated with the Bouligand-Severi contingent cone to the epigraph (respectively, hypograph) of the function. These tools of nonsmooth and set-valued analysis have a number of applications, see, for example, [AC84, AF90, Au91] and the references therein. The use of the Dini derivative for Hamilton-Jacobi equations goes back to Subbotin [SUB0, Su84] for Lipschitz functions, and the generalized version appears in papers of his school [AT87, RS88] and of Frankowska [Pr87, Fr89b]. The equivalence between Dini-minimax and viscosity solutions was proved by many authors in the Lipschitz case by various methods under different assumptions [EI84, LS85, ST86]. In the general case it was observed in [Fr89b] that Dini solutions are viscosity solutions (Lemma 2.37 here), and Subbotin proved the equivalence [Su93b] by means of the property of the subdifferential described in Remark 2.44, see also [ClaL94] for another proof of this rather deep property. The main reference on the theory of minimax solutions is the monograph [Su95], we give some more historical information in Chapter VIII in connection with the theory of differential games. The regularity results Propositions 2.45 and 2.47 have been improved by Hijab [Hi91] and Lions and Souganidis [LS93] to show that for smooth Hamiltonian and discount rate ,k large enough, the solution of the HJB equation becomes smooth and it solves the equation in the classical sense. Subsection 2.5 originates from the papers of Frankowska [Fr89b] and Berkovitz [Be89], but it contains several additional results and our presentation is different (see [B95] for a more concise account). Some related results are in [CF91, RV91, Z93, Te95] and in [ZYL97] for the stochastic case; see also [Mi93b] where different methods are used. The existence of optimal open-loop controls, which is the main assumption for Theorem 2.61 on the existence of optimal multivalued feedbacks, was studied in depth by many authors under a number of different assumptions; see, for example, [Ce83, Bat78, BPi95] and the references therein. For a discussion of the classical problem of the synthesis of optimal single-valued feedbacks and some recent advances towards its solution we refer, respectively, to [FR75] and to [Mi90a, Sus90, PSus96]. Subsection 3.2 is taken from [I84]. Even if the results of w allow the cost functional and the value function to be unbounded without restrictions on their growth, they do not cover the case of a linear system with unbounded control set; in particular, they exclude the classical LQ problem (linear system and quadratic cost, see, e.g., [FR75, Son90, BDDM93]). Incidentally, this is essentially the only problem where the classical DP method works, because one finds a smooth (quadratic) solution of the HJB equation by means of a Riceati equation, and the optimal feedback obtained from the argmin of the Hamiltonian turns out to be linear. The characterization of the value function as the unique locally Lipschitz viscosity solution for some class of nonlinear non-quadratic problems containing the LQ problem can be found in [Ben88], in [CDaP90] for infinite dimensional systems, and in [BDL96, I97] where comparison theorems are proved. Similar results for

6. B I B L I O G R A P H I C A L NOTES

225

the LQ problem with infinite horizon require more assumptions, because the HJB equation for this problem has two quadratic solutions; see [CDaP89, DL96]. The extensions to time-dependent Hamiltonians outlined in Exercises 3.5, 3.6, 3.7 can be carried further to treat the case of a merely measurable dependence on the time variable in the system f and therefore in H. This requires a generalization in the notion of viscosity solution, see [I85a, LP87, BJ87]. Semiconcavity results such as Theorem 3.26 in w and more general, can be proved either by working on the HJB equation, as in w in Chapter II and in [Kr60, Kr64, L82], or by working on the definition of value, as in [Kr67, CS87, CF9], Sin95]; our proof here follows the second strategy, but is different from the previous ones. The regularity Theorem 3.32 is based on the work of Subbotin and Subbotina [SS83]; see also [Ja77, Sua89]. Theorem 3.30 can be found in the book of Warga [Wa72] (see also [Ja77]) and Example 3.34 is in [Ja77]. The characterization of the differentiability of the value function in terms of uniqueness of the optimal trajectory is in Fleming [F69] for problems in the Calculus of Variations, where sharper smoothness results and more references can be found; we refer to Cannarsa and Frankowska [CF91] for some results on control problems. The classical reference for the Pontryagin Maximum Principle is the pioneering monograph [PBGM62], but some version of the P M P can be found in every book on optimal control and we do not attempt to survey the many different proofs available in the literature. The connection between the P M P and the Dynamic Programming method in the case of smooth value was pointed out in Chapter 1 of [PBGM62]; see also [FR75]. The version of the PMP in w Theorem 3.42, in addition to the classical statement that (3.79) is a necessary condition of optimality, provides the connection (3.81) between the costate and the value, as well as the sufficiency of these two conditions for optimality. This formulation is due to Subbotina [Sua89], and our proof 1 follows hers with some variants, whereas proof 3 is based on Zhou [Z90]. On the other hand, the first proof of the P M P by means of viscosity solutions is due to Barton and Jensen [BJ86]. Other results on the connection between the P M P and DP can be found in [Fr89b, CF91, Z93, Te95] in the framework of viscosity solutions and in [ClaV87, VW90b, Mi92a] where different tools of nonsmooth analysis are used. Remark 3.49 suggests that the results of w are related to the classical method of characteristics (see w For some recent developments on generalizations of this method for HJB equations see [Mi90a, Mi93a, Sua91, Su95] and the references therein. Our formulation of the monotone control problem in w is essentially the intlnite horizon version of the evolution problem considered in [Bn85], see also [BJ80]. However, the proof of the comparison Theorem 4.6 is different from the one in [Bn85] since no use is made of smoothing techniques. The optimal stopping problem in w can be seen as the prototype of optimal control problems whose value is characterized by a variational inequality. In the particular case where no control acts on the dynamics, this problem was analyzed by Dynamic Programming methods before the introduction of viscosity solutions (see, e.g., [Me80a, CDM81, Me82]). The synthesis procedure in this case,

226

III. PROBLEMS WITH UNRESTRICTEDSTATE SPACE

Theorem 4.12, is based on the knowledge of the free boundary, a typical feature of variational and quasivariational inequalities. HJ variational inequalities also arise in some questions in singular control, see [FH94]. A general reference for this type of problems in the control of nondegenerate diffusion processes is the book [BL82]. The material in w is taken from [Ba85a, Ba85b]. The impulse control problem was considered also in a more general setting in [MosT6, Me80b]. Impulsive problems of different nature are considered in [BJM93] and [MR96a]. The problem of minimizing the maximum cost gives rise to a quasivariational inequality which was studied by the viscosity solution method in IBIS9] and applied to the relaxation problem in [BJ95]. Let us also mention that some connections between the impulse and the monotone control problem with controls of bounded total variation are pointed out in [BJS0]. The optimal switching problem in w is perhaps the simplest problem in optimal control giving rise to a system of quasivariational inequalities. It was analyzed first by viscosity solutions in [CDE84], which we follow in our presentation; see also [E83]. The results of w were extended in various ways. We refer, for example, to [Y89] dealing with switching costs and usual as well as impulse controls, and [Le87] where the dynamics is piecewise deterministic; for the theory of systems of HJB equations see also [ELgl, I92b]. Let us mention, finally, that many results of this chapter were extended to the optimal control of piecewise deterministic processes and/or jump-Markov processes: in these cases the first order HJB equation involves some integrodifferential terms. This was done in the above-cited [Le87] and also [Le89] (for impulse control), [$88] and [Say91] for the finite and the infinite horizon problem, respectively. A more recent paper on integrodifferential equations is [A1T96].

CHAPTERIV

Optimal control problems with continuous value functions: restricted state space

In this chapter we continue the study of optimal control problems with continuous value functions and consider cost functionals involving the exit time from a given domain, in particular time-optimal control, and infinite horizon problems with constraints on the state variables. The continuity of the value function for these problems is not as easy as in the previous chapter. For time-optimal control this is essentially the problem of smallotime local controllability. We give the proof of just a few simple results on this topic, and state without proof several others. For each problem we characterize the value function as the unique viscosity solution of the appropriate Hamilton-Jacobi-Bellman equation and b o u n d a r y conditions. We do not give all the applications of this theory, as verification functions and conditions of optimality: most of them can be obtained by the arguments of Chapter III and are left as exercises for the reader. The system (S) and hypotheses (A0)-(A3) are the same as in Section 1 of Chapter III. We recall the basic estimates on the trajectories:

(0.1) (0.2) (0.3)

lyz(t, a) - x I <_ 21lzt

for all a E .4 and t E [0, 1/Mx],

lYx(t, ~)1 -< (Ixl + ~ - t ) e Kt

for all a E ,4 and t > 0,

lyx(t,a) - yz(t,c~)l <_ e L t l x - zl

for all a E . 4 and t > 0 .

1. S m a l l - t i m e

controllability

and minimal

time functions

Throughout the first four sections of this chapter we are given a target set T C R N satisfying (AT)

T is closed with compact b o u n d a r y 07-.

We study problems with initial state x in 7-c : = ]~N \ 7-, whose dynamics is stopped and the payoff computed when the system reaches either 7- or its interior int 7-.

228

IV. P R O B L E M S W I T H R E S T R I C T E D S T A T E S P A C E

Therefore we define, for a E .4, the exit time from T c, or entry time in 7-,

tx(a) :=

{+~c min{t : yx(t,a) E T }

if {t : yx(t,a) E T } = O, otherwise,

(where it is easy to show that the rain exists), and the exit time from T-~, or entry time in int T,

t'x(a) :=

{+~ inf{t:yz(t,a) E intT}

if {t : yx(t,a) E intT} = 0, otherwise.

In this section we discuss some continuity properties of the minimal time function

T(x) := inf tx(a),

x E ~N

aEA

and the minimal interior-time function $ ( x ) := inf ~x(~),

x E R~

aEA

Here we are extending the use of the term minimal time function to general targets. This name is commonly used in the literature for the case T = {0}, and in this special case T -- +co, because the interior of the target, int T, is empty. Note that the term minimal may seem inappropriate if a minimizing control does not exist. However, by standard results, a minimizing control for the first problem exists in .4 if f ( x , A) is convex, and it always exists among relaxed controls .4r. We introduce the notations n ( t ) :-~ { X E ~ N : T ( x ) < t },

t > O,

Tr := U n ( t ) = { x E ~(N : T(x) < +oc }, t>0

and 7~(t) and 7~ are defined by replacing T with T. The letter 7r stands for reachable: T~(t) is the set of points reachable from the target in time less then t by the backward system yr = _ f ( y , a), as well as the set of starting points from which the system can reach the target in time less than t. DEFINITION 1.1. The system (f, A) is small-time controllable on 7- (briefly STCT") if T C int T~(t) for all t > 0. If T = {0} this property is called small-time local controllability (STLC). The system is small-time controllable on int T, briefly o

STCT, if T c_ int ~ ( t ) for all t > 0.

<3

Note that STLC is equivalent to the continuity of T in 0 if "/- = {0}, since T(0) = 0 by definition. The next proposition extends this observation to the general case. Throughout the section we use the notation

d(x) := dist(x, T),

x

E ]I~.N .

1. SMALL-TIME CONTROLLABILITY

229

PROPOSITION 1.2. Assume (A0), (A1), (Aa), and (AT). Then the following state-

ments are equivalent: (i)

the system is S T C T ;

(ii)

T is continuous in x for all x E 07--;

(iii)

there exists 5 > 0 and WT : [0, 5] ~ [0, +oc[ such that lims_0+ WT(S) = 0 and T(x) <_wT(d(x) ) for all x E B(7-, 5). 1

PROOF. (i) =V (ii) For fixed ~ > 0 and x0 E 07-, since 7" C int 7~(z) there exists 5 such t h a t Ix - x o l < 5 implies x E 7~(~), t h a t is, T(x) < z. This gives the conclusion because T(x) >_ 0 and T(xo) = O. (ii) =v (iii) Define w(s) := sup{ T(x) : x such t h a t d(x) < s }, for s _> 0. We have to show t h a t lims~0+ w(s) = 0. We fix ~ > 0 and for any xo E 07- let (f(xo) > 0 be such t h a t T(x) < ~ for all x E B(xo,5(xo)). Since Off- is compact we can select xi E 07-, i = 1 . . . . . n, such t h a t uin__l B(xi, 15(xi)) D_ 07-. Now set ~ := mini=i,...,n 5(xi)/2 and observe t h a t d(x) < a implies x E X := Uin__lB(xi, 5(xi)). Therefore, by the construction of X , T(x) < ~ if d(x) < a, and then w(s) <_ s for s < a, which proves our claim. (iii) ~ (i) Fix t > 0 and let s > 0 be such t h a t WT(S) < t. T h e n T(x) < t if d(x) <_ rain{s, 5}, and so T~(t) _D B(7-, rain{s, 5}), which i m m e d i a t e l y gives S T C T . 9 REMARK 1.3. Under the a s s u m p t i o n s of P r o p o s i t i o n 1.2, T(x) > 0 if and only if x ~ 7-. In fact, by (0.1), for any a E .4

d(x) <_ lyx(tx(~), ~) - xl < Mxt~(~), and d(x) > 0 for x ~ 7- because 7- is closed.

<~

REMARK 1.4. P r o p o s i t i o n 1.2, with S T C T replaced by STCT-, holds for :T under the a d d i t i o n a l a s s u m p t i o n t h a t 7- is the closure of its interior, T = int 7-. In fact, the proofs t h a t (iii) ~ (i) ~ (ii) keep holding essentially unchanged, and the proof t h a t (ii) =~ (iii) holds because any x E 07- is the limit of a sequence in int T , thus the continuity of :F at such a point x implies T ( x ) = 0 because :F --- 0 in int 7-. Note t h a t the condition T = int 7- is necessary for STC~-. In fact if the distance c of x from i n t T is positive and less t h a n 1, by estimate (0.1) tz(a) >_ c/Mz for all a E . 4 , and then x ~ ~ ( t ) for any t <_ c/Mz. Now it is easy to remove the restriction c _< 1 and see t h a t no set strictly larger t h a n int 7- can be contained in 7~(t) for all t > 0. <~ REMARK 1.5. It is clear t h a t STC~- implies S T C T . The reverse implication does not hold in general, even if 7" = int T, as we see in the next example. In Ii(2 take 7 " - - { ( x l , x 2 ) : Ixll _< Ix21 or [Xll ___ 1} and the system f ( ( x i , x 2 ) , + ) -~ (1,0), f ( ( x l , x 2 ) , - ) = ( - 1 , 0), A = ( + , - } ( w e a d v i s e the reader to draw a picture). It is easy to c o m p u t e explicitly T and T and see t h a t T is discontinuous on

1B(X, 5) :=

( x E R N : d i s t ( x , X ) < 5}.

230

IV. PROBLEMSWITH RESTRICTEDSTATESPACE

X :-= { ( x l , 0 ) : Ix11 < 1/2}, in particular :F(0,0) -- 1 while limx~0T -- 0. Note x~X also that T is lipschitzean and :F = T off X. In Chapter V we see that if 07" is locally the graph of a Lipschitz function (a property failing only at the origin in the example), then STC~- and S T C T are equivalent.

<~

The next result lists some consequences of STC:r, which could be summarized by saying that T is a continuous function of ]~N into [O~--[-o(D], PROPOSITION 1.6. A s s u m e (A0), (A1), (A3), (AT), and STCT-. Then (i)

T~ is open;

(ii)

T is continuous in ~ ;

(iii)

lim T ( x ) = +oc for any xo E O k . X~X

0

PROOF. Fix z > 0 and 5 E 7r and choose ~ E ,4 such that t := t~(~) _< T(5) + s. By (0.3), if Ix - 5 1 <_ 5 e -n~, then (1.1)

]Yx(t,~) - y~(t.-5)] <_ Ix - 51 e Lf <_ 5,

and thus d ( y x ( t , 5 ) ) < 5 because y~(t,~) e 7-, where ~ is as in property (iii) of Proposition 1.2. Then (1.2)

T(x) < t + T(yx(t)) ___t + ~ T ( d ( ~ ( t ) ) ) ,

where the first inequality is a half Dynamic Programming Principle whose easy proof is left to the reader, and the second inequality follows from (iii) of Proposition 1.2. By (1.2), x E TO, which proves (i). Moreover, the choice of ~, (1.1), and (1.2) give

T(x) - T ( 5 ) < ~ ( I x

- 51 e L(r(~)+~)) + ~,

and by letting s ~ 0 + and exchanging the roles of x and 5 we get

(1.3)

IT(z)

-

T(5)I <_~ r ( l ~ - 51 eL(~(~)vr(~))),

which proves (ii). To prove (iii) we fix x0 E 0/~ and we suppose by contradiction there exist a constant M and a sequence Xn E 7~ such that [xn - x0] _< 1/n and T ( x n ) ~ M for all n. We choose ~ such that eL(M+l)/~ < 5 and ~ such that t := tx~ (-5) < T ( x n ) § 1 < M § 1. Then, by (0.3), ]yz0(t,~) -- yxn(t~'5)l < Ixo - x~] e Lf < 5, which implies d(y~o(t, ~)) < 5 and thus T ( x o ) < t + wT(5) by Proposition 1.2 (iii). This gives x0 E ~ , a contradiction to (i). 9 REMARK 1.7. Inequality (1.3) allows us to estimate the modulus of continuity of T by the uniform modulus at the points of OT, WT, on sets where T is bounded. For instance, WT(S) = C s ~ for some C > 0, 0 < -y < 1, implies the HSlder continuity of exponent 7 (Lipschitz continuity if "7 = 1) of T on compact subsets of T~. <~

1. SMALL-TIME CONTROLLABILITY

231

REMARK 1.8. Proposition 1.6 holds for T and 7~ if S T C T is replaced by STC~" and T = int T. The proof needs just a bit more care because t'x is defined as an inf instead of a min. < Next we give a brief account of the theory of controllability. First note that it is very easy to give examples where STC7. fails for any compact 7" because the system cannot go to sufficiently many directions (take, for instance, A a singleton and f constant ... ). In Chapter V we study the time-optimal control of noncontrollable systems. We begin by listing without proof some controllability conditions in the case 7" = {0}, which is the most deeply studied in the literature. We give in each case the corresponding regularity of the minimal time function. THEOREM 1.9. Assume 7" = {0} and the system is linear, i.e., f ( y , a) = M y + Qa,

A = [-1, 1]'n,

where M is an N x N matrix and Q an N x m matrix. Then STCT" holds if and only if the N x ( N m ) controllability matrix [Q M Q

M2Q

...

MN-1Q]

has full rank N. Moreover, T is locally HSlder continuous in Tt of exponent 1/(j + 1), where j is the least integer such that rank[Q

MQ

...

MJQ] = N .

THEOREM 1.10. Assume 7" ----{0} and consider the linear system f ( y , a) = M y + a. Assume A C ]~g is a compact convex neighborhood of the origin, and OA is a differentiable hypersurface (i.e., for each point of OA there is a unique supporting hyperplane for A). Then T 9 C 1 ( ~ "-, {0}). THEOREM 1.11. Assume 7"-- {0}, A -- [-1, 1]m, f 9 C 1, and f(0,0) = 0. If the linearized system M y + Qa, where

/'~ij =- ~ ( 0 , 0 ) ,

Qik = ~ k ( 0 , 0 ) ,

is STCT", then f is STCT. If, moreover, f 9 C ~176then T is locally HSlder continuous in Tr of exponent 1/(2j + 1 + e) for any E > O, where j is defined as in Theorem 1.9. THEOREM 1.12. Assume (Ao)-(A3) and T = {0}. If (1.4)

inf f(0, a) 93' < 0

for any unit vector %

aEA

then STCT. holds. Moreover, (1.4) is necessary and sufficient for the local Lipschitz continuity of T in T~.

232

IV. PROBLEMSWITH RESTRICTEDSTATESPACE

The proof of the sufficiency of (1.4) for S T C T and the Lipschitz continuity of T are postponed to Section 4. PROOF OF THE NECESSITY OF (1.4) FOR THE LIPSCHITZ CONTINUITY OF T. We assume by contradiction that there exists 7, 1"71= 1, such that (1.5)

f(0, a ) . ~ _ > 0

Let M := s u p { ] f ( z , a ) ] : t<_ 1 / M ,

for all a E A .

]z] <_ 2}, and note that for any Ix I < 1, a E A, and

ly~(t)l < 2, lyx(t)l < x + rio t f(x,e~(s)) ds + t w / ( M t , 2 ) , where we have used (A2) and (0.1). Then, by (A2) again, for any a E A, (1.6)

If(yx(t),e~)-f(O,~)l

<_wl(Ixl+Mt+o(t),2

)

as t - * 0 + .

Now pick ~ such that tx(~) _< T ( x ) + Ixl; then x

(1.7)

--

x -- - ~0 t (a) f ( y x ( t ) , "J(t)) dt,

and, by the Lipschitz continuity of T, (1.8)

tx(~) < (c + 1)Ix I .

Plug (1.6) into (1.7) and use (1.8) to get (1.9)

x = - f0 tx (~) f ( O , ~ ( t ) ) d t + t x ( - 5 ) o ( 1 )

as Ix[--*0.

Next we choose x = Ixl-y, take the scalar product of both sides of (1.9) with ~/and use (1.5) and (1.8) to get

Izl _< (c + 1)lxl o(1)

as Ixl --,

0,

which gives the desired contradiction.

~l

REMARK 1.13. Condition (1.4) is called the positive basis condition (see Exercise 1.1 for the motivation of the name) and it is equivalent to the existence of r > 0 such that ~6f(0, A) 2 B(0, r). <~ PROPOSITION 1.14. A s s u m e (Ao)-(A3) and 2r = {0}. Then a necessary condition for S T C T is (1.10)

inf f(O,a) .~/ <_ 0 aEA

for any unit vector ? .

1. S M A L L - T I M E C O N T R O L L A B I L I T Y

233

PROOF. The argument is essentially the same as in the previous proof. By contradiction we assume the existence of 3, and ~ > 0 such that

(1.11)

f(O,a).~>E

for all a E A .

By assumption T is continuous, thus T(x) <_ WT(IXl) by Proposition 1.2, and we can replace (1.8) by tx(~) = o(1)

as

Ixl

-~ o .

This is enough to get (1.9) from (1.6) and (1.7). By using (1.11) in (1.9) we get Ixl < t x ( ~ ) ( - z + o(1))

as Ix[ --* 0

and the right-hand side can be made negative for x :~ 0 small enough, a contradiction. 9 Before stating next result we recall the definition of the Lie bracket, or Jacobi bracket, of two vector fields gl, g2 E C1: [gl, g2] (x) = Dg2 (x)gl (x) - Dgl (x)g2 (x), where Dgi(x) is the Jacobian matrix of gi at the point x. We say (by induction t h a t a vector field F is generated by the vector fields Qi, i = 1 , . . . , m, by 0 bracket operations if it is a linear combination of the Qi, and by k bracket operations if F is a linear combination of brackets [G1, G2] with G1, G2 generated by k - 1 bracket operations. THEOREM 1.15. Assume T = {0} and the system is symmetric, i.e., n

f ( y , a ) = Q(y)a = Z a i Q i ( y ) ,

A = [-1,1] m,

i=1

where Q is an N x m matrix valued C ~ function and the vector fields Qi are its columns. If there exist N vector fields F 1 , . . . , FN generated by Qi, i = 1 . . . . . m, by bracket operations and such that s p a n { / 1 ( 0 ) . . . . . FN(O) } : ~ N ,

then S T C T holds. Moreover, T is locally H61der continuous with exponent 1/ (j + 1) where j is the minimum number of bracket operations necessary to generate all of the fields F 1 , . . . , FN. Now we prove some results for the case of a "smooth target". The smoothness of T we need is expressed in terms of the distance function from T, d( 9 ), as follows: (1.12)

d(- ) can be redefined in int T as a C 1 function on B(OT, c) for some c > 0 .

234

IV. P R O B L E M S W I T H R E S T R I C T E D

STATE SPACE

This holds for instance if 7- = int T (which is a necessary condition for (1.12)) and 0q- is a C 2 manifold: it is enough to redefine d(x) in int 7- as - d i s t ( x , OT) (this new function d( 9 ) is the signed distance from cgT). See also Remarks 1.20 and 1.21 for some generalizations. Note that under assumption (1.12) there exists an exterior normal n(x) = Dd(x) for all x E 0:1-. The first sufficient condition for controllability we consider is a very natural and simple one, though rather restrictive: (1.13)

inf f ( x , a). n(x) < 0

for all x E OT,

aEA

t h a t is, at each point of the boundary of 7- the system can choose a direction pointing to the interior of 7-. Note the similarity between (1.13) and (1.4). THEOREM 1.16. Assume (A0)-(A3), (AT), and (1.12). Then (1.13) implies S T C T

and STC~r. Moreover, (1.13) is necessary and sufficient for the local Lipschitz continuity of T in 7r PROOF. We first show that there exist constants C, ~ > 0 such that (1.14)

T(x) <_ T(x) < C d(x)

for all x 9 B ( T , (~) \ int q-.

o

This implies STCT, STC:r, and the local Lipschitz continuity of T by Proposition 1.2 (with WT(S) = Cs) and Remarks 1.4, 1.7 and 1.8. We fix x0 9 0q- and use (1.12) and (1.13) to choose a0 9 A, ~0 > 0, 0 < r0 _< C such t h a t (1.15)

f ( x , ao)" Dd(x) < -~0

for all x 9 B(xo, ro) .

Now let a0(s) = a0 for all s and y(s) := yx(s, a0). In the following we restrict s to be less than 1, so y(s) remains in a bounded set independent of the choice of x in B(OT, C), by (0.2) and (AT). Thus we have (1.16)

ly(s) - x] < M s

for a l l x E B ( 0 T ,

C), f o r 0 < s <

1.

By the Taylor expansion for d( 9 ) and (1.16) we get d(y(s)) = d(x) + (1.17)

os f(y(s), ao) ds. Dd(x) + o(Ms)

-- d(x) + s f ( x , ao). Dd(x) + o(cls) < d(x)

-

l~os

for 0 < s <_~,

where in the second equality we have used (A2) and cl is independent of x, while in the inequality we have used (1.15) and 0 < g < 1 is independent of x. Now we set Sz := 2d(x)/~o, we note that sx _< ~ if d(x) <_~ 0 / 2 , and then apply (1.17) to get d(y(sz)) < 0; this implies t h a t y(Sx) c i n t 7-, thus

T(x) < :Y(x) < s~ = C d(x),

for all x E B(xo, Ro) \ int q-,

1. SMALL-TIME C O N T R O L L A B I L I T Y

235

where C = 2/~0 and R0 = min{r0,~0/2}. The compactness of 07" allows us to complete the proof of (1.14) by choosing a finite number of balls B ( x i , R i / 2 ) covering 07. and setting (f = mini=0,_.,n Ri/2, as in the proof of Proposition 1.2. The proof that (1.13) is necessary for the Lipschitz continuity is similar to the corresponding proof in Theorem 1.12. We assume by contradiction there exists x0 where (1.13) is violated and we choose new coordinates such that x0 = 0, so we have (1.5) with "7 = n(xo). As before we seek a contradiction by looking at a sequence of points x,~ = "7/n. Let an be such that t x , ( a n ) < T ( x n ) + Ixnl so that (1.18)

xn +

f

t.~(,~.)

f ( Y x , (t), a,~(t)) dt =: Yn E 0 7 . .

JO

We use (1.6) and (1.8), take the scalar product with "7 and use (1.5) to get 1 - = Xn ""7 < Yn'"7 + o(1/n) . n

(1.19) Now we estimate [y.i<_iyn

x.I+ixnl<_Mtx.(an)+

1 n

< (MC+M+

1) 1 n

where M is a local bound on Ifl and C a local Lipschitz constant for T. Then Yn ""7 = o(1/n) because Yn E 07. and "7 = n(0) is orthogonal to 07. at 0. Thus (1.19) gives a contradiction. The next result is the counterpart of Proposition 1.14 for smooth targets. Its proof is very similar and it is left to the reader. PROPOSITION 1.17. Assume (A0)-(Aa), (AT), and (1.12). Then a necessary condition f o r STCT. is (1.20)

inf f ( x , a ) . n(x) <_ 0

f o r all x E 0 7 . .

aEA

Now we give another sufficient condition for controllability which is less restrictive than (1.13). This is similar to the hypotheses of Theorem 1.15, in particular we assume the system is symmetric. THEOREM 1.18. Assume (A0)-(A3), (AT), and (1.12). A s s u m e that, in a neighborhood of any point x E 07. where (1.13) fails, the system is symmetric and there is a vector field F generated by Qi, i = 1 , . . . , m , by bracket operations such that (1.21)

F ( x ) . n ( x ) # O.

Then STCT. and STC~r hold. If, moreover, the number of bracket operations necessary to generate F can be bounded by j f o r all x E 07., then T is locally HSlder continuous in T~ with exponent 1 / ( j + 1).

For the proof we need the following result of geometric control theory.

236

IV. P R O B L E M S W I T H R E S T R I C T E D STATE SPACE

LEMMA 1.19. If the system is symmetric and F is generated by j bracket operations by Qi, i = 1 , . . . , m, then there is C > 0 and, for each s > O, a piecewise constant

control c~ E P such that (1.22)

y z ( s , o~) = x ~- C F ( x ) 8j+l -~- 0(8j+l)

as S --~ 0 +

.

PROOF OF THE LEMMA IN THE CASE j = 1. It is not restrictive to assume F = [Qi, Qk]. Denote a (i) E A the vector whose c o m p o n e n t s are all 0 except the i-th. T h e n set

a(O a(t) =

O
a (k) _a(i )

s/4
- a (k)

3s/4 < t <_ s .

A straightforward b u t tedious c o m p u t a t i o n using the Taylor expansions of the trajectory gives S2

y~(s,a)=x+T~[Qi,Qk](x)+o(s2 )

ass-~0 +.

4

PROOF OF THEOREM 1.18. W i t h the ]emma available the proof is very similar to that of Theorem 1.16. Given x0 E c9~, by (1.21) and the symmetry of the system, we can find ~0 > 0, 0 < r0 _< c such that

F ( x ) . Dd(x) < - ~ 0

for all x e B(xo, ro) 9

T h e n we combine the Taylor expansion of d( 9 ) with (1.22) to get

d(y~(s, a)) = d(x) + C sJ+lF(x) 9Dd(x) + o(s j+l) < d(x) - C~o s j+l

for 0 < s < ~,

which is the c o u n t e r p a r t of (1.17). Now we choose

(2d(x)

Sx

1

:= \ C~0 ]

a n d get

T ( x ) <_ ~ ( x ) < s~ = C ' d l / ( J + l ) ( x ) . The conclusion is now achieved as in T h e o r e m 1.16; we leave the details to the reader. .q REMARK 1.20. Theorems 1.16 a n d 1.18 can be extended to targets with piecewise s m o o t h b o u n d a r y in the following sense: 2r = {x : gi(x) <_ 0 for all i = 1 . . . . , M } , where gi C CI(]R N) a n d IDgi(x)l > 0 for all x such t h a t gi(x) = 0. It is enough to modify (1.13) a n d (1.21) by replacing n(x) with Dgi(x) for all the "active constraints", t h a t is, for all i such that gi (x) -- 0. The proof is essentially the same with d replaced by gi for all i active at the given point x0 E cgq-. In Section 4 we extend T h e o r e m 1.16 to targets whose b o u n d a r y has no smoothness properties. <~

1. SMALL-TIME CONTROLLABILITY

237

REMARK 1.21. Most of the preceding controllability results hold if measurable controls are replaced by piecewise constant controls and T, T are replaced by T ~ := inf~e-p tx(C~), T~ := inf~ept'x(~ ). This is the case, for instance, of Theorems 1.9, 1.12, 1.15, 1.16, and 1.18. <3 The most restrictive assumption of this section is (A3). In fact the minimum time problem makes perfect sense without knowing a priori that the trajectories of the system exist for a given time, so (A3) can be replaced by (1.23)

( f ( x , a ) - f ( y , a ) ) . ( x - y) < L R ] x - yI2

for all a E A, ]xl, lYl <- R (which is satisfied, for instance, by any f of class C 1 if A is compact) because (1.23) is enough to ensure local existence and uniqueness of the trajectories (see Remark 5.7 in the appendix of Chapter III). Most of the results of this section are of local nature and keep holding under this assumption. However, in this case S T C T does not imply the continuity of T in all of ~ (cfr. Proposition 1.6 (ii)) as shown by the example in Exercise 1.3. Nonetheless the continuity of T propagates in the so-called escape controllability domain which we define next. We restrict ourselves for simplicity to the case S E C1(1~ N • ~M),

(1.24)

A C_ R M compact,

T = {0} .

Let T~(< t) :~- { x E ]~N : ~:x(O~) < t for some c~ E ,4 } .

DEFINITION 1.22. The escape time is To := sup{ t : 7~(_< t) is bounded } . The escape controllability domain is 7~0 := 7~(T0) = { x : T ( x ) < To

}.

It is easy to prove that

/

+~z To = liminfT(x) I. I x l ~

if "R is bounded, otherwise. <3

The next result, whose proof we omit, is an analogue of Proposition 1.6. THEOREM 1.23. Assume (1.24) and STCT. Then (i)

T~o is open;

(ii)

T is continuous in T~o;

(iii)

nolirnxo T ( x ) = To for any xo E OTto;

(iv)

lim T ( x ) = To. Ixl~ xET~o

See Exercises 1.3, 1.4, 1.5, and Example 2.10 for some explicit examples of "R0.

238

IV. P R O B L E M S W I T H R E S T R I C T E D STATE SPACE

Exercises

1.1. A set of vectors qi E N N, i = 1,... ,m, is called a positive basis if for any p E ~N there are )~i _~ 0, i = 1,.. m, such that p = ~-~i=1 iqi. Prove that condition (1.4) holds if and only if f(0, A) contains a positive basis. Prove that another equivalent condition is ~6f(0, A) _DB(0, r) for some r > 0. (See [Pe68].) 1.2. Assume (A0)-(A3), T closed and unbounded, (1.12) and (1.13). Prove that T and T are locally Lipschitz continuous, ~ is open, and T ( x ) ~ +oc as x ---* xo E OT~.

1.3. Consider the system in l~2 5

y' = ~_. aifi(y),

aEA,

i=1

where

fl(Y) =(-y2(1 -y~),yl(l+y~)), I2(Y)= (-1,0), f3(y)=(-(y2 - 8)(y1+1/2),0),

/4(Y)=(0,1), f5(Y)=(O,-1), and A is the convex closure of the canonical basis of R 5. Take T = {0}. Prove that (i)

the trajectories corresponding to the control (1,0,0,0,0) contained in the strip S = {(Xl, x2): Ix1 [ < 1} are all the ellipses x~ + cx~ = 1 - c for c E ]0, 1[, and the time taken to turn around each ellipse is 2~r;

(ii)

S C_ ~ (_< 2rr + 1) (then the escape time To _< 2~r + 1);

(iii) if y(_l,0) (t, a) e 8 - := {(Xl,X2) :x2 _< 8} for t < to, then y(0,1)(t,a) ~ S for t < to; (iv) if y(_l,0)(t,a ) C S + := {(xl,x2) : x2 _> 8}, then t > 8; (v) (vi)

t(_l,O)(a ) > 8 for all a C .A; lim

S3x--(--1,0)

T ( x ) _< 2zr + 1 < 8 <_ T ( - 1 . 0 ) (then T is discontinuous at ( - 1 , 0)).

1.4. Assume (A0), (A3), T = {0} and [ f ( x , a ) l < k ( l + lx[)

Prove that

for all x e R N , a E A .

lim T ( x ) = + o c and then the escape time To = +oc and ~0 = ~ .

1.5. Let N = 1, h(x) > 0 be a smooth function such that h(x) - 0 for x _< 0 and h(x) =- 1 for x _> 1. Consider the system f ( x , +) = - x ~ h ( x ) , f ( x , - ) = 1, A = { + , - } , and the target T = {1}. Prove that

2. HJB EQUATIONSFOR THE MINIMALTIME FUNCTION (i)

239

T(x) = ( x - 1)/x for x _> 1, T(x) = 1 - x for x _< 1, so t h a t To = 1, 7~0 = lo, +oo[, 7~ = ~;

(ii) the system satisfies (Ao)-(A3). 2. H J B e q u a t i o n s for the minimal

and boundary value problems time function: basic theory

In this section we formulate the Dynamic Programming Principle for the minimum time problem and derive a b o u n d a r y value problem for the Hamilton-JacobiBellman equation associated with the minimal time function

T(x) := inf tx(a), c~cA where t~(c~) is the first time the system hits a given closed target T. This boundary value problem has infinite d a t a on 07r where ~ is the set of starting points from which the system can reach the target. We also introduce also a rescaling of T which solves another b o u n d a r y value problem where the (unknown) set ~ does not appear. We use the notation 1

Xx(X)=

ifxEX,

0 ifx~X.

PROPOSITION 2.1 (DYNAMIC PROGRAMMING PRINCIPLE). Assume (A0), (A1), (A3). Then, for all x E T~, (2.1)

T(x) = inf { t A tz(a) + X{t<_t,(a)}T(y~(t,a)) }, c~EA

for all t > 0

and

(2.2)

T(x) = inf { t + T(y~(t, c~)) }, c,cA

for all t C [0, T(x)] .

PROOF. Let B and C denote, respectively, the right-hand sides of (2.1) and (2.2). Note that (2.1) reduces to the definition of T(x) for t > T(x). For t <_ T(x) we have B < C because the quantity in brackets in (2.1) is less than or equal to the quantity in brackets in (2.2). To show that T(x) > C we observe that for all (~ E .A and t < T(x) < t~(a), by setting ~(s) = c~(t + s), we have t~(a) = t + tyx(t,~)(~ ) >_ t + T(y~(t, ~)) > C, and we reach the conclusion by taking the inf over .4. To show that T(x) < B, we fix ~ E A, t <_ T(x), set z : = yx(t, a), and fix s > 0 and a 1 C A such t h a t T(z) > t~(a 1) - ~. We define [a(s) = ~'LozX(s- t)

s _< t, s :> t .

240

IV. PROBLEMS WITH RESTRICTED STATE SPACE

Since t _< tx(~) we have

T ( x ) <_ tx(-a) = t + tyx(t,o)(a 1) <_ t + T(yx(t, a)) + r = t A tx(a) + X{t<_tx(a)}T(yz(y, c~)) + z because t <_ tx(a). We conclude the proof by taking the inf over .4 and then letting z go to 0. 9 REMARK 2.2. The Dynamic Programming Principle for the minimal interior time function :F corresponding to the target int T is given by the same formulae (2.1) and (2.2), where T is replaced by T and t~ by t'x. The proof is exactly the same provided one checks that for t < t'~(a) = t +

9 + t)).

We also note that in (2.1) we can replace X{t_
t = tz(a~ y~(t,a) E T and T - 0 on T. The same statement holds for T and t'~, even if T may be positive on 07", because T ( y ~ ( t ' ( a ) , a ) ) = 0 for all x and a, as the reader can easily check. <~ PROPOSITION 2.3. A s s u m e (A0)-(A3), (AT), ~ \ T open, and T E C(7~ \ 7") (e.g., STCT" holds). Then T is a viscosity solution of (2.3)

sup{ - f ( x , a). D T } - 1 = 0

in 7~ \ 7".

aEA

PROOF. Let p E C 1(T~ \ 7") and let x be a local maximum point of T - p. Fix a E A and let yx(t) be the trajectory corresponding to the constant control function c~(t) _= a. Then for t > 0 small enough

p ( x ) - p(yx(t) ) <_ T ( x ) - T(y~(t) ), and by the D P P (2.2) (note that T ( x ) > 0 by Remark 1.3) -

t

_< 1

By letting t --* 0 + we get

-D~(x).

f ( x , a ) <_ 1.

We can take now the sup over A in the left-hand side, by the arbitrariness of a, and conclude that T is a subsolution of (2.3). Next assume x is a local minimum point of T - r By (0.1) there is to such that ~(x) - ~(yx(t, a)) ~ T ( x ) - T(yx(t, a)),

for all t E [0, to], a E . 4 .

For each r > 0 and t E ]0, T(x)], by the D P P (2.2), there is ~ E .A and corresponding trajectory ~ such that

T ( x ) >_ t + T(-~x(t)) - st,

2. H J B E Q U A T I O N S F O R T H E MINIMAL TIME F U N C T I O N

241

and then ~a(x)

~(Yx(t)) > 1 - ~. t

As in the proof of Proposition III.2.8, (A2) and (0.1) imply ~(x) - ~(~x(t)) = -

/0'

D ~ ( x ) - f ( x , ~(s)) ds + o(t),

as t --* 0 +

and thus sup{ - D w ( x ) . f ( x , a) } > - 7 aEA

>_ 1

l forD ~ ( x ) . f(x,-~(s)) ds

-

~ + o(1)

as t ~

0+ .

After letting t and c go to 0 we obtain that T is a viscosity supersolution of (2.3). 9 REMARK 2.4. Under the assumptions (A0)-(A3), (AT), 7~ -. T open and T C(7~ \ T), we have that T is a viscosity solution of (2.3) in 7~ \ T, and the proof is the same of Proposition 2.3 by means of Remark 2.2. <~ EXAMPLE 2.5. If f ( x , a) = a, A is the closed unit ball, then it is easy to prove that T ( x ) = dist(x, T ) and in this case the HJB equation (2.3) becomes the eikonal equation IDT] - 1 = 0 in T c (cfr. Corollary II.2.16). < Now we want to characterize T as the unique solution of the HJB equation (2.3) with suitable b o u n d a r y conditions, namely,

(2.4)

T = 0

T ( x ) ----* +oc

on OT,

as x ---, xo E O n .

Let us first give a simple result where the set ~ is supposed to be known. This is reasonable because the determination of T~ is a controllability problem: one has to find all the points from which there is a trajectory reaching T. This can be considered as a preliminary investigation before looking for time-optimal trajectories. However, this is not completely satisfactory because ~ is not a d a t u m of the problem. Later in this section we make a sort of resealing of the cost functional leading to another b o u n d a r y value problem whose solution simultaneously gives both T and 7~. In Section 4 we also show t h a t 0T~ is a free b o u n d a r y for the problem (2.3)-(2.4). THEOREM 2.6. Assume (A0)-(A3), (AT), and STCT-. Then T is the unique viscosity solution of (2.3) continuous in 7-4, bounded below and satisfying (2.4). PROOF. Let Ul and u2 be, respectively, a subsolution and a supersolution of (2.3). Define

(2.5)

v~(z)

= 1 - e -u'(x~,

i = 1, 2 .

242

IV. PROBLEMS WITH RESTRICTED STATE SPACE

By Proposition I1.2.5 vl and v2 are, respectively, a sub- and a supersolution of (2.6)

v + sup{-f(x,a). Dv} - 1 = 0

in 7~ \ T .

acA

Moreover, vi are bounded if ui are bounded below and, by the boundary condition (2.4) on cgT~, vi can be extended in a unique way to vi E B C ( 7 r \ T ) satisfying the boundary conditions (2.7)

vi = 0

on O T ,

vi = 1

on 07~.

Then we can apply the comparison results of Chapter III, more precisely Remark III.2.14, to get that vl _< v2 and therefore uj < u2. We obtain the conclusion by exchanging the roles of Ul and u2. EXAMPLE 2.7. In this example we apply Theorem 2.6 to compute the minimal time flmction for the rocket railroad car. The system is ~ = a E [-1, 1] = A, and the target is to reach 0 with 0 speed. Then N = 2, f ( ( y l , Y 2 ) , a ) = (y2, a), 7" = {(0,0)} and the HJB equation (2.3) is (2.8)

]Tx2] - x 2 T x l - 1 = O .

The trajectories corresponding to the constant controls - 1 and +1 are, respec1 2 + constant and Yl = ~Y2 1 2 + constant. The initial tively, the parabolas Yl = -~Y2 points from where the system can reach 7- with no switchings are 1 2

r _ :~-~ { ( X l , X 2 ) : 321 ---- - - ~ X 2 ,

12 r + := { (:cl, z2) : z l = ~x2, z2

X 2 ___~0 } ,

_< 0 } ,

and to minimize time we must choose a = - 1 on F_ and a = +1 on F+. Define

R_ := { (xl,x2): xl >

-89

R-p : = { ( X l , X 2 ) : X 1 <

--lx2[x21 }.

},

It is easy to see that starting in R_ (respectively, in R+) and using the control a = - 1 (respectively, a = +1) the system reaches F+ (respectively, F_) in finite time. Then 77. = R 2 and a single switching is enough to reach T. We claim that the feedback map (see w (I)(x) = ~ - 1 ( +1

in R_ t 2 r _ , in R+ tJF+,

is optimal (we advise the reader to draw a picture of the trajectories corresponding to tiP). In fact it is clearly admissible, and a straightforward computation shows that the time taken by the system to reach 7" by using 9 is

-(x)=

{ x2+22. --X 2-}-

1 V2~ 2 --X 1 ~

X E R+ 9

2. HJB EQUATIONS FOR T H E MINIMAL TIME F U N C T I O N

243

By construction, u :> T. Next we show t h a t u is a subsolution of (2.3) in 7 -c, so t h a t u < T by the proof of Theorem 2.6 and then T = u (actually it is not hard to check directly that u is a supersolution, see Exercise 2.4). It is easy to check t h a t u is a classical solution for x ~ F := F_ U F+. As in Proposition II.2.9 we compute the superdifferential of u at x C F by decomposing its elements in the components tangential to F, P T D u ( x ) , and orthogonal to F. The normal unit vector to F pointing inside R+ is n(x) := ~)(-1,-ix21),

Q := (1 + x~) -]/2 .

If we denote u_ := ul~_, u+ := Ul.~+ we have that Du_(x).n(x)

= f-oc t y/x2

for x e F _ , forxEF+,

SQ/x2

for x e F _ ,

(+~

for x C F+,

Du+(x) . n(x)

Then D + u ( x ) = ~ and so u is a viscosity subsolution of (2.8).

<~

The exponential transformation (2.5), also named the KraSkov transform, simultaneously reached two goals in the proof of Theorem 2.6: transforming the original HJB equation (2.3) into the P D E (2.6) with the good structure to apply the comparison results, and changing the singular boundary condition on cOT~(2.4) into a s t a n d a r d one (2.7). Next we push the consequences of this transformation further by observing that it amounts to considering the following payoff

J(x, a)

:=

frot~(a) e -t dt = 1 - e -tx(~),

which is a rescaled and discounted version of the original one tx(a). The value function associated with the new payoff is (2.9)

v ( x ) : = inf J ( x , a ) - ~A

~! -e-T(x) ll

ifxc~, if x it 74,

which satisfies 0 < v(x) <_ 1 for all x E t~g . Note that, once v is known, both T and 7~ are immediately recovered by the formulae T ( x ) = - l o g ( 1 - v(x)),

7r = { x : v(x) < 1 } .

The HJB equation for v is given by the following LEMMA 2.8. A s s u m e (A0)-(A3), (AT), and v C BC(]~N). solution of (2.10)

v +sup{-f(x,a). aEA

Dv} - 1 = 0

Then v is a viscosity

in T c : = ]Ry \ :7".

244

IV. PROBLEMSWITH RESTRICTEDSTATESPACE

PROOF 1. The Dynamic Programming Principle for this problem is

v(x)

r ftAt~(~)

= inf

~I

e-s ds + X{t
c~E.A ~ J o

)

for all t > 0. The proof is essentially the same as those of Propositions III.2.5 and 2.1. The proof of (2.10) is then the same as those of Propositions III.2.8 and 2.3.

9

PROOF 2. In the interior of T~ r v -- i, so v is a classical solution of (2.10). Moreover, v is a solution in 7~ \ T by Propositions 2.3 and II.2.5. Finally, let x 6 0~ and observe that yx(t, ~) E T~ c for all ~ and all t, so v(x) - v(yx(t)) - 0 for all t and (~, in particular for a( - ) -- a. If r E C I and x is a maximum point for v - ~b we get &(x) - r _< 0, then -Dr f(x, a) <_ 0 for all a E A, and this says that

v is a subsolution of (2.10) because v(x) = 1. Similarly, if x is minimum point for v - r we have r - r >_0, - D e - f ( x , a ) > 0 for all a C A, and then v is a supersolution of (2.10). 9 As in Chapter III we want to compare v with any subsolution Ul and supersolution u2 in BC(~ N) of the boundary value problem

{~+sup{-f(x,a)'Du} -1=0 (2.11)

acA = 0

inTc, on

Of.

By this we mean that ul, us are, respectively, a sub- and a supersolution of the HJB equation such that Ul _< 0 on cgT, u2 >_0 on OT. We say that v is a complete solution of (2.11) in BC(R N) if it is the maximal subsolution and the minimal supersolution in BC(]~N) (in particular, it is the unique solution in this set).

Assume (A0)-(Aa), (AT), and S T C T . solution of (2.11) in BC(R:v).

THEOREM 2.9.

Then v is the complete

PROOF. By (2.9) and Proposition 1.6, v E Bc(RN). By Lemma 2.8 v is a solution of (2.11). The Hamiltonian in (2.11) is a special case of (III.2.9) ($ -- 1) and we get the conclusion by Remark III.2.14. 9 As in Chapter III we can draw several consequences from this theorem: verification results such as Corollary III.2.18, where sub- and supersolutions of (2.11) are, respectively, verification and falsification functions; the equality v = v ~ (: indicates the value function associated with piecewise constant controls) if the system is also small-time controllable to T by means of piecewise constant controls (see Remark 1.21), and then T = T ~ and T~ = 7~:; the equality v = ~ under the additional assumption STC~r, and then T = T and 7~ = ~ ; relaxation and perturbation results. We leave the precise statements as exercises for the reader. Many of these results can be improved by the theory of discontinuous viscosity solutions we present in next chapter and we give some explicit applications there. For instance, in w we show that, for targets with Lipschitz boundary, the property

2. HJB EQUATIONS FOR THE MINIMAL TIME FUNCTION

245

of small-time controllability STC7- is enough to prove that T = T ~ and T = T, and therefore it is equivalent to STC~r for such targets (cfr. Remark 1.5). Let us make some comments on the assumptions of Theorems 2.6 and 2.9. The boundedness of 07- in (AT) was assumed to simplify the study of STCT- in Section 1 and it plays no role in the comparison theory. Indeed Theorems 2.6 and 2.9 hold for unbounded 7-, with STCT- replaced by the assumption that 74 is open and T is continuous in T4. The uniqueness assertion in Theorem 2.6 is false if extended to solutions unbounded below. Here is the counterexample: f(x, a) = a, A is the closed unit ball in I~N~ 7- = {0}; the b o u n d a r y value problem IDul - 1 -- o ~(0)

=

in ~ N \ { 0 }

o

has the classical solutions T(x) = Ixl and - I x l . Sections 3.2 and 5.1 of Chapter V are devoted to extending the theory to non-controllable systems, t h a t is, to the case when STCT- fails. We have seen at the end of Section 1 t h a t if we relax (A3) to a local condition such as (1.23) or (1.24) and STCT- holds, the minimum time function T is not necessarily continuous in all 74, but it is continuous in the escape controllability domain 74o- By Theorem 1.23, under the assumptions (1.24) and STCT, T is a viscosity solution of the boundary value problem

~

ma~{ - f ( x , a ) . D T } - 1 = 0

(2.12)

[ T ( 0 ) = 0,

in T~o \ {0},

ralimxoT(x ) = To if x0 e 0740,

where To is the escape time. If (Aa) holds, it is easy to see that T is the unique solution continuous and bounded below of (2.12) (note that it may be 740 C 74 even if (A3) holds, see Exercise 1.5) but this is not the most interesting case. If we drop (A3), uniqueness may fail as the following example shows. EXAMPLE 2.10. Let N = 1, f(x~ a) = x2a, A = { - 1 , +1}, 7- = {1}. It is easy to check t h a t 74 = ]0,+oo[, T(x) = 1 / x - 1 for x < 1, T(x) = 1 - 1/x for x > 1, and so To = 1 and 740 = ]1/2, +oc[. A second solution, continuous and bounded below, of

~x2[u'l = 1, (2.13)

[ u ( 1 ) = 0,

in 740 \ {1}, u(1/2) = 1,

is u(x) = 1/x - 1. Note that both solutions are classical.

<~

However, Theorem 1.23 also gives the following information on the behavior of T at infinity: (2.14)

lira T(x) = To. xETr

The last result of this section says that if we add this b o u n d a r y condition at infinity to the b o u n d a r y value problem (2.12), then T is the unique solution.

246

IV. P R O B L E M S W I T H R E S T R I C T E D S T A T E S P A C E

THEOREM 2.11. Assume (1.24) and S T C T . (2.12) and (2.14) that is continuous in 740.

Then T is the unique solution of

PROOF. Let u E C(740) be another solution of (2.12). Extend T and u by continuity to 74o and take their exponential transformations y := 1

-

e -T,

w :=

1 -

e -u

.

By Proposition II.2.5 they solve equation (2.6) in 740 "- T. We apply the comparison principle of Exercise II.3.6 to the bounded open set f~r := { x E ~ 0 : 0 < Ix[ < r } and get (2.15)

m_ax(v - w) + = max(v - w) + =

max

(v - w) + .

Now we let r ~ ~ so that the sets f~ invade T~0. Since T and u satisfy (2.14), the right-hand side of (2.15) tends to 0, and we obtain v < w in 7~0. The proof is completed by reversing the roles of v and w. ~1 The main drawback of this result is that it characterizes T only once 7~0 is known, even if it is not a d a t u m of the problem. In Section 4 we give a uniqueness result where ~ 0 appears as an unknown, so that 0740 can be considered as a free boundary. The last remark is that one might suspect that T can be characterized as the unique solution of (2.12) by adding the natural sign condition, T(x) > 0 for x # 0, instead of (2.14). The next example shows that this is not the case. EXAMPLE 2.12. Consider the system and target of Example 2.10 and define

~T(x), if x < 1, u),(x) = l A - I T ( x ) - AI, if x > l . It is easy to check that, for all A > 0, ux is a solution of (2.13), and, for 1/2 < A < 1, u~(x) > 0 for x # 1 and u:~(x) < T(x) for x > 1/(1 - A). <3 Exercises 2.1. In the assumptions of Proposition 2.1 let a function S : I~ N ---4 [0, -l-oc] satisfy the D P P at a given point x E 74, that is S(x) = inf{t A t~(~) + x{t<_txt~)}S(y~(t, a))},

for all t k 0 .

Prove that S(x) = T(x).

2.2. Assume (A0)-(A3), 74 open, and T continuous in 74. Show that T is a solution of (2.3) in int T and that at any point of 07- it satisfies the definition of subsolution and does not satisfy the definition of supersolution. [Hint: look at D+T(x) and D - T ( x ) by using Remark 1.3.]

3. PROBLEMS WITH NON-ZERO TERMINAL COST

247

2.3. In the assumptions of Remark 2.4 let x 9 OT a point where 5~(x) > 0 but ~T~\int'/" is continuous. Show that if ~b 9 CI(IR N) and x is a maximum point for ( ? 2 - r then sup~{-f(x,a). De(x)} _< 1. This remark suggests an interpretation of the Dirichlet boundary condition in a weak sense which is studied in Section 4 of Chapter V. 2.4. In the setting of Example 2.7 show that

D-u(x) = { PTDu(x) + ;~u(x) : )t E A(x) }, ~]-~z,~/x2] if x 9 F_, A(x) := [ [ O / x 2 , + o c [

PTDu(x) = ~ ~2(-x2' 1) [p2(-x2,-1)

i f x 9 F+, if x 9 F_, if x 9 F+,

and check that u is a supersolution of (2.8). 2.5. Assume (A0)-(Aa) and let T and Tp be, respectively, the minimal time functions associated with the targets {0} and {p}, for a given p r 0. Assume small-time controllability on both targets and the existence of trajectories joining 0 with p and p with 0, so that the reachable sets for the two targets coincide:

7r := {x: T(x) < + ~ } = {x: Tp(~) < + ~ } . Does

u(x) := Tp(x) - Tp(O) give a counterexample to Theorem 2.6? 3. P r o b l e m s

with exit times and non-zero terminal

cost

In this and the next section the cost to minimize is

JJo f r t ~ ( ae(Yx(S)'a(s))e-'~Sds ) +e-~tx(a)g(yx(tx(a))),

iftx(ct) < +oc,

J ( x , c,) := ~ r

[J0

f(Yx(S)'~

ds,

if

tx(a)= +oc,

where tx(cr) is the first time the trajectory Yz(",a) hits a given target 27-, as in Sections 1 and 2; the running cost g and the constant interest rate A satisfy

~g : R N x A --~ R is continuous; there are a modulus wt and a constant M such that (3.1)

~It.(x,a)-~(y,a)l < wt(ix-y]) and 1 < ~(x,a) <__hi for I a l l x , y E IRN, a E A; I,A > o;

the terminal cost g satisfies (3.2)

gE

C(T) and g(x) > 0 for all z E T .

248

IV. PROBLEMS WITH RESTRICTED STATE SPACE

First of all we remark that, if A > 0, the assumptions g _> 1 and g > 0 are not more restrictive than the boundedness of ~ and g from below. In fact, if they are not satisfied we define C := max{ 1 -

inf f , - A i n f g } , ~N • A

0q-

~-= ~ + C, ~ = g + C/A, ~ the corresponding value function, and it is straightforward to check that ~ and ~ satisfy (3.1) and (3.2) and that Y(x) = v(x) + C for all x. In the case A = 0 the same remark holds for g, but the condition ~ _> 1 can only be replaced by infRNxA e > 0. In this case, on the other hand, the upper bounds on ~ and g can be dropped in some results (see, e.g., Exercises 4.1 and 4.1). The hypotheses on the sign of ~ and g are used only in some of the following results, they are irrelevant for others such as, for instance, the Dynamic Programming Principle. The value function of the problem is

v(x) := inf d ( x , a ) . c~EA

We also state some simple results as the problem of minimizing 07 , which is obtained by replacing tx by t'x in the definition of J, whose value function is denoted by ~. However, the main results on this problem are postponed to Chapter V. 3.1. C o m p a t i b l e t e r m i n a l c o s t a n d c o n t i n u i t y o f t h e v a l u e f u n c t i o n The study of the continuity of v here is harder than in Section 1 because of the non-zero terminal cost g. We first prove that continuity at all points of a T implies continuity everywhere, then we give necessary a n d / o r sufficient conditions for upper and lower semicontinuity at points of 07-. The property of small-time controllability on T, STCT, still plays an important role, but, in contrast to the case g -- 0, it is neither sufficient nor necessary for the continuity of v as we show in Examples 3.1 and 3.2. At the end of the section we give a necessary and sufficient condition for the continuity of v in terms of sub- and supersolutions of the HJB equation. EXAMPLE 3.1 ( S T C T IS NOT S U F F I C I E N T F O R THE C O N T I N U I T Y O F v ) . Assume N = 2, A = {1,2}, f(x, 1) = (0,1), f(x, 2) = ( - 1 , 0 ) , ~ = 1, A = 0 , 7 - = { x : Xl _< 0 and :c2 > 0},

g(x) =

x2

x2 <_ 1, if xl = 0 , x 2 > l . i f X l - - 0, 0 _<

It is easy to see that S T C T holds and T(x) = x l v 0 - x 2 A 0 . It is also clear that v(x) = T(x) = -x2 for xl <_ 0. On the other hand, if Xl > 0 and x2 _< 0 we use the control c~(t) = 1 for 0 < t < - x 2 + 1, a(t) = 2 for t > - x 2 + 1, so that tx(a) = - x 2 + 1 + Xl, and y~(tx(a)) = (0, 1). Then

v(x) < J(x, c~) = xl - x2 - 1

f o r Xl > 0, X 2 _~ 0,

3. PROBLEMS W I T H NON-ZERO T E R M I N A L COST

249

a n d therefore v is discontinuous on the half-line {(0,x2) : x2 ~ 0}. It is not hard to modify this example so t h a t O T is b o u n d e d a n d v has the same lime of discontinuities. <~ EXAMPLE 3.2 (STCT- IS NOT NECESSARY FOR THE CONTINUITY OF v). The most trivial example is f ( x , a) =-- 0; t h e n for a n y 7- a n d g

v ( x ) = inf ~(x, a) - : ~(x) aE A

)~

for x E T c, thus v is continuous if a n d only if g = gl:r. A less degenerate example is, for N = 2, f ( x , a ) - (1,0), g =- 1, A = 0, T = {x : IxI _> 1 or x l >_ 0}, g(x) = 0 if Xl = 0. T h e n v ( x ) = - x l for x E T r and it is continuous if a n d only if g(x) = - x l for IxI = 1. <~ PROPOSITION 3.3. A s s u m e (A0), (A1), (A3), (AT), (3.1), (3.2), and v continuous at each point of 07-. (i)

/ f ~ > 0, then v E B U C ( • N ) ;

(ii)

/f A = 0, then T~ = { x : v ( x ) < +~z}, l i m ~ x 0 v ( x ) = +~c f o r any xo E 0 ~ , and v E BUC(]C) for any closed set ~ where the m i n i m u m time function T is bounded.

PROOF. First of all we observe t h a t there exists a m o d u l u s w such t h a t

Ig(x) - v(z)[ < w(]x - zl)

(3.3)

for all x E 07-, z E ]~N .

It is e n o u g h to d e f i n e w ( s ) : = s u p { I g ( x ) - v ( z ) l : l x - z I < s, x E 07-, z E ~ N } a n d use the continuity of v on 07-, the b o u n d e d n e s s of 07-, a n d a compactness a r g u m e n t as in the proof of P r o p o s i t i o n 1.2. To prove (i) we note t h a t the b o u n d e d n e s s of v is trivial, we fix z > 0 and x, z E 7-c we assume v ( x ) > v ( z ) a n d pick K E .4 such t h a t v ( z ) > J(z,-5) - e. As in the proof of the c o n t i n u i t y of the value function for the infinite horizon problem, Proposition III.2.1, we fix ~- such t h a t f ~ 25I e -~t dt < z, where M is a b o u n d on t~. If t ~ ( 6 ) > ~- the t r a j e c t o r y y z ( t , ~ ) stays at a positive distance from 7- for t < T a n d therefore so does the t r a j e c t o r y y x ( t : ~ ) if x is sufficiently close to z, by (0.3). T h u s also t~(~) > T, a n d the e s t i m a t e v ( x ) - v ( z ) < 3~ for Ix - z I small enough is found exactly as in the proof of Proposition III.2.1. If, instead, t~(~) < 7, we first consider the case that

:= tx(~) < t~(~) =: ~. We define ~ = ~ ( . + ~), ~ : = y~ (~, ~), ~ : = y~ (~, ~) E 07-, a n d c o m p u t e

~(z) _>

f/ fo

e(yz,~))e-~d~+

//

e(y~,~)~-~ds+g(y~(~,~)e-~-e

gf(y~,-5)e-'X~ ds + e-'X~J(F,,8)

e,

250

IV. PROBLEMS WITH RESTRICTED STATE SPACE

so that

(3.4)

v(x)-v(z)

<

]g(yx:.5)-g(yz,.5)le-~Sds+e-~(g(5)-

< ~(eL~lx

v(~)) + E

- zl) + ~ ( e L ~ l x - zl) + ~,

where in the last inequality we have used (0.3), (3.1) and ~ < T. Since the righthand side of (3.4) can be made _< 2z for Ix - z] small enough we reach the conclusion in this case. On the other hand, if t < ~ we define 9 := Yx(t, a ) and use a similar calculation (which in this case is just a piece of the DPP, see Proposition 3.9) to get v(x) - v(z) <_

/o

e(yx,-5) e -:~s ds + v(-2) e -;~ - v(z),

which is _< the right-hand side of (3.4) by the definition of .5, (0.3)~ (3.3), and < v. This completes the proof of (i). To prove (ii), we note that the obvious inequalities T(x)<_v(x)<_MT(x)+maxg

02r

for all x E ~ N

give the first two statements. Now set C 1 :~- J ~ / s u p

xE~:

T(x) + m~_g(x)

so t h a t v(x) < C 1 for x EIC. Next f i x E E ] 0 , 1 ] , x , z E / C \ 7 - , and . s E f l , such that v(z) > J(z,.5) - ~. Then, by the lower bounds on g and g, we have t~(.5) < J(z,.5) <_ C1 + 1 =: ~-. From now on the proof is exactly the same as the second part of the proof of (i).

9

PROPOSITION 3.4. A s s u m e (A0), (A1), (A3), (A~r), (3.1), (3.2), and STCT-. Then, for every z E Off-, v is upper semicontinuous at z, that is, l i m s u p v ( x ) < v(z) = g(z) .

x~z

PROOF. For x ~ ~- and ~ > 0 we pick .5 E A such that t~(.5) _< T ( x ) + s. Then v(x) - g(z) <_ fo t~(~) e ( y ~ , . 5 ) e - ~ ds + g(y~(t~(-~))) - g(z)

<_M(T(x) + ~) +

~(ly~(t~(.5))

- xl + Ix - ~1)

and the right-hand side of the last inequality tends to zero as x ~ z and E --~ 0 +, thanks to (0.1) and because T ( x ) --~ 0 by S T C T . 9

3. P R O B L E M S W I T H N O N - Z E R O T E R M I N A L C O S T

251

DEFINITION 3.5. We say that the terminal cost g is compatible (with the continuity of v) if g has an extension to a neighborhood B ( T , 5) of T, still named g, which is lower semicontinuous at points of OT and such that g(x) <_ J(x; a) for all a E .4 and x E B ( T , 6). Clearly, the compatibility of g is equivalent to the lower semicontinuity of v at all points of z E T. In fact, it is equivalent to the condition g < v in B ( T , 5); thus, for any z E T,

v(z) = g(z) < liminfg(x) < liminfv(z), which proves the sufficiency, and the necessity is obvious. Therefore we have the following THEOREM 3.6. Assume (A0), (A1), (A3), (AT), (3.1), (3.2), and STCT. Then the compatibility of g is equivalent to v E BUC(]~ N) i r a > O, and to v E C(T~) if A--0. PROOF. If A > 0 it is enough to put together Propositions 3.3 and 3.4. If A = 0 we have to recall that S T C T implies T~ open and T continuous in N, by Proposition 1.6, so that Proposition 3.3 (ii) applies to some neighborhood of each point of "/~. 9 In the case A = 0 the compatibility condition can be rewritten in a different form by introducing the function r

ft.(a)

L(x, z ) : = inf~[]~

"1

~(yx, a ) d s : a E ,4 such that yz(t~(a)) = z ) <_ +oc,

for x ~ T, z E OT, which represents the minimal cost to join x and z staying out of T. It is easy to prove that

v(x) = inf { L(x, z) + g(z) },

(3.5)

zEO'Y

and then the compatibility condition g < v is equivalent to (3.6)

g(x) - g(z) <_ L(x, z)

for all x E B ( T , 6) \ T, z E O T .

In this case we can give some more explicit sufficient conditions for the compatibility of g. PROPOSITION 3.7. Assume (A0), (A1), (A3), (AT), (3.1), (3.2), and A = O. If

either g can be extended to B ( T , 5) so that (3.7)

L_Lx_-zl

Ig(x) - g(z)l <_ L -1 log(1 +

SUPaEA

If(x,a)l]

for all x E B ( T , 6) \ T , z E OT, where L is the constant in (A3), or

(3.8)

Ig(x) - g(z)l _< ( sup Ill \T c x A

then g is compatible.

)-1

Ix - zl

/ o r all ~, z e 0 7 ,

252

IV. PROBLEMS W I T H R E S T R I C T E D STATE SPACE

Note that (3.8) is just an upper bound on the Lipschitz constant of g on 07., and therefore it is much easier to check, but it is very restrictive and it requires that g is a constant if f is unbounded. However, (3.8) is also necessary for compatibility in the special case f ( x , a) = a, A = B(O~ 1), g = 1 and 7.c convex, see Exercise 3.2. PROOF OF PROPOSITION 3.7. Set z = yx(tx(~)). Remark III.5.6 1 Iz - xl < ~ sup

If(x,

By the estimate (III.5.11) in

)l (e L~(~ - 1),

9

A

and thus tx(a) >_ the right-hand side of (3.7). Now fix ~ > 0 and ~ E ,4 such that L(x,z) >

~ot~(~) g(yx,-~)ds

-z

> tx(-~) - ~.

Then, by the arbitrariness of s, (3.7) implies (3.6). Moreover, it is clear that (3.7) implies the continuity of the extended g at points of 07-. Now we assume (3.8) and f bounded (if f is unbounded the conclusion is very easy). Then g is Lipschitz on 07" and can be extended off 7- preserving the same Lipschitz constant. Moreover, Ix - z[ _< sup ]fl t~(~) (for x ~ 7- and z as before) implies L ( x , z) >_ (sup I f l ) - l l x - z] and therefore (3.6). 4 Next we give a necessary and sufficient condition for the compatibility of g which looks like (3.6) but can be checked only at points x. z of 07", so that one does not have to look for extensions o f g satisfying (3.6). To do this we first extend L ( . , 9) as follows L ( x , z) := inf l~

J0"

e(y~, a) ds : a E A such that yx(T) = z and y~(t) ~ T for all t E ]0, ~-[ }

for x, z E 7.r The candidate compatibility condition is (3.9)

g(x) - g(z) <_ L ( x , z)

for all x, z E 0 7 . .

PROPOSITION 3.8. A s s u m e (A0), (A1), (A3), (A:r), (3.1), (3.2), and A = O. Then (i) (3.9) is necessary for the compatibility of g; (ii) (3.9) is sufficient for the compatibility of g if in addition f o r all xo E 07. either (3.10)

lim L(xo, x) = 0

x~x

o

or the minimal-time function T is continuous at xo (e.g., STCT" holds) and f o r some r > 0

(3.11)

L(z,x) < L(x,z)

f o r aU z E O7., x C 7-C, x, z C B(xo, r) .

3. PROBLEMSWITH NON-ZEROTERMINALCOST

253

Condition (3.10) is the small-time controllability on x0 of the backward system y~ = - f ( y , a) with the state-constraint that the trajectories stay in 7-c until they reach x0. A simple sufficient condition for it is that the boundary 07" is smooth (e.g., a Lipschitz surface) and f(xo, A) D_ B(0,r) for some r :> 0, see Exercise 3.3. Condition (3.11) is satisfied, for instance, by symmetric systems, that is, systems whose trajectories can be run backwards at the same speed, if t ( . , - ) depends only on the state variable. PROOF OF PROPOSITION 3.8. To prove (i), we fix x0 E c~7" and first observe that (3.12)

liminfL(x, z) <_ L(xo, z) . X~X 0

In fact, for any E > 0 there is a0 such that yxo(t) ~ 7" for t E ]0, r[, yxo(T) = z and 'oT t(Yxo, n0) ds _< L(x0,

Z)

+ s.

We take Xn = yxo(1/n,c~0), a,~(-) = a0(- + l/n), so that yz~,(T -- 1 / n , ~ n ) = Z and therefore L(xn, z) <_

Jo /,-1/,~

g(yx,, an) ds - - ~

f0"

e(Yxo,ao) ds,

which proves (3.12) by the arbitrariness of r Now we recall that the compatibility of g gives g(xo) <_ liminf v(x), x~x

0

and then (3.5) and (3.12) imply (since "lim inf inf < inf lira inf" ) g(xo) <_ inf {L(x0, z ) + g ( z ) } . zEO'lThis gives (3.9) by the arbitrariness of x0. To prove (ii) we first observe that (3.13)

L(xo, z) < L(x0, x) + L(x, z)

for all zo, z E 7"r x ~ 7",

because the right-hand side is the minimal cost to join x0 and z touching x. Then (3.9) gives (3.14)

g(xo) - g(z) - L(x, z) <_ L(xo, z) - L(x, z) < L(x0, x)

for xo, z E 07", x ~ 7". Since v = g on 07" and g is continuous it is enough to prove that g(xo) <_ liminfn v(xn) for any sequence x,, ~ x0, xn ~ "1- for all n. By (3.5) and (3.14) we get g(xo) - lim inf v(x~) = lim sup sup { g(xo) - g(z) - n(x~, z) } n

zEOT

lim sup L(xo, xn) = O,

254

IV. PROBLEMS WITH RESTRICTED STATE SPACE

where in the last equality we have used the assumption (3.10). To treat the case where (3.10) does not hold we fix E > 0 and an E .4 such that

txo(~) <_ T(x~) +

~,

and set zn := yzn(t~,(c~,)) so that, by (3.1) and the definition of L,

L(xn, zn) < M ( T ( x n ) + E) .

(3.15)

By (3.2), (3.9) and (3.13) we get

g(xo) - g(z) - n(xn, z) <_ wg(ixo - zn[) + g(z,~) - g(z) - L(x~, z) < ~g(Ix0 - z~l) + L(z,~, z) - L(xn, z)

< ~g(IXo - z,~l) + L(zn, Xn)

for all z E OT~

where ~g is the modulus of continuity of g on OT. Since (3.5) gives

g(xo) - v(xn) <_ sup {g(x0) - g(z) - L ( x , , z)}, zEOT we can now obtain, also using (3.11) and (3.15),

g(xo) - v(xn) <_ wg(Ixo - z~l) + M ( T ( x , )

+ z),

where the right-hand side tends to 0 as n ---* oc and s --* 0 + because T ( x n ) ---* T(xo) = 0 and then zn ~ x0 as well, as it is easy to show by contradiction. 9 3.2. T h e H J B e q u a t i o n a n d a s u p e r o p t i m a l i t y

principle

Next we derive the HJB equation for the optimal control problem of this section, we consider the Dirichlet boundary value problem for this equation with the terminal cost g as b o u n d a r y data, and compare the value function with sub- and supersolutions of this Dirichlet problem. PROPOSITION 3.9 (DYNAMIC PROGRAMMING PRINCIPLE). A s s u m e (A3), (AT), (3.1), and (3.2). Then

(3.16)

f

v(x) =

inf,/

(A0), (A1),

ftAt~(~)

~ ( y ~ , ~ ) e - ~ ds

aEALJO

+ X{ttx(a)}g(yx(tx(c~))) e -~tx(a) } for all x E ~ N if )~ > O, for all x E 7"~ if % = O, and for all t > O. The proof of Proposition 3.9 is essentially the same as the proofs of Propositions III2.5 and 2.1 and we omit it (see Exercise 3.5). Note that (3.16) can be rewritten as (3.17)

ftAtx(c~) V(X) = i n f ~ / ~(y~,a)e-~ds+v(y~(tAt~(a)))e aEA(JO

} -~(t^t~(~))

9

3. PROBLEMS W I T H NON-ZERO TERi~IINAL COST

255

REMARK 3.10. The D P P for ~ is given by the same formula (3.16) where v is replaced by ~ and tx by ~'x. <~ The Hamiltonian in the HJB equation for the problem with exit time is the same as for the infinite horizon problem, that is, (3.18)

g ( x , p ) := sup{ - f ( x , a ) . p

-

g(x,a)},

x,p E •N.

aEA

PROPOSITION 3.11. Assume (A0)-(A3), (AT), (3.1), (3.2), ~ > 0, and v E C(•c). Then v is a viscosity solution of (3.19)

)w + H ( x , Dv) = 0

in T c .

The proof of Proposition 3.11 is the same as that of Proposition III.2.8. For the case )~ -- 0 we proceed as for the minimal time function by considering the nonlinear transformation of v (3.20)

ifx~y~,ifxET~'

V(x):={ll-e-V(X)

which is the value function of the problem whose cost functional is

This is a problem with nonconstant tonian for this problem is

interest rate, see Exercise III.2.4. The Hamil-

~ ( x , u,p) := sup{ - f ( x , a) . p - g(x, a) + (t~(x, a) - 1)u }, aEA

for u E t~, x : p E R N. PROPOSITION 3.12. Assume (A0)-(A~), (A~r), (3.1), (3.2), and ~ = O. (i) IfT~ is open and v E C(T~ \ T ) , then v is a viscosity solution of (3.21)

H ( x , Dv) = 0

in ~ \ 7 ;

(ii) /f V E C ( T c ) , then V is a viscosity solution of (3.22)

Y + 7-/(x, V, D V ) = 0

in T c .

The continuity assumptions on v and V in the last two propositions are satisfied if g is compatible and S T C T holds, by Theorem 3.6. The proof of (i) is the same as that of Proposition III.2.8, the proof of (ii) is the same as that of Lemma 2.8. As usual we characterize the value function v as the maximal subsolution and the

256

IV. PROBLEMS WITH RESTRICTED STATE SPACE

minimal supersolution, that is, the complete solution, in a suitable set of functions, of the HJB equation with the natural boundary conditions. By subsolution (respectively, supersolution) of the Dirichlet condition (3.23)

u = g

on OT,

we mean a function < g (respectively, > g) at each point of OT. PROPOSITION 3.13. A s s u m e (A0)-(A3), (A~-), (3.1), (3.2), STCT, and g compatible. Then, (i)

if A > O, v is the complete solution in PC(7----~) of (3.19) with the boundary condition (3.23);

(ii)

if A = O, v is the unique solution of (3.21) continuous in T~ \ int T , bounded below, and satisfying (3.23) and v(x) ~ +~c as x --~ xo E 07~;

(iii)

if A = O, V defined by (3.20) is the complete solution in B C ( "T~ ) of (3.22) with the boundary condition V = 1 - e -g on OT.

PROOF. (i) It is enough to put together Theorem 3.6, Proposition 3.11, and the comparison result in Remark III.2.14. (ii) v is a solution of the HJB equation with the desired properties because v > 0 and by Theorem 3.6 and Proposition 3.11. Uniqueness follows from the proof of Theorem 2.6. (iii) The comparison assertion for equation (3.22) is an easy extension of Theorem III.2.12; see Exercise III.2.5 and Remark III.2.14. 9 Next we give two comparison results where no continuity assumptions on the value function v are made a priori. They have several consequences, in particular necessary and sufficient conditions for the continuity of v in terms of existence of suitable sub- and supersolutions of the HJB equation. The first one is an immediate consequence of the global principle of suboptimality, Theorem III.2.33. COROLLARY 3.14. A s s u m e (A0)-(A3), (AT), (3.1), and (3.2). I f u C C ( T c) is a subsolution of (3.19) such that u < g on 07- and u is bounded above if )~ > O, then u < v in T c. In particular, the existence of such a u, satisfying in addition u = g on OT, is sufficient for the compatibility of g, and it is also necessary if S T C T holds. PROOF. As we noted in Remark III.2.34, the assumption )~ > 0 in Theorem III.2.33 can be dropped with no changes in the proof. If a C ,4 is such that tx(a) < +co we take s = 0, t = tx(c~) in statement (ii) of Theorem III.2.33 and use u < g on 07- to get u(x) <_ J ( x , a ) . If tz(c~) = +~z, then for/k = 0 we have Y(x,c~) = +z~, while for A > 0 we let t --* +~c in statement (ii) of Theorem III.2.33 and use the boundedness of u from above to get u(x) < J(x,c~). The other statements are obvious, by the definition of compatible cost g and Theorem 3.6. 4 The next theorem is one of the main results of this section.

3. PROBLEMSWITH NON-ZEROTERMINALCOST

257

THEOREM 3.15. Assume (A0)-(Aa), ( A t ) , (3.1), and (3.2). If u E C ( T c) is a supersolution of (3.19) bounded from below, then

u(x) >_ inf G(x, a) aEA

(3.24)

for all x E 7-c, where { ft~(~)

/o

e(yx~)e-~Sd*+e-~<~

/ftx(~) < + ~

G(x,(~) := ~0 ~r g(y~, c~) e-~ks ds,

if t,(c~) = +~c .

If, in addition, u >_ g on 07-, then u >_ v in T r REMARK 3.16. In the proof of the theorem we also show the global superoptimality principle (3.25)

(ftAt~(o) u(x) >_ inf ~ [ ~(Yx, a) e -;~s ds + e-~(tAt~(a))u(yx(t A tx(a)))} o E A ~ Jo

for all x E 7-r and t > 0, under the same assumptions, except the boundedness below of u which is useless here. <~ PROOF OF THEOREM 3.15. For simplicity we first give the proof under the additional assumption that u is bounded above, because in this case we can use the argument in the proof of Theorem III.2.32 and Remark III.2.34. Then we give the proof in the general case. We recall from Chapter III the formula (III.2.46) in Remark III.2.34: (3.26)

u(x) > i n f I ( x , LcQ

for all x E 7-r O < t < T~(x), 5 > 0,

C~

where (3.27)

I(x, t, ~) := fo t e(y~(s, ~), ~(s)) e -~s & + ~(y~(t, ~)) e - ~ ,

and

Ts(x) := inf{t : dist(yx(t,(~),Of~) ~ 5, c~ E .,4}. We extend I to t = + z c by setting

I(x, + ~ , ~) :=

/7

e(y~(s, ~), ~(s)) e -~s ds.

For given x E T r and e > 0 we are going to construct ~ E .,4 such that (3.28)

u(x) _> I(x, tx(~),~) - E,

258

IV. PROBLEMS WITH RESTRICTED STATE SPACE

which i m m e d i a t e l y gives the desired inequality

u(x) >_ inf I(x, t x ( a ) , a) = inf G(x, a) . We define x l := x, ~'1 := Tl(Xl) and use (3.26) to get a l E A such t h a t

u(x:) > I(x:,r:,~l)

-

r

N o t e t h a t Y=I (T1, a l ) E T c. By induction we define

Xk := Yx,_, (7-k-1, O~k-1),

Tk : = T1/k(xk),

and since Xk E T r we use (3.26) to get ak E .4 such t h a t

U(Zk) > I(xic, TIC,aic) -- : 2 -k 9

(3.29)

We also set aic := 7"1 -~ " * ' -'}"- Tic, -~

=

limit aic,

'al(S)

i f 0 <_ s < a l ,

a2(s - a l ) ~(~)

:=

if or1 < s < ~r2,

--.

aic(s - ale-l)

if Ok_ 1 ~ 8 < Ok, i f ~ _< s,

where the last line makes sense w h e n P < + o c , and ~ E A is arbitrary. definition of xic and ~,

(3.30)

yz(s,~)=yxk(s-ak_l,aic)ET

c

B y the

for all s < ~

and

~OTk e ( u ~ , ~k) e - ~ ds = ~ k

e- ~ - ~

1 ~(y~ ~) e-~

ds.

T h e n we use (3.27) and (3.29) to get

u(z) >

(3.31)

e(yx,-~) e -~s ds + e - ~ l u ( x 2 )

- e/2

>- ~oa2 ~(Yx,-5) e-~S ds + e-~a2u(x3) - r

>_ I(x, a k , ~ ) -- r

-- 2 - k ) .

If ~ -----t-oc we use the boundedness from below of u to pass to the limit as k --~ oc in (3.31) and get (3.28), because t~(~) -- + o o by (3.30) (note t h a t this case cannot occur if A = 0 because g > 1 by (3.1)).

3. PROBLEMS WITH NON-ZERO TERMINAL COST

259

On the other hand, in the case ~ < +o~ we claim that lira yx(t,-~) = 9 E cgT . t/e

(3.32)

Then again we can let k --~ oc in (3.31) and obtain (3.28) because tx(G) = by (3.30). To prove the claim we first note that the limit in (3.32) exists because any trajectory of the system is locally Lipschitz. Moreover ~ E 7 "c. Let us assume by contradiction t h a t ~ E T c. We take 5 > 0 such t h a t B : = B(~, 35) C T c and set C : = suPB• If[ < + o c by (A1). For any z E B(~, 5), any t r a j e c t o r y starting at z remains in B(5, 25) at least for the time 5C -1, and then T , ( z ) >_ 5C -1. Thus there is k such that

~'k >- 5C -1

for all k > k,

and therefore ~ = +oz. This gives the desired contradiction and completes the proof if u is bounded above. To prove the global superoptimality principle in Remark 3.16, we fix x E T c, r > 0, t > 0, and modify slightly the construction of ~ to get

u(x) >_ I ( x , t A tx(-5),~) - z .

(3.33)

Of course, if t >_ Y = tx(~), there is nothing to do. If t < ~, let j be such that O'j_ 1 < t < O'j, ~ j :~- t -- O'j-1 <_ Tj and ~ j E .4 such that

u ( x j ) >_ I ( x j , ~ , ~ S ) - ~ 2 - J . Now redefine ~ as before for s < aj-1, -~(s) := -~j(s - aj-1) for aj-1 <_ s < t, and extend it arbitrarily for s > t. Then the calculation in (3.31) now gives (3.33). The additional assumption t h a t u is bounded above has been used only to get (3.26), so it is enough to refine the cutoff argument of Theorem III.2.32 to prove (3.26) in the general case. For e > 0 we set f2e : = { x E T o : dist(x, c0T) > e and IxI < 1/e } and take he, we E C 1 ( ~ N) such t h a t O
i n R N,

he-=l

in f2e,

he -= 0

in (f23~/4) c,

0 _< We < 1

in R N,

We -- 1

in Qe/2,

We -= 0

in (f~e/4) c.

Define

ue(x):={(o~eU)(X)

ininTr

By multiplying the HJB equation (3.19) by w~hE it is easy to see that uc is a supersolution of (3.34)

Aw + sup{ - r e ( x , a ) . D w - ge(x, a) } = 0 aEA

in ]i{N,

260

IV. PROBLEMS W I T H R E S T R I C T E D STATE SPACE

where fs := p s h s f and gs := ~ s h E t ~ (1 - he))~uc. Now we proceed as in the proof of Theorem III.2.32: we consider the solution we(x,t) of the evolution equation associated with (3.34) (see (III.2.41)) with initial data we(x, O) = us(x), and we observe that us E BC(N N) is a (stationary) supersolution of this initial value problem and that the assumption of the comparison Theorem III.3.7 and Remark III.3.8 hold so that (3.35)

u(x) > us(x) > we(x, t)

for x e ~N., t _> 0, E > 0,

and (3.36)

we(x, t) = inf

{/o'

1

fs(y (e), c~) e -~s ds + us(y(S)(t)) e -At ,

where y(S) denotes the trajectories of the system associated with re. For x E T c, s > 0 , and t < T s ( x ) , y(xC)(s,a) = y x ( s , a ) E ~s for any a and a l l s < t . Since t~s = t~ and us = u in ~e we combine (3.35) and (3.36) to get (3.26). 9 A straightforward consequence of Corollary 3.14 and Theorem 3.15 is a representation formula for any bounded and continuous solution of the Dirichlet boundary value problem (3.37)

~)~u+H(x, Du)=O / (u=g,

i n T c,

for the Hamiltonian given by (3.18). As a uniqueness result it improves Proposition 3.13 considerably because the hypotheses of compatibility of g and S T C T are dropped. COROLLARY 3.17. Assume (A0)-(A3), (AT), (3.1), (3.2), and )~ > O. If u E BC(7 -c) is a solution of (3.37), then u(x) = v ( x ) : = i n f { fo t~ e(yx, ~) e - ~ s d s + g ( y x ( t x ) ) e - ~

}

for all x. REMARK 3.18. Under the additional assumption S T C 7 there is an easy proof of Corollary 3.17 that avoids the (hard) Theorem 3.15. By Corollary 3.14, u < v, so g = ulo:l- is compatible by definition and we conclude by Proposition 3.13. <~ We end this section with a necessary and sufficient condition for the continuity of v which is an immediate consequence of Corollary 3.14, Theorem 3.15, and Proposition 3.3. COROLLARY 3.19. Under the assumptions of Corollary 3.17, the value function v is continuous in T c if and only if there exist u, w E C(Tc), u a subsolution bounded above of the HJB equation (3.19), w a supersolution bounded below of (3.19), u = w = g on OT. Moreover, under this condition, v is the complete solution in B e ( 2 -c) of (3.37).

4. FREE BOUNDARIES AND LOCALCOMPARISON RESULTS

261

The statements of the last corollary also hold in the case A = 0 without the assumption that the subsolution u is bounded above. However, the result is applicable only in the case that ~ = ~N. In the next section we give more precise results for the case A = 0. Exercises 3.1. Compute :F and ~ in Example (3.1). 3.2. Let f ( x , a) = a, A -- B(O, 1), h -- 1, T c convex. Prove that

Ig(x) - g(z)l < Ix - z I

for all x, z 9 07-

is necessary and sufficient for the compatibility of g. 3.3. Let x0 E 02-. Assume there is R > 0, G : B(xo, R) ~ X C ~ N Lipschitz continuous with a Lipschitz inverse, where X is a neighborhood of G(xo) = 0, G(7- A B(x0, R)) -- {x E X : Xl _< 0}, where xl is the first component of x, and G(07- N B(xo,r)) -- {x 9 X : xl -- 0} (G "straightens the boundary" near x0). Assume there is r > 0 such that f(xo, A) D_ B(O, r), Show that x ~-* L(x0, x) and x ~ L(x, xo) are Lipschitz continuous in a neighborhood of x0. 3.4. Under the assumptions of Exercise 3.3 show that

L(x, z) = L(z, z) := inf{

/0"

e(yx, a) ds :c~ 9 .,4 such that yx(T) = z and y~(t) 9 -T c for all t }

for all x, z E T-?'. 3.5. Prove Propositions 3.9 and 3.11 and Remark 3.10. Search for weaker assumptions on the data such that (3.16) remains true. 3.6. Let

:= {x E 0 7 - : x = yz('[z(a),~) for some z 9 I~N, a 9 ,4}. Assume ~ is continuous at each point of U. Prove that (i) if A > 0, then ~ 9 BUC(RN); (ii) if A -- 0, then ~ 9 BUC(1C) for any closed set/C where iF is bounded. 4. F r e e b o u n d a r i e s a n d l o c a l c o m p a r i s o n r e s u l t s f o r undiscounted problems with exit times In this section we improve the basic theory for the HJB equations of problems with exit times developed in Sections 2 and 3, in the special case of null interest rate A, so that the value function of the previous sections becomes r

V(X) :---- i n f ~

ft~(~)

]

a 6 A ~ JO

"1

[(yx(s),a(s))ds+g(yx(tx(a)))~,

262

IV. PROBLEMS W I T H R E S T R I C T E D STATE SPACE

for all x such t h a t t~(a) < + c o for some a , v(x) = +~c otherwise. We first state the m a i n results a n d t h e n c o m m e n t on them. Let us recall t h a t the H a m i l t o n i a n is (4.1)

H(x,p) :=sup~-f(x,a).p-g(x,a)

},

x , p 9 ]l(N .

acA ~

THEOREM 4.1. Assume (A0)-(A3), (AT), (3.1), (3.2). Let 12 C_ •N set, wo C ]~ U { +oc }, w 9 C(fl) be bounded below and satisfying

H(x, Dw)>O

in~\IT,

w > g

on l2 A IT,

be an open

and (4.2)

lim w ( x ) = wo Z~X

(4.3)

f o r all xo C 012,

0

w(x) < wo

for all x E ~ .

Then v(x) < w(x) for all x E ~. THEOREM 4.2. Under the assumptions of Theorem 4.1 let f~ be an open set such that IT C ~ , w o e ]R U { +oc }, w 9 C(l)) satisfy

H ( x , Dw)<_O

inf'\IT,

w <_g

on IT,

(4.2), and (4.3). Then ~ D_ { x : v(x) < wo } and v(x) >_ w(x) for all x 9 ~. These comparison results are different from usual because we are comparing the value function v with a super- or subsolution w on an open set f~ with no informations on the behavior of v on 0 ~ , or on the continuity of v. Instead, we know that fl is a sublevel set for w. A n i m m e d i a t e consequence is the following uniqueness result, which is the c o u n t e r p a r t of Corollary 3.17 for A = 0. COROLLARY 4.3. Assume (Ao)-(A3), (AT), (3.1), (3.2), and wo 9 ]R U { +oc }. I f there is a pair (fl, w) such that ~ D_ IT is open, w 9 C(12) is bounded below and satisfies (4.4)

SH(x, Dw)=O

inf'\IT,

(w

on T,

= g

(4.2), and (4.3), then 12 = { x : v(x) < wo } and w = v in f~. Note t h a t in the special case of t i m e - o p t i m a l control, g -= 1, g =- 0, the conclusion of Corollary 4.3 is t h a t w = T in f~ a n d fl = { ~ ( w 0 ) i f

w0 < +c~, if w0 = +oc,

4. F R E E B O U N D A R I E S A N D L O C A L C O M P A R I S O N R E S U L T S

263

where T is the minimal time function and 7~(t), T~ are the "reachable sets" defined in Section 1. Therefore Corollary 4.3 strengthens considerably the previous uniqueness results for (4.4), namely Theorem 2.6 and Proposition 3.13 (ii). In fact, for w0 = +oc, (4.2) and (4.3) reduce to (4.5)

lim w(x) = +oz

for all x0 e 0ff,

X---*X0

and we see t h a t the correct notion of solution of the boundary value problem (4.4), (4.5) is the pair (~, w), not just the function w. Moreover, if such a solution pair exists it must be (7~, v), without assuming a priori that v is continuous. Thus COT~ is a free boundary for this problem, t h a t is, a piece of the boundary which is not given but is part of the solution to be found. Next we give the proofs of Theorem 4.1 and 4.2, then we turn to other applications and related results. PROOF OF THEOREM 4.1. Case 1 : w 0 < +oc. By (4.2) we can extend w continuously to ~ by setting (4.6)

w(x) : = w0

for all x E 0 ~ .

Let ~'x(a) be the exit time of the system from the open set f \ :r. By Theorem 3.15 we have

(4.7)

w(x) > inf~ f r x ( ~ ) f(Yx, a) ds + w(yx(~:x(c~))) } , o "J0

so if we prove that the infimum on the right-hand side does not change when we restrict ourselves to the controls a such that 7x(a) = tx(a), we immediately reach the conclusion because w > g on cOT. Let us consider a control such that Tx := Vx(a) < + o c and yx(Tx) E cO~. Since * > 1 we get from (4.6)

~oor~ ~(yx, a) ds + w(yx(Tx) ) _> ~-z + w0 > w0, while w(x) < wo for x E f by (4.3), so such a control is irrelevant for the calculation of the inf in (4.7). Case 2 : w 0 = +oc. We recall the exponential (Kru2kov) transformation (4.8)

~(t)

: = 1 - e -t..

t ~ R.,

and compare W(x) := qA(w(x)) and V(x) := q2(v(x)) after extending W continuously to ~ by setting (4.9)

W(x) := 1

for all x E COl,

which is possible by (4.2). Since W is a supersolution of the HJB equation associated with V, the rest of the proof is similar to that of Case 1. We detail only the

264

IV. P R O B L E M S W I T H R E S T R I C T E D

STATE SPACE

case g =_ 1. (For the general case, involving a problem with n o n c o n s t a n t interest rate, see Exercise 4.1.) By Proposition II.2.5, W satisfies

DW}-

W +sup{-f(x,a)-

1 >_ 0

in ft \ T ,

aff A

so by T h e o r e m 3.15, for all x E f~ \ T ,

W(x) > inf~ f r = e - S d s + e - r ~ V ( y x ( T x ) ) } . c~ ~ d O

If yx(T=) E 0fL by (4.9) the t e r m in brackets is 1, while W(x) < 1 for x E Q, so the i n f i m u m does not change if we restrict ourselves to controls such t h a t ~-x = tx, as in the proof of Case 1. Since w _> g on OT we t h e n get

W(x) >- inf{ ~ot~ e - s : inf{1-

ds + e-t~tg(yx(t=)))

e-t=e -9(y=(t=)) } :

}

O ( v ( x ) ) : V(x).

PROOF OF THEOREM 4.2. We prove only the case w0 < +0% since the case w0 = + e c is o b t a i n e d in a similar way after t a k i n g the exponential transforms (4.8) of w a n d v as in the previous proof: we leave this part as a n exercise for the reader. By (4.2) we can extend w continuously to ~ by setting

w(x) : = wo

(4.10)

for all x E Oft .

Let 7~(a) be the exit time of the system from the open set f~ \ T . Corollary 3.14 yields, for all x E s

(4.11)

w(x) <

inf{ f'~ g(Yx,~)ds + w(y~(Tz))}. a "JO

To prove the first s t a t e m e n t we assume by contradiction there is x ~ f~ such that v(x) < wo, a n d fix ~ E .4 such t h a t

g(y~,-5) ds + g(yz(tz(-~))) = wo - 6.

(4.12)

for some 5 > 0. From g _> 1 we get t=(~) < +~c. Since T c f~ is closed and f~ is open, there is t* E [0, t=(~)[ such t h a t z : = y=(t*,~) E Of~ a n d y=(t,-~) E f t for all t E ]t*,tx(-5)]. Therefore, zn : = y=(t* + 1/n,-d) E Q satisfies r,,(a,) = tz,,(a~) for a~( 9 ) : = -~( 9 + t* + 1/n). For these points we have, by (4.11),

~(z,~) <_

e(y .... ~ , ) d s + ~(yx(tx(~),-a)) .

Since s > 0, w < g, a n d by construction of z~, c~, we get from (4.12) w ( z n ) < wo - 5 .

4. FREE BOUNDARIESAND LOCAL COMPARISONRESULTS

265

But w ( z ) = wo by (4.10) and we get a contradiction by letting n ---* ~c because w(zn) -~ w ( z ) . Therefore ~ _~ { x : v ( x ) < w0}. In view of (4.11) and w < g on OT, we can prove that w < v in ~ by showing that the infimum in the definition of v does not change if it is restricted to controls a such that ~'x(a) = t x ( a ) . To this end we consider a E ,4 such that

f0 t~(~) ~(yx, ~) ds + g(yx(tx(~))) _< v(x) + E for some e > 0 to be chosen, and ~-x(a) < t~(a). Then the integral in the left-hand side can be split as f o ~ + jt~ and we rewrite

ftx(~) .'T~(c,)

fot~(~)

where z := y~(Tz(Ol)) E CO~ and K(- ) := (~(- + Tz(o~)), to get

v(x) + e >_

ff

e(yx, ~) ds + ~(z) > wo,

where the last inequality follows from { x : v(x) < w0 } _C ft and ~ _> 0. Now we observe that if v ( x ) > wo we get v ( x ) > w ( x ) from (4.3). In the opposite case we choose ~ < w0 - v(x), and we reach a contradiction if there exists a control as above. Then v ( x ) >_ w ( x ) for all x E ~. 9 Next we consider the m i n i m u m time problem under merely local assumption on the system f , namely,

( f ( x , a) - f ( y , a)) . (x - y) <

(4.13)

LRI x -

yl 2

for a l l a C A, R > 0, Ixi,[yl < R, instead of (A3). At the end of Section 2 we considered a boundary value problem for the minimal time function T in the escape controllability domain R0, and we proved the uniqueness of the solution under suitable assumptions, see Theorem 2.11. The next result is an improvement in the spirit of the previous Corollary 4.3: we show that ~ 0 can be considered as part of the solution to be found, instead of a datum, so that 07~0 is a free boundary. THEOREM 4.4. A s s u m e (Ao)-(A2), (4.13), and (A~r). I f there is a triple (t2, wo, w) such that ~ D_ T is open and unbounded, wo E ]R U { +oc }, w E C ( ~ ) is bounded below and satisfies [supa~A{--f(x,a). Dw} - 1 = 0

w=O

in f~ \ T , in T ,

w(x) --~ ~o

as x --* xo E Of~ and as Ixl --* 9c,

~ ( x ) < ~o

for all x E f~,

then ~ = 7~o, wo = To (the escape time) and w = T in ~.

266

IV. PROBLEMS WITH RESTRICTED STATE SPACE

PROOF. Since w < w0, l i m l z l _ ~ w(x) = wo if a n d only if the sets

~t := { x e ~ : w(x) < t} are b o u n d e d for all t < w0. Therefore, once we have fixed such a t, we can extend the restriction of the system f to i2t x A to a new system f* : R N x A ~ ] ~ g satisfying the global condition (A3) (for instance, f*(x, a) := f(x, a)h(x) with h E CI(]~N), 0 < h < 1, ]Dh] < 1, h -- 1 in f~t and h -- 0 off some b o u n d e d set). Let y~( 9 ) a n d T* be, respectively, the trajectories a n d the m i n i m a l time function corresponding to f*, a n d

7r

:= { x e •N: T*(~) < t } .

We can apply Corollary 4.3 in f~t a n d get ~t = T~* a n d w = T* in gh. We are now going to prove T*(x) = T(x) for all x E i2t. Since T*(x) < t, the only relevant controls a E ,4 are those such that t*(a) < t, and for these controls y*(s) e TC*(t) for all 0 < s < t~(a) because t*(a) -- s + t*y*~.s.(a(. § s)). Since j these trajectories r e m a i n in 7r = fit, they are also trajectories for the original system f, so T(x) < T*(x). To prove the converse inequality, we argue by contradiction. If T(x) < T* (x) for some x E/r let ~ E .A be such t h a t tx(~) < T*(x), so that the corresponding t r a j e c t o r y y(. ) = Yz(" ,-5) exits 7r Since T c 7~*(t) is closed a n d ~ * ( t ) is open, there is s' > 0 such t h a t z := y(s') E OTC*(t) a n d y(s) E T~*(t) for all s e ]s',tx(~)] • @. T h e n

t > T*(x) > t=(~) > tz(-~(" + s')) = t*(~( 9 + s')) > T*(z), so z E T~*(t), a contradiction to z E 0?~*(t), because T~*(t) is open. Therefore T* ~ T in fit. Now we can conclude t h a t w = T in fl because f~ = U0 To. Next we show t h a t TO(t) = f~t for all t < wo, so t h a t wo = To because ftt are b o u n d e d for all t, a n d 1~0 = f~ because 7r = Uo<_t
~ H ( x , Dw) : 0 in B(xo, r) \ T, (4.14)

[w = 0

o n q-,

where x0 E 07" a n d H is given by (4.1). See Exercise 4.2 for extensions.

4. FREE BOUNDARIES AND LOCAL COMPARISON RESULTS

267

THEOREM 4.5. Assume (A0)-(A3), (AT), (3.1), xo E OT, r > O, w 9 C(/3(x0, r)). If w is a subsolution of (4.14) (respectively, a supersolution such that w(xo) = 0 and w(x) > 0 for Ix - z0[ = r), then there exists s > 0 such that w(x) < v(x) (respectively, >_) for all x 9 B(xo, s). PROOF. For mere n o t a t i o n a l simplicity we assume g ~ 1 a n d x0 = 0. We denote with zx = "rx(a) the exit t i m e of the system from the open set B(0, r) \ 7". We begin with the case of the supersolution. By T h e o r e m 3.15 (4.15)

w(x) >_ inf{ Tx(a) + w(yz(Tx(a) ) ) } =: U(X) c~

a n d we w a n t to show t h a t , for ]x] small enough, the i n f i m u m does not change if it is restricted to controls such t h a t zz(a) = tz(a). T h e n u(x) >_ T(x) because w _> 0 on 07". D e n o t e

Itfll:=

sup

B(0,r)xA

If[

a n d note that, for a n y control a a n d for Ixl < r, [yx(t, ~) - xl < Irfl] t as long as the t r a j e c t o r y r e m a i n s in B(0, r). Observe t h a t the existence of the supersolution w implies Llf[] > 0 because H ( x , p ) - - 1 if f - 0. If [x I _< s a n d -r~(a) # tx(a), t h e n [yx(Tx(a))[ = r a n d we can e s t i m a t e r--8

Tx(a) > [Ill--/- "

(4.16)

Now we use w(xo) = 0 a n d the c o n t i n u i t y of w to choose s > 0 small enough so that r--s

(4.17)

sup w < - - . r--s

We also choose 0 < 5 < 2~---~' a n d take a n y a 9 .4 such t h a t ~x(~) + w(yx(~x(a))) < u(x) + 5 . If Ixl < s a n d ~-z(c~) r t x ( a ) , t h e n (4.15), (4.16) a n d w(z) > 0 for Izl = r imply r--s

w(x) >_ IIfI---T-- 5

r--s

>

21ill----~,

a c o n t r a d i c t i o n to (4.17). Therefore u(x) > T(x) for all x 9 s), which is the desired conclusion in the case where w is a supersolution. In the case where w is a s u b s o l u t i o n we choose s satisfying (4.17") as before a n d assume by c o n t r a d i c t i o n w(x) > T(x) for some x 9 B(0, s) (in this case we can assume [Ifll > 0 because T - ~ in B(0, r) \ T if Ilfll = o). T h e n there is ~ 9 ,4 such t h a t

(4.18)

t~(~) < ~(~).

268

IV. P R O B L E M S W I T H R E S T R I C T E D

STATE SPACE

By Corollary 3.14 w(x) < inf{ % ( a ) + w ( y x ( r x ( a ) ) ) } cl

_< inf{ t z ( a ) : a E A , t z ( a ) = rz(a) }, where we used w _< 0 on 0"1- in the second inequality. Then, by (4.18), rx(~) < tx(-5) and we can use (4.16) to get r--s - -

IIfH

<

U,(X),

which gives a contradiction to (4.17). Therefore w < T in B(0, s).

9

Now we t u r n to some applications of the previous theory. As we explained in ~2.2 of C h a p t e r III, a comparison result for the HJB equation can usually be t r a n s l a t e d into a verification theorem, where subsolutions are verification functions a n d supersolutions are falsification fimctions. For instance, if fl and w are as in T h e o r e m 4.2. a n d

Y(x, ~) :=

~00tx(~)f(y~, ~) ds + g(y~(tx(~)))

< w(x)

for some x E ~t, t h e n ~ is a n optimal control for x. This sufficient condition of optimality is also necessary if the value fnnction v is continuous, by t a k i n g w = v a n d fl = { z : v(z) < w0 } with w0 > v(x). Similarly, for f~, w as in T h e o r e m 4.1, a control ~ is not optimal for x C f~ if J ( x , ~ ) > w(x), a n d the condition of nono p t i m a l i t y is necessary as well if v is continuous. T h e second application concerns the c o n t i n u i t y of the value function v. A first simple consequence of T h e o r e m 4.2 is that the t e r m i n a l cost g is compatible (see Definition 3.5) if, in addition to the hypotheses of the theorem, w = g on 02-. Therefore in this case v is continuous in T / i f the system is small-time controllable on 2- (see Definition 1.1) by T h e o r e m 3.6. The next result characterizes the continuity of v even if STC2- fails. It is a c o u n t e r p a r t of Corollary 3.19 for ~ = 0; note t h a t here we can use local, instead of global, sub- a n d supersolutions. Its proof is trivial by Theorems 4.1 and 4.2 and Proposition 3.3. COROLLARY 4.6. Under the assumptions (A0)-(A3), (AT), (3.1), and (3.2), T~ is open and the value function v is continuous in T4 if and only if there exist f~, ~' open sets containing 2- and w E C(fl), u E C ( f Y ) , such that H ( x , D w ) > 0 in gt \ 2-, H ( x , D u ) < 0 in t2' \ T , w = u = g on 2-, w satisfies (4.2), (4.3) for some wo <_ +ae, and u < uo, limx~x0 u(x) = uo f o r all xo E Of Y, for some constant uo <_ +oc. In the special case g = 0 one obviously takes u - 0 and gets a characterization of small-time controllability STC2-. A better result in this case is immediately obtained from T h e o r e m 4.5, Proposition 1.2, a n d a s t a n d a r d compactness argument.

4. FREE BOUNDARIES AND LOCAL COMPARISON RESULTS

269

COROLLARY 4.7. Assume (A0)-(A3), and (AT). Then the system (f. A) is S T C T if and only if for all xo E 07- there exist r > 0 and w 9 C(B(xo, r)) satisfying

>_0

supaeA - f ( x , a ) - D w } - I (4.19)

w(z) > 0

kw(x0)

=

in B ( x o , r ) \ 7-, for x C O(B(xo, r) \ T),

0.

Since the result is of local nature we can actually replace (A3) with (4.13). Some explicit applications of Corollary 4.7 can be easily given. The first is in the case 7- = { 0 }. Theorem 1.12 says that the minimal time function T is locally lipschitzean if and only if (4.20)

inf f(0, a) - 7 < 0

for any unit vector 7 9

aEA

In Section 1 we proved the necessity part. Here is a short proof of the sufficiency part. END OF THE PROOF OF THEOREM 1.12. By (4.20), (A2), and a continuity argument, there are r, 5 > 0 such that inf f ( x . a ) . ~ < - 5

for Ixl < r,l~/I = 1.

aEA

Then w(x) := [x]/5 is a classical solution of (4.19) with x0 = 0, T = {0}. By Corollary 4.7 the system is STCT-. Moreover, T ( x ) < w(x) implies the local Lipschitz continuity of T by Remark 1.7. ~l Now we extend this simple argument to an arbitrary closed and bounded target 7-. We know from Theorem 1.16 that, for smooth targets, T is locally Lipschitz if and only if inf f ( x , a). n(x) < 0

for all x C OT,

acA

where n(x) is the exterior normal to 7-. The next result uses a similar condition where n(x) is replaced by the generalized exterior normals introduced in Chapter II, Definition II.2.17. It includes Theorems 1.12, 1.16, and the extension to piecewise smooth targets of Remark 1.21 as special cases. We recall that N ( x ) denotes the set of all the exterior normals to 7- at x. THEOREM 4.8. Assume (A0)-(A3), (AT), and xo 9 07-. such that (4.21)

If there exist s, 5 > 0

inf f(xo, a) 9L, <_ - ~ aEA

for any ~, E N ( x ) with x E 07- A B(xo, s), then the minimal time .function T is locally lipschitzean in a neighborhood of xo.

270

IV. PROBLEMS WITH RESTRICTED STATE SPACE

PROOF. Define

F ( x , p ) := inf f ( x , a ) . p aEA

and observe that it is continuous in ]~2N by (A2). Let iV" := { v 9 N ( x ) : x 9

07- N B(x0, s) }. Since Iv[ = 1 for any normal, the function G(x) := m a x F ( x , p ) pE~ exists and is continuous. Moreover, (4.21) implies G(xo) < - 5 , so there is r > 0, r < s/2, such that

c(~) < -5

5

for all x 9 t~(~o, r )

Then (4.22)

sup{-f(x,a), u} > 5 aE A

for all u EAf, x E B(xo, r) .

- - "~

We claim that w(x) := (2/5)dist(x~ 7-) satisfies (4.19). In fact, we have only to check the differential inequality. By Proposition II.2.14, either x has a unique projection onto T, p(x), and then D - w ( x ) = { (2/5)u* }, where u* -

x - p(x) is Ix - p(x)[

a normal at p(x), or D - w ( x ) = 0. In the latter case there is nothing to prove. In the former case, if x E B(xo, r), then Ixo - p(x)l ~ [xo - x{ + Ix - p(x)l < 2r < s, so u* E A f and the claim follows from (4.22). Therefore Theorem 4.5 gives T ( x ) < w(x) in some ball B(xo, r~), r ~ > 0, and this gives the Lipsehitz continuity of T in a smaller ball by the argument of the proof of Proposition 1.6. 4 REMARK 4.9. It is easy to see that the main assumption of Theorem 4.8 is equivalent to the existence of ~r, tt > 0 such that (4.23)

inf f ( x , a) . u <_ - #

for all u E N ( x ) ,

aEA

for all x E 07"M B(x0, 0). By the methods of Section 1 it can also be proved that T is locally lipschitzean in all 7~ if (4.23) holds for all x E OT. Actually, the converse is also true: if T is lipschitzean in a neighborhood of T, then (4.23) holds for all x E 07-, see Exercise 4.5. <3 Exercises 4.1. Complete the proofs of Theorems 4.1 and 4.2 in the case w0 = +~c. Show that both theorems hold if the following assumptions are dropped:

5. P R O B L E M S W I T H S T A T E C O N S T R A I N T S

(i)

the boundedness of 7-;

(ii)

g _> 0, provided g is bounded from below;

271

(iii) (3.1), provided f : ]~Y • A --~ 1~ is continuous, f ( x , a ) > f0 > 0 for all x, a, If(x, a) - f(y, a)l < w(Ix - YI~R),

for all a E A, Ix t, lYl -< R,

R > 0, for a suitable modulus w (and this follows from continuity if A is compact). The result can be checked in [Sor93a]. 4.2. Extend Theorem 4.5 to the case of non-null continuous boundary d a t a g, satisfying

g(xo) < ~ where I]fll := sup/~(x0,~)•

r

+ wm,

g(x) >win

for a l l x E 0 7 - ,

Ifl and

wm :=

min

O(B(xo,r)\T)

w.

Show also that Theorem 4.5 holds in the generality of Exercise 4.1. (See [BSgla].) 4.3. Suppose 7---- {0}, f ~ 1, g ~ 0, f~ open, 0 E f~ and let w E C(f~) solve H(x, Dw) = 0 in f~ and w(x) > 0 for x ~ 0, w(0) = 0. Use Theorems 4.1, 4.2, 4.5 and their proofs to find the largest set where we can ensure that w = T.

4.4. Assume (AT-) and let gt ~ 7" be open and connected. Give an example of system f satisfying (A0)- (A3) such that 7~ = f2. (This example says that the free b o u n d a r y 07~ does not have any regularity in general). 4.5. Assume f satisfies (A0) with A compact and is lipschitzean with respect to the state variable~ uniformly in the control variable. Let 7- be any closed set. Prove t h a t the minimal time function T is locally lipschitzean in ?~ if and only if for all R > 0 there exists #R :> 0 such that

minf(x,a)'u<_--#n

VuEN(x), VxeOT-NB(O,R).

aEA

5. P r o b l e m s

with state constraints

In this section we consider a problem where the trajectories of the system must verify a state-space constraint, that is, they have to remain in a given set for all time. More precisely, we are given an open set ~, and for the initial position x E we consider the following set of controls Ax:={aE.4:yx(t,a)

Ef~

for a l l t _ > O } .

The problem is to minimize the infinite horizon cost

J(x, ~)

:=

//

e(~(t), ~(t))e - ~ dt

272

IV. PROBLEMS WITH RESTRICTED STATE SPACE

over the set of admissible controls Ax which keep the state of the system in the closed set ~ for all time. Therefore the value function is inf J ( x ~ a ) .

v(x):=

,~EA~:

The basic assumption on g and A is (A4), see Section 2.1 of Chapter III. We first study the continuity of v under the following additional assumptions (5.1)

0 ~ is of class C 2 and compact;

(5.2)

K :-- sup IX] < +~c;

(5.3)

inf f ( x , a). , ( x ) < 0

R ~ xA

for all x E 0f~;

aEA

where n(x) is the exterior normal to f~ at x. Note that (5.3) means that at any point of the boundary of ~ the system can choose a direction pointing inward ft. It is rather intuitive that under this condition admissible controls exist, that is, (5.4)

A~ # 0

for all x E ~ .

We first give a technical lemma, which in particular proves (5.4). LEMMA 5.1. Assume (A0)-(A4) with wt(r) = Ltr, and (5.1)-(5.3). Then there exist t*, C* > 0 such that for any x E f~ and a E .4 there is ~ E Ax satisfying

(5.5)

IJt*(x,~)

- -

gt*(x,•)l

< C*

s u p dist(y~(t,a),f~), 0
where (5.6)

J t . ( x , a ) :=

fo

e(yx(t,a),c~(t))e-;~t dt.

PROOF. We define the signed distance from Off

d(x) :=

t

-

dist(x, 0f~),

if x E ~

dist(x, 0f~),

ifx ~,

and recall that it is of class C 2 in a neighborhood of 0t2 by (5.1). By (5.3), (As) and the compactness of 0fL there are ~i, r > 0 such that, for any x0 E 0fi there is a(xo) E A satisfying (5.7)

f ( x , a(xo)). Dd(x) > r

for all x E U(x0, 5).

Note also that by (5.2), for any a E .A (5.8)

lye(t, a) - x[ < K t

for all t > 0.

5. P R O B L E M S W I T H STATE C O N S T R A I N T S

273

We want to d e t e r m i n e two c o n s t a n t s t*, k > 0 such that for a n y x E 9 a n d a E A, if we set

to := t* A tx(a),

z:=yx(to)~

~:=

sup d i s t ( y x ( t , ~ ) , ~ ) , 0
where tx(a) is the exit time from ~t, a n d define

{

~-

a

if to = t*,

~(t) : = a(t)X[o.to[ + a(z)k[to,to+le~ ] + a(t - ka)X}t0+kz,oc[

if to = tx(a),

t h e n y~(t,~) E -~ for all t E [0, t*]. This is proved by showing t h a t d(yx(t~-5)) > 0 for any such t. By r e p e a t i n g this construction on the intervals [nt*, (n + 1)t*] we o b t a i n a control in A~. We indicate for brevity y(t) :=

yx(t, ~),

~(t) = y~(t~ ~)

First observe t h a t for d(x) > 5/2 we can choose t* < 5 / 2 K a n d get from (5.8) y(t) E ~ for t E [0, t*]. So in this case to = t* a n d ~ = a. Now suppose d(x) < 5/2 a n d to = t~(c~) < t* < 5/214. T h e n If(t) - x] < (5/2 by (5.8) for t <_ t*, a n d thus [~(t) - z[ < 5

for t E [0, t*]~

where z = y(to) = ~(t0) E 0~2. Therefore (5.7) and the definition of E give

f ( ~ ( t ) , - a ( t ) ) . Dd(~(t)) > r

for to < t < t* A (to + k z ) .

We can use Taylor's formula for d o ~ centered at to to get

d(-O(t)) > ~ ( t - to) >_ 0

for to < t < t* A (to + kz)

and, in the case t* > to + kg,

d(f(t)) > Cks + I(t)

for t e [to + k~, t*],

where

I(t) : =

Dd(~(s)). f(~(s),-~(s))ds. +ke

We estimate the i n t e g r a n d function in I(t) by adding and s u b t r a c t i n g Dd(~(s)) 9 f ( y ( s - kz),-5(s)) a n d D d ( y ( s - kr . f ( y ( s - k~), ~(s)):

Dd(~(s) ) . f(~(s),-a(s) ) >_ - ( L + C)]~(s) - y(s - k~)l + (d o y)'(s - k~), where C is a Lipschitz c o n s t a n t for D d in B ( z , 5). Now we use Gronwal] inequality to e s t i m a t e (5.9)

I~(s) - ~ ( s - k~)l < I~(t0 + k~) - zl e L(s-~~ ~ K k ~ e L(s-t~162

for s > to + kz,

274

IV. PROBLEMS W I T H R E S T R I C T E D STATE SPACE

where the last inequality follows from (5.8). constant C1 and t E [to + ks, t*], d(y(t)) >__ke[~ - C l ( e 5(t-t~

Therefore we get, for a suitable - 1)] + d(y(t - ke)) .

Consequently, d(~(t)) > _ ~ks~ - s if we choose t* := rain { ~--~, (f ~1 log(1 + ~ C 1 }) and use the definition of s and d. T h e n we choose k := 2/~ to o b t a i n d(y(t)) >_ 0 for all t c [0, t*]. This proves t h a t .5 E A s . It remains to prove (5.5) and we do it in the case t* > to + ks, since the other case is similar and easier. By (A4) and the definition of .5 we get ]Jt.(x,.5) - J t . ( x , a ) [

<_

fi

<

f toro+k6 Ig(~,-5)1 dt + ~t]*+ks Ig(~(s), .5(s)) - g(y(s J

+

le(~,.5) - e(y, ~)1 e - ~ dt

f:

ks), .5(s)) I ds

le(Y, (~)l dt

<_ M k E + t*Le

sup

to+ke<s
]y(s) - y ( s - ks)l + M k s < k s ( 2 M + t*LeKeLt*),

where we have used we(r) = L~r and (5.9). B y the definition of ~ the proof is now complete. THEOREM 5.2. A s s u m e (A0)-(A4) with ~r value function v is in B U C ( - ~ ) .

= L~r, and (5.1)-(5.3).

Then the

PROOF. We assume for simplicity s = 1. Let x, z E ~ with Ix - z I < r. For any > 0 there is c~ C .A~ such t h a t (5.10)

J~*(z,~) + e - % ( y z ( t * , ~ ) )

< v(z) + 5~

where t* is as in L e m m a 5.1. Since y~(t:c~) C ~ , by the estimate (0.3) (5.11)

sup d i s t ( y x ( t ~ a ) , ~ ) < re Lt* .

O<_t~t*

Now let .5 E ~4~ be as in L e m m a 5.1 so t h a t (5.12)

IJt*(x,.5)

-

J t * ( x , a ) l <_ Cor

for a suitable constant Co. B y the definition of .5 and the estimates (5.9) and (5.11) there is also C1 > 0 such t h a t lye(t,.5) - y~(t, a)l <_ C l r

for t e [0, t*].

5. PROBLEMS WITH STATE CONSTRAINTS

275

T h e n , by (0.3), (5.13)

[yz(t,~) - yz(t, ~)I < (Cl + eLt)r

for t e [0,t*];

by (A4) with we(r) -: L~r a n d (5.12) there is C > 0 such t h a t

IJr.(x,-5) - J t . ( z , a ) J < O r .

(5.14) Now let, for r > 0,

w(r) = sup{ Iv(x)

-

v ( z ) l : x , z 9 fl, Ix -

zI<

r }.

Our goal is to show t h a t w ~ 0 as r --~ 0 +, a n d for this it is enough to find rn -~ 0 + such t h a t w(rn) --* 0 as n --~ oc. We use the D y n a m i c P r o g r a m m i n g Principle (see Proposition 5.5), (5.10), (5.13) a n d (5.14) to get

v(x) - v(z) < Jr.(x,-5) + e - t * v ( y x ( t * , 5 ) ) - Jt*(z, c~) - e - t * v ( y z ( t *, a) ) + 6 < C r + e - t * w ( C r ) + 6, where C := C1 + e Lt*. Therefore

Now set r~ : = C - n : since C > 1, r,~ ---* 0 as n --~ oc. By an easy i n d u c t i o n a r g u m e n t we o b t a i n n--1

~z(rn)< Crn E(ce-t')k + e-nt'~z(1). k:O

Now we assume, w i t h o u t loss of generality, Ce -t* • 1 to get ~(~n) < c

e -nt* - rn _ + e-~t%(i), Ce -t* - 1

so that w(rn) ~ 0 as n ~ oo.

.q

A n extension of T h e o r e m 5.2 to t h e case of u n b o u n d e d c9V2is in Exercise 5.1. The a s s u m p t i o n (5.3) in T h e o r e m 5.2 looks rather strong and one m a y wonder whether it can be replaced by a weaker one, for instance by (5.4). The p r o p e r t y (5.4) of existence of admissible controls is thoroughly studied in the literature u n d e r the n a m e of viability of the set 9 for the system (f, A). T h e celebrated Viability T h e o r e m gives a necessary a n d sufficient condition for a closed set to be viable for a differential inclusion with convex r i g h t - h a n d side, in particular, for a controlled system such t h a t f ( x , A) is a convex set for all x 9 ~ g . In our special case where [~ has a s m o o t h boundary, this condition is (5.15)

inf f ( x , a) 9n ( x ) < 0 aEA

for all x 9 09t.

The next example shows t h a t the value function v can be discontinuous if (5.15) holds, even if the strict inequality in (5.3) fails at just one point. Therefore the viability p r o p e r t y is not a sufficient c o n d i t i o n for the continuity of v, t h o u g h it is obviously necessary.

276

IV. PROBLEMS WITH RESTRICTED STATE SPACE

EXAMPLE 5.3. Let I2 = { (xl,x2) E R 2 : x 2 < x~}, A = { - 1 , 1}, f ( x , a ) = (a, 0), for all x E R 2, A = 1 and for all a E A

~(x,a)=

-xl

0_<x1_<1, Xl>l.

It is easy to see that an optimal control is a - - 1 if Xl < 0 and x2 > 0, a -= 1 for all other points, and 01,

Xl > 1,x2 E [~, Xl < O~x2 > O.

v(x)=

1-eXl(1-e-1), [ ex~-I - Xl,

Xl<0, x2_0, 0 <_ Xl <_ 1,x2 E R .

Thus v is lower semicontinuous and discontinuous at all points (Xl, 0) with Xl _< 0. Note that all the assumptions of Theorem 5.2 are satisfied except (5.3) at x = (0, 0), where just (5.15) holds, and the boundedness of 0 ~ , but it is easy to modify the example to one with the same features and bounded f~. Note also t h a t the value function does not change if we convexify the system by taking A = [ - 1 , 1]. <~ The next result is in the same spirit as Theorems 1.15 and 1.18 for time-optimal control. Near any point where (5.3) fails (i.e., the system can not choose a direction pointing inward f~), we assume the system is symmetric and it generates a Lie bracket pointing inward f~. The rather long and technical proof is omitted. THEOREM 5.4. Assume (A0), A is a compact symmetric subset of ]~m, (5.16)

If(x,a)-f(y,a)l
f o r a l l x , y,a,

(A4) with w~(r) = Let, (5.1), (5.2), and (5.15). Suppose that for any x E On such that minaeA f ( x , a ) 9n(x) = O, there is a neighborhood of x where the system is symmetric, that is, f ( y , a ) = Q(y)a with the N x m matrix Q of class C 1, and there are two columns Qi, Qj of Q such that

[Q~, Qj](x) 9~(x) # 0. Then the value function v is in BUC(~).

Note that the conclusion of the theorem is the uniform continuity of v in f~, not in f~, and it cannot be improved in view of the example in Exercise 5.3. Next we turn to the study of the HJB equation for this problem. The first result is standard and we omit the proof (see Proposition III.2.5). (A0), (A1),

PROPOSITION 5.5 (DYNAMIC PROGRAMMING PRINCIPLE). Assume

(A3), and (A4). Then, for all x e -~ and t > O, v(x)

=

inf aEA.

{/o

e(yx(,), ~ ( s ) ) e -x~ d~ + ~(y~(t, ~))e -a'

.

5. PROBLEMS WITH STATE CONSTRAINTS

277

l~rom the D P P one deduces by a s t a n d a r d argument that the value function v solves (5.17)

s

+ H(x,Dv)

= 0

in f~, where H ( x , p ) = supaeA { - f ( x ; a ) . p - - g ( x , a ) } (see Proposition III.2.8). However in this problem it is not clear what b o u n d a r y condition v should satisfy. While in problems with exit times v equals the terminal cost on 0f~ by definition, here we do not know it a priori on the boundary, so the Dirichlet problem is not appropriate. To derive the correct boundary condition let us assume 0 ~ is C 1, v is C 1 in ~ , and there exists a continuous optimal feedback m a p ~. Then v satisfies (5.17) in f~ in the classical sense and g ( x , D r ( x ) ) = - f ( x , a2(x) ) . D r ( x ) - g(x, r The state constraint imposes f ( x , 4p(x)). n ( x ) < 0 for x E 0 ~ , so, for any/3 > 0,

(5.1s)

H (x, D r ( x ) ) <__- f ( x , a2(x) ) . (Dv(x) + / 3 n ( x ) ) - e(x, ~(x)) <_ g ( x , D v ( z ) + / 3 n ( x ) )

for x E c~f~.

Now note that for any smooth ~ such t h a t v - c~ has a minimum on ~ at x E 0f~, by the Lagrange Multiplier Rule, D ~ ( x ) = D v ( x ) + / 3 n ( x ) for some/3 > 0. Therefore (5.18) and (5.17) give (5.19)

s

+ g ( x , D ~ ( x ) ) >_ O .

This leads to the following DEFINITtON 5.6. Let u ~ C ( ~ ) . We say that (i) u is a viscosity supersolution of (5.17) in ~ if (5.19) holds for any ~ E CI(]R N) such that u - c2 has a local minimum, relative to ~ , at x E ~ ; (ii) u is a constrained viscosity solution of (5.17) in ~ if it is a subsolution in and a supersolution in f~. <1 The reader must be warned that this definition has to be handled with some care because some obvious properties of supersolutions in open sets do not hold for viscosity supersolutions in closed sets. Consider, for instance, the equation [u' I = 1. The classical solution u(x) = x is not a viscosity supersolution in ~ = [0, 1]. Indeed, if we take ~ - 0, u - ~ has a minimum at 0 but [~'(0)[ > 1 is false. Therefore (a)

a classical supersolution up to the b o u n d a r y is not necessarily a viscosity supersolution in ~ ,

(b) a solution (even classical) in an open set f~' D ~ is not necessarily a viscosity supersolution in ~ . Now we can give the good properties of constrained (viscosity) solutions.

278

IV. PROBLEMS WITH RESTRICTED STATE SPACE m

THEOREM 5.7. Assume (A0)-(A4). If the value function v is in C(f~), then it is a constrained viscosity solution of the HJB equation (5.17) in ~. PROOF. We only indicate the changes with respect to the proof of Proposition III.2.8. Assume x E 0f~ is a local minimum point of v - p relative to ft, t h a t is, for some r > 0 (5.20)

v(x) - v(z) <_ p(x) - ~(z)

for all z E B(x, r) N-~.

The difference with respect to the previous proof is the additional restriction z E in (5.20). In the continuation of the proof (5.20) is used only with z = yz(t,-~), where ~ is a control function suboptimal in the DPP. In particular, in the present case, ~ E Ax, thus yx(t, ~) E ~ and we can use (5.20) as in the previous proof. 4 The comparison theorem for the current problem requires the following regularity assumption on the set fk (5.21)

there are 77 : ~ ---, R N bounded and uniformly continuous and c > 0 such that B ( x + t~(x),ct) C_ f~ for all x E ~ , 0 < t < c .

Condition (5.21) has an easy geometric interpretation: for each x E ~ there is a truncated open cone with vertex in x contained in f~, these cones depend continuously on x and have a uniform size. W h e n applied to x E O~ this suggests that (5.21) is equivalent to saying that OQ is locally the graph of Lipschitzean functions satisfying a uniform bound on their Lipschitz constants. This statement is made precise in Exercise 5.6. For instance, (5.21) is satisfied if O~ is C 1 and bounded. THEOREM 5.8. Assume (A0), (5.2), (5.16), (A4), and (5.21). / f Ul E BUC(f~) is a subsolution of (5.17) in f~ and u2 E BC(-~) is a supersolution of (5.17) in -~,

then ul <_ u2. These two results allow us to characterize the value function in terms of solutions of the HJB equation as follows. COROLLARY 5.9. Under the assumptions of Theorem 5.8, if the value function v is in BUC(f~), then it is the unique constrained viscosity solution of (5.17) in

BUC(-~). Moreover, it is the maximal subsolution in f~ belonging to BUC(~) and the minimal supersolution in ~ belonging to BC(-~). The proof of the Corollary is immediate because (5.2) (5.16) imply (A1)-(A3). Note that the value function has the regularity required in Corollary 5.9 under the assumptions of Theorem 5.2, but not under those of Theorem 5.4. For this case we need the following finer result whose proof involves the theory of discontinuous solutions which is developed in Chapter V. THEOREM 5.10. Under the assumptions of Theorem 5.8, if v E BUC(Q), then its

unique continuous extension to Q is the unique constrained viscosity solution of (5.17) in BUC(~).

5. PROBLEMS W I T H STATE C O N S T R A I N T S

279

Before turning to the proof of Theorem 5.8 we recall the usual consequences of comparison and uniqueness results such as Corollary 5.9 and Theorem 5.10: relaxation and p e r t u r b a t i o n results, as well as verification theorems with subsolutions in ~ playing the role of verification functions and supersolutions in ~ as falsification functions. The statements and proofs of these results are easy exercises, following the arguments of w in C h a p t e r III. PROOF OF THEOREM 5.8. The proof is based on the usual idea of looking at maximum points of an auxiliary function O(x, y). Now, however, we have no information on the subsolution ul at boundary points, so we have to avoid the possibility t h a t the m a x i m u m (5, ~) of r is such t h a t 5 C 0ft. This is achieved by adding an e x t r a term to ~ involving the function U of assumption (5.21). Assume by contradiction m := sup~(ul - u2) > 0 and pick z E ~ such t h a t

(5.22)

Ul(Z) - u2(z) > m - 5,

where 5 > 0 will be chosen small enough later. Define 9 (x,y)

:= ul(x)

- u2(y) -

x ---~Y~- v ( z ) ~ - ly - zl 2,

where s > 0 will be chosen small enough. First we are going to show t h a t achieves its maximum. To this end we choose E < c, so that z + Eq(z) E ~, and name wl the modulus of continuity of Ul to compute (5.23)

~(z +cq(z),z)=ul(z +eq(z))-u2(z) >_m-5-wl(CS),

where C :-- sup~ I~]l and we have used (5.22). On the other hand,

9 (x,y) < m + ~ l ( I X -

yL) -

z___~y

~(z) 2 _ ly - zl ~.

If (x, y) is such that (~(x, y) > ~ ( z + ~q(z), z) we have

(5.24)

ly-zl2+

L-~_

~(z) 2 < 5 + ~ l ( C ~ ) + ~ ( I x - y l ) .

Since wl is bounded, the points x, y satisfying this inequality are in B(z, R) for a suitable R, so the search for the maximum of 9 can be restricted to a compact set. Let (5,~) be such a point:

4~(5,~) = m a x , . gtxfl

By (5.24) we get for some constant C1

(5.25)

15 - ~1 < C ~ ,

and plugging this into (5.24) (5.26)

l Y - Zl 2 "1- X --Y~ -- ?'](Z) 2 -~< 5 "1"-~

"}- ~

9

280

IV. PROBLEMS WITH RESTRICTED STATE SPACE

From this inequality we deduce that Z E ~. In fact, there is ~ > 0 such that C

b ( x ) - ~(~)1 < 5

if I~ - ~1 < Q;

also we can choose 6 and ~ such that the right-hand side of (5.26) is less than min{ #2 c2/4 }. Therefore (5.26) yields lY - zl -< ~) and, consequently, C

(5.27) by (5.26) we also have

C

This, combined with (5.27), implies

Thus 5 E B ( y + E~l(y), cs) C_ ~ by (5.21). Now we can use the fact that ul is a subsolution of (5.17) in 9 and that Ul has a maximum at ~, where ~(x) :=

x~

~?(z)

to get

On the other hand, u2 is a supersolution in 12 and u2 - ~ has a minimum at ~, where

~(y) :: - ~

Y

~(z) 2 _ b - zl<

E

so that Au2(~) + Next we subtract this inequality from (5.28) and estimate U l ( ~ ) -- U2(~). To this end we observe that (5.2), (5.16), and (A4) imply g ( x , p ) - H ( y , q ) <_ g [ p - ql + LIx - Y] [P[ + w~([x - Yl), see Lemma III.2.11. We assume for simplicity A = 1 and get ul(~)

-

u2(Y) <_ 2 K I z

-

2L '2 Yl + 7-1

-

x - y Yl - - 7 - -

-

~(z) + ~ ( l ~ -

Yl),

and observe that by (5.25) and (5.26) the right-hand side can be estimated by ~ ( v q + o(1))

as ~ ~ 0%

for a suitable constant C. Now we use (5.23) to obtain rn-6-o(1)

< (I)(g,~) _< Ul(~ ) - u2(~) <_ C ( v ~ + o(1))

which gives a contradiction to rn > 0 for e, 6 small enough.

as z ---*0+,

5. PROBLEMS W I T H STATE CONSTRAINTS

281

REMARK 5.11. Comparison results still hold between subsolutions in fl and supersolutions in f~ of general Hamilton-Jacobi equations of the form u(x) + H ( x , D u ( x ) ) = O .

For example, comparison holds if f~ is bounded and H satisfies conditions (Hi) and (H3) of w in Chapter II (see Exercise 5.7). Note that no comparison between sub- and supersolution on 0f~ is required whereas (H3) is needed (compare with Theorem II.3.1). <~ Exercises

5.1. Extend Theorem 5.2 to the case that (5.1) and (5.3) are dropped and replaced by the following assumptions: the signed distance d( - ) from Oft is of class C 1 in X := B(Of~, r) for some r > 0, and D d is lipschitzean in X: there is a modulus such t h a t I f ( x , a ) - f ( y , a ) l < ~ ( I x - Yl) for all x , y E X and a E A; there is ~ > 0 such t h a t infae A f ( x , a) 9n ( x ) < - ~ for all x E cgf~. 5.2. Prove that v is lipschitzean under the assumptions of Theorem 5.2 and for A large enough. (Cfr. [LOT94].) 5.3. Consider gt = { (xl, x2) E R 2 : x~ < x2 }, f ( x , a) = (2al, 2(xl + x2)a2), A = { (1,0), ( - 1 , 0 ) , (0, 1), ( 0 , - 1 ) }, A = 1, g ( x , a ) = ~(x2) where r E C ~ ( [ 0 , + o c [ ) is nonincreasing, ~ ( r ) =- 1 if 0 < r < 1, ~ ( r ) - 0 if r >_ 2. Show that: (i)

the assumptions of Theorem 5.4 are satisfied;

(ii) v is discontinuous at (0,0) (err. [Mo95]). 5.4. Prove that, under the assumptions of Theorem 5.4, the set of points x E 0f~ such that minaeA f ( x , a) 9 n ( x ) = 0 has e m p t y relative interior. 5.5. Assume H is differentiable with respect to p, cgf~ is C 1 and u E C I ( ~ ) is a constrained viscosity solution of (5.17). Prove that (i)

u satisfies the nonlinear oblique derivative boundary condition OH (x D u ( x ) ) . n ( x ) > O.

x E OQ:

(ii) if in addition H is nondecreasing with respect to IPl, then u satisfies the Neumann b o u n d a r y condition

Ou (x

On

)_>0,

z E 0f~.

5.6. Assume f~ C Ii~N is open and Oft is bounded. Prove that ft satisfies (5.21) if and only if there exists xi E Oft, i = 1 , . . . , m, unit vectors 71i, Lipschitz continuous functions fi : {v : v - rli = 0} ~ N, and r > 0 such that

U

B(x,,,-/2) ~_ on

i=l,...,m

and a n

B(x. 2r)

= {t~i + v: v . Vi = 0 and t > fi(v)} n B(xi, 2r) for i = 1 . . . . . m.

282

IV. PROBLEMSWITH RESTRICTEDSTATE SPACE

5.7. Assume ft bounded and let ul, u2 E C(fI) be, respectively, a subsolution in ft and a supersolution in ~ of

u(x) + H(x, Du(x)) = O . Prove that, under conditions (H1) and (H3) in w

ul _< u2 in K

5.8. Assume f~ bounded and starshaped with respect to the origin. Assume also that, for some k > 0, dist(x,~) > k~,

Vx C (1 + E ) 0 ~ , V ~ > 0 .

Prove that if (H1) of Chapter II holds then the same result as in Exercise 5.7 is true. [Hint: use the auxiliary function @(z,y)=(l+a)ul

(~ x

) - u2(y) - ~-fiz l 2 l x _ yl2

and look for a maximum point (~,~) of 4~ over (1 + E ) f / x ft (cfr. [CDL90]).] 5.9. Consider equation (5.17) in a bounded set D with A = 0. Assume the existence of u_0 E C 1(~) such that

g ( x , Duo(z)) < O,

x e "~.

Prove that the comparison result in Exercise 5.7 still holds in this case. [Hint: mimic the proof of Theorem 5.9 in Chapter II.] 5.10. Assume (A0)-(A4) and let v be the value function of the infinite horizon problem without state constraints, that is, v(x) = inf~6 A d(x, a). Given D0 C ]~N the attainable set from ft0 is the set :={z:z=yx(t,~)

for s o m e x 6 f ~ 0 , t _ > 0 , ~ 6 A } .

Prove that: (i)

v is a viscosity solution of (5.17) in ~, that is, it is a supersolution in ~ according to Definition 5.6 and for all ~ 6 C1(• N) such that v - ~ has maximum, relative to f~, at x 6 f~ Av(x) + H(x, D~(x)) <_ 0;

(ii) v is the unique bounded and continuous viscosity solution of (5.17) in ~ (cfr. [IMZ91]). 6. B i b l i o g r a p h i c a l

notes

Time-optimal control is one of the most classical problems in control theory (it appears already in Carath6odory's book [Ca35]) and it has a large literature (see, e.g., [PBGM62, Lei66, LM67, HL69, FR75, Co85]). The property of small-time local controllability (STLC) is one of the main subjects of the so-called differential geometric control theory; introductory surveys are, for example, [Sus83, Bac86].

6. B I B L I O G R A P H I C A L NOTES

283

It has also different names, for instance it is called N-local controllability in the Russian literature. The characterization of controllability for linear systems in Theorem 1.9 is classical (see, e.g., [MS82, Bi83, Co85]) and the Hhlder continuity of the minimal time function T in this case can be found in [LivS0, Ra82, Gy84]. Theorem 1.10 can be found in [KP72, BreS0], see also the book [Co85] for more informations on controllability of linear systems, and [CSi95a] for semiconvexity and semiconcavity properties of T. The controllability statement in Theorem 1.11 on linearized systems is again classical [MS82], and the estimate of the Hhlder exponent of T was proved by Bianchini and Stefani [BiSg0]. Theorem 1.12 is due to Petrov [Pe68, PeT0a], the proof we present here is new. Theorem 1.15 is the prototype result in geometric theory of controllability, usually called Chow~s theorem, see [Bac86] for its history. There is a large literature on small-time controllability for point-shaped targets and systems nonlinear in the state and affine in the control variables, see [Sus87, Kaw90, He91, BiS93] and the references therein. For the related Hhlder continuity properties of T the reader should consult [Liv80, St91, BiS90] and the references therein. There are not so many results for general targets. If the boundary is smooth, the sufficient condition for controllability (1.13) is very natural and it appears, for example, in IF61, Fri74] in the more general context of pursuit-evasion games. Theorem 1.16 and Remark 1.20 are taken from [BF90a], and the extension to general targets Theorem 4.8 can be found in [Sor93a], see also [Ve96]. Theorem 1.18 is taken from [BS91c] where necessary conditions for 1/2-Hhlder continuity are also studied. Lemma 1.19 is taken from [HH70] and it is implicit in Hhrmander's work on hypoelliptic PDEs [Hor68], its proof for j > 1 can be deduced, for example, from Theorem 3.4 in IBIS93]. The Hhlder continuity of T when the target is a lowerdimensional manifold is studied in [Sor92a]. For further regularity properties of the minimal time function and some applications we refer to [CSi95a, CSi95b, BLP97]. Definition 1.22, Theorem 1.23 and the example in Exercise 1.3 can be found in IBIS90], see also [Pe70b]. Section 2 is based on [B89]; observe that the example reported in Exercise 2.5 was proposed in [Ha91] as a possible counterexample to Theorem 2.6. Theorems 2.9 and 2.11 are taken, respectively, from [BF90a] and [BS92]. For linear systems with target the origin, the minimal time function is a bilateral solution of the HJB equation (see Remark 2.5 in Chapter VI and [Hagl]), and it also solves a different PDE derived by H~jek, see [Sta89]. For some earlier connections between the linear time-optimal problem and other notions of solution of the HJB equation we refer to [Co85, MP86]. Some of the results in Sections 1 and 2 are extended to systems with unbounded controls in [RS96]. The material of w is mostly new and it is based on unpublished joint work with Soravia. It extends the analysis in Chapter 5 of P.L. Lions' book [L82] where problems of the Calculus of Variations in smooth domains are treated. The

284

IV. PROBLEMS W I T H R E S T R I C T E D STATE SPACE

compatibility conditions on the boundary data are formulated in [L82] by means of a comparison function different from L ( . , 9 ), namely, the function L ( . , 9 ) defined in Exercise 3.4, and they can be recovered from our results by Exercises 3.3 and 3.4. Some related results for controlled diffusions are in [L83a]. A different approach to the compatibility of the data in the Dirichlet problem for HJ equations is in [En86]. The main result in w is Theorem 3.15, whose proof follows Evans and Ishii [EI84], see also [LS85] (both papers also treat differential games). A uniqueness theorem such as Corollary 3.17 was proved by P.L. Lions [LS3a] in the case of controlled degenerate diffusion processes as well. Section 4 follows the papers of Bardi and Soravia [BS91a, Sor93a, BS92]. A complete proof of Corollary 4.3 in the case w0 = +co is in [BS91a] for the more general problem of differential games (see also [EJ89] for a similar result in some special cases); Theorems 4.1 and 4.2 are taken from [Sor93a] (where differential games are studied), Theorem 4.4 can be found in [BS92]. Theorem 4.5 is due to Evans and James [EJ89], see also [BS91a] for the case of non-zero terminal cost and [He88] for a connection of this result with the synthesis of optimal feedbacks. Several results of Sections 1-4 were extended to the case of state-dependent constraints on the control in [Bo93], see also [K96a]. The problem of maximizing the blowup time of a controlled system gives rise to a free boundary problem similar to those of w which was studied in [BL96a]. Section 5 follows Soner [$86]. The proof of Lemma 5.1 using the signed distance from the boundary is new (see, e.g., the appendix to Chapter 14 in [GT83] for the smoothness of the distance function). The viability theorem can be found, for example, in the books [AC84, AF90, Au91]. Example 5.3 was shown to us by Soravia. Theorems 5.4 and 5.10 are due to Motta [Mo95]. Further properties of the value function of the infinite horizon problem with state constraints are studied in [Lo87, LOT94, ArL96], and the boundary condition is reformulated and improved in [IK96]. Finite horizon problems with state constraints, unbounded controls, and non-coercive cost were studied in [MR96b, MR96c]. The theory of constrained viscosity solutions was developed in several directions in [CDL90]. The condition satisfied at boundary points by constrained solutions is related to oblique derivative problems, as indicated by Exercise 5.5. Hamilton-Jacobi equations with Neumann boundary conditions were first studied by P.L. Lions [L85b, L86], linear oblique derivative problems in nonsmooth domains in [DIg0, DI91], and fully nonlinear oblique derivative problems in [BaL91, I91].

CHAPTER V D i s c o n t i n u o u s viscosity s o l u t i o n s and applications

In this chapter we extend the theory of continuous viscosity solutions developed in Chapter II to include solutions that are not necessarily continuous. This has two motivations. The first is that many optimal control problems have a discontinuous value function and we want to extend to these problems the results of Chapters III and IV, in particular the characterization of the value function as the unique solution of the Hamilton-Jacobi-Bellman equation with suitable boundary conditions. The second motivation is more technical: viscosity solutions are stable with respect to certain relaxed semi-limits, that we call weak limits in the viscosity sense, which are semicontinuous sub- or supersolutions. These weak limits are used extensively in Chapters VI and VII to study the convergence of approximation schemes and several asymptotic limits, even for control problems where the value function is continuous. In Section 1 we present a comparison result for semicontinuous sub- and supersolutions and prove the stability of viscosity solutions with respect to the weak limits. The reader interested in the approximation and asymptotic results of the following chapters should read this section. Section 2 is devoted to Ishii's definition of a non-continuous, (i.e., not necessarily continuous) viscosity solution. We prove the existence of such solutions by means of Perron's method, and we remark that there is no uniqueness for the standard boundary value problems without some additional conditions. In the remaining sections we describe three different ways to get uniqueness in classes of non-continuous functions. In Section 3 we reformulate Subbotin's generalized solutions in the viscosity language; we call them envelope solutions. In Section 4 we discuss a weak interpretation of Dirichlet boundary conditions which is related to the vanishing viscosity method. In Section 5 we describe the BarronJensen definition of discontinuous solution, that we call bilateral supersolutions. In all three sections the theory is applied to control problems involving the exit times of the system from a domain, in particular to the minimum time problem. In Section 5 we also consider the finite horizon problem with constraints on the endpoint of the trajectories.

286

V. DISCONTINUOUSVISCOSITYSOLUTIONS 1. S e m i c o n t i n u o u s s u b - a n d s u p e r s o l u t i o n s , weak limits, and stability

We begin with some notation for the spaces of semicontinuous functions on a set E C RN:

USC(E) LSC(E) BUSC(E) BLSC(E)

:= { u : E -~ ]~ upper semicontinuous }, : = { u : E --* R lower semicontinuous }, := { u E USC(E) bounded }, := { u E LSC(E) bounded }.

We recall that u : E --~ ~ is upper (respectively, lower) semicontinuous if for any x E E and r > 0 there is (f such that f(y) < f(x) § ~ (respectively, f(y) > f(x) - ~) for all y E E N B(x, (f); Weierstrass' Theorem on the existence of maxima (respectively, minima) on compact sets holds for upper (respectively, lower) semicontinuous functions. The definition of continuous viscosity sub- and supersolutions of a first order equation (1.1)

F(x, u, Dx) = 0

in ~,

with ~ C_ ]~N open and F : fl • ]~ • ~ N __, R continuous, extends naturally to semicontinuous functions as follows. DEFINITION 1.1. A function u E USC(fl) (respectively, LSC(~)) is a viscosity subsolution (respectively, supersolution) of (1.1) if, for any ~ E C l ( f l ) and x E 12 such that u - ~ has a local maximum (respectively, minimum) at x,

F(x,u(x),D~(x)) < 0

(respectively, >_ 0 ) .

,~

As for continuous semisolutions, there is an equivalent definition by means of the semidifferentials of u instead of test functions. In fact the definition of D+u(x) and D-u(x) makes sense for any real valued function, and u E USC(~) (respectively, LSC(~)) is a viscosity subsolution (respectively, supersolution) of (1.1) if and only if, for any x E ~,

F(x,u,p) < 0

(respectively, _> 0)

for all p E D+u(x) (respectively, D-u(x)), as the reader can prove by the arguments of Chapter II, see Exercise 1.3. Later in this section we show how the proofs of the comparison theorems of Chapters II and III can be modified to get some assertions for these larger classes of sub- and supersolutions. For future reference we make precise what we mean by comparison principle and give an example (whose proof is postponed to the end of the section).

I. SEMICONTINUOUS SUB-AND SUPERSOLUTIONS

287

DEFINITION 1.2. We say that the Dirichlet problem for equation (1.1) satisfies the comparison principle or, briefly, t h a t the comparison principle holds for F , if for any ul E BUSC(~) and u~ C BLSC(~) which are, respectively, a viscosity subsolution and supersolution of (1.1) such that Ul _< u2 on 0tl, we have ul _< u2 in ft. THEOREM 1.3. Assume H : ft • lt~N --+ ]~ satisfies either the assumptions of Theorem II.3.5 or those of Theorem III.2.12. Then the Dirichlet problem for the equation

u + H(x, Du) = 0

(1.2)

in il

satisfies the comparison principle. This result, whose proof is postponed to the end of this section, adds nothing to the problem of uniqueness of viscosity solutions, because a function which is simultaneously a subsolution and a supersolution according to Definition 1.1 is indeed continuous, and thus it is a solution in the same sense as in the previous chapters. However, Theorem 1.3 has other important consequences. A first simple one is the strengthening of the notion of complete solution we used in Chapters II and III. Consider the boundary value problem of equation (1.2) with the boundary condition (1.3)

u = g

on 0il,

and call a subsolution (respectively, supersolution) of the Dirichlet problem (1.2)(1.3) a subsolution of (1.2) in BUSC(ft) (respectively BLSC(ft)) which is <: g (respectively, > g) on Oft. We call a complete solution of (1.2)-(1.3) a function which is at the same time the maximal subsolution and the minimal supersolution. Of course such a function is a solution in BC(-~). Theorem 1.3 implies that the converse is also true: a solution of (1.2)-(1.3) in BC(il) is the complete solution. In Chapters III and IV we have seen a variety of optimal control problems whose value function is the continuous solution of an initial-boundary value problem for a Hamilton-Jacobi-Bellman equation. For instance the value function of infinite horizon problems fits within Theorem 1.3 with il = I~ N (see III.2), and so does the value function of exit time problems with terminal cost g (see IV.3), under suitable assumptions. Theorem 1.3 and the fact that the value function is the complete solution give verification theorems, and then sufficient conditions of optimality, with merely upper semicontinuous (and bounded) verification functions, and they give falsification theorems, and then necessary conditions of optimality, with merely lower semicontinuous falsification functions. The precise statements of these results can be easily deduced from the arguments of w A much more interesting and far-reaching application of Theorem 1.3 is the extension of the stability properties of viscosity solutions. Let us recall briefly what was proved on this m a t t e r in Chapter II. Let F~ : ft x I~ x 1i~N ----+ ]I~, 0 < ~ _~ 1, be continuous and (1.4)

F~ =a F

on compacta of il x 1~ x ]~Y as E ~

0+ ,

288

V. DISCONTINUOUS VISCOSITY SOLUTIONS

(:=t means uniform convergence), u~ E C(f~) be solutions of (1.5)

Fs(x, us, Dus) = 0

in f~

and (1.6)

us ~ u

on c o m p a c t a of f~ as r ---, 0 + .

T h e n the limit function u is a solution of the limit equation (1.7)

F ( x , u, D u ) = 0

in ~,

see Proposition II.2.2. If one wants to use this result to pass to the limit in the P D E (1.5), a s s u m i n g (1.4) is known, the n a t u r a l procedure is t h e following. First prove t h a t us satisfy on each c o m p a c t u m K: c_ ~ the u n i f o r m estimates (1.8)

sup lusl < C~: tc

(1.9)

lug(x) - us(y)l _< ~ ( I x

- yl)

for all ~ y e t:,

where the c o n s t a n t C~: a n d t h e m o d u l u s wK: do not d e p e n d on r t h e n use the Ascoli-Arzelk compactness t h e o r e m a n d a diagonal procedure to extract a sequence uE~ ~ u on all compacta of ~t a n d so conclude that u solves (1.7). In the special case ~ = N N a uniqueness t h e o r e m for (1.7) would t h e n ensure the convergence (1.6) for the whole net us a n d not j u s t for a subsequence; the same can be done when (1.7) is coupled with a b o u n d a r y condition as (1.3), provided this b o u n d a r y value problem satisfies a uniqueness theorem, Usloa -* g, a n d the estimates (1.8), (1.9) hold on each c o m p a c t u m K C_ ~. In next results we want to dispense with the equicontinuity estimate (1.9). DEFINITION 1.4. For the functions uE : E --~ ~, 0 < E < 1, E C_ N N, the lower weak limit in E as e --* 0 + at the point x E E is u_(x) = l i m i n f , u s ( x ) : = s u p i n f { us(y) : y E E , Ix - Yl < 6 a n d 0 < r < 6 }~ s~ 0 +

6> 0

or, more concisely, u(x)=

l i m i n f uE(y); (y,~)~(x,O+)

the upper weak limit is ~(x) = limsup*ue(x) := l i m s u p uE(y) . e~0 + (y,s)~(~,0+)

<~

If the functions us are locally uniformly b o u n d e d , t h a t is, (1.8) holds for each c o m p a c t u m / C , the weak limits are finite a n d define two functions u, ~ : E --* R. Their m a i n properties are described by the next two lemmata.

1. SEMICONTINUOUSSUB* AND SUPERSOLUTIONS

289

LEMMA 1.5. The lower weak limit u is lower semicontinuous and the upper weak limit ~ is upper semicontinuous. PROOF. We only show t h a t ~ is u p p e r semicontinuous. We assume by contradiction xn ~ x a n d ~(xn) _> ~(x) + 5 for some (f > 0 a n d all n large enough. By definition of ~(x,~) there are Yn, z,~, ~ such that lY,~ - xnl < l / n , Zn --* 0 +, a n d 5 (f uc, (Yn) >_~(xn) - 5 >- ~(x) + -~

for n > ~ .

T h e n l i m i n f n u~,(yn) > ~(x), a contradiction to the definition of ~(x) because Yn ----~x. LEMMA 1.6. Let u~ e USC( E) (respectively, LSC(E) ) be locally uniformly bounded, ~ E C I(E), B : = / ~ ( x 0 , r) ~ E be closed, and assume xo E B is a strict maximum (respectively, minimum) point for ~ - ~ (respectively, u_ - ~) on B. Then there exists a sequence {xn} in B and zn --* 0+ such that xn is a maximum (respectively, minimum) point for us, - ~ on B and limxn = x0,

l i m u e , ( x n ) = ~(x0)

(respectively, _u(x0)).

PROOF. We prove only the s t a t e m e n t a b o u t the upper limit. Let Zn --~ 0 + a n d {x (n)} be sequences such t h a t (1.10)

x (n) ----, x0,

u,(x

(n)) - - ~ ~ ( x 0 )

as n ~

~.

Let x,~ be a m a x i m u m point of u~n - ~ on B and extract subsequences, still denoted {Xn} a n d {~n}, such t h a t

(1.11)

xn ---, ~,

u~o(x~)

~

s.

Since (uE, - p)(x,~) > (uz~ - ~)(x(n)), we o b t a i n from (1.11), the definition of ~, a n d (1.10), (1.12)

(~ - ~)(~)

> s - ~(~) > ( ~ - ~ ) ( x 0 ) .

Since xo is a strict m a x i m u m point for ~ - p we get t h a t ~ = x0. T h e n (1.12) also gives s = ~(x0). .,, The next result states t h a t viscosity sub- a n d supersolutions are stable with respect to upper a n d lower weak limits, respectively. THEOREM 1.7. Assume F~ is continuous for any E E ]0, 1], (1.4) holds, and let uc : ~ ---* 1~ satisfy (1.8) for any compactum ]C C 12 and all ~. (i)

I f ue C USC(12) is a subsolution of (1.5) for all e, then the upper weak limit E USC(ft) is a subsolution of (1.7);

(ii)

if uE E LSC(~) is a supersolution of (1.5) for all z, then the lower weak limit u E LSC(~) is a supersolution of (1.7).

290

v. DISCONTINUOUSVISCOSITYSOLUTIONS

PROOF. By Lemma 1.5, ~ and u have the correct semicontinuity. The proof t h a t they are, respectively, a subsolution and a supersolution follows from the definitions and a straightforward passage to the limit, just as in the proof of the previous stability result Proposition II.2.2, once we have replaced Lemma II.2.4 with Lemma 1.6. 4 This result is particularly powerful when it can be used in conjunction with a comparison principle such as Theorem 1.3. In fact, we can prove the local convergence of the whole net us under the same assumptions as before except for the equicontinuity estimate (1.9), which can be dropped altogether if gt = ~ g and replaced by similar estimates just near the boundary in the general case. COROLLARY 1.8. A s s u m e Fs is continuous f o r any E E ]0, 1], the comparison principle holds f o r F , and (1.4). Let us : ~ --* ]~ be continuous in ~ and satisfy Fe(x, u~, Due) = 0

in l-~,

and f o r some constant C

suplusl < C

(1.13)

for all z > 0 .

I f ~ ~ ~ N , assume in addition the existence of the limit

(1.14) Then uE ~ B C ( ~ ) of

lim us(y) =: g(x) (y,s)-~(x,0+) u on compacta of ~

(i.i5)

for any x E 0 ~ .

as z --* 0 +, where u is the unique solution in

F(x,u, Du)=O

inl2,

u = g

on OQ .

PROOF. By definition u < ~. By (1.13) u and ~ are bounded, they are semicontinuous in ~ by Lemma 1.5, and (1.14) gives u = ~ = g on cOi-t. Thus, by Theorem 1.7, the comparison principle gives ~ < u and therefore u_=~--:u is the solution of (1.15). The local uniform convergence of u~ to u is the following elementary Lemma 1.9. 4 LEMMA 1.9. A s s u m e that the functions us as in Definition 1.4 satisfy (1.8) on a compact set ~ and

(1.16) Then us ~ u on tg as z -~ O+.

u = ~ =: u

on ~ .

1. SEMICONTINUOUS SUB- AND SUPERSOLUTIONS

291

PROOF. Since u is both upper and lower semicontinuous (by Lemma 1.5), it is continuous. Assume by contradiction there exist 5 > 0, En ~ 0 +, and xn E IC such that either us,(xn) - u ( x n ) > 5 or u ~ ( x ~ ) - u ( x , ) < -5. We may also assume Xn ---* x and use the continuity of u to get either ~(x) - u(x) >_ 5 or u(x) - u(x) < - 5 , which are both in contradiction to (1.16). 9 As a first simple application of the new stability results to control theory, we improve Corollary III.2.22 on the continuous dependence of the value function Vh of the infinite horizon problem with respect to a small parameter h _> 0 measuring the size of some p e r t u r b a t i o n of the data: the system's dynamics fh, the control set Ah, the r u n n i n g cost gh and the interest rate ,kh. As in w we assume Ah C_ A C_ R M are compact for all h, the Hausdorff distance ds(Ah, Ao) tends to 0 as h --~ 0 +, fh ~ f and gh --* g uniformly on ~ x A for each compact K: __ ]RJr, limh~0 ,~h = A0 > 0. We also assume the data satisfy for all h > 0 the structural assumptions (A0)-(A4) of Chapter III with the same A and a uniform bound M for the r u n n i n g costs gh, but now we allow the constant L measuring the regularity of fh in (A3) and the modulus of continuity we of gh in (A4) to depend on h. In other words, we are now requiring the perturbations to be small just in the sup-norm without bounds on their oscillations. For instance,

gh(X, a) = tO(Z, a) + h p ( x / h L a) with c2 lipschitzean and bounded is allowed for all constants ~,, but it satisfies the assumptions of w only if ~, _< 1 (or if ~ is constant with respect to x . . . ). THEOREM 1.10. Under the preceding assumptions, Vh converges to Vo as h --* 0 +,

uniformly on compact sets of R N. PROOF. We know from w

that Vh satisfies a HJB equation of the form

,'~hVh Jr- gh(x, Dvh) = 0

in ]~N,

where Hh ~ Ho on compact sets of IR2N by the proof of Corollary III.2.22. The limit equation

,kou + Ho(x, Du) = 0

in IRN

satisfies the comparison principle by Theorem 1.3, since A0 > 0. Moreover, sup ~N ]Vh] _< M ~--~

for all h,

and we have a uniform bound because "~h --* A0 > 0. Thus we can apply Corollary 1.8 and get the conclusion. 9 REMARK 1.11. When Corollary 1.8 is applied to problems with ft r N N it may not be easy to prove the convergence of us near the boundary, as required by (1.14). In Section 4 we show that this can be avoided if one knows that the limit problem (1.15) has a solution u E BC(-~). This is done by combining the weak limit technique with a weak notion of b o u n d a r y condition, the Dirichlet b o u n d a r y condition in the viscosity sense. <1

292

v. DISCONTINUOUSVISCOSITY SOLUTIONS

Next we show how we can a d a p t the proofs of the comparison theorems in Chapters II and III to semicontinuous semisolutions. We first consider the simpler case t h a t ~ is b o u n d e d where the technicalities are kept to a m i n i m u m . PROOF OF THEOREM 1.3 FOR ~ BOUNDED. We are going to indicate the changes with respect to the proof of T h e o r e m II.3.1. T h e continuity of Ul a n d u2 is used three times in t h a t proof. T h e first time it is used to produce a m a x i m u m point (xE, y~) for the auxiliary function

9 ~(~, y) = Ul(X)

-

u2(y)

Ix -

yl 2

a n d this still works if (I)~ is upper semicontinuous, as it is under the semicontinuity assumptions on Ul a n d u2. It is used for the second time to deduce (1.17)

Ix~ - y~I2 2E

§0

Ix~ -

0

as ~ ---* 0 +

from (1.18)

Y~I ~

as ~ --, 0 § .

Now we argue as follows. We observe t h a t ul - u2 E U S C ( ~ ) and S : = m_ax(ul - u2) _< ~ ( x ~ , YE) _< Ul(Xe) - u2(ye) ~2 immediately gives (1.17) if we prove that (1.19)

Ul(Xe) - u2(Ye) ~

S

as c ~ O+ .

For this purpose, we assume by contradiction t h a t (1.19) does not hold, so by the compactness of ~ there exists ~k --* 0 +, xek --+ x*, ye~ ---* y* such t h a t e := lim(ul(x~) - u2(y~)) > S. B u t (1.18) implies x* = y*, a n d the upper semicontinuity of (x, y) ~ u l ( x ) - u2(y) gives g < Ul(X*) - u2(x*) _< S, a contradiction. T h e continuity of ul and u2 is used for the last time to prove t h a t g : = l i m s u p (I)~k(x~k, Y~k) --< 0 k for every sequence ek -~ 0+ such t h a t (x~k ~YEk) E O(f~ • f~). To o b t a i n the same conclusion we assume by contradiction t h a t ~ > 0 for some Ek --~ 0 +. By the compactness of f~ a n d (1.18) we can assume xz k --~ ~, Y~k --* ~. T h e n we use the upper semicontinuity of (x, y) ~ Ul(X) - u2(y) to get g < limsup(ul(xE~) -- u2(Y~k)) <_ u1(2) -- u2(~) <_ O, k where the last inequality holds because 5 E cgl't. We have reached a contradiction and completed the proof. 9

1. SEMICONTINUOUSSUB- AND SUPERSOLUTIONS

293

PROOF OF THEOREM 1.3 FOR ~ UNBOUNDED. We outline the variants to the proof of T h e o r e m III.2.12 (see also R e m a r k III.2.14); the case of a H a m i l t o n i a n as in T h e o r e m II.3.5 is left as a n exercise for the reader. In this proof the auxiliary function is

~ ( z , y ) := ~l(x) - ~2(y)

I x - yl 2

~((x)m + (~)m),

2~ where (x) = (1 + Ix12) 1/2, m = min{1, l / K } , a n d K > 0 is a c o n s t a n t appearing in the a s s u m p t i o n s on the H a m i l t o n i a n . We assume by c o n t r a d i c t i o n t h a t u l - u2 is larger t h a n 5 > 0 somewhere, a n d d e t e r m i n e / 3 > 0 so t h a t s u p O > ~/2. Since ~ is u.s.c., a n d negative for large Ixl a n d lYl, it achieves the m a x i m u m at some point (E, ~) = (xE, y~). T h u s the goal is to get a contradiction to 5 ~(xs, YE) _> ~ > 0

(1.20)

by t a k i n g e > 0 very small. Since m a n d ~ have b e e n fixed, it is easy to see t h a t (1.20) a n d the b o u n d e d hess of u l , u2 force xe and y~ to r e m a i n in a compact set 1C for all E > 0 (cfr. R e m a r k III.2.13). T h e b o u n d e d n e s s of u l , u2 implies as before (1.21)

Ix~ - Y~I

,0

as ~ --* 0 +,

~0

as ~ -~ 0 + .

a n d we want to deduce t h a t

Ix~ - y~l 2

(1.22)

2s We f o l l o w as m u c h as possible t h e p r o o f o f t h e p r e v i o u s case w h e r e s was b o u n d e d

by setting

s := ~ 2 [ ( ~

- ~ ) ( x ) - 2~<~> ~1

a n d observing t h a t s < r

yE) < ~ 1 ( ~ ) - ~2(y~) - ~ ( ( ~ ) ~

+ (y~)~) = : r

B y t h e d e f i n i t i o n o f 9 we get (1.22) i f we show t h a t tb(E) -~ S as ~ --* 0 +, a n d

to prove this we assume by c o n t r a d i c t i o n the existence of Ek --* 0 + such t h a t limk ~(zk) > S. By the compactness o f / C a n d (1.21) we can assume xE~ --~ x* a n d YEk ---* X*, thus the s e m i c o n t i n u i t y of u l a n d u2 gives l i m k r < ut(x*) -u2(x*) - 2~(x*) m < S, which is the desired contradiction. If x~k , Yek E ~ for all k of some sequence Ek --~ 0 +, we conclude as in the proof of T h e o r e m III.2.12. Otherwise we use the compactness of/C and (1.21) to select Zk ---* 0 + such t h a t XEk --* X*, Y~k --+ x" a n d observe t h a t x* E O f / b e c a u s e (xe, y~) E 0(ft x f~) for all ~ small. By the s e m i c o n t i n u i t y of ut a n d u2 we get l i m s u p 4~(x~, y ~ ) < l i m s u p ( u l ( x ~ ) - u2(ye~)) ~ Ul(X*) - u2(x*) ~ 0, k k where the last inequality comes from the b o u n d a r y condition u~ < u2 on 0f/. We have got a contradiction to (1.20) which completes the proof. 9

294

V. DISCONTINUOUSVISCOSITYSOLUTIONS

Also the comparison theorems on initial value problems for evolutive HJB equations, Theorems II.3.7, III.3.7, and III.3.15, are still valid if the assumption of continuity of the semisolutions to compare is dropped and we suppose just the upper semicontinuity of the subsolution and the lower semicontinuity of the supersolution. The proofs are obtained by the arguments given above and we leave t h e m to the reader, as well as some extensions: see Exercises 1.5, 1.6, 1.7. The reader might now believe in the validity of a metatheorem saying t h a t every comparison result for continuous viscosity semisolutions extends to semicontinuous ones. This is essentially true when the boundary conditions are achieved in the classical pointwise sense, but it can be false if the b o u n d a r y conditions are understood in some weak sense. For instance, the comparison result for constrained viscosity solutions (Theorem IV.5.8) does not hold for a subsolution in ft, ul E BUSC(~), and a supersolution in ~ , u2 E BLSC(Q), of u + H(x, Du) = O. (The definition of u.s.c, supersolution in ~ is the natural one: just add p E C 1(~) and x E ~ in Definition 1.1.) The counterexample is provided by Example IV.5.3. In fact it is not hard to check (either directly or by a Dynamic Programming argument) that the value functions v~ and vn for the problem with state constraints respectively in ft and ft are, respectively, a 1.s.c. supersolution in ft and an u.s.c, subsolution in ft (both bounded), but v ~ < vn on the half-line of discontinuity. In Section 3 we study the possibility of comparing semisolutions satisfying weak b o u n d a r y conditions. For the constrained boundary value problem we prove that Theorem IV.5.8 holds for semicontinuous semisolutions, provided the subsolution is continuous at each point of the boundary of ft. Exercises

1.1. Let u : ft --* IR be an a r b i t r a r y function. Prove that if D+u(x) r ~ (respectively, D-u(x) r 0), then u is u.s.c, at x (respectively, u is 1.s.c. at x). Compute the sub- and superdifferentials of the characteristic functions of R+ and of Q. 1.2. Discuss the validity of Lemma II.1.8 on the properties of D+u and D - u without the continuity assumptions on u. 1.3. Prove the following extensions of Lemma II.1.7. Assume u E USC(ft) (respectively, u E LSC(ft)). Then p E D+u(x) (respectively, p E D-u(x)) if and only if there exists c2 E c l ( f t ) such that D~(x) = p and u - ~ has a local maximum (respectively, minimum) at x. 1.4 (LIPSCHITZ CONTINUITY OF SUBSOLUTIONS FOR COERCIVE HAMILTONIANS). Assume t h a t for some constants C and K

F(x,r,p) > 0

Vx E a, r E [-K,K], Ipl -> C .

Prove that a subsolution u E USC(ft) of (1.1), such that lu[ _< K , satisfies lu(x) u(y)[ _< C Ix - y[ for all x, y such that the line segment joining them is contained in ft. In particular u is locally lipsehitzean (err. Proposition II.4.1).

2. N O N - C O N T I N U O U S

SOLUTIONS

295

1.5 (COMPARISON PRINCIPLE FOR CAUCHY-DIRICHLET PROBLEMS). Assume T E ]0. ff-oKD],,,~ > 0, ~ C ]~g is open, H ( x , t, p) is continuous on ]~2N+l and satisfies the structure condition (2.18) of Chapter III for each fixed t with the same constants L . K and modulus we. Let Ul 9 B U S C ( ~ x [0, T D and u2 9 B L S C ( ~ x [0, T D be, respectively, a sub- and supersolution of ut + Au + H ( z , t, D x u ) = 0

in ~t x ]0, T[

such that Ul _< u2 on ~ x {0} and on 0~t x [0-T[. Prove that ul _< u2 on ~ x [0~ T[. [Hints: one has to modify appropriately the proof of Theorem III.3.7 using the tricks of the proof of Theorem 1.3. There are two additional difficulties. The first is how to compare at t = T. We suggest to replace ul by ul - s / ( T - t) and then let s ---* 0 +. Alternatively, one can extend ul~ u2 to t = T by taking their semicontinuous envelopes and then prove an analogue of Lemma II.2.10 for these extensions (we warn the reader that this lemma does not hold for arbitrary semicontinuous extensions to f~ x [0~ T]). The second difficulty is the dependence of H on t. This can be overcome by replacing the term (Ix - yl 2 + tt - sl2)/2~ with Ix - y12/2s + It - sl2/2~ in the definition of the auxiliary function (I), and letting r/ go to 0 + before ~.] <~ 1.6 (LOCAL COMPARISON FOR INITIAL "vALUE PROBLEMS). Prove that Theorem III.3.12 remains valid if the continuity assumption on Ul,U~ is dropped and replaced by ul 9 BUSC(C) and u2 9 BLSC(-C). 1.7 (COMPARISON OF LOCALLY BOUNDED SOLUTIONS TO INITIAL VALUE PROBLEMS). Prove that Theorem III.3.15 remains valid if the continuity assumption on ul, u2 is dropped and replaced by Ul 9 USC(]~y x [O, T]), u2 9 LSC(]~ y x [0, T]), Ul - u2 bounded on K: x [0, T] for any compact K:. < 1.8. Prove the following converse of Lemma 1.9: if u~ converge uniformly on a compact K: to a continuous function u as z --* 0 +. then lira i n f , u , = limsup*u, = u . X ~ 0 ~-

~-~0 +

1.9. Let I G R be an open interval. Prove that (i)

u 9 USC(I) is nondecreasing if and only if it is a supersolution of u ~ = 0 in I;

(ii)

u 9 L S C ( I ) is nondecreasing if and only if it is a subsolution of - u ~ = 0 in I (cfr. Lemma II.5.15). 2. N o n - c o n t i n u o u s

solutions

2.1. Definitions, basic properties, and examples In this subsection we extend the notion of viscosity solution to functions that are not necessarily continuous. Note that the extension of the definition of sub- and supersolutions to semicontinuous functions in Section 1 does not change the concept of solution, because a function that is simultaneously a sub- and a supersolution

296

V. D I S C O N T I N U O U S V I S C O S I T Y S O L U T I O N S

according to Definition 1.1 is a u t o m a t i c a l l y continuous. Therefore we need an additional tool, which is the notion of upper and lower semicontinuous envelope of a function u : X --~ ~ = [ - o c , 4-cc], X C_ ]~g. These new functions are, respectively,

u*(x) : = l i m s u p u ( y ) : = lim s u p { u ( y ) : y e X, l Y - xl -< r } , y~x

r~0 +

u , ( x ) := liminf u(y) := lim i n f { u ( y ) : y 9 y---*x

l y - xl <_ r } .

r--~O +

The next proposition collects the p r o p e r t i e s of the semicontinuous envelopes. PROPOSITION 2.1. (i)

u,
(ii)

U,=--(--u)*;

if u is locally bounded, then u*, u, : X ---* ~ satisfy (iii)

(iv) (v)

Su*(x) = inf{ v(x) : v 9 C ( X ) , v >_ u }, Lu,(x) sup{v(x) :v 9 c(x), v < u}; Su*(x) = min{ v(x) : v 9 USC(X), v >_ u },

(u,(x)

max{v(x):v

9 L S C ( X ) , v <_ u};

u is u.s.c, at xo if and only if u(xo) = u*(xo), u is l.s.c, at xo if and only if u(x0) = u,(x0).

PROOF. (i) and (ii) are trivial. T h e most i m p o r t a n t s t a t e m e n t of this proposition is t h a t u* 9 USC(X) and u, 9 L S C ( X ) . This can be deduced from (iii), and then (iv) follows i m m e d i a t e l y from the characterization of upper semicontinuity of v at x by means of the equality v(x) = l i m s u p y ~ v(y). Instead of giving the proof of (iii), which is left as an exercise for the reader, we prove directly t h a t u* is u.s.c. By definition, for any x 9 X and r > 0, there is r > 0 such t h a t

u*(x) + E > s u p { u ( y ) : l Y - xl < r } > s u p { u ( y ) : l Y - z] < r -

Ix -

z] } > u*(z)

for z E X such t h a t Iz - x I < r. T h e n limsupz_~ z u*(z) < u*(x), and so u* is u.s.c. at x. Finally (v) follows easily from the definitions. 4 Now we can give the definition of non-continuous viscosity solution for the equation (2.1)

F ( x , u, Du) = 0

in

with ft C ]~N open and F : ~ x ]R x ]l~N ~ ]l~ continuous. DEFINITION 2.2. A locally b o u n d e d function u : ~ --~ ]R is a non-continuous viscosity solution of (2.1) if u* is a subsolution of (2.1) and u, is a supersolution of (2.1) according to Definition 1.1. <~

2. NON-CONTINUOUS SOLUTIONS

297

This definition makes sense because u* E USC(~) and u, E L S C ( ~ ) by Proposition 2.1 (iv). Note t h a t here the word non-continuous means "not necessarily continuous". In fact, if u E C(f~), then u = u* = u, and the definition coincides with the one used in the previous chapters. From now on the word solution means non-continuous viscosity solution according to Definition 2.2. It is interesting to note that, for Hamiltonians satisfying the structural assumptions of any comparison theorem, a solution of (2.1) may be a discontinuous function only if it is discontinuous at some boundary point. This fact was proved directly for the value functions of optimal control problems with exit times in Proposition IV.3.3, but it is indeed a general property of Hamilton-Jacobi equations, as the next result shows. THEOREM 2.3. A s s u m e the Dirichlet problem for (2.1) satisfies a comparison principle (Definition 1.2). I f u : -~ --* ~ is a bounded solution of (2.1) which is continuous at all x E Of~, then u E C(f}). In particular, f o r any g E C(Of~), there is at most one such solution satisfying u = g on 0 ~ . PROOF. By definition, u* E B U S C ( ~ ) is a subsolution of (2.1) and u, E BLSC(-~) is a supersolution. By Proposition 2.1 (v), the continuity of u at b o u n d a r y points gives u*(x) = u(x) = u , ( x )

for all x E 0f~.

Then, by the comparison principle, u* _< u,

in fl .

Since the opposite inequality is trivial by definition, we obtain u* -- u, and thus u is continuous by Proposition 2.1 (i) and (v). Next we give some simple examples showing that the notion of non-continuous solution is rather weak. The first one shows that nowhere continuous functions can be solutions in t h a t sense of very simple equations such as (2.2)

u' = 0

in ]~.

EXAMPLE 2.4. Let u ( x ) - 1 if x E Q, u(x) - 0 if x E R \ Q, that is, u = XQ. Then u* - 1 and u, - 0, so u is a solution of (2.2). (See Exercise 2.9 for some properties of the solutions of (2.2).) <~ The second example shows t h a t a b o u n d a r y value problem with a unique continuous solution m a y have infinitely many discontinuous solutions. EXAMPLE 2.5. The Dirichlet problem (2.3)

u'=l

in]0,1[,

u(0)=l,

u(1)=2,

has the continuous solution u(x) = x + 1, which is unique by Theorem II.5.9. On the other hand, w ( x ) = x + XQ(X) is also a solution because w*(x) = x + 1

298

V. DISCONTINUOUS VISCOSITY SOLUTIONS

and w.(x) = x are both classical solutions of the equations and the boundary conditions are satisfied. Similarly, for any dense subset X of [0, 1] such that 0, 1 E X , x + ~(x(X) is a solution of (2.3). However, it is enough to require the continuity of the solutions at the boundary points 0 and 1 to have uniqueness as it is easy to prove by the comparison theorems for semicontinuous semisolutions, Theorem 1.3, and the change of variables v := 1 - e -h (Kru~kov transform, see w <~ Next we show that discontinuous value functions of optimal control problems satisfy the associated HJB equation in the sense of Definition 2.2. We have seen in Chapter IV examples of discontinuous value functions for problems with exit times or state constraints. Here we do not refer to a particular control problem because it is enough to assume a local optimality principle. In fact, in Chapter IV we proved the Dynamic Programming Principle without assuming the continuity of the value functions (Propositions IV.2.1 and IV.3.9). We assume that the system with dynamics f and control constraint A satisfies the basic assumptions (A0)-(A3) of Chapter III, the running cost g satisfies (A4) in Chapter III, and A > 0. We give the result for the stationary equation (2.4)

Au + H(x, Du) = 0

in ~,

with (2.5)

H(x,p) := sup{ - f ( x , a ) . p - g(x, a) }, aEA

and ~ c ]I~N open. A similar result holds for evolutive equations, see Exercise 2.1. We recall that .4 denotes the set of measurable control functions, and Yx( 9 a) is the trajectory of the system starting at x and corresponding to a E .4. THEOREM 2.6. Under the preceding hypotheses, assume u is locally bounded in

and that for all x there exists ~- > 0 such that, for all 0 < t < % (2.6)

u(x) = ~cA ~ fotg(yx, c~)e-~S ds + u(yx(t,c~))e-At I .

Then u is a solution of (2.4). PROOF. As in the proof of Proposition III.2.8, the local suboptimality principle (i.e., the inequality < in (2.6)) implies that u* is a subsolution, while the local superoptimality principle (i.e., the inequality > in (2.6)) gives that u. is a supersolution. To prove that u. is a supersolution we take ~ E Cl(fl) and z E ~ such that u.(z) = ~(z) and u,(x) >_ ~(x) for all x in a neighborhood of z. We assume by contradiction A~(z) + H(z, D~(z)) < 0. Then, for some ~ > 0, (2.7)

) ~ ( x ) + H(x, D~(x)) <__- ~

for all x E B(z, z) C_ ~ .

2. N O N - C O N T I N U O U S SOLUTIONS

299

By t h e a s s u m p t i o n s on f a n d A, there is t E ]0, T[ such t h a t yx(s, a) C B ( z , E) for all x E B ( z , E / 2 ) , 0 < s < t, a n d all ~ E A. We fix such a t a n d set

~E(I -

e-~t)/2A

6 := let~2

if A > O, if A = 0 .

By the inequality _> in (2.6), for a n y x there is a E .4 such t h a t

u(x) > - 6 +

1'

/(Yx, a ) e -;~s ds + u(yx(t, a)) e -:~t .

Since u _> u. _> p a n d

d ~ ( ~ ( 8 ) ) e-As = e-~q-~v(y~) + Dv(yx). y'] a.e., we get, for x E B(z, ~/2),

u(x) - ~(x) > - 5 + (2.8)

_> - 5 -

>_ - 5 +

I' /0' /:

e-a~[e(yx, ~) - , ~ ( ~ ) + D~,(y~) 9f(y~, ~)1(8) as e - ~ [ A ~ ( y x ) + H(y~, D p ( y ~ ) ) ] ( s ) d s

e e - ~ ds = 6,

where in the last inequality we used (2.7) a n d the choice of t. T h e n

u . ( z ) = l i m i n f u(x) > ~(z) + 5, a c o n t r a d i c t i o n to the choice of ~ because 5 > 0. T h e proof t h a t u* is a s u b s o l u t i o n is similar. We fix z E f2 a n d ~, E C 1 ( ~ ) such t h a t u*(z) = ~(z) a n d u*(x) <_ ~ ( x ) for all x near z. We assume by contradiction t h a t A~(z) + H ( z , D ~ ( z ) ) > 0 a n d fix z > 0, g E A such t h a t (2.9)

A~(x) - f(x,-a) . O~o(x) - e(x, ~) >_ s

for all x 9 B ( z , E) C_ f t .

We choose t small as before a n d apply the inequality < in (2.6) to the c o n s t a n t control a ( t ) = ~ to get

I' e(yx,~)e-~&+~(yx(t,~))~-~.

~(~) <_

Now we use the inequalities u _< u* _< ~ to perform the same calculation as in (2.8) a n d get, by (2.9),

u(x) - T(x) _< - 2 5 < 0

for all x 9 B ( z , E / 2 ) .

By t a k i n g the l i m s u p as x --+ z we o b t a i n u*(z) < ~(z), a contradiction to the choice of ~o. 9

300

V. DISCONTINUOUSVISCOSITYSOLUTIONS

Now we can give some interesting examples of discontinuous value functions solving the HJB equation. If we are given a closed target T with compact boundary, we recall that tx(a) indicates the first time the trajectory associated with a E .2[ and starting at x hits T, that

T(a) := inf tx((~) aEA

is the minimum time function, and (2.10)

v(x) := inf Jo ftx(~)e-tdt=

{:

ifif+ T(x) O C ,=
is the Kru2kov transform of T. In Section IV.1 we proved that v is continuous in T c if and only if the system is small-time controllable on T (STCT), and in this case v is the complete solution in BC(q -c) of

[u+sup{-f(x,a).Du}-l=O

inT r

(u = 0

on Off".

If S T C T fails, by Theorem 2.6 v is still a solution in the sense of Definition 2.2. Thus we have the following necessary condition of optimality for control laws also in this case, as in w COROLLARY 2.7. A control law A(x) = ax E A defined on an open set ~ C ]~N is time-optimal, that is, T(x) = tx(ax) for all x E ~, only if u(x) := 1 - e -t~(a~) is a viscosity solution in ~ of the HJB equation in (2.11). PROOF. If A is optimal, then u = v in fi and it is enough to apply the D P P Proposition IV.3.9 and Theorem 2.6. ~1 If S T C T fails, however, the problem of uniqueness and comparison for (2.11), related to sufficient conditions of optimality for control laws, is much harder. This is because v is discontinuous at all points of OT where S T C T fails, so the b o u n d a r y condition v = 0 at such points gives very poor information. In fact it is easy to construct a counterexample to the uniqueness for (2.11) similar to Example 2.5, even in cases where there is a complete continuous solution, see Exercise 2.2. In Sections 3, 4, and 5 we see some appropriate interpretations of the boundary conditions which give uniqueness of solution of (2.11) in classes of discontinuous functions. To further motivate our interest in discontinuous value functions we give some explicit examples of time-optimal control problems missing STCT. The first is one of the most classical problems in optimal control theory. EXAMPLE 2.8 (ZERMELO NAVIGATION PROBLEM). Consider a boat moving with

velocity of constant magnitude, which we normalize to 1, relative to a stream of constant velocity. We want to reach in minimum time a given compact target T.

2. NON-CONTINUOUS SOLUTIONS

301

We choose the axes in I~2 so t h a t the s t r e a m velocity is (a, 0) with a > 0. T h e n the system is y~ ---- ~ + a l ~ y~ ---- a2~

with al2 + a 2 = 1. It is easy to see t h a t the system is small-time controllable on a n y target 2- if ~ < 1, so we consider only the case a > 1. Let us first take 7- = {0}. It is easy to see by a n e l e m e n t a r y geometric a r g u m e n t t h a t the reachable set 7 4 : = { z : T ( x ) < + o c } is ~'~ = { X : X 1 < 0 or xt = x2 = 0 }

Ix2]<--Xl(a2--1) -1/2}

74={x:x1<0,

if a = 1, ifa>l.

T h u s in b o t h cases 2- is not contained in the interior of 74 a n d v is discontinuous at least in 0. Actually, it is not hard to c o m p u t e T by s t a n d a r d control-theoretic methods a n d get f - Ix12

if a = l ,

2Xl

T(x)

=

/

--XiCr -- [X22(1 -- 0-2) -F X12]1/2

(

a2- 1

if (T > 1,

see Exercise 2.3. Then, for a = 1, v is discontinuous only at 0, whereas for a > 1 it is discontinuous at each point of 074 (in fact at such points T ( x ) = - x l a / ( a 2 - 1)). If the target is 2- = / ~ ( 0 , 1), by similar a r g u m e n t s we c o m p u t e for a > 1 n = " f i N { X : X 1 "~ 1/a, Ix21 <

(~2 _

1 ) / a + ( 1 / a - x l ) ( a 2 - 1) - 1 / 2 },

and again v j u m p s at all points of 07~.

<~

EXAMPLE 2.9. Consider the system

yl = - a y 2 + 1 - l a l , Y'2 = ayl, with a 9 { - 1 , 0 , 1}, a n d 2- = {0}. The only way to reach 2- by a piecewise s m o o t h control is by first reaching the half-axis S : = { (Xl,0) : x l < 0} a n d t h e n using the control a -- 0. T h u s one guesses t h a t the optimal control is to use a = 1 in the half-plane where x2 > 0 a n d a = - 1 in the half-plane where x2 < 0. A n easy c o m p u t a t i o n shows t h a t the time needed to reach S by such controls does not d e p e n d on Ixl, a n d t h a t the payoff associated with the c a n d i d a t e o p t i m a l control is tx(a) = Ixl + a r c c o s ( - x l / N ) . It is not hard to prove t h a t this is, in fact, the m i n i m u m time function (at least if we restrict ourselves to piecewise s m o o t h controls). Note t h a t in this p r o b l e m 74 = 1~2 the value function is discontinuous at 0, since it has a j u m p of height 7r along the Xl axis at the origin, a n d it is continuous elsewhere. <~

302

v. DISCONTINUOUSVISCOSITY SOLUTIONS

EXAMPLE 2.10. Consider the system of E x a m p l e 2.9 with a 9 {0, 1} a n d 2- = {(1, 0)}. Now one can reach 2- either from the half-axis { (Xl, 0) : xl < 1 }, by using a = 0 as before, or from the arc {Xl = (1 - x~) 1/2 : - 1 _< x2 < 0} by choosing a = 1. Again 74 = R 2. By analyzing the possible trajectories one realizes t h a t the m i n i m u m time function is discontinuous at all points (Xl,0) with 0 <_ xl _ 1. In fact, for xl _< 1 it is easy to see t h a t

T(xl,0) = 1- xl, while for x2 > 0 the best behavior is first using a -- 1 until one reaches x2 -- 0, so

T ( x l , x 2 ) = Ix{ + 1 + a r c c o s ( - x l / I X l ) . T h e points of discontinuity of the value function in lines a n d they are parts of 0T4, in E x a m p l e 2.10 they interior of 74. In any case, a set of discontinuities of T t h a t is not barrier, because no optimal trajectory can cross it. A n p r o b l e m whose target is a half-space instead of a single is in Exercise 2.6.

<3

E x a m p l e 2.8 are two halfare a b o u n d e d arc in the contained in 2- is called a example of a t i m e - o p t i m a l point and having a barrier

2.2. Existence of solutions by Perron's m e t h o d We begin with some properties of envelopes of viscosity sub- or supersolutions that generalize Proposition II.2.1. PROPOSITION 2.11. (a)

Let ,S be a set of functions such that w* is subsolution of (2.1) for all w E S, and define u(x) := sup w ( z ) , wES

x 9 ft.

I f u is locally bounded, then u* is a subsolution of (2.1). (b)

Let Z be a set of functions such that w, is supersolution of (2.1) f o r all w 9 Z , and define u(x) : = inf w(x), wEZ

x 9 ft.

I f u is locally bounded, then u, is a supersolution of (2.1). PROOf. We only prove (a). Indeed (b) can be o b t a i n e d by a similar a r g u m e n t or using the fact that, for a subsolution w of (2.1), v -- - w is a supersolution of

-F(x,-v,-Dv)

= 0

in f~

and vice versa. We fix y 9 fl, ~2 9 C l ( f t ) such t h a t u* - ~2 has a local m a x i m u m at y a n d u* (y) -- ~(y). It is not restrictive to assume, for some r > 0, (u* - ~)(x) _< - I x - y]2

for all x 9 ]~(y, r)

2. N O N - C O N T I N U O U S

SOLUTIONS

303

by a d d i n g Ix - yl 2 to ~. By definition of u*, there is a sequence xn ~ y such t h a t (u - ~)(x,~) > - -

1 n

for all n .

By definition of u there is a sequence {v,~} with v,~ E S such t h a t 1

u(xn) - - < v n ( x , )

for all n .

n

T h e n ( v , - ~ ) ( x , ) > - 2 / n for all n and

(~,

- ~ ) ( x ) < - I x - yl 2

for x e ~ ( y , r ) ,

and thus t h e same inequalities hold w i t h v , replaced by v*. Therefore a m a x i m u m point yn of v* - p i n / ~ ( y , r) satisfies, for n large enough,

--

2 n

< (v* - ~ ) ( y ~ ) ~ - l Y ~ - Yl 2,

and l e t t i n g n ---* + o z we get

and

Yn -"-'* Y

v;~(y~) - - ~ ~ ( y ) = ~ * ( y ) .

Since v* is a subsolution of (2.1) we have F(y~, v*(y~), D~(yn)) < 0, and letting n ---~ oo

F ( y , u * ( y ) , D ~ ( y ) ) <_ O .

~,

T h e following technical lemma, s o m e t i m e s called the Bump Lemma, will allow us to prove t h a t in some cases the sup of subsolution or the inf of supersolution are indeed solutions of the equations. We give it in a form t h a t makes it usable in different situations. LEMMA 2.12. Let u be a function such that u* is subsolution of (2.1) and u. fails

to be a supersolution at some point y. Then, for any 5 > 0 there is ~/ > 0 such that, f o r all r < 7, there exists a function w with the following properties: w* is a subsolution of (2.1), w > u, (2.12)

s u p ( w - u) > 0 fl

and, f o r all x C ft, (2.13)

w(x) = u(x)

if Jx - ~f > r,

(2.14)

w ( x ) < u ( x ) v ( u . ( y ) + 5)

if Ix - yl < r.

PROOF. T h e a r g u m e n t is essentially t h e s a m e as in the proof of P r o p o s i t i o n II.2.1(c). It is not restrictive to assume y = 0. Since u. fails to be a supersolution at this point, there exists T E C l ( f l ) such t h a t 4(0) = u.(0),

~(x) < u.(x)

for small Ixl,

F(O,u.(O),D~(O)) < O.

304

V. DISCONTINUOUS VISCOSITY SOLUTIONS

Define, for E > O,

v(x) := ~(x) + ~ - I x l 2, and observe that, by continuity, v is a classical subsolution of F ( x , v, Dr) = 0 in B(0, r) for sufficiently small E, r > 0. We choose r small enough to have, in addition, (2.15)

v(x) <_ u,(0) + ~ + ~

for ]x I < r .

Note that (v - u)(z) < (v - u,)(x) < (v - ~)(x) <_ 0 for ix] > ~1/2, and thus (2.16)

v(x) < u(x)

for Ix] > r/2

if we choose ~ < r2/4. Moreover, if xn --+ 0 is such that u(x,~) --~ u,(0), we have lim~(v - u)(xn) = E > 0, so, for all r > 0,

(2.17)

sup(v-u)>0. B(0,r)

Now we define W(X) :.~_

and claim by (2.16). cides with Moreover, < ~/2.

fu(x) Vv(x) [u(x)

if Ix I < r , otherwise,

that w has the desired properties. In fact, w(x) = u(x) for Ixl > r/2 Then (2.13) holds and w* is a subsolution of (2.1), because it coinu* for Ixl > r/2, while for ix I < r we can apply Proposition 2.11. (2.12) follows from (2.17), and (2.14) follows from (2.15) if we choose

REMARK 2.13. Of course there is an analogue of Lemma 2.12 for the case when u, is a supersolution and u* fails to be a subsolution. < The first consequence of the last two results is a very general existence theorem for equation (2.1). This is the adaptation to first order equations of the classical Perron's Method for the Laplace equation. THEOREM 2.14. Assume there exists a subsolution ul and a supersolution u2 of (2.1) such that U1 ~ U2. Then the functions (2.18)

U(x) := sup{ w(x) : u 1 ~ W ~ U2, W* subsolution of (2.1) },

(2.19)

W ( x ) := inf{ w ( x ) : ul <_ w <_ u2, w, supersolution of (2.1) },

are (non-continuous viscosity) solutions of (2.1). PROOF. By Proposition 2.11 (a), U* is a subsolution of (2.1). To prove that U, is a supersolution we fix y c ~ and first consider the case that U,(y) = u2(y).

2. NON-CONTINUOUS SOLUTIONS

305

Let ~ E C l ( f t ) be such that U, - p has a local m i n i m u m at y and U,(y) = p(y). Them in a neighborhood of y, (u2 - ~ ) ( x )

> (v, - ~)(x)

_> 0 = (u2 - ~ , ) ( y ) ,

so u2 - c2 has a local m i n i m u m at y. Since u2 is a supersolution we get F(y, u2(y), D~(y)) _> 0, which completes the proof in the case U,(y) = u2(y). The other possible case is

U,(y) + 6 <_ u2(y) - 5

(2.20)

for some 6 > O. If we assume by contradiction that U. fails to be a supersolution at y, then by Lemma 2.12 there is a subsolution w* such that w > U at some point of ~2, and w > U > ul everywhere. Moreover, since u2 is lower semicontinuous, we can choose r > 0 small enough such that

u2(y) - 6 < u2(x)

for all x C ~2, I x - y[ < r .

Then we use (2.20), the properties (2.13) and (2.14) of w, and U < u2, to get w < u2. Then, by the definition of U, w < U in f~, a contradiction to the fact that w > U at some point of ft. Thus U is a solution of (2.1). The proof that W is a solution as well is similar, by means of Proposition 2.11 (b) and Remark 2.13. 9 It is easy to deduce from Theorem 2.14 existence results for initial or boundary value problems associated with (2.1). In fact, if we assume in addition that on F ___Of~ (2.21)

lim u2(x) = lim u l ( x ) = g(y)

X~y

X~y

for all y C F,

then the solutions U and W of (2.1) defined by (2.18) and (2.19) have unique extensions U, W to ft U F, continuous at each point of F, satisfying the boundary condition

U=W=g

onF.

In particular, we have the following result on the Dirichlet problem for (2.1). COROLLARY 2.15. Assume there exist a subsolution Ul and a supersolution u2 of (2.1), both bounded and satisfying (2.21) with F = Of~ and g C C(Oft). Suppose the Dirichlet problem for (2.1) satisfies a comparison principle. Then there exists a (unique) solution u E B C ( ~ ) of (2.1) satisfying the boundary condition u = g on OfL PROOF. We extend ul and u2 by setting ul = u2 = g on Oft. The continuity of g and (2.21) imply ul E BUSC(-~) and u2 E BLSC(-~), thus ul _< us by the comparison principle. Then we can apply Theorem 2.14 to get a solution of (2.1) which can be extended continuously to Oft by (2.21). The conclusion follows from Theorem 2.3. 9

306

v. DISCONTINUOUSVISCOSITYSOLUTIONS

Next we give a stability result for discontinuous viscosity solutions with respect to pointwise convergence of monotone sequences. It is reminiscent of Harnack's convergence theorem for harmonic functions. PROPOSITION 2.16. (i)

If un E US C( ~ ) (respectively, u~ E L S C( ~ ) ), n E N, are subsolutions (respectively, supersolutions) of (2.1) such that u , ( x ) "~ u(x) (respectively, un(x) 7 u(x) ) where u is locally bounded, then u is a subsolution (respectively, supersolution) of (2.1).

(ii)

If, in addition, un are solutions of (2.1), then u is a solution of (2.1).

PROOF. First note that (ii) follows immediately from (i) and Proposition 2.11. We prove (i) for subsolutions. Since u(x) = infun(x) for all x and un e USC(~), we have u E USC(~) (this follows immediately from the fact that u is u.s.c, if and only i f { x : u ( x ) < t } is open for a l l t E ~ ) . Let p 9 be such that u - p has a local max at y, u(y) = p(y). We can assume (u-p)(x)<-]x-yl

2

forx 9

Ix-yl_
by adding Ix - yl 2 to p. Consider the functions

w~(x) := ( u . ( x ) - ~(x) + ix - yl2) + They are u.s.a, functions such that wn(x) --+ 0 + o n / ~ ( y , r). We can apply Dini~s lemma: if a sequence of u.s.c, w~ is such that w , ( x ) "~ w(x) for all x in a compact set and w is continuous, then the convergence is uniform. Then, after relabeling if necessary, we may assume that, for all n,

u~(x) - ~(x) <_ -l - [x - yl 2 n

forx 9

Let yn be a maximum point of un - p i n / ~ ( y , r). Then, 0 = (u - ~ ) ( y ) < (u~ - ~ ) ( y ) < (u~ - ~ ) ( y . )

< 1 n

ly.

yl 2

Hence y , -+ y and u~(yn) --+ u(y). Since un is a subsolution of (2.1)

F(y~, un(y~)~ D~(y~)) < O, so we let n -+ oc and get

F ( y , u ( y ) , D ~ ( y ) ) < O.

4

REMARK 2.17. Another proof of Proposition 2.16 is easily obtained by combining Theorem 1.7 with next elementary lemma, whose proof is left to the reader. This lemma is of independent interest because it relates pointwise monotone limits with the weak limits of Definition 1.4. <~

2. NON-CONTINUOUSSOLUTIONS

307

LEMMA 2.18. A s s u m e the functions uz : E -~ ~, 0 < z < 1, E C_ R N, are upper semicontinuous, us >_ uE, if E >_ ~', and set

w ( x ) = inf ue(x) . z>0

Then w --- w* = limsup*ur E~O +

w. = liminf.uE . r

+

Exercises

2.1 (THEOREM 2.4 FOR EVOLUTIVE H J B EQUATIONS). Assume u is locally bounded in 12 x ]0, T[ and for all (x. t) there exists T > 0 such that, for all 0 < 0 < T,

u(x, t) = inf ~" ~

g(Yx, a) e -As ds + u(yx(~), t - ~) e -~~ ~.. )

aEA ~

Prove t h a t u is a solution of

ut + Au + H ( x , D~u) = 0

in f~ x ]0, T[,

where H is given by (2.5). 2.2. Take g = 1, A = [ - 1 , 1], f ( x , a) = a, T = ] - o c , - 1 ] U [1, +oc[. Check that in this case (2.11) is u +

lu'l

-

1= 0

in ] - 1 , 1 [ ,

u ( - 1 ) = u(1) = o,

and find the unique solution of this problem in C([0, 1]). Show that there are infinitely many discontinuous solutions. 2.3. Compute the optimal controls and the value function for the Zermelo Navigation Problem, Example 2.8, for T = {0} and all a > 0. [Hint: use the Pontryagin Maximum Principle and the existence of optimal controls; check the result in Section 1.17 of [Lei66].] 2.4. Prove that the minimum time function of Example 2.9 is C 1 except on the Xl axis and locally Hhlder continuous of exponent 1/2 in R 2 \ T. 2.5. Guess an explicit formula for the minimum time function of Example 2.10, by analyzing the t r a j e c t o r y associated with piecewise smooth controls. 2.6. Consider the system y~ = y2, y~ = a, - 1 < a < 1 and the target T = { x : xl _< x2 }. Prove that the minimum time function is discontinuous at the points of O T such that x I X2 ~> 1 and on the barrier { x : Xl 1/2 + x~/2, 0 <_ x2 <_ 1 }, and continuous elsewhere. [Hint: use the Pontryagin Maximum Principle and the existence of optimal controls; check the result in Section 5.5.3 of [BaO82].] =

=

2.7. Check directly t h a t the discontinuous value functions found in Exercise 2.3, Example 2.9, and Exercises 2.5 and 2.6 are viscosity solutions of the corresponding HJB equations.

308

V. DISCONTINUOUSVISCOSITYSOLUTIONS

2.8. Give another proof of the sufficiency statement in Petrov's Theorem IV.1.12 by using the comparison principle for non-continuous semisolutions, Theorem 1.3. [Hint: show that u(x) = 1 - e Clxl for Ix] < R, u(x) = 1 elsewhere, is a supersolution of (2.11) for a suitable choice of C and R.] 2.9. Let u : ]R --+ ]R locally bounded. Prove that (i)

if u' = 0 in ]R (in the sense of Definition 2.2), then there exists a E [ - o c , +oo] such that u is discontinuous at all x E ] - a t , a[ and u is constant in ]a, +oc[;

(ii)

if - u ' = 0 in IR, then there is a E [-oo, +~c] such that u is discontinuous at all x E ]a, + c o l and u is constant in ] - c e , a[;

(iii) if u' = 0 and - u ' = 0 in an open interval I, then u is either constant in I or discontinuous at each point of I. 3. E n v e l o p e

solutions of Dirichlet

problems

In this section we consider the Dirichlet boundary value problem

(3.1)

f F ( x , u , Du) =O

in ft,

t.u = g

on 0~t,

with ft C_ ]RN open, F : ~ x ]R • ]l~ N ---+ ]~ continuous and g : Oft --+ JR. Motivated by the theory of optimal control problems with exit times in Chapter IV and the further discussion in Section 2, we restrict ourselves to the case t h a t (3.1) satisfies the comparison principle (see Definition 1.2), so that

(3.2)

if ul and u2 are, respectively, a sub- and a supersolution of (3.1), then ul _< u2 9

We recall that u is a subsolution (respectively, supersolution) of (3.1) if u E BUSC(-~) (respectively, BLSC(-~)) is a viscosity subsolution (respectively, supersolution) of the differential equation F = 0 in fl, which is < g (respectively, > g) on ~9. Our model problem is F(x, u, p) = u + H(x, p), with H satisfying the structural assumptions quoted in Theorem 1.3. This is the case for the HJB equations arising in control problems with payoff involving the exit time of the system from the domain ft, see Chapter IV. We know from Theorem 2.6 that the value function of such a problem is a (non-continuous viscosity) solution of the corresponding HJB equation. It also satisfies the boundary condition in (3.1), where g is the terminal cost to be paid when the system hits the boundary cofL However, this is not very meaningful if it is discontinuous at boundary points, and in fact we saw in Example 2.5 that there are infinitely many such solutions in a very simple problem satisfying the comparison principle. On the other hand, if we require continuity at the boundary, that is, u. = u* = g on 0f~, then any solution is automatically continuous in ~ by Theorem 2.3, and this rules out many interesting problems. In fact, the value functions in Examples 2.8-2.10 are discontinuous at some point of

3. E N V E L O P E SOLUTIONS OF D I R I C H L E T PROBLEMS

309

0~, and indeed we know from Chapter IV t h a t the value function v of time-optimal problems (g --- 0) for systems which are not small-time controllable on the target satisfies v* > 0 at some boundary point, while in problems with incompatible terminal cost g we have v. < g. Therefore we seek a notion of generalized solution of (3.1), and in particular of generalized Dirichlet b o u n d a r y condition, which singles out the value function even when it is discontinuous. There are several ways to do this, and no one is better than the others in all situations. Sections 4 and 5 are devoted to this problem by different methods. The approach we describe in the present section is global, in the sense t h a t we define a generalized solution of the whole b o u n d a r y problem (3.1) and not of the Dirichlet b o u n d a r y condition by itself. The first subsection is devoted to the general theory, and the second deals with the application to the minimum time problem. 3.1. E x i s t e n c e a n d u n i q u e n e s s o f e - s o l u t i o n s We denote S :-- { subsolutions of (3.1) },

Z := { supersolutions of (3.1) }.

DEFINITION 3.1. Let u : ~ --- I~ be locally bounded. (i)

u is an envelope viscosity subsolution of (3.1), briefly, e-subsolution, if there exists S(u) C S, S(u) ~ 9, such t h a t

u(x) = sup w(x),

x E ~;

wes(u) (ii)

u is an e-supersolution of (3.1) if there exists Z(u) C Z, Z(u) ~ 9, such that

u(x) =

inf w(x), wEz(u)

x 9 ~;

(iii) u is an e-solution of (3.1) if it is e-subsolution and e-supersolution.

<3

A trivial example of an e-subsolution of (3.1) is a subsolution u, by taking 8 ( u ) : {u}. Note that, if u is also a supersolution of (3.1), we can take S(u) : S as well (and Z(u) : Z), because in this ease u is indeed the complete solution of (3.1) by the comparison principle (3.2), that is, u : max w = min w . wES

wEZ

At this point it is not apparent whether a discontinuous value function of a timeoptimal problem is an e-solution of the corresponding b o u n d a r y value problem. This is a nontrivial result that we postpone to the next subsection. At the moment the main motivation of the definition of e-solution comes from Perron's method of Section 2, since this method allows us to construct solutions of the equation (3.3)

F(x, u, Du) = 0

in 12

310

V. DISCONTINUOUS VISCOSITY SOLUTIONS

as envelopes of certain sets of sub- or supersolutions. As a first application of the results of w we get some additional consistency properties of e-semisolutions with viscosity semisolutions. PROPOSITION 3.2. (i)

I f u is an e-subsolution of (3.1), then u* is a subsolution of (3.3) and u(x) < g(x) for all x E 0 ~ ;

(ii)

if u is an e-supersolution of (3.1), then u, is a supersolution of (3.3) and u(x) > g(x) for all x E OR;

(iii)

if u is an e-solution of (3.1), then u is a (non-continuous viscosity) solution of (3.3) and u = g on O~t.

PROOF. To prove (i) we observe that u* is a subsolution of (3.3) by Proposition 2.11 (a). If x 9 0~t, then w(x) < g(x) for all w 9 S and so u(x) < g(x). The proof of (ii) is analogous and (iii) is a trivial consequence of Definition 2.2. 9 It is easy to see that the converse of each statement in Proposition 3.2 is false, see Exercise 3.1. Comparison and uniqueness for e-solutions are immediately obtained by the very definition and the comparison principle. PROPOSITION 3.3. Assume (3.2). Then (i)

any e-subsolution u and e-supersolution U of (3.1) satisfy u <_ U in -~;

(ii) /f there exists an e-solution v of (3.1), then it is unique and v=supw= wES

inf W . WEZ

PROOF. For any w E S ( u ) and W C Z ( U ) , w <_ W by (3.2), so that u=

sup w < inf W = U wcS(u) w~z(u)

and (i) is proved; (ii) is an easy consequence of (i).

9

The main result of this subsection is the following existence theorem for esolutions. THEOREIvl 3.4. Assume (3.2) and S ~ O, Z ~ O. (i) I f there exists w E ,9 continuous at each point of O~ and such that w = g on 0 ~ , then (3.4)

U(x) := sup w ( x ) -- rain W ( x ) wE.S

WEZ

is the unique e-solution of (3.1), and there exists a sequence Wn C 8 such that wn/zU.

3. E N V E L O P E SOLUTIONS OF D I R I C H L E T PROBLEMS

311

(ii) If there exists ~ E Z continuous at each point of OR and such that ~ = g on OR, then (3.5)

u(x) := inf W ( x ) = m a x w ( x ) WEZ

wES

is the unique e-solution of (3.1), and there exists a sequence Wn E Z such that

Wn\u. Before proving the theorem we make some comments on its assumptions and conclusions. The assumption 8 ~ 0, Z ~ 0 is of course necessary for the existence of e-solutions. An easy sufficient condition is

F ( x , M, O) > O,

F ( x , - M , O) < 0

for all x r ~t,

for some large constant h i . This is satisfied in the HJB equation (3.6)

Au + sup{ - f ( x , a). Du - e(x, a) } = 0, aEA

if ~ is bounded, by M = A-1 sUp~N• A It]. The assumption on the existence of a continuous sub- or supersolution is not easy to verify for general boundary d a t a g, but it is easily obtained in the homogeneous case g - 0. In fact, w - 0 is a subsolution if F ( x , 0, 0) < 0, and this is indeed the case for time-optimal control, whose equation is (3.6) with ~ - 1. See also Remark 3.5 for a weaker assumption. The characterization (3.4) or (3.5) for the e-solution is optimal. In fact, when a continuous solution exists it is m a x w ( x ) = rain W ( x ) , wE8

WEZ

m

but this formula implies continuity (because s N Z c_ C(Ft)), so it is false in general. In (3.4) we weaken it just to allow U ~ 8, and in (3.5) we allow u ~ Z . However, U E Z and u C 8 so, in particular, U is lower semicontinuous and bounded. Next we turn to the proof, which is a variant of Perron's method. PROOF OF THEOREM 3.4 (i). The function U defined by (3.4) is below any W e Z, by the comparison principle (3.2), so it is bounded and therefore it is an esubsolution by its very definition. We get all the conclusions if we prove that U E Z , and we do it by constructing V E Z such that U :> V, so that U = V by Proposition 3.3 (i). We define, for e > 0 and x r ~ ,

ve(x) := sup{ w ( x ) : w e 8, w(x) = w(x) if dist(x, OQ) < e }, and observe that this function is bounded. Moreover, v~ >_ v5 if E < ~, so we can define V(x) :=

312

V. DISCONTINUOUSVISCOSITYSOLUTIONS

Note that U > vs > (v~). by definition, so U > V. We want to prove that V E Z. For this purpose we define ~tE := { x E ~ : d i s t ( x , 0 f t ) > z}, and claim that (v~). is a supersolution of F = 0 in ~E. Then V is also supersolution in f~ by Proposition 2.16 (recall that (vl/n). ,7 V), thus it is a supersolution of F -- 0 in ft. Moreover, for x E c9~, (vs).(x) = (ve)*(x) = g(x) for all ~ because ve - w in a neighborhood of x, and w is continuous at x with w = g(x), so we conclude that V E Z. To prove the claim, we first observe that (vc)* is a subsolution of (3.3) by Proposition 2.11. We assume by contradiction that (%). fails to be a supersolution at y E ~ . We can apply Lemma 2.12 to get, for any small r > 0, a subsolution w* of (3.3) such that (3.7)

sup(w - vE) > O, f~

and w ( x ) = v~(x)

for Ix - ~l > r

We choose r < dist(y, 0~) - E. This gives w(x) = w(x) if dist(x~ 0f~) < E. Since w E USC(~) w e g e t w*(x) = w(x) if dist(x, 0ft) < ~, and so w* E $. Therefore, by definition of vs, we have w* < v~. This is a contradiction to (3.7), which proves the claim. To complete the proof we define wn :-- (vu,~)*, and observe that this is a nondecreasing sequence in S whose pointwise limit is > V by the definition of V. On the other hand Wn _< U by the definition of U, and we have shown that U -- V, so

w~TU.

9

REMARK 3.5. We can prove (3.4) and the fact that U is the e-solution of (3.1) under the weaker assumptions that S and Z are nonempty and for all z E 0f~ there exists wz E S such that l i m x ~ wz(x) = g(z). The proof is indeed simpler than that of Theorem 3.4. In fact, by the definition of U, wz <_ Us so U.(z) >_ g(z) for all z E Oft. By Theorem 2.14, U. is a supersolution of (3.3), thus U. E Z. This implies U. = U E Z, which completes the proof. <~ The next result gives a characterization of the e-solution in terms of pointwise limits of sequences in G and Z. PROPOSITION 3.6. Under the assumptions of Theorem 3.4, a function v is the esolution of (3.1) if and only if there exist sequences Wn E 8, Wn E Z, such that

w,~(x) --* v(x) and Wn(x) --* v(x) for all x E -~. PROOF. The "only if" part follows immediately from Theorem 3.4 and in fact one of the two sequences can be chosen constant and the other monotone. To prove the converse we observe that there cannot be any point y where v(y) < infwe z W ( y ) or v(y) > supwes w(y). By Theorem 3.4 the e-solution is infw~ z W = supwes w, so v must coincide with it. 9

3. ENVELOPE SOLUTIONS OF DIRICHLET PROBLEMS

3.2. T i m e - o p t i m a l

313

problems lacking controllability

The main examples of discontinuous solutions of an HJB equation in Section 2 were the minimum time function T and its discounted version (or Kru2kov transform) v, see (2.10) and Chapter IV. The Dirichlet boundary value problem associated with v is

(3.8)

:+H(x, Du)=O = 0

i n : / - c, on 07.,

with (3.9)

H(x,p) : = s u p { - f ( x , a ) . p }

- 1,

aEA

where f is the dynamics of the controlled system and T the target. We assume f : ]I~y X A ~ ]~N continuous, T closed and nonempty, and, for simplicity, (3.10)

]f(x,a)-f(y,a)l<_Lix-y I

forallx, y,a,

A and 07. compact. Exercise 3.3 gives some extensions, and in Section 5 we study by different methods the case of nonconstant running and terminal costs (see Corollary 5.10). The main result of this subsection is the following. THEOREM 3.7. Under the previous assumptions v. is the e-solution of (3.8). Before turning to the proof we make some comments. The Dirichlet problem (3.8) satisfies the comparison principle (3.2) by Theorem 1.3, and u - 0 is a continuous subsolntion of (3.8), so the assumptions of Theorem 3.4 (i) are satisfied. Therefore the e-solution is a supersolution of (3.8) (in particular, it is ].s.c.), so v. is the natural candidate. In fact it is a supersolution of the P D E by Theorem 2.6, and v. _> 0 on 07" because v > 0 everywhere by definition. If, in particular, the system is convex (i.e., f(x, A) is a convex set for all x E ]~N), then T is lower semicontinuous (see, e.g., [HL69]) and so v = v.. W h a t is left to prove is t h a t v. is an e-subsolution, that is, it can be approximated from below by subsolutions of (3.8). We construct approximations having a control-theoretic interpretation: roughly speaking, we increase the directions available to the system so that it becomes controllable and the value function is Lipschitz continuous. The first preliminary result says t h a t the minimum time function does not change if we add the null vector field to the system. PROPOSITION 3.8. L e t A = AU{~}, ~ ~ A, jT(x, a) = f(x,a) ira E A, f ( x , ~ ) ~ 0 , for all x. Then the minimum time function T associated with the system (f, A) coincides with the minimum time function T associated with (f, A). SKETCH OF THE PROOF. The result is rather intuitive. Of course T _< T. Fix x such t h a t T(x) < +oc, e > 0, and c~ E .A such that t~(a) < T ( z ) + e. We are going to construct a control ~ E .A which, roughly speaking, skips all the times t such

314

V. DISCONTINUOUSVISCOSITYSOLUTIONS

that c~(t) -- ~. Since at such times the velocity of the system is null, we expect to reach the target in a time not larger than tx(c~). We define J := {t > 0: a(t) = ~ } ,

Jr := [0, oc[ \ J,

r(t):=t-meas([O,t]nJ)=

Xjc(s)ds,

T-l(s):=min{t>_O:T(t)=s},

:=

We claim that yx(T(t),/3) = yx(t, a) for all t > 0. Then T(x) <_ t=(~) <_ T(tx(a)) < t=(a) < T(x) + z and we conclude because z is arbitrary. The proof of the claim is left to the reader (see Exercise 3.2). We approximate the system (f, A) with the controllable system (3.11)

~y'=f(y,a)+h~, [y(0)

= x,

where h > 0. The control functions are (c~,/3) e A x/3, where B :-- { ~ : [0, oc[ ---, /3(0, 1) measurable }, and the trajectories are denoted by yh(. a, ~). The value functions are

Th(x) := inf t h, .AxB

Vh(X) := inf

e -t dt,

~xt3 JO

where t h -= th(a,/3) is the entry time in 7- of the trajectory of (3.11). PROPOSITION 3.9. For all h > O, Vh E BC(]~ N) is a subsolution of (3.8) such that

Vh -- 0 on T. PROOF. Since the system (3.11) can go in any direction, Th is locally Lipschitz continuous by Theorem IV.4.8 (in special cases simpler controllability results suffice: Theorem IV.l.12 if T = {0}, Theorem IV.1.16 if the target is smooth). Thus Vh is a continuous solution of

u + ~I(x, Du) + h IDu[ = 0

in T r

where ~I(x,p) = H ( x , p ) v ( - 1 ) is the Hamiltonian associated with the system (f, A) as in (3.9). Therefore Vh is a subsolution of (3.8). The proof of Theorem 3.7 follows immediately from the following result, which is of independent interest. THEOREM 3.10. Under the assumptions of Theorem 3.7

v.(x) = supvh(z) h>0

for all x E ]~N .

3. E N V E L O P E SOLUTIONS OF D I R I C H L E T PROBLEMS

315

PROOF. By P r o p o s i t i o n 3.8 we can a s s u m e w i t h o u t loss of generality t h a t there is E A such t h a t f(x,~d) = 0 for all x, a n d drop the - on f a n d A. Since vh _< v for all h a n d Vh is continuous, we have suPh v h ~_ v,. Let us assume by contradiction there exists 5 such t h a t (3.12)

sup Vh(~) < v , ( ~ ) . h

T h e n there exists e, h* > 0 a n d (ah, ~h) E ,4 X B such t h a t (3.13)

1 - e -t~(ah'~h) < Vh(-2) + e < v , ( ~ ) -- e

for 0 < h < h* .

We simplify the n o t a t i o n s as follows:

~h := t~(~h,Zh),

yh(t) := y~(t, ~h,~h).

If Vh(~) --- 1 as h -* 0 + we get a c o n t r a d i c t i o n to (3.12) because v, < 1. Otherwise, there exists 5 > 0 such t h a t Vh(5) < 1 -- 5 for a n y h > 0, so T := suph>o 7h < + o z a n d there exists ~h E 0 T such t h a t yh(Th) = ~h. By the compactness of 0 T we m a y assume ~h ~ ~0 E O T as h --- 0 +. Now we solve the system (3.11) backward s t a r t i n g at ~0, t h a t is, we consider the solution yh( " ) of y' = f ( y , ah) + h~h,

Y(~h) = ~o, a n d the solution Yh( 9 ) of y' = I ( y , ~ h ) , y(~h) = ~ o . For 0 < t < 7h we get from (3.10) lyh(t) -- yh(t)l <_

L lyh(s) -- yh(s)l ds + hT,

hence, by Gronwall's inequality, lyh(t) - yh(t)l <_ hTe L(rh-t). T h e n we set C1 : = "re Lr a n d get (3.14)

]yh(0) -- ~h(0)] _< C l h .

O n the other hand, since Yh a n d yh are the t r a j e c t o r y of system (3.11) passing at time Vh at, respectively, ~0 a n d ~h, a s t a n d a r d estimate gives ]yh(t) -- yh(t)] <_ I~O -- ~hl e l f . Then, for C2 : = e L~, we have lYh(0) - 51 = lYh(0) -- ~h(0)l _< C21~0 - ~hl P u t t i n g this estimate together with (3.14) we get (3.15)

~h(0) ~

~

as h -*

0 +

.

By definition of Yh we have Sh : = t~h(O)(C~h) <_ vh = t h ( a h , ~ h ) ; hence, by (3.13), v,(~) - v(~h(0)) :> e + 1 - e -~h - (1 - e -~h) > e . Then, by (3.15), v,(~) > v , ( ~ ) + e, a contradiction.

4

316

v. DISCONTINUOUSVISCOSITY SOLUTIONS

REMARK 3.11. It is easy to give verification theorems for time-optimal controls similar to Corollaries III.2.18-2.19 by taking e-subsolutions of (3.8) as verification functions and e-supersolutions as falsification functions. <~ Exercises

3.1. Find a counterexample to the converse of each statement of Proposition 3.2. [Hint: use Example 2.5.] 3.2. Complete the proof of Proposition 3.8. (This is essentially an exercise in measure theory. The result can be checked on [BSt93].) 3.3. Extend Theorem 3.7 to the case where the Hamiltonian is H(x,p)

=

sup{

- f(x~a)

.p -

((x,a)

},

(1

where e _> 1 is a running cost satisfying the hypotheses of (A4) in C h a p t e r III, to the case of cOT unbounded, and to systems (f, A) satisfying the assumptions (A0)-(A3) of Chapter III. 3.4. Consider the necessary condition of time-optimality for control laws given in Corollary 2.7. (i)

Find a stronger necessary condition which is also sufficient;

(ii) state precisely the verification theorems alluded to in Remark 3.11. 4. B o u n d a r y

conditions

in the viscosity sense

In this section we continue the study of the Dirichlet boundary value problem

(4.1)

{

F(x,u~Du)=O

in~,

u = g

on 0~.

in cases where a solution satisfying the boundary condition in the classical sense (i.e., u* = u, = g on 0 ~ ) may not exist. While in Section 3 we gave a global notion of generalized solution of (4.1), here we give a local definition of noncontinuous viscosity solution of the boundary condition, and use the definition of solution given in Section 2 for the PDE. In subsection 4.1 we show that these definitions have good stability properties and apply to value functions via standard Dynamic Programming arguments. In subsection 4.2 we give some comparison and uniqueness results when 012 is a Lipschitz hypersurface and give some applications. The first is a further study of the value functions of control problems involving exit times from f~. The second is a convergence theorem that completes the stability results of w In subsection 4.3 we study the non-continuous complete solution of the boundary value problem for time-optimal control. Throughout the section we assume ~t C_ ~ N open, F : ~ • ]~ • ~ N __, ~ continuous, g : c9~ --, ~ continuous.

4. BOUNDARY CONDITIONS IN T H E VISCOSITY SENSE

317

4.1. M o t i v a t i o n s a n d b a s i c p r o p e r t i e s We begin with an example of an optimal control problem which provides a first motivation to the definition of the Dirichlet boundary condition in the viscosity sense. Consider a system with dynamics f and control constraint A satisfying the basic assumptions of (A0)-(A3) of Chapter III. Let 7- be a closed target with nonempty interior int 7- and denote by t'x(a) the entry time in int 7. of the t r a j e c t o r y y~( 9 ) associated with the control function a E ,4 as in w of Chapter IV. Consider the value function (4.2)

~(x) := inf {t'~(c~) + g(yx(t'x(c~))) } aE.A

and assume it is continuous and finite everywhere. In w of Chapter IV we gave the Dynamic Programming Principle for ~ which can be written as (4.3)

~(x) = inf { t + ~(yx(t, ~)) }

if 0 < t < t ' x ( a ) .

aEA

By a s t a n d a r d argument we deduce from it the HJB equation in the viscosity sense

H(x, D~) = 0

(4.4)

in ft := 7-c

where

H(x,p) :-- s u p { - f ( x , a ) . p }

- 1.

aEA

We claim t h a t ~ satisfies (4.5)

(V - g) V H(x, Dr) > 0

on 0f~,

in the viscosity sense: see the definition of constrained viscosity solution in w of Chapter IV. In fact, if F is the part of 0f~ where ~ < g, it is not profitable to enter int 7- through F and pay the terminal cost g. Therefore, the control problem with the state constraint t h a t forbids trajectories to enter int 7- through F has the same value function. Then ~ satisfies the boundary condition H(x, D~) >_ 0 of state constrained problems on F. This proves (4.5). Next we claim t h a t ~ also satisfies (4.6)

(~ - g) A H(x, D~) <_0

on 0f~,

in the viscosity sense. To see this, let x E 0f~ be such that ~(x) > g(x). Then the system cannot enter intT- at x, t h a t is~ t'x(c~) > 0 and yx(t,(~) E ~ for all c~ E .4 and t > 0 small. Thus for a C 1 function p such that ~ - ~1~ has a max at x we have, for small t, (~-~)(yx(t,~))

< (~-~)(x)

.

Now the D P P (4.3) gives

~(x) - ~(yx(t, ~)) < t

for 0 < t < t'x(~),

318

V. DISCONTINUOUS VISCOSITY SOLUTIONS

and by the usual a r g u m e n t (divide by t > 0 and let t --~ 0 +) we get H(x, D~(x)) < 0, which gives H(x, D~) < 0 on 0~t in the viscosity sense and proves (4.6). In conclusion, ~ satisfies the b o u n d a r y value problem (4.4) (4.5) (4.6), which we s u m m a r i z e with the following n o t a t i o n

H(x, Du) = 0 u=gorH(x,

in Ft, Du)=0

on0f~.

M o t i v a t e d by this example we give the following DEFINITION 4.1. A function u E USC(Ft) (respectively, LSC(~)) is a (viscosity) subsolution (respectively, supersolution ) of (4.7)

u=g

or

F(x,u, Du)=O

on0fl

if, for any ~2 E C 1 (~) and x e 0~t such t h a t u - ~ has a local m a x i m u m (respectively, m i n i m u m ) at x,

(u - g)(x) < 0 (resp., _> 0)

or

F(x, u(x), D~(x)) < 0 (resp., :> 0).

We also express this p r o p e r t y by saying t h a t subsolutions (respectively, supersolutions) of (4.7) satisfy

(u - g) A F(x, u, Du) <_0

on OFt

(u - g) v F(x, u, Du) >_0

on OFt)

(respectively,

m the viscosity sense. We say t h a t u is a v-subsolution (respectively, v-supersolution) of the Dirichlet p r o b l e m (4.1) if it is a (viscosity) subsolution (respectively, supersolution) of

~F(x,u, Du) = 0 (4.8)

[u=gorF(x,u,

Du)=0

in Ft, on0f~.

Finally, a locally b o u n d e d function u : fl ---* ]R is a (non-continuous viscosity) solution of (4.7) (respectively, (4.8)) if u* is a subsolution and u. is a supersolution of (4.7) (respectively, (4.8)). <~ REMARK 4.2. Definition 4.1 can be reformulated more concisely. Define G : ~ x R x R g ~ ]R by setting

~F(x,u,p) G(z,u,p)= ( ( u - g ) ( x )

i f x c f~, ifxE0~.

Note t h a t G . = G* = F in Ft, while G.(x,u,p)= ( u - g(x))A F(x,u,p) and G*(x,u,p) --- ( u - g(x))v F(x, u,p) on OFt. T h e n Definition 4.1 is equivalent to the following definition of viscosity subsolution (respectively, supersolution) of (4.9)

G(x, u, Du) = 0

in ~ .

4. BOUNDARY C O N D I T I O N S IN T H E VISCOSITY SENSE

319

A function u E USC(~) is a subsolution (respectively, u E LSC(~) is a supersolution) if for any ~ E C 1 ( ~ ) such t h a t u - ~ has a local maximum (respectively, minimum) at x G . (x, u(x), D ~ ( x ) ) < 0

(respectively, G* (x, u(x), D ~ ( x ) ) > O) .

Therefore interpreting the b o u n d a r y condition in the viscosity sense is a special case of defining viscosity solutions of discontinuous equations in non-open sets, see Exercise 4.1. <3 Note t h a t subsolutions (respectively, supersolutions) of (4.1) as defined and used in Sections 1 and 3 are obviously v-subsolutions (respectively, v-supersolutions) but the converse is not true in general. For instance, the value function ~ of the previous motivating example is a v-subsolution but not a subsolution on a part of 0 T where the system is not small-time locally controllable. A more explicit example is the following. EXAMPLE 4.3. Consider the Dirichlet problem u' - 1 = 0 u(0) = O, u(1) = c .

(4.10)

in ]0, 1[,

This is a special case of the motivating example by taking f ( x ) ~ - 1 and A a singleton. Clearly, ~(x) = x for all c E ~ , so ~ is not a subsolution of (4.10) in the sense of w if c < 1, and it is not a supersolution if c > 1. It is easy to check directly t h a t ~ is a v-solution of (4.10). <3 The notion of v-subsolution (respectively, v-supersolution) of (4.1) is also weaker t h a n t h a t of e-subsolution (respectively, e-supersolution) given in w as the following proposition and example show. PROPOSITION 4.4. I f u is an e-subsolution (respectively, e-supersolution) of (4.1), then u* is a v-subsolution (respectively, u. is a v-supersolution) of (4.1), that is, u* is a viscosity subsolution (respectively, u. is a supersolution) of (4.8). PROOF. By Proposition 3.2 we must check only the b o u n d a r y condition. By definition, u(x) = suPwes(u ) w(x), where each w E S(u) is a subsolution of the PDE in (4.1) and w < g on 0~. The proof is a variant of that of Proposition 2.11. We fix y E 0 ~ such t h a t u*(y) > g(y) and take ~ E C1(~) such t h a t u*(y) = p(y) and, without loss of generality,

(u*-~)(x)<_-Ix-yl

2

forxE~NB(y,r).

We must prove t h a t (4.11)

F(y, u*(y), D p ( y ) ) <_ O .

Select a sequence xn -~ y such t h a t (u - ~)(x~) _> - -

1 n

for all n,

320

V. D I S C O N T I N U O U S V I S C O S I T Y S O L U T I O N S

and v,~ C S ( u ) such that 1 u ( x , ) - - < vn(xn)

for all n .

n

Then a maximum point y~ of Vn - ~ in ft A B ( y , r) satisfies, for n large, 2

- - - - < (Vn -- ~ ) ( Y n ) ~ --lYn -- Yl 2,

which gives

y. ~

y

and

v~(y~)

, ~ ( y ) = u*(y) > g ( y ) .

Thus Yn ~ Of~ for n large enough, because Yn E Of~ would imply v,(y,~) < g(yn) and therefore a contradiction to the continuity of g by letting n ~ +ec. Then F(y~; v~(y~), D ~ ( y ~ ) ) <_ 0 and we get (4.11) in the limit.

9

EXAMPLE 4.5. Consider the Dirichlet problem

(4.12)

fu+lDul-l=O inRN\{0}, [u(O)

=

0

The complete solution in B C ( ~ N) of this problem is v(x) = 1 - e-lXJ thus it is also the e-solution of the problem. Any constant function w - c _< 1 is a v-subsolution of (4.12) but it is not an e-subsolution because it is above v in some neighborhood of 0. Note that w = 1 is indeed a v-solution of (4.12) and w > v everywhere. <~ The key point of this example is that ~ = ]~N that is, in terms of the target for time-optimal control, int 7- = 0. See Exercise 4.2 for a generalization. A n example where i n t ~ = ~, that is, int 7- = 7-, will be given soon (see Example 4.15). The next property of v-sub- and v-supersolutions has a proof very similar to Propositions 2.11 and 4.4 and we leave it to the reader. PROPOSITION 4.6. Let :F be a set of functions such that w* is a subsolution (respectively, w , is a supersolution) of (4.8) for all w E .T. Set u(x) := sup w ( x )

(respectively, inf)

f o r x E f~,

wE.T

and assume u is locally bounded. supersolution) of (4.8).

Then u* is a subsolution (respectively, u, zs a

Next we show the very good stability properties of v-solutions. This is one of their main features, since the weak limits of essentially any perturbation of (4.1) are v-sub- or v-supersolutions of (4.1). In the next subsection we give comparison

4. BOUNDARY CONDITIONS IN T H E VISCOSITY SENSE

321

theorems for v-semisolutions t h a t allow us to prove, u n d e r some additional assumptions, the coincidence of the weak limits a n d therefore the uniform convergence of the solutions of the following three p e r t u r b e d problems (see Corollary 4.27). We begin with regular p e r t u r b a t i o n s , as in Section 1. Assume F~ : ~ x R x N N ---* N a n d g~ : Oft --~ R are continuous, 0 < E _< 1, and (4.13)

f F ~ ::t F

on c o m p a c t a of ~ • R x ~ g ,

[ g s :=~ g

on c o m p a c t a of 0ft, as E ~ 0 + .

We consider the b o u n d a r y value problem

(P~)

~Fs(x, u, Du) = 0 I.u=g~ or F~(x,u, D u ) = O

in ft, on0ft.

For the function ue : ~ ---* R assume for each c o m p a c t u m K: C_ (4.14)

sup [us[ _< C~:

for 0 < z < 1,

t:

so t h a t the weak limits u_ = l i m i n f . u e ,

~ = lim sup*ue

s--*0+

s---*0+

are locally b o u n d e d in ~ (see Definition 1.4). PROPOSITION 4.7. Assume (4.13).

If uE satisfying (4.14) are such that u~ is a subsolution (respectively, us. is a supersolution) of (Pz) for s > O, then the upper weak limit ~ is a subsolution of (4.8) (respectively, the lower weak limit u_ is a supersolution of (4.8)). This result is b e t t e r u n d e r s t o o d if we look at a simple example where u~ = g on 0ft in the classical sense for all e > 0. EXAMPLE 4.8. Consider the Dirichlet problem max{u',-zu'}-l=0

in]0,1[,

u(O) = u ( 1 ) = O.

This is the HJB e q u a t i o n for a t i m e - o p t i m a l control problem whose u n i q u e viscosity solution in BC([0, 1]) is us(x) = x A (1 -- x)/z. T h e n ~(x) = x, so K satisfies the b o u n d a r y condition at x = 1 only in the viscosity sense. Note t h a t ~ is indeed a continuous v-solution of the limit p r o b l e m

(4.15)

[u

'-I =0 [u = 0

in ]0, 1[, on { 0 , 1 } .

O n the other hand, u_(x) = g(x) for x < 1, _u(1) = 0, so _u is the pointwise limit a n d the s u p r e m u m of us as ~ --~ 0 +. Since uz is a subsolution of u' - 1 = 0, u is the e-solution of (4.15), as well as a v-solution. <~

322

v. DISCONTINUOUSVISCOSITYSOLUTIONS

Before giving the proof of Proposition 4.7 we extend a bit Lemma 1.6. Since the proof requires minor modifications we leave it to the reader. LEMMA 4.9. Let uE : ~ ~ tR satisfy (4.14) for any compactum ]C C_ ~ , ~ E C1(~), and assume ~ - ~ attains its m a x i m u m in 12 N B(xo,r) only at xo. Then there exist sequences Cn --~ 0 + and xn such that xn is a m a x i m u m point f o r u*~, - ~ in f~ N B ( x o , r) and

xn ----* xo

and

u*~,(xn) ~

~(xo)

as n --* c o .

PROOF OF PROPOSITION 4.7. The proof is a variant of that of Theorem 1.7. Let us check the boundary condition. Let c2 E C 1(~) be such that x0 E 012 is the unique maximum of ~ - ~ in f~ M B ( x o , r) and consider the sequences cn, Xn produced by Lemma 4.9. If there are subsequences, still named En, x~, such that xn E 0f~ and u;n(x~) < gE,(xn), then ~(x0) < g(xo) because g~ ~ g. For all other cases

F~o(x~, uio(x~), D ~ ( ~ ) ) _< 0 because u* is subsolution of (PE), and we conclude by (4.13). The next example concerns singular perturbations of the PDE via vanishing viscosity. Suppose u~ E C2(f~) A C(~) are solutions of the elliptic boundary value problem

(4.16)

S-zaij(x)u~,,j (u = g

+ F ( x , u, Du) = 0

in ft, on 012,

satisfying the uniform bound (4.14) in any compact K: C_ ~. Here z > 0, the coefficients aij are bounded in 12; the matrix (aij(x)) is nonnegative definite for all x E f~, we have used the summation convention, and ux~xj = 02u/OxiOxj. The most classical example is aijux~xj ~ A n , the Laplacian of u, and the term - ~ A u is called artificial viscosity. For the existence of such solutions we refer, for instance, to [GT83, Kry87]. PROPOSITION 4.10. Under the previous assumptions, the upper weak limit ~ of the solutions of (4.16) is a subsolution of (4.8) and the lower weak limit u_ is a supersolution of (4.8). PROOF. Let ~ E C2(~) be such that x0 is the unique maximum point of ~ in ~ (1 B(xo, r) and consider the points xn and the sequence En produced by Lemma 4.9. If x~ e 0~2 o n a subsequence, then uE~(xn) = g(xn), and we get ~(xo) = g(xo). Otherwise, for all small z, xn E 12 is an interior maximum of up. -- ~. Then D u ~ (x~) -- D ~ ( x n ) and the Hessian matrix D 2(u~n - ~2)(x~) is nonpositive definite. Since aij (x) is nonnegative definite, by standard linear algebra

--aij(xn)(uE. -- ~)~i~j(xn) _> 0,

4. B O U N D A R Y C O N D I T I O N S IN T H E V I S C O S I T Y SENSE

323

and from the P D E in (4.16) we get

-:a~j(x~)~x,~ (~) + F(x~, u~, (x~), D~(x~)) <_O. Finally, we let :~ --+ 0 + and use the boundedness of aij to obtain

F ( x o , g(xo), D ~ ( x o ) ) < O . Proposition 4.10 gives a motivation for naming "viscosity sense" the boundary condition defined in Definition 4.1. In fact, this boundary condition is satisfied by the limit of the vanishing viscosity approximation, even when a boundary layer occurs. We give a simple one-dimensional example to better illustrate this phenomenon. EXAMPLE 4.11. Consider the problem

-:u"-u'+u-l=O

in ]0, 1[,

u(O) = u ( 1 ) = O.

The solution of this problem is

u~(x) = 1 + (e x - e " ) - l [ e Ax+" - e "x+A - e )~x + e"X], where - 1 + ~/1 + 4 : =

A(:) =

2:

- 1 - ~/1 + 4~ '

~ =

~(:) -

2:

A s : --+ 0 +, ~ --+ 1 and p -+ - o % so ~(x) = 1 - e x - 1 satisfies the b o u n d a r y condition at x = 0 only in the viscosity sense. Note that u(x) = ~(x) for x > 0, u(0) = 0,

so both ~ and u__are solutions of the limit problem (4.8) with F ( x , u, p) = - p + u - 1, g = 0, fl = ]0, 1[. The boundary layer is the small interval on the right of 0 where u: is not close to g even for : > 0 small, because it is "pulled down" by the boundary condition. <~ The last example of stability property in this section concerns perturbations of the domain. Assume flE C_ f~ are open, 0 < : _< 1, and their complements TE := f~c are bounded and converge to T := f~c in the Hausdorff metrics (see w in Chapter III), that is, lim d H ( ~ , T ) = 0 . S~0 +

Assume g E C(~), and let u~ be solutions of (4.17)

I F ( x , u, D u ) = 0 l u=g

or

F(x,u, Du)=O

in f~:, on0f~s,

for : > 0. Extend u~ to ~ by setting us := g off f~s and assume us are uniformly bounded for : > 0 on every compact K: C_ ~, that is, (4.14) holds.

324

V. DISCONTINUOUS VISCOSITY SOLUTIONS

PROPOSITION 4.12. Under the previous assumptions, the upper weak limit ~ of the solutions of (4.17) is a subsolution of (4.8), and the lower weak limit u is a supersolution of (4.8). The proof of this result is a variant of previous ones~ and we leave it as an exercise for the reader. We end this subsection with the proof that the value functions v and ~ of the control problems studied in w are solutions of the boundary value problem

~ )~u + H(x, Du) = 0 [u=g or A u + g ( x , Du)=O

(4.18)

in f~, on Off,

where

H(x,p) := sup{ - f ( x , a ) . p - e(x,a) },

(4.19)

aEA

even if they are not continuous. The payoffs of the optimal control problems under consideration are

J(x, ~)

:=

if0tx(a) ~(yx(s)~ ~(s)) e -~s ds + e-~t~(~

where tx(a) is the first time the trajectory Yx( ", ~) hits the target :Y := f~c and J(x, a) is obtained by replacing tx(a) with the entry time in int 2r, t'~(a), in the definition of J. We recall that .4 := { a : [0, ~c[ ~ A measurable }. THEOREM 4.13. Assume the hypotheses (A0)-(A4) of Chapter III and g e BC(Of~).

Then the value functions v(x) = inf J(x, a),

~(x) = inf J(x, ~)~

wEA

c~E.A

are both viscosity solutions of (4.18) with H defined by (4.19). PROOF. Since ,~ > 0 we can assume without loss of generality A = 1. Under the current assumptions v and ~ are bounded and satisfy the Dynamic Programming Principle, Proposition IV.3.9. Then they satisfy the PDE in (4.18) by Theorem 2.6. We prove that v is a supersolution of the boundary condition in (4.18), all the other proofs being similar (and sometimes easier). This proof is a refinement of that of Theorem 2.6. We take z E 0f~ and 9~ E C1(~) such that v.(z) = 9~(z) and v.(x) >_~(x) for all x in a neighborhood of z. We assume by contradiction 9~(z) + H(z, D~(z)) < 0 and ~(z) < g(z). Then, for some z > 0, (4.20)

(4.21)

~(x) + H(x, D~(x)) <_ - z

( ~ - g)(x)

<_ - ~

for x E B(z, ~) N ~,

for z ~ B ( ~ , E ) n 0 ~ .

4. BOUNDARY CONDITIONS IN T H E VISCOSITY SENSE

325

Now we fix t > 0 such that y z ( s , a ) E B(z,E) for all x E B ( z , s / 2 ) , 0 < s _< t and a E A, and set 5 := s(1 - e-t)~2. By the DPP, Proposition IV.3.9, for any x C B(z, z/2) there is a E A such that { - 5 + fJ0 t f(y~, a) e -s ds + v(yz(t)) e -t

(4.22) v(x) >

if t < t~(a),

tx(~) -5 +/o

f(Yx, a) e -s ds + g(yx(tx(a))) e -tx(~)

if t >_ t x ( a ) .

In the first case, t < t~(a), we use v(yz) >_ v.(y~) >_ P(Yx) into the right-hand side and repeat the calculation (2.8) in the proof of Theorem 2.6 (based on (4.20) and the choice of t and 5) to get

v(x)

(4.23)

~(x) > 5.

-

In the second case, t > t z ( a ) , we use (4.21) and the choice of t to observe that

and plug this into the right-hand side of (4.22). Then we can repeat the calculation (2.8) in the proof of Theorem 2.6 to get

v(x) - ~(x) > - 5 +

e e -s ds + z e -t~ = - 5 + e > 5 .

Therefore (4.23) holds for all x C B ( z , r diction to the choice of 4.

so v.(z) _> ~(z) + 5 > p(z), a contra9

REMARK 4.14. The argument of the proof of Theorem 4.13 is local and uses only the local boundedness of v and the Dynamic Programming Principle. Therefore it applies to other situations, for instance, to the case A -- 0. <] EXAMPLE 4.15. Consider as in Remark IV.1.5, T={(xl,x2)

e R 2 : I x l I _ < Ix21or [xll_> 1}

and the system f ( x , 1) = (1,0), f ( x , 2 ) = ( - 1 , 0 ) , A = {1,2}. It is easy to see that the minimal time function is T(x) = (Ixll - Ix21) A (1 -- Ixll) and the minimal interior-time function is

T(x) = T(x)

if x2 • 0,

"~(xl, 0) = 1 - IXl I-

Their associated Kru2kov transforms, v = 1 - e -T, ~ = 1 - e -~., both satisfy Theorem 4.13. Note that v is the complete solution in B c ( ~ N ) , and therefore the e-solution, of

(4.24)

-l=0 ~u+lux~l [u = 0

i n ~ 2 = T c, on 0f~,

while ~ is a discontinuous v-solution of this problem.

<~

326

V. D I S C O N T I N U O U S V I S C O S I T Y S O L U T I O N S

4.2. C o m p a r i s o n results a n d a p p l i c a t i o n s

to e x i t - t i m e p r o b l e m s and stability In this subsection we prove some comparison and uniqueness results for the boundary value problem (4.25)

S u + H(x, Du) = 0 [u=g or u + H ( x , D u ) = O

in Ft, on Oft,

where ft C_ ]~N is open, H : ~ x ] ~ g ~ ]~ is continuous, and g C C(Oft), under some structural assumptions on H, some regularity conditions on Oft, and some continuity assumptions at 0s for the semisolutions to compare. We show on examples the optimality of most of these conditions. Then we apply these results to minimal time and minimal interior-time functions T and T and to the value functions v and ~ of problems with terminal cost. No uniqueness theorem in this subsection allows the solution to be discontinuous: such results are postponed to the next subsection. Finally we prove a convergence theorem that completes the stability results in w We begin with a result for Ft bounded where the assumptions and proof are simpler. Here are the hypotheses on H: (4.26)

[H(x,p) - H ( y , p ) I < L IPl Ix - Yl + wl(I x - Yl),

where wl is a modulus, and

IH(x,p) - H(x, q)l -< M IP - ql,

(4.27)

for all x~ y E ~ , p, q E ]RN. They are satisfied by the Hamiltonian of control theory (4.28)

H ( x , p ) = sup{ - f ( x , a) .p - g(x, a) }, acA

where Ft is bounded, if, for instance, A is compact, f and ~ are continuous, and (4.29)

] f ( x , a ) - f(y,a)] <_ L I x - y I ,

Ig(x,a)-g(y,a)] ~_wl(ix-yI),

for all x , y E it, a c A. The assumption on cOit is to be a Lipschitz surface, as in w (4.30)

or, more precisely,

there are c > 0 and ~7 : ~ --* R N c o n t i n u o u s such that B ( x + t~(x),ct) C_ ft for all x E ft, 0 < t _< c .

(4.3o) and ft bounded. Suppose ul C USC(~) and u2 E LSC(~) are, respectively, a subsolution and a supersolution of

THEOREM 4.16. Assume (4.26), (4.27),

(4.31)

u + H(x, Du) = 0

in Ft,

Ul is continuous at each point of Oit and u2 satisfies (4.32)

(u2 - ul) V (u2 + S ( x , Due)) >_ 0

on OFt

4. BOUNDARY CONDITIONS IN THE VISCOSITY SENSE

327

in the viscosity sense (Definition 4.1). Then ul <_ u2 in f~. The same conclusion holds if u2 instead of ul is assumed continuous at the points of O~ and ul satisfies (tt 1

-

-

U2) A (U 1 -'[- H(x, Dul)) <__0

on O ~ .

PROOF. We consider first the case m : = maxo~(Ul - u2) < m a x , ( u 1 - u2). If m < 0 we conclude i m m e d i a t e l y by T h e o r e m 1.3. If m > 0, t h e n w = u2 + m has the s a m e properties of u2, and, in addition, ul _< w on 0f~. T h e n ul _< w in ~ by T h e o r e m 1.3, so Ul - u2 _< m, a contradiction. T h e o t h e r case is m a x o ~ ( u l - u2) = m a x , ( u 1 - u2). We assume by c o n t r a d i c t i o n (4.33)

(u 1 - u2)(z ) : m a x ( u 1 - u2) > 0,

for some z E 0f~. Define, for 0 < z < c (see (4.30)), 9 (x, y) : = ~ l ( x )

- us(y) -

5-~

- ~ ( z ) 2 _ ly -

Z[2~

and observe t h a t 9 is u.s.c, in ~ x ~. Let (xe, ye) be a point such t h a t ~(x~, ye) = m a x ~ . ~xgt By (4.30) z + z~(z) E ~, so ~ ( z + s~(z), z) < ~ ( x e , YE) and we obtain, using the continuity of Ul at z,

(4.34)

Ix z - y e e

rl(z) 2 + l y --zl2<_Ul(X~)--u2(yE)--Ul(Z+eTl(z))+u2(z) Ul(Xe)

-

-

u2(ye)

-

-

(U 1 -- U2)(Z ) ~- ~(~')

where ,a is a modulus. Since ul and - u s are b o u n d e d above, we get (4.35)

x~ - y~ 2 <_ C E

for all ~ .

T h e n x~ - y ~ --* 0 as c ~ 0 +, hence lira sup~_0-~ (ul (xz) -u~(yc)) <_ m a x ~ ( u l - u2) by t h e c o m p a c t n e s s of s and the u p p e r semicontinuity of Ul - u s . Now we use (4.33) and (4.34) to get (4.36)

xz-yz

~rl(z )

and

y~

*z

asE~0

+.

This implies xc = ys + ~Tl(z) + o(e) = ye + sr](y~) + o(z)

as e --* 0 +,

so we can use (4.30), for z small enough, to get x~ G ~. T h e n we can exploit the fact t h a t Ul is a subsolution of (4.31) and o b t a i n (4.37)

(

u~(x~) + H x~, ~

328

V. DISCONTINUOUS VISCOSITY SOLUTIONS

O n the other hand, (4.34) also implies lt2(ye) ~ U2(Z ) ~- Ul(Xr

-- Ul(Z ) -I- 03(E),

so we use lime_0+ xe = z a n d the semicontinuity of Ul to get (4.38)

lim sup u2(ye) <<_u2(z) . e~0 +

Thus, by (4.36), the c o n t i n u i t y of u l at z, a n d (4.33) we o b t a i n (u2 - Ul)(ye) < 0 for all E small, a n d we can exploit either the P D E (4.31) or the b o u n d a r y condition (4.32) to get

C o m b i n i n g this with (4.37) a n d using (4.26) a n d (4.27) we o b t a i n

Ul(X~) - u2(ye) <_ 2 M Iz - Yel + 2L jxe - Yel xe - ye

~?(z) + wl(Ix~ - Yel)

a n d the r i g h t - h a n d side tends to 0 as E --~ 0 + by (4.35) a n d (4.36). Since (4.34) gives (ul - us)(z)

< Ul(Xe) - u2(ye)

+ ~(~),

for small ~, we get a contradiction to (4.33). T h e second s t a t e m e n t of the t h e o r e m is obtained by replacing the t e r m lY - zl s in the auxiliary function 9 with Ix - zl s and repeating the previous a r g u m e n t with the roles of the variables x a n d y a n d of the ffmctions u] and us exchanged. 9 Note t h a t T h e o r e m 4.16 implies a comparison result between a subsolution of (4.31) (continuous at 0 ~ ) a n d a constrained supersolution, t h a t is, a supersolution of u + H ( x , Du) = 0 in ~ , which extends the theory of w In fact, the b o u n d a r y condition in this case is stronger t h a n (4.32). The next result is a small variant of the previous one. THEOREM 4.17. A s s u m e (4.26), (4.27), (4.30) and ~ bounded. I f Ul and u2 are, respectively, a subsolution and a supersolution of (4.25), and they are continuous at each point of Of~, then Ul <_ u2 in -~. PROOF. If m a x o o ( u l -- us) < m a x , ( u 1 -- u2) we argue as in T h e o r e m 4.16. In the opposite case we assume by contradiction (4.33), as before. We have to deal with two cases: (i) u s ( z ) < g(z), a n d (ii) Ul(Z) > g(z). In case (i) we proceed as before until (4.37). Next we want to get (4.39) to reach a contradiction as before. If Ye E fl, we have (4.39) from the P D E (4.31) a n d the definition of Ye. If Ye E 0f~, we use (4.38), (4.36), a n d the continuity of g to get l i m s u p ( u 2 - g)(Ye) = (us - g)(z) < O. r + Thus, for c small enough, the b o u n d a r y condition (4.32) gives (4.39). Case (ii) is treated similarly, as for the second s t a t e m e n t of T h e o r e m 4.16.

9

4. BOUNDARYCONDITIONSIN THE VISCOSITYSENSE

329

Before turning to the case of unbounded f and to uniqueness results we discuss the optimality of the assumptions of the two comparison theorems. The new hypotheses with respect to the comparison principle for the standard Dirichlet problem (see Chapters II and III and w are (i) the regularity of Off, (4.30); and (ii) the continuity of the semisolutions to compare. The next examples show that neither of these assumptions can be removed. EXAMPLE 4.18. Let g ( x , p ) = ]Pl - 1, f = B(O, 1) \ {0}, g(0) = 0, 9(x) = 1 for Ixl = 1. Here v(x) = 1 - e-IX{ is a Lipschitz continuous solution of (4.25) such that v = g on On. Any w -= c < 1 is a subsolution of (4.25), and if c > 0 it violates the conclusions of Theorems 4.16 and 4.17. Note that ~(x) - 1 is another continuous solution of (4.25) (cfr. Example 4.5 and Exercise 4.2). Even if (4.30) is violated only at 0, the domain is rather bad because int ~ D fl. The domain is more regular in Example 4.15 because int f = f and (4.30) does not hold just at 0; in such an example, however, the semisolutions violating the conclusions of Theorems 4.16 and 4.17 are not both continuous. <~ It is easy to give examples where the conclusions of the comparison theorems are false because the semisolutions to compare are discontinuous at the boundary, even for smooth domains. In fact, it is enough that small-time controllability on 0 f be violated and we have T.(x) < T*(x) at some point x E Of/, because the continuity of the minimal time function T at each point of the boundary of the target is equivalent to small-time controllability on the target, see Proposition IV. 1.2. Then v. and v* (v = 1 - e -T) do not satisfy v. > v*, even though they are a super- and subsolution of (4.25) with g --= 0, H given by (4.28) with g = 1, by Theorem 4.13. If il is bounded and nonconvex, one can easily choose a system with a single, constant direction and get the example (here the PDE is linear with constant coefficients). We propose an explicit example where the supersolution fails to be continuous at just one point of 0il, but the conclusion of Theorem 4.16 is false. EXAMPLE 4.19. Take the system of Example 4.15, g = 0, and il = { (Xl,X2) e R 2 : }zll < 1,

Ix1{-

1 < x2 <

lXll } .

The minimal time function T is as in Example 4.15 for x2 k 0, and

T(x) = 1 -

lxlI+

x2

for x2 < 0 .

Thus it is continuous in ~ except on X := { (xl,0) : - 1 / 2 < Xl < 1/2}. We take v := 1 - e -T. By Theorem 4.13 v = v. is a v-supersolution of (4.24) and ~ = v* is a v-subsolution, and both are v-solutions of (4.24). For every x C X , v.(x) < v*(x), so the conclusions of Theorems 4.16 and 4.17 do not hold. Here the domain il satisfies the regularity assumption (4.30), and it is easy to see that the corners of f can be smoothed to make 011 a C ~ curve while keeping the set of discontinuities of v unchanged. <~ If f~ is unbounded, we give the assumptions for the comparison theorem only for the Bellman's Hamiltonian (4.28). For the general structural hypotheses see

330

V. DISCONTINUOUS VISCOSITY SOLUTIONS

Exercise 4.7. We do not give the proof since it is rather long and technical. It can be obtained by the methods of proof of Theorem 4.16 and of w THEOREM 4.20. The comparison statements in Theorems 4.16 and 4.17 remain valid if ~t is unbounded, provided that (i)

(4.30) holds with ~ bounded and uniformly continuous;

(ii)

ui and u2 are bounded;

(iii) H is given by (4.28) with f : -~ • A --~ ~ Y and ~ : -~ • A --~ I~Iv continuous and satisfying (4.29) for all x, y E ~ , a E A, A compact, f bounded in On • A. Note that the assumptions in (iii) imply (4.26). However (iii) does not imply (4.27) unless f is bounded, and this is a restrictive assumption we want to avoid. Next we turn to the uniqueness theorem for the boundary value problem (4.25). Denote B ( ~ ) := { u : ~ ~ R bounded }, BCo(~) := { u E B(fl) : u continuous at each point of 0 ~ } . THEOREM 4.21. Assume the hypotheses (i) and (iii) of Theorem 4.20 and suppose there exists a solution u E BCo(-~) of (4.25). Then (a)

u E C(fl) and it is the unique solution, the maximal subsolution and the minimal supersolution of (4.25) in BCo(~);

(b)

if, in addition, u = g on O~, then it is the unique solution, maximal subsolution and minimal supersolution of (4.25) in B ( ~ ) , and u = v = ~ in -~, where v and ~ are the value functions defined in Theorem 4.13 with ~ = 1.

PROOF. Since u.,u* E BCo(~) are, respectively, a super- and a subsolution of (4.25), Theorems 4.17 and 4.20 imply u. _> u*, thus u is continuous. Then by the same theorems u is the maximal subsolution and minimal supersolution in BCo(~). If w is another solution in BCo(~), then w* < u < w. and so u = w. This proves (a). We prove (b) in the same way by replacing the comparison statement in Theorem 4.17 with that of Theorem 4.16. The value functions v and are bounded solutions by Theorem 4.13, thus they coincide with u. 9 The statement u = v in (b) had already been proved under weaker assumptions in Chapter IV. On the other hand, the statement v = ~ was not studied in Chapter IV and it deserves a more explicit statement. COROLLARY 4.22. Assume hypotheses (i) and (iii) of Theorem 4.20, g E BC(O~), and let v, ~ be the value functions defined in Theorem 4.13 with )~ = 1. I f v is continuous at each point of O~, then v = ~. This occurs in particular if either (j) or

the system ( f , A) is small-time locally controllable on :1- := ~c (STCT~ Definition IV.I.1) and the terminal data g are compatible (Definition IV.3.5),

4. BOUNDARYCONDITIONS IN THE VISCOSITYSENSE

331

(jj) there exist u, ~ E BCo(-~) respectively a sub- and supersolution of (4.25) such that u = ~ = g on O f . PROOF. The first statement follows immediately from Theorems 4.13 and 4.21. The sufficiency of (j) for the continuity of v follows from Theorem IV.3.6 (this theorem is stated under the simplifying assumption that OT is bounded which can be easily dropped). The sufficiency of (jj) is easily proved by the comparison theorems 4.16 and 4.20 or by Perron's method (Corollary 2.15). The following control-theoretic consequence of the previous theory was announced in Chapter IV. COROLLARY 4.23. Assume the hypotheses of Corollary 4.22 and Oil bounded. Then system (f, A) is small-time locally controllable on T := f c if and only if it is smallo

time locally controllable on int 7- ( S T C T , Definition IV.I.1). Here the assumption t h a t 0il be bounded can be dropped; it is enough to extend a bit two results of w where this assumption was made only to simplify some statements and proofs. On the other hand, we observed in Remark IV.1.4 t h a t some regularity on OT such as (4.30) is necessary. PROOF OF COROLLARY 4.23. STC~- implies S T C T by definition. If we assume S T C T we have the continuity of the minimal-time function T by Proposition IV. 1.6. Thus v = 1 - e - T is continuous, and therefore T is continuous by Corollary 4.22. Since (4.30) implies 7" = i n t T (see Exercise 4.8) we can use Remark IV.1.4 and O

get STCT-. We have not yet described in some generality the situation where there exists a unique continuous solution of (4.25) which does not satisfy u = g on 011, so t h a t u ~ v, even though this was what happened in the simple one-dimensional Examples 4.3 and 4.11. The next result describes a situation where 7, instead of v, is the unique continuous solution of (4.25) and it does not satisfy ~ = g on 0il, as in the Examples 4.3 and 4.11 (other examples are in Exercise 4.13). We define the usable part of cO~ as the set of terminal points of trajectories of the system, t h a t is, b[ := { z E Oil : z = y z ( t z ( a ) ) for some x E i~, a E . 4 } . The following theorem deals with time-optimal control and says, roughly speaking, t h a t ~ is continuous if there is a target ~- containing b/, smaller than the true target T = f c , and an extension of the system inside T, such that STC :T holds even if S T C T does not. THEOREM 4.24. Assume the hypotheses of Corollary 4.22 and ~ -- 1, g - O. Suppose there exist an open set 5 2 il and a system ] : 5 • A ---+]~N such that f = f on i~ • A, satisfying the structural assumptions (A0)-(A3) of Chapter I I I and such that ( f , A) is small-time controllable on ~" := fie. Assume also ~- D Lt and O ~ bounded. Then

332

v. DISCONTINUOUSVISCOSITYSOLUTIONS m

(i)

V E C([}) is the solution of (4.25);

(ii)

V(x)=v(x)forallxE~;

(iii) for x E O~, ~(x) = v(x) = 0 if and only if x E ~. PROOF. Let :F be the minimal time function for T and ~ = 1 - e - ~ .

By the

controllability assumption on (f, A) we have ~ E BC(-~). To prove that (4.40)

~* < ~

in ~ ,

we can use Theorems 4.16 and 4.20. In fact ~ E BC(~) is a supersolution of the HJB equation in fl, and for x e 0 ~ such that ~*(x) > ~(x) we have ~* + H(x, D~*) G 0 at x in the viscosity sense, because ~(x) _> 0 and ~* is a subsolution of (4.25) with g - 0. Then we can apply the comparison results and get (4.40). Next we claim that (4.41)

~ ___~

in ~ .

This gives g < ~, and then ~ = ~ in ~ by (4.40). To prove (4.41) we need the following representation formula for g: (4.42)

~ ( x ) = ~A'inf ~ footX(~)e-" ds + ~(yx(t,(a)))e -t'(~) },

which holds by Corollary IV.3.17. We assume by contradiction t h a t ~(x) < ~(x). Then there is ~ E A such that 1 - e -~(~)

< ~(z),

and t'x(~) < +cx~ because ~ < 1. Thus we use (4.42) and t'x _> tx to get ~(yx(tx(~)))

> 0.

This implies yx(tx(-~)) ~t T, a contradiction to L / C T. This proves the claim and therefore ~ E C ( ~ ) . In a similar way we obtain ~ < v in ~, so ~ = v in ~ because v < ~ by definition. To prove (iii), we first observe that ~ = 0 on b/ C "Y, so v = v = 0 on ~ by continuity. On the other hand, if x ~ ~ let r :-- d i s t ( x , ~ ) > 0. For any z E B(x,r/2) and c~ E A, we have Iz - yz(tz(c~))[ > r/2 by definition of H, so that, for some c > 0, v(z) > c for all such z. Then l i m s u p z ~ v(z ) > 0 = v(x). For the case of general running and terminal costs we give without proof two results under controllability conditions. In the first one we make an explicit assumption on the directions of the vector fields at On, in the second one we assume essentially a uniform small-time local controllability on each point of ~ (cfr. Exercise 4.15).

4. BOUNDARY C O N D I T I O N S IN T H E V I S C O S I T Y SENSE

333

THEOREM 4.25. Assume the hypotheses of Corollary 4.22 and 0[2 a compact manifold of class C 2. Suppose for all x E 0[2

inf f ( x , a ) , n(x) < 0,

s u p f ( x , a ) , n(x) > O,

acA

aEA

where n(x) is the interior normal to 0[2. Then ~ C C(-~) is the solution of (4.25).

THEOREM 4.26. Assume the hypotheses of Corollary 4.22 and that there exist constants C , r > 0 and a modulus w such that, for all x , z C [2, Ix - z I < r, there is c~ E .4 satisfying yx(t,c~) = z for some t <_ ~v(Ix - z[) A tx(a). Then ~ E C(-~) is the solution of (4.25) and V(x) = v(x) for all x E [2.

We end this subsection with a convergence theorem that completes the stability results on the three perturbation problems studied in the previous subsection: regular perturbations (Proposition 4.7), singular perturbations (Proposition 4.10) and perturbations of the domain (Proposition 4.12). We summarize all the results in a single statement because the proof is exactly the same in the three cases. COROLLARY 4.27. Assume either (4.26), (4.27), (4.30) and [2 bounded or the hypotheses (i) and (iii) of Theorem 4.20. Suppose there exists a solution u E BC([2) of (4.25) such that u = g on 0[2. Set F(x~r~p) = r + H ( x , p ) and assume either the hypotheses of Proposition 4.7, or those of Proposition 4.10, or those of Proposition 4.12. Then uz ~ u on compacta of[2. PROOF. The three propositions imply in each case that the upper and lower weak limits, ~ and u, are, respectively, a subsolution and a supersolution of (4.25). By the comparison theorems 4.16 and 4.20 we get ~
Since u < ~ by definition, we obtain ~ = u = u, and this implies the uniform convergence of u~ to u on compacta by Lemma 1.9. The reader should compare this result with Corollary 1.8, see Remark 1.11. Corollary 4.27 can be applied to prove the stability of the value function of problems with restricted state space (either involving exit times or with state constraints) under perturbations of the data, as we did in Theorem 1.10 for problems in all ]RN, see Exercise 4.16. The most restrictive assumption of Corollary 4.27 is the existence of a continuous solution of the limit problem. This will be dropped in the next subsection for the problem of time-optimal control. 4.3. U n i q u e n e s s a n d c o m p l e t e s o l u t i o n for t i m e - o p t i m a l c o n t r o l In this subsection we give some uniqueness and stability results for the boundary value problem (4.25) that allow the solution to be discontinuous. We limit ourselves to the time-optimal control problem and consider the discounted value functions ^

(4.43)

f

v(x) = inf t ~ ( ~ ) e -s ds, oEA Jo

~(x) = inf ~0 t~(~) e -~ ds, aEA

334

v. DISCONTINUOUSVISCOSITYSOLUTIONS

where tx and t'x are the first entry time of the system in 7- and int 7", respectively. We assume the same hypotheses as in w namely, the function f defining the system is continuous in all variables and lipschitzean in the state variables, the control set A is compact, the target 7- is closed and nonempty and its boundary OT is bounded (see Exercise 4.17 for some extensions). We know that both v and ~ solve the boundary value problem

u + H(x, Du) = 0

(4.44)

~u,

in 7-c =: ~, on aT,

> 0

( u * A ( u + H ( x , Du))<_O

onOT

with H given by (3.9), where the HJB equation in 7-c is in the sense of noncontinuous viscosity solutions, Definition 2.2, u, and u* are the semicontinuous envelopes, and the second boundary condition on 07- is in the viscosity sense, Definition 4.1. This follows from Theorem 4.13 and from the fact that v, ~ >_ 0, so that we do not need to consider supersolutions in the viscosity sense at the boundary. We call a supersolution of (4.44) a supersolution ~ E BLSC(~) of the HJB equation satisfying the first boundary condition ~ > 0, we call a subsolution of (4.44) a subsolution u E BUSC(-~) of the HJB equation satisfying the second boundary condition u A(u + H ) < 0, and a solution of (4.44) is a function u such that u, is a supersolution and u* is a subsolution. Recall that in Chapters III and IV we called complete solution of a b o u n d a r y value problem a function t h a t is the minimal supersolution and the maximal subsolution. The natural extension of this notion to non-continuous solutions is the following. DEFINITION 4.28. A locally bounded function w is a complete solution of (4.44) if w, is the minimal supersolution and w* is the maximal subsolution; that is, if w is a solution of (4.44) and any subsolution u and supersolution ~ of (4.44) satisfy w,<~,

u_<w*.

<

The main result of this subsection is the following. THEOREM 4.29. Under the previous assumptions ~ is the maximal subsolution of (4.44). If, in addition, 7- is the closure of its interior, (4.45)

7- = int 7-,

then ~ is a complete solution. Before giving the proof we describe several consequences. COROLLARY 4.30. Under the previous assumptions and (4.45) (i)

~ is the unique upper semicontinuous complete solution of (4.44);

(ii)

for any solution u of (4.44), ~,<_u,
4. BOUNDARY CONDITIONS IN THE VISCOSITY SENSE

(iii)

335

~, = v, is the unique lower semicontinuous solution of (4.44); moreover, it has a meager I set of points of discontinuity and a dense set of points of continuity.

PROOF. (i) ~ is upper semicontinuous because it is a subsolution by Theorem 4.29, and there is at most one u.s.c complete solution because it must be the maximal subsolution. (ii) is a trivial consequence of the definition of complete solution. (iii) The equality ~, = v, is essentially the second statement of Theorem 4.29 because v, is the minimal snpersolution of (4.44) by Theorem 3.7. To prove uniqueness, let w = w, be a solution of (4.44). By definition of complete solution ~, < w

and

w* < ~ * .

The second inequality, w 1.s.c. and ~ u.s.c., imply

w_< (~*), _< ~,, which gives w = ~,. To prove the regularity result, set u = ~, = v, and consider the sets

C~ := { x : u*(x) - u(x) < l / n },

neN,

which are open because u* - u is u.s.c. Of course u is continuous at x if and only if x E C , for all n. We are going to prove that each Cn is dense and we observe first that (4.46)

(u*), = u .

In fact the inequality > follows from the lower semicontinuity of u, and the opposite one from taking the 1.s.c. envelope of both sides of the inequality u* ___~* = 9. If Cn were not dense, for some x0 it would be u*(x) >>u(x) + 1In for all x near x0. Thus (u*),(x0) _> u(xo) + l / n , because u is ks.c, and we have reached a contradiction to (4.46). The intersection of all Cn is dense by Baire's theorem (see, e.g., [nu86]). 4 Note that a complete solution is unique if and only if it is continuous, because otherwise there are infinitely many functions with the same semicontinuous envelopes. We remark that under the assumptions of Theorem 4.29 a lower sdmicontinuous complete solution of (4.44) may not exist. In fact, the only possible candidate is ~, = v,, and in Example 4.15 we have a target satisfying (4.45), a linear system, and a continuous v such that v(x) < ~(x) on a line segment. This example also says that v may not be a complete solution of (4.44), even in a case where it is the complete solution of the usual Dirichlet problem for the HJB equation. 1A set is m e a g e r (or B a i r e ' s first category) if it is a c o u n t a b l e u n i o n of nowhere dense sets (i.e., sets w i t h e m p t y interior), or, equivalently, if its c o m p l e m e n t is a c o u n t a b l e intersection of dense open sets.

336

V. D I S C O N T I N U O U S V I S C O S I T Y S O L U T I O N S

Note also that Corollary 4.30 (ii) says t h a t ~ is the unique solution in the sense of the graph, that is, the graph of any solution u of (4.44) is contained in the completed graph of

c-graph ~ := { (x, r) ~ ~ • ~ : V,(x) < r < V*(x) }. Theorem 4.29 is particularly fit for the study of the stability of the value functions with respect to perturbations. In fact, if uE are solutions of any of the p e r t u r b e d problems studied in w in the special case F ( x , r , p ) = r + H ( x , p ) and b o u n d a r y d a t a g - 0, then the upper weak limit ~ is a subsolution of (4.44) (see Propositions 4.7, 4.10, 4.12), while the lower weak limit _u is a supersolution of (4.44) provided u ~ _> 0 in a neighborhood of the boundary. Then

by Theorem 4.29, so the weak limits coincide at any point where ~ is continuous, and therefore ue ~ ~ on compact sets where ~ is continuous. This is the natural extension of Corollaries 1.8 and 4.27 to the case that the value function of the limit problem is discontinuous. Next we give two explicit examples of applications. The first is the analogue for time-optimal control of Theorem 1.10 for the infinite horizon problem. For each > 0 we are given a system with continuous dynamics fe : R N x A -~ R N, with compact control set Ae _ A and satisfying

If~(x,a)-fe(y,a)l<_Lelx-y

I

forallx, yc]RN, aEAE.

We assume fr ~ f0 on ~ • A for each compact ~ C_ ~.N and that the Hausdorff distance d~I(AE, A0) --~ 0 as ~ --~ 0 +. For each E we are given a compact target T~ _~ T and assume ds(T~, TO) ~ 0 as s --~ 0 +. Let ve,~e be the discounted value functions of the time-optimal control problems corresponding to E _> 0, that is, they are given by (4.43) with ~4 replaced by As and the entry times tx, t'z corresponding to the system fs and the target "YE. THEOREM 4.31. Assume the preceding hypotheses and let TO be the closure of its interior. Then vc and ~z converge uniformly to vo = vo on every compact set where ~o is continuous. PROOF. We know that vE and ~e satisfy for all x > 0 problem (4.44) with H replaced by tl~e Hamiltonian He associated with the system (re, Ae) and 7- replaced by T~. It is easy to see, as in the proof of Corollary III.2.22, that Hr ::~ H0 on compacta of ~2N. By the arguments of the proofs of Propositions 4.7 and 4.12 and the obvious inequalities 0 <_ vr < 1, the weak limits u_ = liminf.v~, e~0 +

~ = lim sup*rE ~0

+

(see Definition 1.4) are, respectively, a supersolution and a subsolution of the limit problem (4.44) with H -- H0 and T = To. By Theorem 4.29 ~0 is a complete

4. BOUNDARY CONDITIONS IN THE VISCOSITY SENSE

337

solution of this problem, so the general argument explained above gives the uniform convergence of vs to ~0 on compacta where ~0 is continuous, and on these sets v0 = ~0 by Corollary 4.30. The proof for vs is obtained in the same way by taking the corresponding weak limit. 9 The second application is to the vanishing viscosity approximation. Suppose us E C2(f~) N C ( ~ ) are solutions of (4.47)

-eAu+u+H(x,

Du)=O

u = 0

i n f t = 7 -c. on 0ft,

for e > 0, and assume ft is bounded. For the existence of such solutions, for example, in the case ft has smooth boundary, we refer to [GT83, Kry87]. COROLLARY 4.32. Assume the hypotheses of Theorem 4.29, including (4.45). Then uE ~ ~ as E ---* 0 + on compact sets where ~ is continuous. PROOF. We claim that 0 < us < 1 for all r By contradiction assume ue is negative somewhere. By the compactness of ~ and the boundary condition there is x0 E f~ where ue attains a negative minimum. Since Dus(xo) = O, Aus(x0) > 0 and H(xo, 0) = - 1 , we have - ~ A u s ( x 0 ) + us(xo) + H(xo,Dus(xo)) < - 1 , a contradiction to the PDE in (4.47). Next assume by contradiction us is larger than 1 somewhere. Then there is a maximum point Xl c f t where us(x1) > 1, so - e A u s ( x l ) + u~(xl) + H ( x l , D u s ( x l ) ) >_ us(x1) - 1 > 0, a contradiction again. Now we can apply Proposition 4.10 to get that the weak limits u and g are, respectively, a supersolution and a subsolution of (4.44). By Theorem 4.29 and the general stability argument explained above we can conclude. 9 Now we t u r n to the proof of Theorem 4.29, which is a simple consequence of the following two propositions. PROPOSITION 4.33. The value function ~ is the maximal subsolution of (4.44). PROOF. We assume int T ~ ~ because the case int T = ~} is much easier and we leave it as an exercise for the reader. Let w be a subsolution of (4.44) and let us first observe that it is not restrictive to assume w >_ 0. In fact, if this is not the case, we can replace w by w + = w V 0 which is a viscosity subsolution of (4.44) by Proposition 4.6, since it is u.s.c, and w and 0 are subsolutions. Define for h > 0

Th:--{xET:dist(x,

OT)>h}

and

vh(x) :=

inf l t h ( a ) e - S d s , aEA J0

338

V. DISCONTINUOUSVISCOSITY SOLUTIONS

where txh is the first entry t i m e of the s y s t e m in Th. By T h e o r e m 2.6 supersolution of the Dirichlet p r o b l e m

(4.48)

: + H ( x , Du)=O = 0

in RN --. 7-h, on 0Th .

We e x t e n d w in intT- by setting w - 0 and note t h a t w E t h a t w is a subsolution of (4.48) and t h a t (4.49)

~(x) = inf vh(x) h>0

(vh), is a

for all x E • N

U,-,qC(]t~N). We claim

.

T h e n we get w _< ~ because the c o m p a r i s o n principle, T h e o r e m 1.3, gives w <_ (vh), in R N for all h. Since ~* is a subsolution of (4.44) by T h e o r e m 4.13, we conclude t h a t ~ = ~* is the m a x i m a l subsolution. To prove the first claim we first observe t h a t w is trivially a subsolution of the b o u n d a r y condition in (4.48) and of t h e H J B e q u a t i o n in T r and in int T \ Th. N e x t let x0 E 07- be a m a x i m u m point of w - c2 with p E C I ( R N \ Th). T h e r e are two possible cases. T h e first is t h a t w(xo) > 0, and we get the conclusion because w satisfies the second b o u n d a r y condition in (4.44), hence (4.50)

w(xo) + H(xo, Dp(xo) ) <_O.

T h e second case is t h a t w(xo) = 0. Since we can assume ~(x0) = w(xo) = 0 and ~(x) > w(x) > 0 near x0, we have D~(xo) = 0 and t h e n (4.50) is satisfied. It remains to prove (4.49). N o t e t h a t by definition ~ _< v h for all h. A s s u m e by contradiction there is 5 such t h a t 9(~) < infh> 0 vh(2). T h e n there is ~ > 0 such t h a t (4.51)

~(~) + 2E < v h (5)

for all h > 0 .

In particular, ~(~) + 2~ < 1; therefore, given ~ E .4 such t h a t (4.52)

1 - e -i~(~) < ~(~) + e,

there is t_> 0 such t h a t z : = y~(t.G) E i n t T and (4.53)

1-e -~
-i~(~)+E.

B y taking h < d i s t ( z , 0 T ) we have z E Th, hence and (4.53) we get a contradiction to (4.51). PROPOSITION 4.34.

vh(~) _< 1 -- e -~ and by (4.52)

If 7- is the closure of its interior, then v,(x) = v,(x)

.for ~ u x ~ ~ N .

9

4. BOUNDARY CONDITIONS IN THE VISCOSITY SENSE

339

PROOF. Since v _< ~ we have v, < 9.. We are going to prove that ~. < v, which immediately gives the conclusion. Fix x E ]~Y and a C A such that tz(a) < +oc (if such an a does not exist then v(x) = 1 and the desired inequality is verified). Set T := tx(c~), ~ := yx(T,C~) E 7-, and choose a sequence ~n E int7- such that 5n --* 5. Let zn( 9) be the solution of

{

z' = f(z,~) z(r) =

:n

0_
Then, for all 0 < t < r, [z~(t) - yx(t, o~)I <_ I:n - :1 + and by Gronwall's inequality

l

L Izn(s) - y~(s, c~)l ds,

I z ~ ( t ) - yx(t,a)l < IS~ - ~le Lr . By plugging t = 0 into this inequality we get that x,~ := z,~(O) converge to x as n ~ oz. Moreover, t'x. (a) _< ~- = tz(a) for all n, so ~,(x) < liminf(1 - e -i~(~)) _< limninf(1 - e -~x.(~)) <_ 1 - e -t~(a) . y~x

Since a is arbitrary we have proved that ~.(x) < v(x).

~,

PROOF OF THEOREM 4.29. The first statement of the theorem is Proposition 4.33. In order to show that ~ is a complete solution it remains to prove that 9. is the minimal supersolution of (4.44), and this follows from Proposition 4.34 and Theorem 3.7. Exercises 4.1. Motivated by Remark 4.2, we define a viscosity subsolution (respectively, supersolution) of the equation G(x, u, D u ) = 0 in a set X _C ~ g with G : X • ~ • ]l~N ~ I--OO, + ~ ] not necessarily continuous, a locally bounded function u such that G , ( x , u(x), D ~ ( x ) ) < 0 (respectively, G*(x, u(x), D ~ ( x ) ) > 0) for every ~ G C : ( X ) and x local maximum (respectively, minimum) point for u - ~. Consider the infinite horizon problem with f ( x , a) = a, A = ~N, g(x, a) = g(x) with ~ E B c ( R N ) , and A = 1. Compute the Hamiltonian H associated with this problem and the value function v, and observe that H is discontinuous and v is smooth. Prove that v is a viscosity solution of the HJB equation u + H ( x , D u ) -- 0 in ]t~N in the sense just defined; however, if g is not a constant, v(x) + H ( x , D v ( x ) ) < 0 at some point x, so v does not solve the HJB equation in the classical sense. 4.2. Consider 7- closed, int 7- = O, d(x) = dist(x, T), O open, O _D T, k:=inf{d(x):xE00}>0. Consider the problem

0 d

or

u+lDul-1--O

on07-, on O 0 .

340

V. DISCONTINUOUS VISCOSITY SOLUTIONS

(This is a problem with mixed boundary conditions, but it is obvious how to interpret them.) Find a lipschitzean solution of this problem satisfying u = 0 on 0T. Prove that any function w -- c < k A 1 is a subsolution of the problem. 4.3. Prove that the function tt in Example 4.11 is the e-solution of - u ' + u - 1 = 0 in ]0, 1[, u(0) = u(1) = 0. 4.4. Under the hypotheses (A0)-(A4) of Chapter III consider

and the value function w(x) = inf{ J(x, ~, T ) : c~ E .,4, tz(c~) _< T _< t'x(C~)}. Prove that w is a viscosity solution of (4.18), (4.19). 4.5. If we drop the continuity assumption on the boundary d a t a g, Definition 4.1 can be modified by replacing g with g* in the definition of a subsolution and with g, in the definition of a supersolution. Prove t h a t Propositions 4.4 and 4.6 remain valid in this case. Check that this definition is consistent with that of Remark 4.2. 4.6. Use Theorems 4.16 and 4.20 to give verification results for infinite horizon problems with state constraints using semicontinuous verification and falsification functions. 4.7. (a) Assume A is compact, f : ~ x A ---, R N is continuous, it satisfies ( f ( x , a ) - f ( y , a ) ) . ( x - y) < L l x -

yl 2

for all x, y, a and (4.29) for all a and x, y 9 B(O~. 5), 5 > 0, and it is bounded in 0f~ • A. Assume g is continuous and satisfies (4.29) for all x, y, a. Show that H defined by (4.28) satisfies (4.26), (4.27) for all p, q and x, y 9 B(0fl, 5/2), and the inequality (2.7) in Chapter III for all x, y 9 ~. (b) Prove Theorem 4.20 for g satisfying the structural conditions in (a). (This proof is rather long and delicate, the reader can check the result in [BS94].) 4.8. Prove that (4.30) implies int ~ = ft. 4.9. Use Theorem 4.21 to give verification theorems for the problems of minimizing J and J. 4.10. Let T' be the set of piecewise constant controls as in Chapter III. Let v ~(x) := inf~c p J(x, a). Assume the hypotheses of Corollary 4.22 and v continuous. Prove that v~ = v. 4.11. Assume the hypotheses (i) and (iii) of Theorem 4.20 and g - 0. Suppose there exists ~ 9 C ( ~ ) such that ~(x) = v ( x ) for all x 9 ft. Prove that ~ = ~ . (Check the result in [BS94].) 4.12. Assume the hypotheses of Corollary 4.22 and S T C T . Suppose there exists V E C ( ~ ) such that ~(x) = v(x) for all x 9 fL Prove that V -- ~.

4. BOUNDARY CONDITIONS IN THE VISCOSITY SENSE

4.13. Let f ( x , a ) = f ( x ) = ( 1 , O )

for a l l x E ~ 2

341

7 " = ([0,112) c , t ~ - l , g m 0 .

(i)

Compute v and ~;

(ii)

show t h a t the assumptions of Theorem 4.24 hold;

(iii) do the same for 7" : B(0, 1) c. 4.14. Prove Theorem 4.26. [Hint: use Exercise 4.12.] 4.15. Give some explicit conditions on the system (f, A) ensuring the assumption of Theorem 4.26. [Hint: use the results on small-time local controllability of Chapter IV, e.g., Theorems I V . I . l l , IV.I.12, and IV.I.15.] 4.16. Consider control systems o n ]I~N with dynamics fh and control set Ah, 0 _< h _< 1, target sets Th, and value functions l~'h( x )

ft"

= inf ~ J0 ~ ( y ~ h ) o)

e-~"

d. + . - ~ h % h t y . ( t . ( ~ ) ) )

eEAh ~

},

where y(h)(. ) is the trajectory of the system corresponding to a and tx = t (h) (e) is its first entry time in Th. Assume fh, Ah, gh, Ah are continuous with respect to h as in Theorem 1.10, gh E C(R N) converge uniformly to go, and Th _~ To are compact and dn(Th, To) ---* 0 as h --* 0 +, where dH is the Hausdorff distance. Assume T0r satisfies (4.30), (4.54)

I f o ( x , a ) - fo(y,a)] < L i x -

y]

for all x, y C B(OT,~), a E A,

for some 5 > 0, and vo C BC(T--~). Prove t h a t Vh ~ Vo on compacta of R g . [Hint: if (4.54) holds for all x, y C ~ N it is enough to check the hypotheses of Corollary 4.27, otherwise one has to use Exercise 4.7 instead of Theorem 4.20.] 4.17. Extend Theorem 4.29 to systems (f, A) satisfying the assumptions (A0)-(A3) of Chapter III and to the case of unbounded 07". 4.18. Extend Corollary 4.32 to the case that fl is unbounded and uc is a classical solution of - a l ; )uxix~ + u + g ( x , Du) = 0

in fl,

u = gE

on 0 0 ,

0 < l i m i n f u ( x ) < l i m s u p u ( x ) _ 1. where the matrices (al~)(x)) are positive semidefinite for all x E Q and r > 0, a(~) ij ::~ 0 on compacta, and gz ~ 0 on compacta of 0~, 0 _< gr < 1. 4.19. Consider the targets Th introduced in the proof of Proposition 4.33. Find systems (fh, Ah) such that the corresponding value functions v h are Lipschitz continuous and satisfy (4.49). 4.20. Under the assumptions of Corollary 4.30 prove that

for all x.

v, (x) = lim inf v(y) y~X Y#~

342

v. DISCONTINUOUSVISCOSITYSOLUTIONS 5. B i l a t e r a l s u p e r s o l u t i o n s

In Chapter II, w we saw that if the Hamiltonian H is convex with respect to the p variables, then a continuous function u is a viscosity solution of

Au + H(x, Du)

(5.1)

= 0

in 12,

where 12 c_ ~N is open, if and only if it is a bilateral supersolution, that is, (5.2)

)~u(x) + H(x,p)

= 0

for all x E l~ and p E

D-u(x).

In Chapter III, w we also saw that this formulation is natural for value functions in view of the Backward Dynamic Programming Principle. In this section we exploit the fact that the notion of bilateral supersolution extends naturally to lower semicontinuous functions without taking semicontinuous envelopes, and prove uniqueness results in this framework with appropriate boundary conditions. DEFINITION 5.1. A function u E LSC(12) is a bilateral (non-continuous viscosity) supersolution of (5.1) if (5.2) holds, or, equivalently, for any p E C1(12) and x E 9t such that u - p has a local m i n i m u m at x,

Au(x) + H(x, Dp(x))

= 0,

that is, equivalently, if u is a supersolution of both (5.1) and

-s

(5.3)

- H(x, Du)

= 0

in f~

(Definition 1.1).

<~

In the first subsection we consider the Dirichlet boundary value problem, with lower semicontinuous boundary data satisfying a sort of compatibility condition. This theory can be compared with those of w167 and 4.3 on time-optimal control. Here we consider nonconstant running and terminal costs and the target T = fU is an arbitrary closed set as in w without the assumption T = int 7- of w In the second subsection we consider the initial value problem for an evolutive HJB equation with lower semicontinuous initial data. This corresponds to finite horizon problems with lower semicontinuous terminal cost, and this generality includes problems with a constraint on the endpoint of the trajectories by assigning a large cost to trajectories violating the constraint. We also prove a result on the penalization of the constraint. This allows to approximate the constrained problem with unconstrained ones fitting into the theory of necessary and sufficient conditions of optimality given in w of Chapter III and satisfying the Extended Pontryagin Maximum Principle. 5.1. P r o b l e m s w i t h e x i t t i m e s a n d g e n e r a l targets We consider the boundary value problem (5.4)

{~+H(x, Du)=O = g

inl2, on 012,

5. BILATERAL SUPERSOLUTIONS

343

with the Hamiltonian of the form

(5.5)

H(x,p) := sup{ - f ( x , a) .p - $(x, a) }, aEA

associated with the problem of minimizing the cost functional f ft~(c~)

//

:= [t Jo fo+,

(yx,

e - t dt +

if tx(a) < +oc,

_,

if tx(a) = +oc,

over ,4 := { a : [0,+co[--~ A, a measurable}, where tx(a) is the first exit time from the open set gt of the t r a j e c t o r y Yx( 9 a) of the system r = f(y,

(s)

y(0) = x.

We assume t h a t ' f : ]~N X A ----+ I~ N is continuous,

(5.6)

I f ( x , a ) - f(z,a)l < L I x - zi, If(x,a)l < k(1 + Ixl), '~ : ]l~.N X A

(5.7)

for all x,z,a;

> R is continuous,

]t(x,a) - ~(z,a)[ < n Ix - z[, le(x,a)l < M,

for all x,z,a;

"g E BLSC(~ N) satisfies a global principle of suboptimality, (5.8)

i.e., for all x c ~ N and t > 0,

g(x) < inf ~fo g(Yx. a) e -s ds + e-tg(yx(t, a)) } --

aEA

"

This assumption should be compared with the compatibility condition on the terminal cost of Definition IV.3.5. A trivial example satisfying (5.8) for any system (f, A) and for ~ > 0 is g =- 0. In particular, the discounted version of the minimum time problem, that is, g - 1 and g -- 0 (see Section IV.2), satisfies (5.7) and (5.8). We interpret the Dirichlet boundary value problem (5.4) in the following sense. DEFINITION 5.2. Given g E LSC(RN), a function u E LSC(R N) is a bilateral supersolution of (5.4) if it is a bilateral supersolution of the HJB equation in ~, u(x) = g(x) for all x C ~r and it is a supersolution of (5.9)

-u-

H(x, Du) > 0

in ]RN .

<~

If g satisfies (5.8) the definition can be rephrased as follows. A function u E

LSC(-~) which is a bilateral supersolution of the HJB equation in ~, and such that

344

v. DISCONTINUOUSVISCOSITYSOLUTIONS

u = g on Off, can be extended to a bilateral supersolution of (5.4) by defining u = g on (~)r if and only if this extension satisfies (5.10)

u + H ( x , p ) <_ 0

for all x E 0fl and p E D - u ( x ) .

In fact the requirement at points x ~ ft is automatically fulfilled by assumption (5.8) (see Exercise 5.1). PROPOSITION 5.3. Assume (5.6), (5.7). (5.8) and consider the value function v(x) := inf J ( x , a ) . c~EA Then its lower semicontinuous envelope v, is a bilateral supersolution of (5.4). PROOF. We know from Theorem 2.6 that v, is a supersolution of the equation. The following Backward Dynamic Programming Principle implies that v, is a supersolution of (5.9) by the arguments of the proofs of Corollary III.2.28 and Theorem 2.6. Finally, v, _< v = g on f~r by definition, and v, > g on R N because v > g by (5.8) and g is 1.s.c. ~1 LEMMA 5.4 (BACKWARD D P P ) .

Under the assumptions of Proposition 5.3,

v(x) >_ V ( y x ( - t , ~ ) ) e ~ -

/0'

e(yx(-s),~(-s))e

~ ds

for all x E R N, t > 0 and c~ : ] - ~ c , 0] --~ A measurable. PROOF. We claim that for all x E R N, t > 0, c~ E A (5.11)

v(x) <_

2

e(yx, c~) e -~ ds + e - t v ( y x ( t , c~)).

In fact the usual D P P gives

(5.12)

v(x) <_

fJ0

tAt~ (a) e(y~, ~) e-" ds +

e-tA~(~

where z = y~(t A tx(a), a). This proves the claim if t _< tx(a). If t > tx(a), we use that v(z) = g(z), the principle of suboptimality in (5.8) at the point z, and the inequality g _< v (obvious from (5.8)) to get (5.11). Now the conclusion is reached exactly as in the proof of the Backward D P P Proposition III.2.25. ~1 The main result of this section is the following. THEOREM 5.5. Assume (5.6), (5.7), and (5.8). Then v, is the minimal bounded supersolution of (5.4) and the maximal bounded supersolution of (5.9) such that u < g on f~c. In particular, v, is the unique bounded bilateral supersolution 4 (5.4).

5. BILATERAL S U P E R S O L U T I O N S

345

The first statement of the theorem is already known from Section 3 in the case of time-optimal control, f -- 1, g - 0, because v. is the e-solution of (5.4), but the proof we give here is independent and completely different. In the proof we need a one-parameter family of inf-convolution type approximations of a lower semicontinuous function. LEMMA 5.6. Let u E B L S C ( ~ N ) , G C > O, and define for (x, t) E R N+I

uc(x,t) := inf { u ( y ) + e - C t l x - y ] 2 / ~ 2 } .

(5.13)

y6~ N

Then

(i)

uc is locally Lipschitz continuous;

(ii)

for all (x, t) 9 l~ N+i, ue(x, t) 7 u(x) as c --* 0+;

(iii)

M c ( x , t ) := argmin{ u(y) + e - C t l x - y12/r 2 } ~ 0 for all x,r yE]~N

(iv)

for any Yc E 2lIc(x~t)

Ix - ycl -< ~ ect/2(211ull~r

(5.14)

(v)

D - u e ( x , t) = 0 if 5I~(x, t) is not a singleton; if ~Ic(x, t) -- {Yc}, then D-uc(x,t)

= { ( p c , - C e - C t [ x - yc[2/s 2) }

with Pc := 2 e - C t ( x - Yc)/e 2, and in this case Pc E D - u ( y c ) . PROOF. (i) By the argument of Lemma II.4.11 (a), uc( 9 t) is semiconcave, and the local Lipschitz continuity with respect to t is easily checked directly. (ii) The proof is essentially the same as that of Lemma II.4.11 (b). (iii-iv) As in the proof of Lemma II.4.12 it is easy to see that the search for the inf in (5.13) can be restricted to points y = Yc satisfying (5.14), so the minimum is attained by Weierstrass' theorem. (v) We write uc(x,t) -- m i n y e B g ( x , t , y ) , with B = B ( x , eeCt/2(2[[ullor 1/2) and g(x, t, y) := u(y) + e-Ct]x - yj2/E2, and apply the results on marginal functions of Chapter II, in particular Proposition II.2.13. The required assumptions are satisfied because

Dxg(X, t, y) = 2e -Ct (x - y ) / e 2,

Dtg(x, t, y) = - C e -Ct Ix - y12/s 2,

and a straightforward calculation gives

g(x + h, t + s, y) - g(x, t, y) - G g ( x , t, y ) . h - D,g(x, t, y)s -

e-Ct ~2 (Ihl 2 + ( e - c " - 1)Ix + h - yl 2 +

Cslx

-

yl 2) < C K I h l 2 + s 2)

for all h and s small and a constant C1 independent of y 6 B, so g is differentiable with respect to (x, t) uniformly in y. This gives the formula for D - u ~ ( x , t). The last statement is obtained easily as in the proof of Lemma II.4.12 (iii). 4

346

V. DISCONTINUOUS VISCOSITY SOLUTIONS

PROOF OF THEOREM 5.5. We show t h a t if u E BLSC(NN), w E BLSC(-~) are, respectively, a supersolution of (5.9) a n d of

w + H(x, Dw) > O

(5.15)

inf,,

such t h a t u _< w on 012, t h e n u _< w in fL All the conclusions t h e n follow from Proposition 5.3, since v is b o u n d e d . First note t h a t the H a m i l t o n i a n satisfies (5.16)

[H(x,p) - H(y,p)[ <_ L Ix - yl(1 + IPl).

Consider the a p p r o x i m a t i o n ue of u defined by (5.13) with a c o n s t a n t C to be chosen appropriately. We claim t h a t ue is a subsolution of (5.17)

Ut + U + H(x, DxU) <_ Lms ct/2

in ]I~N+I,

where m := (211u11~)1/2. Since ue is locally lipschitzean and H(x, . ) is convex, by Proposition II.5.1 it is e n o u g h to check (5.17) at all points of differentiability of ue. At such a point (x,t), Due is the single element of D - u e c o m p u t e d in L e m m a 5.6 (v) a n d

~e(x, t) = ~(ye) + ~ - c ~ Ix - yel ~ s

By (5.9)

u(ye) + g ( y e , p e ) <_0

(5.18)

because Ps c D - u ( y e ) (see L e m m a 5.6 (v)). T h e n the left-hand side of (5.17) for U = ue is ( - C + 1) e -Ct -Ix -- yel2 + u(ys) + H ( x , p e ) s

< (-c --

+ 1 + 2L) e - c t Ix - yel_____2___2+ L Ix - ~e[ < L m s s

ct/2,

--

where in the first inequality we have used (5.16), (5.18), a n d to get the last inequality we have chosen C -- 2L + 1 a n d used (5.14). This proves the claim. Next define We(x, t) := w(x) + 2Lms eCt/2 which is a supersolution of

Ut + U + H(x, D~U) >_ Line e Ct/2

in f~ x ]R

by (5.15). Choose te large e n o u g h to have We(x, te) > Ilull~r >_ ue(x, te) (this is possible if m > 0; if m = 0, t h e n u - 0 a n d the conclusion is reached by a s t a n d a r d comparison t h e o r e m for (5.4)). T h e n ue _< We on 0 ( a x ] - e c , te[) because for

x c Of~ u e ( x , t ) <_ u(x) < w(x) < W e ( x , t ) for all t E N. Since ue a n d We are b o u n d e d a n d they are, respectively, a subsolution and a supersolution of the same H a m i l t o n - J a c o b i equation in l] x ] - o c , tE[, we can apply the comparison T h e o r e m 1.3 a n d get ue <_ We for x E f~ a n d t < tE. Now we fix a n y such t a n d let s -~ 0 + to get u < w by L e m m a 5.6 (ii).

5. BILATERAL S U P E R S O L U T I O N S

347"

REMARK 5.7. In the next subsection we show that, for the Cauchy problem, the unusual boundary condition (5.10) is equivalent to a more classical-looking one, see Definition 5.14. For the Dirichlet problem one might hope to replace (5.10) with the condition (5.19)

lim inf u(y) = g(x)

7"~y-~x

for all x C 0~t,

where 7~ c Ct is the set of starting points from which the system can reach 0~. However, the following example shows t h a t this b o u n d a r y condition does not ensure uniqueness. Consider the equation u + IOu/Oxll -- 1 in ~ = ~ N \ {0} and the terminal cost g - 0. Then v(x) = 1 - e x p ( - I X l l ) on the xl axis, and v - 1 elsewhere (so v = v.). A different bilateral solution u of this HJB equation satisfying (5.19) is obtained by setting u -- v except at the points of the x] axis where x l < 0, and u = 1 at such points. <] REMARK 5.8. It is easy to modify the proof of Theorem 5.5 to get the following comparison principle. If (5.6) and (5.7) hold, 5 > 0, and u e B L S C ( B ( ~ , 5 ) ) , w E BLSC(-~) are supersolutions, respectively, of - u - H ( x , Du) > 0 in B ( ~ , (~) and of (5.15), such t h a t u _< w on 0 ~ , then u < w in ~. In fact, it is enough to modify the approximating u~ by taking the infimum over B ( ~ , 5) instead of ]~N in their definition (5.13), then observe t h a t u ~ / z u(x) as E ---* 0 + for all (x, t) E ~ • ]~ and check t h a t uE satisfies the inequality (5.17) in ~ • I~ for all E < e-Ct/25/m. <3 REMARK 5.9. If the running cost ~ does not depend on a, A is compact and the system is convex (i.e., f ( x , A) is convex for all x, as it happens, for instance, with relaxed controls, see w then v = v., see Exercises 5.9, 5.10. Therefore we can use Theorem 5.5 to prove verification theorems such as Corollaries 2.18 and 2.19 of Chapter III, by taking bounded supersolutions of (5.4) as falsification functions and bounded supersolutions u of (5.9), u < g on 0 ~ , as verification functions. <] Next we discuss the connections among the bilateral supersolutions and the other notions of non-continuous solution introduced in this chapter. We begin with the envelope solutions of Section 2. COROLLARY 5.10. Assume (5.6), (5.7), (5.8) and g continuous at each point of 0~. Then v. is the e-solution of (5.4). PROOF. By (5.8) and Theorem 2.6 g* is a subsolution of the HJB equation in (5.4). By the continuity of g on 0 ~ , g* is a subsolution of the Dirichlet problem (5.4) which is continuous at the points of 0 ~ , so the assumptions of Theorem 3.4 (i) are verified. Then the minimal bounded supersolution of (5.4) is the unique e-solution of (5.4) and we conclude by Theorem 5.5. *1 We omit the proof of the next result, which compares supersolutions of (5.9) with subsolutions of the HJB equation and with v-subsolutions of the boundary conditions defined in Section 4.

348

V. DISCONTINUOUS VISCOSITY SOLUTIONS

THEOREM 5.11. Assume (5.5), (5.6), (5.7), A _> 0, and f~ C_]~N open. Then (i)

for any u E LSC(f)) supersolution of (5.3), u* is a subsolution of (5.1);

(ii)

for any u e USC(~) subsolution of (5.1), u, is a supersolution of (5.3);

(iii)

for any solution of (5.1) (Definition 2.2) such that (u*), = u,, u, is a bilateral supersolution of (5.1).

Assume, in addition, that g is continuous at each point of Of~. Then

(iv)

for any u E LSC(]~ N) supersolution of (5.9), u* is a v-subsolution of (5.4) (see Definition 4.1);

(v)

a bilateral supersolution of (5.4) is a v-solution of (5.4).

Note that (iii) and (v) are immediate consequences of (ii) and (iv), respectively. Some hints for the proofs of the other statements are given in Exercises 5.5 and 5.6. Since the bilateral supersolution of (5.4) is unique, while v-solutions are not unique, the implications in (iv) and (v) cannot be inverted. Here is an explicit example, which is essentially Example 4.18. EXAMPLE 5.12. The bilateral supersolution of u+]u'[=l

inR\{O},

u(0) = 0, is 1 - e -Ixl, whereas w(x) = 1 for x ~ 0, w(0) = 0, is a lower semicontinuous v-solution, but it does not satisfy - u - lull + 1 _> 0 at 0. <~ We end this subsection with a stability result whose easy proof is left to the reader. PROPOSITION 5.13. Assume (5.6), (5.7), (5.8), and let Hn be a sequence of continuous HamiItonians converging to H uniformly on compacta of ]~2N. If un is a bilateral supersolution of (5.4) with H replaced by Ha, the sequence Un is locally uniformly bounded and in a neighborhood of Of) Un >_ g for all n, then the lower weak limit limninf,un(x ) is the bilateral supersolution of the limit problem (5.4). 5.2. F i n i t e h o r i z o n p r o b l e m s w i t h c o n s t r a i n t s on t h e e n d p o i n t o f t h e t r a j e c t o r i e s

In this subsection we consider the initial value problem

(5.20)

fu~ + )~u + H(x, Du) = 0

in ]~N

"Lu(x, 0) = g(x)

in •N,

•

]0, T[,

where ut = On~Or, Du = D~u, A >_ O, T C ]0, + ~ ] , the Hamiltonian is given by (5.5) and g is lower semicontinuous. The associated optimal control problem is the minimization of the finite horizon cost functional

(5.21)

J( ~ , t , ~ ) :=

i'

e(y~,~)e-~'ds+e-~My~(t,~))

5. BILATERALSUPERSOLUTIONS

349

over .4, with the same notations of the previous subsection. In w of Chapter III we studied this problem for continuous terminal cost g, whereas here we assume g merely lower semicontinuous. This allows us to consider problems with the constraint that the endpoint of the trajectory yx(t, a) belongs to a given closed target set T: given a terminal cost defined on T, ~ C B L S C ( T ) , we want to minimize (5.21): with g replaced by ~, over controls a E .4 such that

(5.22)

yx(t, ~) e ~-.

This constrained minimization problem can be rewritten as a free minimization problem, provided we are interested only in horizons t < T for a given T < +c~. In fact we define

f'~(x) g ( x ) : = le;~Tc (5.23)

if x C T, ifx~T,

f s~_p I~'1+ 2 T s u p [el + 1

if A = 0 ,

C := ~ 1 e -:~T [s~_P]~l+ - ~ 2suple[+l

ifA>0,

and consider the problem of minimizing (5.21) with such g over all .4. Note that g c BLSC(]~N), because T is closed, and g _< C. Clearly the value function of this problem

v(x, t) := inf J ( x , t, a) aEA coincides with the value function of the constrained problem for all (x, t) such t h a t there exists an admissible control, that is, a control a satisfying the terminal constraint (5.22). On the other hand, it is easy to see that v(x, t) > C' if and only if an admissible control does not exist, where s

[~] + T sup le] CI = ~-P 1 - e -AT [ s u p [gl + - sup It[ ~:r A

if A = 0, if A > 0

In other words, the big constants C and C ~ replace + o c when the terminal constraint cannot be fulfilled by any control a. In view of the poor regularity of the initial d a t a g we do not expect solutions of (5.20) to be b e t t e r than lower semicontinuous. Since the HJB equation in (5.20) can be rewritten in the form (5.1), the definition 5.1 of bilateral supersolution applies. The initial condition will be interpreted as follows. DEFINITION 5.14. of (5.20) if it is a coincides with g at for all n, such t h a t

(5.24)

We say t h a t u E L S C ( R N • [0, T D is a bilateral supersolution bilateral supersolution of the HJB equation in R N • ]0, T[, it t = 0, and for any x C ]I~N there exist xn -* x, tn --~ 0 +, tn > 0 g(x) = limn u ( x , , tn); in other words,

g(x) = ~( x , o ) = ( u , ~ x ] O , T [ ) , ( ~ , 0 ) .

<

350

V. DISCONTINUOUS VISCOSITY SOLUTIONS

We assume for simplicity that g - 0 so that the Hamiltonian becomes

H ( x , p ) := sup{ - f ( x , a ) . p } .

(5.25)

acA

We know from w of Chapter III that we can always reduce to this case if the running cost g satisfies (5.7) and A -- 0. For the general case see Exercise 5.11. Under this assumption, if we find the value function

v~(x, t ) : = inf e-~t g(yx(t, a))

(5.26)

aEA

for a given value of A, we can easily get it for all A because

v~(x, t) = e-~tvo(x, t). Moreover, we can consider a very general g satisfying merely (5.27)

g : ]RN - - * l~ tJ { - c o , +oo}

lower semicontinuous,

and transform the problem into an equivalent one with terminal cost in BLSC(RN). It is enough to take a strictly increasing function ~ mapping the extended real line into a compact interval, for instance, (5.28)

r

2

~b(-oc)=-l,

7r

~p(+oc)=l.

If we take ~ o g C BLSC(II~ N) as a new terminal cost and, for instance, A = 0, the new value function is

V(x, t) := inf r

a))) = r

t)),

aEA

where v(x,t) : inf~emg(yx(t, a)) is the original value function. Thus, once we know V, we can immediately recover v by taking ~b-l:

v(x,t) :

-co

if V(x, t) = - 1 ,

tan( 89

i f l Y ( x , t ) ] 74 1, if V ( x , t ) = 1 .

I.+~

Therefore the following theory is developed just for a bounded terminal cost, without loss of generality. Note that with a g as general as in (5.27) we can cover the case of the constraint (5.22) on the endpoint of the trajectories even for an unbounded ~ E LSC(T), by extending it to g = +oc on T c. PROPOSITION 5.15. Assume (5.6), (5.25),)~ > O, g E L S c ( ~ N ) , and consider the value function v : v~ given by (5.26). Then its lower semicontinuous envelope v. is a bilateral supersolution of (5.20) with T : +oc.

5. BILATERAL SUPERSOLUTIONS

351

PROOF. By s t a n d a r d arguments (i.e., the proofs of Proposition III.3.5 and Theorem 2.6) v, is a supersolution of the HJB equation. The Backward Dynamic P r o g r a m m i n g Principle is

v(x, t) > e~tv(yx(-~ , ~), t + ~-) for all x E RN, t, T > 0, a : ]--Oc~ 0] --* A measurable, see Proposition III.3.20. Its proof under the current assumptions is easily obtained as in Chapter III. We deduce from it t h a t v, is a supersolution of

- u t - ~u - H ( x , D u ) >_ 0

in ~ N • ]0, +r

by s t a n d a r d arguments. It remains to check the initial condition. By definition v(x, O) = g(x), so

v,(x,O) <_ g ( x ) .

(5.29)

We claim t h a t for any x~ --+ x, tn -+ 0 +., (5.30)

lira inf v(xn, tn) >_ g(x), n

and so the equality holds in (5.29). Moreover, it is easy to construct a sequence with tn > 0 for all n such t h a t the equality holds in (5.30). It is enough to fix z E ~ N c~ E A and t > 0 such that y~(t,c~) = x and choose x~ :-- y~(t - 1In, a). Then there is a t r a j e c t o r y starting from xn and reaching x in time 1/n, thus v(x,~, l / n ) <_ g(x)e -~/'~ and by (5.30) we get lim,~ v(x,~, l / n ) = g(x), which completes the proof

of (5.24). To prove the preceding claim we choose an E Jl such that

v(x,~,tn) > e-~t"g(yxn(t~,a,~)) - 1 n By the assumption (5.6) and the standard estimates on the trajectories, there is a constant c such that (5.31)

lYxo(tn, an) - x ~ I < ctn

for all n

and so yx,(tn, an) ---* x as n --* oc. Therefore, by taking the liminf of both sides of (5.31) and using the semicontinuity of g, we obtain (5.30). The next result is the main one of this subsection. Some of its assumptions are weakened in a subsequent remark. THEOREM 5.16. A s s u m e (5.6), (5.25), ~ _> 0, T E ]0, +oc], and g E B L S C ( R N ) . Then v, is the minimal bounded supersolution of (5.20) and the maximal supersolution u C B L S C ( R N x [0, T D of (5.32)

- u t - Au - H ( x , Du) >_ 0

in ]~N • ]0, T[, such that u ( . , O) <_ g and (5.33)

u = (ul~tN •

9

In particular, v, is the unique bounded bilateral supersolution of (5.20).

352

V. DISCONTINUOUS VISCOSITY SOLUTIONS

We r e m a r k t h a t the regularity a s s u m p t i o n on u at time t = 0 (5.33) cannot be dropped. A trivial c o u n t e r e x a m p l e when A = 0 is w := v. + 1 if t > 0, w := g if t=0. PROOF OF THEOREM 5.16. We split the proof in two steps. The goal of Step 1 is to extend a solution of (5.32) in R N x ]0, T[ satisfying (5.33) to a solution of (5.32) in ]~N x ]--1, T[. T h e n what remains to prove is a comparison s t a t e m e n t very similar to T h e o r e m 5.5, which is tackled in Step 2. Step 1. The a r g u m e n t of this step is an extension to semicontinuous functions of L e m m a II.2.10. Let u E B L S C ( R N x [0, T[) satisfy (5.33). Define ~ u ( x , t) U ( x , t ) := 'l e-At sup u(x,O) ( xeR N

if t _> 0, if t < 0,

and note t h a t U E BLSC(]R N • ] - 1 , T D. Clearly U solves (5.32) for t < 0. To prove t h a t U solves (5.32) at t = 0 take ~ G C1(]1~N-F1) such t h a t U - ~ has a strict m i n i m u m i n / ~ ( 5 , r) x [ - r , r] at (5, 0). B y (5.33) there are sequences xn --* 7, tn --~ 0, tn > 0 for all n such t h a t u(x•, tn) ---* u(5, 0). Define for t > 0 Cn(x, t) = u(x, t) - (~(x, t) - t ~ / t ) , and denote by (zn, s~) a m i n i m u m point of r i n / ~ ( 5 , r) x ]O, r]. We claim t h a t there is a subsequence, which we do not relabel, such t h a t (z,~, s~) --+ (5, 0) and u ( z , , sn) ---* u(5, 0) = V(5, 0). T h e n we can use (5.32) for t > 0 to get t2 - Au - H ( z ~ , D ~ ) > O. - ~ t - s--~n - , where all the functions are evaluated at (z,~, sn). Now we can erase the negative 2 2 and let n ~ + o c to o b t a i n term -tn/S,~ - ~ t (5, 0) - AU(~, 0) - H ( 5 , D ~ ( 5 , 0)) > 0, which is the desired inequality. To prove the claim assume (zn, s,~) --* (5, t') and ~Pn(zn, sn) -* b. Since ~p,~ _> u - ~, (5.34)

b _> lirninf(u - ~)(Zn,Sn) >_ (u - ~ ) ( 5 , ~ ) .

On the other hand, we can pass to the limit in ~ ( z ~ , Sn) < ~ ( x ~ , t , ) = (u - ~)(x~, tn) + t~ to get (5.35)

b _< (u - p ) ( 5 , 0 ) .

5. B I L A T E R A L

SUPERSOLUTIONS

353

Since (5, O) is a strict m i n i m u m point of u - ~ i n / ~ ( ~ , r) x [0, r], (5.34) and (5.35) give (~, t ) = (~, 0) and b = ( u - ~)(~, 0). Finally (u - p)(~, 0) _%
wt + )~w + H(x, Dw) >_ 0

in R N x ]0, T[

and u( 9 0) ___ w ( . , 0), then u _< w in ]RN x [0, T[. All the conclusions t h e n follow from Step 1 and P r o p o s i t i o n 5.15. The case T = + o c is covered by the comparison principle in R e m a r k 5.8. This is clear if ,k > 0, while for A = 0 we replace u and w with e-tu and e-tw, respectively, which satisfy (5.32) and (5.36), respectively, with ,k = 1, by P r o p o s i t i o n II.2.7 (which is easily seen to work under the current assumptions). Now consider the case T < + o c and restrict ourselves to A = 0. For the general case it is enough to replace u and w with e~ttt and e~tw, respectively. The proof is very similar to t h a t of T h e o r e m 5.5, and actually simpler. The main difference is the replacement of the a p p r o x i m a t i o n of L e m m a 5.6 with the following one (5.37)

uE(x,t) := uelI ~Ninf { u ( y , s ) + e - a [x-Yl~E~-g---+ ~[t-sl2 }, -l<s
where we choose c = 2L. By the a r g u m e n t s of L e m m a 5.6 and L e m m a II.4.12 it is easy to see t h a t ue is locally lipschitzean, ue(x,t) 7 u(x,t) as r --, 0 + for all x E R N, 0 < t < T, and for E < [(T - t) A 1]/m, m = (21luH~r 1/2, the inf in (5.37) is a t t a i n e d at some (Ye, se). Moreover, at any point of differentiability of u~,

Due(x, t) - 2e-a(x - y~) s

'

Ou~ (x, t) = 2 t - s~ Ot ~,2

ce-alx - y~l 2 s '

and (Due(x, t), 2(t - s~)/z 2) e D~,,)u(y,, ss). T h e n we use (5.32), the inequality IH(x,p) - H(y,p)] <_ L Ix - Yl IP], and the choice c = 2L to get

(5.38)

00---~ ~ + H(x, DuE) <_ 0

in A N x ]0, T - ern[

at every point where u~ is differentiable. Thus (5.38) holds in the viscosity sense as well, by the convexity of H(x, . ) and P r o p o s i t i o n II.5.1. Since for all x

u~(~, 0) <_ u(~, o) _< w(~, 0), we can use the comparison principle for initial value problems (see T h e o r e m III.3.7 and Exercise 1.5) to get uE(x, t) <_ w(x, t) in A N x [0, T - r and we conclude by letting E --* 0 +. 9

354

V. DISCONTINUOUS VISCOSITY SOLUTIONS

REMARK 5.17. By a classical compactness result for relaxed controls the bilateral supersolution of (5.20) v, is, in fact, the value function v r corresponding to relaxed controls .4 r. The compactness theorem states that, if A C_C_R M is compact, then for any sequence Pn E .A.r there is a subsequence, which we do not relabel, such t h a t Pn converge weak star on [0, T] to # C fir and the corresponding trajectories Yx,(" ,#n) converge uniformly to Yx(" ,p) if xn -~ x (see w of Chapter III for definition and notations and [Wa72] for more details and the proof). In order to show that v, = v r it is enough to show that v r is lower semicontinuous~ by the uniqueness Theorem 5.16 and the fact that relaxed controls give rise to the same Hamiltonian H (see w Take (xn, tn) ---* (x, t) and Pn C .4 r such t h a t vr(xn, tn) = g(Yxn (tn, pn))e -~t~ (such optimal controls exist by the previous compactness theorem, but c-optimal controls would do the job anyway). By extracting a subsequence as before and using the semicontinuity of g we get l i m i n f v r ( x n , t n ) > g(yx(t,#))e -~t > vr(x,t), which completes the proof. It can also be shown t h a t for any convex system (i.e., such that f ( x , A) is convex for any x) the value function is lower semicontinuous, thus it is the bilateral solution of (5.20) (see Exercise 5.8). <3 REMARK 5.18. By Theorem 5.16 and Remark 5.17 we can get verification theorems as in w see Remark 5.9 and Exercise 5.14. <3 REMARK 5.19. Theorem 5.16 admits an extension to the case of a locally lipschitzean system f. The different proof also allows us to compare v, with locally bounded supersolutions of (5.20) and (5.32). Here is a more precise statement. Assume g E L S C ( ~ 2r is locally bounded, f satisfies (5.6) with the global Lipschitz condition replaced by the following: for all R > 0 there is LR such that for all x, y C B(0, R) and a E A, If(x, a) - f ( Y , a ) l <- LRI x -

Yl,

and (5.25), A = 0, T E ]0, +oc[. Then v, is the unique locally bounded bilateral supersolution of (5.20). This follows from a comparison principle on the cones c~0 = { ( x , t ) : 0 < t < T, Ix - x0l _< c(T - t) }

for Hamiltonians such that ]H(x,p) - H ( x , q)I -< clP - q] as in the proof of Theorem III.3.15. Here is the local comparison principle: if u is a BLSC supersolution of (5.32) in a neighborhood of Cx0 and w E BLSC(C=o) is supersolution of w t + H ( x , D w ) >_OinC~ o such that u ( - , 0 ) < w ( - , 0 ) , t h e n u _ < w i n C x 0 . This can be proved as in Step 2 of the proof of Theorem 5.16 by means of the local comparison principle between sub- and supersolutions, see Theorem III.3.12 and Exercise 1.6. We leave the details to the interested reader. <3 Next we give a stability result and discuss some applications. THEOREM 5.20. Under the hypotheses of Theorem 5.16 assume gn E BLSC(RN), gn / z g as n --* +co. Let Un and u be, respectively, the bounded bilateral solution of (5.20) with initial data g~ and g. Then un / z u as n ---* +~c.

5. BILATERALSUPERSOLUTIONS

355

PROOF. Extend u and all un for t < 0 by setting t h e m equal to e -At supg. Then all un solve (5.32) in ]~N X ]--OO, T[ by Step 1 of the proof of Theorem 5.16. This theorem implies that u,~ is a nondecreasing sequence, so we can define w :-- sup~ u~. By Lemma 2.18, w is the lower weak limit of {un}, thus w E BLSC(]~ g x ] - 1 , T D is a supersolution of both (5.32) in I~g • ] - 1 , T[ and (5.36) by the stability result Theorem 1.7. Moreover, w(x,O) = g(x) by definition. By Step 2 of the proof of Theorem 5.16 there is at most one solution to this problem, and by Step 1 the bilateral supersolution of (5.20) is such a solution. Then w = u and the proof is complete. 9 A first consequence of this stability theorem is that we can approximate the value function of a finite horizon problem with discontinuous terminal cost g by much smoother value functions corresponding to more regular costs g,~. If we define g~ as the inf-convolution of g E BLSC(I~ N) corresponding to the parameter a = an, with an ---* 0 +, we have that gn ,7 g and each gn is semiconcave in ]1~N by the proof of Lemma II.4.11. The value function

vu(x, t) = inf e-~tgn(yx(t~ a)) aEA is locally lipschitzean and it is the bilateral supersolution of (5.20) with g replaced by gn, by Theorem 5.16. By Theorem 5.20 v~ / v. as n --~ +~c where v is the value function with cost g, (5.26). This fact is useful because we can apply to the approximating problems the theory on necessary and sufficient conditions of optimality developed in w of Chapter III: see Theorem III.3.37 on necessary conditions where the continuity of vn is enough, and Theorem III.3.38 on sufficient conditions where the Lipschitz continuity of v~ is needed. If, in addition, the system f satisfies (5.39)

[f(x - h, a) - 2f(x, a) + f ( x + h, a)l < M Ihl 2

(e.g., if f is C 2 with respect to x with bounded second derivatives), then each v,~ is locally semiconcave by Theorem III.3.26. Therefore we can apply to the approximating problems the necessary and sufficient conditions of optimality in the form of the Minimum Principle of Theorem III.3.39. If instead of (5.39) we assume that f ( - , a ) is C 1 for all a, then the Extended Pontryagin Maximum Principle, Theorem III.3.44, applies to the approximating problems. The finite horizon problem with the constraint (5.22) on the endpoint of the trajectories can be approximated in a more explicit way by penalizing smoothly the violation of the constraint. We assume the terminal cost ~ is the restriction to the target 2- of a semiconcave function ~ : ]~N ___+1~. Then we approximate the discontinuous terminal cost g defined by (5.23) with (5.40)

gn(x) :----min{ ~(x) + n dist2(x, 2-); e~Tc } .

Clearly g n / z g, and it is not hard to show that all g , are semiconcave. Therefore all the previous remarks on the regularity of vn and on the applicability of the theory of Chapter III still hold.

356

V. DISCONTINUOUS VISCOSITY SOLUTIONS

Exercises 5.1. Prove that a function g satisfying (5.8) is a supersolution of (5.9). 5.2. Prove Theorem 5.5 under the assumptions (A0)-(A3) of Chapter III on the system (f, A) instead of (5.6). 5.3. Extend Theorem 5.5 to the equation H(x, u, Du) = 0 with

g ( x , u,p) := sup{ - f ( x , a) .p - e(x, a) + k(x, a)u }, a6A

where f satisfies (5.6) (or the assumptions of Exercise 5.2), g satisfies (5.7) except the condition of boundedness which can be replaced by ]g(x, a)] < M(k(x, a) + 1) for all x, a, and k : ]1~g X A --+ R is continuous, bounded above in K: x A for all compact/C, bounded below by a constant k0 > 0, and lipschitzean in x uniformly with respect to a. 5.4. Prove that the uniqueness statement of Theorem 5.5 is false if the equation

u + H(x, Du) = 0 is replaced by H(x, Du) = O. [Hint: use the example in Remark II.5.11.] 5.5. Prove the statements (i) and (iv) of Theorem 5.11. [Hint: consider the approximation of Lemma 5.6 and use Lemma 2.18. The result can be checked in [Sor93b].] 5.6. Prove statement (ii) of Theorem 5.11. [Hint: use the approximation u (E)(x) = supye~ { u(y) - Ix - yl~/s ~ } for a suitable choice of y > 0. The result can be checked in [Ba93].] 5.7. State and prove some result on the stability of the bilateral supersolution of (5.4) with respect to vanishing viscosity approximations and to perturbations of the domain, such as Propositions 4.10 and 4.12 for v-solutions. 5.8. Prove that the value function of the finite horizon problem is lower semicontinuous if the running cost t~ does not depend on a, A is compact and f ( x , A) is convex for all x. [Hint: show that f ( x , A) = fr(x, Ar), where "r" indicates relaxed controls, and use a suitable selection lemma, such as Lemma B in the Appendix of [FR75], see also the proof of Corollary 1.4 in Chapter VI.] 5.9. Assume A is compact and g does not depend on a. Prove that the relaxed value function for the problem with exit times of w is lower semicontinuous. [Hint: see the proof in Remark 5.17.] 5.10. Prove the first statement of Remark 5.9. [Hint: use Exercise 5.9 and the hint to Exercise 5.8.] 5.11. Extend Theorem 5.16 and Remark 5.19 to the case of Hamiltonian H given by (5.5) instead of (5.25) with g satisfying (5.7). [Hint: in Step 1 extend u for t < 0 as Ge -kt for suitable large constants G and k. In Step 2 add to w a term of the form zke ht for suitable constants k and h.]

6. BIBLIOGRAPHICALNOTES

357

5.12. Give a direct proof that u E BLSC(s • [0, TD, ~ _c ~N open, solution of (5.32) in ft • ]0, T[ can be extended to a solution on f~ • ] - o c , T[ if and only if u(x, O) -- (UlI~N• 0) for all x such that D~x,t)u(x, 0) ~ O, or for all x such that D-;u(x, O) ~ O. 5.13. Assume (5.25), A = 0, T E ]0, +oc[, g e LSC(~N), f satisfies (5.6) with global Lipschitz continuity replaced by local, as in Remark 5.19. Let u E LSC(]R g • [0, T D be a supersolution of (5.20) (respectively, of (5.32) such that u ( - , 0 ) < g and (5.33) holds). Prove that u >_ v. (respectively, <). [Hint: use the change of variables (5.28).] 5.14. Give the verification theorems for the finite horizon problem of w corresponding to each comparison result of this subsection: Theorem 5.16, Remark 5.19 and Exercises 5.11 and 5.13. 5.15. Extend Theorem 5.20 to bilateral solutions un of (5.20) with H replaced by H , satisfying the assumptions of Theorem 5.16 a n d / I n "~ H. 5.16. Under the assumption of Theorem 5.16 let gn E BLSC(R N) be uniformly b o u n d e d , / I n E C(]~2N)~ Hn ~ H on compacta. Let u~ be bilateral supersolutions of (5.20) with H, g replaced b y / I n , gn and set u = lim inf.un. Prove that u is the bilateral supersolution of (5.20) with g := u ( . , 0). n 5.17. Extend the results of w

to systems f(t, y, a) depending continuously on t.

6. B i b l i o g r a p h i c a l

notes

The possibility of extending the definition of viscosity sub- and supersolutions and the comparison principles to semicontinuous functions was remarked in [CEL84, CIL87, I87a], see in particular [I88b]. The "half-relaxed" semilimits that we simply call weak limits in Definition 1.4 are a special case of the general notion of F-limit in the Calculus of Variations (see, e.g., [DM93]). They were introduced in the theory of viscosity solutions by Barles and Perthame who proved the stability results Theorem 1.7 [BaP87] and Corollary 1.8 [BaP88]; this method is very powerful, as we show in Chapters VI and VII, and it is sometimes called the Barles-Perthame procedure. The definition of non-continuous viscosity solution 2.2 is due to Ishii [I87a], as well as the connection with optimal control, Theorem 2.6 [I89]. The Zermelo Navigation Problem, one of the most classical problems in Optimal Control, was studied by Carath~odory [Ca35] and many other authors in the '30s (see the references in [BSt93]) and can be found in the books [Lei66, Ce83]. Example 2.10 is taken from [Sus89b]. Section 2.2 follows Ishii [I87a], Proposition 2.16 appears in [I88b]. Perron's method is a very versatile tool in the existence theory of viscosity solutions; it was used in [CDL90] for constrained solutions and extended to weakly coupled systems of HJ equations [EL91], integrodifferential equations [LeYgl], as well as second order degenerate elliptic equations, see [CIL92] and the references therein.

358

V. DISCONTINUOUSVISCOSITYSOLUTIONS

Section 3.1 gives a viscosity solutions approach to a result obtained by Subbotin [Su93a, Su95] in the framework of minimax solutions, see also [RS88]. The definition of envelope solution is new and it is inspired by the classical Wiener's notion of generalized solution of the Dirichlet problem for the Laplace equation in nonsmooth domains [W25]; Subbotin's original definition of (generalized) minimax solution is essentially the equivalent property of Proposition 3.6. Theorem 3.4 was extended to second order degenerate elliptic equations in [BB95] where the stability properties of e-solutions are also studied. Section 3.2 follows [BSt93]. The boundary conditions in the viscosity sense of w appear in [BaP87] and are systematically studied by Ishii [I89] and Barles and Perthame [BaP88]. Proposition 4.12 and Theorem 4.13 are taken, respectively, from [Sa90] and [I89]. The definition of solution for discontinuous Hamiltonians in Exercise 4.1 is also in [I89], while the example is taken from [Bag0a]; in [Ba90a] it is also proved that for a large class of unbounded control problems with discontinuous Hamiltonian the value function is the maximal subsolution. Section 4.2 follows [I89] in the case of bounded domains, Theorem 4.16, and [BS94] for unbounded domains, Theorem 4.20. The definition of usable part of the boundary comes from the theory of differential games [Is65]; Theorem 4.24 is taken from [BS94]. The proof of Theorem 4.25 can be found in [CGS91]. Section 4.3 follows [BSt93]; statement (iii) of Corollary 4.30 is due to Barles and Perthame [BaP87] and it is the first uniqueness theorem for discontinuous viscosity solutions in the literature. Related papers are [BaP90~ T92]. The theory of bilateral supersolutions in w originates from the work of Barron and Jensen [BJ90] as revisited by Barles [Ba93]. Section 5.1 follows Soravia [Sor93b] and extends a bit his results because we do not assume the continuity of the boundary data. Discontinuous data that do not satisfy (5.8) were considered in [Bla97]. Some boundary conditions different from (5.10) but equivalent to it were studied recently in [WZ97] and [KN97]. The proof of Theorem 5.11 is in [Ba93]. Section 5.2 gives a new proof of the main result of [BJg0, BJ91]. The methods of this section were applied to equations with singular coefficients in [K96b, CamS95]. A different approach to control problems with semicontinuous value function was pursued by Frankowska using contingent derivatives and viability theory, see [Fr89b, Fr93]; a uniqueness theorem for the HJB equation and connections with the viscosity solutions theory of w are in [Fr93] (see also [FPR94]). The same approach was used in [CQS94b] for time-optimal control with state constraints. Further recent results on discontinuous viscosity solutions are in [BL96b, BL97, Sor96b, BG97].

CHAPTER VI

Approximation and perturbation problems In this chapter we consider some approximation and perturbation problems for Hamilton-Jacobi equations. The results are presented for the model case Au(x) + sup{ - f ( x , a). Du(x) - g(x, a) } = 0

(HJB)

in IRN,

aEA

although many of them hold for more general Hamiltonians and boundary value problems, as indicated in the remarks. Singularly perturbed versions of (HJB) motivated by discrete time and stochastic control are analyzed in w and w respectively. On the other hand, in w we consider regular perturbations of (HJB). The methods of viscosity solutions and, in particular, the weak limits technique described in Chapter V, w (see Definition V.1.4, Theorem V.1.7 and Corollary V.1.8), prove to be effective in providing simple proofs of general convergence and stability results in the uniform norm. Moreover, estimates on the rate of convergence are established via appropriate implementations of the basic ideas appearing in the proofs of comparison theorems for viscosity solutions. 1. S e m i d i s c r e t e

approximation and s-optimal f e e d b a c k s

In this section we develop a theme already touched upon in Chapter I, w namely, the convergence as h --. 0+ of the solutions of the functional equation (HJB)h

uh(x)+sup{-(1-)~h)uh(x+hf(x,a))-ht(x,a)}=O,

xe]R N

aEA

to the viscosity solution of (HJB)

)~u(x) + sup{ - f ( x , a). Du(x) - g(x, a) } = 0,

x E N N,

aEA

and its application to the approximate synthesis of optimal feedback maps for the infinite horizon problem. The existence, uniqueness and regularity theory for (HJB)h, as well as its relationship with discrete time dynamic programming, are developed in the appendix

360

VI. A P P R O X I M A T I O N

AND PERTURBATION

PROBLEMS

of this chapter, Section 4. Equation (HJB)h can be interpreted as an approximation of (HJB). Indeed, if u E CI(RN), then its directional derivative in the direction f ( x , a) satisfies

- h f ( x , a ) . Du(x) = u(x) - u(x + h f ( x , a ) ) + o(h) as h ~ 0 +. Hence, if u is a classical solution of (HJB), then

sup{ Ahu(x) + u(x) - u(x + h f ( x , a ) ) - h*(x,a) } = o(h) aEA

for all x E R y. So, if we compare the preceding with (HJB)h, it is reasonable to guess that Uh will be close to u for small h > 0. The section is divided into three parts. In 1.1 we prove (Theorem 1.1) the convergence of uh to u under a set of assumptions which guarantee that the unique viscosity solution of (HJB) is the value function of the infinite horizon problem (see Chapter III). This result has some relevant applications to the synthesis of z-optimal controls and to the existence of optimal controls for the infinite horizon problem (see Theorem 1.3 and Corollary 1.4). In 1.2 and 1.3 some quantitative results on the rate of convergence are established, showing that sUp~N lU(X) -- Uh(X)[ behaves, for small h, as h ~ for some a E ]0, 1], depending on the regularity of u and uh. The theory of this section provides the theoretical basis to some computational techniques explained in greater detail in Appendix A. The following conditions on the data are assumed throughout the present section (1.1)

f : Rg • A

(1.2) (1.3)

If(x,a)-f(y,a)I < LIx-y It(x,a)-t(y,a)i_L1]x-y[,

~

]~N, ~ : ]~N • A --* R continuous, A compact,

I, ]t(x,a)] _< M,

for x,y E R N, a E A, and (1.4)

A > 0.

Note that (1.1) and (1.2) imply (1.5)

If(x,a)l < K(1 + Ix]).

Under these assumptions, equation (HJB) has a unique viscosity solution in BUC(]~ N) (see Chapter III), whereas (HJB)h has a unique solution in BUC(]R N) for each h E ]0, l/A[ (see the appendix of this chapter, Section 4). 1.1. A p p r o x i m a t i o n o f t h e value f u n c t i o n and construction of optimal controls The next result shows that the viscosity solution u of (HJB) (that is, the value of the infinite horizon problem) can be uniformly approximated by the solution of (HJB)h, for small h. The proof employs the weak limits technique of Chapter V.

1. S E M I D I S C R E T E

APPROXIMATION

361

THEOREM 1.1. Let us assume (1.1)-(1.4). Then sup luh(x) - u(x)l ---* 0

as

h --~+,

tC

for every compact ~ C_ N N, where Uh, u E B U C ( R N) are, respectively, the solution of (HJB)h and the viscosity solution of (HJB).

PROOF. It is easy to check t h a t M / A and - M / A , with M as in (1.3), axe superand subsolutions of (HJB)h, for any h. Hence, by the results of w sup luh(x)l <_ M / A . Ng Let us define then functions u. g by u_(z)=

liminf

uh(y),

g(x)=

limsup

(Y,h)--(x, 0+)

uh(y).

(y,h)~(x,O + )

By L e m m a V.1.5, u and g are, respectively, lower and upper semicontinuous and satisfy (1.6)

u < g

in

]I~ y

.

Let us assume temporarily t h a t u is a viscosity supersolution and g is a viscosity subsolution of (HJB) in the generalized sense of Definition V.I.1. Then, by the comparison principle, Theorem V.1.3, we obtain < u

in

][~N .

Taking (1.6) into account, then u - g. Therefore, u := g - u turns out to be a continuous viscosity solution of (HJB) and also by Lemma V.1.9, Uh

,u

locally uniformly in

]I~ N a s

h ~ 0 +.,

which proves the theorem. Let us show then t h a t _u is a viscosity supersolution of (HJB). Let xl be a strict minimum point for u - T i n / ~ = / ? ( x l , r ) , ~ E CI(NN). By Lemma V.1.6, there exist xn E / ~ and hn --+ 0 + such that (1.7)

(Uh, -- T ) ( x n ) = n ~ n ( u h , -- c2),

Xn ~

Xl,

Uhn(Xn) -----+U(Xl) .

Since uh solves (HJB)h we obtain Uh,(X,~) -- (1 -- )~hn)Uh,(Xn + h,~f(xn, an)) - hne(xn,a,~) = O,

for some a,~ = an(x,~) E A. A d d and subtract .'~hnUh, (Xn) and use the first relation in (1.7) to get (1 - Ahn)[~(xn) - ~ ( x n + h n f ( x n , an))] + ,'~hnUhn (Xn) -- h n f ( x n , an) >_ O,

362

VI. APPROXIMATION AND PERTURBATION PROBLEMS

for sufficiently small h,~. Divide now by h,~, take a subsequence rtk such that ank --+ -6 E A and let k ---* + ~ to obtain by (1.7)

-Dp(xl).

f(xl,a)

-]-/~/t(Xl) - e(xl,a)

___ 0 .

This implies that _u is a viscosity supersolution of (HJB). Now let x0 be a strict maximum point for ~ - ~ o n / ~ : = / ~ ( x 0 , r). By Lemma V.1.6 again, there exist xn E B, hn --+ 0 +, such t h a t (1.8)

(Uh, -- F)(x~) = n~ax(uhn -- ~),

xn

' x0,

Uhn(Xn)

' ~(Xo).

From (HJB)h we obtain the inequality

Uhn(X,~) -- (1 -- Ahn)uhn(Xn + h n f ( x n , a ) ) - h,~s

< 0,

for all a E A. Then, by (1.8), (1 - )~hn)[p(x~) - r

+ h n f (xn, a))] + Ah,~uh~ (xn) - hnt(xn, a) <_ 0

for all a E A. The same argument as above shows then t h a t ~ is a viscosity subsolution of (HJB). We proceed now to explain how the previous results can be applied to the construction of an approximate optimal control for the infinite horizon problem. An important tool is the following representation formula for uh:

(1.9)

Uh(X) = inf J h ( x , a ) ,

x E ~N.

aEAh

Here

Ah := { a E A : a(s) =-- a(kh), s E [kh,(k + 1)h[, k E N},

oc Jh(x, o~) := h E g(yh(k), k=0

a(kh))(1

-

Ah) k,

where yh(k) is recursively defined by

yh(O) = x,

yh(k + 1) = yh(k) + h f ( y h ( k ) , c~(kh)) .

The proof of (1.9) can be found in w Note t h a t this discrete time system is the Euler approximation scheme with step h associated with the continuous time system (S) of Chapter III, while Jh is a sort of trapezoidal approximation of the integral functional

J ( x , a) --

fO~

g(yx(t), a ( t ) ) e -At dr.

1. SEMIDISCRETE A P P R O X I M A T I O N

363

The algorithm for the construction of an approximated optimal control is as follows: let Uh be the solution of (HJB)h and, for each fixed x, pick a*a(x) E A such that (1.10)

a*h(X) E arg max{ --(1 -- )~h)uh(x + h f(x, a)) - h s aEA

a) } .

Hence a *h( X ) satisfies Uh(Z) -- (1 -- A h ) u h ( X + h f ( x , a~(x)) - h e ( x , a ~ ( x ) ) )

= 0

for all x e ]~g. Define then y~(k) by (1.11)

y~(O) = x,

y~(k + 1) = y~(k) + h f(y~(k), a~(y~(k)))

for k = 0 , 1 , 2 , . . . , and a~ E Ah by c~(s) := a~(y~([s/h]))

(1.12)

(Is~h] is the largest integer less than or equal to s/h). From (HJB)h, (1.9) and (1.10) it follows that Uh(X) = Jh(X, Ol*h) = inf Jh(x, a ) . aEAh

The simple proof of this fact is detailed in w Theorem 1.1, we have

Therefore, in the framework of

Jh(x,a*h) ----+ inf J ( z , a ) , aEA

uniformly as h -~ 0 +. The preceding discussion says that a~ can be viewed as an approximate (piecewise constant) optimal control for the infinite horizon problem and that its construction is reduced to the solution of (HJB)h. The computational aspects are discussed in Appendix A. We turn our attention now to the behavior of the piecewise constant controls a~ and the corresponding y~ as h --~ 0 +. The next lemma and Theorem 1.1 imply that {a~} is a minimizing sequence for the infinite horizon problems that is,

J(x, ~ ) ---+ inf J(x, a ) .

(1.13)

aEA

LEMMA 1.2. Assume (1.1)-(1.4). Then lim sup [ J ( x , a ) - Jh(x,a)l = O.

h~O+ aEAh

If, moreover, either )~ > L and f is bounded or ;~ > L + K, where K is the constant in (1.5), then I J ( x , a ) - - J h ( x , a ) l <_Ch for some C > O.

forall~eAh,

h e ]0,1/~[

364

VI. A P P R O X I M A T I O N AND P E R T U R B A T I O N PROBLEMS

PROOF. For fixed a E .Ah we set 9(s) = yz(s, a) and y'(s) = yh([s/h]), where yh(k) is the solution of the discrete time system starting at x, defined above. Observe that ~ can be expressed as

f[s/h]h ~(s) = x + ./o

f ( ~ ( r ) , a(T)) dr,

s > O.

Then M s ) - ~(s)t

<

~0 [s/h]h I f ( y ( 7 ) , a(r)) - f(ff(7), a(T)) I d T + jf[s~ If(y(T), a(T)) I dT. /h]h

Hence, using (1.2) and (1.5), ly(s) - ~(s)l <_ L

/0

ly(r) - ~(r)[ dr + K

f;

/h]h

(1 + ly(r)l) d r .

By estimate (III.1.4) the above gives ]y(s) - ~(s)l < L

ly(T) - ~(T)I dr + K(1 + (Ixl +

2v/'~K~s)eKS)h

(where we used the inequalities [s/h]h < s < ([s/h] + 1)h). Hence, by Gronwall's Lemma, (1.14)

lY(S) - g(s)l _< K h ( 1 + (Ixl

2v/~-88)eKs)eLs .

+

Now a simple computation shows that (1.15)

IJ(x,~)--Jh(x,c~)l<Xl+X2+X3;

VctEAh,

with X1 =

h IT~hi-1 fL T e(y(s), a(s)) e -;~s ds , ~ /(g(kh), a(kh))/~ k do

k=0

X2 = h

X3 =

M e -~s ds .

k= T/h] Here M is given by (1.3), we set/3 = 1 - Ah, and T > 0 is arbitrary. Let us proceed now to estimate the Xi's (i = 1, 2, 3). It is straightforward to check that X2 + X3 < M h (1 - / ~ h ) [T/h] + M -e--AT Ah A

Since [T/h] log(1 - ,th) --~ - A T as h --+ 0 +, we conclude that for any z > 0 there exist h > 0, T > 0 such that X2 + X 3 < z,

for all 0 < h < h, T > T .

1. SEMIDISCRETEAPPROXIMATION

365

Observe next that

r T/h h X1 ~-- /0 t(~(S),C~(S))~ls/hl-t(Y(S)'a(s))e-)~S[ds+

r~ J[T Me-~Sds /h]h

so that

(1.16)

X I <_ X4 + X5 + M h

with

x 4 = f0 T le(y(s), ~ ( s ) ) -

25 =

/0

e(y(s),

It(y(,), a(s))l le -as -

~ ( s ) ) l e -as ds,

e-)~O[s/hlhIds.

Here tO--

log(l-/kh)

Now, by (1.3) and (1.14) X4 _< fo T L1 {~(s) - y(s)l e -~s ds <_ L1K(1 + Y ) h e LT < for sufficiently small h. The estimate for X5 is obtained by an application of the mean value theorem to the function s ~ e -~s. Since [s/h]h < s < (Is/h] + 1)h this gives

X5 <_ T M A Is - ~)[s/h]h I <_ TMA((1 - O)T + Oh) . Now 0 --* 1 as h ~ 0 + and consequently X5 _< ~ for sufficiently small h. The first statement now follows from (1.15), (1.16). In order to prove the remaining part of the statement, observe that

IJ(x, c O - J h ( x , a ) l < X + Y

Vc~Ah,

where

X = ~o + ~c [g(y(s), c,(s)) - e(~(s), a(s))l e -A~ ds, Y -= ~0 + ~c le(y(s), ct(s)) I e- ,~s - e -o'~[s/hlh d s . Now (1.3) and (1.14) yield

X < KLlh

(1 + (Ix I + ~ ) e K s ) e

(L-)Os d s .

366

VI. APPROXIMATION AND P E R T U R B A T I O N PROBLEMS

If A > K + L one easily obtains from the preceding that

X <_ Ch

for some C > 0 .

In order to estimate Y let us observe t h a t by (1.3) and the mean value theorem Y _< M

max{e-P's; e-;~O[s/hlh}]As -- A~)[s/h]h] ds.

Since z9 > 1 and s <_ (Is~hi + 1)h we have

max{e-;~S; e-),O[~/h]h} < e,~),he-),s " This, together with the inequality Is - ~)[s/h]h[ < (~) - 1 + h)(s + 1), gives

Y < Mt(v~ - 1 + h)e ~

(s + 1)e -xs ds.

From the above and the fact that limh~0+(0 -- 1)/h = ,~/2 one concludes that

Y < Ch

for some C > 0 .

The preceding and the similar estimate previously obtained for X complete the proof in the ease ,~ > K + L. Under the stronger assumption M I := sup [fl < +~c the estimate ly(s)

-

~(s)l _< 3 b h e Ls

holds instead of (1.14). In this case the term X is bounded above by

L1MIh

e (L-a)~ ds

and a f o r t i o r i , if A > L, by L1M/h/(A - L).

.9

It is well known that minimizing sequences of optimal control problems may not have a limit in any classical sense due to a highly oscillatory behavior. Therefore it seems natural to approach the question of convergence of a~ as h --+ 0 + in the framework of relaxed controls. We refer to III.2.2 for the notations employed hereafter. The next result affirms, roughly speaking, that the discrete time optimal control problem infaeAh Jh(x, a) converges, in an appropriately weak sense, to the relaxation of the infinite horizon problem inf~e A J(x, a). More precisely, if we denote by y~ the solution of

(1.17)

~](t) ----f(y(t), a~(t)),

y(o) = ~,

with ~ given by (1.12), the theorem says that the pair (a~, y~) converges to an optimal relaxed pair (#*, y*) for the infinite horizon problem.

1. S E M I D I S C R E T E A P P R O X I M A T I O N

367

THEOREM 1.3. Let us assume (1.1)-(1.4). Let a*h E .Ah be the control defined by (1.10), (1.11), and (1.12), where Uh E BUC(~ N) is the solution of (HJB)h, and let y~ be the solution of (1.17). Then for any x E ~ g there exist #* E .4 r, y* : [0, +oc[ --+ 1Rg and a sequence h, --+ 0 as n ---++oc such that, for any T > O,

LT~(s,a~(s))ds--~

(1.18)

LTIA~(s,a)dp*(s)ds

for all ~ E Li(0, T; C(A)) (i.e., for all 4 : [0, T] -~ C(A) Lebesgue integrable), and Y~n ---+ Y*

(1.19)

uniformly in [0, T] .

In addition, (1.20) (1.21)

y*(t) = x +

f(y*(s),a)dp*(s)ds

lim Jh,(X, al,)----

n~+~c

fort>_O,

lira J(x,c~*h.) = Jr(x,,*) = inf J ( x , a ) .

n~+oc

aEA

PROOF. The crucial point is that for any T > 0 the restrictions to [0, T] of measures p E .4 r form a (convex) sequentially compact subset of L~162 T; A r) equipped with the weak-, topology (see [Wa72]). This means that we can select h,~ --~ 0 as n --~ +oc and/z* E .Ar in such a way that (1.18) holds true. By definition Y*hn satisfies (1.22)

y~n(t) = x +

f(y~,(s),ai,(s))ds,

t > O.

From (1.2) it follows that [y~n(t) - x[ _< L and, consequently,

/0' 2 i' ly~n(s)-xlds+Zsuplf(x,a)l, [y~n(s) - x[ ds +

If(z, a*(s))[ ds

ly~.(t)- xl < L

aEA

for t E [0, T]. By Gronwall's inequality, {y*h, } is a uniformly bounded and equicontinuous sequence of functions on [0, T]. Therefore (1.19) holds, modulo the extraction of a subsequence. To prove (1.20), observe that (1.23)

L'

[f(y~ (s), a*h, (s)) -- f(y* (s), a;~ (s))] ds ----+ 0

as n ---++oc, as a consequence of (1.2) and (1.19). The choice ~o(s, a ) : = f(y*(s), a) in (1.18) yields

So'

f(y*(s),a*h.(s))ds ---*

So'l.

f(y*(s),a)d#*(s)ds .

368

VI. APPROXIMATION

AND PERTURBATION

PROBLEMS

Combining the above with (1.23) gives (1.20). A similar argument shows that (1.24)

So'

io'i.

g(y~,(s),a*h~(s))e-;~ds - - - +

g(y*(s),a)e-S~Sd#*(s)ds,

uniformly in [0: T]. To prove (1.21) observe first IJ(x, a ~ ) - Jr(x, P*)l <- X~ + Y~ with

x~= L T ~(yL(s),~L(s))- L e(y*(s),a) d#*(s) e-~ds, :

(s),

e

-

ds.

By (1.24), Xn ~ 0 as n ~ +oo; on the other hand,

IY~I <_ 2M

/F

e -)'s ds,

as follows from (1.3). Consequently lim J(x.a~ ) = Jr(x, It*).

rt --+ - i - ~

'

n

By Lemma 1.2, limn-~+or J(x, a~n ) = limn-+oc Jhn(x, a~). From Theorem 1.1 we know that lim~_+~ Jhn(x, ~ ) = inf~e A J(x, a), so the proof is complete. As an application of Theorem 1.3 we have an existenceexistence of an optimal control result of an optimal control for the infinite horizon problem. COROLLARY 1.4. Assume (1.1)-(1.4) and (1.25)

f(x, A) • g(x, A) is convex for all x E ]I~N

9

Then there exists ~ * E ~4 such that J(x, a*) = inf J(x, ~ ) . aE.4

PROOF. Let y*, #* be as in Theorem 1.3. We observe first that (1.1) and (1.25) imply that f(x, A) x g(x, A) is convex and compact for each x E R N. On the other hand,

if(x, A r) x ~r(x, A r) = ~'6(f(x, A) x g(x, a)) for all x E ]I~ N (see Exercise III.2.10) so this set coincides with f(x, A) x g(x, A). By definition, y* and #* satisfy, for all s > 0,

L f(y*(s), a) d#*(s) E fr(y*(s), Ar),

IA

g(Y*(S), a) dp*(s) e er(y*(8), At).

1. SEMIDISCRETEAPPROXIMATION

369

This means t h a t for any s > 0 there exists a*(s) 9 A such t h a t

f(y*(s), a*(s)) = f~ f(y*(s), a) d~*(s)~ g(y* (s), a*(s)) = / A g(y*(s): a) d # * ( s ) . By general measurable selection results (see, e.g., Appendix B in [FR75]) we can assume t h a t s ~ a*(s) is a measurable function a*. By (1.20) y*(t) = x +

f(y*(s),a*(s))ds,

and therefore J(x,a*) =

/o

e -~s

fJA f ( y * ( s ) , a ) d # * ( s ) d s

= Jr(x,,*).

Then (1.21) of Theorem 1.3 gives the conclusion.

9

1.2. A f i r s t r e s u l t o n t h e r a t e o f c o n v e r g e n c e We have seen in Theorem 1.1 that supK: tUh - u I ~ 0 as h ---* 0 + for every compact ]C C_ R y . The next result shows, under an extra regularity assumption on u, that the convergence is uniform over all ~N and the rate of convergence can be estimated. THEOREM 1.5. Let us assume (1.1)-(1.4) and (1.26)

I f ( x . a ) l <_ M

for ( x , a ) 9 R N x A .

A s s u m e also

(1.27)

u E C~

for some "~ 9 ]0, 1] .

N)

Then

sup lUh(X) -- u(x)l <_ C h ~/2

for all h E ]0, l/A[,

for some constant C > 0 depending on the various bounds on the data but not on h.

PROOF. Consider for ~, ~ > 0 the auxiliary function 9 (x, y) = u ( x ) - uh(~)

Ix - yS 2 2z

~(~(x) + ~(y)),

where # E c i ( ~ N) is chosen to satisfy (1.28)

#_> 0,

#(0) = 0,

ID#l _< 1,

lim #(x) = +oc~ Ixl--*+or

370

VI. A P P R O X I M A T I O N AND P E R T U R B A T I O N P R O B L E M S

(see Exercise 1.2). It is not hard to check that the boundedness of u, Uh implies the existence of some (5, ~) (depending on 6, 5) such that

q!(g,9) >_ ~g(x,y)

for all x, y C N N .

The inequality k~(5,9) _> ~(0, 0) immediately gives I~ - 915 2------7 < C,

(1.29)

with C = 2(supRN lu(z)l + sup~N ]Uh(X)l ). Now use k~(5,9) > ~(9,9) to get, on the account of (1.27), (1.28), (1.30)

91_____~_ I~ ~ < u(~) - u(~) + 5(~(~) - ~(5)) < c 15 - ~l ~ + ~ 1 5 - 91. 26 --

From (1.29) it follows that 15 - Yl <-- v / ~ estimate

< 1 for small 6. Hence (1.30) gives the

15 - Y] < C s l / ( 2 - ' 9 9

(1.31)

From equation (HJB)h we have

Uh (~) -- (1 -- Ah )uh (~ + h f (~, a* ) ) - h e(9, a* ) = 0

(1.32)

for some a* = a*(9) E A. On the other hand, since x,

, u(x) -

[Uh(9) + -lx - ~-ff-?L2 + 5(~,(x) + ~,(9))j]

has a maximum at 5, from (HJB) we obtain

Au(5) < f ( 2 , a*). ( x - y + 5 D # ( 5 ) ) + 2(2, a*).

(1.33)

E

The inequality ~(5, Y) -> gz(5, ~ + h f(~, a*)) gives

Uh(~ + h f ( ~ , a *)) >_ uh(y) +

}5 -- ~I 2 - ]5 - 9 - h f(9, a*)] 2 + 5X1, 26

with X1 = #(~) - P(Y + h f(~, a*)) . Consequently,

uh(~ + h f ( ~ , a *)) > Uh(~) + 2h f(~,a*) . (5 - 9 ) - h21f(Y,a*)[ 2 --k 5X1 9 26 -

Multiply the above inequality by 1 - Ah and substitute the result into (1.32) to obtain Uh(9) _> (1 -- ;~h) [Uh(9) +

2h f(~,a*) . ( 5 - Y) - h2[f(Y,a*)l 2 + 5X1]] + h g(9, a* ) . 26

1. S E M I D I S C R E T E A P P R O X I M A T I O N

371

Hence, dividing by h, -AUh(~) < -(1 - Ah)[f(~,a*). x-

h

y

X 1

a*)l 2 + 5 --~-]j - e(~, a * ) . 2s [f(~,

E

Now, add this to (1.33) to get, using (1.3), (1.34) where X2 = ( f ( ~ , a * ) - f ( ~ , a * ) ) . - -x - y ,

X a = Ah f ( ~ , a * ) . x -

E

h X 4 ----~~

Y E

X5 = 5 f ( 2 , a*) . D # ( 2 ) + ~ 5 X 1

If(Y, a*)l 2,

.

Let us proceed then to estimate the various terms on the right-hand side of (1.34). We have

Ix21 < L I ~ - ~ l 2 E

h

I)(41 -< K Ml'

Ix3I < ~Mh I ~ - ~ l E

IXsl <_ 2 5 C M .

Therefore, using

(1.31)

(1.35)

u(X) - uh(~) < C(h ~/2 + h (~+1)/~ + h ~/2 + 5),

and choosing e = h (2-~)/2, we obtain

for a suitable constant C. Hence, for small h,

~(~) - ~h(~) <_ c ( h ~/2 + 5 ) . Then the inequality ~(~, ~) > ffJ(x, x) yields u(x) - ~h(x) - 2 5 # ( x ) <_ ~(~) - ~h(~) <_ C ( h ~/2 + 5),

for all x E R N. Let 5 --* 0 in the above to conclude that sup(u(x) - uh(x) ) < C h ~/2 .

To prove the inequality Uh(X) -- u(x) _< C h "~/2 it is enough to interchange the roles of u and u h in the auxiliary function 9 and proceed as before. 41 REMARK 1.6. From the theory of Chapter III we know that (1.27) holds with "~ = 1 if A > L, ~ = A / L if A < L and ~ / = 1 - 5 for any 5 > 0 if A = L. <1

372

VI. A P P R O X I M A T I O N A N D P E R T U R B A T I O N P R O B L E M S

1.3. I m p r o v i n g t h e r a t e o f c o n v e r g e n c e The estimate of the rate of convergence given by Theorem 1.5 can be improved under a uniform semiconcavity assumption on the solutions of (HJB)h. We have the following result. THEOREM 1.7. Let us assume (1.1)-(1.4), A > L, and (1.36)

If(x, a)l < M

f o r (x, a) e ]~N • A .

Assume also that the solution u of (HJB) is Lipsehitz continuous, and that the solution Uh of (HJB)h satisfies

(1.37)

uh(x + z) - 2Uh(X) + u h ( x -- z) <_ Clzl 2

for some C > 0 independent of h. such that

for all x. z E ]~g,

Then there exists C ~ > 0 independent of h

sup lu(x) - uh(x)l <_ C'h

f o r all h e ]0, 1//~[.

In the proof we need the following technical lemma which exploits a useful consequence of (1.37). LEMMA 1.8. Let Uh be the solution of (HJB)h. I f (1.37) holds, then f o r any a E A (1.38)

U h ( ~ q - h f ( 2 , a)) _ _ ~ < Uh(~) + h f ( 5 , a). x - y + Ch21f(~,a)12 + 6 h l f ( . ~ , a ) l , E

at any global maximum point (-2, ~) of ~(x,~)

:= uh(x) -- u(y)

I x - ~12

~(~(x) + ~(y)),

2r

where p C C I ( R N) satisfies (1.28).

PROOF. First use the inequality k~(~,ff) _> ~ ( ~ + x, ff), with arbitrary x E]R N, to obtain (1.39)

Uh(Z + x) -- Uh(5) -- x"

x-y z

Ixl 2 28

5 Dp(-~) . x < 0

for all x c R N. Now consider the function ~(x) := u ~ ( ~ +

x) - u ~( ~) - ( ~ \

y + E

~Ds(~))x /

Using assumption (1.37) and inequality (1.39) it is easy to check that r/(x + z) - 2r/(x) + ~(x - z) = uh(F + x + z) -- 2uh(7 + x) + uh(~ + x -- z) <_ C l z l L

Lr/(0) = 0,

limsup f f - ~ < 0 . ix,-~0

I I -

1. SEMIDISCRETE A P P R O X I M A T I O N

373

This implies C r](x) < -~ [xl 2

(1.40)

for all x e R N,

see Exercise 1.3. Let us now evaluate ~] at x = h f ( : , a) with a r b i t r a r y a G A to obtain using (1.40)

Since ID#I _< 1, from the above inequality the statement follows.

4

PROOF OF THEOREM 1.7. Here we denote as C any constant independent of h. Let us show first t h a t

(1.41)

sup(uh(x) - u ( x ) )

< Ch.

For this purpose, if ~ is as in the proof of Theorem 1.5 and ( : ~ : ) is a global maximum point for kt,, then from (HJB) it follows that (1.42)

Au(:)-f(:,a*).

(x-y

~D#(:)) -f.(:,a*)>_ O,

for some a* -- a*(:) e A. On the other hand, (HJB)h gives

Uh(~) -- (1 -- Ah)uh(2 + h f(2, a*)) - hg(2, a*) < O . By Lemma 1.8 and multiplication by (1 - .~h)/h, then

(1.43)

~Uh(:) < ( 1 - , X h ) f ( : , a * ) .

-

z-y E

-

+

c

(l-,~h)hlI(:a*)l

2

+ (1 - ah)5 II(:, a*)l + e ( : a*).

From now on the argument is exactly the same as in the proof of Theorem 1.5: subtracting (1.42) from (1.43) and estimating the various terms we obtain (1.44)

Uh(:) -

u(~)_
+l:-~l+h+

e] 9

Now the Lipschitz continuity of u yields the estimate

l : - :1 < CE; see (1.31) in the proof of Theorem 1.5. This, combined with (1.44) and the choice : = h, proves (1.41). The inequality

(1.45)

sup (~(~) - ~h(x)) < C h xE~N

374

vI. APPROXIMATIONAND PERTURBATIONPROBLEMS

is a consequence of the representation of u as the value function of the infinite horizon problem (1.46)

u(x) = inf J(x,c~) aEA

(see Chapter III), as well of the analogous one for Uh, (1.47)

inf Jh(x,c~) aCAh

Uh(X)=

(the proof of (1.47) can be found in the appendix, w from (1.46), (1.47) we obtain u(x) - Uh(X) < inf J ( x , a) -

Theorem 4.2). Indeed,

inf 3"h(x, c~) _< sup IJ(x, a) - & ( z , c~)l.

aE.Ah

a~

h

o~.A h

The estimate (1.45) then follows by Lemma 1.2. REMARK 1.9. A sufficient condition for the Lipschitz continuity of u is A > L (see Proposition III.2.1). On the other hand, the uniform one-sided estimate (1.37) holds if the further conditions A>2L and If(x + z,a) - 2f(x,a) + f(x-

z,a)l <_ C l z l 2,

g(x + z, a) - 2g(x, a) + g(x - z, a) <_ Q Izl2,

hold for some C, Q > 0 and all x, z E II~N, a E A. To check this it is enough to apply Proposition 4.5 in the appendix with f = hf,

~=l-Ah,

F = hg,

hE]0,1/~[.

Observe for this purpose that the constants Q, L, L1, and C in the estimate (4.10) there must be multiplied by h. Observe also that )~ > 2L implies ( 1 - ) ~ h ) . (1 + Lh) 2 < 1 for h E ]0, l/A[, thus condition (4.9) holds. Let us point out that under the above assumptions on A, f, g the result of Theorem 1.7 is optimal, as shown by the following example. Take N = 1, A a singleton and let f ( . , a), g ( . , a) be C ~ functions on R such that f ( x , a) = - x , g(x, a) = x for x E [0, 1]. It is easy to check that X

Uh(X) -- l + ~ -

X

Ah'

u(x) -- I + A '

forxE[0,1].

This gives sup

xE[0,1]

I~(x) - ~h(x)l

=

~h (1 + A)2 _ Ah(1 + ,~)

and, consequently, sup xe[0,1] diverges as h -+ 0 +, for any a > 0.

I~(x) - ~h(X)l hl+a

<1

1. SEMIDISCRETE APPROXIMATION

375

Exercises 1.1. Prove that Theorem 1.5 implies the following: sup lu - Uhl <_ C h 1/2

if )~ > L,

sup lu - Uhl <_ Ch'X/2L

if A < L .

1.2. Check that

~'-Alxl3 + Blxl 2 + Clxl

for Ix] < e, for Ixl ~ e,

p(x) = [ l o g Ix[

satisfies conditions (1.28), (2.13), and (3.7) for a suitable choice of positive constants A, B, C. 1.3. Assume that r/: R N ~ R satisfies, for some C > 0, ~(x + z) - 2r/(x) + , ( x - z) _< C }zl1+" for all x, z 9 ][~Y and some ~- 9 ]0, 1], as well as 7/(0) = 0,

lira sup ~ Ixl-0 I x l

< 0

B

Prove that ( x )

r/(x)

C 2 l+r2r--2-r(k-1)

for any x • O, k = O, 1 , . . . . Deduce from this the inequality r](X) < C IXlI+r ' 2(2 r - 1)

VX 9 R N .

1.4. Assume that Uh satisfies Uh(X + z) -- 2Uh(X) + Uh(X -- z) < C [z]1§

r 9 10, 1]

for all x, z 9 N N. Use Exercise 1.3 to prove that the estimate in Theorem 1.7 holds in the form sup [u -- Uh[ <_ C h r 9 1.5. Prove the analogue of Theorem 1.1 for solutions Uh of sup{(1 + ,Xh)uh(X) -- Uh(X + h f ( x , a)) - h e ( x , aEA

a)} = 0 .

376

VI. A P P R O X I M A T I O N AND P E R T U R B A T I O N PROBLEMS

2. R e g u l a r p e r t u r b a t i o n s In this section we address the following question: how much is the value function of an optimal control problem modified by perturbations of the various data not affecting their regularity or their structural properties? The approach adopted here is to look at the (HJB) equation satisfied by the value function and prove general convergence results for a class of such equations whose Hamiltonians satisfy the same structural conditions. We present a typical result in this direction, which applies to the infinite horizon problem (see Theorem 2.1) and a variant of it (Theorem 2.4) motivated by a regularization question in minimal time theory. Let us consider then the equations X C ~N,

uo(x) + sup { - f o ( x , a). Duo(x) - to(x, a) } = O,

(HJB)0

aCAo

and X E ]~ N.

ua(x) + sup { - f h ( x , a). Dub(x) -- fh(x, a) } = O,

(HJB)h

aEAh

We assume, for h 9 [0, 1], (2.1)

fh : ~N X A ~ ]~g

(2.2)

I.fh(x,a) -- fh(y,a)[ <_ L Ix - Yl, [th(X, a) -- th(y, a) I <_ M Ix - Yl, leh(x,a)l < M,

(2.3)

(2.4) for x, y

E

th : ]~Y • A ~ ]I~ continuous,

]I~N, a E A, with constants L, M independent of h c [0, 1] and

Ah C_ A,

(2.5)

A compact.

Since we are interested in the rate of convergence of Uh to u0 as h --+ 0 +, the perturbed data Ah, fh, th are assumed to be of order 1 in h. More precisely, (2.6)

sup{Ifh(x,a)--fo(x,a)l+lth(x,a)--to(x,a)]

(2.7)

dH(Ah, Ao) < Ch,

I~N

}
for all a E A,

for some C > 0. The notation dH stands for the Hausdorff distance between the sets Ah and A0, that is

dH(Ah, A0) = max~'l max d(a, Ao), maxd(a, Ah) }, aCAh

aEAo

where d is the usual Euclidean distance. Let us recall that the uniform convergence of u h to u0 has been proved in III.2.2. In the next result, the rate of convergence is established under slightly more restrictive assumptions.

2. R E G U L A R P E R T U R B A T I O N S

377

THEOREM 2.1. Let us assume (2.1)-(2.7) and (2.8)

Ifh(x,a)-- fh(x,a')l+lth(x,a)--gh(x,a')l

f o r all a,a' E A, h E [0, 1]. /fu0 E C~

<_nla-a']

N) f o r some 7 e ]0, 1], then

sup I~0(x) - ~ h ( x ) l < C m

for h E [0, 1],

]1~N

f o r some constant C > O.

PROOF. Let us set for h E [0, 1] H h ( x , p ) = sup { - - f h ( x , a ) " p - - t h ( x , a ) } . aEAh

The first step is to show that, for any h E [0, 1], (2.9)

I H h ( x , p ) - Ho(x,p)l < Ch(Ipl + 1)

for all (x, p)

E

]I~N X R N .

For fixed ( x , p ) let g E A0 be such that Ho( x, p) =

- fo

( x,-5) . p - Q ( x, -g) .

By (2.7) there exists a E Ah such that

I~ -a I _< Ch;

(2.10) by definition of Hh, then

g h ( x , p ) >_ --fh(X,~d) . p -- th(X,'~),

so that Ho(x,p)

- Hh(z,p)

< (h(x,~)

- S 0 ( x , ~ ) ) . P + ( e h ( X , ~ ) -- t 0 ( z , ~ ) ) 9

It is now easy to show, using (2.10) and assumptions (2.6) and (2.8) that H o ( x , p ) - H h ( x , p ) <_ Ch(Ip I + 1),

for some C. The other half of inequality (2.9) can be established in a similar way. Simple computations show that, under the assumptions made, the Hamiltonians Hh satisfy (2.11) (2.12)

[Hh(x,p) - Hh(y,p)[ <_ L l x - yl(1 + [p{) I g h ( x , p ) -- g h ( x , q ) ] <_ C[p - ql(1 + Ixl),

for some C, for all h E [0, 1] and x, y,p, q in IRN. Let us consider now the auxiliary function ,1,(5, y) = no(x) - ~h ( v )

Ix - vl 2

2r

,~(.(:~) + . ( y ) ) ,

378

VI. APPROXIMATION AND PERTURBATION PROBLEMS

with tL ~ CI(]~ N) such that

#_>O, #(0)=0, (2.13)

[ I D p ( x ) l _< 1,

lim p ( x ) = + o c ,

Ix[~+cr

IxllDp(x)l < 1

(see Exercise 1.2). The proof of Theorem 1.5 shows that (2.14)

1.2- Yl - C~1/(2-~),

where .2, y is a global maximum point for k9 on ~N x R g. Since u0 is a viscosity subsolution of (HJB)0 and Uh is a viscosity supersolution of (HJB)h, we have (2.15)

uo(.2) + Ho ( .2, x - y _ + 5 D # ( . 2 ) ) <0,_

(2.16)

Uh(y)+Hh(y,

x-y

5D#(y))

>0.

Therefore (2.17)

(

Uo(.2) _ Uh(Y) < Hh ~, x -- y

5 Dp(y

)) - Ho (-2,

+ 5 D#(.2

)) .

The right-hand side of the preceding inequality can be written as R1 + R2 with R t = Hh Y, x

(

s

R2 = Hh .2, x -

y

5 Dp

- Hh .2, - s

+5D#(.2

)) - .o( .2'

z

+ 5 D#(.2 +5D#(.2

,

)) .

Now (2.9), (2.11), and (2.12) provide the estimates IR~I < n ].2 - Yl (1 + .2 : y - 5 D # ( ~ ) ) + C5 ]D#(.2) + D p ( y ) l (1 + 1.21),

IR21 _< ch(1 + x - y +60.(.2) E

)

Using (2.13) and (2.14) we obtain, for some C > O,

IR, t <_ C(~ '/(2-~) + ~/(:-~) + 5z 11(:-~) + 6) (observe that ID#(~)[ $.2] < ]D#(~)[([.2 - ~[ + lYl)), IR2] < C ( h Jr- hE ( 3 - 1 ) / ( 2 - ~ ) -F 5 h ) .

Let us now choose ~ -- h 2-7 in the above and plug the previous estimates into (2.17) to get (2.18)

u0(.2) - ~.(~) _< c ( h ~ + h + 5h + 6) < c ( h ~ + 5h + 5),

for small h. Finally, the inequality LI,(.2,y) > q2(x, x) and (2.18) yield ~0(x) - uh(x) - 2 5 ~ ( ~ ) < ~o(~) - uh(~) < C(h~ + 5h + 5).

Let 5 ---+0 in the above to obtain uo(x) - uh(x) <_ C h ~

for all x

E ]~N .

The inequality uh(x) -- Uo(X) < C h ~ is established in a similar way.

4

2. REGULAR PERTURBATIONS

379

REMARK 2.2. A simple situation where the preceding result applies is as follows. Let A0, f0, g0 be given and define Ah = A0 • h/3(0, 1), fh(X, a, v) = f0(x, a) + by,

gh(X, a, v) = g0(x, a) + h Ivl2, for any a E A0, v E/3(0, 1) and h E [0, 1]. This amounts to a "fattening" of the original control set A0, giving rise to a new control system (fh, Ah) with better controllability properties (see Corollary 111.2.23). <] REMARK 2.3. If the control system is unperturbed (i.e., fh ----fo, Ah -- A0), then the estimate is sup luo(x) - Uh(X)l < Ch

~N

(see Exercise 2.2).

<~

An important observation is that the special structure of the Hamiltonians (namely, the convexity in p) does not play any role. Actually, Theorem 2.1 remains true for general Hamiltonians satisfying (2.9), (2.11), and (2.12). This fact has application to differential games problems (see Chapter VIII). Next we describe an application of Theorem 2.1 (rather, a slight variant of it) that provides the justification of a natural regularization procedure for the minimal time function. We confine ourselves to the case of linear control systems and target 7- = {0}. With the notations of w in Chapter IV, let

T(x) = inf tx(a),

x C ]~N,

~.A

where tx(a) is the entry time in the target 7- = {0} for the control system

(F, A)

fg(t) = Fy(t) + ~(t), l y ( o ) = ~.

Here, F is a given N • N matrix, a E .A = { a : [0, +co[ --~ A c_ ~ Y

c~ measurable }.

Let us consider, for h E ]0, 1], the perturbed control systems

(F, Ah)

S flb(t) = Fyh(t) + ah(t). [ y h ( 0 ) = x,

with C~h C .Ah = { a : [0, +co[ --~ Ah := A + h/3(0, 1), a measurable } and the corresponding minimM time function

Th(x) = inf t z ( a ) . aE.A h

380

VI. A P P R O X I M A T I O N A N D P E R T U R B A T I O N P R O B L E M S

If we assume (2.19)

A compact and convex,

0 E A,

then, for any h > 0, Ah is compact and strictly convex with smooth OAh and Ah D_ /~(0, h). Hence Th E C1(7~h \ {0}) by Theorem IV.I.10 and by Proposition IV.2.3 it is a classical solution of sup { - ( F x + a). DTh(x) } = 1 aEAh

in T~h \ {0}.

We also know that T E C(7~ \ {0}) and is a viscosity solution of s u p { - ( F x + a ) . DT(x) } = 1 aEA

in ~ \ {0},

provided the system (F, A) is small-time locally controllable on the origin, see Definition IV.I.1 and Proposition IV.2.3. The next result shows that if T is v-H61der continuous in a neighborhood of {0}, then T ~ Th + Ch ~, locally in 7"4, for small h > 0. This fact has some useful consequence for the synthesis of time-optimal control for the system (F, A) (see Remark 2.5). THEOREM 2.4. Let us assume A compact and (2.20)

321f > 0, ~t e ]0, 11 such that T(x) <_ ~tl IxI", for all x in a neighborhood of {0} .

Then for every compact K C 7~ there exists C = C ( K ) such that s u p gT(x)

-

Th(z)l <

Ch~

.

K

PROOF. Since A C A h we have Th < T, so that it is enough to prove sup(T(x) - Th(x)) <_ Ch ~ .

(2.21)

K

Let us define for x E

]I~N n e w

unknown functions

Uh(X ) = X(Th(x)),

u ( z ) -.~ x ( T ( z ) )

via the Kru~kov transformation ( 1 - e-~r

r < +co,

tX

r = +~

with A > 0 to be chosen later. By Propositions IV.1.2 and IV.1.6 Uh, u belong to BC(]~ N) and satisfy, respectively, (H3B)h (HJB)

AUh(X) + sup { - ( F x + a). Dub(x) } = 1 aEAh )~u(x) + sup{ - ( F x + a). Du(x) } = 1 acA

in ~N \ {0}, in ]~N

\

{0},

2. REGULAR PERTURBATIONS

381

in the viscosity sense. Let us observe t h a t in the present case (i.e., Ah = A + h/~(0, 1), f h ( x , a ) = F x + a , for any a E A + / ~ ( 0 , 1), gh(x,a) -- 1) the assumptions of Theorem 2.1 are trivially satisfied. Now, from Propositions IV.1.2 and IV.1.6 and Remark IV.1.7, it follows that (F, A) satisfies (STLC) and T E C~ for all compact K C 7~. Then it is easy to prove that u = x ( T ) E C~ : T ( x ) < 2}). We claim that this implies u E C~ N) if A > "yL, where L is the Euclidean norm of the matrix F . To prove the claim we consider the set ~t := {x E ~ : T ( x ) > 1} and denote with w and J , respectively, the modulus of continuity of u in ]I~y and in f~. Then

(2.22)

.~(~) < ~'(~) + c r y ,

~ > 0.

We fix x, z E f~ such t h a t u(z) <_ u(x), and ~ > 0. By the Dynamic Programming Principle for the problem of minimizing J ( x , c~) := fo ~(a) e -As ds, there exists c~ E .4 such t h a t

~(~) _>

e -A~ d~ + ~-~u(y~(1, ~)) - e,

~(~) <_

e-A~ d~ + ~-~(y~(1, ~)).

Then

~(x) - Kz) < e - ~ ( b ~ ( i , ~) - yx(1, ~)1) + c and by s t a n d a r d estimates on the trajectories of the system we get

By (2.22) and the arbitrariness of z we obtain (2.23)

w(r) << e - A w ( e L r ) + e r r .

Given 0 < r < 1, we fix n E N such t h a t e -(n+l)L ~ r < e -nL . We use (2.23) n + 1 times and compute n

.4r) < .~(e -'~L) <_ e-(n+l)A.~(e L) + C ~ " e -~L~+(j-n)A 9 j=l

Since n

e-JL'~+(J-n)A ~- e-n~ E 6~('k-L~/) j=l j=l = (e - L ~ ( n + i ) + ~

- e (1-€

(e ~ - L ~ -

i)-i,

382

VI. APPROXIMATION AND PERTURBATION PROBLEMS

by the choice A > "~L we obtain, for some constant C',

~ ( r ) ~ Vie -(n§

< Clr~,

which proves the claim. Let us consider now the auxiliary function @ employed in the proof of Theorem 2.1. If sup~• ~ is attained at some (~,~) E (~Y \ {0}) x (R g \ {0}), then u(x) - Uh(X) <_ C'h ~

for all x E ]~Y

for some C ' > 0. This follows by the same argument as in the proof of Theorem 2.1. If, on the contrary, 9 -- 0, we get from the inequality @(x, x) < @(~,~), u(x) - uh(x) - 2~p(x) _< 0

for all x E ]~N .

Hence, letting 6 ~ 0 +, u(x) - Uh(X) < 0

for all x E ~ N .

Since Ah D_ A, we have Th <_ T and therefore Uh <_ u because X is increasing. Thus we conclude that (2.24)

0 < sup(u(x) - Uh(X)) < C'h ~ . Ii~N

For any x E T~ we have Th(x) <_ T ( x ) < +~c. The mean value p r o p e r t y for X, when applied with r = T ( x ) , s = Th(x) gives u(x) -- uh(x) = e - ~ r

- Th(x) )

for some ~ = ~(x) E (Th(x), T ( x ) ) . Consequently, (2.24) and the above yield T ( x ) - Th(x) <_ e~T(x)(u(x) -- uh(x)) <_ C'e~T(x)h "~ 9 Inequality (2.21) now follows with C = C ' e "ksupK T.

9

REMARK 2.5. Since Th E C 1 when (2.19) holds, the classical synthesis procedure applies to equation (HJB)h. This allows us to construct some a~ E Ah such that Th(x) = tx(a~). From Theorem 2.4 it follows that T ( x ) < t~(a*h) + C h ~. Note also that T is a bilateral solution of the HJB equation (see Definition III.2.27), because it is a uniform limit of classical solutions. <~ REMARK 2.6. Sufficient conditions for the validity of (2.20) are provided by the results of Chapter IV (see, for example, Theorem IV.1.9). <~

3. STOCHASTIC CONTROL WITH SMALLNOISE

383

Exercises 2.1. For A = [ - 1 , 1]N, set f o ( x , a ) = a ( x ) a , a E A, where a ( x ) = (aij(x)) is a semidefinite positive N • N matrix for each x E ~Y. Set, for h E [0, 1], fh(X, a) = (r

+ hi)a,

gh(x,a)--l.

Assume t h a t for some L > 0 [aij(x) - aij(Y)l <_ L Ix - y[

V x , y E ]~N .

Show t h a t the assumptions of Theorem 2.1 are satisfied. Discuss the STC conditions of IV.1 for the dynamics fh. 2.2. In the setting of Theorem 2.1 take fh = fo, Ah -- Ao. Check t h a t IHn(x,p) - Ho(x.p)[ <_ C h .

Show that, in this case, snp lu0(x) - uh(x)l < Ch. ~N

[Hint: Observe t h a t the term ]R21 in the proof is estimated by Ch; choose then E = h(2-~)/%] 2.3. Consider continuous functions Hh : f~ • R N --* ~ , h E [0, 1], satisfying conditions (2.9), (2.11), and (2.12) for any x , y E ~, a bounded set in R y . Let uh,uo be, respectively, viscosity solutions of Uh + Hh(x, D u h ) = 0

in ~, h > 0,

uo -r Ho ( x, Duo) = 0

in f~,

with Uh = Uo = 0 on 0f~. Assume that u E C~ ]Uh(X) -- uh(Y)i <_ C l x - - y] ~,

for some -y E ]0, 1] and that for x E f2, y E Oft .

Follow the lines of the proof of Theorem 2.1 to show t h a t sup lUh -- u0l _< CIh "y, for some constant C ~ independent of h. 3. S t o c h a s t i c

control with small noise and vanishing

viscosity

Here we return to the topic outlined in Chapter I, w In the spirit of the present chapter the aim is to prove the convergence and estimates for the vanishing viscosity approximation

(VHJB)E

--eAuE(x) + Au~(x) + sup{ - f ( x , a ) . Du~(x) - f ( x , a) } = 0 aEA

384

VI. APPROXIMATION AND PERTURBATION PROBLEMS

of equation (HJB)

Au(x) + sup{ - f ( x , a) . D u ( x ) - e(x, a) } = O . aEA

Despite the different nature of the perturbed equation, the results in this section exhibit a strong analogy with those of w For example, a common feature is the loss of one half in the rate of convergence (see Theorems 1.5 and 3.2), due to the singular character shared by the perturbed equations (HJB)h and (VHJB)e. We consider here equations (HJB) and (VHJB)~ in ]I~.g and assume that the Hamiltonian H ( x , p ) = sup{ - f ( x , a ) . p -

g(x, a) }

aEA

satisfies (3.1) (3.2)

IH(x,p) - H(y,p)I <_ L Ix - y[(1 + IPl), ]H(x,p) - H(x,q)l <_ M l p - q l ,

IH(x,0)I _< M,

(3.3)

for any x,y,p,q. These conditions are fulfilled, for example, if f and g satisfy (1.1), (1.2), (1.3), and (1.26) in w It is well known from classical results on quasilinear elliptic equations (see, for example, [L82, GT83, Kry87]) that for any z > 0 (VHJB)~ has a unique bounded solution ue e C2(~N). Moreover, standard arguments in stochastic control theory show that

(3.4)

u~(x)

=

infEx

e(~(t), ~(t)) e -at at,

where Y~ is the solution of the Ito's stochastic differential equation dy(t) = f(y(t), c~(t)) dt + v ~ d w ( t ) , y(o)

= x.

Here w is an N-dimensional standard Brownian motion, Ez denotes the expectation, and the infimum in (3.4) is taken on the class of progressively measurable processes with values in A (see [FR75, Kry80, FS93]). On the other hand, by the theory of Chapter III, the value function of the infinite horizon problem, namely, (3.5)

u(x) = inf f+~r t(y~:(t), a(t)) e -;~t dt aEA Jo

is the unique viscosity solution of (HJB) in BUC(~N). THEOREM 3.1. Let us assume (3.1)-(3.3) and A > O. Then sup ludx) - u(x)l ~

for every compact ]C C ~ g .

0

a s z - ~ 0 +,

3. S T O C H A S T I C C O N T R O L W I T H SMALL NOISE

385

PROOF. It is straightforward to check t h a t the constants + M / A with M as in (3.3) are, respectively, a sub- and a supersolution of (VHJB)s, for any E > 0. Through the maximum principle for elliptic equations we get then the uniform bound sup luE(x)l _< M / A . Then set l i m i n f us(y), (~,s)~(x,0§

u_(x)=

~(x)=

limsup

X E ]~ N .

us(y),

Functions u, ~ are, respectively, lower and upper semicontinuous. The claim is that u is a supersolution and ~ is a subsolution of (HJB) according to the definition in Chapter V, w To prove the claim, let p E C2(R N) and x be a strict minimum for _u - ~ i n / ~ = / ~ ( x , r). By Lemma V.1.6 there exist xn ~ x and ~n ~ 0 + such that

(~o - ~ ) ( x ~ ) = m~n(~so - 4 ) ,

~o (z~) ~

~(x).

From the minimum property above it follows that -/\us~

< -A~(x~);

hence (VHJB)~ gives

- ~ J , ~ ( x ~ ) + ~ ~so(x~) + g ( ~ , D ~ ( ~ ) ) > 0. Now let n ---* ~ to obtain A_u(x) + H ( x , D p ( x ) ) _> 0 . The proof that ~ is a viscosity subsolution of (HJB) is completely similar. By the comparison principle, Theorem V.1.3, the conclusion follows. 9 An extra regularity assumption on u allows us to get the uniform convergence over all ~ N and to estimate the rate of convergence. THEOREM 3.2. Let us assume (3.1)-(3.3), A > 0 and (3.6)

u E C~

f o r some ? E ]0, 1] .

Then, f o r some C > O,

sup [ue (x) - u(x)[ _< CE ~/2 ~N

as ~ --* 0 + .

386

VI. A P P R O X I M A T I O N AND PERTURBATION PROBLEMS

PROOF. Consider ~'(x, y) = u d x ) -- u(y)

Ix -- y12

with a , 5 > 0 and # E C~(X N) such t h a t ~/~ > 0,

(3.7)

p(0) = 0,

limlxl~+~ r

[ ID,(x)I + I/'~(x)l < 1,

= +cr

Ixl ID,(x)l <_ 1.

If ("2, ~) is a global m a x i m u m for 9 , then D~tI,("2,y) = Du~("2)

(3.8)

x-

y

5Dlz(Y) = 0,

O~

(3.9)

Axk0("2,~ ) = Aur

- N _ 5 A/z(-2) < O.

Hence (VHJB)~ yields Aue("2) + H ( "2, x - ay

(3.10)

+ S D p ( " 2 ) ) <_ N za- + e S A p ( " 2 ) .

On the other hand, ~ minimizes - 9 ( " 2 , y). F r o m (HJB) it follows t h a t

A u(y) + H (y, -x - - y - 5D#(~))

_> O.

OL

Therefore A(u~("2) - u(~))

(

< H ~, x - Y - 5 D # ( ~

)) - H (x-y 2,

+ 5 D#("2

)) + N

+r

Using assumptions (3.1) and (3.2) exactly as in the proof of T h e o r e m 2.1 we obtain, for some C > 0,

(3.11)

u,("2) - u(~) < c(~'/(2-~>

+

~/(2-~>

+

a~'/(~-~)

+

a)

+

~(N ~- + ~)

since 1"2 - Yl < Cal/(2-'~). The inequality 9("2, ~) > @(x, x) and (3.11) then give u~(x) - u(x) - 2 5 # ( x ) < C ( a U(2-7) + a ~/(2-~) + 5a U(2-'~) + 5 + z + r for any x E I~N. Let us choose now a = E(2-~)/2 in the preceding to o b t a i n

u~(x) - u(x) - 2 5 # ( x ) <_ C(r 1/2 + E"~/~ + &1/2 + 5 + r

+ ~)

for all x E ~ N . If we send 5 --~ 0 + in the above we conclude t h a t sup(uE(x) - u(x) ) < CE "r/2 RN for small ~ > O. The inequality supRN (u(x) -- u d x ) ) < Cz ~/~ is proved by exchanging the roles of u and u~ in 9.

3. STOCHASTIC CONTROL WITH SMALL NOISE

387"

REMARK 3.3. Let us point out that the particular structure of H (namely, its convexity in p) is irrelevant in the proofs of Theorems 3.1 and 3.2. Then the results hold for general H satisfying (3.1), (3.2), and (3.3). < REMARK 3.4. Standard verification arguments in stochastic control show that any feedback map a~ such that a~(x) 9 arg max{ - f ( x , a). Due(x) - ~(x, a) } aEA

is optimal for problem (3.4). From this point of view one can see Theorems 3.1 and 3.2 as the basis for the justification of those numerical methods for deterministic control problems relying on the viscosity approximation. <~ The final result of this section is analogous to Theorem 1.7. We assume that the viscosity solution u of (HJB) is Lipschitz continuous and (3.12)

3C > 0 such that Au~(x) < C for all x E ]~Y and z > 0 .

Condition (3.12) amounts to the uniform semiconcavity of uz (compare with (1.37)). The validity of (3.12) can be ensured under further assumptions on H. This can be proved by elliptic PDEs techniques (see, e.g., w167 and 6.2 in [L82]) which are outside the scope of this book. Exercise 3.1 gives just a flavor of this type of result. THEOREM 3.5. Let us assume (3.1)-(3.3), (3.12) and (3.13)

U 9 C0'l(]~ N) .

Then sup lu~ - u(x)l < Cs, ~N .for some C > O.

PROOF. Proceed as in the proof of Theorem 3.2 and observe that because of (3.8) and (3.12), from (VHJB)~ it follows that

(3.14)

~ uE(2) + g ( 5 , x - y + 5 D p ( 2 ) ) = z Au~(2) _< C e . C~

As a consequence of this and (3.13), inequality (3.11) now becomes uE(~) - u(~) < c ( ~ + ~ + 5~ + 5) + ~ c ~ .

At this point choose (~ = ~ and conclude as in the proof of Theorem 3.2.

9

REMARK 3.6. For the Dirichlet boundary value problem associated with (HJB), some results similar to Theorem 3.1 on the convergence of the vanishing viscosity approximation have been proved in Chapter V, see Corollaries V.4.27 and V.4.32. <3

388

VI. APPROXIMATIONAND PERTURBATION PROBLEMS Exercises

3.1. Assume • > 0, F C C2(]~) convex, g E C2(R) semiconcave. Suppose that the equations

- : u " + F(u') + ~u = t

in ]R

have a solution u = uE E Ca(R) • B(A) for each : > 0. Differentiate the above to prove that w = u" satisfies for some C > 0

-Ew" + F'(u')w' + Aw <_ C

(i)

in ~ .

Deduce from (i) that supA u e" _
Dynamic

Time Systems

This appendix contains a few results on the discrete time infinite horizon problem

(4.1)

inf J(x,~),

J(~,~) := ~ t(yn,a~)9 n .

~EA

n=O

Here, Yn is given by the recursion

lyn+:=yn+f(y~,an),

(S)

n = 0,1,2,...

l, Y0 = x E ~ N where an E A, a topological space, and f : R N x A -* R N is a given mapping. We denote by A the set of all sequences a -- {an} _C A and by y,~(x, c~) the corresponding trajectory of (S). Problem (4.1) can be seen as a discrete time version of the infinite horizon problem (see Chapter III). Some relevant properties of the value function are proved and the Bellman functional equation characterizing the value is established. A main feature of the discrete time case is t h a t the synthesis of optimal feedback controls can be easily performed starting from the discrete Bellman equation. Let us recall that we applied the results presented here in w of this chapter. More precisely, this was done in the case

fh = h f,

gh = hg,

13h = 1 -- Ah

where h E ]0, l/A[, A > 0. We assume that g and fl satisfy (A:)

~ E ]0,1[;

9M>_O:]~(x,a)I<M,

VxERN,VaEA.

Consider now the value function

v(x) := inf J(x, 4 ) . oEA

The next result is the statement of the Dynamic Programming optimality Principle.

4. APPENDIX: DYNAMIC PROGRAM*.IING FOR DISCRETE TI~.IE SYSTEhIS

389

PROPOSITION 4.1. Assume (A1). Then v satisfies

v(x) = inf {/3v(x + f(x, a)) + g(x, a) },

x 9 ~N.

aEA

PROOF. Fix any x 9 I~N and any a -- { a 0 , a l , . . . } 9 .4. It is easy to check that

J(x, a) = s

ao) +/3J(x + f(x, ao),-~)

with ~ = {al, a 2 , . . . }. The above identity follows from the fact t h a t

Yn+l (x, a) -- Yn(Yl (x, a),-~) = yn(x + f(x, an), "~) . By definition of v

J(x, c~) >_g(x, ao) +/3v(x + f(x, ao) ) . Hence

v(x) > inf { g(x, a) + :3v(x + f(x, a)) } . aEA

To prove the reverse inequality, set 2 = x + f(x, a) with arbitrary fixed a E A. By r definition of v(z), for any e > 0 there exists a * = { a,~ } E .4 such that

(4.2)

v(z) > J(z, d ) -

~.

Observe now t h a t

J(x, d~) -- g(x, a) +/3J(x + f ( x , a), a) where ~ = {a, a0, a t , . . . }, a E A. This identity with a -- a E and (4.2) gives

v(x) <_ g(x, a) + 13v(x + f(x, a)) +/3e. Since a E A and e > 0 were arbitrary,

v(x) < i n f { g ( x , a ) + •v(x + f ( x , a ) ) }.

9

aEA

Proposition 4.1 asserts t h a t v satisfies the Dynamic Programming equation, or Bellman functional equation (DBE)

u(x) = i n f { / ~ u ( x + f ( x , a ) ) + ~(x,a) },

x 9 ]~N,

aEA

a discrete version of the Hamilton-Jacobi-Bellman equation for the infinite horizon problem (see C h a p t e r I, Chapter III w Chapter VI w Denoting by G the nonlinear operator on the right-hand side of (DBE), any function u such t h a t

~(x) < (cu)(x)

Vx 9 ~:~

is called a subsolution of (DBE). Similarly, u is a supersolution provided

lz(x) > (Gu)(x)

Vx 9 ~N .

Under the assumptions made, v is in fact characterized as the unique solution of (DBE). Moreover, a comparison principle between sub- and supersolutions of (DBE) holds.

390

VI. APPROXIMATION AND PERTURBATION PROBLEMS

THEOREM 4.2. Assume (A1). Then v E B(]~ N) and v is the unique solution of (DBE) in B(]~N). Also, / f u r , u 2 E B(R N) are, respectively, a sub- and a supersolution of (DBE) then

ut <_u2

(4.3)

in

]~N .

PROOF. Observe that from (A1) and the definition of v

I~(x)I_
V x e ~ ~,

n:O

so t h a t v E B(RN). By Proposition 4.1 v is both a sub- and a supersolution of (DBE); from the comparison principle (4.3) it follows that v is the unique solution of (DBE) in B(RN). In order to prove (4.3), observe t h a t by definition of supersolution for any z > 0 and x E ~ g there exists a ~ = aE(x) E A such that u2(x) > 13u2(x + f(x, aS)) + [(x,a E) - E. On the other hand ~tl(X ) <~ ~3~tl(X -Jr-f(x,

ae)) --~~(x, ar

since Ul is a subsolution. Hence

~l(x) - ~2(x) < Zul(x + f(x,a~)) - Zu2(x +/(z,a~)) + ~,

V z e ~ N,

so that (4.4)

sup(ul -- u2) _< ~ s u p ( u l -- u2) + ~. ~N ~.~

Since/3 E ]0, 1[ this implies (4.3).

4,

REMARK 4.3. For /3 = 1 uniqueness does not hold. Actually, if u E B(]~ N) is a solution of (DBE) the same is true for w(x) = u(x) + c for any constant c. <~ In order to proceed in our program leading to the description of the synthesis procedure (see Theorem 4.6) we make some further assumptions on the data, namely (A2)

3L>_O:lf(x,a)-f(z,a)l<_L[x-zl, Ig(x,a)-[(z,a)}~w(Jx-zl),

(A3)

Vx, z E R N , V a E A , Vz, z e ] ~ Y , V a e A ,

for some modulus w. Observe that (A2) implies (4.5)

ly,~(x,a)- y~(z,a)l <_ (l + L)'~lz- zl,

Then we have

Y a e A, n=O, 1,2 ....

4. APPENDIX: DYNAMIC PROGRAMMING FOR DISCRETE TIME SYSTEMS

391

PROPOSITION 4.4. Assume (A1), (A2), and (A3). Then v E BUC(]~N). Moreover, /f (A3) holds with co(r) = Llr and 0 < / 3 < 1/(1 + L), then (4.6)

Iv(x) - v(z)] < 1 - / 3 (L1 1 + L) Ix - zI,

'7'x. z E ]~N

9

"

PROOF. We have seen already t h a t ~I Iv(x)l <- 1 - 13

V x E R N,

i.e., v E B(NN). To prove the uniform continuity, observe t h a t for fixed z in IRN a n d a r b i t r a r y E > 0 there is c~e = { a cn } E .4 such t h a t

v(z) >_ J(z, as)

3 "

Hence

v(x) - v ( z ) <_ J ( x , d ) - J ( z , c d ) + -~,

V x . z in R N . .

Consequently, by (A1),

v(x) - v(z) < E

[g(yn(x, as), a~) - g(y,(z, ac), a~) 113n +

2M13" +

n=0

n=k+l

for a n y k C N. Let us choose now k such t h a t ~-~n~Z=k+l2M13 n < z / 3 a n d use (As) a n d (4.5) to get k

v(x) -

. ( z ) < ~12 co((1 + L)~I 9 - zl)Z ~ + 2~ --

3

n=O

'

Since co(r) --* 0 as r -~ 0, it is easy to check t h a t this s u m can be made smaller t h a n z / 3 for Ix - z I small enough. Therefore, v is uniformly continuous. Let us proceed now to the proof of (4.6). A r g u i n g as before, we o b t a i n easily k

v(x) -- v(z) < E LI(1 + L)n/3nix - z I + 2__~ 3 n=O for large enough k. Since 0 < 13 < 1/(1 + L), we can let k ~ + o c to o b t a i n v(x)--v(z) <

L1 1 -- 13(1 + L)

Is - z

I+-

2a 3

Since ~ was a r b i t r a r y this proves (4.6). T h e next proposition states sufficient conditions for the semiconcavity of v, (see C h a p t e r II, w C h a p t e r III, a n d w in this chapter).

392

VI. APPROXIMATION AND PERTURBATION PROBLEMS

PROPOSITION 4.5. A s s u m e (A1), (A~), and (A3) with w(r) = L l r . A s s u m e also (4.7)

SC>0:

(4.8)

3Q > O:

If(x+z,a)-2f(x,a)+f(x-z,a)[<_Clzl

2,

$(x + z,a) - 2~(x,a) + e ( x - z,a) _< Qlzl 2,

for all x , z E 1 ~ , a E A, and

(4.9)

O
1 -

(1 + L) 2 " -

Then 1

(4.10)

Q+

v(x+z)-2v(x)+v(x-z)<

1-fl(l+L)

2 Iz12

for all x, z C ~ N .

PROOF. A simple consequence of the definition of v is that v ( x + z) - 2v(x) + v ( x -

z) <_ sup[g(x + z , a ) - 2 J ( x , a ) + J ( x -

z,a)].

aEA

Observe that for fixed a = {a,} E A we have J ( x + z, a) - 2 J ( x , a) + J ( x - z, a) = E

Anti",

n~O

where x a An := f(y~ + (,Ynx+z - Yn), n) - 2g(y~, a , ) + ~(y,x - (y~+Z _ y ,x) , a n) q_ f(yX-Z an) - e(y x - (yXn+Z -- yX), a n ) .

In the above we used the notation y~ = Yn(G a). From assumptions (4.8) and (A3) we obtain (4.11)

AnKQ

ly~+Z --2y x + y ~ - z l .

x+z --yXl2+L1 Yn

Let us use now (4.7) and (A2) to estimate the quantity x--z

~n := lY~+z--2Y~ + Yn

I.

By definition of y~ we have a.§

2~4-Z

< an + Lf(Y,, _


x +

,an) - 2 f ( y ~ , a n ) + f ( Y n x+z

2

X--Z x

,an)l (

(Yn - Y x ) , a , ) - f(Yn, an)+f(Y~n - , y n + If(y,~ -~, an) - f(y,~ - " ~yn~+z - ~ n )~, a n)l -

x+z

x

--Yn),an)[

Therefore, using also estimate (4.5), C~n+ 1 ~__ a n q- C [yX+Z _ yX[2 _~_ L o .

_< (1 + L)c~. + C(1 + L)2"lzl 2 .

4. APPENDIX: DYNAMIC P R O G R A M M I N G FOR DISCRETE T I M E SYSTEMS

393

By recursion this implies ~

< c

Izl2 2~-1 y]~ (1

+ L) k = C

iz[2 (1 + L) 2n - (1 + L) L

,

n = 1, 2 . . . .

k=l

The previous inequality, together with (4.11), yields

A~ < (Q + ~ - ~ ) ( I + L)2n]z[2,

n=l,2,...

Hence, using assumption (4.9),

v(x+z)-2v(x)+v(x-z)_<

AnZ~-<

Q+

1-•(I+L)

2

'

n=0

In the next result we state some sufficient conditions for the existence of an optimal feedback control for the infinite horizon problem and give the procedure for its construction from the equation (DBE). THEOREM 4.6. Assume (A1), (A2), and (A3) and (4.12)

a ~-~ ~(x, a)

(4.13) (4.14)

a ~ f(x, a) A is compact.

is lower semicontinuous for each fixed x, is continuous for each fixed x,

Then the control a* = { a~ } E A defined by

a*eargmin{e(y,,.)+Zv(y,

f(y,,.))}

(n=0,1,...),

A

where y~ = x and { y~ } is the trajectory of (S) corresponding to a *, is optimal f o r X.

PROOF. Under the assumptions made, v is the unique solution of (DBE) in BUC(]~ N) (see Theorem 4.2 and Proposition 4.4). By (4.12), (4.13) the function a,

, Zv(x + f(x,

a)) + ~(x, a)

is lower semicontinuous for each fixed x. Since A is compact, for every x E there exists a(x) E A such that

l~ N

Zv(x + f(x, a(x))) + g(x, a(x)) = inf { ~v(x + f(x, a)) + g(x, a) } . aEA

On the account of (DBE) this means that the set

F(x) = {a E A: v(z) = Zv(x + f(x,a)) + t ( x , a ) } is nonempty for all x E R g. For fixed x, set y~ = x and choose any a~ E F(x). Define then recursively the sequences { y~ } C ll~N and a* = { a~ } by setting y* =

* * a* Y,z-1 + f(Yn-1, ,-1),

* a N E F(y*)

394

VI. APPROXIMATION

for n = 1., 2.. . . .

AND

PERTURBATION

PROBLEMS

By definition of F, Y,~+I,* y,,* a s* are related by

Z%(u;,)

-

~n+lyt

*

n

,,

*

*

tun+lJ = Z e(~,. an),

,~ = o, 1 . . . . .

Therefore ~n[V[

*~

*

n=O

*

*

n

n=O

~'~(v(y~) -/%(Y~+1)) = v(x); hence

It is straightforward to check that ~ n=O

v(x) = ~ e(y;. a;)~ ~ = J(x, ~*) n:0

and the proof is complete. Exercises 4.1. Consider the value function V h ( X ) ~-

inf Yh(x,a), c~E.Ah

where Jh and Ah are defined in w Use Theorem 4.2, Propositions 4.4 and 4.5 to deduce that for h E ]0, l/A[ the following estimates hold: (a)

M ]Vh(X) I <_ - ~

(b)

tVh(X) -- Vh(Z)l <-- A L1 -- L

(c)

Vh(X + z) -- 2Vh(X) + Vh(X -- z) ~ L.hl + L1C iz[2 L(A - 2L)

if ,~ > 0;

Ix-z[

ifA>L; if A > 2L

for all x, z C It~N. 4.2. Take A = IRN, f ( x , a) = M x + a, g(x, a) -- Ix]2 + lal 2 where M is a N • N matrix. The DBE equation is

u(x) = inf {/~u((I+.hl)x + a) + [xl ~ + [a[~}. aE~ N

(i)

Show that u(x) := Q x . x is a solution provided that Q satisfies the matrix Riccati equation

Q = I + ~(I + M)t(I + ~Q)-IQ(I +.hi). (ii) Compute Q in the case N --- 1 and show that

u(x) = ((1 + .hi) 2 + 1 \

-

1/~ + V/((1 + ill) 2 + 1

-

1//~) 2 + 4 / ~

X2 /

T

5. BIBLIOGRAPHICAL NOTES

395

4.3. For any control a = {an} e ,4 and trajectory yn(x,w) of system (S) define ~min{n E {0, 1,2,... }: y,(x,a) E T}, n(x,c,) : : ( + o c ify,~(x,a) ~ 7- for all n, where 7- is a given subset of ]~N. Consider the minimum time function

N(x) := minn(x, c~) c~EA

and the reachable set := { x e

: N(x) <

}.

Show that if A is compact, f continuous, and T closed, then N satisfies

N(x) = 1 + m i n N ( x + f(x,a)),

VxET\T.

aEA

4.4. Consider the minimum time function N in the case 7- = B(0, 1), A =/~(0, R), f(x, a) = a. Show that the mapping x ~ a*(x) := - R x / H is an optimal feedback map and compute N(x). 5. B i b l i o g r a p h i c a l n o t e s The problem of convergence of semidiscrete approximations of Hamilton-JacobiBellman equations can be traced back to Bellman's own work [Be157, BelD62]. Our presentation in w basically follows [CD83] and [CDI84] (see also [F61, F64, Cu69, M79, CDM81] for previous convergence results in particular cases, and [CL84, CDF89], where connections with different approximations procedures are discussed). Note, however, that the proof of convergence in Theorem 1.1 uses ideas of [BF90a], where the weak limits technique of Barles-Perthame [BaP87, BaP88] was first applied to the discretization of HJB equations. The estimates in Theorem 1.5 were proved in [CDI84, Sou85a]; the role of semiconcavity to improve estimates as well as the convergence of suboptimal feedbacks to optimal relaxed controls are results from [CDI84]. We refer to the papers [Kr66b, CL84, Sou85a, Sou85b, BaSo91] for general consistency and convergence results. Section 1 also includes a simple but apparently new constructive proof of existence of optimal controls for the infinite horizon problem, Corollary 1.4. Semidiscrete approximation schemes of the type described in w have been applied to several optimal control problems; see, for example, the papers [BF90a, BF90b] for the minimum time, [LOT94, CFa96] for the case of state constraints, [Cam96] for T/~ control, [CFa95b] for impulse problems, and [BS91b, BFS94, Sor97b, BFS97] for some differential games. Higher order schemes have been proposed in [FaF90~ FaF94, GTi92]. Extensions to the case of discontinuous value functions can be found in [BFS94, BBF95]. Other related papers are [GR85, MeR86, Lo86, RVgl, Ta94, AL94, DMG95, LS95], and [RT92] for problems in

396

VI. APPROXIMATIONAND PERTURBATIONPROBLEMS

shape from shading, [LOS93] for some infinite dimensional case. Recently, approximation schemes based on the viability theory have been proposed in [CQS94a, CQS94b, CQS97]. Similar approximation schemes have been considered and their convergence proved in the stochastic case. Let us mention in this respect the books [Kus77, KD92, FS93] and the papers [Me89, CFa95a]. The material of w is from the papers [BSa91a, BSa92], where more general problems in optimal control and differential games are also considered. Theorem 2.4 refines a result in [BBi90]; see also the references therein. Related works are [Ja77, Pe79, Sa90]. For a different approach to well-posedness in optimal control see the recent book [DZ93] and, for perturbation results, [Ben88]. The vanishing viscosity approximation to Hamilton-Jacobi equations studied in w is very classical (see, e.g., [Kr64, Kr66a, F69]). Theorems 3.2 and 3.5 slightly improve some results in [L82] (see also IN91]). A more extensive treatment of this important topic, including in particular the connections with stochastic control, is outside the scope of the present book. The reader is referred to the book by Fleming and Soner [FS93] (see also [FR75, BL82]). The related topic of boundary conditions in the viscosity sense is treated in Chapter V. We refer to [FSo86] for asymptotic expansions, to [PS88] and [CLS89] for the vanishing viscosity, respectively, in the Neumann boundary value problem and in the Cauchy problem with infinite data, and to [LSV87, EM94, BM96] for approximation of Hamilton-Jacobi equations via nonlinear degenerate diffusion. The use of viscosity solutions methods in large deviations problems was initiated by Evans and Ishii in [EI85] and surveys on this topic can be found in the books [E90, FS93, Ba94]; other results on the exponential decay of solutions in singular perturbation problems for elliptic equations are in [Kam84, Kam86, Kam88, B87, BP90, BP91, BaP88, BaP90, Ei87, Ei90, Ei93, IK91a].

CHAPTER

VII

Asymptotic problems

In this chapter we consider several asymptotic problems in optimal control. Our approach is to pass to the limit as the relevant p a r a m e t e r goes to zero in the Hamilton-Jacobi-Bellman equation satisfied by the value function and characterize the limit value function as the viscosity solution of the limit equation. A useful tool in this approach is the weak limit technique described in w The main advantage provided by this technique in the context of the present chapter lies in the fact t h a t it requires only a priori bounds in the supremum norm. Throughout the chapter we assume without explicit mention that conditions (A0)-(A4) of Chapter II hold. For simplicity, we assume also that the control set A is compact and that (A4) is satisfied with w~(r) = Lr, even though most of the results remain valid in greater generality.

1. E r g o d i c

problems

In this section we investigate the limiting behavior of the value function v~ of the infinite horizon problem as the discount rate A --~ 0 +. In w we consider the case of state constraints and show that, under a controllability assumption, Av~ converges to some constant X0 t h a t can be evaluated as the infimum of a long run averaged cost criterion (see Theorem 1.1 and Proposition 1.3). Section 1.2 deals with the unconstrained case, with special attention to a particular case, namely, that of the optimal stopping time. In this case a dissipativity condition on the dynamics allows us to obtain the same results as in the constrained case (Theorem 1.6). Moreover, an estimate on the rate of convergence is established (Proposition 1.9).

1.1. V a n i s h i n g d i s c o u n t in t h e s t a t e c o n s t r a i n e d p r o b l e m We consider here the limiting behavior as A --~ 0 + of the value function v~ of the infinite horizon problem with state constraints discussed in C h a p t e r IV, w Let us

398

VII. ASYMPTOTIC PROBLEMS

recall for convenience that, for A > 0, v~ is defined by

v~(x) = inf

g(yx(S, a), a(s)) e -~s ds,

x E -~,

aEA~

where f~ is a bounded open set in 1t~N with smooth boundary and

Ax={aEA:yx(s,a)

e-~, Vs___0}.

Let us also recall that the following b o u n d a r y condition on the dynamics

inf f ( x , a ) . n ( x ) < O

(1.1)

VxEOfL

aEA

where n(x) is the exterior normal to f~ at x, and the standard assumptions on g, f , imply t h a t v~ E C ( ~ ) and it is the unique constrained viscosity solution of the Bellman equation (HJB)~

Av~(x) + s u p { - / ( x , a ) . OvA(x) - g(x, a)} = 0

x E

aEA

(i.e., v.~ is a viscosity subsolution in f~ and a viscosity supersolution in f~ of (HJB)~). As a preliminary remark, let us observe t h a t the Hamiltonian

F~(x,r,p) = Ar + s u p { - f ( x , a ) . p

- g(x,a)}

aEA

converges locally uniformly as A --* 0 + to F0(x, r,p) = s u p { - f ( x , a) .p - g(x, a ) } . aEA

Therefore, by Proposition II.2.2, one can pass to the limit in (HJB)~ if v~ is known to converge locally uniformly in N g as A ---* 0 +. However this cannot be true in general. Indeed, if g is a constant, then Iv~[ = Igl/,k --* +oc as A --~ 0 +. Let us make the controllability assumption (1.2)

3r > 0 such that B(0, l / r ) C_ ~-6f(x, A), for all x E ~ .

If (1.2) holds, then (1.3)

Ivy(x)

-

va(z)l < M ~ Ix - zl

vx, z 9 a,

where M = sup~u • Ig( x , a)l, as it follows easily from Proposition III.2.3. The uniform Lipschitz estimate (1.3) suggests the introduction of the function

w~(x) := v~(x)

-

v~(x0),

where x0 is arbitrarily fixed in ~ . It is easy to check that w~ satisfies (HJB)~

Awa(x) + s u p { - f ( x , a ) . D w a ( x ) - g(x, a)} + )~va(xo) = O,

x 9

aEA

in the constrained viscosity sense. The asymptotic behavior of (HJB)~ as )~ ---*0 + is described by the next result.

1. E R G O D I C PROBLEMS

399

THEOREM 1.1. Let us a s s u m e (1.1) and (1.2). Then there exists Xo E ~ such that (1.4)

lira Ave(x) = Xo,

uniformly in ft .

A~0 +

Moreo?yer

r

lira w)~,~= wo,

Tt ~ -- OC

uniformly in -~,

for some An --~ O, and wo is a constrained viscosity solution of (HJB)~

sup{ - f ( x , a) . Dwo - g(x, a) } + Xo = 0

in ft.

aEA

PROOF. From (1.3) it follows immediately that I ~ v ~ ( x ) - Av~(z)l < ~ M r l x

(1.5)

- zl,

Iw~(x)l < Mrlx - x0t, Iw~(x) - ~ ( z ) l < Mrl~ - zl,

for any x, z, x0 in it. Also, the uniform bound

SAv~(x)l < sup [~(x,a)l ~ x xA

is easily obtained either directly from the definition of vx or by comparison results in Chapter III. Therefore, by the Ascoli-Arzel~ Theorem, lira Anv~. ----+ ~(0 E C(it),

lira w~, ----* w0 E C(ft),

uniformly in it, for some sequence An --~ 0 as n ~ +~c. It is an immediate consequence of the first inequality in (1.5) that X0 is a constant, and consequently, Anw~,(x) ~ 0 uniformly in ~. Let us check now that (~(0, w0) satisfy (HJB)~. By Proposition II.2.2 we can pass to the limit in (HJB)~, as n ---* ~z to obtain sup{ - f ( x , a ) . Dwo - g(x, a) } + X0 = 0

in ft

aEA

in the viscosity sense. In order to check the supersolution condition on Oft, let x E Of2 be a local strict m i n i m u m for w0 - cy, p E C 1(~). Then by Lemma II.2.4 there exist local m i n i m u m points x~,, for w ~ - ~2 such that X)~ n ~

X,

W~n(X~rl)

"----:t W o ( X )

.

Since wA is a supersolution of (HJB)k on ~,

~ w ~ . ( ~ . ) + sup{ - y ( x ~ o , a). D ~ ( ~ . )

- e ( x ~ , a) } + A~v~. (x0) > 0

aEA

and the desired statement follows by letting n --* + ~

in the above.

400

VII. ASYMPTOTIC PROBLEMS

It remains to be proved t h a t X0 is uniquely defined by (HJB)~) and t h a t the whole family { Av,~ } converges to X0. For this purpose, let H(x, p) = SUPa~A{ --f(x, a). p -- g(x, a) } and consider the equation

H(x, Du) + a = 0

(1.6)

in ~ .

The claim is that there is a unique a E ~ for which (1.6) has a constrained viscosity solution u E C(~). Indeed, suppose that u and ~ are constrained viscosity solutions of (1.6) corresponding to a, 5 with G < a. Fix 5 > 0 such that - ~ + 5~(x) > - a + 5(u(x) + C),

Vx e

with C > sup~ I~(x) - u(x)l. Observe now that ~ is a constrained solution of

d~ + H(x, D~) = - ~ + 6~

in

while u + C satisfies

5(u + C) + H ( x , D ( u + C)) = - a + 5(u + C). By the comparison result for constrained solution (see w

we conclude

"5>_u+C, a contradiction to the choice of C. Therefore ~ > a. Reversing the roles of c~, the claim is proved. Using this fact for equation (HJB)~, the proof is easily completed. 9 REMARK 1.2. Uniqueness does not hold in general for equation (HJB)~. Indeed, it is clear that u + C is a solution of (HJB)~) if u is such and C is any constant. The next simple example shows that (HJB)~) can have a very large set of constrained viscosity solutions. Let f~ be any smooth bounded open set in ]~.N and A = { a E 11~g : lal < 1 }. For f(x, a) =- - a and e(x, a) - lal it is easy to compute sup{ - f ( x , a) . p -

e ( x , a ) } = (]Pl - 1) + .

aEA

The unique constrained viscosity solution of (HJB)~, ,~ > 0, is v~ = 0 so that )/0 = 0 and wa = 0 in this case. Since H(x,p) = (rPl-1) + > 0 for a l l p , any continuous function is a supersolution on f~ of (HJB)~; on the other hand it is easy to check that any Lipschitz continuous u with ]Du(x)l < 1 a.e. in ft is a subsolution of (HJB)~. Because of the lack of uniqueness for (HJB)~ we do not know in general if the whole family { w~ } converges to w0. <3 The next result states t h a t the constant X0 appearing in (HJB)~ can be interpreted in terms of a long run averaged cost criterion for the infinite horizon problem with state constraints.

1. ERGODIC PROBLEMS

401

PROPOSITION 1.3. Under the assumptions of Theorem 1.1, the constant Xo is given by X0 = c*EA~ inf f~l T--,+oc i m i n f Tl f0 r ~.(yx(8), ol(8)) d8 } . PROOF. By the DPP (see Proposition IV.5.5), a(vA(z) - e-~Tv~(y~(t, ~))) < ,~

(1.7)

e(~x(s), c~(s)) ds + a r

for any x E ~, T > O, ,~ > 0 and ~ E Ax, where ~(T) =

(e - ~ - 1)e(yx(s), ~ ( s ) ) d s .

An immediate computation shows that (1 I_~T ) Mb(T)_<M,~ ; - ~ e -T ,

M=

sup ]f(x,a)l. N['" x A

Let us fix 6 > 0. For ,k, T such that AT = 6, (1.7) yields

~

(v~/r(~) - ~ - % / r ( ~ ( T ,

~))) < ~

e(y~(s), ~(~)) & + ~ ( 1 - e -~ - ~).

Thanks to the uniform convergence in (1.4) and the compactness of ~ we obtain from the above, as T --~ +0% (1 -

e-5)Xo _ 61iminf T-~+oc T1 ~0 T ~(yx(s), c~(s)) ds +/11(1 - e -~ - 5).

The preceding yields, dividing by 5 and letting 5 ~ 0 +, inf "T 1 9s T ~(yx(S), ct(s)) ds, Xo _< lim T--+~c

V a 9 Ax .

To prove the opposite inequality, fix again 5 > 0. By Proposition IV.5.5, there exists ~ 9 ~x such that 5 2 +~(v,X(x) -- e-;~Tv,~(yx(T,-~))) > )~

t(yds),~(~))&+~r

and it is easy to see that ~ can be chosen independent of T (see the proof of the DPP, Proposition III.2.5). For ,~T = 5 this gives

g (ve,/T(X) e_Sv,vT(yx(T_~))) >_

52 + ~

_

g(yz(s),-~(s))

d s - ~I(1

-

e -~

-

5)

402

VII. ASYMPTOTIC PROBLEMS

for any T > 0. Hence, by the same arguments as in the first part of the proof, as T --* + 2 we obtain 52 + (1 - e-~)Xo >_ 61iminf T ~ + ~ T1 ~0 T g(yx(s),-~(s)) ds - M(1 - e -~ - 6 ) . Now take the inf over ,4x, divide by 5 and let 6 ~ 0 in the above, to get inf ~'liminf 1 ~ T } X0 > - ~eA tT--.+oc T g(y~(s),a(s))ds .

9

REMARK 1.4. The constant X0 is also related to the asymptotic behavior as T --* +~c of the finite horizon problem with state constraint ~ , for any terminal cost g. The value function of this problem is ,

=

g(y~(s,(~),c~(s))ds+g(yz(T,a))} aE.A~ ~ Jo

and it is not hard to show that X0 = limT~+or T - l v ( x , T ) Exercise 1.1).

uniformly in ~ (see <1

1.2. V a n i s h i n g d i s c o u n t in t h e u n c o n s t r a i n e d c a s e : o p t i m a l s t o p p i n g We consider now the same limit problem as in w in the unconstrained case 9 = R y . In this setting a result of the type of Theorem 1.1 still holds true. Namely, if we consider vA(x) = inf f + ~ o E A Jo

g(yz(s, o0, o~(s)) e -As ds

x e ]~N

the value function of the unconstrained infinite horizon problem, we have THEOREM 1.5. Let us assume (1.2). Then there exists a constant Xo such that Ave(x) ---* ~0, locally uniformly in ~ N as A --* 0 +. Moreover, for any fixed xo E ]~N, w A , ( x ) = vA. (x) -- vA~ (Xo) converges locally uniformly in R N f o r some i n --* 0 + to wo E C(]~ N) such that

s u p { - f ( x , a ) . D w o ( x ) - t ( x , a)} + X0 = 0 aEA

in R N

in the viscosity sense.

The proof of the preceding result is essentially the same as that of Theorem 1.1. A major difficulty arises, however, if one tries to establish an analogue of Proposition 1.3, because in this case the controlled trajectories are not necessarily bounded as T --* +c~ and the limit function w0 is only locally bounded in ~ N .

1. E R G O D I C P R O B L E M S

403

We describe in what follows some ergodic type results in the unconstrained case on a simplified model, namely, the optimal stopping problem (see w The value function is then (1.8)

vx(x) = inf

gl(yx(s))e-XSds+go(yx(O))e -~~

O>O~dO

}= inf Jx~(0) 0>0

and is the unique viscosity solution of (1.9)

max{Av~(x) - fl(x) 9OvA(x) - t~l(x); v)~(x) - g0(x)} = 0

x e RN .

Let us observe first that in this case the system is governed by the discrete control set A = {0, 1} with dynamics f(x.O) = 0, f ( x , 1) = fl(x). Hence ~-6f(x~A) is the line segment joining 0 with fl(x) and the controllability assumption (1.2) does not hold. We replace it with a dissipativity condition on fl, namely, (1.10) 3 F > O s u c h t h a t ( f l ( x ) - f l ( z ) ) . ( x - z ) < _ - F I x - z l

2

Vx~zE]~ N.

This condition implies, via Gronwall~s inequality, that (1.11)

ly~(t)-y~(t)l<e-Ftlx-zl

Vx, z e ~ N, Vt>0:

where yx(t) is the solution of y'(t) = A(~(t)),

y(0) = x

Moreover, if fl is Lipschitz continuous, then 3!x0 such that fl(xo) = 0

(1.12)

(this is a standard result in monotone operator theory, see Exercise 1.2). Properties (1.11) and (1.12)imply

(1.13)

lyx(t)

-

xol _< e-F'lx - xol _< Ix - :~ol

Vx 9 ~N

Vt > 0,

or, in other words, that any closed ball /~(x0, R) is invariant for the flow defined

byA. The limiting behavior of (1.8) is described by the next result. We denote by w~ the function

~(~)

= v~(~) - v~(x0),

where x0 is as in (1.12). THEOREM 1.6. Let us assume (1.10). Then lim )w~(x) = X0 -- min{gl(X0); 0} .

A---+0+

404

VII. A S Y M P T O T I C

PROBLEMS

Moreover, if gl (xo) >_ O, then at least a subsequence of {va} converges locally uniformly as A ---+0 + to a viscosity solution vo of (1.14)

m a x { - / l ( x ) . Dvo(x) - gl(x); vo(x) - g0(x)} = 0

x E ]1~N .

If, on the contrary, gl(xo) < O, then {w;~} converges locally uniformly as k ~ 0 + to the unique viscosity solution wo of

f -fl(x)"

(1.15)

Dwo(x) = h ( x ) - el(X0)

X

E ]~N,

wo(xo) = O.

PROOF. Let M

=

max{lsuplgl(X)l;suplgo(x)[}. RN

]I~N

It is easy to check that M is a supersolution of (1.9) whereas - M is a subsolution of (1.9)o Hence, by the comparison Theorem II.3.5, Iv:,(x)l <_ M

V x c IoN,

so that

(1.16)

IAvA(x)I <

max~sup<~N Asup]eO(X)]} I~l(X)];

VX E R N 9

~N

In order to obtain uniform Lipschitz estimates on {Ave} we apply Proposition II.4.2. For this purpose, let us observe that uA = (A + F)v~x satisfies (see Exercise 1.7) u~(x) + HA(x, Du~(x)) = 0

x 6 R N,

in the viscosity sense, where H~(x,p) = m a x { - l f l ( x ) . p -

~h(x);-(l

Also, for any C > 0, x, z, p c R N, (1.17)

H:~(x, Cp) - HA(z, Cp)

>_min{ C ( f i ( z ) - f l ( x ) ) . p - -

A+F

"1

= - ( e l ( x ) - el(z)); (~ + F)(eo(z) - to(x)) }) , 4

X--Z

(see again Exercise 1.7). For p = ]x - z]' the above and (1.10) yield

_> min

{oF A

A+ F L a ; - ( A + F)Lo ~ }Ix - z I ) A

1. ERGODIC PROBLEMS

405

where L0, L1 are Lipschitz constants for 2o, 21. Choosing C k max{ (1 + F)Lo; LI } we get X--Z

X--Z

H),(x, CT--~_ z[) - H)~(z, C ~ _ z] ) > -C]x - z],

VAE ]0,1[.

Therefore, by Proposition II.4.2, (A + F)lv~(x) - v~(z)l _< c l ~ - zl

vA e ]0,1[.

This implies, of course, (1.18)

I~A(x)- vA(z)l < yC l x - z I

VA e l 0 , 1 [

and

(1.19)

AC I~,~,(x) - ~xvz,(z)l ~ --Fix - zl

v~x e ]o, 1[

From (1.16) and (1.19) it is easy to deduce that a subsequence of Av~ converges to some constant XoLet us now show that X0 = min{21(x0); 0}. The representation formula (1.8) at the equilibrium point x0 gives

v;~(xo) =

inf ~21 (xo) j~o0 e -As ds

+ 2o(xo) e -A~ } .

OkO ~

An elementary computation then yields f2o(xo)

if 2 1 ( x o ) - Ato(xo) > 0,

VA(X0) Hence, for sufficiently small A > O,

~ A20(x0)

(1.20)

Av~(xo) = ~ A min{2o(x0); (21(X0)

0}

if 21(x0) > 0, if 21 (xo) = 0, if 21(X0) < O,

and the formula for X0 is proved. To prove the second part of the theorem, let us first treat the case gl(X0) > 0. From (1.18) and (1.20) it follows that (1.21)

Iv,~(x)l _< Iv~(xo)l +

C

ylx

- xop _< t6(xo)l +

C

yIx

- *of.

This uniform estimate, together with (1.18), allows us to apply the Ascoli-Arzel~ theorem and conclude that at least a subsequence of {v~} converges uniformly to

406

VII. A S Y M P T O T I C P R O B L E M S

some v0 as A --* 0 +. T h e assertion t h a t v0 is a viscosity solution of (1.14) is t h e n a direct consequence of Proposition II.2.2. Now consider the case Gl(X0) < 0; it is i m m e d i a t e to check t h a t wA = v~ - vA(xo) is a viscosity solution of (1.22)

m a x {Aw~(x) - f l ( x ) " D w ~ ( x ) - t l ( x ) + Ave(x0); w~(x) - eo(x) + v ~ ( x 0 ) } = 0

x 9 RN

As a consequence of (1.18), {v~} is locally equibounded and equi-Lipschitz continuous. Let w0 be any subsequential uniform limit of {w~} as A --~ 0 +. In the present case (see (1.20)) v ~ ( x o ) = A-1Gl(X0) ~ - o c as A --+ 0 +. Hence, for sufficiently small A, the m a x in equation (1.22) is a t t a i n e d by the first m e m b e r at any x 9 It~N, so t h a t wA is a viscosity solution of the linear equation A w ~ ( x ) - f l ( x ) " D w ~ ( x ) - GI(x) + A v ~ ( x o ) = 0

x 9 ]I~N .

The conclusion t h a t w0 is a viscosity solution of (1.15) is again a consequence of P r o p o s i t i o n II.2.2. As for the uniqueness of the solution of (1.15), let us observe t h a t ui(x) =

/o t(Gl(yx(s))

- Gl(X0)) d s + u i ( y x ( t ) )

Vt > 0, x 9 ~ N

if ui (i = 1, 2) are viscosity solutions of (1.15) (by Proposition II.5.18 and some calculations, or by R e m a r k III.2.34). Hence (~1 - ~ 2 ) ( x ) = (Ul - u 2 ) ( y x ( t ) )

vt>0,x 9

~

By a s s u m p t i o n (1.10) and its consequence (1.13), we can let t ~ + o c and o b t a i n (U 1 -- It2)(X ) :

(lt 1 -- U2)(X0)

:

0

VX 9 ]I~N,

and the proof is complete.

9

REMARK 1.7. E q u a t i o n (1.14) does not have a unique solution. As an example, suppose t h a t the stopping cost G0 satisfies -fl(x).

DQ(x)

- GI(X) <_ 0

x 9 ]~N

in the viscosity sense. T h e n u -- G0 is a solution of (1.14), but u does not necessarily coincide with v0 = lima_0+ v~. Actually, if Gl(x0) = 0 and G0(x0) > 0, then (1.20) gives vA(xo) = 0 for all )~, so t h a t 0 = vo(xo) < u ( x o ) = G0(x0). <~ We conclude this section with some properties of the undiscounted value: (1.23)

v0(x) := i n f ~ / 0 ~ Gl ( y x ( s ) )e - ~ d s + G0(yx(~))) } = inf J ~ 0_>0 ~

and an estimate on v~ - v0.

0>0

1. ERGODICPROBLEMS

407

PROPOSITION 1.8. Let us assume (1.10) and

(1.24)

~l(xo) > o .

Then for any R > 0 there exists T > 0 such that (1.25)

vo(x) =

inf J~ O
Vx e B(xo, R ) .

Moreover, for some K > O,

Vx, z 9 ~(xo, R ) .

Ivo(x) - vo(z)] _< K I x - zl,

PROOF. By the continuity of 61 and (1.24) we can choose 5 > 0 such that 1 ~I(Y) > ~ i ( x o )

Vy 9 B(xo,5) .

Thanks to (1.13), there exists T1 > 0 independent on x 9 y~(s) 9 B(xo,5),

V s >_ T1,

R) such that

V x 9 B(xo, R ) .

For ~ > T1 we have J~

= J~

+

~l(yx(S)) d~ + eo(ux(~)) - eo(u~(r~))

J~

>_ J~

so that 1

+ -~ e,(xo)(O - T~) - 2Mo,

where Mo = sup~(~o.R) I~o(~)l. Hence, for T = T1 + ~

4

supB(~o,R) reo(~)],

inf jo(o) _> jo(T). O_>T On the other hand, since T1 < T, J~ and (1.25) is proved. z E/~(x0, R). Then

>_ inf j o ( o ) 0
In order to prove the Lipsehitz continuity of v0, let x,

j o ( ~ ) _ j o ( e ) < LI

1~(8) - yz(8)l e~ + Lolyx(~) - yz(0)l;

where Lo, L1 are Lipschitz constant for ~~ fl- Hence, from (1.11), jo(~)) _ j o ( o ) < ( L I T + Lo){x - zl

V0 < 0 < T

and, by (1.25), this implies Ivo(x) -- vo(z)] <_ ( L I T + Lo)]x - z I,

Vx, z E/~(xo, R ) .

408

VII. ASYMPTOTIC PROBLEMS

PROPOSITION 1.9. Let us assume (1.10) and (1.24). Then for any R > 0 there exists CR > 0 such that (1.26)

]v~(x) - vo(x)[ _< eRA

Vx 9

R)

for sufficiently small A > O.

PROOF. Let us fix R > 0. For any x 9 that (1.27)

R) and ~ > 0 let O* = O*~.x,E be such

J~(O*) < v,x(x) + e .

We claim that (1.28)

limsup0*h_o+-< gl ( ~ o ) (2M~ + - ~

+ s).

Indeed, (1.27) yields for 31o _> sup [go[, f0 *

-- e-A0*

A

Hence, by (1.11),

f/"

M o + e _> - L ~ l x - x o ]

e-(F+;gSds+gl(XO) 1-e-)~~

Moe-''.

"~l~176 ;

this implies g l ( x o ) l - e -A )'~

LI

-< z + 2 3 I ~

Ix - x o [ _ < r

L1R

for all x E/3(xo, R). Therefore 0"_<-~ and the claim follows. For any x E/3(xo~. R) and 0 _> 0 we have (1.29)

IJxA(O) -- S~

~ ]~/'1 ~ ~

- e -As) d8 + 310(1 - e -X0) : : c2(A, O) .

Thanks to (1.25), (1.28), for small A we have sup}Jx~(O)

0>0

- J~

=

sup

O
[J~(O) - 3"~

for some TM < +~c independent of x E/3(xo, R). Hence there exists OM 6 [0, TM] such that Ivy(x)

-

vo(x)l < sup IJ~(0) 0>0

-

J~

< ~(A, 0M).

Since v(A, OM) = m l

(1.26) follows.

AOM + e -~~

- 1

+ -alo(1 - e-A~ ~I

1. E R G O D I C P R O B L E M S

409

Exercises 1.1. Consider the finite horizon problem

v(x, t) = inf

g(yx(S, a), a(s)) ds + 9(yx(T, a)),

c~EA~

where g is any given bounded continuous function on ]~N. Use the results of w (see, in particular, Remark III.3.10) to show that )% =

lim

T~+oc

1

~v(x,O),

uniformly in 12.

1.2. Suppose t h a t f l satisfies condition (1.10) and [fl(z)

-

fl(z)[ _< L[x

-

z[,

Vx,

Z

E ~N.

Show t h a t TQ(x) := x + ofl(x) (Q > 0) satisfies [TQ(x) - Te(z)[ 2 < (1 - 2•F + n2oa)[x - zl 2 . Deduce from this that the equation f l (X) : 0 has a unique solution. 1.3. For f l as in Exercise 1.2, let x0 be the unique point with fl(xo) = 0. Show that 1

6

tx >_ - - f log ~,

Vx E/~(z0,R),

where t~ = i n f ( t _> 0 : y~(t) E B(xo, 6) }. 1.4. Let u E C ( ~ ) N Lip(f~) be a solution of

sup{-f(x,a). Du(x) } = 0

in f~.

aEA

1 (i) Show t h a t z~ := ~ u is a solution of

Aza + sup{ - f ( x , a). Dza(x) - u(x) } = 0

in f~

aEA

and therefore

z;~(x) = inf f + ~ c u(yx(s, c~))e -~s ds; oLEAxJo

(ii) show, under the assumptions of Theorem 1.1, t h a t u is a constant.

410

VII. A S Y M P T O T I C P R O B L E M S

1.5. Consider the Dirichlet problem -f(x).

D u ( x ) - ~?(x) = 0 u = 0

in ft, on F0,

whereF0:={xE(9~:tx = 0 } and t z : = i n f { t k 0 : y x ( t ) ~}. Let S : = { x E f~ : tx = +oo }. Use the results of w to show that the preceding has at most one viscosity solution u E BC(I~ N) that satisfies the additional condition lim T1 ~0T u(yx(S)) ds = 0 r~+~c

uniformly in S .

1.6. Consider f E C 1 and suppose that S := { x E f~ : tx = + e c } is measurable. Show that either S has zero measure or there exists U C_ S with positive measure such that d i v f ( x ) < 0,

Vx E U,

where d i v f denotes the divergence of the vector field f. [Hint: set U(t) = { y E ]y~N : y = yx(t), x E U } and use Liouville's volume formula meas(U(t)) =

fu et div f(yz(t) ) dx.] 1.7. Use the elementary facts max{ a; b } = 0 r

max{ a; 7b } = 0

max{ 5a + c; ~a + d } = ~a + max{ c; d } max{ a; b } - max{ c; d } _> min{ a - c; b - d }, for a, b, c, d E 1~ and 7, 6 > 0, to complete the first part of the proof of Theorem 1.6. 1.8. Assume the existence of some p E C1(~) such that max{ - f l ( x ) 9D ~ ( x ) - gl(x); ~2(x) - e0(x) } < 0, p _< v0, in/~(x0, R), where v0 is given by (1.25). Prove under the standard assumption of f, gl,/?0, that v0 _> u for any viscosity solution u of max{ - f l ( x ) 9D u ( x ) - gl(x); u(x) - to(x) } _ 0

in/~(x0, R),

u(xo) < eo(xo). [Hint: modify the proof of Theorem II. 5.9 also taking into account Theorem IV.5.8.] 1.9. In the framework of Proposition 1.9 prove that if gl(x) :> el(x0) Vx E Ii~N, then the following variants of estimates (1.28) and (1.26) hold: 1 + z), o* ~ eC3~x~)(2Mo

Vx E ~ g .

Ivy(x) - vo(x)l < C)~,

Vx ~ ~ N .

9

2. V A N I S H I N G S W I T C H I N G

2. V a n i s h i n g

COSTS

switching

411

costs

In this section we analyze the limiting behavior of the value function v~ = ( v l . . . v~) of the optimal switching problem when the positive switching costs k~(i, j ) --+ 0 as E ---~ 0 + "

Let us recall from w to which we refer for notations, assumptions and basic results, that vE is defined in the case of discount rate A = 1 as oo

aLg"

v~(x) = ~e~ ~01=( ]

,-1 g(Yx(S'C~)'an-1)e-S ds + k ~ ( a n - l ' a n ) e - ~

= inf JE(x,(~),

for i E A = { 1 . . . . . m } .

aE;ol

and it is the unique solution of the system of quasivariational inequalities (2.1)

max{ %~ _ fi 9D v t - g , vi.r i

mi#n(v{ + k , ( i , j ) ) }

= 0

inlR m .

Setting formally ~ ~ 0 + in (2.1) we find that the limit vector ~ = lim,__,0+ vE, if any, should satisfy (2.2)

max{ yi _ f i . o ~ i _ g i; Vi _ m i n ~ J } = 0 j#i

in l~N,

for all i E { 1 , . . . , m }. The above yields, in particular, W <_ minj#i~J for all i E { 1 , . . . , m } so t h a t necessarily E1 - - ~ 2

~

...

=~;rn

.

Denoting by v the common value of W, i E { 1 . . . . , m }, from (2.2) we deduce that v should satisfy

max

{ v - fi . D v - gi } = 0

ie{ a ...... }

in

]I~N .

This is the Hamilton-Jacobi-Bellman equation for the infinite horizon problem with control set A = { 1 , . . . , m }. By the theory in w the guess is t h a t

v(x) = inf f + o c g(yx(S, (~), a ( s ) ) e -s ds .

(2.3)

aE'P dO

The preceding heuristics are confirmed by the next result. THEOREM 2.1. Let us assume

k~(i,j)>O, (2.4)

k~(i,i)=O,

kc(i,j)
lim k ~ ( i , j ) = O

~0 +

Vi, j E { 1

..... m}.

Then lira

%i = v

Vi E

{1,.. ., m},

~-..0 +

locally uniformly in ~ N , where v E BUC(]R N) is given by (2.3).

i#j#e,

412

VII. ASYMPTOTIC PROBLEMS

PROOF. The first step is to prove that

(2.5)

sup 14(x)l < M ~N

for some constant M independent of z, i. This follows from the fact that M = max sup Igi(x)l l < i < m ~N

satisfies (2.6) m a x { M - g i ( x ) ; M - m i n ( M + k ~ ( i , j ) ) } jCi

>_M-P(x)>O.

'r

--

N.

On the other hand, --~l--ei(x)
'~xE]I~ N, i E { 1 , . . . , m }

- M - m i n { - M + ks(i,j)} < O. jCi

so that (2.7)

m a x { - M - e l ( x ) ; - M - m i n { - M + ks(i,j)}} <__O. jT~i

Vx E a N .

Inequalities (2.6) and (2.7) show that M , - M are, respectively, super- and subsolutions of (2.1). Hence, by comparison, (2.5) follows. Define then u__i(x)=

l i m i n f vi(y), (~,y)--,(0+,x)

gi(x)=

l i m s u p vi(y). (~,y)--,(0+,y)

Let us observe at this point that for any i E { 1 , . . . ,m},

vie(x) < vJ(x) + ke(i,j)

(2.8)

Vx E ]RN, Y j 7~ i.

Actually, if this were not true, then there would exist x0 E ]l~Y~ i0 ~ j 0 E { 1 , . . . ~ m} such t h a t

vi~

(2.9)

> ~~

+ k~(io,jo)

for x in an open neighborhood B of x0. It is not difficult to show the existence of c2 E C I ( R N) such that v/~ - c~ has a local maximum at some xl E/~. Since ve is a subsolution of (2.1), this implies

viE~

<_jmin{vJ(xl) + ke(io,j)} -< yJo(r e [ X l ),

"4- k~(io,jo),

a contradiction to (2.9). From (2.8) and (2.4) it follows that ill ~ 112 ~ . . . __~ilm

~1 ~ ~2 ~ . . . ~ ~ n .

2. V A N I S H I N G S W I T C H I N G C O S T S

413

Let us denote b y u_(x) and g(x), respectively, the c o m m o n values of u_i(x) and W(x). By construction, u and g are, respectively, lower and u p p e r semicontinuous and (2.10) The

_u < ~ .

claim is that u, ~ are, respectively, a super- and a subsolution

(2.11)

u(x) +

max

{-fi(x).

Du(x)-

t~i(x)} = 0

of

x E ~N,

iE(1 ...... }

in the viscosity sense (see Definition V.1.1). A d m i t t i n g this claim, the comparison T h e o r e m V.1.3 and (2.10) imply u = g. Therefore, if u denotes the c o m m o n value of u and g, u is a b o u n d e d continuous viscosity solution of (2.11). Moreover, by the stability results of w (see L e m m a 1.9), i ?3E

) tt

locally uniformly in R N, for all i E { 1 , . . . , m}. By the theory of C h a p t e r III, w u = v and the proof is complete. In order to prove the claim, let 4 E C I ( R N) and let 5 be a strict m a x i m u m point for g - 4 i n / ~ = / ~ ( ~ , R). T h e n by L e m m a V.1.6, there exist en ~ 0 + and, for all i, a sequence ( X in)~eBI i n / ~ such t h a t ( v i - 4 ) ( x / ) = max~}(v~n - 4), (2.12)

x~ ---* 5,

v E~ i (xD -+

~i(~) = ~(~),

a s ,~ --, o r

Since v~ = (v~ . . . . , v~n) is a viscosity subsolution of (2.1),

i 9 D4(xin) --ei(xin) _< O, v,i~ ( x ni) - f i (Xn)

Vi E { 1,.. . , m } .

A passage to the limit as n ~ ec in the above proves, by (2.12), t h a t g is a viscosity subsolution of (2.11). On the other hand, let 4 E C 1 (N N) and x be a strict local m i n i m u m for u_- 4. As before, we can find & --* 0 + and (xi),~eN i n / ~ =/~(x_, R) such t h a t ( v i - 4 ) ( x ~ ) = min/)(vin - 4), (2.13)

x~ - - - . x__, v~~ , CxnJ ~~ ~

_ ~ ( x ) = _~(x),

as n - ~ ~ .

Now choose is E { 1 . . . . . m } such t h a t

(v~: - 4)(x~') = and set x~ := x~'. Then, by (2.14)

rain

m_in( v/ - 4),

jE{1 ...... }B

(2.4),

vi:(x~) < v~,~(xn) < V~n(Xn ) + ken(in,j),

Since v~ = ( V e1 , . . . , have

Ve m

n

V j r in.

) is a viscosity supersolution of (2.1), and (2.14) holds, we

~:(~)

- P(~).

D4(x~)

- e~(~)

> 0.

414

VII. ASYMPTOTIC PROBLEMS

Passing if necessary to a subsequence we may assume that in --~ i, and so in = i for all n large enough. Thus we can send n ~ cc in the inequality above and use (2.13) to get _u(_x) - f i ( x ) - Dp(_x) - tt(_x) _> O. Consequently, u is a viscosity supersolution of (2.11) and the proof is complete.

4

Exercises

2.1. Generalize Theorem 2.1 to the case where the switching costs kE(i,j) may depend on x. 2.2. Give a direct proof (i.e., without using the Hamilton-Jacobi-Bellman equations (2.1)) of Theorem 2.1. 2.3. Try to estimate the rate of convergence of v~ to v in the case k e ( i , j ) = k~ ~ for all i, j where k and -y are positive constants. 3. P e n a l i z a t i o n The purpose of this section is to indicate how classical penalization methods can be easily implemented in the framework of Hamilton-Jacobi-Bellman equations through the technique of viscosity solutions. Two examples are treated: penalization of unilateral constraints on the value function arising in optimal stopping problems and penalization of state constraints. Let us mention here that also the limit problems considered in w can be regarded as examples of penalization techniques. 3.1. P e n a l i z a t i o n o f s t o p p i n g c o s t s Let us consider the value function of the optimal stopping problem with discount -- 1 defined as

(31/

v(x) inf F e~(y~(~))e-~d~+eo(~(O))e =

9>O~dO

We know that in the framework of w of (3.2)

-~} .

v 9 BUC(]~ N) and is the unique solution

max{ v(x) - f ( x ) . D v ( x ) - ga (x); v(x) - g0(x)} = 0,

x 9 ]~N.

Let us associate with (3.2) the semilinear HJB equation (3.3)

1 u~(x) - f ( x ) . DuG(x) -- gl(x) + -~(u -- go)+(x) = O,

x 9 ]~N,

with z > 0. The simple observation that l(r-go(x))

+=

sup

a(r-go(x))

Vz 9

r9

3. PENALIZATION

415

allows us to rewrite (3.3) as (3.4)

sup {(1 + a)uE(x) - f ( x ) . Due(x) - (el(x) + ag0(x))} = 0, aE[o,1/~]

X E ]I~N. The usual dynamic programming technique (see Chapter III) shows that

(3.5)

vE(x) := inf f + , c (gl(Yx(t)) + a(t)eo(y~(t))) e -t-f~(~)d~ dr,

aEA~Jo

where .AE = { a : [0, +c~[ -+ [0, 1/El, a measurable}, is a viscosity solution of (3.4) (indeed, its unique viscosity solution in view of the uniqueness results in w Hence the representation formula (3.5) shows that v E can be interpreted as the value of a discounted infinite horizon problem with the time varying discount factor 9 ( ( 1 + a(s)) ds and the running cost

~(x,a) --~-~l(X)~-aeo(x),

(x,a)

e]l~ N x [0, 1/El,

which incorporates the stopping cost f0, after multiplication by the penalization factor a E [0, l/z]. Let us observe that the penalizing effect in (3.3) occurs only at those points x where uE(x) > e0(x) and becomes stronger and stronger as E --~ 0 +. Concerning the behavior of v E as z ---* 0 + we have the following quite natural result: THEOREM 3.1. Under the assumptions on f, e, go as in w v, locally uniformly in R N, with v given by (3.1).

we have lim v E = ~0+

PROOF. It is a simple matter to check that the constant /1J = max.~supR N leo(x)[; s~p [~1 (X)[ } is a supersolution of (3.3) for any z > 0. Similarly, - M is a subsolution of (3.3), for any E > 0. Hence, by comparison (see Chapter III, w ---~J ___~rE(X) < M

V x E ]~N, s > 0 .

Let us introduce then the weak limits u , ~ studied in w u(x)----

liminf v~(y), (E,y)---.(0+,~)

~(x)=

limsup v~(y). (E,y)~(0+,x)

416

VII. A S Y M P T O T I C PROBLEMS

By L e m m a V. 1.5, u, ~ are, respectively, lower a n d upper semicontinuous a n d satisfy (3.6)

u < ~

in ll~N .

Let us show t h a t ~ is a subsolution of (3.2) in the generalized viscosity sense of C h a p t e r V. For this purpose, let p E C 1 ( ~ N) a n d ~ a strict m a x i m u m point for - ~ in /~ = / ~ ( ~ , r). T h e n by L e m m a V.1.6 there exists x ~ c / ~ a n d Cn ~ 0 + such that (3.7)

(v E~ - ~ ) ( x '~) = rn~x(v ~ - ~),

xn

,5,

v ~'~ (x n) -----+ ~ ( 2 ) .

Since v e is a viscosity subsolution of (3.4), sup

{(1 + a)v En - f . 0 4 -

(el + ago)} < 0

at x n .

Let us choose a = 0 in the above to o b t a i n (3.8)

vr

n) - f ( x n ) 9 D ~ ( x n) - g l ( x n) <_ O .

On the other hand, the choice a = 1/en gives (1 + 1/En)ven(x n) - f ( x n ) 9 D ~ ( x n) - g l ( x n) - 1 Q ( x n )

< 0,

yielding (3.9)

EnVzn(X '~) + (V e" -- go)(X n) < ~,~f(xn) 9 D ~ ( x " ) + E n g l ( X n ) .

Now let n --~ + o c in (3.8) a n d (3.9) to obtain, t h a n k s to (3.7), ~(~) - f ( ~ ) .

D~v(5) - g l ( ~ ) < O,

( ~ - g 0 ) ( ~ ) _< 0 .

The two inequalities show t h a t ~ is an u p p e r semicontinuous viscosity subsolution of (3.2). We proceed now to prove t h a t _u is a lower semicontinuous viscosity supersolution of (3.2). Let t h e n ~ E CI(]R N) a n d z be a strict m i n i m u m point for _u - ~ in B = B(x_,r). By L e m m a V.1.6 again, there exist x '~ E / ~ a n d Cn --+ 0 + such t h a t (3.10)

(v ~ -- ~ ) ( x n ) = rn~n(vE~ -- ~2),

xn ~

x,

vEn(x n)

, u(x).

If u ( x ) > g0(x), t h e n trivially max{_u(x) - f(_x). D T ( x ) - el(x);_u(x) - g0(x)} _> O, so t h a t u_ is a viscosity supersolution of (3.2). If, on the contrary, u(x_) < g0(x), t h e n by (3.10) a n d the continuity of go,

(3.11)

!(vEn

-

e0)+(x ~) =

o,

3. PENALIZATION

417

for n large enough. Since v * is a viscosity supersolution of (3.3) we deduce v ~ ( x ~) - f ( x ~ ) 9D ~ ( x ~) - ~l(X n) > 0. Let n -+ +~c in the preceding to obtain u(x) - f ( x ) . D~(x_) - gl (x) > 0 and, a fortiori, max{ u -

f. D~-

g l ; u _ - * 0 ) _> 0

at x .

This shows t h a t u_ is a lower semicontinuous viscosity supersolution of (3.2). At this point we observe t h a t (3.2) is equivalent to (3.12)

max { v(x) - f ( x , a). D v ( x ) - g(x, a) } = 0 ae{0,1}

x 9 I~N,

with

f ( x , o) = o,

~(x~ o) = e0(x),

f(~, 1) = / ( x ) ,

~(z, 1) = el(x).

Therefore we can apply the comparison principle in the discontinuous case (see Chapter V, w to conclude t h a t _< u

in ]~N .

Taking (3.6) into account, ~ -- u. We have therefore proved t h a t u = ~ --- u is a bounded continuous viscosity solution of (3.2). Since the viscosity solution of (3.2) is unique, u coincides with v given by (3.1); by Lemma V.1.9 the convergence of v ~ to v is locally uniform and the proof is complete. 9 3.2. P e n a l i z a t i o n o f s t a t e c o n s t r a i n t s Let us consider now the infinite horizon problem with state constraints as described in w (in the case of discount rate A = 1). Its value function is defined for x E as (3.13)

v~(x)=

inf e ( y x ( t , a ) , a ( t ) ) e - t d t - = inf J ( x , a ) , aEA~ fO +:~ ~EA~

where ~ is an open bounded subset of ]~N with sufficiently smooth b o u n d a r y 0 n and Ax={t~EA:y~(t,~)C~,

Vt>0}.

We know t h a t under suitable assumptions on the d a t a (see w (3.14)

inf f ( x , a ) , n ( x ) < 0 aEA

V x E On,

including

418

VII. A S Y M P T O T I C PROBLEMS

where n(x) is the outward normal at x E 0f~, v ~ E BUC(-~) and is characterized as the unique solution of (3.15)

u(x) + sup{ - f ( x , a). Du(x) - g(x; a) } = O,

x E f~,

aEA

(3.16)

u(x) + sup{ - f ( x, a). Du(x) - ~(x, a) } > O,

x E f~,

aEA

in the viscosity sense. A first obvious remark is that

v~(x) _> v(x)

v:~ ~

~,

where v is the value function of the corresponding unconstrained problem; that is

v(x) = inf

e(yx(t,~),~(t))e-tat=

aE.A J0

inf J(x,c~). aE..4

A rather natural question, both from a theoretical and numerical point of view, is to approximate v ~ by the value function v e of a suitable penalization of the corresponding unconstrained problem. One possibility is to look at

ve(x) = aeA inf { J(x,c~) + 1s Jo f + c r p(yx(t, 0)) e -t dt

} =: ~exinf J~(x, a),

x E IRN .

Here e is a positive parameter and p is a distance-like function vanishing on f~, whose effect is to penalize the running cost of those trajectories that do not satisfy the state constraint. More precisely, we assume that p satisfies (3.17)

p E BUC(IRN),

p =- 0 in ~ ,

t h a t v ~ E BUC(]R N) and is the unique

We know then by the results in w viscosity solution of (3.18)

p > 0 in R N -.. ~ .

1

vE(x) + sup{ - f ( x , a) - DvS(x) - e(x, a) - -~p(x) } = O,

x E NN .

aEA

The next result describes the asymptotics of v ~ as e --+ 0 +. THEORElVI 3.2. Under the assumptions on f,g as in w (3.19) (3.20)

v(x)

< v~' (x) < vS(x)

< v~(x)

and (3.17)

V x E -~, 0 < e < E',

lim v ~ = v-5, uniformly in f~ .

PROOF. The inequalities (3.19) are the simple consequence of the definitions of v, ve,v~. In particular, the inequality v E < v~ in ~ follows from the fact that JS(x, ~) = J(x, c~), for any x 9 Q and c~ 9 Ax; hence

ve(x) = min / inf JS(x,~); " c~EAx

=min{v~(x);

inf

J~(x, ct) }

~EA\.A~

inf QEA'..A~

J~(x, oO } <_v'~(x ).

3. P E N A L I Z A T I O N

419

In order to prove (3.20), let us i n t r o d u c e the weak limits u_(x)=

l i m i n f ve(y), (~,u)~(0+.x)

g(x)=

limsup

(s,u)~(o+,x)

vE(y),

as in w F u n c t i o n s u, g are well-defined in ~ on the basis of (3.19). By definition, u, g are, respectively, lower a n d u p p e r semicontinuous on 12 and satisfy u__
g<_v~ (recall t h a t (3.14) implies the c o n t i n u i t y of v ~ on ~ ) . T h e plan of the proof is to show t h a t u is a viscosity s u p e r s o l u t i o n of (3.16). At this point the comparison T h e o r e m V.4.16 shows t h a t

u>v~ Hence _u = ~ = v ~ a n d v E converge uniformly to v ~ in ~ as ~ --~ 0 + (see L e m m a V.l.9). Let us prove t h e n t h a t u satisfies (3.16) in the viscosity sense. For this purpose, let x_ be a strict m i n i m u m of _u - ~ in B(_x, r) A ~ , ~ E C t (~), r > 0. E x t e n d ~ as a C 1 function on ~ x , still denoted by ~, a n d set ~(x) = ~ ( x ) - k]x - xl ~ where k > O will be selected later. Let x z be a m i n i m u m point of v E - ~ i n / ~ ( x , r); we can extract a sequence r --* 0 + such t h a t xn : = x ~n converges to some x* a n d v E" (xn) converges to some 7. We claim now t h a t x,~ E B(x_, r) for all n if k is chosen large e n o u g h a n d t h a t x* E ~ . Indeed, x~ E OB(x_, r) would imply by (3.19) t h a t

(v ~" - ~)(x~) > (v - c2)(x~) + kr 2 > _min (v - ~) + kr2; B(x,r)

on the other hand, using (3.19) again, ( v ~~ - ~ ) ( x n )

< ( v ~n - ~)(~_) < ( v ~ -

~)(~).

These two inequalities are c o n t r a d i c t o r y for sufficiently large k. Therefore the first claim is proved a n d this implies t h a t xn is a local m i n i m u m point for v E" - ~. Since v En is a s u p e r s o l u t i o n of (3.18), t h e n (3.21)

v ~" (x~) + H ( x n , D ~ ( x n ) ) > l p ( x , ~ )

with H ( x , q) = SUPaeA { - - f ( x , a ) . q -- t(x, a) }.

420

VII. A S Y M P T O T I C P R O B L E M S

To prove the second claim, assume by contradiction that x* ~ ~. Hence there is ~ > 0 such that p(xn) >_ 5 for n > n~, as a consequence of (3.17). This leads to a contradiction to the fact that the left-hand side of (3.21) is bounded uniformly with respect to n, as follows from (3.19). Let us now show that x* = _x and 7 = u_(x). Indeed, by the definition of_u there exists a sequence zn --~ x such that v E~ (zn) ~ _u(__x). From the choice of xn (v ~ - ~ ) ( x ~ )

~ (v ~ - ~ ) ( z ~ )

whence, by definition of % (u - ~)(x*) < 7 - ~(x*) < (u - ~)(x). Since we know that x* E ~ and x is a strict minimum for u - ~ in B(x, r) A ~ as well, the above yields x* = x and ? = u(x). At this point we let n --* +co in (3.21) to obtain

_u(x_)+ H(x_, D~(_x)) _> l i m s u p l p ( x n )

> 0,

showing t h a t u is a supersolution of (3.16).

Exercises 3.1. Give a direct proof of the fact that v ~ defined by (3.5) converges to v given b y (3.1) a s ~ - ~ 0.

3.2. Generalize the result of Theorem 3.1 to the case where

v(x) = inf inf ~ f e el(yx(s, ~), ~(s))e -s ds + e0(yx(0, ~))e-o } . aE.A O>_O " JO

3.3. In the framework of Theorem 3.1, try to estimate in terms of E sup~N IVY(X) --

v(x)l

a s E - ~ 0.

3.4. Give a direct proof of Theorem 3.2. 4. S i n g u l a r p e r t u r b a t i o n

problems

We analyze here a quite general singular perturbation problem in optimal control and some related topics concerning the approximation of the infinite horizon problem and of the monotone control problem (see w and w with similar problems where feasible controls satisfy a bound on the time derivative. 4.1. T h e i n f i n i t e h o r i z o n p r o b l e m for systems with fast components We consider here a system described by N + M state variables whose evolution is governed by

y(t) = y(y(t), ~(t), ~(t)) (4.1)

z~(t) = g(y(t), ~(t), ~(t))

t > 0,

4. S I N G U L A R P E R T U R B A T I O N P R O B L E M S

421

with initial conditions (4.2)

y(0) = x,

r

= z.

Here, f : ]RN x 1RM x A ~ ]I~N, g : ~:~N• ]~2~I • B ~ IRM, E > 0 is a small p a r a m e t e r and A, B are given sets where the controls c~, ~ take their values. For small values of the p a r a m e t e r E > 0 the y components of the state vary slowly in comparison with the fast variables ~. This situation is common in many applications and it is a convenient approach to reduced order modeling. Indeed, setting ~ -- 0 reduces (4.1) to (4.3)

y(t) = f ( y ( t ) , r

~(t)),

0 = g(y(t), r

9(t)).

Problem (4.3) is no longer a system of ODE's: this explains the terminology singular perturbation. Observe t h a t if the algebraic equation (4.4)

g(x, z, b) = 0

has a unique solution ~ = ~(x, b) E R M for any fixed x E R N 9b E B, then a substitution in (4.3) yields the reduced N-dimensional system (4.5)

y(t) = f(y(t),-2(y(t), ~(t)), a(t)), y(0) = ~ .

The optimal control problem t h a t we consider in this section is to minimize the performance index (4.6)

J~(x, z, ~, Z) =

f0

t(y~,z(t), e L ( t ) , ~(t)) e - ~ dr,

where Y~,z, ~ , z is the solution of (4.1), (4.2), over (.4 x B) ..... = { ( a , ~ ) : [0, +oc[ ~ A x B, ~xe,~(t) 9 O, Vt > 0 } . Here, O is a connected subset of ]RM with smooth boundary 0 0 playing the role of a state constraint on the fast variable ~. We also assume that O is open. We assume the controllability condition on the fast variables (4.7)

/~(0, 1) c ~6g(x, z, B )

V x E g{N z 9 -0,

as well as the uniform Lipschitz continuity assumption (4.8)

If(x, z, a) - f ( x ' , z', a)I + Ig(x, z, b) - g(x', z', b)[ < L(Ix - x'[ + Iz - z'l) ,

for all x, x ~ 9 ]RN, z, z ~ 9 O, with L independent of the control settings (a, b) E A • B. We assume also that

(4.9)

If(x,z,a) - e(x',z',a)l

< L ( [ x - x'[ + Iz -

z'[),

422

VII. ASYMPTOTIC PROBLEMS

for all x, x' in I~N, Z, ZI E 9 , with L independent of a E A, and

O<e(x,z,a)<M

(4.10)

V x E R 27,9 z E O ,

aEA.

The question that we address here is the limiting behavior of the value function (4.11)

v~(x,z) =

inf J~(x,z,a,3), (AxB).....

(x,z) E ~ N x

as e ~ 0 +. Note t h a t under this set of assumptions nothing can be said about the solvability of the algebraic equation (4.4). Nonetheless we are able to find a reduced N-dimensional system describing the limit problem. In fact, the controllability condition (4.7) allows the system to reach any point Zl E (9 from any starting point z in a lap of time of order [z - zll/z. Therefore the value function v E is expected to be less and less sensitive to z as e decreases, and to be independent of z in the limit s --* 0 +. Moreover, since the control ,8 does not appear in the equations for the slow variables y, nor in the cost functional j r , we expect its role reduces to driving optimally the ~ variables within (9. Since this can be done arbitrarily fast as s ---* 0 +, we expect/~ to disappear in the limit problem, and to take the role of a control varying in O. The preceding heuristic is confirmed by the following result. THEOREM 4.1. Let us assume (4.7), (4.8), (4.9), (4.10), and A > O. Then lim v~(x, z) ~0

=

v(x),

locally uniformly in ]~g

x

(.9,

+

with v given by (4.12)

v(x) = inf f + o o g(y~(t),r

x E R N,

aEA Jo

CEZ,

where yx(t) = yx(t, ~, c~) satisfies

(4.13)

~Ix(t) = f(y~(t), r

c~(t)),

y~(O) = x,

with r in the set z = { r : [0, + o o [ - - , o : r measurable}. PROOF. Let us start by observing t h a t the exterior normal vector to 0(]~ N • -0) at (x, z) is given by n(x, z) = (0g, n(z)) where 0 y is the null vector in ]~N and n(z) is the exterior normal to 0(9 at z. Moreover, by (4.7) and Carathdodory's theorem on convex hulls, it is not hard to show that for all (x, z) E 0 ( ~ y • -0) there exists b = b(x, z) E B such t h a t

g(x, z, ~). ~(z) < - M +----7'

4. SINGULAR PERTURBATION PROBLEMS

423

where M is the dimension of the vector g. Hence (4.14)

(f(x;z:a), !g(x,z,-b)) . n(x,z) : ~,(x,z,-b> . n(z) <

~(M + 1)

for all (x, z) E Oq(]~N • 9 ) . Therefore, by the results on the infinite horizon problem with state constraints of Chapter IV, w (see also Exercise IV.5.1), the value function v E E BUC(~ g • "0) and satisfies (4.15)

inf g(x, z, b) 9Dzv ~ = 0 Av~ + aeAsup{--f(x, z, a). Dxv ~ - t(x, z, a) } - 71 beS

in R N • O, as well as (4.16)

Av~ + s u p { - - f ( x , z , a ) . D ~ v ~ - - t ( x , z , a ) } - - - ~ b E1Binf g(x, z,b) 9Dzv ~ > 0 aEA

in IRN z 0 0 , in the viscosity sense. On the other hand, by the theory of Chapter III, v E BUC(II~ N) and is the unique viscosity solution of (4.17)

Av(x)+

sup

{-f(x,z,a).Dv(x)-g(x,z,a)}=O

inR N.

(a,z)EAxO

The uniform bound ]v~(x,z)I < M/.~ for all (x,z) E ]RN • O is an immediate consequence of assumption (4.10). We then define the weak limits _u(x, z) =

liminf (E,y,r

re(y, ~)

~(x, z) =

lim sup (~,y,r

ve(y, r

Functions u, ~ are, respectively, lower and upper semicontinuous and satisfy (4.18)

u<~

inR N •

The plan of the remaining part of the proof is to show that

(4.19)

v(x) < u__(x,z)

(4.20)

5(x, z) = 5(x)

(4.21)

~(z) _< v(x)

V(x,z) E ]RN x-O, V (x, z) e ~U • O,

Vx ~ ~ N .

Prom these and (4.18) the conclusion follows by Lemma V.1.9, as in the proofs of the various asymptotic results in this chapter. In order to prove (4.19), let us observe that v satisfies in the viscosity sense (4.22)

Av(x) + sup{ - f ( x , z, a). Dv(x) - g(x, z, a) } < 0 aEA

i n ]~N,

424

VII. A S Y M P T O T I C PROBLEMS

for all z E 59. This is a straightforward consequence of (4.17). Also, since v does not depend on z E 59, _ 1 inf g(x, z, b). D~v(x) = 0

(4.23)

in R N x 59

E bEB

in the classical and, afortiori, viscosity sense. Hence v is a viscosity subsolution of (4.15), so that by the comparison Theorem IV.5.8,

v(x)<:(x,z)

v(x,z) ~ ~:: • ~, v ~ > 0 ,

and (4.19) follows, by the continuity of v. Let us proceed now to show that the ~ is a subsolution of

ID~(x, z)l = 0

(4.24)

in ]1~N • 59

in the viscosity sense. For this purpose, let ~ E C1(~ N x 59) and (~,~) be a strict maximum point for ~ - ~ i n / ~ = / ~ ( ( ~ , ~), r). By Lemma V.1.6 there exist Zn ~ 0 + and (x n, z n) E / ~ such that

(:o

-

:)(:,:)

=

m_a~(:~

-

:),

B,.

(:,:)

--~ (~,~),

:~

: ) ---~ ~(~,~).

Then, from (4.15)

(4.25) ~:~(:,:) + sup{ -f(:, z", a). D,:(:, z") - t(x",z", a) ) 1

aEA

< - - inf g(x '~, z n, b). Dz~(X n, z'~). -- g n bEB

The right-hand side of the above is the same as 1 -

inf

-

~. Dz~(x '~, z~);

En ~TE'f"6g(xn z,~,B)

by assumption (4.7) we obtain 1

inf

~n ~?E~6g(xn,z~,B)

~7" Dz~( xn, z~) <- 1

inf

7/. Dz~(X n, z n)

~n rlE/~(0,1 )

1

= - - - IDz~(Z '~, zn)l. ~n

The left-hand side of (4.25) is bounded uniformly in n; hence letting n --+ +co in (4.25) yields IDz~(~,~)l < 0, which proves that ~ is a subsolution of (4.24). This implies that ~ is locally Lipschitz continuous as a function of z E O (see Chapter V, Exercise 1.4). This easily implies, using (4.24) again, that ~ is constant with respect to z, and we obtain (4.20). The proof will be complete after we have established the validity of (4.21). To this end we show that ~ = ~(x) is a viscosity subsolution of (4.17). Indeed, if

4. S I N G U L A R P E R T U R B A T I O N

PROBLEMS

425

~o E C1(~ N) and 9 is a strict local maximum point for ~ - ~, then (~, -2) is a strict maximum for ~ - p - Iz _-2]2 in/3~ :=/3((5,-2),r), for any fixed -2 E (9. Hence, by Lemma V.1.6, there exist (x n, z n) -o (~,-2) and sn --+ 0+ such that

(~-

_ ~)(~,

v ~o ( ~ ,

z ~ ) = m ~ x ( ~ ~o - ~ ) ,

zn) --+ ~(~).

Since v ~ is a viscosity subsolution of (4.15) and z ~ E (9 for large n, (4.26)

Av e" (x n, z n) + sup{ - f ( x " , z", a). D~(x n) - e(x n, z n, a) } aEA

1

<--inf

g(x~,z~,b).2(z ~ - - 2 ) .

Cn bEB

Now (4.7) gives inf g(x n,z n,b). (z n --2) < b cB

inf

9 (z n - ~ )

<__o.

r/E/)(0,1)

Letting n --- + e c in (4.26) we conclude that A~(5) + sup{ - f ( ~ , -2, a)- D~(5) - g(5, 2, a) } < 0

V~E(9.

aEA q

This shows, after taking the supremum over (9, that ~ is a viscosity subsolution of (4.17). By the comparison Theorem V.1.3 we obtain (4.21) and the proof is complete. 9 The result of Theorem 4.1 can be improved by showing that v c converges to v locally uniformly in ]t~N X O (see Exercise 4.2). The same remark applies to Corollary 4.2. To conclude this section we discuss the limiting behavior as e ~ 0 + of the value functions of the Lipschitz control problem with prescribed Lipschitz constant 1/z. The Lipschitz control problem models the optimal regulation of those control systems that do not allow, for technological reasons, an instantaneous switching between different control settings. Its value function is (4.27)

v~(x, z) := inf J(z, ~) := inf

g(yx(t, (), ~(t)) e -~t dt,

where Yx( ' , ~) solves

ft(t) = f ( y ( t ) , ~(t)),

(4.28)

y(0) = x,

and s

= { r : [0, + ~ [

--+ D : [r

- r

_< e - l i t - t'l, r

= z }

426

VII. ASYMPTOTICPROBLEMS

is the class of Lipschitz controls with prescribed initial value z 9 O and fixed Lipschitz constant 1/z > 0. The behavior of v ~ as z --* 0 + can be analyzed in the framework of Theorem 4.1. Indeed, a feasible 4 9 s can be interpreted as a fast state variable driven by the state equation i 9(t),

(4.29)

~(t) = r

= z 9 o,

with /3 E B:,r = {/3 : [0,+oo[ ~ / ~ ( 0 , 1 ) : 4~(t) E O for t > 0 } . COROLLARY 4.2. Under the assumptions (4.8), (4.9), (4.10), and A > O, v ~ given by (4.27) converge locally uniformly in ]t~g • 0 as E ---* 0 + to v given by v ( x ) := inf J ( x , 4) CEZ where Z = { 4 : [0, +oc[---, O : 4 measurable }.

PROOF. The statement follows from Theorem (4.1) when applied with A a singleton, B =/~(0, 1) C ]~M f ( x , z, a) = f ( x , z), g(x, z, b) -- b. Observe that with these choices condition (4.7) is trivially satisfied. 9 From this result we obtain easily that the value function in the infinite horizon problem does not change if we restrict the set of admissible controls from merely measurable functions to Lipschitz continuous functions with given initial point, provided there is no bound on the Lipschitz constant. More precisely (4.30)

v(x) =

inf g(x, ~) = inf{ 3(x, ~): ~ E ZLip, ~(0) ~EZLip

=

Z

},

V Z 9 O,

where ZLip = { ; : [0, +oc[--* O : 4 Lipschitz continuous }. 4.2. A s y m p t o t i c s for t h e m o n o t o n e c o n t r o l p r o b l e m An approximation of the monotone control problem which is appropriate in the modeling of control systems that do not allow instantaneous switching between different control setting is (4.31)

v~(x,z) =

inf J ( x , 4 )

where J is as in (4.27) and M~ = { r : [0, +oo[ --. ]R: 0 < 4(t) - ;(t') << (t - t ' ) / s for t > t', 4(0) = z } is the class of one-dimensional monotone nondecreasing controls with fixed upper bound 1/s on the time derivative.

4. SINGULAR PERTURBATION PROBLEMS

427

The preceding model is appropriate for systems whose evolution is driven by a non renewable resource (see Chapter III, w Much in the same way as for the case of Lipschitz controls (see the previous subsection) we interpret any ~ E A4~ as a '"fast" variable driven by the dynamics 1 ~(t),

=

r

= z ~

~,

with/3 e / 3 = { f~: [0, +c~[ --* [0,1] :/3 measurable }. We again take A as a singleton, f ( x : z, a) = f ( x, z), g(x, z, b) = b, but now B is the closed interval [0, 1] as dictated by the monotonicity condition. Hence, condition (4.7) does not hold. Consequently, the behavior of v e as ~ --* 0 + cannot be deduced from Theorem 4.1. However, a somewhat similar asymptotic result holds as indicated in the following. Heuristically, as E ~ 0 + the upper bound on the time derivative of feasible controls becomes larger and larger so that one expects v E to converge to (4.32)

v(x,z) =

r

inf J ( x , ~ )

with A4z = { r [ 0 , + o c [ - - ~ : 0 < r 1 6 2 fort>t',r Unlike the previous example, the limit function here depends on z, due to the unilateral constraint on r As expected, this is reflected by a unilateral constraint on Ov/Oz, namely, Ov

~ ( x , z) < o,

(4.33) in the viscosity sense.

THEOREM 4.3. Let us assume (4.8), (4.9), (4.10) and )~ > O. Then lira v~(x, z) = v(x, z)

locally uniformly in ]~N •

e--~O +

where v is given by (4.32). PROOF. Since v s is viewed as the value function of a standard infinite horizon problem with state variables (y, ~), by the results of w v ~ e BUC(]~ y • R) and satisfies equation (4.15) which in the present case (recall that f ( x, z, a) = f ( x , z), g(x, z, b) = b and B = [0, 1]) takes the form (4.34)

Av~(x,z)_f(x,z).Dxv~(x,z)_g(x,z)

in ~ N X ]I~. By the arguments of w Bellman equation (4.35)

+ _~[_._~.~z Ov c ( x , z ) + = 0

we see that v satisfies the following

max{ Av(x,z)- /(x,z). Dxv(x,z)-e(x,z);--5-~zOv (X,Z) } =0

428

VII. ASYMPTOTIC PROBLEMS

in ]~N • ~, in the viscosity sense. Observe that any viscosity subsolution of the above satisfies (4.33). It is immediate to check that M = sup~N• I is a supersolution of (4.34), whereas - M is a subsolution. Hence, by comparison, lyE(x, z)] < M for any (x, z) E ]~N X ]~ and r > 0. We then define _u(x, z) =

lim inf vE(y, ~), (~,y,~)~(0+,x,z)

~(x, z) =

lira sup rE(y, ~ ) . (~,y,~)~(0+,x,z)

Functions u, ~ are, respectively, lower and upper semicontinuous and satisfy u_<~

in ]~Y x ]~.

The claim now is that ~ is an upper semicontinuous viscosity subsolution of (4.35). For this purpose, let ~ E C 1(•g x ]~) and (E, 3) be a strict maximum point for ~ - ~p in/3~ :=/3(~, r) where ~ = (~,~). By Lemma V.1.6, there exist ~n E / ~ and r --~ 0 + such that

( ~ ~ _ ~)(r

= m _ a x ( ~ - ~),

~

~~

B~

(4.36)

, ~,

~

~(~).

Since v e is a viscosity subsolution of (4.34), we have (4.37) (4.38)

Av~ (~,~) _ f(~,~). Dz~2(~n) _ g(~n) < 0, _

(~)

< _sn()wz,(~n)

_ f(~,~) . D ~ : ( ~ n ) _ g(~n)) .

Thanks to (4.36) and the uniform bound on v ~, we can let n --~ +c~ in (4.37), (4.38) to obtain

~(~) - f(~). n~(~)

- e(~) < 0,

0: (-~(~))

+

< 0

Hence - ~ - ~ (~) < 0. This implies that ~ is a viscosity subsolution of (4.35). Let us proceed now to show that u is a viscosity supersolution of (4.35). If ~_= (x,z) is a strict minimum point for u - ~, ~ E C1(]~ g x ]~), and - ~--~-~(~_) > 0, then the supersolution condition is trivially satisfied. If, on the contrary, -~---~(_~) < 0, then by Lemma V.1.6 there exist ~ ~/~r : = / ~ ( ~ , r ) and r

~ 0 + such that

(v~ - ~)(~") = m_in(v~- - ~), B~

(4.39) C - ~ ~,

~- (~) ~

_~(~) 9

0~ ~ ~ + By the continuity of O ~ / O z we have ~ (-~zz()/, = 0 for sufficiently large n, that (4.34) yields

SO

5. BIBLIOGRAPHICALNOTES

429

If we let n --+ + o c in the preceding we obtain, by (4.39), A_u(~) - f(_~) 9Dz~(~_) - t(~_) > 0 and, afortiori, max{ Au_(x_,z__)- f(x_,_z) - D x ~ ( x , z ) - t(x, z _ ) ; - - ~ ( x , z) } _> 0 . Now we claim that a comparison principle holds for semicontinuous sub- and supersolutions of (4.35): it can be obtained by combining the arguments of the proofs of Theorem III.4.6 and Theorem V.1.3 and we leave it as an exercise for the reader. Then _u = ~ = v, and this implies the local uniform convergence of v ~ to v by L e m m a V.1.9. 4 Exercises

4.1. Assume t h a t B(0,1) C g(x,z,B), Vx 9 R N, Vz 9 -0. Prove that for any ( x, Z) 9 ]1:~N • 0 ~ there exists t* > 0 and ~/: [0, t*] ~ ]~N X ~ such that: V is continuous,

-~(0) = ( x, z),

7(t) 9

Yt 9

N•

v~(x,z)-vE(7(t))<_2Mt,

VtE[O,t*],VO
l.

4.2. Use Exercise 4.1 to show that v E converges pointwise to v in ] ~ g X 9 . Conclude by a monotonicity argument t h a t v s converges to v locally uniformly in ]l~N X 9 . This improves the result of Theorem 4.1. 4.3. Show t h a t the convergence in Corollary 4.2 is monotone and prove (4.30). 5. B i b l i o g r a p h i c a l

notes

Our presentation of the vanishing discount problem in presence of state constraints in w is modeled on that in [CDL90] with a more explicit use of controllability conditions and with new proofs based on the weak limit technique described in Chapter V. Recent results on the asymptotics as A tends to zero for equation (HJB)~ with different boundary conditions (modeling, for example, the reflection of the state when it hits the boundary) can be found in [Ar95a, Ar95b]. The role of dissipativity conditions on the dynamics in the ergodic problem for the unconstrained optimal stopping time problem was pointed out in [CDMe86]. Our treatment in w is inspired by [CDMe88], with partly different proofs, and by some unpublished work of Capuzzo Dolcetta and H. Ishii on the estimate of the rate of convergence. It is worthwhile pointing out that similar ergodic problems arise in connection with ~0r control (see for example [FM92b, FM95, Cam96]). Different results on the ergodic problem for the infinite horizon problems can be found in [Co189]; we refer also to the book [CH87] for some related topics.

430

VII. A S Y M P T O T I C P R O B L E M S

Let us mention also that there is a wide literature on ergodic problems for second order Hamilton-Jacobi equations connected with optimal control of stochastic processes (see, for example, [Gi85, LP86, CDG86] and [Ro83] for a survey). The convergence result for vanishing switching costs in w is taken from [CDE84] with a slightly different proof. The penalization technique has been widely employed in the domain of elliptic and parabolic variational and quasivariational inequalities connected with free boundary problems, see [BL82] as a general reference on this topic. Subsection 3.1 can be seen as a variation of this theme in connection with variational inequalities for a first-order partial differential operator. Related results can be found in [CDMe88]. In w a different kind of problem is considered where the penalization term bears on the boundary condition. The asymptotic result there is taken from [CDLg0] with a new proof using weak limits, as usual in this chapter. Singular perturbation is an important topic in system theory and in optimal control, see, e.g., [Kok84, Ben88] and the references therein. The result on optireal control of systems with fast components in w seems to be new; our PDE treatment is inspired by [L85c] (see also [$93] for a similar approach to a stochastic problem). For some extensions see [Bag93, BBCD94, BagB96, An96]. The Lipschitz control problem that we study as a byproduct is also considered in [BEJ84] for differential games. The asymptotic result for the monotone control problem in w is essentially taken from the original paper [Bn85] with a new proof. To conclude, let us mention that many asymptotic problems of different nature have been analyzed by means of viscosity solutions. We refer, for example, to [LPV86, E89, E92, B90, Alv97b, HI97] for homogenization, to [Bn93] for the related problem of averaging, to [FrSo86, Di92] for blowup of solutions, to [L82, Ba85c, DR93] for large time behavior in evolutive HJ equations, and to [DISg0] for an asymptotic analysis of queuing systems. Viscosity solutions methods were applied to the "geometric optics" approximation of wavefronts propagation in reactiondiffusion equations and systems [ES89, BES90, MSou94], as well as to several other asymptotic problems related to the propagation of fronts (such as De Giorgi's definition of the motion by mean curvature via singular perturbations [DeG90, DeG92]); see, for instance, [ESS92, BSS93, SSou96] and the survey paper [Sou95].

CHAPTER

VIII

Differential Games

In this chapter we consider two-person zero-sum differential games. Let us describe them. We are given a system (0.1)

SY'(t ) = f(y(t), a(t), b(t)),

l~(0)

t~O,

x,

where

(0.2)

f : ]I~N X A x B ---* ][~N is continuous, A, B are compact metric spaces,

and b = b(- ) 9 B := { measurable functions [0, +cc[ ~ B } and a = a ( . ) 9 .A are, respectively, the control functions of the second and the first player. The solution of (0.1) is denoted by Yx(" ;a, b). We are also given a cost functional J(x, a(. ), b(- )), which the first player wants to minimize and the second player wants to maximize. In other words, - J is the cost the second player has to pay, so the sum of the costs of the two players is null, which explains the name "zero-sum". Finally we are given an information pattern for the two players prescribing, roughly speaking, t h a t each of them choose his or her own control at each instant of time without knowing the future choices of the opponent. This is made rigorous in Section 1 by introducing the notion of nonanticipating strategy, where one player knows the current and past choices of the control made by his opponent. In Section 3 we assume instead t h a t one player knows just the current state and its past history, and model this information p a t t e r n by means of feedback strategies (strategies are denoted by Greek letters, a and 3, t h a t is why we have changed notation with respect to the previous chapters and we now denote control functions by a and b). These dynamic information patterns are more interesting for applications than the one where the first player chooses over A and the second player over B, namely, the static game whose lower and upper value functions vs and us are (0.3)

vs(x) := sup inf J(x,a,b) < inf supJ(x,a,b) =: us(x). bEB aEA

aEA bEB

432

VIII. DIFFERENTIALGAMES

Note t h a t the lower value corresponds to a game where the second player makes his choice with the information of the whole future response of the first player to any control function b E B, and the roles are reversed in the case of the upper value. The static game is actually a two-person zero-sum game over the infinite dimensional sets .4 and B in the sense of classical game theory, but it is not fit to be analyzed by the Dynamic Programming method. There are two main motivations to the theory of (two-person, zero-sum) differential games. The first is modeling a real conflict situation where two controllers have opposite interests, so that the loss of one coincides with the gain of the other. The most classical example is the pursuit-evasion game where the first N / 2 state coordinates describe a moving object controlled by the first player, the other coordinates describe another object controlled by the second player, and the cost J is the capture time, that is, the first time when some prescribed coordinates of the first object get close enough to the corresponding coordinates of the second object. The second motivation comes from optimal control problems with a single controller where, however, the system a n d / o r the cost functional are affected by some unknown disturbance b. The traditional approach of stochastic control theory consists of choosing a statistical model for the disturbance and minimizing the expected value of the cost. In some cases this approach may not be appropriate because minimizing the expected value does not give any information on the dispersion of the realizations, and does not guarantee against some possible "catastrophic" events. Moreover, the statistical model may not be good enough. In these cases it may be useful to adopt a pessimistic, risk-averse approach and model the disturbance as the control of a second player willing to maximize the cost. This leads to a two-person zero-sum differential game, and it is equivalent to minimizing the cost in the case of the worst possible behavior of the disturbance. We refer the reader to Appendix B for the connections with robust optimal control and ~ control. In this chapter we study only the infinite horizon differential game, where the cost functional is

(o.4)

J(x, a, b) :=

e(Vx(t), a(t),

b(t))e -at dt,

where yx(t) = yz(t; a, b) is the solution of (0.1) corresponding to a E ..4 and b E B. The reason for this choice is that the infinite horizon is our model problem in Chapter I and it is the first one studied in Chapter III. However, all the other cost functionals considered in this book can be studied in the context of differential games by the same methods, as well as many approximation, perturbation, and asymptotic problems of Chapters VI and VII. In fact, the proofs of the main results of the book are mostly based on the HJB equations, and similar equations are satisfied in the viscosity sense by the value functions of differential games. As an important example we prove in Section 3 the convergence of approximations by discrete time games. We assume throughout the chapter that the system satisfies, for some constant L, (0.5)

( f ( x , a, b) - f ( y , a, b)) . (x - y) <_ L I x

-

yl 2

1. DP FOR LOWER AND UPPER VALUES

433

for all x, y E ~:~N, a E A, b E B. Under this assumption we have the usual estimates on the trajectories: (0.6) (0.7)

lyx(t,a,b) - yz(t,a,b)l

~ eLtix-- z],

t>0,

lye(t, a, b) - x I <_ Mxt,

t E [0, 1/Mx],

lyx(t,a,b)[ <_ (1~1 + 4 Y ~ ) eKt,

(0.8)

for all a E .A, b E B, where

M~ := max{ lf(z,a,b)l : lx - zl < l, a E A, b E B }, K := L + m a x { i f ( O , a , b ) l : a 1. D y n a m i c

Programming

E A,b E B}.

for lower and upper

values

We begin with the definition of nonanticipating (or causal) strategy. DEFINITION 1.1. A strategy for the first player is a map a : B --~ .4; it is nonanticipating, if, for any t > 0 and b, b E 13, b(s) = b(s) for all s < t implies a[b](s) = a[b](s) for all s _< t. We denote with F the set of nonanticipating strategies for the first player. Similarly, the set of nonanticipating strategies for the second player is A := { ~ : A --, t~ : a(s) = ~(s) for all s < t implies 13[a](s) = 13[~](s) for all s < t } .

<~

EXAMPLE 1.2. A trivial example of strategy a E F is a constant one: for a fixed E ,4, a[b] -- ~ for all b E B. Another example of nonanticipating strategy is c~[b](s) = ~(b(s)) where ~ : B --~ A is a given map such t h a t ~ ( b ( . )) is measurable for any b E B. <~ EXAMPLE 1.3. A time-dependent feedback map 9 : ]~lv x [0. oc[ --+ A is admissible for x if for every b : [0, T[ --~ B measurable and T E ]0, +oc] there exists a unique solution of

q'(t) = f(q(t), ~(q(t), t), b(t)),

q(o)

0 < t < T,

x,

such t h a t t ~-* ~(q(t), t) is measurable. We associate with ~ (and x) the strategy a[b](t) : = ~(q(t),t), which is nonanticipating and such t h a t Y x ( ' , a[b], b) = q( 9 ) for all b E B.

<J

EXAMPLE 1.4. One may want to apply a feedback map to some weighted average of the past history of the state instead of its current value. Then one supposes t h a t for any measurable b and T E ]0, +oc] there exists a unique solution of

(q'(t) = f(q(t),e~(I(t),t),b(t)), (1.1)

/"} I(t)

=

fot

( q ( 0 ) = x,

q(s) d#t,

0 < t <_ T,

434

VIII. D I F F E R E N T I A L GAMES

such t h a t t ~ ~(I(t), t) is measurable, where, for all t, Pt is a given probability measure on [0, t]. Given 4~, Pt, t _> 0, a n d x, the strategy a[b](t) : = ~ ( I ( t ) , t ) is n o n a n t i c i p a t i n g a n d such t h a t yx( 9 ; a[b], b) = q( 9 ) for all b E B.

<3

EXAMPLE 1.5. A strategy t h a t is n o t n o n a n t i c i p a t i n g is, for instance, a[b](s) : = ~(b(s+7)) for any fixed r > 0 a n d ~5 : B --* A such t h a t ~ ( b ( . + 7)) is measurable for all b E 13. <3 Other examples are given in w Now we can define the lower a n d u p p e r values of a differential game. DEFINITION 1.6. The lower value of a game with cost functional J : ~ N • .A • • R is

v(x) :: inf sup J ( x a[b]. b); c~EF bEB

the upper value is

u(x) : = sup inf J(x, a,/3[a]). ,~EA aEA

If v(x) = u(x), we say t h a t the game with initial point x has a value.

<3

Note that (1.2)

vs _< v _< us

and

v, _< u _< us,

where Us, Vs are the upper and lower values of the static game, see (0.3). In fact, for a n y x E N N a n d ~ > 0 there is ~ E F such t h a t

v(x) + E > J(x, ~[b], b) > inf J(x, a, b)

for all b E B,

aEA

which proves the inequality vs <_ v, while the inequality v < Us follows from E x a m ple 1.2, because F contains a copy of ,4. T h e inequalities a b o u t u are proved similarly. Note also t h a t the inequality (1.3)

v(x) < u(x)

for all x,

which would justify the terms "lower" a n d "upper", is not obvious at first glance. (Actually one might first guess the opposite inequality because inf sup > sup inf if they are taken over the same sets; here, however, the inf in the definition of v is taken over strategies whereas in the definition of u it is taken over controls, and similarly the sup is taken over different sets in the two definitions, so the inequality v > u is false in general, see E x a m p l e 1.7.) We prove (1.3) for the infinite horizon problem in a rather indirect way, by using the Hamilton-Jacobi equations associated with the game. A n intuitive reason why (1.3) is true is t h a t ,

1. DP FOR L O W E R AND U P P E R VALUES

435

even if the players choose simultaneously with no knowledge of the future behavior of the opponent, in the definition of lower value the first player makes his choice of the strategy a with the information of the second player's optimal response to any a. Roughly speaking, at each instant of time the first player knows the choice the second one is making at the same time. This informational advantage goes to the second player in the definition of upper value, so (1.3) is plausible. The reader can be easily convinced of this argument by looking at Example 1.7, where actually (1.3) holds with the strict inequality. The information p a t t e r n described is not always realistic for the applications because of the advantage given to the player using strategies. However, it is reasonable to believe t h a t any more fair game has an outcome between v(x) and u(x). Indeed, this is the case for the game with feedback strategies t h a t we study in Section 3 (see also Exercise 1.5). For these reasons it is interesting to give conditions ensuring the existence of a value, that is, the equality v(x) = u(x). From now on we study the lower and upper values for the infinite horizon problem, t h a t is, with the cost functional given by (0.4), under the following assumptions on the running cost and the discount rate:

/

~ : ]l~N iX A x B --' ]R is continuous, there are a modulus we and a constant M such that

(1.4)

le(x, a, b) - et(y, a, b)] _< le(x,a)l <_ M, for all x , y

dlx E

yl) and

][~.N, a

E

A, b E B;

A>O.

Therefore in the rest of this chapter the lower and upper values are

v(x) := inf sup

e(y.(t),a[b](t),b(t))e-atdt,

a E F bEB JO

u(x) := sup inf

e(y~(t),a(t),Z[a](t))e-atdt,

~E A aEA

for z E ]~N. First we give an example where v(x) < u(x) for all x. (We refer to Exercise 2.1 for an example where vs(x) < v(x) = u(x) < us(x) for some x.) EXAMPLE 1.7. Let N = 1, A = 1, f ( z , a, 5) = (a - b) 2, A = B = [-1, 1], ~(x, a, 5) = g(x) = s g n x 9 (1 - e-I*l). Since the running cost f is strictly increasing and y' = f > 0, the best move of the minimizing player "a" is to keep the system still, so his optimal strategy is a*[b](t) = b(t)

for all t .

W i t h this choice J(x, a[b], b) = e(x) for all b, thus v ( z ) = e(x)

for z e ~ .

436

VIII. DIFFERENTIALGAMES

On the other hand the best choice of the maximizing player "b" is to make (a - b)2 as large as possible, so his optimal strategy is { 3*[a](t) =

: _

ifa(t) O,

and the best choice of the first player under this strategy is a -= O. Then we compute

J(x, 8" [0], O) =

e(x + t)e -t dt = 1 - --if-

for x >_ 0,

and claim that e-X

u(x) = l

e-x

2 = g(x) + -~ > v(x),

forx_>O.

A similar calculation can be made for x < 0. This claim can be checked by the verification theorems of the next section, for instance. <3 As usual, we begin with a regularity result for the value functions. PROPOSITION 1.8. Assume (0.2), (0.5), and (1.4). T h e n v andu are in BUC(RN).

If, moreover, wt(r) = Lgr (i.e., g is Iipschitzean in y uniformly in a and b), then v and u are HSlder continuous with the following exponent "~: 1

7=

any T < l ( A/L

/fA > L,

irA=L, if A < L .

P R o o f . The estimate ]v(x)t < M / A is an immediate consequence of (1.4). We fix x, z E ~N, e > 0 and first pick ~ C F such that

v(z) > sup Z(z, ~[b], b) - e/2, bE•

then b E B such that

v(x) < sup J(x,-~[b], b) < J(x, ~[b], b) + ~ / 2 . bE~

Thus

v(x) - v(,) _<

/?

e-a~le(~x, ~[b], b) - e(y~, ~[b], b)] at + ~,

and from now on the proof is exactly the same as that of Proposition 2.1 in Chapter III, in view of the assumption (1.4) and the estimate (0.6). The proof for the upper value is analogous. The next result is the cornerstone of this section.

1. DP FOR L O W E R AND U P P E R VALUES

437

THEOREM 1.9 (DYNAMIC PROGRAMMING PRINCIPLE). Assume (0.2), (0.5), and (1.4). Then for all x E ]~Y and t > O, (1.5)

v(x) = inf sup~ ,s0ft~(yx(s)'a[b](s)'b(s))e-aS ds + v(yx(t; c~[b],b))e -at }, acF bEB "

(1.6)

u(x) = sup inf ~ fO t g(yz(s), a(s), 13[a](s))e -As ds + u(yz(t; a, lg[a]))e -;~t }. 13EA aE.A k

PROOF. We give only the proof of (1.5), since the other is similar. Denote by w(x) the right-hand side of (1.5). We fix z > 0 and for any z E ]1{N we pick az E F such that

v(z) >_ sup J(z, c~z[b],b) - r

(1.7)

bEB

For simplicity we set ,k = 1. We first prove that v(x) <_ w(x). We choose ~ E F such that (1.8)

w(x ) >_ sup~ ,softt(yx'-5[b]'b)e-" ds + v(yx(t; -5[b], b) )e -t } - s . bE IZ ~"

Now we define 5 C F as follows /~[b](s), ~[b](s) := (~=[b](. + t ) ] ( s - t),

s <__t, s > t,

with z : : yx(t, ~[b], b). Note that

yx(s+t;5[b],b):yz(s;az[b](.

+t)],b(. +t))

for s > O,

so by the change of variables r = s + t

a(z,c~z[b](-+t)},b(. +t))=

e(yx(r),5[b](r),b(r))et-'dr.

Then by (1.8) and (1.7) we obtain w(~) > sup bEB

//

e(yx, ~[b], b)e -~ as - 2E _> v(~) - 2e,

which gives the desired inequality because e is arbitrary. To prove the inequality w(x) <_ v(x) we pick bl C B such that

(1.9)

w(x) <

/0't(y~, C~x[bl],

bl)e -s ds + v(y~(t; ax[bl], bl))e -t + ~,

where c~ is defined as in (1.7). Now for each b E B define b E B by (1.10)

b(s) : : [ b l ( s ) ,

(b(s-t),

S

< t,

s > t,

438

VIII. D I F F E R E N T I A L GAMES

and then define g E F by

(i.II)

~[b](~) := ~x[~](~ + t).

Next set (1.12)

z := yx(t;c~[bl],bl)

and choose b2 E B such that

v(z) < J(z, ~[b2], b2) + E.

(1.13) We claim that

w ( x ) < J(x, C~x[b2], b2) + 2s, which gives, by (1.7), w ( x ) < v(x) + 3~, and thus the conclusion by the arbitrariness of z > 0. To prove the claim observe that, by (1.10) and (1.11), - ~yx(T,a~[bl],bl), yx(r;ax[b2],b2) = [ y z ( r - t , ~ [ b 2 l , b 2 ) ,

r <_ t, T > t,

SO by the change of variables 7- = s + t

(1.14)

J(z, ~[b21, b2) =

t(yx(r), ax[b2](~-),g2(~-))e t-" d r .

Now we use (1.9), (1.10), (1.12), (1.13) and (1.14) to get w ( x ) <_

g(yx(T), ax~2](T),b2(v))e-" dt + 2z,

which is the claim.

4

We proceed as in the previous chapters and derive a partial differential equation for each value. For differential games the equations are called Hamilton-JacobiIsaacs (briefly, HJI) and they involve the following Hamiltonians (1.15)

H ( x , p) := minbeBmca~{ - f (x, a, b) . p - ~(x, a, b) },

(1.16)

~I(x,p) := max min{ - f ( x , a, b) . p - g(x, a, b) } . aEA bEB-

THEOREM 1.10. Assume (0.2), (0.5), and (1.4). viscosity solution of (1.17)

Av + H ( x , Dv) = 0

Then the lower value v is a

in RN;

and the upper value u is a viscosity solution of (1.18)

Au + ~r(x, Du) = 0

in N N .

1. DP FOR LOWER AND UPPER VALUES

439

The equation in (1.17) is called the lower H J I equation and t h a t in (1.18) is the upper H J I equation. We prove only that the lower value solves the lower HJI equation, the other proof being similar. For differential games, unlike the case of a single controller, proving t h a t v is a subsolution is not easier than proving it is a supersolution. Indeed, for the first part we need the following LEMMA 1.11. Under the assumptions of Theorem 1.10, let x E 11~N and r E C 1 (JRN) be such that (1.19)

,kr

+ H(x,Dr

) = 0 > O.

Then there exists c~* E F such that for all b E B and t > 0 small enough t

(1.20)

fo

{ f(yx, a*[b],b) + f ( y x , a*[bl,b) . DO(yx) - Ar

Ot }e-a* ds <_ - - ~ ,

where Yx = Y~ ( ' , a* [b], b). PROOF. Set for simplicity A = 1 and define A(z, a, b) := r

- g(z, a, b) - f ( z , a, b) . D e ( z ) .

By the definition of H and (1.19) m i n m a x A ( x , a, b) = ~), bEB aEA SO for any b E B there is a = a(b) E A such that a ( x , a, b) > tg. Since A(x, a, 9 ) is uniformly continuous, we have in fact A(x, a, ~) > __3t9 -

for all ~ E B(b, r) N B

4

for some r = r(b) > 0. Because B is compact, there exist finitely many points b l , . . . , b n E B and r l , . . . , r n > O such that B c 0 B(bi,ri), i=I and for ai = a(bi) A(x, ai, ~) > 3t9 -

for ~ E B ( b i , r i ) n B .

4

Define ~5 : B -+ A by setting ~(b) = ak

k-1 if b e B(bk, rk) ", U B(bi, r i ) . i=1

440

viii. DIFFERENTIAL GAMES

It is easy to prove that, for a n y b E B, ~5(b(. )) is measurable, so we can define a* E F by setting a*[b](s) : = ~(b(s)), see Example 1.2. By definition of 3v9 A(x, ~5(b), b) _> ~ -

for all b E B,

a n d by the continuity of A a n d estimate (0.7) there is t > 0 such that

A(yx(s), a*[bl(s), b(s)) > -~

(1.21)

for 0 < s < t

a n d for each b E B. Now we m u l t i p l y b o t h sides of (1.21) by e - s a n d integrate from 0 to t to o b t a i n (1.20) for t small enough. .,, PROOF OF THEOREM 1.10. We set A = 1 a n d we begin with the proof t h a t v is a subsolution of (1.17). Let r E C I ( R N) a n d x be such t h a t v - r has a local m a x i m u m at x a n d v(x) = r We assume by contradiction that (1.19) holds a n d use L e m m a 1.11 to find a* E F such t h a t for all b E B and all t small enough t

fo

(1.22)

g(Yx, a*[b], b)e -s ds + e-tr

Ot

- O(x) <_ ----~,

where Yx = Y z ( ' , a*[b], b). Since v - O has a local m a x i m u m at x a n d v(x) = O(x), by (0.7) we have

e-~O(ux(t)) - O(x) _> e-~v(ux(t)) - v(x) for t small enough. By plugging this into (1.22) we o b t a i n

" ~0t e(yx, a[b], b)e -~ ds + e-tv(yx(t, a[b], b)) } -

~erinfsuPtbE~

v(x) <_ --~-Ot< O,

which is a contradiction to the inequality ':<" in the D P P (1.5). T h e n v is a subsolution of (1.17). Next we show t h a t v is a supersolution of (1.17). Let O E C I ( N N ) , x be such t h a t v - O has a local m i n i m u m at x, a n d v(x) = O(x). Assume by contradiction O(x) + H(x, DO(x)) = - 0 < 0 . By the definition of H there exists b* E B such that r

- f(x, a, b*) . DO(x) - f(x, a, b*) < - 0

for all a E A. For t > 0 small enough a n d any a E F r

- f(yx(s), a[b*](s), b*) . DO(yx(s)) - e(yx(s), a[b*](s), b*) < --~

1. DP F O R L O W E R AND U P P E R VALUES

441

for 0 < s < t, where yx(s) = y~(s; a[b*], b*). We multiply by e -~ and integrate from 0 to t to get (1.23)

r

- f0 t g(y~, c~[b*],b*)e -~ ds <_ - - tgt ~ ,

- e-tr

for t small enough. From r

- e-t r

) > v(x) - e-tv(yx(t) )

we obtain v~t + v(x) e-tv(yx(t)) + f0 t e(y~, c~[b*], b*)e -s ds >_ -~ and thus inf sup~ fote(yx, a[b],b)e -s ds + v(y~(t;c@],b))e -t } > v ( x ) . c~EF bEN ~

This is a contradiction to the D P P (1.5) and completes the proof that v is a viscosity solution of (1.17). 9 Exercises

1.1. Define F* := {c~ : B --* A}, the set of all strategies for the first player (not necessarily nonanticipating). Prove that inf sup J(x, c~[b],b) = v~(x), aEF* bEB

where vs is defined by (0.3). 1.2. Replace the assumption that A and B are compact with the following: A and B are separable metric spaces; for all bounded X c IRN f is bounded on X x A x B; ]f(x,a,b) - f ( y , a , b ) l <_ w I ( l x - y I , R ) for all x , y E B(O,R), a E A, b E B and R > 0, where ws is a modulus (cfr. (A1) and (A2) in Chapter III). Redefine H and H by replacing min max with inf sup in (1.15) and max min with sup inf in (1.16). Prove that all the results of this section remain true in this more general setting. [Hint: in the proof of Lemma 1.11 use Lindel5ff's theorem, that is, any open cover of a separable metric space has a countable subcover.] 1.3. In analogy with the definitions of w of Chapter III we call presynthesis of strategies for the first player a map a : ]I~N ~ ~ ~ 1" and set Ja(x) := sup J(x, a(x)[b], b). bE6

We say a is optimal at x if Ja(x) = v(x). Prove that a necessary condition of optimality for a at all points of an open set f~l is that Ja is a viscosity solution of the lower HJI equation in f~l. Similarly, for a presynthesis b : f~ ---* B for the second player we set Jb(X) := infest J(x, a[b(x)], b(x)) and say it is optimal at x if Jb(X) = v(x). Find a necessary condition of optimality for b in an open set.

442

VIII. D I F F E R E N T I A L GAMES

1.4. Compute u(x), for x < 0, and vs, us, H, H in Example 1.7. 1.5. A strategy a such that, for any t > 0 and b,b E 13, b(s) = "b(s) for all s < t implies a[bl(s) = ~[b](s) for all s _< t, is called a slightly delayed strategy and it is clearly nonanticipating. (i)

Find which strategies in the Examples 1.2-1.4 are slightly delayed.

(ii) Define the lower and upper values Vd and ud by replacing F and A in Definition 1.6 with the corresponding sets of slightly delayed strategies, and prove they satisfy the Dynamic Programming Principle. (iii) Prove that Vd and Ud are supersolutions of (1.17) and subsolutions of (1.18). (iv) Compute Vd and Ud in Example 1.7. 1.6. Adopt all the assumptions of Exercise 1.2, except the first, and replace it by: A, B topological spaces. Prove that u and v are supersolutions of the lower HJI equation (1.17) and subsolutions of the upper HJI equation (1.18). 1.7. Assume w e C ( ~ N) satisfies for all x E ]Rx and small t > 0,

w(x) > inf sup~ f t g(yx(s), a[b](s), b(s))e - ~ ds + w(yx(t; c~[b],b))e -~t } a E F ~ b E B ' ~ Jo (respectively, _~), where 13' C_ 13 contains the constant control functions and F ~ _C F contains all the constant strategies of the type a[b](t) = ~ for all t, for some ~ E A. Prove that w is a supersolution of the lower HJI equation (1.17) (respectively, subsolution of upper HJI equation (1.18)). 2. E x i s t e n c e o f a v a l u e , r e l a x a t i o n ,

verification theorems

In this section we characterize the lower and upper values for the infinite horizon problem as the unique solution of the lower and the upper HJI equations, respectively. From this result it is easy to derive the announced inequality v < u, and the equality under the assumption H -- H, so that the game has a value. As a consequence, we show that any game has a value if both players use relaxed controls. Finally, we give verification theorems in the form of necessary and sufficient conditions of optimality for strategies and controls. THEOREM 2.1. Assume (0.2), (0.5), and (1.4). Then (i)

the lower value v is the complete solution (i.e., the minimal supersolution and maximal subsolution) of (1.17) in BC(]RN); in particular, it is the unique solution of (1.17) in BC(]RN);

(ii)

the upper value u is the complete solution of (1.18) in BC(I~N).

PROOF. By Proposition 1.8 and Theorem 1.10 it is enough to check the assumptions of a suitable comparison theorem for sub- and supersolutions of (1.17) and

2. E X I S T E N C E OF A VALUE, R E L A X A T I O N , V E R I F I C A T I O N

443

(1.18). We want to apply Theorem 2.12 and Remark 2.13 of Chapter III, so we have to prove that H and Lr satisfy the inequality (2.1)

g ( y , # ( x - y) - 7-y) - H ( x , # ( x - y) + 7x) <_ p L i x _ y]2 + 7 K ( 1 + ly]2) + 7K(1 + IxI9) + w ( I x - yl)

for some constants L, K and a modulus w, for all x, y E R N, p, T, 7 > 0. We fix U E B such that

H ( x , p ) >_ - f ( x , a , b ' )

. p - ~ . ( x , a , b ~)

for all a E A,

then we pick a ~ E A such that

H ( y , q) <_ - f ( y , a', b'). q - g(y, a', b'), where p = #(x - y) + 7x, q = #(x - y) - ~-y. The left-hand side of (2.1) is less than or equal to # ( f ( x , a', b') - f ( y , a', b')) . (z - y) + r f ( y , a', b') . y

+ ~/f(x, a', b'). x + g(z, a', b') - g(y, a',/)1), which can be estimated by the right-hand side of (2.1) where L is the constant in (0.5), w = we, and

K := L + K ' , In fact, ( f ( z , a , b ) - f ( O , a , b ) ) . z

K ' := m a x { If(O,a,b)l : a E A, b E B } . < Lizl 2 by (0.5), thus

f ( z , a , b ) . z < LIz] 2 + K ' l z ] < K(1 + Iz[2). The proof of (2.1) with H replaced by H is similar.

9

COROLLARY 2.2. A s s u m e (0.2), (0.5), and (1.4). Then

v(x) < u(x)

for all x 9 ~ N .

If, in addition, (2.2)

H(x,p) = H(x,p)

for all x , p 9 l~ N,

then v -= u and the game has a value for all initial points. PROOF. The Hamiltonians H and ~r defined by (1.15) and (1.16) are, respectively, minB maxA and maxA minB of the same function. Therefore

H(x,p) > H(x,p)

for all x , p

9 ]~N .

Thus v is a subsolution of the upper HJI equation (1.18) and u is a supersolution of the lower HJI equation (1.17). By either (i) or (ii) of Theorem 2.1 we conclude that v _< u. The second statement follows immediately from Theorem 2.1 because the two HJI equations (1.17) and (1.18) coincide if H = ~r. 9

444

VIII. D I F F E R E N T I A L GAMES

The sufficient condition (2.2) for the existence of a value is called Isaacs' condition or solvability of the small game. It is easy to give conditions on the d a t a A, B, f , and ~ implying (2.2). In fact, if we set

F(a, b) := - f ( x , a: b) . p - t(x, a, b), (2.2) says that F has the property min max F(a, b) = max min F(a, b) .

(2.3)

bEB aEA

aEA bEB

The simplest case when this occurs is for any system and cost with the separation property

f(x,a,b)=

fl(x,a)+

f2(x,b),

~(x,a,b)=~l(x,a)+~2(x,b),

which is often verified in the applications. More generally, (2.3) is equivalent to saying that the static (two-person, zero sum) game over the sets A and B with payoff F has a saddle. This is one of the basic problems in classical game theory and it has been studied in depth after the first pioneering result of yon Neumann. The minimax theorem we report without proof is not the most general in the literature but it is sufficient for giving two more examples of games with the Isaacs' condition. THEOREM 2.3. Assume F := A x B ~ R is continuous, A, B are convex compact spaces, the sets { b E B : F(-d, b) > t } and { a E A : F(a, "b) < t } are convex for all t E ~, ~ E A, -b E B. Then (2.3) holds. EXAMPLE 2.4. Let the control sets be finite dimensional, A _c ~m, B C_ R k, the system be affine in the control variables, that is,

f ( x , a, b) = f l ( x ) + f2(x)a + fa(x)b, where f2(x) is an N x m matrix and f3(x) is an N x k matrix, and the cost be bilinear in the control variables, that is, g(x, a, b) = f l ( x ) + a . e2(x)b, where g2(x) is an m x k matrix. Then it is straightforward to check that Theorem 2.3 applies for all x , p E NN, so that H = ~r and the game has a value. More generally, the same holds if g(x, 9 b) is convex for all x, b, and t~(x, a, 9 ) is concave for all x, a. <3

Next we consider relaxed controls as we did in w of Chapter III for one-person problems. As we proved there, for those problems the value does not change using relaxed controls, and the advantage in using them is the existence of an optimal control which we proved in w of Chapter VI. For two-person games we show that the value always exists if we use relaxed controls for b o t h players. In particular, if the original game does not have a value, then either the lower or the upper value must be different from the relaxed value.

2. EXISTENCEOF A VALUE,RELAXATION,VERIFICATION

445

We denote by A r (respectively, B r) the set of Radon probability measures on A (respectively, B) and define, for x E ~N, m E A r, n E B r,

fr(x,m,n):=/B/Af(x,a,b)drndn,

gr(x,m~n):-=/B~Ag(x,a,b)dmdn,

where the order of integration is irrelevant by Fubini's Theorem. Note that if(x, 5~, (fb) ----f(x, a, b), where 5c is the Dirac measure concentrated at c. A similar formula holds for g, so that f r and gr can be viewed as extensions of f and g, respectively. As in Section III.2.2, A r and B r are viewed as subsets of the dual spaces of C(A) and C(B), respectively, with the weak star topology. LEMMA 2.5. (0.2), (0.5),

If A, B , f , g satisfy and (1.4) as well.

(0.2), (0.5),

and

(1.4),

then Ar, Br, fr,g r satisfy

PROOF. The compactness of A r and B r with the weak star topology is a classical result in functional analysis; see [Wa72]. The other properties are straightforward to check, as in the proof of Lemma III.2.20. 4 By this lemma there exists a unique trajectory of the relaxed system, that is, a solution of y'(t) = f ( y ( t ) , ~(t), ~(t)),

t > 0,

y(0) = x, for every # E ,4 ~, u E B r := { measurable functions [0, +~c[ ---* B r }, and we denote it by yz(t) = yx(t; #, ~'). Then we can extend the cost functional to relaxed controls:

Jr(x, Ia, u)

:=

//

e-~tgr(yx(t), p(t), u(t)) dt.

Now we can consider the lower and upper value functions corresponding to the following three games: (i) only the first player can use relaxed controls, and the lower and upper value are denoted by v~ and u[4; (ii) only the second player can use relaxed controls, and the values are v~, u~; (iii) both players can use relaxed controls and the values are vr, u r. The next result compares these six values and the original v and u. Note that some inequalities are obvious, for instance v < v~,

(2.4) THEOREM 2.6. (2.5) (2.6)

In particular,

Assume

(0.2), (0.5),

urA < U.

and

(1.4).

Then

v = v~, u -- u ~ , v~ = v r = u r = u[4.

446

VIII. DIFFERENTIAL GAMES

(a)

the game where both players can use relaxed controls has a value;

(b)

if the original game has a value, then it coincides with the relaxed value v r.

PROOF. To prove the first formula in (2.5) we observe that for all b E B max{ - i f ( x , m, b) .p - er(x. m, b) } = max{ - f ( x , a, b) .p - f.(x, a, b) }

mEA r

"

aEA

by the proof of Corollary III.2.21. Then

HrA(x,p) := min max{ --if(x, m, b) .p - er(x, m, b) } = H ( x , p ) , bEB rnEA r-

so b o t h v and v~ are solutions of the lower HJI equation (1.17), by Theorem 1.10. Thus they coincide by the uniqueness Theorem 2.1. The proof of the second formula in (2.5) is similar, by using the upper HJI equation (1.18). Next we prove that the game where both players use relaxed controls has a value. By Corollary 2.2 it is enough to show that this game satisfies the Isaacs' condition, namely, t h a t the following upper and lower Hamiltonians coincide,

Hr(x, p ) : = min max F ( m , n),

H r ( x , p ) := max min F ( m , n),

nEB r mEA r

rnEA r nEB r

where F ( m , n) : = - f r ( x , m, n ) . p - gr(x, m, n). We claim that the assumptions of the minimax Theorem 2.3 hold true. In fact, F is continuous and A r, B r are compact by Lemma 2.5. Moreover, it is straightforward to check that A r, B r are convex and F is bilinear. By Theorem 2.3 we get H r = ~ r and then v r = u r. The identity v~ = v r is obtained by applying the first formula in (2.5) to the game on A and B r. Similarly u~ = u r follows from the second formula in (2.5) applied to the game on A r and B. The last conclusion (b) (i.e., v = u implies v = v r) follows immediately from (2.4) and (2.6). ~1

Before giving a verification theorem for the lower value we need some definitions. DEFINITION 2.7. A strategy a* E F is optimal at the initial point x if

v(x) = supJ(x,c~*[b],b).

<~

bet3

DEFINITION 2.8. A control b* E B is optimal at x if

v(x) = inf J(x,a[b*],b*). aEF

Note that if c~* and b* are optimal at x, then

J ( x , a *[b],b) < v(x) = J(x,a*[b*],b*) <_ g(x,a[b*],b*)

2. EXISTENCE OF A VALUE, RELAXATION, VERIFICATION

447

for all b E B and a E F. This means that the first (respectively, second) player cannot improve his guaranteed outcome, represented by v(x), by a unilateral deviation from his optimal strategy (respectively, control). Thus the pair (a*, b*) is a Nash equilibrium, in the terminology of the general theory of noncooperative games. <~ PROPOSITION 2.9. Assume (0.2), (0.5), and (1.4). Then the following statements

are equivalent: (i)

a* E F is optimal at xo;

(ii)

there exists u E BUC(IR N) subsolution of the lower HJI equation (1.17) such that (2.7)

sup a(x0, a*[b], b) _< u(x0); bEl3

(iii)

there exists u E BUSC(~ N) subsolution of (1.17) satisfying (2.7).

Also, the following statements are equivalent: (j)

b* E 13 is optimal at xo;

(jj)

there exists w E BUC(I~ N) supersolution of (1.17) such that (2.8)

w(xo) <_ inf J(x0, Rib*], b*); ~EF

(jjj)

there exists w E BLSC(R N) supersolution of (1.17) satisfying (2.8).

PROOF. (i) implies (ii) by taking u = v, the lower value function. Obviously (ii) implies (iii). Now we assume (iii) and recall the comparison principle for semicontinuous sub- and supersolutions in Chapter V (see Definition V.1.2). By the proof of Theorem 2.1 the Hamiltonian H of the lower HJI equation has enough regularity to apply Theorem V.1.3 and get u(x) <_ v(x) for all x. Thus by (2.7) and the definition of v we obtain that a* is optimal at x0. The proof of the equivalence between (i), (jj) and (jjj) is analogous. 9 We see from this result that viscosity subsolutions of the lower HJI equation can be used as verification functions for the optimality of strategies of the first player, whereas supersolutions are verification functions for controls of the second player. It is obvious how to define optimality for strategies/3 E A of the second player and controls a E ,4 of the first. It is easy to give a verification theorem for them similar to Proposition 2.9, by using super- and subsolutions of the upper HJI equation (1.18) as verification functions. Exercises

2.1. Consider the infinite horizon differential game with N -- 4, A -- 1, A = B =

{-1,1}, f ( x ) = (--ax2, a x l , -bx4, bx3), g(x) = arctan((yl - y3) 2 + (Y2 - Y4)2) .

448 (i)

viii. DIFFERENTIALGAMES Prove that u(x) = v(x) = $(x) for all x;

(ii) find some x such that vs(x) < v(x) < us(x), where vs and us are defined by

(0.3). 2.2. Prove that a function F : A x B --* ~ satisfies (2.3) if and only if it has a saddle, namely, a pair a*, b* such that F(a, b*) < F(a*, b*) < F(a*, b) for all a E A, b E B . 2.3. State and prove a verification theorem for strategies/3 E A and controls a E `4. 2.4. Assume that the static game over .4 a n d / 3 with payoff J ( x , ., . ) has a saddle (a*, b*). Prove that the strategy a*[b] = a*, for all b, and the control b* are optimal at x according to Definitions 2.7 and 2.8. 2.5. Compute all the relaxed upper and lower value functions for Example 1.7. Show in particular that vr = u. 2.6. Consider the lower and upper values Vd, Ud defined in Exercise 1.5 using the slightly delayed strategies. Prove that (i)

V < Vd <_ u and v <_ ud <__u;

(ii) v = va = Ud if Isaacs' condition (2.2) holds. 2.7. Assume .4~ c .4, B ~ C/3 are closed with respect to translation in time (a E .4, r > 0 implies a ( . + r) E .4') and concatenation (al,a2 C .4', r > 0, a(t) = al(t) for t <_ r, a(t) = a~(t - r) for t > r, implies a E A'), as, for instance, piecewise constant controls. Replace `4, /3 by `4~, ~ , respectively, in the definition of nonanticipating strategy and of lower and upper value, and call v~ and u ~ the new values. Prove that v ~ = v and u ~ = u.

3. Comparison w i t h other information patterns and other notions of value 3.1. Feedback strategies In Section 1 and 2 we used the notion of nonanticipating strategy which models a situation where a player knows the current and past choices of the control made by his or her opponent. This is not the case in many practical examples where a player knows at most the current state of the system, and perhaps can keep track of the past history of the state. To model this different information pattern we introduce the feedback strategies. We denote by Y the set of trajectories of the system, that is, y :-- { y : [0,+*r

•N: 3x E R Y , a E `4, b E Bsuch that y(t) = y x ( t , a , b ) f o r all t } .

3. OTHER INFORMATION PATTERNS

449

DEFINITION 3.1. A m a p r : y ~ A is a feedback (or closed-loop) strategy (with perfect memory and perfect state measurement) for the first player if it is nonanticipating (or causal) (i.e., for any t > 0 and y, ~ E Y, y(s) = ~(s) for all s _< t implies ~[y](s) = ~[y~(s) for all s < t), and playable, that is, for any x E ]~N, b: [0, T[ ~ B measurable and T E ]0, +c~] there exists a unique solution of (3.1)

Sq'(t ) ----f(q(t),~[q](t),b(t)), tq(0)

0 < t < T,

= x.

We denote this solution by Yx( 9 ; r b) (with a slight abuse of notation) and the corresponding control function ~[q] by ax('; ~, b). A feedback strategy ~ for the second player is defined in the obvious symmetric way, the trajectory and the control function corresponding to ~, x, and a E .4 are denoted, respectively, by Yx( 9 ; a, ~) and ~x( 9 ; a, ~). We denote by 5v and ~, respectively, the set of feedback strategies for the first and the second player. ,3 The most trivial example of feedback strategy is a constant one: for a given E .4, ~[y] =- ~ for all y E Y. Here are some more interesting examples. EXAMPLE 3.2. Let 4) : ]~g X [0, OO[ ---+ A be a time-dependent feedback map, admissible for every x E ]~g as defined in Example 1.3. We can associate with 4) a feedback strategy by setting

r

:= 4)(y(t)~ t),

provided ~[y] is measurable for all y E Y. Note that this strategy is memoryless in the following sense: for any t > 0 and y, ~ E Y, y(s) = ~(s) for all s _> t implies ~[y](s) -- C[y-](s) for all s _> t. <3 EXAMPLE 3.3. Let 4) be a feedback m a p acting on some average of the history of the state, as in Example 1.4. We can associate with 4) a feedback strategy by setting

r

:= 4)

(/0'

y(s) dut,

provided ([y] is measurable for all y E Y and there is a unique solution of (1.1) for every b E B, T E ]0, +cc] and x E ]~.N. <3 EXAMPLE 3.4. Given a m a p 4) : ~ N _.~ A and an increasing sequence of times { tn } such that to = 0 and limn tn = +oo, we define a feedback strategy as follows, ([y](t) = 4)(y(tn))

if tn _< t < t n + l .

We call such a strategy sampled-data, but it is also named K - s t r a t e g y in Isaacs' terminology. Note that in this case the trajectory y = ( - ; r exists unique for all b E B and x E 11~N, and it generates a piecewise constant control function ~x ( "; ~, b), with no assumption whatsoever on 4). <~

450

VIII. D I F F E R E N T I A L GAMES

The next lemma gives the connection between feedback strategies and nonanticipating strategies in the sense of Definition 1.1. LEMMA 3.5. For any given ~ E ,T and x E ]~g, the strategy a : 13 ---, A, oz[b] = as(. ; ~, b) is nonanticipating and Yx(', a[b], b) = y~(. ; ~, b) for all b E 13. PROOF. If b,b e B, b(s) = "b(s) for all s < T, then yx(S; ~, b) = yx(s; ~,'b) for all s < T because of the uniqueness requirement in Definition 3.1. Thus a~(s; ~, b) = C~x(S;~,b) for all s < T because ~ is nonanticipating. Therefore a is slightly delayed (see Exercise 1.5) and, afortiori, it is nonanticipating. 9 Now we define the feedback values and compare them to the values of Definition 1.6. DEFINITION 3.6. The A-feedback value of a game with cost functional 3 : ]RN • Ax/3~Ris

vA(x) := inf supJ(x, ax(. ;~;b),b), ~Et bEl3

and the B-feedback value is

VB(X) := sup inf 3(x, a, 13x( .; a, ~)). ~,E~ aEA

If VA(X) = VB(X) we say that the game with initial point x has a feedback value, which is denoted by vF(x). <~ By Lemma 3.5 we immediately obtain (3.2)

v(x) ~ VA(X )

and

VB(X ) < u(x)

for all x,

where v and u are the usual lower and upper values of the game (Definition 1.6). Note that, in general, there are nonanticipating strategies t h a t do not correspond to a feedback strategy, for instance, the second in Example 1.2. Therefore the inequalities in (3.2) might be strict at some point. In the rest of this subsection we study the infinite horizon problem, that is, the cost functional is given by (0.4). The main result is that under Isaacs' condition the feedback value exists and coincides with the usual value. We introduce the notation

Jr(x; ~, b) := J~(x; a, ~) :--

I' I'

e(yx(s; r b), c~x(s; r b), b(s))e -~8 ds,

e(~(s; a, ~), a(s), ~ ( s ; a, ~))e -~s as,

for t E [0,+oc], x E I R N, ~ E ~ ' , b E B , ~ E ~ , a E A . subsection the A-feedback and B-feedback values are

VA(X) : = inf sup 3 ~ ( x ; ~, b), CEY bEB

VB(X) : =

sup inf

J=(x; a, ~).

Thus in the rest of this

3. OTHER INFORMATION PATTERNS

451

PROPOSITION 3.7. Assume (0.2), (0.5), and (1.4). Then VA, VB E BUC(]RN). If,

moreover, w~(r) = L~r, then VA and VB are H6lder continuous with the exponent given in Proposition 1.8. PROOF. It is essentially the same as that of Proposition 1.8.

4

As in Section 1 the D P P is the cornerstone of our theory. THEOREM 3.8 (DYNAMIC PROGRAMMING PRINCIPLE). Assume (0.2), (0.5), and

(1.4). Then for all x E ]~N and t > 0 (3.3)

VA(X) = inf sup{ Jt(x; ~, b) + vA(Yx(t; (, b))e -~t },

(3.4)

vB(x) = sup inf { Jr(x; a, ~) + vB(yz(t; a~ ~) )e -)'t } .

(E~" bEB ~E6 aEA

PROOF. We prove only (3.3) for A = 1. We follow the scheme of the proof of Theorem 1.9 and fix z > 0, z E ]RN, and (~ E .T" such that

VA(Z) >_ sup J ~ ( z ; (z~ b) - s .

(3.5)

bEB

We call w(x) the right-hand side of (3.3) and begin with the proof that VA(X ) w(x). We choose ( E ~- such that (3.6)

w(x) > sup{ Jr(x; ~, b) + vA(yx(t; (, b) )e -t } - e, bEB

and we define

~[y](s), ~[Y](s) := (r

+ t)](s - t),

s _< t, ~ > t,

for any y E Y. We claim that a E 5v. Clearly a is nonanticipating, and it is not hard to check that the trajectory associated with a exists unique and is given by

y~(s; cr, b)

fyx(~;(,b),

[yz(s-t;r

s < t,

+t)),

~>t~

where

z := yx(t; (, b), for a n y b E B a n d x E I R from (3.5) and (3.6)

N. Proceeding as in the proof of Theorem 1.9 we deduce

w(x) >_sup J~c(x; a, b) - 2e ~ VA(X) -- 2e, bEB

which gives the desired inequality.

452

viii. DIFFERENTIALGAMES

To prove the inequality

(3.7)

w(x) < VA(X) we pick bl E B such that

w(x) <_ Jr(x; ~, bl) + VA(Z)e -t + ~,

where z :=

y=(t; ~=, bl) .

Now for any y E Y such that y(O) = z we define ~" E Y by

[yx(S;~x,bl), ~(s) := LY( s _ t),

s <_ t, ~ > t,

then we fix any g E ~4 and define r ~

Sr

+ t),

[~(s),

It is not hard to see t h a t ~ E ~- and

(3.8)

ify(O)=z, otherwise.

yp(s; ~, b) = yp(s; "5, b) if p r z, and

y~(~;~b)=yx(t+s;r

for ~ > 0 ,

where for any b E B

~bl(s), ~(s) := l b ( s -

s <_t, t),

s > t.

The rest of the proof is essentially the same as that of Theorem 1.9: we choose b2 E B such that

VA(Z) <_ goc(z; ~,b2) + E, and use (3.7) and (3.8) to obtain, after some calculations,

w(x) < J ~ ( x ; (x,b2) + 2~. Then by (3.5)

w(x) < vA(x) + 3e, which gives the desired inequality by the arbitrariness of E. REMARK 3.9. It would be desirable to be able to prove the D P P for the value function defined by means of some subset of feedback strategies, such as strategies generated by feedback maps (see Example 3.2), or sampled-data strategies (see Example 3.4). A set of strategies smaller than 5v for which the previous proof works is the set of feedback strategies with finite memory. These strategies are those ~ E 5r with the property that for all y E Y there exists a set of "times to keep in memory" S((,y) C_ [0, +oc[ such that for any t > 0 S(~,y) n [O,t] is finite, and

3. OTHER INFORMATIONPATTERNS

453

E Y, ~(s) = y(s) for all s _> t and all s E S(~,y) implies ([y-'](s) = ~[y](s) for all s > t. Clearly, a memoryless strategy has finite memory (take S(~, y) = ~ for any y). Thus the strategies of Example 3.2 have finite memory. Also the sampled-data strategies of Example 3.4 have finite memory, by taking S(~, y) = { t~ : n E N } for any y E Y. On the other hand, the strategies of Example 3.3 do not have finite memory if, for instance, the probability pt has a density for some t. <~ COROLLARY 3.10. Assume (0.2), (0.5), and (1.4). Then VA and VB are supersolutions of the lower HJI equation (1.17) and subsolutions of the upper HJI equation (1.18). PROOF. We set ~ = 1 and we begin with the proof that VA is a supersolution of (1.17), which is essentially the same as the proof that v is a supersolution in Theorem 1.10. In fact, after fixing h E Ci(]R N) and a local m i n i m u m point x of VA -- h such that VA(X) = h(x), we assume by contradiction h(x) + H(x, Dh(x)) = - 0 < 0, and obtain from the definition of H the existence of b* E B such that

~(x) - f ( x , a , b * ) . Oh(x) - g ( x , a , b * ) <_ - 0

for all a E A .

Then for t > 0 small enough and any ( E 5r h(yx(s)) - f (yx( s), as(s), b*) . Dh(y~( s) ) - t(y~( s), C~x(S), b*) <_ --~ for 0 < s < t, where yz(s) := yx(s;~,b*) and a s ( s ) : = az(s;~,b*). Now we perform exactly the same calculations as in the proof of Theorem 1.10 and reach a contradiction to the inequality ">__" in the D P P (3.3). To prove that VA is a subsolution of (1.18) we fix h E C I ( R N) and x such that VA -- h has a local maximum at x and VA(X) = h(x). We assume by contradiction h(x) + -~(x, Oh(x)) = 0 > 0 and use the definition of H, (1.16), to find a* E A such that h(x) - f ( x , a*, b). Oh(x) - g(x, a*, b) _> ~

for all b E B .

Now we define the constant strategy (* E ~" as ~*[y](s) = a*

for all y E Y and s > 0 .

For t > 0 small enough and any b E/3 we have 0

h(yx(s)) - f(yx(s), a*, b(s)) . Dh(yx(s)) - g(yx(s), a*, b(s)) >_ -~ for 0 < s < t, where yx(s) := yz(s;~*,b). We multiply both sides by e - s and integrate from 0 to t to obtain an inequality such as (1.20), with a*[b] replaced by a*. So we can repeat the first part of the proof of Theorem 1.10 and get a contradiction to the inequality "<" in the D P P (3.3). This completes the proof of the statements about VA. The proofs concerning VB are similar and we leave them to the reader.

454

VIII. DIFFERENTIAL GAMES

Here is the main result of this subsection. THEOREM 3.11. A s s u m e (0.2), (0.5), and (1.4). Then (3.9)

V
V
In particular, if H = ~I, then VA = VB = v = u and the game has a feedback value for all initial points.

PROOF. Some of the inequalities in (3.9) are already known, see (3.2). All of them follow from the comparison theorem for HJI equations, Theorem 2.1, from Corollary 3.10 and the continuity of VA and VB, Proposition 3.7. If Isaacs' condition holds, then v = u by Corollary 2.2, and (3.9) then implies V A : Y : V B . 4 REMARK 3.12. This result applies for instance to the game where both players use relaxed controls: by Theorem 2.6 this game has a value, thus it also has a feedback value and they coincide. <1 REMARK 3.13. Theorem 3.11 remains valid if in the definition of VA and vB we use only the sets of feedback strategies with finite memory instead of the whole .T and G. In fact, by Remark 3.9 the D P P still holds in this case, and it is immediate to check that the proofs of Corollary 3.10 and Theorem 3.11 work as well.
As in Section 2 we say that a strategy ~* E ~ (respectively, ~* E G) is optimal at the initial point x if (3.10)

VA(X) = supJ~c(x;~*,b)

(resp., VB(X) = inf J~c(x;a,~*)),

bEI3

aEA

and we can easily prove a verification theorem similar to Proposition 2.9(a) using subsolutions of the lower HJI equation as verification functions. If one has a pair of optimal strategies ~* E .T, ~* c G, a natural question to ask is whether they can be played one against the other and form a saddle point for a game with payoff J~r over some pair of sets of strategies ~ ' C ~ and G' _c G. To make this precise we say t h a t 9c ' and G' are compatible if for any x E ] ~ N ~ E ~ / and ~ E ~', there exists a unique solution of (3.11)

{

q'=f(q,~[q],~[q]),

t>O,

q(O)=x,

and define

J(x; ;, ~) := J(x, ;[q], ~[q]). Now we can give the answer to the previous question.

3. O T H E R I N F O R M A T I O N PATTERNS

455

PROPOSITION 3.14. Assume x 9 ~1v is a point where the game has a feedback value VF(X) := VA(X) = VB(X) and the strategies ~* E .T and ~* E 6 are optimal. If ~* 9 7 G 3v and ~* E 6' C 6 with 3rl and 6' compatible, then

vg(x) = J(x; ~*, ~*) = max min J ( x ; r ~)

(3.12)

= min m a x J ( x ; ~, ~). r v' ~E~' PROOF. The existence of the feedback value and the optimality of ~*, ~* give

J~c(x;r

forallaEA, b 9

Plug into this inequality first b = ~* [q*], then a = ~* [q*], where q* is the solution of (3.11) corresponding to ~* and ~*, and get

Joc(x;~*,b)<__J(x;r

forallaEA, bEl3.

This proves the first equality in (3.12). Now take any ~ 9 ~ ' and plug a = ~[~] into the last inequality, where ~ is the solution of (3.11) corresponding to ~ and ~*, to obtain J(x; ~*, ~*) _< J(x; ~, ~*)

for all ~ 9 ~-'.

J(z,r ,~) < J ( z ; C , C )

for all ~ 9 6',

Similarly we get

and we reach the conclusion by an easy standard argument (see Exercise 2.2).

9

An example of compatible sets .TI, 61 is the sampled-data feedback strategies of Example 3.4. With this choice the term on the right-hand side of the last equality in (3.12) is the value function used in the pioneering book of Ismucs. Many authors define the value as the payoff J computed at a saddle point over a suitable pair of sets of compatible strategies. Any such definition is equivalent to the feedback value by Proposition 3.14 and the following converse. PROPOSITION 3.15. Assume ~t C .T and 6' C 6 are compatible and contain the constant strategies. Suppose ~* E .Y' and ~* E 6 r are such that for some x E IRN, (3.13)

-J(x,9~ *, ~) < -J(z,9~ *, ~*) < J(x; ~, ~*)

for all ~ E .T 1, ~ 9 6 ' .

If the feedback value exists at x, then (3.12) holds and ~*, ~* are optimal at x, that is, (3.10) holds. PROOF. Since 5r ' contains the constant strategies, (3.13) implies

J ( x , ~ ,~*) < inf Joc(x;a,~*) < VB(X), aEA

and similarly m

*

*

VA(X) < sup Jo~(x,~ ,b) < J(x;~ ,~ ). bEB

Then all the conclusions follow from the existence of the feedback value, i.e., vA(x) = vB(~) =: v r ( ~ ) .

9

456

VIII. D I F F E R E N T I A L GAMES

See Exercises 3.4 and 3.5 for extensions of Propositions 3.14 and 3.15 where the optimal strategies are replaced by optimizing sequences of strategies.

Some authors consider different notions of strategies that can be played one against the other by taking uniform limits of trajectories, usually called motions of the game generated by a pair of strategies. This is the case in the theory of positional differential games by Krasovskii and Subbotin and in Berkovitz's theory. Following these approaches one can define a new lower value ~ and a new upper value ~ as, respectively, the sup inf and the inf sup over the same pair of the sets of strategies. Therefore one automatically has the inequality ~ <_ ~, and if the new value exists (i.e., ~ = ~), it makes sense looking for a saddle point and indeed the existence of saddle points can be proved under suitable assumptions. On the other hand, given a pair of strategies, the motion and payoff generated by them may not be unique. The description of these theories goes beyond the scope of this book. However, we explain briefly how they can be connected to the formulations of differential games presented in this chapter. This is done by proving that ~ and satisfy the following weak super- and suboptimality principles, respectively, (3.14)

~(x) > sup inf bEB #E.A r ~

(3.15)

f' r(yx(s)

~(x) _< inf s u p ~ f t g r ( y x ( s ) , a , ~ , ( s ) ) e - ~ d s + ~ ( y x ( t ; a , , ) ) e - ~ t ~ , a E A u E B r ~" j0 j

for all 0 < t < to, where A r and B r denote the sets of relaxed control functions and fr the relaxed running cost. The next result says that any two functions with these properties coincide with the value of the game. THEOREM 3.16. Assume (0.2), (0.5), and (1.4). Let ~,~ C BC(R N) be such that ~_ ~ and satisfy, respectively, (3.14) and (3.15). If the game has a value (i.e., v = u, e.g., / f H = H), then v = u = v. PROOF. We claim that ~ is a supersolution of the lower HJI equation (1.17) and is a subsolution of the upper HJI equation (1.18). Then by Theorem 2.1 v < and ~ <_ u and we get the conclusion. To prove the claim we first observe, as in the proof of Theorem 2.6, that the Hamiltonian H coincides with the Hamiltonian H~ of the lower HJI equation corresponding to the game where the first player uses relaxed controls and the second player standard controls. With this remark the proof of the claim is essentially the same as that of Corollary 3.10, with (3.14) and (3.15) replacing the D P P (3.3). 9 REMARK 3.17. In the theory of positional differential games inequalities of the form of the weak super- and suboptimality principles (3.14) (3.15) are called, respectively, stability of ~ (respectively, ~) with respect to the first (respectively, second) player, briefly a-stability (respectively, b-stability). The motivation of this

3. OTHER INFORMATIONPATTERNS

457

name can be explained by considering the corresponding inequalities for games with finite horizon and null running cost and interest rate, that is, with payoff

J(x, t, a, b) : = g(yx(t; a, b)) for a given g E c ( R N ) . Then the weak superoptimality principle is g(x, t) >_ sup inf g(yx(s; #, b), t - s)

b6B p6.Ar

for all 0 < s _< t, which is usually proved by considering any level r e ]I( and constructing for any b C B and (x, t) in the sublevel set of ~',

s(r) := { (x,t): ~(x,t) < r }, a # 6 , 4 r such t h a t (yx(s; #, b), t - s) 6 S(r). This says t h a t the first player can keep the system in the set S(r) for any choice of a constant control of second player. In the terminology of Krasovskii and Subbotin the set S ( r ) is an a-stable bridge. Similarly, the weak suboptimality principle for ~ is related to the b-stability of the superlevel sets of ~. <~ 3.2. A p p r o x i m a t i o n

by discrete time games

In this subsection we consider some discrete time games associated with the infinite horizon differential game by a s t a n d a r d approximation scheme with step h. The main result is the convergence of the values of a m a j o r a n t and a minorant discrete time game to the upper and lower value defined in w as h tends to 0. It is the extension to two-person games of Theorem 1.1 of Chapter VI. The interest of this result is threefold. As for problems with one player in w of Chapter VI, the discrete time games have an optimal strategy and an optimal control in feedback form t h a t can be synthesized from the solution of the (discrete) Dynamic Programming equation, so we have approximated optimal feedbacks for the differential game. Moreover, the discrete time game is the first step of a fully discrete approximation scheme suitable for the numerical solution of the differential game: see Appendix A. Finally, we recall t h a t limits of values of discrete time games were used for quite a while to define the value of a differential game, so the convergence result gives the equivalence of these definitions with the one of w We denote by A N and B l~ the sets of sequences taking values, respectively, in A and B and consider for { a n } = a E A N, { b n } = b E B N,and h>0thediscrete time system

y(k + 1) = y(k) + hf(y(k), ak, bk),

k e N,

y(o) = x,

whose trajectory is denoted by yh ( k ) ----yh ( k; x, a, b). The payoff functional is OC

Jh(x, a, b) := --~ he(yh(k), ak, bk)a k, k=0

a

:=

1 -

),h.

458

VIII. D I F F E R E N T I A L GAMES

The fair discrete time game has the following rules: at each step k the first player chooses ak and the second player chooses bk simultaneously; both know all previous moves and in particular the current state of the game yh(k). In the minorant game the rules are the same except that the first player knows bk before choosing ak, that is, the second player must commit himself first at each move. In the majorant game the advantage in the information p a t t e r n is reversed and goes to the second player because the first must commit himself first at each move. The value of the minorant game vh(x) can be formalized by defining a nonanticipating discrete strategy for the first player as a mapping a : B N --~ A N such that, for any k E N, bj = bj for all j <_ k implies a[b]j = a[b]j for all j <_k, and setting

vh(x) := inf sup Jh(x, ~[b], b), c~EA bE B ~

where A denotes the set of nonanticipating discrete strategies for the first player. Similarly, the value of the m a j o r a n t game Uh(X) is defined as

Uh(X) := sup inf Jh(x,a,~3[a]), BEe a E A N

where O denotes the set of nonanticipating strategies for the second player. Clearly, any reasonable notion of value for the fair game should belong to the interval [Vh(X), Uh(X)]. Next we characterize Vh and Uh by means of the lower and upper

Dynamic Programming equations (3.16)

Vh(X) = m a x m i n { aVh(X + hf(x, a. b)) + he(x, a, b) },

(3.17)

uh(x)=minmax{~uh(x+hf(x,a.b))+hg(x,a,b)}

bEB aEA

aEA bEB

9

which are discrete versions of the lower and upper HJI equations (1.17) and (1.18). THEOREM 3.18. Assume (0.2), (1.4), h E ]0, l/,k[, and that for some L, (3.18)

If(z,a,b)-f(y,a,b)l<_Lix-Y]

forallx, y E ~ N , a E d , b E B .

Then the value function vh (respectively, Uh) of the minorant (respectively, majorant) game is the unique solution in BUC(~ N) of the lower (respectively, upper) DP equation (3.16) (respectively, (3.17)). PROOF. We begin with the proof that vh E BUC(]~N). The estimate Ivh(x)] < M / ( 1 - 0) is an immediate consequence of (1.4). We fix x, z E]R N, ~ > 0 and pick E A such that

Vh(Z) >_ sup dh(z,~[b],b)- E/6, bEB N

then b E B N such that

Vh(X) < Jh(x,~[~],~) + ~/6.

3. O T H E R I N F O R M A T I O N P A T T E R N S

459

Thus

Vh(X) -- Vh(Z) <_ Jh(x,-~[b], b) - Jh(Z,~[b],b) + z/3, and from now on the proof is exactly the same as that of Proposition VI.4.4. Next we prove that vh satisfies (3.16). Call w(x) the right-hand side of (3.16). Observe that the rain in (3.16) exists because Vh, f, and ~ are continuous and A is compact. Moreover, it is easy to deduce from (0.2) and (1.4) that the function b ~-* minaeA { . . . } is continuous, so the max also exists because B is compact. We fix z > 0 and for any z E R g we pick az E A such that

Vh(Z) >_ sup Jh(z, O~z[b],b) - ~.

(3.19)

bEB N

We first prove that va(x) _< w(x). By definition of w, for all b E B there exists ~(b) E A such that

w(x) >_ r

(3.20)

+ hf(x,~(b),b) + hf(x,~(b),b)) .

We define 5 E A as follows:

(5[b])0 := ~(b0),

(5[bJ)k:= (az [{ bn+l }])k-1

if k > 1,

with z := x + h/(x,5(bo), bo),

(3.21)

where { bn+l } E B N denotes the sequence n ~-* bn+l. Note that, if we set z(n) = yh(n; z, az [{ bn+l }1, { b,~+l }), we have oc

dh(X, 5[b], b) = he(x,~(bo), bo) + E he(z(n - 1), (5[b])~, b,,)G" n=l

= hi(x, "d(bo), bo) + o',]h(z, az [{ bn+l }], { bn+l })Thus, by (3.19),

Vh(X) <_ sup dh(x, 6[b], b) < sup { hf(x,-~(bo), bo) + O'Vh(Z) + bEB N

aE

boEB

and we obtain vh(x) <_ w(x) by (3.20), (3.21) and the arbitrariness of z. To prove the inequality w(x) < vh(x) we pick b E B such that (3.22)

w(x) = min { aVh(X + hf(x, a,-b)) + h~(x, a,-b) } aEA

and define for any b E B N

{~, bk:=

bk-1,

k=0, k>l.

},

460

VIII. D I F F E R E N T I A L GAMES

Observe that (ax[b])0 depends only on b because the strategy az defined by (3.19) is nonanticipating, and set (3.23)

z:=x+hf(x,'ff,-b),

-5 := (o~z[b])o 9

Define also ~ E A by (3.24)

(~[bl)~ := (~x[~])k+l,

and choose b* E B 1~ such that (3.25)

vh(z) <_ Jh(z, ~[b*], b*) + z .

We claim that

w(x) < Jh(x,,~x[b*],b*) +~,

(3.26) which gives, by (3.19)

w(x) < Vh(Z) + 2~, and thus the conclusion by the arbitrariness of z > 0. To prove the claim note that, by (3.22) and (3.23), (3.27)

w(x) <_aVh(Z) + hg(x,-5, b)

and that, by (3.23) and (3.24) (3.28)

O'Jh(z, ~[b*], b*) + hg.(z, 5,-b) = Jh(X, az[b*], b*).

By putting together (3.27), (3.25) and (3.28) we get the claim (3.26). This completes the proof that vh satisfies (3.16). Next we prove that Vh is the unique solution of (3.16) by showing that two bounded functions u and w such that (3.29)

u(x) < m a x m i n { a u ( x + hf(x,a,b)) + ht(x,a,b) }

(3.30)

w(x)>_maxmin{o'w(x+hf(x,a,b))+ht(x,a,b)}

bEB aEA bEB aEA

satisfy u < w. In fact, by (3.29) there exists b E B such that

u(x)<_au(x+hf(x,a,b))+hg(x,a,-b)

for all a E A,

and by (3.30) there is ~ E A such that

w(x) > aw(x + hf(x, -5, b)) + hg(x, 5, b) . Putting the last two inequalities together we get

u(x) - w(x) < a sup (u - w)(z) < +oc

for all x E ~ N .

zE~g

Since a < 1 we conclude that s u p ~ (u - w) _< 0. This completes the proof of all statements concerning Vh. The results about uh can be proved in a similar way.

3. O T H E R I N F O R M A T I O N PATTERNS

Next we give the main result of this subsection. equations of Theorem 1.1 of Chapter VI.

461

It is an extension to HJI

THEOREM 3.19. Assume (0.2), (1.4), and (3.18). Then the value function Vh (re-

spectively, Uh) of the minorant (respectively, majorant) game converge uniformly on every compact set o f R N as h ---+ 0 + to the lower (respectively, upper) value v (respectively, u) of the game (see Definition 1.6). REMARK 3.20. Ifvh(x) and Uh(X) converge to the same limit w(x) as h ~ 0 +, then w(x) is called the Fleming's value of the game. The last theorem implies that when the value defined in w exists (e.g., if the Isaacs' condition holds, by Corollary 2.2), then the Fleming's value also exists and the two values coincide. <1 PROOF OF THEOREM 3.19. We prove only the statement about Vh and leave the part on Uh as an exercise for the reader. The proof is similar to that of Theorem VI.I.1. We observe t h a t Vh, h > 0, are uniformly bounded and consider the weak limits

u(x) :=

liminf

vh(y),

(Y,h )~(x, 0+ )

g(x) :=

limsup

vh(y).

(y,h ) ~ (x,O+)

We show t h a t uu_ E BLSC(]R N) is a supersolution and g E BUSC(]R N) is a subsolution of the lower HJI equation (1.17). Then the comparison principle Theorem V.1.3 (whose assumptions are verified by the proof of Theorem 2.1) implies that ~
and both inequalities are actually equalities because u < K by definition. Thus Vh converge uniformly to v on compact sets by the properties of weak limits, see Lemma V.1.9. We begin with the proof that u_ is a supersolution of (1.17). The semicontinuity of u follows from L e m m a V.1.5. By (3.16) Vh satisfies

Vh(X) + max { --aVh(X + f(x, a, b)) - hr.(x, a, b) } > 0 aEA

for all b E B,

so it is a supersolution of the discrete Bellman equation (HJB)h of Chapter VI for any fixed value of b E B. Thus by the proof of Theorem VI.I.1 for any ~ E C I ( N N) and xl, a strict local minimum point of u - ~, there exists ~ = ~(b) such t h a t

Au(xl) - D ~ ( x l ) . f(xl,-& b) - g(xl,-& b) > O . Then Au_(xl) + min max { - D ~ ( x l ) 9f ( x l , a, b) - g(xl, a, b) } > O, bEB aEA

which is the desired inequality.

462

VIII. D I F F E R E N T I A L GAMES

Next we consider ~ and a strict maximum point x0 for ~ - ~ on S : = / 3 ( x 0 , r). By Lemma V.1.6 there exist x,~ E S, hn --* 0 + such that (3.31)

(Vh, -- ~)(xn) = n~x(vhn -- 4),

xn ~ xo,

Vh, (xn) ~ ~(Xo).

By (3.16) there exist bn E B such that

Vh, (xn) -- avh, (Zn + hnf(x,,, a, b~)) - hng(z~, a, bn) <_ 0

for all a E A .

We use the first relation in (3.31) and the definition of a to get (1 -- Ahn) [~2(Xn) -- ~(Xn + hnf(Xn, a, bn))] + Ahnvh, (Xn) - hng(xn, a, bn) ~_ O . Now we extract a subsequence of bn convergent to b E B, and without relabeling, we divide by hn and pass to the limit in the last inequality to obtain, by (3.31),

- D p ( x o ) . f(xo, a, b) + .k~(xo) - g(xo, a, b) <__0

for all a E A .

Then, A~(x0) + m i n m a x { - D ~ ( x o ) . f(xo,a,b) - g(xo,a,b) } _< 0, bEB aEA

which proves that ~ is a subsolution of (1.17).

9

Now we explain how the preceding results can be applied to constructing approximated optimal strategies for the infinite horizon differential game. The next result shows how to synthesize an optimal strategy in feedback form for the minorant game from the solution of the lower DP equation (3.16). It is the extension to games of a standard result in the theory of optimal control for discrete time systems, see the Appendix to Chapter VI. Of course a similar result holds for the majorant game, by using the upper DP equation (3.17). THEOREM 3.21. Let h E ]0, 1/)~[, w E BC(]~ N) be a solution of (3.16), and F : X B ~ A be such that

~N

F(y, b) E arg min { a w ( y + h f ( y , a, b)) + hg(y, a, b) } . aEA

For any sequence b E B I~l and any x E defined by

R y

consider the sequence { zn[b] } in

l~ N

Zn+l = zn + hf(zn, F(zn, bn),bn), Zo ~ X~

and define a*[b],, := F(zn,bn). Then a* E A is optimal at the initial point x, and (3.32)

Vh(X) = max Jh(x, a*[b], b) = Jh(X, a*[b*], b*) bEB 1~

for any b* E B I~t such that bn E G(zn[b*]) for all n, where G(y) := arg max nfin { a w ( y + h f ( y , a, b)) + hg(y, a, b) } . bEB

aEA

3. O T H E R INFORMATION PATTERNS

463

Note t h a t the optimal strategy a* for the minorant game, and the corresponding maximizing control b*, can be computed if one knows a solution of (3.16), namely the value function Vh, and if one can solve for every y E ]~N the problem max min { avh(y 4, hf(y, a, b)) 4- h~(y, a, b) } bEB aEA

'

which is the right-hand side of the lower DP equation (3.16). PROOF OF THEOREIvl 3.21. For any b E B N by definition of zn[b] and (3.16),

w(zn) >_~w(zn+l) + he(z~, F(z~, b~), b~) and the equality holds if and only if b~ E G(zn). Thus we get, by induction, n--1

w(x) >_a~w(z~) 4- h Z akf(zk' F(zk, bk), bk), k=0

and, letting n -o co,

w(x) >_ Jh(x,a*[b], b), where the equality holds if bn E G(zn) for all n. There is a natural way to extend the optimal discrete time strategy c~* E A and the control b* E B N to continuous time, by defining b~ E B as

b~(t) : = b*[t/h],

(3.33)

where [-] denotes the integer part of a real number, and a~ E F as

(3.34)

c~[b](t) := ct [b][t/h],

where for any b E B we set bn := b(nh) . Note t h a t b~ E/3h where

Bh : = {b E B : b(s) = b(kh) for all s E [kh, (k + 1)h[, k E N } , and that a~[b] E Ah for all b E B. The next result says that in some sense a~ is an approximately optimal strategy for the infinite horizon game with continuous time. COROLLARY 3.22. Under the assumptions of Theorem 3.19 we have

(3.35)

lim J(x, c~[b;], b~) = v(x),

h~0 +

and for any ~ > 0 (3.36)

sup J(x, a~[b], b) < v(x) 4. c bEl~h

for all h sufficiently small.

464

VIII, D I F F E R E N T I A L GAMES

PROOF. It is not hard to show, as in the proof of Lemma VI.1.2, that (0.2), (1.4), and (3.18) imply (3.37)

IJ(x, a, b) - Jh(x,~,b)] _< ~(h)

for all a E Ah, b E B h

where w is a modulus. Then

IJ(x, a~[b~], b~) - Jh(x, a*[b*], b*)l < w(h), and we get the first conclusion (3.35) from (3.32) and Theorem 3.19. Now observe that (3.32) and (3.37) imply sup d(x, a~[b], b) <_ J(x, a~[b~], b'h) + 2~(h). bESh

Then we can use (3.35) to get the last conclusion (3.36).

9

Under some additional assumptions we can refine Theorem 3.19 and Corollary 3.22 by giving an estimate of the error made if the value function v is replaced by Vh and if the approximately optimal strategy a~ is chosen by the first player. THEOREM 3.23. In addition to (0.2), (1.4), and (3.18) assume f is bounded, g is lipschitzean in y uniformly in a and b (i.e., we(r) = Ler), and let "y be the exponent of H61der continuity of v (see Proposition 1.8). Then there is a constant C such that (3.38)

[Vh(X) -- v(x)[ _< Ch ~/2,

(3.39)

Is(x, ~[b~], b~) - v(x) I < Ch~/~,

(3.40)

]sup J(x, a*h[b], b) - v(x)] < Ch "Y/2, bEBh

for all x E ~ N and h E ]0, 1/A[, where a~ and b*h are defined by (3.34), (3.33), and (3.32). SKETCH OF THE PROOF. The proof of (3.38) is very similar to that of rem VI.1.5 and it is left as an exercise for the reader (see Exercise 3.8). the additional assumption on g, the modulus w in (3.37) is w(h) = Ch. (3.39) and (3.40) are immediately obtained by the argument of the proof rollary 3.22.

TheoUnder Then of Co-

"k~'k

We end this section with a brief discussion of the approximation procedure which is the basis of Friedman's theory. A detailed description of this method goes beyond the scope of the book, but we explain how it is connected to the various formulations of differential games presented in this chapter. In Friedman's theory the differential game is approximated by a lower and an upper &game, where 5 > 0 is a small parameter. In these games the state is governed by the

3. O T H E R I N F O R M A T I O N

PATTERNS

465

continuous-time system (0.1), but decisions are taken only at the discrete times ~n, n E N. The information pattern is the following: in the lower 5-game at time 5n the second player chooses his control in the interval of time ]hn, ~(n + 1)], and with this information available the first player makes his choice for the same time interval; in the upper 5-game the information advantage is reversed. This can be formalized by introducing the lower and upper 5-strategies and leads to the definition of a lower 5-value v~(x) and an upper 5-value u~(x), where x is the initial state of the system. It is rather intuitive that v~ < v and u < u~ for all 5 :> 0, and that, setting 5n --- T / n for a fixed T, the sequence v~, is nondecreasing and u~, is nonincreasing. The limits of these sequences are, respectively, Friedman's lower and upper value, and if they coincide we have the existence of Friedman's value. To compare these values with the notions introduced previously in this chapter we need only to observe that the following form of the Dynamic Programming Principle holds for the lower and upper 5-game, respectively: (3.41) v~(x) ---- inf sup~ fo ~ e(yx(s), a[b](s), b(s))e -~s ds + v~(y~(5; a[b]. b))e - ~ }: a E F ~ bEB~ ~

(3.42)

=

~(y~(s), a(s), Z[a](s))e -~s ds + u~(y.(~; a, Z[a]))e -~ !.~, )

~EA~ aEA~ ~ JO

where .A~ := ( a : [0, 5] --~ d measurable },

/ ~ : = { b: [0, 5] --+ B measurable },

r~ : = { ~ : B~ ~ A~ },

A~ := { Z : A~ ~ ~ } .

See Exercise 3.9 for a simple definition of v~ and u~ with these properties. THEOREM 3.24. Assume (0.2), (0.5), and (1.4). For all small 5 > 0 let va and u5 be uniformly bounded functions satisfying, respectively, (3.41) and (3.42). Then v~ and u~ converge uniformly on every compact set of ]~N as 5 ---+0 +, respectively, to the lower value v and the upper value u of the game, see Definition 1.6. SKETCH OF THE PROOF. To study the limit ofv~ we first show that (3.41) implies (3.43)

v~(x) =

sup inf ~ .Iof6~.(y.(s), a(s),

b(s))e-'xs ds + v~(y.(5;

a,

b))e-'x~ }.

bE/3 a E A ~

Then we consider the weak limits v(x) :=

liminf v~(y), (y,~)~(x~0+)

~(x) :=

limsup v~(y), (y,~)~(~,0+)

and show that v is a supersolution and ~ a subsolution of the lower HJI equation (1.17). This is achieved by the arguments of the proof of Theorem 3.19, after replacing the discrete DP equation (3.16) with (3.43). The conclusion follows from the comparison principle Theorem 1.3 which gives V < v < _v, and the inequality v _< ~ is trivial.

466

VIII. D I F F E R E N T I A L GAMES

Exercises 3.1. Let { ~n } be a sequence of maps N N ~ A and { tn } be an increasing sequence such that to --- 0 and limn t~ -= +cr Define for y E 3)

r

= ~(y(tn))

if tn < t < t n + l .

Prove that ~ is a feedback strategy with finite memory. 3.2. Prove that the D P P still holds i f / ~ and G are replaced by the feedback strategies with finite memory in the definition of VA and vB. [Hint: follow the proof of Theorem 3.8 and show that S(a,y) = { ti E S(-~,y) : ti < t } 0 { t } U { ti + t :

t~ e S(r

y(- + t)) }.1

3.3. State and prove a verification theorem for feedback strategies using subsolutions of the HJI equations as verification functions. 3.4. We say that a sequence of strategies { ~,~ } in .T" (respectively, { {n } in G) is optimizing at the initial point x if

VA(X) = lira sup J~(x; Cn, b) n

bEN

(respectively, vB(x) = limn infae.4 Joe(x; a, (~)). Prove the following extension of Proposition 3.14. Assume x is a point where the game has feedback value and there are optimizing sequences { • } in 7 c_ 5c and { ~n } in 61 C__G, with f~ and Gp compatible. Then (3.44)

VF(X) = limJ(x;~n,~n) = sup inf J ( x ; ( , ( ) = inf sup J(x; ~, ().

3.5. Prove the following extension of Proposition 3.15. Assume the game has feedback value at x and that 7 , G' are compatible, contain the constant strategies, and satisfy sup inf J(x; ~', ~) = inf sup J(x; r ~). Then there exist optimizing sequences of strategies (see Exercise 3.4) { ~,~ } in 7 and { ~n } in G' such that (3.44) holds. 3.6. Given g : ~N X B --~ ]t~P, set l/Y := { g(y(" ), b(. )) : y E Y, b 9 } and define an output state feedback strategy as a nonanticipating map a : W ---* A such that for any x 9 ]~Y, b : [0, T[ --~ B measurable and T 9 ]0, +cr there exists a unique solution of

f q'(t) = f(q(t),a[w](t),b(t)), ~ w(t) = g(q(t), b(t) ), |

(3.45)

Lq(O) =

~.

0 < t < T,

3. O T H E R I N F O R M A T I O N PATTERNS

467

The variable w is called the output or observed state. We indicate with 2" the set of these strategies. Since for a general g the state q cannot be reconstructed from the o u t p u t w, this information p a t t e r n is called imperfect state measurement and we have a game with partial observation. Define

W ( x ) := inf sup 2(x, a[w], b), o'EZ bEl3

where w comes from the solution of (3.45) for given, x, or, and b. Prove that (i)

if g(q, b) = b, then W = v, the lower value of the game, and if g(q, b) = q, then W = VA, the A-feedback value;

(ii)

Z contains the constant strategies;

(iii) for any a E 2" there exists ~ E F such that, for any b E B, a[w] = a[b], where w solves (3.45); (iv) v < W _< us, the upper value of the static game; (v)

ifg(q,b) = ~(q), then VA <_ W;

(vi) if the Isaacs' condition holds and Us = u, the upper value of the game, then W = vF, the feedback value (results of this kind are sometimes called Certainty Equivalence Principle); (vii) W is a supersolution of the lower HJI equation (1.17). 3.7. State and prove a result for the majorant game corresponding to Theorem 3.21 and a result for the upper value corresponding to Corollary 3.22. 3.8. Prove Theorem 3.23. [Hint: for the proof of (3.38) note that the proof of Theorem VI.1.5 has to be modified only at two points, namely, formulas (VI.1.32) and (VI.1.33), where one must choose first a suitable b* E B and then a*.] 3.9. Define the strategies of the first player anticipating the second player's moves for a time at most 5 > O, as, A := { a : B ~ A : for any n E N and b,b E B, b(s) = b(s) for all s < n5 implies [bi(s) =

for all

< n5

},

and denote with H the strategies of second player with the same property. Define

v,~(x) :--- inf supJ(x,a[b],b), o~EA bEB

u~(x) : = sup inf J(x, a, ~3[a]) . ~EYI aE.A

Prove t h a t v~ and u5 satisfy, respectively, (3.41) and (3.42).

468

viii. DIFFERENTIALGAMES 4. B i b l i o g r a p h i c a l n o t e s

The theory of (two-person, zero-sum) differential games started at the beginning of the '60s with the work of Isaacs [Is65] in the U.S.A. and of Pontryagin and his school [PBGM62, P66] in the Soviet Union. The main motivation at that time was the study of military problems, and pursuit-evasion games were the most important examples. Isaacs based his work on the Dynamic Programming method. He analyzed many special cases of the PDE that is now called Hamilton-Jacobi-Isaacs, trying to solve them more or less explicitly and synthesizing optimal feedbacks from the solution. He began a study of singular surfaces (which are essentially the sets where the value function is discontinuous or not differentiable) which was continued mainly by Breakwell, Bernhard [Ber77, BrB90, Ber92], and Lewin [Lew94], and led to the explicit solution of some low-dimensional but highly nontrivial games. We refer to the recent book [Lew94] for a survey of this theory and to [Br89, Me194] for some connections with viscosity solutions. As for the second motivation for two-person, zero-sum differential games, that is, the worst case analysis of a controlled system with a disturbance, it is already present at the early stages of the theory [F61]. It has been made rigorous more recently by Barron and Jensen within the theory of risk-aversion [BJ89] (see also [BSa91b]) and by many authors in connection with 7-/~ and robust control. For this topic we refer to the book of Basar and Bernhard [BaBegl] in the case of linear systems and to [FM92a, FM95, Jam92, Jam93, Sor96a, DME97] and Appendix B of this book for nonlinear systems and the use of viscosity solutions. The notions of strategies and value of Sections 1 and 2 are due to Varaiya [Va67], Roxin [Ro69], and Elliott and Kalton [EK72, EK74], and Evans and Souganidis [ES84] first proved that such a value is a viscosity solution of the Hamilton-JacobiIsaacs equation for finite horizon problems. Theorems 1.10 and 2.1 are adaptations of their results to the infinite horizon problem. (Indeed, to our knowledge, the first connection between differential games and viscosity solutions was made by Souganidis in his Ph.D. thesis of 1983 for the Fleming and Friedman's values). The book [E187] contains a presentation of this material. Similar results have been obtained for many different problems, indeed almost any cost functional considered in Chapters III and IV has been studied in the context of differential games by these methods. We mention [EI84] for problems with exit times in bounded domains, [L85b] for games with reflected processes, [BS89] for time-optimal control (i.e., pursuit-evasion games) and [BSgla, Sor93a] for more general problems with exit times and the study of free boundaries as in w (Soravia applied some of these results to the study of the stability of dynamical systems with two competing controllers [Sor93a, Sor95] and to some models of front propagation [Sor94a]). We refer to [A191, K95, BKS96] for problems with state constraints, [Bn90] for games with maximum cost, [Y90, EL91, IK91b] for problems with switching cost and [LeY92] if the process is in addition piecewise deterministic, and [BJM93,

4. B I B L I O G R A P H I C A L NOTES

469

Y94, Ra95] for systems with impulses. Example 1.7 is adapted from [Be64]. The classical minimax Theorem 2.3 can be found in [AE84], for instance. The notion of feedback strategy of w is taken from Soravia [Sor94b], see also [E177] and [BaBe91]. The results of this subsection are new; Propositions 3.14 and 3.15 are an attempt to make a connection with the Isaacs-Breakwell theory, inspired by Bernhard's lectures [Ber92]. The theory of positionM differential games was developed in the '70s by N.N. Krasovskii and Subbotin [KS74]. The value of the game in this theory is characterized as the unique function that is stable with respect to both players (i.e., which satisfies weak super- and suboptimality principles of the form (3.14) and (3.15)), see Remark 3.17). Subbotin [Su80] characterized these properties in terms of inequalities for Dini directional derivatives, in the case of locally Lipschitz functions (see also [SS78] and [Su84]). The equivalence between functions satisfying these inequalities and viscosity sub- and supersolutions was proved in different ways, see [EI84, LS85, ST86]. Later this notion of solution was extended to general HJ equations and to merely continuous functions [Su91a, Su91b] by means of the generalized Dini derivatives, see Definitions 2.36 and 2.38 in Chapter III. The equivalence between minimax and viscosity solutions was proved in [Su93b]; see also w in Chapter III and [ClaL94]. The theory was extended to discontinuous solutions [RS88, Su93a]. A comprehensive reference for the theory of minimax solutions is the book [Su95]. Some more recent papers on constructive and numerical methods within this theory are [Su97, CLS97, ChSu95, Ta94, Ta97, PBKTZ94]. For a survey of other developments of the theory of positional differential games we refer to the book of N.N. Krasovskii and A.N. Krasovskii [KK95]. The Berkovitz's theory starts with the paper [Be85] and combines features of the Isaacs, Friedman and Krasovskii-Subbotin approaches. The connection with minimax and viscosity solutions was established in [Be88], see also [Gh91] for games with state constraints. We refer to [Be94] for a survey of this theory. Subsection 3.2 adapts to the infinite horizon problem the results obtained for pursuit-evasion games in [BS91b], see also [Sor92b]. Fleming introduced his notion of value in 1961 IF61] and proved its existence for finite horizon problems and for some special cases of pursuit-evasion games; see also [F64]. These were the first general and rigorous results about the existence of a value. The connection of this definition of value with viscosity solutions can be found in [Sou85b] for the case of finite horizon. Friedman's theory was developed in the early '70s and is summarized in the books [FriT1, Fri74]. The connection with viscosity solutions was studied in [BE J84] for the finite horizon problem, and in [Sor92b] for games with a target. Many results of Chapters V and VI were extended to differential games. Indeed many classical examples of pursuit-evasion games have discontinuous value, and this was one of the main motivations for developing the theory of discontinuous viscosity solutions of Chapter V. The characterization of the value function as the unique e-solution of w holds for pursuit-evasion games where the pursuer uses relaxed controls [BFS97], see also [Su95, BBF95] for other notions of value.

470

VIII. D I F F E R E N T I A L GAMES

A related result for problems with open targets has been proved in [BG97]. Some applications in w have been carried over to games in [BS94], whereas the uniqueness result of w is currently open for games. The theory of bilateral supersolutions described in w does not seem to extend to games because it uses heavily the convexity of the Hamiltonian. The semidiscrete approximations of w and w do not converge in general if the value function is discontinuous. In this case the scheme was modified in [BBF95] to get the convergence in a suitable sense. This paper also studies a fully discrete scheme, as in [BFS94] (see also Appendix A of the present book). Estimates of the rate of convergence such as in w have been first studied for games by Souganidis [Sou85b], see also [Sou85a, AT87, Bot94] and [Sor97b]. Results similar to those of w can be found in [BSa91a, BSa92] for games with targets. Differential games with Lipschitz controls are studied in [BEJ84], and the results extend Corollary VII.4.2. Stochastic two-person zero-sum differential games have been studied in the framework of viscosity solutions by Fleming and Souganidis [FSo89], see also [BaSo91, Ka95, Sw96b]; for differential games involving infinite dimensional systems we refer to [KSS97] and the reference therein. Finally we give some additional references. A book devoted to pursuit-evasion games is [Ha75]; the volume [Pg0] contains several contributions by authors of Pontryagin's school; some recent papers on numerical methods are [TiG94, Ti95, Pes94b]. The theory of viability has been applied to differential games by J.P. Aubin and his school [Au91, CQS94a]; let us mention [CQS97] for an algorithm to compute the possibly discontinuous value function of pursuit-evasion games with state constraints. For the general theory of N-person differential games we refer to the books [BaO82] and [Kle93]. This theory has a wide range of applications from economics to environmental management, ecology, and biology, see [BAH94, O195] and the references therein.

APPENDIX A Numerical Solution of Dynamic Programming Equations by Maurizio Falcone

As shown in the book, the Dynamic Programming approach to the solution of deterministic optimal control problems is essentially based on the characterization of the value function in terms of a partial differential equation, the Hamilton-JacobiBellman equation. In the case of deterministic optimal control problems, which we consider here, this is a nonlinear first order equation H(x,u(x), Du(x)) = 0 where H is convex with respect to Du. However, this approach is very flexible since a similar characterization can be obtained for the value flmction of stochastic optimal control problems (in which case the equation will be of the second order and/or an integro-differential equation depending on the stochastic process describing the dynamics) and of differential games (in which case we still have a first order equation but we lose the convexity with respect to Du). One feature of this approach is particularly interesting in many applications: it permits computation of approximate optimal controls in feedback form and, as a consequence, approximate optimal trajectories. Although these are very appealing properties if compared to the results given by the open-loop approach based on the Pontryagin Maximum Principle, the Dynamic Programming approach suffers the problem of the "rise of dimension". In fact, to compute the value function we need to solve a Hamilton-Jacobi-Bellman equation in the domain where the initial condition for the dynamics is taken. On the other hand, the necessary conditions of the Pontryagin Maximum Principle often correspond to the solution of a two-point boundary value problem for a system of ordinary differential equations involving the state and the adjoint variables. This simple observation implies that the number of computations needed by the Dynamic Programming approach can be huge when the state variable belongs to a space of high dimension (R g with N = 4, 6 or more). This is one of the main motivations to develop efficient algorithms to solve equations of this type. Of course, when comparing the Dynamic Programming approach with the Pontryagin

472

A. NUMERICAL SOLUTION OF DYNAMIC PROGRAMMING EQUATIONS

Maximum Principle one has to remember that the former gives a global minimum while the latter gives only a necessary condition which is satisfied by local extrema. The basic idea in the foreground of the numerical methods we present in this Appendix is to start the construction of the algorithm by a discretization of the original control problem and to use a discrete version of the Dynamic Programming Principle. Of course, this is not the only possible approach and some references to other methods will be given in the last section containing also some historical remarks. The approximation scheme is obtained in two steps. First we look at the original control problem and make a time discretization with time step h = At. The value function Vh corresponding to this new discrete-time optimal control problem is a natural candidate to approximate v (see Chapter VI). A discrete Dynamic Programming Principle (see Chapter VI, Appendix) gives the characterization of Vh and, at the same time, a semi-discrete approximation scheme. The second step consists in a space discretization with space step k = Ax and it results in a finite dimensional problem that we can finally solve. The schemes obtained by this technique have a nice "built-in" control interpretation which is also useful to establish convergence results and a priori estimates in L ~ , both for the semi-discrete and the fully-discrete schemes. Moreover, the synthesis of approximate feedback controls is simple and effective. The aim of this appendix is mainly expository, so we give here only some representative results. The interested reader should look at the papers mentioned in the references and in our last section for the most general version of the results. In Section 1 we give a short presentation of the results related to the approximation of the unconstrained infinite horizon problem that we will use as our model problem. In that section the main results related to our discretization technique will be presented in detail. Section 2 is devoted to the modifications needed to deal with problems with state constraints. Section 3 deals with "target" problems, such as the minimum time problem and pursuit-evasion games. In Section 4 we give some hints for the construction of the algorithms. Finally, Section 5 contains some commented references and historical remarks. 1. T h e i n f i n i t e h o r i z o n p r o b l e m We consider the following system of controlled differential equations in ]I~N

(1.1)

f y'(t) = f ( y ( t ) ,

a(t))

Ly(0) = x

where a ( t ) C .4 -- L~162 +co[; A) and A is a compact subset of ~M. We will refer to the set .4 as the set of admissible controls. The cost functional related to the infinite horizon problem is (1.2)

Jx(~) -

//

e ( ~( s ) , ~ ( s ) ) e - ~ d s .

1. T H E I N F I N I T E H O R I Z O N P R O B L E M

473

We will make the following assumptions on the data (1.3) (1.4) (1.5)

f

:

]~N X A

~

]~g

and e : ]~N • A ~ ~ are continuous;

I f ( x l , a ) - f(x2,a)l < LflXl - x2i for any a E A and llfll~z < Mf; Ig(xl;a)-s

a)l
Assumptions (1.4) guarantee that there exists a unique solution trajectory y defined in [0, +cc[ for any fixed control a. The value function v, defined for any initial state x, is

v(x) = inf J z ( a ) . aEA

1.1. The Dynamic Programming equation Let us recall from Chapter III that the value function v of the unconstrained problem above is the unique viscosity solution of (H J)

)~u(x) + s u p { - f ( x , a)Du(x) - g(x, a)} = 0,

for

X

E ]~N.

aEA

Making a discretization in time of the original control problem, which consists in replacing the dynamics (1.1) by a one-step scheme (e.g., by the Euler method) and the cost functional (1.2) by its discretization by a quadrature formula (e.g., the rectangle rule), one can get a new control problem in discrete time. The value function vh for this problem (as we have seen in Chapter VI) satisfies a discrete dynamic programming principle which gives the following approximation scheme, (HJ)h

uh(x)=ma~,~{(1--~h)uh(x+hf(x~a))+he(x~a)},

forxER N.

Under our assumptions (1.3)-(1.5) and for A > Lf, the family of functions Vh is equibounded (by Mr~A) and equicontinuous L~ ]vh(x) -- vh(y)] <_ ~ _ n l Ix - y] .

(1.6)

Then, by the Ascoli-Arzel~ theorem we can pass to the limit and prove that it converges locally uniformly to v for h going to 0. Moreover, the following estimate holds true, (1.7)

IIv - v h l l ~ < c h 1/~

(see w In order to compute an approximate value function and solve (HJ)h we have to make a further step: a discretization in space. We start building a grid in the state space and, to simplify our presentation, we will assume that there exists a bounded polyhedron f~ c R N such that, for h sufficiently small (1.8)

x§

E-~

VxE~

VaEA

474

A. NUMERICALSOLUTIONOF DYNAMICPROGRAMMINGEQUATIONS

i.e., f} contains the discrete controlled trajectories for all times t _> 0. We will see in the next section how we can get rid of this assumption and treat different boundary conditions on 0f~. We construct a regular triangulation of Q made by

a family of simplices Sj, such that f~ = Uj Sj, denoting by xi, i -= 1 , . . . , L, the nodes of the triangulation and by k -= max diam(Sj) 3 the size of the mesh (here and in the sequel diam(B) will denote the diameter of the set B). We look for a solution of (HJ)~

u(xi) = mi~{(1 - Ah)u(xi + hf(xi, a)) + he(xi, a)},

i = 1,..., L

in the space of piecewise linear functions on f~ (1.9)

W k =- {w: a + R : w(. ) 9 C ( a ) and D w ( x ) = cj Y x 9 Sj, V j } .

We will denote by II" I{cr the max norm for vectors, i.e., IIZII~ = maxi=l,...,L IZil. THEOREM 1.1. Assume (1.3)-(1.5) and (1.8). exists a unique solution v~ of (HJ)~ in W k.

Then for any h 9 ]0, l/A] there

PROOF. By our assumption (1.8), starting from any x in f~ we will reach points which still belong to f~. Then, for every u 9 W k we have L u(xi-t- hf(xi, a)) -= E ,~ij(a)u(xj) j=l where ,~ij(a) are the coefficients of the convex combination representing the point xi + h f ( x i , a), i.e., (1.10)

(1.11)

L xi + h f ( x i , a ) = E ~ij(a) xj j=l L 0 ~ .'~ij(a) _< 1 and E..~ij(a) --- 1, for any a 9 A. j=l

Then (HJ)~ is equivalent to the following fixed point problem in finite dimension (1.12)

U = T(U)

where the map T : ]~L __, II~L is defined componentwise as (1.13)

(T(U))i -- amid[(1 - Ah)A(a) U + hF(a)]i,

Ui =- u(xi), Fi(a) - g(xi, a) and A(a) is the L x L matrix of the coefficients Aij satisfying (1.10)-(1.11) for i , j = 1 , . . . , L . Note that the matrix A satisfies the

1. T H E I N F I N I T E H O R I Z O N P R O B L E M

475

same properties of the transition matrix of a Markov chain process with L states. This can be interpreted as saying that the transition probability from the node xi to the node x j under the dynamics f is )~ij(a). This interpretation is particularly useful when one wants to extend the scheme to deal with diffusion control processes (see the Bibliographical notes for some references). T is a contraction mapping. In fact, let ~ be a control giving the m i n i m u m in T ( V ) i , we have IT(U) - T(V)] i < (1 - ,~h)[A(~)(U - V)] i <_ (1 - ,kh)m.axl,~ij(a)[ HU - vll~c z,3

< (1 - ~ h ) l l U - V i l e .

Switching the role of U and V, we can conclude that (1.14)

][T(U) - T(V)]]~ < (1 - Ah)IIU - V[]:r

and the thesis follows for h E ]0, 1/~]. REMARK 1.2. The existence of (at least) one control a* giving the minimum in (1.13) relies on the continuity of the data and on the compactness of the set of controls A. Note also that the search for a global m i n i m u m over A is not an easy task. Let us define

L~(a, U) - (1 - ,kh)Ai(a)U + hFi(a) . To compute the fixed point, we have to find the argument of the m i n i m u m for each component i, i.e., we have to solve the problems min Li (a, U) . aEA

Note that L i ( a , . ) is an affine function for every i and that when A is finite (e.g., A = { a l , . . . , a s } ) , the m i n i m u m can be obtained by direct comparison at each node. So one simple way to solve the problem is to replace A by a finite set of controls constructing a mesh over A. Depending on the structure of the control problem this may or may not affect the result. For example, when the optimal controls are bang-bang a careful discretization (essentially only of the boundary of A) can give very accurate results. <~ Since T is a contraction mapping in ]RN, the sequence (1.15)

U n = T(Un-1),

U~ = Z

will converge to U*, for any Z E ~N. Moreover, the following estimate holds true: IIU~ - V*ll~r _< (1 - ~ h ) n l [ Z

- U*I[~.

We want to give an estimate for the approximation error in the L~

norm.

476

A. NUMERICALSOLUTIONOF DYNAMICPROGRAMMINGEQUATIONS

THEOREM 1.3. Let Vh and v k be the solutions of (HJ)h and (HJ)~. A s s u m e that (1.4), (1.5) and (1.8) hold true and that A > L I , then L~ k i]Vh - v hkl l ~ < A(A - L f ) h "

(1.16)

PROOF. For any x E ~, we can write IVh(X) -- vk(x)l ~-- E A j ( U h ( X ) J

-- Vh(Xj)) "j- E ~ J Y h ( X j ) J

-- y k ( x j ) "

where the Aj are the coefficients of a convex combination. By the equations, we have (1.17) _< (1 - Ah)llVh -- v~ll~

where ~ is a control giving the minimum in (HJ)kh. This implies, as in Theorem 1.1, that [Vh(Xj) -- vk(xj)l

<_ (1 - Ah)[)Vh - vkll~

.

By the Lipschitz continuity of vh stated in (1.6), we get Lt k . ]Vh(X) -- Yh(Xj)] ~-- A -- L /

(1.18) By (1.17) and (1.18), (1.19)

_<

HVh

(1

-

Ah)l[Vh - v h l [ o r + ~

k

.

Then, we conclude (1.20)

IIvh-

v~]lc~

Le k <- A ( A - L I ) h

Coupling (1.20) with the estimate (1.7) we obtain the following global estimate for the numerical solution. THEOREM 1.4. Let v and v k be the solutions of (HJ)h and (HJ)~. A s s u m e that (1.3)-(1.5) and (1.8) hold true and that A > L f , then (1.21)

IIv -

v~llor

< ChV~ +

Le k A(A - Lf) h

The contraction mapping argument used in the proof of Theorem 1.1 does not provide a very efficient method to compute the solution since the contraction coefficient (1 - A h ) tends to 1 as h tends to 0. This means that we would expect

1. THE INFINITE HORIZON PROBLEM

477

to have a very slow convergence whenever we t r y to improve the accuracy of our numerical solution by refining the step sizes. To overcome this difficulty one can accelerate the convergence by exploiting the monotonicity of the map T t h a t is, T ( U ) >_ T ( Z ) if U >_ Z. Let us introduce the basic ideas of the monotone acceleration procedure. We define the set of subsolutions of (1.12) as /A-= {U e If{L: U _< T ( U ) } . It is easy to prove t h a t the set/4 is convex, closed. Moreover, the fixed point U* of the m a p T is also the maximal element of/4, since it is the limit of all monotone nondecreasing sequences starting in U. These properties are fundamental to show that starting from any Z E / g the sequence defined in (1.15) belongs to /4 for every n and converges monotonically (increasing) to U*. We can accelerate the convergence using the contraction mapping iteration just to define a direction of displacement in/d. We will follow that direction up to the b o u n d a r y of U before restarting the algorithm by the contraction mapping iteration. We have then the following ALGORITHM S T E P 0: Take U ~ E ]~L Set n : 0 S T E P 1: Compute Un+l/2 = T ( U '~)

S T E P 2: Compute U '*+1 = U n + -fi(U n+U2 - U n) where ~ - max{# E lt~+ : U n+l E U}

S T E P 3: Stopping criterion: is ][Un+l - un[[oc < z? if YES then S T O P else G O T O S T E P 1 Note that if it is possible to derive an explicit optimal step /z to reach the boundary of the set of subsolutions this will produce a very fast algorithm. W h e n this is not possible, the computation of p will be obtained via an iterative one dimensional zero finding algorithm. In t h a t case, the algorithm is usually faster if one spends more iterations in locating the point on the b o u n d a r y before changing direction. REMARK 1.5. W h e n the set of controls is finite A -- { a l , . . . , a m } we can adopt a different method to find U*. Instead of the contraction mapping argument coupled with the acceleration technique we can apply the simplex method. Note t h a t /g is identified by L linear constraints In fact, /4 = {U C ]1{L : gi(U~a) < 0 for any i = 1 , . . . , L and a E A}, where gi(U, a) = U - (1 - Ah)(A(a)U)i - hFi(a) .

Then, we look for a solution V* of the problem L (1.22) max ~ Vj. VEbI

478

A. NUMERICAL SOLUTION OF DYNAMIC PROGRAMMING EQUATIONS

It is evident by the above remarks that (1.22) can be solved by the simplex method or other methods of linear programming. Moreover, V* and U* must coincide as the following simple argument shows. As we have seen, the fixed point U* is also the maximal element of the set of subsolutions U, then by definition U~* _> V/* so that U/* > V/*; hence, by the definition of V* and the fact that U* e/A,

~-~L=i

~-~L=i

L

(1.2a)

Z i=l

L

= i

In fact, if there is an index i such that V/* < U/*, the maximality of U* would imply L

(1.24)

L

> Z v/* i

i

which contradicts (1.23).

<~

1.2. S y n t h e s i s o f f e e d b a c k c o n t r o l s We go back to one of the most important problems for real applications: the reconstruction of an approximate optimal feedback control. We will construct them by means of our numerical approximation of v. Let us consider equation (HJ)h and define a* = a~(x) E A as the control setting such t h a t

Uh(X) = (1 -- )~h)uh(X --k h f ( x , a * ( x ) ) ) + he(x,a*(x)) (if that control is not unique we can always define a selection). Let our discrete dynamics be obtained by an Euler discretization of (1.1) Yrn+l = Yrn + hf(ym,am),

m EN

YO = X and define a sequence of approximate feedback controls using our equation:

(1.25)

am=--a*h(ym)

m=0,1,2...

We define, (1.26)

Vh(X ) =

inf Jh({am)) {amt

where -~oc

(1.27)

Jh({am}) -= ~

hg.(ym,am)(1 - Ah) m 9

rn=O

In the Appendix to Chapter VI it was proved that

Vh(X) = J h ( { a * } ) 9 In Chapter VI the following results were established.

1. THE INFINITEHORIZONPROBLEM

479

THEOREM 1.6. Let the assumptions (1.3)-(1.5) be satisfied. Let {aL} be the sequence defined in (1.25) and let ~*h(~) =- a * ( ~ )

fo~ any

s 9 [mh, ( m +

1)h[

and m = O~ 1, 2 , . . .

Then J ~h ( a h* ( . ))

,

locally uniformly in ~n

and

inf J x ( a ( ' )) ~(.)~A

J x ( a ~ ( ' )) - - ~ inf Jx(a(" )) ~(.)

as h --~ 0 +.

Now the question arises: can the synthesis of feedback control based on the numerical solution of (HJ)~ also produce an "almost optimal" control for the functional related to the continuous problem? The answer to this question is positive under additional assumptions. Let us define the functions L k : f~ • A ~

L k ( x , a ) -= (1 -- Ah)v~(x + h f ( x , a ) ) + he(x,a) and L : ~

•

L(x, a) -- (1 - Ah)vh(x + h f ( x , a)) + hg(x, a) . Sometimes in the following we will use the short notation w = Vh k when we will not need to emphasize the dependence of the numerical solution from h and k. We can naturally associate with any x E fl a control aw(x) C A such that (1.28)

Lk(x, aw(x)) = min n k ( x , a) = v~(x) . aEA

That control of course could not be unique but we can always select one using a convex criterion (e.g., picking the control with m i n i m u m norm which minimizes Lk(x, 9)). In the same way we can select a (unique) control a(x) such that (1.29)

n ( x , a(x)) = min L(x, a) = Vh(X) . aEA

Let us define the piecewise constant control

a * ( s ) -- aw(ym)

for s 9 [mh, (m + 1)hi

where {ym} is the solution trajectory of the discrete time system Ym+l = Ym + h f ( y m , a w ( y m ) ) ,

Yo = x .

THEOREM 1.7. Let the assumptions (1.3)-(1.5) be satisfied. Moreover, assume that f o r any x there exists a unique a(x) minimizing L. Then, f o r any x 9 * J ;h ( a w ( . ) ) ----* inf d h ( a ( . )) ~(.)

as k --* 0 +

480

A. NUMERICALSOLUTIONOF DYNAMICPROGRAMMINGEQUATIONS

PROOF. The estimate (1.16) shows that Lk converges uniformly on f~ • A to L for k tending to 0. It is clear that the feedback control defined in (1.28) for h and x fixed depends only on k so we will denote it here as ak(x). Since A is compact from any sequence {ak(x)} of controls givingthe minimum in Lk we can always extract a subsequence {an~} convergingto a limit, say {~(x)}. We have then

L'~k(x,a~ ~) - L(x,-d(x))

)0

as k --+ 0. Moreover, by the definition of ak(x) passing to the limit for k tending to 0 one gets

n(x,-6(x)) < L(x, a(x)). By the definition of a(x) the reverse inequality is also true, so that

L(x,'6(x)) = n(x, a(x)) which implies -6(x) = a(x). That argument can be repeated for any subsequence; then we conclude that ak(x) tends to a(x) for k tending to 0. By definition,

hf(x, ak(x)) = Lk(x, ak(x)) -- (1 -- Ah)v~(x + hf(x, ak(x))) for any x E Ft. Taking x = Ym and multiplying by (1 - Ah) m one gets

hg(ym, ak(ym))(1 -- Ah) m . Vhk(Ym)(1 . . . Ah) m

(1

;~h) m + l v hk ( Y m + h f ( Y m , a k ( y m ) ) ) .

Summing on m and passing to the limit for k tending to 0 we get the result.

9

THEOREM 1.8. Let the assumptions (1.3)-(1.5) be satisfied.

Moreover, assume that for any x there exists a unique control a(x) minimizing L. Then, for any xEf~ Jx(a*(. )) ~

inf J~(c~( - )) ~(.)

for h, k and k/h tending to O. PROOF. The proof can be obtained by coupling the a priori error estimates with the traditional approximation of the integral by a series. We want to prove that I J ~ ( a ~ ( - ) ) - v(x)l tends to O. We can write IJx(c~(.

))

- v(x)[

<_ I J x ( c ~ (

. )) - w(x)l

+ Iw(x)

- v(x)l

.

The last term is estimated by the result in Theorem 1.4 and tends to 0. The second term can be estimated by

(1.3o)

IJ~(~w( 9

.

))- ~(~)l <-IJ~(,~,;()) .

_ J ; (ha w ( .9) ) [

+

I J ~h( a ~ ( .9) ) - w ( x ) [ .

Since w = vk the second term of the above sum converges to 0 by Theorem 1.7. Following the lines of Theorem 1.6 one can prove that the first term in the righthand side of (1.30) tends to 0 for h and k going to 0 and this ends the proof. 9

1. THE INFINITE HORIZON PROBLEM 1.3. N u m e r i c a l

481

tests

We present some numerical results in order to show how our numerical approxim a t i o n behaves for s m o o t h a n d n o n s m o o t h solutions. The first two tests are in ~ 1 while the second is in ]~2. TEST 1. We set Ft = ] - 1 , 1[, A = [0,1],

f(x,a) = - x a .

A = 1,

We consider a cost function e which is j u s t Lipschitz continuous, e ( x , a) = x .

T h e viscosity solution in ~t is

v(x) = ~x [ x/2

for x < 0, elsewhere.

T h e o p t i m a l choice is to take a* = 0 whenever x < 0 a n d to move on the left at m a x i m u m speed (a* = 1) whenever x > 0. This causes a kink at x = 0. Figure 1 shows the exact solution (straight line) and the approximate solution (small circles). <~

9.4

0,2

0

-0.2

.0.4

-04

-t -I

--0.'~

*'~5

-0,25

O

O.2S

0.5

0.~

FIGURE 1: Value function (h = 0.1, k = 0.1). TEST 2. We set ~ ----] - 1 , 1[, d = [ - 1 , 1],

)~ ----1,

f(x,a) = a(1 - 1 x 1 ) .

482

A. NUMERICAL SOLUTION OF DYNAMIC PROGRAMMINGEQUATIONS

We consider a cost function g which is j u s t Lipschitz continuous, g(x,y,a)

=

3(1 -Ixl).

For numerical purposes we have extended the cost a n d the vector field outside j u s t setting t h e m to 0. 1.6

1,4

1.2

1

0.8

0.6

0.4

0.2

0 -1.5

-1

-0.5

0

0.5

t

1.5

(a) 1.6 4~ 1.4 o

1.2

i

*

~ o

1 o

9

o 0.8

o

o

o

o

o 0.6

o

o

o

o

o o

o 0.4

o

o

o

o

o

o 0.2

o

o o

o 0 -1.5

0Pd ol

I -0.5

015

I

1.5

(b) FICURE 2: Value functions (h = 0.01), (a) Euler (k = 0.011), (b) R u n g e - K u t t a (k = 0.044).

1. THE INFINITE HORIZON PROBLEM

483

In this test the optimal trajectories are quite simple. For x > 0 the optimal choice is a* = 1 and for x < 0 the optimal choice is a* = - 1 . By a direct computation on the optimal trajectories one can compute the value function. The viscosity solution in ~t is

v(x)=

f (x+l) forx<0, [3(1-x)

elsewhere.

The solution has one sharp kink at the point x = 0. It can be seen t h a t the numerical solution is close to the exact solution also at that point for a discretization based on 201 grid points (Figure 2a). Note t h a t at x = 0 one has two different optimal choices for the control a* = 1 and a* = - 1 and this is the main cause for the j u m p in the derivative of the solution. Using a higher order method for the discretization of the system of differential equations and of the cost functional, the approximate solution can even be improved. In fact, coupling a R u n g e - K u t t a scheme with a trapezoid rule produces an accurate approximation with just 51 grid points (see Figure 2b). Several other couplings between a one-step method for ordinary differential equations and a quadrature formula for the cost are possible (see the hints at the end of this Appendix). <~ TEST 3. Let us consider an example in R 2. We set ~ = ] - 1 , 1[2, A = / ~ ( 0 , 1) c ~2.

fl(x,y.a)=ay,

A= 1

f2(x,y,a)

=0.

FIGURE 3: Value function (h = 0.05, k = 0.025).

484

A. NUMERICAL SOLUTION OF DYNAMIC PROGRAMMING EQUATIONS

We consider the cost function g,

e(x,y,a) = (ix{ - 1) 2 . In this example one can compute the exact solution. The numerical solution represented in Figure 3 has an L m error of 0.025. Note that the algorithm gives an accurate approximate solution also at the origin where the exact solution has a cusp. <~ 2. P r o b l e m s

with state constraints

One major interest in view of real applications is to see how state constraints can be included in the model. In practical problems, the system has often to satisfy some restrictions (e.g., on the velocity or on the acceleration) which can be written as state constraints for the dynamics (1.1). We will introduce such constraints in our infinite horizon model. Let 12 be an open bounded convex subset of ]~N with regular boundary (n(x) being its outward normal at the point x E 0~). For any initial position x E ~, we require that the state remains in ~ for all t > 0. As a consequence we will consider admissible with respect to the state constraint only the (open-loop) control functions such t h a t the corresponding trajectory of (1.1) never leaves ~. We will denote by A ( x ) such a subset of ~4, i.e., for all x E ~ we define A(x) --= {c~(-) E .A: yz(t,a(t)) E -~, Vt > 0}, where yx(t, a(t)) denotes the solution trajectory of (1.1) corresponding to a. The value function for the constrained problem is

v(x)

(2.1)

=

inf

./~(a).

aEA(x)

By the theory of w

v is the unique constrained viscosity solution of

Au(x) + s u p { - f ( x , a). Du(x) - e(x, a)} = O. aEA

An approximation scheme for the constrained problem can be introduced adapting to this situation the scheme for unconstrained problems. In order to understand the scheme it is useful to note that at each internal point we can choose any control in A since, at least for a small time, we can move in any direction without leaving ~. On the other hand, at each point on O~ not all the controls in A are allowed since some of them correspond to directions pointing outward with respect to the constraint ~. This means t h a t the set of admissible controls will depend on x (in a rather irregular way if we do not make additional assumptions on the boundary of ~t), and the "right equation" for the value function should be:

Au(.) +

sup { - f ( x , a ) - D u ( z ) aEA(x)

g(x,a)} = 0

2. PROBLEMS W I T H STATE C O N S T R A I N T S

485

where

A(x) =

A

for x E l ] ,

{a E A : f ( x , a ) points inward the constraint}

for x E 01].

If we assume that the set of constraints has a regular b o u n d a r y so that the outward normal n(x) is uniquely defined at each point of co~, the above definition can be written as

A(x) = {a E A : f ( x , a). n(x) < 0}, for any x 9 01]. Note that the assumption (see w

ma~.~f ( x , a). n(x)

(2.2)

<~

O,

V x 9 cOf~

guarantees that A(x) is not empty for any x 9 ~. Although this is not the easiest way to set the problem from the point of view of the theory of partial differential equations (the dependence of A on x will make it difficult to prove existence and uniqueness results), it can give a hint on the definition of constrained viscosity solution. Assume x 9 f~, then A(x) = A and we see that the equation is satisfied in the open set. Now take x 9 0~, then A(x) c A and we have

Av(x) + s u p { - f ( x , a) . Dv(x) - g(x, a)} aEA

_> Av(x) +

sup { - f ( x , a ) .

D v ( x ) - g(x,a)} = 0

aEA(x)

which means that the constrained viscosity solution is a solution in fl and a supersolution on its boundary. In this way, the state constraints appear in the equation as a constraint on the set of admissible controls at each point. From the point of view of the algorithm, we can still define a discrete time system and a discrete formula for the cost as in Section 2. We just have to define a multivalued map Ah : -~ ~ A which associates with each x the set of controls which are admissible for the discrete system, i.e.,

Ah(x) -----{a E A : x + h f ( x ; a ) E fi}, for every x E ~ . At least when l] is convex and when it has a reasonably smooth boundary it can be proved under assumption (2.2) that there exists a time step h > 0 such that for any h _< h,

Ah(x) ~ O,

x e ft.

The approximation scheme which results from writing that discretization on the mesh is (HJC)hk

u(xi) =

inf aEAh(xi)

{(1-Ah)u(xi +hf(xi,a)) +ht(xi,a)},

i = 1. . . . . L

486

A. NUMERICAL SOLUTION OF DYNAMIC PROGRAMMING EQUATIONS

where xi, i -- 1 , . . . , L is a node of a regular triangulation of fL Let W k denote our usual space of piecewise linear functions defined in (1.9). The following convergence result holds true. THEOREM 2.1. Let (1.3), (1.4), (1.5), and (2.2) be satisfied. Moreover, let A > L f and h e ]0~ l/A]. Then there exists a unique solution v~ of (HJC)~ in W k.

Moreover, v k converges uniformly to v for h, k and k / h tending to O. The proof that (HJC)hk has a unique solution in W k is based on a fixed point argument. In fact, defining Uj = u(xj) and Fj(a) = e(xj,a), it is easy to prove t h a t T : ]I~N ~ ]I~N N

(T(U)) i =- rain ((1 - Ah) E aEAh(xi) l.

Aij(a)Uj + hFi(a)},

iCI

j=l

is a contraction mapping. Also in this case it is crucial to have an acceleration technique to speed up the convergence. TEST 1. We set • --= ]0, 3[2 \ (Q1 [-J Q2), where Q1 = (2, 1.5) + Q,

Q2 = (1,2) + Q A = / ~ ( 0 , 1) c li~~,

and

Q = [-0.2, 0.2] 2,

A = 1,

f ( x l , x 2 , a) = a . Let us consider the cost function 6,

g(x, a) = ((xl - 3) 2 + (x2 - 3)2) 1/2 + 10(1 - x2) + where x - (Xl, x2) C ~. This cost function is very high near the Xl axis and it depends only on the distance from (3, 3) for x2 > 1. The dynamics is such that A(x) is a proper subset of A on each point of the boundary. In the interior points the system changes instantaneously its direction so that it can avoid the obstacles Qi, i = 1, 2, and runs at maximum speed toward (3, 3), the absolute minimum point for the cost. Note that in Figures 4, 5, and 6 the xl-axis is on the left (i.e., it is the vertical axis in Figures 5 and 6) and the value on the obstacles has been set arbitrarily to zero (just to have a better graphical representation). In the computation the unit ball has been approximated by 9 points (8 on the boundary 0B(0, 1) and 1 at the origin). To compute the optimM trajectories starting from previous knowledge of the value function, we have used a one-step approximation scheme for the system of ordinary differential equations. The optimal control to be applied at each point of the trajectory is obtained by looking for a control which gives the minimum in the right-hand side of (HJC) k. T h a t search has been done over 37 controls (36 on the boundary and 1 at the origin). Figure 6 shows the approximate optimal trajectories obtained by that method. They start from several points on the boundary of D and point to the upper right corner, the point (3, 3). Note that they avoid the obstacles and t r y to go around them in a very reasonable way. <~

2. PROBLEMS WITH STATE CONSTRAINTS

F I C U R E 4: V a l u e f u n c t i o n ( h =

0.1, k

F I G U R E 5: L e v e l c u r v e s .

= 0.187).

487

488

A. N U M E R I C A L S O L U T I O N O F DYNAMIC P R O G R A M M I N G EQUATIONS

FIGURE t3: Computed optimal trajectories.

3. M i n i m u m

time problems

and pursuit-evasion

games

3.1. T i m e - o p t i m a l c o n t r o l Another class of classical control problems is that of the so called "target problems". One of the most famous is the following minimum time problem. Assume that the dynamics is given by (1.1) and that the assumptions (1.3)(1.5) are satisfied. We are also given a closed target set if- C IRN, and the problem consists of driving the system to the target in minimum time from its initial position x. It has been shown in Section 1 of Chapter IV that the the minimum time function

T(x) = inf {t > 0: yz(t,~) E T} oE.A

is continuous over the reachable set ~ if a local controllability condition such as

inf f(x,a).n(x)
VxEOT

aEA

is satisfied (here n(x) is the exterior normal to if- at x). Moreover, under that assumption T satisfies the boundary value problem

'ma~{-f(x,a). DT(x)} = 1 in ~ --. T, (3.1)

~[~(xl ; ( : ) = +oc

in'/-, for any x0 E 07~.

3. MINIMUM TIME PROBLEMS AND PURSUIT-EVASION GAMES

489

The following change of variable (see Chapter IV)

v(x) --

{11--e-T(~:) ifT(x)<+oc if T(x) = +oo

is a rescaling of time which is useful to prove that T is the unique viscosity solution of the b o u n d a r y value problem (3.1). In fact, one can prove that the exponential transform v is the unique viscosity solution of the b o u n d a r y value problem

v(x) + sup{-f(x:a). Dv(x)} = 1 in v(x) = 0

~ N \ 'T

for x E

OT.

Note t h a t by the same rescaling we have also eliminated the cumbersome boundary condition on 07~. Applying the Euler discretization to the dynamics we can define a new discrete minimum time problem. T will be replaced by the quantity hNh(x) where Nh(x) is the minimum number of steps necessary to reach the target. As in the continuous problem we can rescale the time, and we can prove by the methods of Chapter VI, Appendix, that the function

{i--e-hlVh(:c) (3.2)

Vh(X) --

ifNh(x)<~oo if Nh(x) ------boo

is the unique bounded solution of (3.3)

I uh(x) = min{e-huh(x + hf(x'a))} + l (Uh(X) = 0

in NN \ T in T .

Naturally, to compute our solution we have to restrict the problem to a bounded set Q and to solve the equation (3.3) in Q \ T. Note t h a t this also means that we need to add an additional boundary condition on OQ. Since the set Q is a r b i t r a r y one can choose a cube in ]~N containing the target set (but of course other choices are possible and can be even more efficient on specific problems). Working on a fixed grid on Q we are lead to the approximation scheme,

[ Uh(Xi) = min{e-huh(xi + hf(x,a))} + 1 - e-h, aEA

~Uh(Xi) = 0 [Uh(Xi) = 1

for i e Iin for i E IT for i E Iout

where /out =- {i : Zi + hf(xi, a) ~ Q for

I7-

-- { i : xi E T N Q }

Iin - i e I ' . . ( /out U IT") .

any a}

490

A. NUMERICALSOLUTION OF DYNAMICPROGRAMMINGEQUATIONS

Naturally, one would like to know if we can apply the same approximation technique to problems where the local controllability assumption is not satisfied and T is discontinuous. In fact, it can be proved that the same scheme can produce a reasonable approximation also in those cases. Let us introduce the time

t(a) = tz(a) = inf{t e [0, + ~ [ : y~(t, a ( . )) 9 int(7-)}, and the function ~(x) = inf{1 ~(-)

e-t(a)},

which corresponds to the exponential transform of the minimum time necessary to enter the interior of 7- and not just 7-. Under rather general hypotheses, it can be shown that our numerical approximation converges to ~ uniformly on any compact set where ~ is continuous. TEST 1. We set Q = ] - 2 , 2[2, the dynamics is ~/= a, A = [-1, 1] and 7- = {0} or 7- = / ~ ( 0 , s). When T = {0} this is the well known example of the rocket railroad car, where the optimal control is bang-bang (see w The optimal trajectories, corresponding to the feedback controls a . = 1 and a* = - 1 , are arcs of parabolas.

FIGURE 7: Value function (h = 0.01, k = 0.15). The graph and the level curves of the computed value function are, respectively, in Figures 7 and 8; Figure 9 shows the optimal trajectories for two slightly different targets. Note that the approximation of the trajectories is also accurate for the pointwise target 7- = {0} (Figure 9a), although this produces a degenerate boundary condition: one can easily recognize the families of parabolas corresponding to the two optimal choices of the control. <~

3. MINIMUM TIME PROBLEMS AND PURSUIT-EVASION GAMES

491

2

1.5

I x

0.5

0

-.,0.5

-1

},,--,

-1..5

-2 -2

-1.5

-1

-05

0

0.5

1

~.5

2

FIGURE 8: Level curves.

(a)

(b)

FIGURE 9: Computed optimal trajectories: (a) 7" = {0} (h ---- 0.01, k = 0.15) and (b) 7" ----/~(0, 1).

TEST 2. We set Q -- ] - 2 , 2[ 2, the dynamics will be J: -- c(x)a, A - - / ~ ( 0 , 1) and T will have different shapes. Let us consider first the simple case c(x) = 1. In that situation the value function coincides with the distance function from the

492

A. NUMERICAL SOLUTION OF DYNAMIC PROGRAMMING EQUATIONS

target. Figure 10 shows the solution for 7- being a rhomb R, the level curves of the minimum time function are smooth as they should be. Figures 11 and 12 shows the m i n i m u m time function corresponding to a nonconvex target L (an L-shaped curve).

FIGURE 10: Minimum time function and level curves for 7- = R and c ( x ) - 1.

FIGURE 11: Minimum time function for 7- = L and c ( x ) - 1 ( h = 0.01, k = 0.25).

3. MINIMUMTIME PROBLEMS AND PURSUIT-EVASIONGAMES

493

,.. .. ....././ ./,.,'/" I ..- .....'-~.~...".,,',,',,.',,,. ', ,. ..." ..." / / ,.'/',,,/.'... .......:..., ', ',,': \ "., ,, , ~

0.75

-" .," 1" ./ /' / ,' //"'/'~"1"~ .... .,."..-'..--"..--.,' ,' ~ ,' '

" " " : ~ ~ ',,!!! i !~~ :' ;

" '"./'-'".f

fill'

0.5

""-"

....

";I

"'./,f ;.''"" J

t

":"////,'" /'////,'~I

0,25

ii',',; ii i ii i i

ildl"

I I I L !

i

0

i

II "

,;i

\ \', ,,,,,,',

i

ll~i

~ i

;

,,'/,.,'//,, ,./ i ,i

...'.//'/',,'/ :" /

. \ \'..',..',..'-..

i

/

i

'

-0.25

", ' " , , "'.... "............................................... ' "

I"

..."" i "

/"

E

/"

t'

-0,5

," ..........~...............ZIZ::I:SZS:SIS::I........... ..-.",.," ,-'" ,." ,,' -0.75

-I

-

-0.75

-0.5

-0.25

0

0.25

0,5

035

'

FIGURE 12: Level curves for IT = / J ( 0 , 1) and c ( x ) =- 1. Let us consider the same problem for IT = / ~ ( 0 , 1) and for c ( x ) = e xlx2. Now the solution does not correspond to the distance function and becomes much more complicated (Figure 13). <~

FICURE 13: Value function and level curves for 2- = / ~ ( 0 , 1) and c(x) = e xlx~.

494

A. NUMERICAL SOLUTION OF DYNAMIC PROGRAMMING EQUATIONS

3.2. P u r s u i t - e v a s i o n g a m e s

Another classical target problem is the so called "pursuit-evasion game". We are given a dynamical system controlled by two players

f[ ~ u )

(3.4)

= x

where, a(t) E A, b(t) E B, A and B are given compact sets. We are also given a closed target set T C R N, and we consider the generalized pursuit-evasion game where the first player "a" seeks to minimize the time T(x) taken by the system to reach 7", while the second player "b" wants to maximize it. The function T(x) represents the lower value function of this game in the sense of Varaiya, Roxin, Elliott and Kalton. As shown in C h a p t e r VIII the Isaacs equation corresponding to the minimum capture time for the game is

minm~l bEB

r(

a,b) DT(x)} 9

=

1

in R \ 7 -

where Tr denotes the reachable set. It was proved in Chapter VIII that the exponential transform of the minimum capture time v is a viscosity solution of the Isaacs equation (3.5)

v(x)+minmax{-f(x,a,b).Dv(x)-l}=O bEB aEA

in ~ N ",. 7- --= 7-c

and that, if v is continuous, it is the unique bounded viscosity solution of (3.5) satisfying the natural Dirichlet boundary condition

v(x) =

0

for x E

07-.

By the standard discretization of the previous sections we obtain the following discrete Isaacs equation (3.6)

~ vh(x)=/3mbaxn~nvh(x+hf(x'a'b))+l-/3

in]~N'7-'

(vh(z)

on 7-,

0

where/3 = e-h. Again, to solve the problem numerically we have to restrict it in a bounded set Q. The choice of this set is arbitrary, the only request being that it should contain T. For example, one can take a cube in ]~N or the intersection of two sets of state constraints Q1 and Q2 (respectively, for the first and the second player). The boundary condition to be imposed on OQ will of course affect the solution (unless Q is invariant with respect to the controlled dynamics), and it can drastically change the result of the game in the interior of Q. W i t h o u t entering into details on the advantages and disadvantages of other possibilities, let us take the same boundary condition we used in the minimum time problem. In the pursuit-evasion game the effect of this boundary condition is more tricky. Setting v = 1 outside

3. M I N I M U M T I M E P R O B L E M S

AND PURSUIT-EVASION

GAMES

495

Q gives an advantage to the evader who can also win by leaving the domain of computation. However, it is sometimes better to know who will have an advantage in the "numerical game" since this will make it possible to interpret correctly the results. For example, with the above "boundary condition" we know that the numerical value function will be greater or equal to the real value function. This means that, if the evader is captured in our simulation, this will also be true in the real game; if not we cannot conclude anything on the real game. Let us consider a mesh over Q and denote by xi its nodes, i = 1 , . . . , L . Let us introduce the following subsets of indices: /out ~ { i : Va e A 3 b 9 B such that zi + hf(xi, a,b) ~ Q}

IT =- {i : xi 9 2- A Q} and /in Let us define the m a p S

: ]l~ L ----* ]l~ L

~

{i : i ~/out tJ I=r} .

componentwise as follows:

{;mbaxminPi(a,b,V)+l-/3 Si(V) =-

ificIin if i 9 lout

if i e lTL

where 13 --= e -h, Pi(a, b, V) = ~ Aij(a, b)Vj and j=l L

L

EA,j(a,b)xj =xi+hf(xi,a,b),

Aij 9 [0,1],

EAij=I. j=l

j=l

Note t h a t the definition of Si for i 9 /out should be interpreted as a discrete boundary condition. It is easy to prove the following PROPOSITION 3.1.

(i)

S : [0, 1] L ~ [0,1]L;

(ii)

S is monotone with respect to the partial order;

(iii) S is a contraction in [0, 1]L with respect to the norm I[yHar =

max IV/I.

i=I,...,L

The unique fixed point of S is our approximate solution on the nodes of the grid belonging to Q. To have an approximate solution defined everywhere we can extend the triangulation over Q and define by linear interpolation an affine function on each simplex; t h a t is, we define w : Q --* [0,1] as the unique function such that

(3.7)

r w(x)= Z Ajw(xj) J ! ~w(xi)=~n~xminw(xi+hf(xi, |w(xi)

= 1

[,w(xi) = 0

if x = E Ajxj,

J

a,b))+l-Z

if i e / i n

if i 9 if i 9 I7 9

496

A. NUMERICALSOLUTIONOF DYNAMICPROGRAMMINGEQUATIONS

The condition w ( x i ) -- 1 off Q means that the first player loses the game if the system exits Q before reaching the target. This is a natural approximation for the case of the single player or for bounded targets, but it is not realistic for true pursuit-evasion games. It would be more realistic to consider a game with state constraints in Q in which each player loses the game if his state variables exit Q first. Under local controllability assumptions v is continuous and our approximation w will converge to v uniformly on any compact set of ~N. The technique of the proof of that convergence result is based on weak limits (see Chapter V). As a final remark, we just observe that the convergence of the same scheme is also valid for games where v is discontinuous (i.e., in presence of barriers). In those cases, we expect the convergence to be uniform only in the regions where the solution is continuous. 3.3. N u m e r i c a l t e s t s Let us analyze the numerical results related to two simple examples of 1-dimensional pursuit-evasion games for which the solution can be computed exactly. In both cases the set Q is invariant for the controlled dynamics so that the changes to the solution due to the artificial boundary condition do not appear. The first game has a continuous value function while the second has a slight modification in the dynamics which produces discontinuities (barriers). TEST 1. We consider the following dynamics for the players x' = a(x

(3.8)

-

1)(x + 1)vl

y' = b(y - 1)(y + 1)v2

where x, y are the (1-dimensional) state variables respectively of the pursuer and the evader and vl, v2 are two positive real parameters representing their relative velocities. We choose A = B - [-1, 1]. We study this game in Q -- [-1,1] 2 which is invariant with respect to the trajectories and we set T - { ( x , y ) : x = y } . Starting the game from (x0,Y0), x0 < y0 the optimal strategies for the evader and the pursuer will be b* - - 1 and a* --- - 1 , since this corresponds for both to the motion toward 1 at the maximum speed. We can solve explicitly (3.8) obtaining 1 - k2e -2vlt x*(t) -

1 + kl e-')vlt

1 - k22e-2v2t '

y*(t) -- 1 + k2e -2v2t '

where kl = ([xo - l[/[xo + 11)1/2 and k2 ~ [Yo - ll/]Yo + 111/2Setting x * ( t ) = y * ( t ) we can compute the value function of the game. When Vl <__v2 that equation implies xo = yo; then the game has no solution and T ( x ) = +ec off the diagonal of Q except for (xo, yo) E OQ A ({y = - 1 } u {y = 1}) where the game always has a solution. For vl > v2 the capture time is t* =

1

2(vl - v2)

in ( k l / k 2 ) ~

3. MINIMUMTIME PROBLEMS AND PURSUIT-EVASIONGAMES

497

and by the exponential transform we get the value

v(xo, yo) = 1 - \ k 2 ] valid on {(x, y) : x < y} N Q. This will also be the unique solution of the Isaacs equation in that subdomain.

FIGURE 14: Value function.

FIGURE 15: Level curves.

498

A. N U M E R I C A L SOLUTION OF DYNAMIC P R O G R A M M I N G EQUATIONS

For x0 > y0, repeating the same argument we get

( k 2 ) 1/(v2-vl) ~1

v(xo, Y o ) = l -

For vl > v2, the exact solution is continuous and the approximation is accurate, in fact the maximum error on the nodes in int(Q) is 0.05962329. Figures 14 and 15 show the numerical results obtained by applying the algorithm on a grid of 1849 nodes (k = 0.04) and h = 0.01. <~ TEST 2. Let the dynamics of the pursuer and the evader be given respectively by

x' = a x ( x

-

1)(x + 1)vi

y' = by(y - 1)(y + 1)v2 where a, b, Vl,V2 have the same meaning as in the previous test and A = B [ - 1 , 1]. We study the game in the square Q -- [-1, 1]2 (which is always invariant for the dynamics) and set 7" - {(x,y) : x = y}. We divide the square into four regions by the lines x = 0 and y = 0. In R2 = [ - 1 , 0] x [0, 1] and R4 = [0, 1] x [-1, 0] the game has no solution and T ( x ) - +oc. In the remaining two squares R1 and R3, the game has a solution depending on the values of Vl and v2.

FIGURE 16: Value function. As in Test 1 we can compute the exact solution, for example, for a starting point (x0, y0) C R1, xo < y0, assuming vl > v2. This solution is

(~I~1/(v2--Vl) v(xo,Yo) = 1 - \k2,]

4. SOME HINTS FOR T H E C O N S T R U C T I O N OF T H E A L G O R I T H M S

499

where

oo- 11 ~/Ix~ - 1[ , k 2 -= x0 Y0 Figures 16 and 17 show the numerical results obtained by applying the algorithm on a grid of 1849 nodes (k = 0.04) and h = 0.01. kl-

FIGURE 17: Level curves. Note that the detection of R2 and R4 is quite accurate (they correspond to the region where the solution is flat since v - 1 there). There is a little smoothing effect which rounds up the discontinuities of the solution, but in R1 and R3 the approximation is accurate when Vl > v2 and the solution is continuous. The maximum error on the nodes belonging to the interior of the regions Ri, i -- 1 , . . . , 4, is 0.05754636. <~

4. S o m e hints for the construction of the algorithms The general algorithm to compute the value function is essentially based on a fixed point argument. Once we have an approximate value function we can use it to construct approximate feedbacks and approximate optimal trajectories at any point in the domain of computation. For example, dealing with the infinite horizon problem this can be simply obtained by taking

a*(x)

-- arg min{(1 aEA

Ah)v(x + hf(x, a)) + hg(x, a)}

500

A. NUMERICALSOLUTION OF DYNAMICPROGRAMMINGEQUATIONS

and in the pursuit-evasion game taking the couples (a*(x), b*(x)) which give the saddle value (maxmin) on the right-hand side of equation (3.6). Note that the algorithm gives an approximate optimal control at each node of the grid without extra computations. In this section we will give some hints and suggestions on different aspects connected to the construction of the algorithm. For a detailed list of references on the single topics we refer to the Bibliographical Notes.

Acceleration algorithm The acceleration technique is crucial to get a fast solution with a reasonable accuracy. There are several ways to accelerate convergence. The first (described in Section 1) is based on the monotonicity properties of the operator T and requires use of the contraction mapping iteration only to recover a direction of displacement and reach the boundary of the set of subsolutions b/. Note that the boundary of L/is where Ui = (T(U))i for at least one i; this implies that it can be reached either by an iterative root finding procedure (e.g., bisection) or by an explicit characterization of ~. Naturally, the second alternative is much more efficient (when feasible); the iterative procedure in fact requires several evaluations of the operator T to determine if we are still in or out of the set U. Practical experience tells us that the algorithm works better when the accuracy in locating points on the boundary of U is high (although this will require more iterations in the root finding procedure). A different acceleration technique which can be coupled with the previous one is based on the so called "iteration in policy space". We fix the controls at the nodes of the grid and compute a value function corresponding to them; then we fix the value and look for the corresponding optimal feedback controls on the grid. An intermediate choice in the algorithm is to compute the solution by the contraction mapping, keeping the approximate feedback controls fixed for a number of iterations (e.g., ten). This choice results in a considerable gain in CPU time.

Boundary conditions We have seen how to treat Dirichlet boundary conditions and state constraints boundary conditions. A simple way to automatically enforce state constraints boundary conditions is to extend the solution outside the domain of computation so that its value is greater t h a n its maximum in ~. Since the fixed point operator always looks for the minimum over the control set, this will imply that only the control satisfying the constraints will really play a role in the computation. This is exactly what we have done setting the solution equal to 1 outside Q in the minimum time problem (remember that in that problem 0 < v < 1). It is also possible to enforce Neumann type boundary conditions. Essentially this can be done by adding to the domain of computation a "frame" of width A x so that one can compute a discrete normal (or directional) derivative.

4. SOME HINTSFOR THE CONSTRUCTIONOF THE ALGORITHMS

501

Evolutive problems In some problems (see w the value function is characterized in terms of an evolutive Hamilton-Jacobi equation and we have to deal with a Cauchy problem

ut + H ( x , t , u , Du) = O,

u(x, o) = uo(x). The scheme corresponding to it is u n+l + H A ( x , t, u, Du) = 0,

u~

= ~0(x),

where the discrete Hamiltonian H A is computed using the space discretization discussed above. Note that the above mentioned evolutive problem can also be seen as a problem in ]~g+l where the state is z = (x,t), just adding to the dynamics one equation, namely t t = 1. The finite horizon problem in [0, T] leads to an evolutive Hamilton-Jacobi equation with a terminal condition. Problems of this kind can be solved backward or can be transformed to a Cauchy problem by a change of variable.

Local g r i d r e f i n e m e n t s When we know (for example on the basis of an analysis of the characteristics) where are the switching curves for the optimal feedback controls, we can use an adapted grid to have a greater accuracy in the computations around those curves. A more sophisticated (and efficient) method consists of modifying the grid as far as the algorithm computes the solution on the basis of a posteriori local error estimators. This method can be particularly efficient, especially when the solution is not differentiable, and can produce accurate results also for meshes with a small number of nodes. High-order methods Although the results which have been presented here through some model problems are based on the simplest approximation schemes for the dynamics and the cost functional (we coupled the Euler method and the rectangle rule), one can improve the global efficiency of the algorithm by coupling different methods. In particular, a careful coupling of different higher order methods can result in very accurate approximations also on course grids. The coupling must satisfy some compatibility conditions and is effective as far as the methods used for both the dynamics and the costs produce local errors of the same order. For example, a good balance is obtained coupling the Heun method with the trapezoid rule. These methods are particularly effective in regions where the value function has some smoothness properties.

502

A. NUMERICAL S O L U T I O N OF DYNAMIC P R O G R A M M I N G EQUATIONS

Synthesis of feedback controls and optimal trajectories The main goal of our computations is to recover approximate controls and optimal trajectories. This task is particularly simple once we know an approximate solution w of the Hamilton-Jacobi equation. Knowing w we can compute the feedback corresponding to point x E Q by simply looking at the set of controls A*(x) giving the minimum (or the maxmin) in the fixed point operator. If that set is not a singleton we can introduce an additional criterion to make a selection. For example, when A is convex we can choose among them the control with minimum norm or the control which minimizes a strictly convex function. The unique feedback control can be used now to obtain an approximate optimal trajectory simply by applying a one step scheme to the dynamics, i.e., (4.1)

~Yn+l = Yn + h ~ ( h , y,~, a*(y,~)) l

(y0 = x where 9 is the numerical reconstruction of the vector field (i.e., the Henrici function of a one-step method). Moreover, along the trajectories we can introduce also a "dynamic selection criterion" which consists of keeping fixed the control as far as the control which was optimal in the last step is still optimal at the new point. This "inertia" has an important stabilization effect on the approximate optimal trajectories. Domain decomposition and parallel algorithms We can considerably enlarge the number of problems which can be solved by the Dynamic Programming approach using a domain decomposition strategy. As we mentioned in the introduction, Dynamic Programming has a major drawback in the "rise of dimension". In many real problems the number of variables is easily greater than three, and solving a partial differential equation in such a large dimension is a very complicated task, easily beyond the possibilities of the last generation computers. However, splitting a problem given over a domain ~ into a sequence of problems set in subdomains of manageable size ~r, r -~ 1 , . . . , d, is the basic idea of domain decomposition techniques and can be really helpful. Roughly speaking, instead of one problem in dimension N we will compute the solutions of d problems each of which has dimension Nr, r = 1 . . . . . d, where N ~ N1 + ' " + Nd. Of course, when splitting the problem, one has to decide which boundary conditions have to be enforced on the internal interfaces. As has been proved recently, there are transmission conditions which guarantee the continuity of the value function. 5. B i b l i o g r a p h i c a l

notes

As we mentioned, other approximation schemes can be adopted. Some of them go back to the Bellman school as [La67] or have appeared before the development of the theory of viscosity solutions. The interested reader should look at the book by Kushner [Kus77] (for stochastic control problems) or at the paper by Gonzalez-Rofman [GR85]. We should also mention that there is a large amount of

5. B I B L I O G R A P H I C A L NOTES

503

literature dealing with numerical methods for optimal control problems based on the Pontryagin principle; we refer for that approach to [BH87, Pes94a] and references therein. More recently, the approach to control problems based on viability theory has also produced some algorithmic results as in [SP94] and [QSP95]. After the foundation of the theory of viscosity solution, the approximation of Hamilton-Jacobi equations was studied by Capuzzo-Dolcetta [CD83], Crandall and Lions [CL84], Souganidis [Sou85a], Capuzzo-Dolcetta and Ishii [CDI84], Falcone and co-authors [Fa87, Fa94, FGL94, BFg0a, BFg0b, FG94] (see also the survey paper [CDF89]), and Rouy [Rou92]. Theorem 1.3 and 1.4 are in [Fa87]. The assumption A > L/may be restrictive in some cases and can be dropped by refining the proofs. The convergence of the scheme can be proved by the method of weak limits in the viscosity sense as in Chapter VI, w see [BF90a, BS91b] and [BFS94] while an estimate of the convergence rate has been obtained by the maximum principle argument of Chapter VI, w see [Sor97b]. High-order approximation schemes have been proposed recently either by adapting schemes used in the study of hyperbolic problems as in [OS91] or by applying Dynamic Programming arguments as in [FaF90, FaF94, FaF96]. More recently, a convergence result for some high-order schemes ("filtered" to preserve an upper bound on weak second order finite differences) has been obtained in [LS95] for Hamilton-Jacobi equations with convex Hamiltonians. Other approximation schemes are based on the discretization of the representation formulae for viscosity solutions of Hamilton-Jacobi equations. They use a fast Legendre-Fenchel transform and their application has been studied mainly for evolutive equations with convex Hamiltonians in [Bren89] and [Cor96]. A result on the convergence of approximation schemes in an abstract setting is contained in Barles and Souganidis [BaSo91]. The acceleration technique was introduced in [Fa87]. Another acceleration technique has been studied in [GS90]. The acceleration in policy space goes back to Howard [Ho60]. A dynamic grid refinement was introduced and implemented by Criine [Gru97]. The convergence analysis and the a priori estimates for the schemes related to constrained problems are contained in [CFa96]. For a similar analysis related to the minimum time problem we refer to [BF90D] (see also [BSt93], [BBF95] and [BFS97] for a convergence result related to the discontinuous case). F~rther extensions to evolutive problems and impulsive control problems have been developed in [FG94, CFa95b] and [Tigl]. The numerical solution of the Isaacs equation was studied by Alziary [A191] for the classical problem of a lion pursuing a slower antelope in a bounded and closed domain. Gonzalez and Tidball [GTig0, Ti91, TIC94, Ti95] worked on deterministic games with stopping times (see also the references therein). The convergence results for the generalized pursuit-evasion game are contained in [BFS94] (for the continuous case) and in [BBF95] (covering the discontinuous case). Let us also mention the paper [Sou85b] dealing with the convergence of approximation schemes for finite horizon games. The domain decomposition strategy with overlapping regions between the subdomains was studied in [FLS94]. An extension to the

504

A. NUMERICALSOLUTIONOF DYNAMICPROGRAMMINGEQUATIONS

case without overlapping was considered in [CFLS94]. Sun [Sun93] studied similar techniques for stochastic control problems. Finally let us mention optimal control problems for diffusion processes and the corresponding second order Hamilton-Jacobi-Bellman equations. Approximation schemes for those problems were studied in [Kus77, KD92, LMrS0, QS0, FS93] and [CFa95a]. Specific applications of multilevel techniques for second order HamiltonJaeobi equations were studied in [Hop86] and [Akg0]. More recently, in [CL96] new schemes for evolutive second order Hamilton-Jacobi equations of degenerate type (with applications to the motion by mean curvature) have been proposed.

Acknowledgments I wish to thank A. Seghini, T. Giorgi~ P. Lanucara, F. Camilli and all the students that have contributed to the development of the codes and to discussions of the theoretical and the algorithmic issues.

APPENDIX B N o n l i n e a r 7-/~ c o n t r o l by Pierpaolo Soravia

This appendix deals with a recent subject of systems theory which has been growing in popularity in recent years. We will present some results concerning the problem of robust attenuation of disturbances in controlled systems and the so called nonlinear T/or control problem. Our goal is to show how the theory of viscosity solutions can be applied to this problem and contributes to its understanding. The idea of the problem is to introduce a quantitative way to measure how disturbances affect a stable or asymptotically stable equilibrium of a controlled dynamical system. The point of view is entirely deterministic as opposed to the probabilistic one of stochastic control; the so called worst case approach, though historically important, is less developed and acknowledged than the probabilistic one, but has recently found a number of important applications in the disciplines that make use of control theory. Despite of the different nature, there is an interesting link between these two methods which is known as risk-sensitive control. This developing chapter of control theory goes however beyond the scope of this appendix. We set ourselves in the general framework of nonlinear systems, where the theory of viscosity solutions plays a crucial role, and refer to the literature for a more detailed discussion of specific results that can be proved for special subclasses of systems. Hence, we do not presume to give a complete treatment of the up-to-date theory of the subject here, but we intend to point out to what extent the theory of viscosity solutions is needed and helpful to approach the problem in great generality. We are concerned only with the state space formulation, while the original setting for linear systems was the frequency domain. We will not discuss the partial information case in great detail, as a fully satisfactory framework for this important side of the problem, for general nonlinear systems, is not yet developed. In the following we indicate the system in the form

(0.1)

{~ =

f ( y , a, b),

e(y,a,b) .

y ( o ) = z c ~t W,

506

B. N O N L I N E A R

7-/~r C O N T R O L

Here as usual y ( . ) = yx(';a, b) denotes the state, a ( . ) the control, b(. ) the deterministic unknown disturbance on the system, or input, and z ( . ) the cost function. Later in the chapter we will add to the system (0.1) the equation (0.2)

w = g(y, a, b),

where w( 9 ) is called the observed output. Very special cases of the above are the following: linear systems

~f(x,

(0.3)

a:

b)=Cx+Pa+Qb,

I,e(x, a, b) = [/(x, a)l",

e(x, a)

=

Dx + Ea;

and nonlinear-affine systems

~ f(x, a, b) = fl (x) + f2(x)a + f3(x)b, [~(x,a,b) = [[~(x,a)lP, [(x,a) = gl(X) +~2(x)a,

(0.4)

where C, P, Q, D, E are matrices and f l , f2, f3, ~1, g2 are suitably defined matrix valued functions. In both examples the function 5 ( . ) = [ ( y ( . ), a ( . )) corresponding to a trajectory y ( . ; a , b ) of (0.1) is called the output of the system. The real number p E [1, +cc[ that appears in (0.3), (0.4) or in the next assumptions, will identify the class of admissible controls and disturbances. Traditionally p = 2, but this choice plays no role in our discussion. 1. D e f i n i t i o n s We now list the basic assumptions we need. In the following we are given the sets A, B, where A c ]RK and 0 E B C 1RM are closed, the maps f : ]~Y • A x B --* I~N, g : N N x A x B ---* ] ~ P ~ ~ : ]~N • A x B --~ ]R are continuous. We assume that the following is satisfied: there are p, q, 0 < q < p, such that

f lf(x,a,b) - f(y,a,b)l <_ L(1 + lal q + Iblq)[x - Yl, | l I ( x , a , b ) l < L ( l + l x l + l a l q + l b l q) for allx, y,a,b, ~lg(x,a,b) - e(y,a,b)[ < LR(1 + laF + IbF)[x - Yl, |C~ - LR(1 + lal q + ]blp) _< e(x,a,b) < Ln(1 + lal p + Iblq), |

(1.1)

!

( f o r some C~ > 0, for all Ixl, lyl -< R a C A, b E B, R > 0 . Moreover the following sign condition on the cost is assumed (1.2)

e(x,a,O) >_ O.

The growth conditions we impose on the system lead naturally to consider the following sets of admissible controls and disturbances .4 = LPoc(IR+, A).

= L~oc(R+, B)'.

1. DEFINITIONS

507

In the following by undisturbed system we mean the controlled system ~ = f(x, a, 0),

(1.3)

y(O) = x,

for a E .4. One of the conditions we want on the undisturbed system (1.3) is t h a t it is open loop Lyapunov stable at the origin in the following sense: for any neighborhood L / o f the origin, there is an open set ft containing the origin such t h a t for all x E f t there is ax E .4 satisfying

yx(t; ax, 0) e U,

for all t >_ 0 .

As a m a t t e r of fact, we will not require the stability of the system in advance but prefer to prove it as a consequence of other assumptions. We will test such stability with the cost function e above, see Definition 1.1 and Remark 1.2. W i t h o u t other assumptions on the cost e t h a n (1.2), open loop stability is the natural request for the undisturbed system. If instead we want the undisturbed system to be open loop asymptotically stable, then it would be more appropriate to suppose that also the following condition holds true, there are Q, 5 > 0 such that if Ix[ < 5, a E .4 satisfy

[y~(t,a,O)iOandt_.+~lim dist(y~(t;a,O),Z)=O

(1.4)

then

lim y~(t;a,O) = O.

t~+oc

In (1.4) we indicate with Z = {x E ]~N : infae A g(x, a, 0) = 0} the degenerate set for the cost. Here and in the following, we implicitly assume that 0 E Z. The condition (1.4) is a modification of the property of controlled systems known as zero state detectability. Note t h a t (1.4) holds, in particular, if 0 is isolated in Z. DEFINITION 1.1. Given 7 > 0, we solve the 7-/cc suboptimal control problem with disturbance attenuation level 7 if there are a nonnegative function U : ] ~ g ...., ]l~, null at the origin and a family of strategies {OLX}xE~NC r (i.e., nonanticipating or causal functionals a : B --~ ,4 in the sense of Chapter VIII) such that the two following conditions are satisfied: (i) the controls az[0] provide the open loop Lyapunov stability of the undisturbed system, (ii)

(1.5)

/0'

e(y~,c~x[b],b)ds<_,~P

/0

IblPds+g(x),

forallxEIRN, t>_O, b E B .

We say t h a t the problem is solved strongly if the following conditions are satisfied. There is a feedback map a : IRN --* A such that for all b E B and x E R N there is a unique absolutely continuous solution of the differential equation

~/= f(y,a(y),b),

y(0) = x,

this closed loop dynamical system is Lyapunov asymptotically stable at the origin when b - 0, and the definition a~[b](t) := a(y(t)), gives the family of strategies solving the problem. ,~

508

B. NONLINEAR 7-/~ CONTROL

REMARK 1.2. As usual in differential games, in full generality, one has to give some advantage to one of the players in order to have a tractable problem by means of the Dynamic Programming approach. Therefore in the definition above we used strategies for the controller and controls for the disturbance. The problem as defined above, is usually referred to in the literature as the full information problem. When the Isaacs condition on the data holds, then one would prefer to restrict the class of admissible strategies to feedback strategies, or causal functionals of the state. This is referred to as the full state information problem. When only property (ii) of Definition 1.1 holds, we say that the system has finite L p gain. The terminology is due to the fact that when the cost function is expressed in terms of an output ~, as in the case of linear (0.3) and nonlinear-affine (0.4) systems, then at x -- 0 the inequality (1.5) can be rewritten as

IIrobIILp(0,t) < ~, IlbllLp(O,t)

for all t > 0, b E B,

where for the given strategy a = a0 the map Fab = ~ is the so called input-output map. Observe that if the system is linear and ct is given by a linear feedback, then also Fa is linear and so y is an upper estimate for the norm of such functional. Actually in this case the real number IIF.II is the best value of "~ one can use in (1.5) for the given strategy a. In this case it is well known that the norm of Ya coincides with the 7-/~c norm of the transfer function in the Hardy space of analytic functions bounded in the right halfplane. As stability is a local property, we can define in a suitable way the local ~cr suboptimal control problem. We do not address here, however, this part of the theory. If we solve the 7-/~c problem in the sense of Definition 1.1, then by the stability of the undisturbed system at x = 0, it is easy to check that Y0( 9; a0, 0) ~- 0, therefore f(0, a0[0](.),0) = 0 a.e. Moreover, by (1.5) and (1.2) we also have g(0, a0[0](. ),0) = 0 a.e. Therefore there is at least one a c A such that simultaneously f(0, a, 0) -- 0 and t~(0,a, 0) = 0. Thus the origin must be an equilibrium point for the controlled system (0.1) when no disturbances are present. This is a necessary condition on our system in order to solve the problem. ,3 We now describe the connection of the problem as formulated above and differential games. It is clear that (1.5) is equivalent to stating that sup sup bEB tEN+

2

(g(y~, ax[b], b) - "/VlblP) ds <_ g(x)

for all x E ]I~N .

We are therefore led to introduce the following (lower) value function for the differential game (0.1) (1.6)

V~(x) = inf sup sup f t ( e ( y x , ax, b) - ~PibIp) ds. ~EF bEB tEN+ ,]0

This value function gives by construction the best value U(x) to be used in (1.5). We skip for the moment the problems related to the existence of the value of the

1. D E F I N I T I O N S

509

game and limit ourselves to the lower value approach. We will come back to this issue later. Observe t h a t by choosing 0 = t C ]~+ in (1.6), we immediately check t h a t V~ > 0, but this function may a t t a i n the value + c o at some points. The previous discussion leads to the following definition. DEFINITION 1.3. Given "y > 0, the ~ suboptimal control problem with level "7 is solvable if the undisturbed system is open loop Lyapunov stable and the value function V~ is finite in R N, and null at the origin. <~ Clearly Definition 1.1 is more restrictive than Definition 1.3. We will show however in the following that the solution of the problem can be pursued by investigating the properties of the value function V~ and that the two definitions are in fact equivalent in some cases. As we will see in the examples, the solvability of the problem will depend on the choice of the index % As a m a t t e r of fact, the main goal of the theory is to compute the real number "y* = inf{'y > 0 : the problem with index "~ can be solved}, since it is a measure of the robustness of the system and, on the opposite hand, of the influence that disturbances have on its behavior. Unfortunately, there are no general effective methods to our knowledge~ to characterize the value "~*, and we will only characterize here the solvability of the problem for a fixed index "7. EXAMPLE 1.4. Consider the following one dimensional system

{ ~ = a y- b( O, ) =2.x C ] R , y 2 + a In this case A -- B = ~ and the system is clearly linear according to (0.3), by choosing D T = (1 0), E T -- (0 1) (T denotes transposition). We set p = 2 and prove that for all "~ > 1 we can solve the ~ r suboptimal control problem. Since we have not developed any better method yet, we guess and use the following strategy, (1.7)

a[b](t) =

-y(t) + b(t)X{b:y(t)b<_o}(b(t) ) ,

where Xs( " ) denotes the characteristic function of the set S. The second term in the right-hand side of (1.7) cancels bad disturbances t h a t push the system away from the origin, while the first is a stabilizing term. It is easy to check that, using the strategy defined in (1.7), the trajectories of the system satisfy (1.8)

[y(t)l _< Ixl exp ( - t ) ,

for t > 0. It is also easy to verify t h a t the function r

2by)C{by
I'

= (X{yb_<0)(b) - ~/2)b2 -

b = y/(1 -.~2).

( 2y2 + (X{yb_
Therefore

510

B. NONLINEARH~r CONTROL

< sup

- reliC_

f0 t (

2y ~ +

< sup fo t x 2 - tEP~+

y2 ....

exp

2y2 ,

1 - ,./2

1~_ ~-g]

2~ 2 - 1

ds 2~,~

(-z~)--~_l ds <_ x 2 2(7 2 --

1 I) '

from which we get the conclusion. Note that since V- r > 0, the previous tion verifies that the strategy a[b] = b is optimal at the origin. The same also optimal at the origin when 7 -- 1 by a direct computation, therefore On the other hand the estimate above does not work for 7 ~ I. We will

computastrategy is I/'i(0) = 0. come back

later to this example.

<3 2. L i n e a r s y s t e m s

For linear systems (0.3) the definition we gave in the previous section can be simplified because, as often in the linear case, information at the origin propagates to the whole space. In this section we assume that

(2.1)

there is a linear feedback a(x) = F x such that the undisturbed, closed loop system = (C + P F ) y , y(O) = x, is asymptotically stable at the origin

and,

(2.2)

for all b E B, the solution of the differential equation = (C + P F ) y + Qb, y(O) = O, satisfies the L p gain condition

/0

I(D + E F ) y f dt <_ 7 v

/0

[b[p dt, for all b E Lv(N+, B ) .

The two previous conditions will be recalled as (2.2). We will see in the following, more precisely in Remark 6.3, that the 7-/or suboptimal problem for linear systems can be solved in the strong sense under certain conditions, and feedbacks solving the problem are linear. Therefore the assumption (2.2) is not restrictive in the linear case. PROPOSITION 2.1. Assume (2.1) and (2.2). Then the 7t~ suboptimal control problem is solved strongly by the feedback map a(x) = Fx. PROOF. We will give the proof only under the following simplifying assumption: the system (2.3)

{ ~ = R yy§( O p) = , X,,sy,

where we indicated by R = C + A F , S = D + E F , is controllable to the origin namely for all x E N N, there is bx E 13 such that the solution Y0( 9 ; bx) of (2.3) with null initial condition satisfies yo(tz; bx) = x for some t~ >_ 0.

2. LINEAR SYSTEMS

511

Let x E NN be fixed. By the simplifying assumption there are a control b E B and T > 0 such t h a t y0(T;b) = x. Now let b E B; we define the admissible control

[b(t), bl(t) = [b(t - T),

0 < t < T, t > T.

Observe that yo(t; bl) = yx(t - T; b) if t > T. By (2.2) we then get for t _> T

o ~ ~0 t (]Syo(s;b~)lP- ~Plb~(s)lP)ds =

([Syo(s; b)l p - 7Plb(s)[ p) ds +

(ISyx(S; b)lP - "~Plb(s)[p) ds,

therefore the L p gain condition (1.5) at x is satisfied with the choice of U(x) = ( - f[(ISyo(s;b)]P - ~/P[-b(s)[p) ds) v 0. Observe that T and b depend only on x. *l REMARK 2.2. Unfortunately, at least to our knowledge, a complete proof of the maybe surprising Proposition 2.1 is not trivial. It also requires some results t h a t we will state in the following, namely Theorem 4.5 and the classical statement of linear state space ~or control theory stating t h a t (2.2) is equivalent to the existence of a positive definite solution of the Riccati equation associated with the problem. To give the flavor of the complete result, we can also make some direct computations based on the linearity of (2.3). In particular for all b C B the solutions of (2.3) satisfy

yx(t; b) = exp (Rt)x + y0(t; b). Therefore

[Sy~(t)[ p = [Sexp (Rt)x + Syo(t)[ p <_ 2 P - l ( I S e x p (Rt)x[ p + ISyo(t)[P) . If we now fix any t _> 0, indicate by U(x) = 2 p - I I s I P f o [exp(Rt)xiPdt, for any b C B denote b = bX[o,t], and use (2.2), we get

fn t [Syx[ pds <_ U(x) q- 2p-1 .~t [SYo]p ds ~ U(x) q- 2P-l~ p

[blp d s = (21-1/vT)v

Ib[pds + g(x),

which ensures t h a t the problem is solved according to Definition 1.1, though with the larger attenuation level index 21-1/P3 ,. <:l The simplified version (2.2) of the definition (with p = 2) is the usual definition one can find in the literature. We will verify later t h a t the result of Proposition 2.1 is false in general if (2.2) is not guaranteed by a linear, asymptotically stabilizing feedback control.

512

B. NONLINEAR ~ =

CONTROL

EXAMPLE 2.3. We now apply Proposition 2.1 to the system of Example 1.4 and solve the corresponding 7-/~r problem strongly. A linear, asymptotically stabilizing feedback control for the undisturbed system has necessarily to be of the form a(x) = - c x , with c > 0. In this case, the trajectory solution with zero initial state is given by the formula y0(t; a, b) = -

//

exp ( - c ( t - s))b(s) d s .

Therefore by the estimates on convolution, we have that, for all b E L2(R+, B),

(y2 + a2)dt = (1 + e2)

y2dt

]bl2dt)

< ( 1 + c 2)

exp(-ct)dt 2 - 1 +c2c 2

Ibl2dt"

From Proposition 2.1 it follows that for any "~ > 1, choosing the constant c satisfying c k 1 / ( ? 2 - 1) 1/2, the feedback above will solve the problem. Actually we find infinitely many feedbacks solving the problem, which is due to the fact that we are not requiring their optimality when the system starts at a point different from the origin. We will see in Example 6.4 that, when doing this, the selection will become much more stringent. Note that the strategy at the point x E I~N corresponding to a choice of c is given by aC[b](t) = a(y(t)) = c

(//

exp ( - c ( t - s))b(s) ds - exp ( - c t ) x

/

,

which, in particular, is optimal at the origin and we can compare with the guess (1.7). Observe also that even if 1/'1(0) = 0, the computation above does not guarantee the assumptions of Proposition 2.1 and therefore we cannot deduce in general the stronger estimate (1.5) when ~ / = 1. <~

3. ~ r

control and differential games

In this section we motivate the introduction of the value function V~ and show that the existence of a solution of the 7-/~r problem can be established by studying V~. The next results also show some simple conditions on V~ under which Definitions 1.1 and 1.3 are equivalent and how stability of the undisturbed system can be proved by the properties of the value function V~. Here and in the following we indicate, as in Chapter V, by (V~).(x) = limr~0 infly_zl<_T V~(y), the lower semicontinuous envelope of the value function V,. LEMMA 3.1. A s s u m e (1.1) and (1.2). Suppose that the system has finite L p gain and the value function is continuous at the origin. A s s u m e that there is a > 0 such that f o r all x e B ( O , a ) \ {0} we have Vv.(x) > O, and V, has an optimal strategy at the origin. Then we can find a family of strategies {ax}xe~N and a

3. 7-/~ CONTROL AND DIFFERENTIAL GAMES

513

function U continuous at the origin, solving the 7-toe suboptimal control problem. If in addition (1.4) holds, then the controls {ax[O]}xe](N provide open loop local asymptotic stability of the system. PROOF. 1. In the assumptions of the statement, we consider a basis of open and bounded neighborhoods of the origin .T = {/4.}he N such t h a t /A~+I CC Ll,~ C B(0, a), for all n E H. Given m0 -- 1, we define E1 = inf{V~(x) : x E 0b/m0}. Note t h a t zl > 0. This follows from the assumption V~. > 0 on B(0, a) \ {0}, the fact that 0 ~ r and V~. is lower semicontinuous. Then set fl~ = {x E b/m0 : V~(x) < ~1} which is a neighborhood of the origin by the continuity assumption. Next given ~t~, we set m l -=- min{k :/4k C ~ 1 } v (rn0 + 1). In the general case, given rnn E N, we proceed and define Zn+l, fle~+~ and m,~+l as above with b/re. instead of b/m0. The family 5r = {~t~,+~ }hen is a nested basis of neighborhoods of the origin. Now we construct the strategies and the function solving the suboptimal problem. For x E ~ N \ ~ , ~ > 0, let ax be such that s u p s u p f/ t (g(y,a~,b) --

721b12)ds _< V~(x) + E,

bEB tEI~- JO

and define U in R N \ ~'~1 by setting U = V~ + ~. For x E gt~n \ 12~,+1, we choose ax such that sup sup ~0 t (e(y, ax, b) - ~/21bI2) ds < E~, beB t e ~

(3.1)

and set U(x) = sn. Finally at x = 0 we choose a0 as an optimal strategy for V~(0) and set U(0) = 0. Clearly by construction cn ---* 0 as n ~ +oc, therefore U is continuous at the origin. 2. We have left to prove that the family of strategies {o~x}xE[(g provides open loop stability, or open loop asymptotic stability of the undisturbed system if (1.4) holds. We start with the stability of the system. Given a neighborhood 1; of the origin, we choose mn such that Umn C 1;, then for any x E ~ , + 1 C Umn, from (3.1) and the definition of V~(yx(t)), it follows that the chosen ax also satisfies

~o

t(t(yx,ax,b)-'~2ibl2)ds+V~(yx(t))<en+l,

for all t_> 0, b E B .

Therefore choosing b = 0 and using the assumption t(z, a, O) _> O, we get

V~(yx(t;az[O],O)) < E,~+, for all t > 0 . We conclude that there is no t such that yx(t;ax[O],O) E OL(m~, as required. 3. To prove the local asymptotic stability, we only need to show that for any x in a suitably small neighborhood 12 of the origin we have lim dist(yx(t;ax,O),Z)=O,

t~-)-or

and

[yx(t, ax,O)[<_Q,

t>_O.

514

B. NONLINEAR7-(or CONTROL

We then reach the conclusion by (1.4) and the stability. Let x E f2s1. By definition of ax and (1.2) we have that

fo +~ ~(Yx, ~x, 0) ds < +co, so we already know that liminft~+~r dist(yz(t; az[0], 0), Z) = 0 and by part 2 of the proof yz(t) E llo for all t >__0. Let us assume by contradiction that for some x E [2~1 we have limsupt__.+~ c dist(yx(t; (~x, 0), Z) = 39 > 0. Since//0 is bounded, we can find an increasing sequence tn --~ +oo such that yx(tn) --~ 5, dist(5, Z) = 39. We can then construct another increasing sequence Sn ~ +cr such that lYx(S2~) xI = P, ]yx(s2n+l) - 5 1 = 2p, lyx(t) - 5 1 E (p, 29) for t E (s2n, s2n+l). Let C o = inf{g(z, a, 0)/(1 + lal 2) : Iz - 51 _< 29, a E A} > 0 by the coercivity of g in (1.1). Therefore we can estimate Co ?1

S~n+X(1 + [C~z[0112)d t <

e(yz, ax [01, 0) dt

"/$2n

n

e(y~,~x[O],O)dt < + ~ ,

<

while on the other hand for each n E N we have

0 < 9 <-- lYx(S2n+l) -- yx(a2n)] < f82 nq-1 If(Yx, as[O], O)l dt n

_<_ Z

f 8 82n§

(1 + 151 + 29 + Ic~x[O]l2) dt _< Z(1 + 151 + 29)

2n

/ 82nq'1

(1 + la[O]l 2) d t .

d 82n

Combining the two estimates, we deduce that +oc = ~

P < n(1 + 15[ + 29)

r$

Z/

82n+1

(1 + [a[0][ 2) dt

n "J82n

_< L(1 + 151 + 29)1Co

e(y~, a~[O], O) dt < +oo,

hence a contradiction. To prove the second statement we need, we can take any x E f~En+x where n is sufficiently large so that/Am~ C V C/B(0, ~i), and use part 2 of this proof. ~1 It is not hard to show that the sign condition on the value function V~ in the statement of Lemma 3.1 follows from (1.2) and (1.1) if the origin is an isolated point of Z, as it happens in many applications. A precise statement is the following. LEMMA 3.2. Assume (1.1), (1.2). The lower semicontinuous envelope V~. may vanish only on the degenerate set Z. PROOF. Observe that by definition we can compute, using (1.2)

V~(x) _> inf f + ~ r e(y~, a, O) dt aEA Jo

4. DYNAMICPROGRAMMINGEQUATION

515

Now note that if ~ ~ Z , then by the assumptions on e (1.1) and (1.2), we have that there is a E (0, 1) sufficiently small such t h a t inf ~(x, a, 0) aEA 1 + lal 2 k a > 0, For ~ > 0 let now V~,(g) < + e c , Xn ~ V~,(~), and an E ,4 be a control such sn > 0 be the first time such t h a t a = never happens), then we use (1.1) for ly~,(s) - xnl <_

/0

for all x E B(5, 2a) .

~ be such that xn E B ( g , a) and V~(xn) --~ that V~(x~) + E > f o + ~ f ( y x , , a n , O ) dt. Let l y ~ , ( s , ) - xnl (and choose sn = +oc if this f and get, for s E [0, sn[ and n large,

L(2 + l-~2l+ lanlq + l y z , ( r ) - x n l ) d r

< L(I~I + 2)s + L

lan[ 2 dr]

q/2sl-q/2 §

-< L(I~I + 2)s + L((V(xn) + c)/a)q/2s 1-q/2 +

lYxn (r) - xn] dr

/o

ly~.(r) - xnl dr.

Hence from Gronwall's Lemma, if s~ < + e c , we obtain t h a t = lYxo(s~) - x,I < L[(2 + I~t)s~ + ((Y(xn) + e)/a)ql2sl-ql2]eLs", and therefore we can find t > 0 independent of n such that s~ _> t for all n. We then conclude t h a t V~(x~) + ~ > fo a(1 + la~i 2) dr > at. As e, n are arbitrary it follows that V~, (5) _> crt > 0. 9 The following result is an immediate consequence of the two lemmas above and the definitions. TUEOREM 3.3. A s s u m e (1.2) and (1.1). The system has finite L p gain if and only if the value function V~ is finite, V~(O) = 0 and Vv has an optimal strategy at the origin. Let moreover the origin be an isolated point of Z . Then the 7-~c control problem has a solution with a function U continuous at the origin if and only if the system has finite L p gain and V~ is continuous at the origin. REMARK 3.4. Note that Definition 1.1 does not require the strategies to be optimal at x r 0. However, as we already mentioned, if we use an optimal strategy, we achieve the best possible estimate in (1.5) with the given disturbance attenuation level ~/by definition of V~. <~ 4. D y n a m i c

Programming

equation

In this section we will s t u d y the properties of the value function V~, t h a t we need in order to solve the 7-/~r problem, by P D E methods. We start introducing the Hamiltonian related to the 7-/oc control problem and the value function (1.6). Precisely we consider the function 7-l(x,p) = inf s u p { - f ( x , a, b) . p - e(x, a, b) + ~/PlbIP} . bEB aEA

516

B. N O N L I N E A R 7-/:r C O N T R O L

The following result is a variant of similar results in Chapter VIII. The proof is therefore omitted. PROPOSITION 4.1. Assume (1.1). continuous.

Then the Hamiltonian 7-l is locally Lipschitz

There are several technical difficulties with the Hamiltonian 7-( and the corresponding Hamilton-Jacobi equation (4.1)

7-l(x, D U ( x ) ) = O ,

inR y.

We mention the unboundedness of the sets of controls and disturbances, which we can take care of in part using the coercivity of the running cost g(x, a, b) - ~P[b]P, see (1.1), and also the unboundedness of the vector field f with respect to the parameters a and b. These facts do not allow Gronwall estimates of the trajectories starting at a given point which are uniform in controls and disturbances. The latter is one of the main technical points when we study the existence of solutions of (4.1) by game theoretic methods. Moreover it is clear that (4.1) does not have a unique solution, since it is invariant by adding a constant to the function U. Even requiring the solution to be nonnegative and null at the origin there is no guarantee that it is unique (in either classical or viscosity sense), due to the weakness of the sign assumption (1.2). The regularity of the Hamiltonian, as deduced from Proposition 4.1 is also very poor to apply the standard theory of the book. Despite all the difficulties mentioned above, the following results can be proved about the equation (4.1). THEOREM 4.2. Assume (1.1) and (1.2) and let f~ C R N be an open set such that V~ is locally bounded in f~. Then V~ is a viscosity solution of 7-l(x, DV~(x)) = O,

in ~ .

In the assumptions of Theorem 4.2, the value V~ may not be even lower semicontinuous in general. Therefore solution has to be meant in the sense of noncontinuous viscosity solutions, as in Chapter V. The following more unusual but very general optimality principle for supersolutions holds as well. THEOREM 4.3. Assume (1.1) and (1.2) and let U : ]~N __, ~ be a continuous viscosity supersolution of (4.1). Then (4.2)

U(x) = inf sup sup ~" f t ( g ( y , a, b) - vPlb[p) ds + g ( y ( t ) ) ~ . ) aEF bE/3 tE~-~Jo

If U is only lower semicontinuous and A is compact, then (4.2) holds if the sets { ( f ( x , a , b ) , g ( x , a , b ) ) : a E A} are convex for all x, b, or provided we substitute the set of strategies F with that of relaxed strategies F r, namely the set of nonanticipating functionals a : B --~ A ~, where .Ar is the set of relaxed controls.

4. DYNAMIC PROGRAMMING EQUATION

517

REMARK 4.4. Since the function V~ is defined as the value function of an optimal stopping time differential game, following Chapter III it could look more natural to consider the following equation, min{7-/(x, D U ) , U} = 0, instead of (4.1). As a matter of fact, due to the sign condition (1.2) they are equivalent for nonnegative solutions, see Exercise 7.3. It could also appear strange that (4.2) is written with the equality sign, while U is only a supersolution. Observe however that one direction (_>) is what one would expect, while the other is obvious by choosing t = 0 in the right-hand side. We also note that the corresponding statement holds for subsolutions as well, substituting supt~R + with inftE~ _. The result of Theorem 4.3 holds true also if U is a supersolution in an open set fl by changing R+ in (4.2) with [0, tx(a,b)[, where tx(a,b) is the exit-time of the trajectory y x ( . ; a, b) from f~. <1 The proofs of Theorems 4.2 and 4.3 are rather technical and will not be given in detail. Let us only sketch the main ideas. First of all, we use a reparametrization procedure. For all a E ..4, b E B, we consider the increasing change of parameter r ( t ) = Ta,b(t )

:

~0t (1

p Jr" la[ p "Jr-Iblp) ds = t + a PL,(Ot) + IIb IlL,(0,t),

and set t(. ) = ta,b(" ) ----r - l ( 9 ). Observe that the reparametrization t(. ) is a nonanticipating functional of the controls. We will briefly denote in the following the reparametrized measurable functions as a(r) = a(t(7)), b(~-) = b(t(~-)), and for such controls solve the dynamical system (4.3)

y'(7) = f ( y ( T ) , a(T), b(7)),

y(O) E R N,

where f : ]I~n • A • B --* ]~N, is defined by f(x, a, b) = f ( x , a, b)/(1 + [a]p + ]blP). Observe that trajectories of the systems (0.1) and (4.3) are equivalent by means of the above introduced change of parameter. Indeed if y(~-) solves (4.3), then y(~-) = y~(t(r); a, b) . By a change of variables in the integral, we can then rewrite the value function as

V (x) =

inf sup sup ~0 r g(y' a[b], b) - "~Plblp

It is now easy to check that, by the assumptions (1.1), the trajectories of the system with reparametrized controls satisfy the following condition lye(s) - x I <_ w ( r ) ,

for all Ixl _< R, s E [0, T], a E A, b E B,

where w : 11~+~ R+ is continuous, increasing and aJ(0) = 0. In particular we have yz(T; a, b) --* x as T ~ O, uniformly for all a E .A, b E B and x bounded. The main

518

B. NONLINEAR "Hor CONTROL

advantage of the reparametrization procedure is indeed the fact that our value function corresponds also to another differential game whose trajectories satisfy such a uniform Gronwall type estimate. The Hamiltonian associated with the reparametrized problem is

7-l#(x,p) = inf s u p l - f ( x , a, b) . p - f(x, a, b) + bEB aEA t

?PlblPlalP + Iblp } 1+

w h e r e / ( x , a, b) = g(x, a, b)/(1 + la] p + ]bIP). It is now ahnost standard to show that the value function V~ satisfies the equation 7-/#(x, D U ) = O, and then conclude the result of Theorem 4.2 with a few simple computations based on the coercivity of the cost g. To prove the optimality principle of Theorem 4.3, further technical ideas are needed since the reparametrized Hamiltonian is still not regular enough to apply standard comparison theorems. Given a flmction U as in the statement of Theorem 4.3, we start with a change of variables. Let g : IR --~ ]R+, be bounded, smooth, such t h a t M :> ~ > 0 and ~)(s) --* 0, as s -~ - o c . Define the nonnegative, bounded function u(x, r) = g(U(x) + r). By the standard formulas of change of variables as in Chapter II, u is a viscosity supersolution of inf s u p { - F ( z , a, b) . D z u } >_ O,

in N N+I,

bEB aEA

where z = (x, r) and F ( z , a, b) = ( f ( x , a, b), ~(x, a, b) - ~/PIbIP/(1 + lal p + IblP)). We proceed as above and consider the following dynamical system with reparametrized controls

z'(r) = F(z(T), a(T), b('r)), z(O) = z E NN+I . It is clear by the definition of F that this system has a unique solution for any

aEA,

bEB.

For a fixed ~ > 0, we now consider a smooth function ~E : IRN+I --' ]R such that 0 < ~s(z) = ~E(x,0) _< 1, ~E = 0 on E~/2, ~ = 1 in Ne, where for a > 0 we define Eo = {z = (x, r) E ]RN+I : Ixl < l / a } . Therefore u is a supersolution of

7-l#~ (x, Dzu) = inf sup {--FE(z, a, b). D z u ( z ) } >_ O,

in RN+I,

bEB aEA

whereF=FEinEexA•215 E c = R N + I \ E . Finally we conclude t h a t for any A E ]0, 1[, the function V = u is a viscosity solution of

)~V + m i n { ~ # E ( x , D ~ V ) , V -

( 1 + A)u} = 0,

in R N+l .

To this last equation we can apply a comparison theorem and state that u is the value function of the corresponding differential game. We then need some careful but tedious computations to take the limits as A --~ 0 first and c --~ 0 next, and complete the proof of the result. W i t h the above results at hand, it is now easy to derive necessary and sufficient conditions for the solvability of the 7-/o~ suboptimal control problem in terms of the equation (4.1). This result completes Theorem 3.3.

4. DYNAMIC PROGRAMMINGEQUATION

519

THEOREM 4.5. Assume (1.1) and (1.2).

If the 7-t~ suboptimal control problem with level "7 is solvable, then there is a nonnegative, lower semicontinuous supersolution, null at the origin of (4.1). If, on the other hand, there is a nonnegative, continuous supersolution U : ]RN -+ ]~ of (4.1), satisfying in a neighborhood of the origin U(x) = 0 only at x = O, then U > V~ and the ~ suboptimal control problem is solvable. If V~ has an optimal strategy at the origin, then there is a solution of the TiM problem. PROOF. The necessary part is a direct consequence of Theorem 4.2, using V~. as the supersolution. For the sufficiency part observe that from (4.2), the fact that U is nonnegative and the definition of V~, we get U _> V~. We are left to show that the undisturbed system is open loop Lyapunov stable. Let {LCe}~>0 be a nested local base of neighborhoods of the origin made of open bounded sets. We then define the sets ~ = {x E Uz : U(x) < min0u U} and observe that for z small they are neighborhoods of the origin and, in fact, {Q~}E is a nested local base of the origin. Fix z > 0 and let x C Q~, by (4.2) we can choose 5 > 0 and a E F so t h a t sap sup/"

ft

bet3 teP~+"do (e(y, a[b], b) - ~/VlblP) dt + U(yx(t))

} _< U(x) + 5 _< minoutU - 5.

In particular for b - O, and since g(x, a, O) is nonnegative, we get suptei?~+ V(yx(t)) <_ minou~ U - 5, and then y(t) E Llz for all t _> O, from which the open loop stability immediately follows. The last statement is a consequence of Lemma 3.1. 9 REMARK 4.6. Theorem 4.3 shows that, when V~ is continuous, it is the smallest nonnegative supersolution of (4.1). There is a gap in Theorem 4.5 between the necessary and the sufficient conditions. As a result of Theorem 4.3, if A is compact, in Theorem 4.5 we can relax continuity of U to lower semicontinuity, still requiring U to be null and continuous at the origin and satisfying in a neighborhood of the origin U(x) = 0 only at x -- 0. This can be done by either considering the value function with relaxed strategies or assuming the convexity of the sets {(f(x, a, b), g(x, a, b)) : a e A} for all x, b. <~ EXAMPLE 4.7. We go back to the system of Example 1.4 and compute the corresponding Hamiltonian. It is immediate to check t h a t Tl(x,p) = 1(1 - 1/72)lpI 2 Ixl 2. Therefore for the choice of 7 > 1, we can check t h a t the function U~(x) = 7/("y 2 - 1)l/2IxI2 is a solution of the equation, while there is clearly no solution if 7 -< 1. By Theorem 4.5 we can now conclude t h a t 7* = 1. By Proposition 2.1 this implies that, for 7 = 1, there is no linear stabilizing feedback control satisfying (2.2) even if VI(0) = 0. <~ The following result is concerned with the existence of optimal strategies and therefore with the actual solution of the 7-/~ suboptimal control problem. It completes Theorem 3.3 and Theorem 4.5, and its proof is an a d a p t a t i o n of classical results in differential games theory. The main ideas of this proof can be also found in Propositions 5.5 and 5.7.

520

B. NONLINEAR 7-/or CONTROL

PROPOSITION 4.8. Assume (1.1) and let A be compact. If the sets

{(f(x,a,b),g(x,a,b)) : a e A} are convex for all x, b, then there is an optimal strategy for VT(x ) at any x C ]~N. Otherwise there is always an optimal relaxed strategy c~ E F r. The theory of linear systems proves that, when (2.2) holds, there is a classical solution of (4.1) which is nonnegative and null at the origin. The next example motivates the use of viscosity solutions. We want to show that when we depart from linear systems, (super) solutions of (4.1) may be not differentiable, even in a neighborhood of the origin and in special cases often used in the applications as the nonlinear-affine systems (0.4). EXAMPLE 4.9. Let us consider the following system

{~=a-b,

y(O)=x,

~(x) 2 + Jail where ~ : ~ N ~ ~ is a nonnegative function null at the origin to be specified. Again it is easy to check that the Hamiltonian is given by ~ ( x , p ) = 1(1 - 1/'~2)1p[ 2 - ~(x) 2 . It is then clear that, when 7 > 1, the Isaacs equation (4.1) is equivalent to (4.4)

2~

IDU(x)l - (7 2 - - i ) l / 2

~(x),

which is the eikonal equation. Our purpose is to show that, with a suitable choice of g, the equation (4.4) does not have a classical nonnegative solution vanishing at the origin. The reader can easily check by control theoretic ideas that if g satisfies (1.4) and we write (4.4) as a Bellman equation, setting Ip[ = maxlal<_l{-a "P}, then the value function of the control problem is nonnegative, continuous and vanishes at the origin, see Exercise 7.4. In this case it can also be proven t h a t there is a unique viscosity solution of (4.4) of this sort. We will now show t h a t this solution will not be smooth for a suitable choice of ~ in any neighborhood of the origin. Consider the function of two variables U(x, y) = (x 2 + y2)(2 - lyl + y2). Such a function is differentiable for all (x, y) such t h a t y # 0 and it is also differentiable at the origin. Note that U(x, y) > 0 if (x, y) # (0, 0). If x # 0 we can compute D+U(x,O) = {(4x, p 2 ) : 192 E [-x2,x2]} and D - V ( x , O ) = 0. Moreover the function f IDU(z, y)[,

g(x,y)= [ixl(16+x2)1/2 '

y # 0, y=0,

is immediately seen to be continuous. By the definition of viscosity solution it follows that IDU[ = g in ~ N , in the viscosity sense. <1

5. ON THE PARTIAL INFORMATION PROBLEM

521

In the case of nonlinear-affine systems, some positive statements concerning the regularity of the value function in a neighborhood of the origin can be found in the literature. We will not give here more details about this matter. If the value of the differential game we studied so far does not exist, we are possibly not completely satisfied by the solution of the problem in the sense of Definitions 1.1 and 1.3 and we may desire also some estimates for the upper value as well. To this end we note t h a t all the results above can be rephrased for the upper value function

-V~(x)

= sup inf sup

fo

(e(y, a, ~[a]) - ~Pl~[a]l p) ds,

~ e A LEA tE~+

where A is the set of nonanticipating or causal functionals f~ : .4 --+ B, and the corresponding Hamiltonian ~ ( x , p) = sup inf { - f ( x , a, b). p - g(x, a, b) + -yPib]P}. LEA bGB

Of course if the Isaacs condition holds (i.e., ~ = 7/) the two approaches turn out to be equivalent even if this is unfortunately not enough to prove the existence of value with the results we presented. As a consequence of Theorem 4.3, however, the following result holds. COROLLARY 4.10. Assume that (1.1), (1.2) and the Isaacs condition hold. If moreover the value functions V.r and V-: are continuous, then the value of the game exists, i.e., V.r = V~. 5. O n t h e p a r t i a l

information

problem

In this section we describe some results concerning the partially observed system (0.1)-(0.2). In this new framework the controller is allowed to use a restricted class of strategies based on the available information. We will denote by )4; -]v[(]~+, RP), the set of measurable observed outputs. DEFINITION 5.1. Let x E ~N; an open loop strategy o E F is said to be an output feedback strategy admissible at x if for all bl, b2 E/3, T _> 0 such that

g(Yx(" ;a[bl],bl),a[bl],bl) = g(Yx(" ;a[b2],b2),a[b2],b2) then ~[bl] = a[b2] a.e. in [0, T].

a.e. in [0, T], <~

We often indicate, with a slight abuse of notation, an output feedback strategy as a functional of the observed output ~ -- ~[w] (which is defined in (0.2)), since if we define

)4;x,~ = {g(Yx(" ; ~[b], b), ~[b], b): b E B} c • , then ~ determines a causal functional 14;x,~ ~ .4. We will also denote by F~ c F the set of all o u t p u t feedback strategies admissible at x.

522

B. NONLINEAR ~ r

CONTROL

REMARK 5.2. We give two explicit examples of output feedback strategies. Suppose that a : W -~ .4 is a causal functional and that for all T > 0 and b E B the m a p LP(O, T; A) -* LP(O, T; A) given by

a,

~ a[g(yx(. ;a,b),a(. ),b(. ))]

has a unique fixed point ~, which is the unique trajectory solution of the problem

(5.1)

'~] = f(y,a,b), w = g(y, a, b),

I

y(0) = x,

(a---The functional a can be viewed as an element of Fx by the position a[b] = ~. Another trivial but helpful example is a constant mapping a[w] --- a E .4, for all w E 142. In particular, in the case with no available information on the system, i.e., g(y, a, b) - const., Fx reduces exactly to the set of all constant strategies, as easily checked. <3 REMARK 5.3. Note that the following useful property holds for output feedback strategies. For all T > 0, a E Fx and bl E L~([O,T],B) then the position ~[b] = a [b], where [bl(t),

-b(t) = ( b ( t - T),

[O,T[, IT, +oc[,

defines an element ~ C Fy3:(T;o,b).

<3

The previous remark allows us to use the dynamic programming approach for the value function of the partially observed game, namely (5.2)

W.~(x) = inf sup sup

(e(y,~,b)-'~PlblP)ds.

c*EF~ bEB tE~-

We assume that the initial state of the system is known, while equation (0.2) provides only partial information about the system as the time proceeds. This is the main reason for us to study the value function in (5.2). Observe that if g(x, a, b) = x, then the set of admissible strategies Fz is the class of feedback strategies, see also Chapter VIII. The notion of solution or of solvability of the "Hoc suboptimal control problem with partial information can be adapted for this new class of admissible strategies for the controller from Definitions 1.1 and 1.3. If the initial state of the system is also unknown, it seems to be more appropriate to reformulate the problem as a full information problem for an infinite dimensional system. For this approach, we refer the reader to the literature. To have a feeling of some additional difficulties involved in the partial information problem, observe that since constant output feedback strategies are admissible at any point, the following estimates hold (5.3)

V~(x) < W~(x) < V~#(x) :-- inf sup sup f t (e(y, ax, b) - ~,Plblp) ds aeA bCB t e ~ - 9'0

5. ON T H E PARTIAL INFORMATION PROBLEM

523

Note that the map V~# actually coincides with the function W~ when g is constant and clearly V~ = W~, when g(x, a, b) = b, therefore the estimate is sharp. On the other hand it is immediately seen that the upper value function satisfies the following inequality

where the inequality may be strict in general. The simple argument to prove the previous inequality is as follows. Fix r > 0, then by definition of upper value there is ~ E A such that

V~(x) - ~ < J ( x , a , fl[a]) := sup tEN+

-

fo'

(e(y, ax, b) -'yPlblP)ds < s u p J ( x , a , b ) , bEB

for all a E .4. The function Tv~# is the upper value function of the static game where no information about the dynamics is available to the players as time proceeds. In the very special case that the payoff of the game J(x, a, b) has a saddle point as a functional of control functions a(. ) E A and disturbances b(. ) E B, then the lower and upper values of the static game coincide and in (5.3) we can substitute inequalities by equalities. In order to solve the partial information 7-/oc problem, by adapting Theorem 3.3 to this case, we need to show that W~ is finite, vanishes, is continuous, and has optimal strategies at the origin. The next result follows as Theorem 4.2 by using Remark 5.3, and shows the relationship between W~ and the Isaacs equation. THEOREM 5.4. A s s u m e (1.1) and (1.2). If W~ is finite, then its lower semicontinuous envelope (W.~). is a nonnegative viscosity supersolution of (4.1). In general W~ is not a solution of equation (4.1) nor a subsolution of the upper value equation at the end of Section 4. Therefore the solvability of the problem cannot be proved by the only equation (4.1). Assume we solved the full information problem; a standard but formal way of proceeding would be the approach known in the literature as worst case certainty equivalence principle. This means that from an optimal strategy of the full information problem, we hope to construct an optimal output feedback strategy as follows. For any fixed x E ]RN, T > 0, control a(. ) and output w(- ), denote the subset of disturbances

Bx,T(a, w) = {b E B : w( . ) = g(Yz( "; a, b), a, b)

a.e. in [0, T]},

which are compatible with the given data. Then assume that there is a unique solution bx,T E/3 of the maximization problem

]tlz,T(a, w) =

sup bEBx,T(a,w)

(h(y, a, b) - .y2[b12) ds + V~ (y(T))

.

The goal, given an optimal strategy a E F of the full information value function V.~(x) is to show that the output feedback strategy ~ defined by the implicit relation ~[w](. ) = a[bx,T](. ),

in [0, T],

524

B. NONLINEAR 7-/~r CONTROL

is optimal for the value W~(x). We do not give details about the actual proof of this statement, which can be found in the literature and requires other assumptions that are in practical cases strong and difficult to check. As a matter of fact, if this procedure can be applied, we end up proving that V~(x) = W~(x) which is a restrictive property of the system. In the following we are concerned with the existence of optimal strategies for the value function W~, at least if the system satisfies some additional structural conditions. For the rest of this section, in addition to (1.1) we require that p = 2 and for all R>0

(5.4)

Iig(x,a,b)] <<_LR(1 + lal k + tbIk), ~lg(xa,b,)" - g(y,a~b)l <_ LR(1 + lal k + tblk)tx - Yl, ItS(x, a, b) _> C~

2 - L(1 + la] q + IbI2),

I

[ A is compact, for some k E [1, 2], q E [1,2[, C o > 0, for all Ixl, iy] -< R, a E A, b E B, so that if j --- 2 / k we have g(yz(. ; a, b)~ a, b) E L~oc(]~+, ]RP) for all x, a, b. We also indicate the space of measured outputs as W = L~oc(~+, ~P). We start considering the relaxation of the original problem, with class of admissible controls .4 r = L~r (lt~+; At), where A r is the set of relaxed controls as in Chapter III, and extend the system defining, for m E A r,

~r(x,m,b) = f A ~ ( x , a , b ) d m , where v E {f,g,g}. Observe that (fr gr gr) satisfy (1.1) and (5.4). Since A is compact, A r is metrizable and compact in the weak* topology as a subset of the dual of C(A; I~). For x E ~N we consider the set F r of relaxed open loop strategies and the set F r of relaxed output feedback strategies admissible at x. Existence of optimal relaxed feedback strategies can be deduced from the following result. PROPOSITION 5.5. Let x E R N and assume (1.1) and (5.4). Suppose that there is a sequence of output feedback strategies an E Fx which can be extended to an equicontinuous family of causal functionals an : W ~ .A and such that

(e(y, an, b) - "~2}bt2) dt < K x + 1/n,

(5.5)

for all b E B, r E N+ .

Then there is a* E F r such that

~0

T(gr(y r, a * , b)

~/21bl2)dt < I(z,

for all b E B, T E ~+

PROOF. In the following we denote Wn(" ) = g(Yx('; an, b),an[b], b) for a fixed b E B. By the weak* compactness of L ~ (0, T; At), the Tychonov theorem applied

5. ON THE PARTIAL INFORMATION PROBLEM

525

to (L~162 T; At)) B • (L~162 T; At)) w tells us that this space is compact in the product topology. In particular we can find a subnet {a~} of {an} such that (at, a~) converges in the product topology to an element ( a * l , a ~1) E (L~(O, 1; Ar)) ~ x (L~162 1; Ar)) W 9

In general, given a subnet {a~} of {an} such that ( a ~ , a k)

> (a*k,a ~k) E (L~

r~ x ( i ~ ( O , k ; A r ) ) w,

we can extract a subnet {a k+l} of {a k} such that a;~k+l , a kA+ l , ) ----+ (a*(k+l),a I~(k+l)) E

(L~

1;Ar)) B • ( L ~ ( O , k + 1; At)) w

Define a*[b](t) = a*(k+l)[b](t) if t E [k, k + 1[ and a ~ similarly, then, by the construction, for all T > 0 we can find some subnet {a~} of {an} such that for all b E B and w E 142 we have a~[b] --+ a*[b] and a~[w] --+ a#[w] weak* in L~(O,T;Ar). In particular a* and a # are indeed causal functionals of their arguments. By definition of weak* convergence, for all T > 0 and 9~ E LI(0, T; C(A)) we get

io"

~o(t, a~[w~](t)) dt =

io"

p(t, ax[b](t)) dt -+

io'(l.

~(t,a~[w](t))dt--*

~(t, a) da*[b](t)(a) dt,

)

c2(t,a)da#[w](t)(a)

dt,

for all b e B and w E W. In particular this leads to the following facts for the relaxed trajectory yrx(. ; a*[b], b) for any fixed b E B (note that (1.1), (5.4), particularly the coercivity of g, and the assumption (5.5), give, for all fixed T > 0, b e B, a uniform bound for the norm of a~[b] in L2(0, T; A)): Yx(" ; a~ [w~], b) ~

y~ (- ; a* [b], b),

locally uniformly,

L

T(hr(yr, a*[b], b) - 9-y2ibl2 ) dt _< Kx 9

j,

Moreover for all ~o E Lloc(~+, ~), j' = j / ( j - 1), we also have

~,(t)g(yx, c~a[wa], b) dt ~

~(t)d(y~, a* [b], b) dt,

and therefore, if we indicate w(t) := gr(yr(t; a* [b], b)a* [b](t), b(t)), for the particular choice c2 = sgn(wx - w)Iw~ - wl j - l , we get

L

T [WA __ W l j

dt

~

0

.

By the equicontinuity of the sequence an, we deduce that a,\[w~] - a~[w] -* 0, so a* [b] = a # [w] and the limit strategy a* depends only on the observed output. Therefore it is a relaxed output feedback strategy admissible at x, i.e., a* E Fr~. ,~

526

B. NONLINEAR ~or CONTROL

REMARK 5.6. Concerning the technical assumption in Proposition 5.5, regarding the family of strategies, observe that if the sequence a,, is given by constant strategies, then it is obviously extendable to a family of constant (therefore equicontinuous) functionals W --~ fit. We need to be careful when passing to limits since it is not true in general that disturbances producing the same output with a given strategy will also enjoy the same property with other strategies. <~ In certain cases, the optimal output feedback strategy provided in Proposition 5.5 can be represented by means of non-relaxed strategies. This fact can be shown by adapting classical selection arguments. To this end we further assume t h a t

'g(x, a, b) = gl(x, b)g2(a), v(x, a, b) = vl (x, b)v2(g(x, a, b), a)

I

(5.6)

( A is compact, $(w) = c o $ ( w )

if v e {f, g}, for all w,

where $(w) = {(f2(w,a),g2(w,a),g2(a)) : a E A}, gl,fl~-~l are matrix valued, g2, f2,g2 are vector valued and all of them are suitably defined and continuous. W i t h co X, we indicate the convex hull of the set X. It is easy to check t h a t if f ( x , a, b), t(x, a, b), g(x, a, b) are nonlinear in x and affine in the controls a, b, and A is convex and compact, then (5.6) is satisfied. As a matter of fact, any function of the type v(x, a, b) = ul(x)u2(a) + u3(x, b), where u 1 is matrix valued and u 2, u 3 are vector valued and v2(A) is convex, is allowed by (5.6). The following result holds. PROPOSITION 5.7. Let x C ]~g. If the assumptions of Proposition 5.5 and (5.6)

hold, then there is an output feedback strategy -~ E Fz such that

/o

T(~(y,-~,b) --

721bl2 ) dt

<_ Kx,

for all b E 13, T E ~+ .

PROOF. The goal here is to define an output feedback strategy with the same behavior of the relaxed strategy a* constructed in Proposition 5.5. Note that by that proof, a* is in fact a causal functional 14; ~ fit. 1. F i x w( 9 ) E 142, w E R P and t > 0, and define the vector valued function

e(w, t; w(. )) = f (f2(w, a), e2( , a), g2(a))

da* [w](t)(a) 9 co $ ( w ) .

Then define the measurable map 9 : R+ ~ 7:'(A) by setting

~w(. )( t) = {a 9 A : (f2(w(t), a), ~2(w(t), a), g2(a) ) = e(w(t), t; w( . ))}. Of course by the assumptions and the construction, ~w(" ) is compact and nonempty valued. Therefore by Filippov's Selection Theorem, we can find a measurable selection a(t) 9 a2(t), thus a 9 fit. We set ~[w] = a.

6. SOLVING THE PROBLEM

527

2. We now proceed with the construction of a causal functional of the output. Assume t h a t a nonanticipating functional has been defined in the set W l C IA2. For w C I/V \ l/V1 we consider T = sup{t : w

=

Wl

a.e. in [0, t], for some

W 1 C ~1/~1} .

If T < +co, for any n E N, n > 2, we determine wn E l/V1 such that w = wn a.e. in [0, (1 - 1/n)T], then set ~[w] = ~[wn],

a.e. in [(1 - 1/(n - 1)T), (1 - 1/n)T],

and define ~[w] in [T, +c~[ by using the argument of part 1. If T = + z ~ the procedure is similar and left to the reader. Zorn's lemma guarantees t h a t indeed can be defined as a nonanticipating functional 142 --+ .4. 3. We can now check that the process leads to what we wanted. Let b E B, X E ]l~N. Denote

~(~, t; ~(. )) = (~:(~, t; ~(. )), ~(~, t; ~(. )), eg(~, t; w(. ))), w ( . ) = g(Yx(" ;c~*, b), (~*, b), and define the causal functional by setting c~#[b] = ~[w]. Any relaxed t r a j e c t o r y of a* is by construction a trajectory of a # and vice versa, since

[ ~ = fr (9, a* [b], b) = fl (9, b) e S (w, t) = f(Y, ~[w], b) = f(9, c~#[b], b), (5.7)

~ hr(9, a*[b],b) = el(9,b)et(w,t) = g(9,~[w],b) = g(9,ae[b],b), "* l = , gr(9, w a [b], b) = gl (9, b)eg(t) = g(9, ~[w], b) = g(9, a # [b], b).

Moreover by its definition ct# depends only on the output, so it is admissible at x. The final inequality of Proposition 5.5 gives the result. 9 6. S o l v i n g t h e p r o b l e m In Section 4 we stated that, under suitable assumptions, optimal strategies for the value function of the full information problem exist. However the proof is not eonstructive and therefore not particularly useful from the practical point of view of the applications. As in the case of standard control theory, one can produce explicit formulas for optimal feedback controls, at least formally. In differential games theory however, this is unfortunately not always likely to succeed. We will consider here only the case of the full information problem. For any (x, q) E IR2N we denote

Fx,q(a, b) = - f ( x , a, b) . q - g(m, a, b) + "/Plb]p, so t h a t 7-/(x, q) = infbe B SUPaeA Fx,q(a, b) and ~ ( x , q) = supaea infbe B Fx,q(a, b). We look for necessary conditions, supposing that the value function is of class C I, which is true for linear systems (0.3) and for the nonlinear-affine systems (0.4) with

528

B. N O N L I N E A R ~zr C O N T R O L

smooth data, at least in a neighborhood of the origin. Assume that the feedback a(x) solves strongly the ~or suboptimal control problem and provides optimal strategies for V~, then by Definition 1.1 we have

fo

t(t(y, a(y), b) - ~Plblp) ds _< Y~(x),

for all b E /3, t >_ O .

Therefore by the definition of V~, for any fixed t > 0, we get

fo

t(t(y,a(y), b) - 7Plblp) ds + V~(y(t)) < V~(x),

for all b E/3, t _> 0.

Dividing by t and letting t ~ 0 +, we conclude

0 <_ F~,DV,(x)(a(x),b),

for all b E B,

and finally we must have 0 < inf Fx,DV,(x)(a(x), b) <_ ~t(x, DV~(x)) <__7-l(x, DV~(x)) = O .

(6.1)

bEB

Therefore equalities hold in (6.1) and in particular V~ has to be also a solution of the upper Isaaes equation. Of course, this is automatically true when the Isaaes condition holds. Moreover, it certainly holds when the value of the game exists. In view of the necessary condition, we proceed and assume that U is a nonnegative, differentiable supersolution of the upper Isaacs equation. Then we define the possibly multivalued map

A(x, q) = arg max{ inf F~,q(a, b)},

(6.2)

aEA

bEB

and assume

[ there is a selection a(x) E A(x, DU(x)) such that the system (6.3)

~y = f(y,a(y),b),

y(0) = x,

[ has a unique absolutely continuous solution for all b E/3 and x E R N . PROPOSITION 6.1. Assume that U is a differentiable, positive definite supersolution of the upper Isaacs equation. Then a feedback control a(x) satisfying (6.3)

solves the ~oc suboptimal control problem. U = V~, then a(x) is optimal.

If the assumptions are satisfied by

PROOF. Let us prove that the position a[b](t) = a(yx(t)) defines a strategy satisfying (1.5). It is clear by the definition of a( 9 ) that inf Fx,DU(x)(a(x), b) -- Tl(x, DU(x)) > O, bEB

(if U = V~, the necessary condition (6.1) then holds by construction) and therefore for all b E 13 the solution guaranteed by (6.3) satisfies

0 < - f ( y , a[b], b). DU(y) - ~(y, a[b], b) + 7P[blp .

6. SOLVINGTHE PROBLEM

529

Integrating on [0, t], we obtain

g(y(t)) +

(e(y, c~, b) - 7Plbp) ds <_ S(x),

for all b e B, t _> 0,

and we conclude the proof since U is nonnegative. If U = V.y the same inequality shows the optimality of a for V~(x). 9 REMARK 6.2. To our knowledge, there are no general results concerning the existence of optimal feedback controls, i.e., under which conditions (6.3) is satisfied. However, one can call the m a p A(x) = A(x, DV~(x)) a multivalued feedback synthesis, in the sense t h a t any solution of the differential inclusion

[1 C f ( y , A ( y ) , b ) ,

y(0) = x,

satisfies the gain condition (to prove this, just apply the argument of Proposition 6.1 to a selection a(t) E A(y(t)) such t h a t ~ = f(y,a,b)). Moreover, since in general the value function is not differentiable, we should really use generalized gradients in some suitable sense, i.e., define the feedback control as a selection a(x) 9 A(x,D• for which (6.3) holds, as in w This is also an open problem. Otherwise A(x) = A(x, D• will again be a multivalued feedback synthesis. <~ REMARK 6.3. In the case of nonlinear-affine systems, when the matrix gT~2 is invertible at each point and p = 2, the above m a p A is luckily single valued and the following explicit formula holds A ( z , q) =

--(eT(x)e2(x))-i(f~ (x)q/2 +

If, in particular, the system is linear and the control sets are finite dimensional vector spaces, it is not restrictive to assume t h a t E T E = I and E T D = 0 (otherwise we change coordinates in the control variable by defining the new control u as u = (ETE)I/2a + ( E T E ) - I / 2 E T D x ) and it is then known t h a t the value function is quadratic, i.e., V~(x) = x T F x , for some nonnegative definite symmetric matrix F . Therefore the optimal feedback control is given by the formula

a(x) = - p T Fx, and so it is linear.

,~

EXAMPLE 6.4. We want to apply the previous Remark 6.3 and compute the optimal linear stabilizing feedback control for the system of Example 1.4 and 7 > 1. In this case E ----(0 1) T, D = (1 0) T, P -~ (1) and F -- ( 7 / ( 7 2 1) 1/2) as we computed in Example 4.7. Therefore the explicit formula for the feedback control is -

a(x) = - x (^/2 _ 1)1/2' which is of course one of those allowed by the computations of Example 2.3.

<~

530

B. N O N L I N E A R ~

CONTROL

7. E x e r c i s e s 7.1. Complete the computations of Example 1.4. 7.2. Prove Proposition 4.1.

7.3. Prove that if U : ]~Y ___,~ is a nonnegative locally bounded function and (1.2) holds, then the following statements are equivalent (i)

U is a viscosity solution of (4.1);

(ii) U is a viscosity solution of min{?-/(x, D U ) , U} = 0, in ]~N. 7.4. As outlined in Example 4.9, prove that the value function of the control problem related to the eikonal equation (4.4) is continuous if (1.4) holds. 7.5. Compute super- and subdifferentials of the function U in Example 4.9. 7.6. Consider the one dimensional, linear system (A = B = JR) { ~ = - y + a -+b , ay (2O. ) = X , y 2

Prove that the Hamilton-Jacobi-Isancs equation of the T/cr suboptimal control problem has a solution if and only if 7 -> 2-1/2- [Hint: look for quadratic solutions and note in particular that there are two such solutions if 7 E (2 -1/2, 1).] Therefore 7" --- 1/v~. Construct the optimal linear, stabilizing feedbacks associated with each acceptable disturbance attenuation level. 7.7. Consider the system as in the previous exercise~ now with bounded control set A = I - r , r], r :> 0. Compute the Hamilton-Jacobi-Isaacs equation and show that for all 7 -> 1 there is a supersolution of the form u ( x ) = cx 2. Prove that for 7 < 1, V ~ - + o c , henceT* = 1. [Hint: for 7 < l a n d fixed x > 0 , try with the disturbance b(t) = - ( t + x) - (r + 1).]

7.8. Consider the two dimensional linear system (perturbed harmonic oscillator) with A = B = 1~

x +a-b, x 2 + y2 .~ a 2 .

Compute the Hamiltonian associated with the 7-/or control problem. Prove that for any 7 > 1 there is a solution of the Isaacs equation of the form U ( x , y) = cx 2 + 2 d x y + ey 2. When 7 = 1, represent the Isaacs equation by a system without controls and then use Theorem 4.3 to prove that there is no nonnegative solution. Therefore 7" = 1. Compute the linear stabilizing feedbacks solving the problem for 7 > 1.

8. BIBLIOGRAPHICALNOTES

531

7.9. Consider the nonlinear-affine system

{~

= -y + f(y)a x 2 q- a2~

- b,

where A = B = ~, N = 1, and f : ~ --* I~ satisfies inf{ (f(x)) 2 : x e R } = k s > 0. Compute the Hamilton-Jacobi equation and show that, if 7 2 > 1/(1 + k2). there is a supersolution of the form u ( x ) = c x 2. Compute the corresponding feedback solving the 7-/o~ problem. Show that, if 7 2 < 1/(1 + k2), then there is no solution to the Hamilton-Jacobi equation, hence 7" = 1/(1 + k2). [Hint: by contradiction, use Remark 4.4 to show that there is no solution of xur -

a + 1 (u,) 2 > x2 ' 4

in an open set, if a > 0.] 8. B i b l i o g r a p h i c a l

notes

The 7-/oc control problem was proposed by Zames in [Za81] for linear systems in the frequency domain. For the solution of the problem for linear systems we refer to [GD88, Sch89] and [DGKF89], also in the output feedback case. General surveys on the linear problem can be found in the books [Fra87] and [Sto92]. The problem for nonlinear-affine systems was studied, assuming smoothness of the solution of the Hamilton-Jacobi equation in [vdS92] and [BHW93]. In [vdS92] a regularity result for the solution of the Hamilton-Jacobi equation is also proved. For an introductory and a fairly complete treatment, see also [vdS93]. The partial information problem for nonlinear affine systems was studied in [IA92a] and [IA92b] again assuming the existence of smooth supersolutions of Hamilton-Jacobi equations. A different approach, based on the introduction of an output state infinite dimensional system was introduced and studied by [JBE94] and [JB96]. The differential games approach to the 7-/cr problem was proposed and studied in the book by Ba~ar-Bernhard [BaBe91]. A study of the problem using the theory of viscosity solutions in the case of dissipative systems (no control dependence of the dynamics), can be found in [Jam93]. The proofs of the statements in Sections 3 and 4, and of other results in this appendix appear in [Sor96a] and [Sor97a]. The results in Section 5 are new, see also [Sor94b]. Numerical studies to compute the 7-/oc norm of a nonlinear system appear in [JY95, Cam96]. For the risk sensitive approach, originally introduced by Jacobson [Jac73] for linear systems and formally by Whittle [Whg0] in the nonlinear case, the reader can refer to [FM92a, FM95] and [Jam92]. The 7-/~ problem for infinite dimensional systems is studied in various generalities in [Bar93b, Bar95a, Bar95b, vKPC93, vK93, KSor96].

Bibliography [AT87]

R.A. ADIATULLINAAND A.M. TARAS~'YEV.A differential game of unlimited duration. J. Appl. Math. Mech., 51:415-420, 1987.

[Ak90]

A.M. AKIAN. Analyse de l'algorithme multigrille FMGH de resolution d'equations de Hamilton-Jacobi-Bellman. In A. Bensoussan and 3.L. Lions, editors, Analysis and Optimization of Systems, volume 144 of Lecture Notes in Control and Inform. Sci., pages 113-122. Springer, Berlin, 1990.

[AGLM93] L. ALVAREZ, F. GUICHARD, P.L. LIONS, AND J.M. MOREL. Axioms and fundamental equations of image processing. Arch. Rat. Mech. Anal., 123:199257, 1993. [Alv97a]

O. ALVAREZ. Bounded from below solutions of Hamilton-Jacobi equations. Differential Integral Equations, to appear.

[Alv97b]

O. ALVAREZ. Homogenization of Hamilton-Jacobi equations in perforated sets. Preprint, 1997.

[ALL97]

O. ALVAREZ, J.~I. LASRY, AND P.L. LIONS. Convex viscosity solutions and state constraints. J. Math. Pures Appl., 76:265-288, 1997.

[A1T96]

O. ALVAREZ AND i . TOURIN. Viscosity solutions of nonlinear integrodifferential equations. Ann. Inst. H. Poincard Anal. Non Lindaire, 13:293-317, 1996.

[AL94]

B. ALZIARY AND P.L. LIONS. A grid refinement method for deterministic control and differential games. Math. Models Methods Appl. Sci., 4(6):899910, 1994.

[A191]

B. ALZIARYDE ROQUEFORT. Jeux diff6rentiels et approximation num~rique de fonctions valeur. 1re partie: ~tude th~orique, 2e pattie: ~tude num~rique. RAIRO Moddl. Math. Anal. Numdr., 25:517-560, 1991.

[ACS93]

L. AMBROSIO, P. CANNARSA, AND H.M. SONER. On the propagation of singularities of semi-convex functions. Ann. Scuola Norm. Sup. Pisa, (IV) 20:597-616, 1993.

[An96]

F. ANTONIALI. Problemi di controllo ottimo con variabili veloci e vincoli di stato: equazioni di Bellman e perturbazione singolare. Thesis, Universit~ di Padova, 1996.

534

BIBLIOGRAPHY

[Ar95a]

M. ARISAWA. Ergodic problem for the Hamilton-Jacobi-Bellman equation - Existence of the ergodic attractor. Cahiers de mathematiques de la decision, CEREMADE, 1995.

[Ar95b]

M. ARISAWA. Ergodic problem for the Hamilton-Jacobi-Bellman equation II. Cahiers de mathematiques de la decision, CEREMADE, 1995.

[ArL96]

M. ARISAWAAND P.L. LIONS. Continuity of admissible trajectories for state constrained control problems. Discrete Continuous Dynamical Systems, 2:297305, 1996.

[ART96]

~,I. ARISAWA AND A. TOURIN. Regularizing effects for a class of first-order Hamilton-Jacobi equations. Nonlinear Anal., 26, 1996.

[Au81]

J.P. AUBIN. Contingent derivatives of set-valued maps and existence of solutions to nonlinear inclusions and differential inclusions. In L. Nachbin, editor, Mathematical Analysis and Applications, Part A, volume 7A of Advances in Math. Supplementary Studies, pages 160-232. Academic Press, 1981.

[Au91]

J.P. AUBIN. Viability theory. Birkh~iuser, Boston, 1991.

[AC841

J . P . AUBIN .AND A. CELLINA. Differential inclusions. Springer, Berlin, 1984.

[AE84]

J.P. AUBIN AND I. EKELAND. Applied nonlinear analysis. Wiley, New York, 1984.

[AF90I

J.P. AUBIN AND H. FRANKOWSKA. Set valued analysis. Birkhhuser, Boston, 1990.

[Say91]

SAYAH AWATIF. t~quations d'Hamilton-Jacobi du premier ordre avec termes int4gro-diff4rentiels: I. Unicit4 des solutions de viscosit6: II. Existence de solutions de viscositY. Comm. Partial Differential Equations, 16:1057-1093, 1991.

[Bac86]

A. BACCIOTTI. Fondamenti geometrici della teoria della controllabilit& Number 31 in Quaderni U.M.I. Pitagora, Bologna, 1986.

[BBi90]

A. BACCIOTTIAND R.M. BIANCHINI. The linear time optimal problem revisited: regularization and approximation. Boll. Un. Mat. Ital., (7) 4-B:245-253, 1990.

[Bad94]

M. BADIALE. Geometrical properties of solutions of fully nonlinear equations and an application to singularities, g. Differential Equations, 112:33-52, 1994.

[BadB90]

hi. BADIALE AND ~I. BARDI. Symmetry properties of solutions of HamiltonJacobi equations. Nonlinear Anal., 15:1031-1043, 1990.

[BadB92]

~I. BADIALE AND 1~,[. BARDI. Asymptotic symmetry of solutions of nonlinear partial differential equations. Comm. Pure Appl. Math., 45:899-921, 1992.

[Bag93]

F. BAGAGIOLO. Soluzioni di viscositg vincolate di equazioni di Bellman e perturbazione singolare in problemi di controllo ottimo. Thesis, Universit~t di Padova, 1993.

[BagB96]

F. BAGAGIOLO AND ~,I. BARDI. Singular perturbation of a finite horizon problem with state-space constraints. Preprint 43, Universit~ di Padova, 1996.

[BBCD94] F. BAGAC-IOLO, ~I. BARDI, AND I. CAPUZZO DOLCETTA. A viscosity solutions approach to some asymptotic problems in optimal control. In G. Da Prato and J.P. Zolesio, editors, Partial differential equation methods in control and shape analysis (Pisa 199~), pages 29-39, New York, 1997. Marcel Dekker.

BIBLIOGRAPHY

535

[BHW93]

J.A. BALL, J.W. HELTON, AND M.L. WALKER. H~c control for nonlinear systems with output feedback. IEEE Trans. Automat. Control, 38:546-559, 1993.

[Bar93a]

V. BARBU. Analysis and control of nonlinear infinite dimensional systems. Academic Press, Boston, 1993.

[Bar93b]

V. BARBU. H~r control for semilinear systems in Hilbert spaces. Systems Control Left., 21:65-72, 1993.

[Bar95a]

V. BARBU. H ~ boundary control with state feedback: the hyperbolic case. SIAM J. Control Optim., 33:684-701: 1995.

[Bar95b]

V. BARBU. The H ~ problem for infinite dimensional semilinear systems. SIAM J. Control Optim., 33:1017-1027, 1995.

[BDaP83]

V. BARBU AND G. DA PRATO. Hamilton-Jacobi equations in Hilbert spaces, volume 86 of Research Notes in Mathematics. Pitman, 1983.

[B85]

hi. BARDI. Geometric properties of solutions of Hamilton-Jacobi equations. J. Differential Equations, 58:364-380, 1985.

[B87]

hi. BARDI. An asymptotic formula for the Green's function of an elliptic operator. Ann. Sc. Norm. Sup. Pisa, (IV) 14:569-586, 1987.

[B89]

hi. BARDI. A boundary value problem for the minimum time function. SIAM J. Control Optim., 26:776-785, 1989.

[B92]

M. BARDI. Viscosity solutions of Isaacs ' equations and existence of a value. In G. Ricci and C. Torricelli, editors, Proceedings of the Summer School on Dynamic Games, 1992.

[B90]

M. BARDI. Homogenization of quasilinear elliptic equations with possibly superqaudratic growth. In A. Marino and M.K.V. hIurthy, editors, Nonlinear variational problems and partial differential equations (Isola d'Elba, 1990), volume 320 of Pitman Res. Notes Math. Set., pages 44-56. Longman Sci. Tech., Harlow, 1995.

[B95]

hi. BARDI. Some applications of viscosity solutions to optimal control and differential games. In I. Capuzzo Dolcetta and P.L. Lions, editors, Viscosity solutions and applications (Montecatini~ 1995), volume 1660 of Lecture Notes in Mathematics, pages 44-97. Springer, Berlin, 1997.

[BB95]

hi. BARDI AND S. BOTTACIN. Discontinuous solutions of degenerate elliptic boundary value problems. Preprint 22, Universit~ di Padova, 1995.

[BBF95]

hi. BARDI, S. BOTTACIN, AND ~I. FALCONE. Convergence of discrete schemes for discontinuous value functions of pursuit-evasion games. In G.J. O1sder, editor, New Trends in Dynamic Games and Application, pages 273-304. Birkh~user, Boston, 1995.

[BDL96]

~i. BARDI AND F. DA LIo. On the Bellman equation for some unbounded control problems. NoDEA Nonlinear Differential Equations Appl., to appear.

[BE84]

hi. BARDI AND L.C. EVANS. On Hopf's formulas for solutions of HamiltonJacobi equations. Nonlinear Anal., 8:1373-1381, 1984.

[BFag96]

hi. BARDI AND S. FAGGIAN. Hopf-type estimates and formulas for non-convex non-concave Hamiton-Jacobi equations. SIAM J. Math. Anal., to appear.

536

BIBLIOGRAPHY

[BF90a]

l~.I. BARDI AND M. FALCONE. An approximation scheme for the minimum time function. SIAM J. Control Optim., 28:950-965, 1990.

[BF90b]

l~I. BARDI AND ~I. FALCONE. Discrete approximation of the minimal time function for systems with regular optimal trajectories. In A. Bensoussan and J.L. Lions, editors, Analysis and Optimization of Systems, volume 144 of Lecture Notes in Control and Inform. Sci., pages 103-112. Springer, Berlin, 1990.

[BFS94]

h{. BARDI, M. FALCONE, AND P. SORAVIA. Fully discrete schemes for the value function of pursuit-evasion games. In T. Basar and A. Haurie, editors, Advances in Dynamic Games and Applications, pages 89-105. Birkhguser, Boston, 1994.

[BFS97]

hi. BARDI, hi. FALCONE, AND P. SORAVIA. Numerical methods for pursuit~ evasion games via viscosity solutions. In M. Bardi, T. Parthasarathy, and T.E.S. Raghavan, editors, Stochastic and differential games: Theory and numerical methods. Birkhs Boston, to appear.

[BG97]

h.I. BARDI AND P. GOATIN. A Dirichlet type problem for nonlinear degenerate elliptic equations arising in time-optimal stochastic control. Preprint, SISSAISAS, Trieste, 1997.

[BKS96]

M. BARDI, S. KOIKE, AND P. SORAVIA. On the pursuit evasion game with state constraints. Preprint, 1996.

[BOs91]

M. BARDI AND S. OSHER. The nonconvex multi-dimensional Riemann problem for Hamilton-Jacobi equations. SIAM g. Math. Anal., 22:344-351, 1991.

[BP90]

.~I. BARDI AND B. PERTHAME. Exponential decay to stable states in phase transitions via a double log-transformation. Comm. Partial Differential Equations, 15:1649-1669, 1990.

[BP91]

~I. BARDI AND B. PERTHAME. Uniform estimates for some degenerating quasilinear elliptic equations and a bound on the Harnack constant for linear equations. Asymptotic Anal., 4:1-16, 1991.

[BSa91a]

.~[. BARDI AND C. SARTORI. Approximations and regular perturbations of optimal control problems via Hamilton-Jacobi theory. Appl. Math. Optim., 24:113-128, 1991.

[BSa91b]

hi. BARDI AND C. SARTORI. Differential games and totally risk-averse optimal control of systems with small disturbances. In R.P. Hamalainen and H.K. Ethamo, editors, Differential Games - - Developments in modeling and computation, volume 156 of Lecture Notes in Control and Inform. Sci., pages 91-99. Springer, Berlin, 1991.

[BSa92]

l~l. BARDI AND C. SARTORI. Convergence results for Hamilton-JacobiBellman equations in variable domains. Differential Integral Equations, 5:805816, 1992.

[BS89]

hi. BARDI AND P. SORAVIA. A PDE framework for differential games of pursuit-evasion type. In T. Basar and P. Bernhard, editors, Differential games and applications, volume 119 of Lecture Notes in Control and Inform. Sci., pages 62-71. Springer, Berlin, 1989.

[BS91a]

hi. BARDI AND P. SORAV1A. Hamilton-Jacobi equations with singular boundary conditions on a free boundary and applications to differential games. Trans. Amer. Math. Soc., 325:205-229, 1991.

BIBLIOGRAPHY

537

[BS91b]

M. BARDI AND P. SORAVIA. Approximation of differential games of pursuitevasion by discrete-time games. In R.P. Hamalainen and H.K. Ethamo, editors, Differential Games - - Developments in modelling and computation, volume 156 of Lecture Notes in Control and Inform. Sci., pages 131-143. Springer, Berlin, 1991.

[BS91c]

~I. BARDI AND P. SORAVIA. Time-optimal control, Lie brackets, and Hamilton-Jacobi equations. Technical report, Universith di Padova, 1991.

[BS921

]kI. BARDI AND P. SORAVIA. Some remarks on the Bellman equation of the minimum time function with superlinear vector fields. Technical report, Universit~ di Padova, 1992.

[BS94]

/~,[. BARDI AND P. SORAVIA. A comparison result for Hamilton-Jacobi equations and applications to some differential games lacking controllability. Funkcial. Ekvac., 37:19-43, 1994.

[BSt93]

M. BARDI AND V. STAICU. The Bellman equation for time-optimal control of non-controllable nonlinear systems. Acta Appl. Math., 31:201-223, 1993.

[Ba84]

G. BARLES. Existence results for first-order Hamilton-Jacobi equations. Ann. Inst. H. Poincarg Anal. Non Lindaire, 1:325-340, 1984.

[Ba85a]

G. BARLES. Deterministic impulse control problems. SIAM J. Control Optim., 23:419-432, 1985.

[Ba85b]

G. BARLES. Quasi-variational inequalities and first-order Hamilton-Jacobi equations. Nonlinear Anal., 9:131-148, 1985.

[Ba85c]

G. BARLES. Asymptotic behavior of viscosity solutions of first order HamiltonJacobi equations. Ricerehe Mat., 2:227-260, 1985.

[Ba90a]

a . BARLES. An approach of deterministic control problems with unbounded data. Ann. Inst. H. Poincard Anal. Non Lindaire, 7:235-258, 1990.

[Ba90b]

a . BARLES. Uniqueness and regularity results for first-order Hamilton-Jacobi equations. Indiana Univ. Math. J., 39:443-466, 1990.

[Bag0c]

G. BARLES. Regularity results for first-order Hamilton-Jacobi equations. Differential Integral Equations, 3:103-125, 1990.

[Ba93]

G. BARLES. Discontinuous viscosity solutions of first order Hamilton-Jacobi equations: a guided visit. Nonlinear Anal., 20:1123-1134, 1993.

[Ba94]

C. BARLES. Solutions de viscositd des dquations de Hamilton-Jacobi, volume 17 of Mathematiques et Applications. Springer, Paris, 1994.

[BES90]

C. BARLES, L.C. EVANS, AND P.E. SOUGANIDIS.Wavefront propagation for reaction-diffusion systems. Duke Math. J., 61:835-858, 1990.

[BaL911

C. BARLES AND P.L. LIONS. Fully nonlinear Neumann type boundary conditions for first-order Hamilton-Jacobi equations. Nonlinear Anal., 16:143-153, 1991.

[BaP87]

G. BARLES AND B. PERTHAME. Discontinuous solutions of deterministic optimal stopping time problems. RAIRO Moddl. Math. Anal. Numgr., 21:557579, 1987.

[BaP88]

G. BARLES AND B. PERTHAME, Exit time problems in optimal control and vanishing viscosity method. SIAM J. Control Optim., 26:1133-1148, 1988.

538

BIBLIOGRAPHY

[BaP90]

G. BARLES AND B. PERTHAME. Comparison principle for Dirichlet-type Hamilton-Jacobi equations and singular perturbations of degenerated elliptic equations. Appl. Math. Optim., 21:21-44, 1990.

[BSS93]

G. BARLES, H.M. SONER, AND P.E. SOUGANIDIS. Front propagation and phase field theory. SIAM J. Control Optim., 31:439-469, 1993.

[BaSo91]

G. BARLES AND P.E. SOUGANIDIS. Convergence of approximation schemes for fully nonlinear second order equations. Asymptotic Anal., 4:271-283, 1991.

[Bn85]

E.N. BARRON. Viscosity solutions for the monotone control problem. SIAM J. Control Optim., 23:161-171, 1985.

[Sn90]

E.N. BARRON. Differential games with maximum cost. 14:971-989, 1990.

[Bn93]

E.N. BARRON. Averaging in Lagrange and minimax problems of optimal control. SIAM J. Control Optim., 31:1630-1652, 1993.

[BEJ84]

E.N. BARRON, L.C. EVANS, AND R. JENSEN. Viscosity solutions of Isaacs' equation and differential games with Lipschitz controls. J. Differential Equations, 53:213-233, 1984.

[BI89]

E.N. BARRON AND H. ISHII. The Bellman equation for minimizing the maximum cost. Nonlinear Anal., 13:1067-1090, 1989.

[BJ80]

E.N. BARRON AND R. JENSEN. Optimal control problems with no turning back. J. Differential Equations, 36:223-248, 1980.

[BJ86]

E.N. BARRON AND R. JENSEN. The Pontryagin maximum principle from dynamic programming and viscosity solutions to first-order partial differential equations. Trans. Amer. Math. Soc., 298:635-641, 1986.

[BJ871

E.N. BARRON AND Ft..~ENSEN. Generalized viscosity solutions for HamiltonJacobi equations with time-measurable Hamiltonians. J. Differential Equations, 68:10-21, 1987.

[BJ89]

E.N. BARRONAND Ft. JENSEN. Total risk aversion, stochastic optimal control, and differential games. Appl. Math. Optim., 19:313 327, 1989.

[BJ90]

E.N. BARRON AND Ft. JENSEN. Semicontinuous viscosity solutions of Hamilton-Jacobi equations with convex Hamiltonians. Comm. Partial Differential Equations, 15:1713-1742, 1990.

[BJ91]

E.N. BARRON AND n . JENSEN. Optimal control and semicontinuous viscosity solutions. Proc. Arner. Math. Soc., 113:397-402, 1991.

[BJ95]

E.N. BARRON AND R. JENSEN. Relaxed minimax control. SIAM J. Control Optim., 33:1028-1039, 1995.

[BJL96]

E.N. BARRON, R. JENSEN, AND W. LIU. Hopf-Lax-type formulas for ut + H(x, Du) -= O. J. Differential Equations, 126:48-61, 1996.

[BJM93]

E.N. BARRON, R. JENSEN, AND J.L. ~IENALDI. Optimal control and differential games with measures. Nonlinear Anal., 21:241 268, 1993.

[BLg6a]

E.N. BARRON AND W. LIU. Optimal control of the blowup time. SIAM J. Control Optim., 34:102-123, 1996.

Nonlinear Anal.,

BIBLIOGRAPHY

539

[BL96b]

E.N. BARRON AND W. LIU. Semicontinuous solutions for Hamilton-Jacobi equations and the L~C-control problem. Appl. Math. Optim., 34:325-360, 1996.

[BL97]

E.N. BARRON AND W. LIu. Semicontinuous and continuous blowup and minimal time functions. J. Differential Equations, to appear.

[BaBe91]

W. BASAR AND P. BERNHARD. HOe-optimal control and related minimax

design problems. Birkhs [BAH94]

Boston, 2 edition, 1995.

T. BASAR AND A. HAURIE, editors. Advances in Dynamic Games and Applications, volume 1 of Ann. Internat. Soc. Dynam. Games, Boston, 1994. Birkhs

[BaO82]

T. BASAR AND G.J. OLSDER. Dynamic non-cooperative game theory. Academic Press, New York, 1982.

[Bat78]

G.R. BATES. Lower closure and existence theorems for optimal control problems with infinite horizon. J. Optim. Theory Appl., 24:639-649, 1978.

[Be157]

R. BELLMAN. Dynamic Programming. Princeton University Press, Princeton, NJ, 1957.

[Bel71]

R. BELLMAN. Introduction to the Mathematical Theory of Control Processes, volume 2. Academic Press, New York, 1971.

[BelD62]

R. E. BELLMAN AND S.E. DREYFUS. Applied Dynamic Programming. Princeton University Press, Princeton, NJ, 1962.

[BM96]

PH. BENILANAND M. ~IALIKI.Approximation de la solution de viscosite d'un probleme d'Hamilton-Jacobi. Rev. Mat. Univ. Complut. Madrid, 9:369-383, 1996.

[Ben88]

A. BENSOUSSAN. Perturbation methods in optimal control. Gauthier-ViUars, Paris, 1988.

[BDDM93] A. BENSOUSSAN,G. DA PRATO, ~I.C. DELFOUR, AND hhTTER S.K. Representation and control of infinite dimensional systems, volume 2. Birkhguser, Boston, 1993. [BL82]

A. BENSOUSSANAND J.L. LIONS. Contrfle impulsionnel et ingquations quasi variationneUes. Dunod, Paris, 1982.

[Bt77]

S. BENTON. The Hamilton-Jacobi equation: a global approach. Academic Press, New York, 1977.

[Be64]

L.D. BERKOVITZ. A differential game with no pure strategy solution. In M. et al. Dresher, editor, Advances in Game Theory, volume 52 of Annals of Mathematical Studies, pages 175-194. Princeton University Press, Princeton, NJ, 1964.

[Be85]

L.D. BERKOVITZ. The existence of value and saddle point in games of fixed duration. SIAM J. Control Optim., 23:172-196, 1985. Errata and addenda in SIAM J. Control Optim. 26:740-742, 1988.

[Be88]

L.D. BERKOVITZ. Characterizations of the values of differential games. Appl. Math. Optim., 17:177-183, 1988.

[Be89]

L.D. BERKOVITZ. OptimM feedback controls. 27:991-1006, 1989.

SIAM .l. Control Optim.,

540

BIBLIOGRAPHY

[Be94]

L.D. BERKOVITZ. A theory of differential games. In T. Basar and A. Haurie, editors, Advances in dynamic games and applications, pages 3-22. Birkhs Boston, 1994.

[Ber76]

P. BERNHARD. Commande optimale, dgcentralization et jeux dynamiques. Dunod, Paris, 1976.

[BerT7]

P. BERNHARD. Singular surfaces in differentiM games. In P. Hagedorn, Knobloch H.W., and Olsder G.J., editors, Differential games and applications, volume 3 of Lecture Notes in Control and Inform. Sci., pages 1-33. Springer, 1977.

[Ber92]

P. BERNHARD. Differential games: lectures notes on the Isaacs-Breakwell theory. In G. Ricci and C. Torricelli, editors, Proceedings of the Summer School on Dynamic Games, 1992.

IBIS90]

R.M. BIANCHINI AND G. STEFANI. Time-optimal problem and time-optimal map. Rend. Sere. Mat. Univ. Pol. Torino, 48:401-429, 1990.

IBIS93]

R.M. BIANCHINI AND G. STEFANI. Controllability along a trajectory: a variational approach. SIAM J. Control Optim., 31:900-927, 1993.

[Bi83]

R.M. BIANCHINI TIBERIO. Instant controllability of linear autonomous systems. J. Optim. Theory Appl., 39:237-250, 1983.

[Bla97]

A.P. BLANC. Deterministic exit time problems with discontinuous exit cost. SIAM J. Control Optim., 35:399-434, 1997.

[Bo166]

V.G. BOLTYANSKII. Sufficient conditions for optimality and the justification of the Dynamic Programming Principle. SIAM J. Control, 4:326-361, 1966.

[BLP97]

B. BONNARD, G. LAUNAY, AND ~,I. PELLETIER. Classification generique de synthese temps minimales avec cible de codimension un et applications. Ann. Inst. H. Poincarg Anal. Non Lingaire, 14:55-102, 1997.

[Bon69]

J.-M. BONY. Principe du maximum, inegalite de Harnack et unicite du probleme de Cauchy pour les operateurs elliptiques degeneres. Ann. Inst. Fourier Grenoble, 19,1:277-304, 1969.

[Bo93]

S. BORTOLETTO.The Bellman equation for constrained deterministic optimal control problems. Differential Integral Equations, 6:905-924, 1993.

[Bot94]

N.D. BOTKIN. Approximation schemes for finding the value functions for differential games with nonterminal payoff functional. Analysis, 14:203-220, 1994.

[BrS9]

J.V. BREAKWELL. Time-optimal pursuit inside a circle. In T. Basar and P. Bernhard, editors, Differential games and applications, volume 119 of Lecture Notes in Control and Inform. Sci., pages 72-85. Springer, Berlin, 1989.

[BrBg0]

J.V. BREAKWELLAND P. BERNHARD. A simple game with a singular focal line. J. Optim. Theory Appl., 64:419-428, 1990.

[Bren89]

Y. BRENIER. Un algorithme rapide pour le calcul de transform~es de Legendre-Fenchel discretes. C. R. Acad. Sci. Paris Sdr. I Math., 308:587589, 1989.

[Bre80]

A. BRESSAN. On two conjectures by Hhjek. Funkcial. Ekvac., 23:221-227, 1980.

BIBLIOGRAPHY

541

[Bre93]

A. BRESSAN. Lecture notes on the mathematical theory of control. International School for Advanced Studies, Trieste, 1993.

[BPi95]

A. BRESSAN AND B. PICCOLI. A Baire category approach to the bang-bang property. J. Differential Equations, 116:318-337, 1995.

[Sru80]

P. BRUNOVSKY. Existence of regular synthesis for general problems. J. Dif-

ferential Equations, 38:317-343, 1980. [BH87]

A.E. BRYSON AND Y.C. Ho. Applied Optimal Control. Hemisphere, Washington DC, 1987.

[CafCa95] L. CAFFARELLIAND X. CABRE. Fully nonlinear elliptic equations, volume 43 of Colloquium Publ. Amer. Math. Soc., Providence, RI, 1995. [CCKS961 L. CAFFARELLI~~I.C. CRANDALL~~I. KOCAN, AND A. SWIECH. On viscosity solutions of fully nonlinear equations with measurable ingredients. Comm. Pure Appl. Math., 49:365-397~ 1996. [Oaf89]

L.A. CAFFARELLI. Interior a priori estimates for solutions of fully non-linear equations. Ann. Math., 130:189-213, 1989.

[Cam96]

F. CAMILLI. Computation of the ~ r norm for nonlinear systems: a convergence result. Systems Control Lett., 28:139-150, 1996.

[CFa95a]

F. CAMILLI AND ~I. FALCONE. An approximation scheme for the optimal control of diffusion processes. RAIRO Mod~l. Math. Anal. Numdr., 29:97-122, 1995.

[CFa95b]

F. CAMILLI AND M. FALCONE. Approximation of control problems involving ordinary and impulsive controls I. Preprint, UniversitY. di Roma "La Sapienza", 1995.

[CFa961

F. CAMILLI AND ~I. FALCONE. Approximation of optimal control problems with state constraints: estimates and applications. In B.S. Mordukhovie and H.J. Sussman, editors~ Nonsmooth analysis and geometric methods in deterministic optimal control, volume 78 of LM.A. Volumes in Applied Mathematics, pages 23-57. Springer, New York, 1996.

[CFLS94]

F. CAMILLI, hi. FALCONE, P. LANUCARA,AND A. SEGHINI. A domain decomposition method for Bellman equations. In D.E. Keyes and Xu J., editors,

Domain Decomposition methods in Scientific and Engineering Computing, volume 180 of Contemp. Math., pages 477-483. Amer. Math. Soc., 1994. [CamS95]

F. CAMILLIAND A. SICONOLFI. Discontinuous solutions of a Hamilton-Jacobi equation with infinite speed of propagation. Preprint, Universith di Roma ':La Sapienza", 1995.

[CDaP89]

P. CANNARSAAND a . DA PRATO. Nonlinear optimal control with infinite horizon for distributed parameter systems and stationary Hamilton-Jacobi equations. SIAM Y. Control Optim., 27:861-875, 1989.

[CDaP90]

P. CANNARSAAND G. DA PRATO. Some results on nonlinear optimal control problems and Hamilton-Jacobi equations in infinite dimensions. Y. Funct. Anal., 90:27-47, 1990.

[CF91]

P. CANNARSA AND H. FRANKOWSKA. Some characterizations of optimal trajectories in control theory. SIAM J. Control Optim., 29:1322-1347, 1991.

542

BIBLIOGRAPHY

[ccs91]

P. CANNARSA, F. GozzI, AND H.M. SONER. A boundary value problem for Hamilton-Jacobi equations in Hilbert spaces. Appl. Math. Optim., 24:197-220, 1991.

[CGS931

P. CANNARSA~ F. GOZZI, AND H.M. SONER. A dynamic programming approach to nonlinear boundary control problems of parabolic type. J. Funct. Anal., 117:25-61, 1993.

[CSi95a]

P. CANNARSA AND C. SINESTRARI. Convexity properties of the minimum time function. Calc. Var., 3:273-298, 1995.

[CSi95b]

P. CANNARSA AND C. SINESTRARI. On a class of minimum time problems.

Discrete Continuous Dynamical Systems, 1:285-300, 1995.

[CS87]

P. CANNARSAAND H.M. SONER. 0 n the singularities of the viscosity solutions to Hamilton-Jacobi-Bellmanequations. Indiana Univ. Math. J., 36:501524, 1987.

[cs89]

P. CANNARSA AND H.M. SONER. Generalized one-sided estimates for solutions of Hamilton-Jacobi equations and applications. Nonlinear Anal., 13(3):305-323, 1989.

[CT96]

P. CANNARSAAND M.E. TESSITORE. Infinite dimensional Hamilton-Jacobi equations and Dirichlet boundary control problems of parabolic type. SIAM Y. Control Optim., 34:1831-1847, 1996.

[CD831

1. CAPUZZO DOLCETTA. On a discrete approximation of the Hamilton-Jacobi equation of dynamic programming. Appl. Math. Optim., 10:367-377, 1983.

[CDE84]

I. CAPUZZO DOLCETTA AND L.C. EVANS. Optimal switching for ordinary differential equations. SIAM J. Control Optim., 22:1133-1148, 1984.

[CDF891

I. CAPUZZO DOLCETTA AND ~I. FALCONE. Viscosity solutions and discrete dynamic programming. Ann. Inst. H. Poincard Anal. Non Lindaire, 6 (Supplement):161-183, 1989.

[CDG86]

I. CAPUZZO DOLCETTA AND ~,I.a. GARRONI. Oblique derivative problems

and invariant measures. Ann. Sc. Norm. Sup. Pisa, 13:689-720, 1986.

[CDISn]

I. CAPUZZO DOLCETTA AND H. ISHII. Approximate solutions of the Bellman equation of deterministic control theory. Appl. Math. Optim., 11:161-181, 1984.

[CDL90]

I. CAPUZZO DOLCETTA AND P.L. LIONS. Hamilton-Jacobi equations with state constraints. Trans. Amer. Math. Soc., 318:643-683, 1990.

[CDL97]

I. CAPUZZO DOLCETTA AND P.L. LIONS, editors. Viscosity solutions and applications (Montecatini, 1995), volume 1660 of Lecture Notes in Mathematics, Berlin, 1997. Springer.

[CDM81]

I. CAPUZZO DOLCETTA AND ~I. I~IATZEU. On the dynamic programming

inequalities associated with the deterministic optimal stopping problem in discrete and continuous time. Num. Funct. Anal. Optim., 3, 1981. [CDMe86] I. CAPUZZO DOLCETTA AND J.L. hIENALDI. On the deterministic optimal stopping time problem in the ergodic case. In C.I. Byrnes and A. Lindquist, editors, Theory and Applications of Nonlinear Control Systems, pages 453460. North-Holland, 1986.

BIBLIOGRAPHY

543

[CDMe88] I. CAPUZZODOLCETTAAND J.L. MENALDI. Asymptotic behavior of the first order obstacle problem. J. Differential Equations, 75(2):303-328, 1988. [CDP96]

I. CAPUZZODOLCETTAAND B. PERTHAME. On some analogy between different approaches to first order PDE~s with non smooth coefficients. Adv. Math. Sci. Appl., 6:689-703, 1996.

[Ca35]

C. CARATHI~ODORY. Calculus of variations and partial differential equations of the first order. Teubner, Berlin, 1935. (In German). Second (revised) English edition: Chelsea, New York, 1982.

[CQS94a]

P. CARDALIAGUET,~'L QUINCAMPOIX, AND V. SAINT-PIERRE. Some algorithms for differential games with two players and one target. RAIRO Modgl. Math. Anal. Numdr., 28:441-461, 1994.

[CQS94b]

P. CARDALIAGUET,~I. QUINCAMPOIX, AND P. SAINT-PIERRE. Temps optimaux pour des probl~mes de controle avec contraintes et sans controlabilit~ locale. C. R. Acad. Sci. Paris Sdr. I Math., 318:607-612, 1994.

[CQS97]

P. CARDALIAGUET,~,I. QUINCAMPOIX, AND P. SAINT-PIERRE. Set-valued numerical analysis for optimal control and differential games. In M. Pardi, T. Parthasarathy, and T.E.S. Raghavan, editors, Stochastic and differential games: Theory and numerical methods. Birkhi~user, Boston, to appear.

[Car93]

F. CARDIN. On viscosity and geometrical solutions of Hamilton-Jacobi equations. Nonlinear Anal., 20:713-719, 1993.

[CH87]

A.D. CARLSON AND A. HAURIE. Infinite horizon optimal control, volume 290 of Lecture Notes in Economics and Mathematical Systems. Springer, 1987.

[c~92]

V. CASELLES. Scalar conservation laws and Hamilton-Jacobi equations in one-space variable. Nonlinear Anal., 18:461-469, 1992.

[Ce83]

L. CESARI. Optimization - - Theory and Applications. Springer, New York, 1983.

[CGG91]

Y.G. CHEN, Y. GIGA, AND S. GOTO. Uniqueness and existence of viscosity solutions of generalized mean curvature flow equations. J. Differential Geom., 33:749-786, 1991.

[ChSu95]

A.G. CHENTSOV AND A.I. SUBBOTIN. Iterative procedure for constructing minimax and viscosity solutions of Hamilton-Jacobi equations. Preprint, 1995. Russian version in Dokl. Akad. Nauk 348:736-739, 1996.

[Cla75]

F.H. CLARKE. Generalized gradients and applications. Trans. Amer. Math. Soc., 205:247-262, 1975.

[ClaS3] [c1~9]

F.H. CLARKE. Optimization and nonsmooth analysis. Wiley, New York, 1983. F.H. CLARKE. Methods of Dynamic and Nonsmooth Optimization, volume 57 of CBMS-NSF Regional Conference Series in Applied Math. SIAM, Philadelfia, 1989.

[ClaL94]

F.H. CLARKEAND Yu.S. LEDYAEV. Mean value inequalities in Hilbert space. Trans. Amer. Math. Soc., 344:307-324, 1994.

[CLSW95]

F.H. CLARKE, Yu.S. LEDYAEV, R.J. STERN, AND P.R. WOLENSKI. Qualitative properties of trajectories of control systems: a survey. J. Dynamical Control Systems, 1:1-48, 1995.

544

BIBLIOGRAPHY

[CLS97]

F.H. CLARKE, Yu.S. LEDYAEV, AND A.I. SUBBOTIN. Universal feedback strategies for differential games of pursuit. SIAM J. Control Optim., 35:552561, 1997.

[ClaV87]

F.H. CLARKE AND R.B. VINTER. The relationship between the maximum principle and dynamic programming. SIAM J. Control Optim., 25:1291-1311, 1987.

[Co189]

F. COLONIUS. Asymptotic behaviour of optimal control systems with low discount rates. Math. Oper. Res., 14:309-316, 1989.

[Co85]

R. CONTI. Processi di controllo in lt~~, volume 30 of Quaderni U.M.I. Pitagora, Bologna, 1985.

[Cor96]

L. CORRIAS. Fast Legendre-Fenchel transform and applications to HamiltonJacobi equations and conservation laws. SIAM J. Numer. Anal., 33:1534-1558: 1996.

[c95]

M.G. CRANDALL. Viscosity solutions: a primer. In I. Capuzzo Dolcetta and P.L. Lions, editors, Viscosity solutions and applications (Montecatini, 1995), volume 1660 of Lecture Notes in Mathematics, pages 1-43. Springer, Berlin, 1997.

[CEL84]

M.G. CRANDALL,L.C. EVANS, AND P.L. LIONS. Some properties of viscosity solutions of Hamilton-Jacobi equations. Trans. Amer. Math. Soc, 282:487502, 1984.

[CIL87]

M.G. CRANDALL, H. ISHII, AND P.L. LIONS. Uniqueness of viscosity solutions revisited. J. Math. Soc. Japan, 39:581-596, 1987.

[CIL92]

M.G. CRANDALL, H. ISHII, AND P.L. LIONS. User's guide to viscosity solutions of second order partial differential equations. Bull. Amer. Math. Soc., 27:1-67, 1992.

[CKSS96]

M.G. CRANDALL, ]kl. KOCAN, P. SORAVIA, AND A. SWIECH. On the equivalence of various weak notions of solutions of elliptic PDEs with measurable ingredients. In A. et al. Alvino, editor, Progress in elliptic and parabolic partial differential equations, volume 350 of Pitman Research Notes, pages 136-162. Longman, Harlow, 1996.

[CL81]

~[.G. CRANDALL AND P.L. LIONS. Conditions d'unicit~ pour les solutions generalises des equations d'Hamilton-Jaeobi du premier ordre. C. R. Acad. Sci. Paris Sgr. I Math., 292:487-502, 1981.

[CL83]

M.G. CRANDALL AND P.L. LIONS. Viscosity solutions of Hamilton-Jacobi equations. Trans. Amer. Math. Soc., 277:1-42, 1983.

[CL84]

~[.G. CRANDALL AND P.L. LIONS. Two approximations of solutions of Hamilton-Jacobi equations. Math. Comp., 43:1-19, 1984.

[CL85]

M.G. CRANDALL AND P.L. LIONS. Hamilton-Jacobi equations in infinite

dimensions. Part I: Uniqueness of solutions. J. Funct. Anal., 62:379-396, 1985. [CL86]

Ik[.G. CRANDALLAND P.L. LIONS. On existence and uniqueness of solutions of Hamilton-Jacobi equations. Nonlinear Anal., 10:353-370, 1986.

[CL87]

]~LG. CRANDALLAND P.L. LIONS. Remarks on the existence and uniqueness of unbounded viscosity solutions of Hamilton-Jacobi equations. Illinois J. Math., 31:665-688, 1987.

BIBLIOGRAPHY

545

[CL94a]

M.G. CRANDALL AND P.L. LIONS. Viscosity solutions of Hamilton-Jacobi equations in infinite dimensions. Part Vh Nonlinear A and Tataru's method refined. In P. Cl@ment and G. Lumer, editors, Evolution equations, control theory, and biomathematics, volume 155 of Lect. Notes Pure Appl. Math., pages 51-89. Marcel Dekker, New York, 1994.

[CL94b]

M.G. CRANDALL AND P.L. LIONS. Viscosity solutions of Hamilton-Jacobi equations in infinite dimensions, part VII: The HJB equation is not always satisfied. J. Funct. Anal., 125:111-148, 1994.

[CL96]

M.G. CRANDALL AND P.L. LIONS. Convergent difference schemes for nonlinear parabolic equations and mean curvature motion. Numer. Math., 75:17-41, 1996.

[CLS89]

hi.G. CRANDALL, P.L. LIONS, AND P.E. SOUGANIDIS. Maximal solutions and universal bounds for some partial differential equations of evolution. Arch. Rat. Mech. Anal., 105:163-190, 1989.

[CN85]

M.G. CRANDALLAND R. NEWCOMB. Viscosity solutions of Hamilton-Jacobi equations at the boundary. Proc. Amer. Math. Soc., 94:283-290, 1985.

[Cu69]

J. CULLUM. Discrete approximations to continuous optimal control problems. SIAM J. Control Optim., 7:32-49, 1969.

[Cut93]

A. CUTRi. Some remarks on Hamilton-Jacobi equations and non convex minimization problems. Rend. Mat. Appl., 13:733-749, 1993.

[DL96]

F. DA LIO. On the Bellman equation for infinite horizon problems with unbounded cost functional. Preprint, Universits di Padova, 1996.

[DAM96]

B. DACOROGNA AND P. ~IARCELLINI. Theoremes d'existence dans les cas scalaire et vectoriel pour les equations de Hamilton-Jacobi. C. R. Acad. Sci. Paris Sdr. I Math., 322:237-240, 1996.

[DM93]

G. DAL MASO. An Introduction to F-convergence. Birkh~iuser, Boston, 1993.

[DeG92]

E. DE GIORGI. New ideas in calculus of variations and geometric measure theory. In G. Buttazzo and A. Visintin, editors, Motion by mean curvature and related topics (Trento, 1992), pages 63-69. de Gruyter, Berlin, 1994.

[DeG90]

E. DE GIORGI. New conjectures on flow by mean curvature. In A. Marino and M.K.V. Murthy, editors, Nonlinear variational problems and partial differential equations (Isola d'Elba, 1990), volume 320 of Pitman Res. Notes Math. Set., pages 120-128. Longman Sci. Tech., Harlow, 1995.

[DeM71]

V . F . DEM;YANOV AND V.N. ~[ALOZEMOV. On the theory of nonlinear minimax problems. Russian Math. Surveys, 26:57-115, 1971.

[DMG95]

S. DI MARCO AND R.L.V. GONZALEZ. Une procedure numerique pour la minimisation du cout maximum. C. R. Acad. Sci. Paris Sgr. I Math., 321:869873, 1995.

[Di92]

G. DfAZ. Blow-up time involved with perturbed Hamilton-Jacobi equations. Proc. Roy. Soc. Edinburgh, 122 A:17-44, 1992.

[DR93]

G. DfAZ AND J.M. REY. Finite extinction time for some perturbed HamiltonJacobi equations. Appl. Math. Optim., 27:1-33, 1993.

546

BIBLIOGRAPHY

[D1877]

U. DINI. Sulle fimzioni finite continue di variabili reali che non hanno mai derivata. Atti R. Acc. Lincei, (3) 1:130-133, 1877.

[DZ93]

A.L. DONTCHEV AND T. ZOLEZZI. Well-Posed Optimization Problems, volume 1543 of Lecture Notes in Mathematics. Springer, 1993.

[Dou61]

A. DOUGLIS.The continuous dependence of generalized solutions of non-linear partial differential equations upon initial data. Comm. Pure Appl. Math., 14:267-284, 1961.

[DGKF89] J.C. DOYLE~ K. GLOVER, P.P. KHARGONEKAR,AND B.A. FRANCIS. Statespace solutions to standard H2 and H~r control problems. IEEE Trans. Automat. Control, 34:831-847, 1989. [DI90]

P. DuPuIS AND H. ISHII. On oblique derivative problems for fully nonlinear second-order elliptic partial differential equations on nonsmooth domains. Nonlinear Anal., 15:1123-1138, 1990.

[DI91]

P. DuPuIs AND H. ISHII. On oblique derivative problems for fully nonlinear second-order elliptic PDEs on domains with corners. Hokkaido Math. J., 20:135-164, 1991.

[DIS90]

P. DUPUIS, H. ISHII, AND H.M. SONER. A viscosity solution approach to the asymptotic analysis of queueing systems. Ann. Probab., 18:226-255, 1990.

[DME97]

P.G. DuPuIS AND W.M. MCENEANEY. Risk-sensitive and robust escape criteria. SIAM J. Control Optim., to appear.

[Ei87]

A. EIZENBERG.The vanishing viscosity method for Hamilton-Jacobi equations with a singular point of attracting type. Comm. Partial Differential Equations, 12:1-20, 1987.

[Ei90]

A. EIZENBERG. The exponential leveling in elliptic singular perturbation pro-

blems with complicated attractors. J. Analyse Math., 55:229-249, 1990. [Ei93]

A. EIZENBERG. Elliptic perturbations for a class of Hamilton-Jacobi equations. Asymptotic Anal., 7:251-285, 1993.

[E177]

R.J. ELLIOTT. Feedback strategies in deterministic differential games. In P. Hagedorn, Knobloch H.W., and Olsder G.J., editors, Differential games and applications, volume 3 of Lecture Notes in Control and Inform. Sci., pages 136-142. Springer, 1977.

[E187]

R.J. ELLIOTT. Viscosity solutions and optimal control, volume 165 of Pitman Research Notes in Math. Longman, Harlow, 1987.

[EK72]

R.J. ELLIOTT AND N.J. KALTON.The existence of value in differential games. Mem. Amer. Math. Soc., 126, 1972.

[EK74]

R.J. ELLIOTT AND N.J. KALTON. Cauchy problems for certain IsaacsBellman equations and games of survival. Trans. Amer. Math. Soc., 198:45-72, 1974.

[En86]

H. ENGLER. On Hamilton-Jacobi equations in bounded domains. Proc. Roy.

Soc. Edinburgh, 102 A:221-242, 1986. [EL91]

g. ENGLER AND S.~L LENHART. Viscosity solutions for weakly coupled systems of Hamilton-Jacobi equations. Proc. London Math. Soc., (3)63:212-240., 1991.

BIBLIOGRAPHY

547

[EM94]

J.R. ESTEBAN AND P. ~.[ARCATI. Approximate solutions to first and second order quasilinear evolution equations via nonlinear viscosity. Trans. Amer. Math. Soc., 342:501-521, 1994.

[E78]

L.C. EVANS. A convergence theorem for solutions of nonlinear 2nd order elliptic equations. Indiana Univ. Math. J., 27:875-887, 1978.

[E80]

L.C. EVANS. On solving certain nonlinear partial differential equations by accretive operator methods. Israel J. Math., 36:225-247, 1980.

[E83]

L.C. EVANS. Nonlinear systems in optimal control theory and related topics. In J.M. Ball, editor, Systems of nonlinear PDEs, pages 95-113. D. Reidel, Dordrecht, 1983.

[E89]

L.C. EVANS. The perturbed test function technique for viscosity solutions of partial differential equations. Proc. Roy. Soc. Edinburgh, 111 A:359-375, 1989.

[E90]

L.C. EVANS. Weak convergence methods for nonlinear partial differential equations, volume 74 of CBMS Regional Conf. Series in Math. Amer. Math. Soc., 1990.

[E92]

L.C. EVANS. Periodic homogenization of certain fully nonlinear partial differential equations. Proc. Roy. Soc. Edinburgh, 120 A:245-265, 1992.

lEG92]

L.C. EVANS AND R.F. GARIEPY. Measure theory and fine properties of functions. CRC Press, Boca Raton, FL, 1992.

[EI84]

L.C. EVANS AND H. IsHII. Differential games and nonlinear first order PDE in bounded domains. Manuscripta Math., 49:109-139, 1984.

[EI851

L.C. EVANS AND H. ISHII. A PDE approach to some asymptotic problems concerning random differential equations with small noise intensities. Ann. Inst. H. Poincard Anal. Non Lindaire, 2:1-20, 1985.

[EJ89]

L.C. EVANS AND M.R. JAMES. The Hamilton-Jacobi-Bellman equation for time optimal control. SIAM J. Control Optim., 27:1477-1489, 1989.

[ESS92]

L.C. EVANS, H.M. SONER, AND P.E. SOUGANIDIS. Phase transitions and generalized motion by mean curvature. Comm. Pure Appl. Math., 45:10971123, 1992.

[ES84]

L.C. EVANS AND P.E. SOUGANIDIS. Differential games and representation formulas for solutions of Hamilton-Jacobi equations. Indiana Univ. Math. J., 33:773-797, 1984.

[ES89]

L.C. EVANS AND P.E. SOUGANIDIS.A PDE approach to geometric optics for certain semilinear parabolic equations. Indiana Univ. Math. J., 38:141-172, 1989.

[ESp91]

L.C. EVANS AND J. SPRUCK. Motion of level sets by mean curvature. I. J. Differential Geom., 33:635-681, 1991.

[ESp951

L.C. EVANS AND J. SPRUCK. Motion of level sets by mean curvature. IV. J. Geom. Anal., 5:77-114, 1995.

[Fa8r]

M. FALCONE. A numerical approach to the infinite horizon problem of deterministic control theory. Appl. Math. Optim., 15:1-13, 1987. Corrigenda in Appl. Math. Optim. 23: 213-214, 1991.

548

BIBLIOGRAPHY

[Fa94]

h'I. FALCONE. The minimum time problem and its applications to front propagation. In A.Visintin and G.Buttazzo, editors, Motion by mean curvature and related topics, pages 70-88. De Gruyter verlag, Berlin, 1994.

[FaF94]

,~[. FALCONE AND R. FERRETTI. Discrete time high-order schemes for viscosity solutions of Hamilton-Jacobi-Bellman equations. Numer. Math., 67:315344, 1994.

[FaF90]

hi. FALCONE AND R. FERRETTI. High order approximations for viscosity solutions of Hamilton-Jacobi-Bellman equations. In A. Marino and M.K.V. Murthy, editors, Nonlinear variational problems and partial differential equations (Isola d'Elba, 1990), volume 320 of Pitman Res. Notes Math. Ser., pages 197-209. Longman Sci. Tech., Harlow, 1995.

[FaF96]

M. FALCONEAND R. FERRETTI. A class of fldly discrete high-order schemes for advection equations. SIAM J. Numerical Analysis, to appear.

[FC941

~[. FALCONE AND T. GIORGI. An approximation scheme for evolutive Hamilton-Jacobi equations. Technical report, Universith di Roma "La Sapienza", 1994.

[FCL941

hi. FALCONE, T. CIORGI, AND P. LORETI. Level sets of viscosity solutions: some applications to fronts and rendez-vous problems. SIAM Y. Appl. Math., 54:1335-1354, 1994.

[FLS94]

hi. FALCONE, P. LANUCARA, AND A. SEGHINI. A splitting algorithm for Hamilton-Jacobi-Bellman equations. Appl. Numer. Math., 15:207-218, 1994.

[FH94]

C. FERREYRAAND O. HI JAB. A simple free boundary problem in ~ . J. Control Optim., 32:501-515, 1994.

[F61]

W.H. FLEMING. The convergence problem for differential games. J. Math. Anal. Appl., 3:102-116, 1961.

[F641

W.H. FLEMING. The convergence problem for differential games. II. In M. et al. Dresher, editor, Advances in Game Theory, volume 52 of Annals of Mathematical Studies, pages 195-210. Princeton University Press, Princeton, N J, 1964.

[F69]

W.H. FLEMING. The Cauchy problem for a nonlinear first order partial differential equation. J. Differential Equations, 5:515-530, 1969.

[FM92a]

W.H. FLEMING AND W.~i. MCENEANEY. Risk sensitive optimal control and differential games. In Stochastic theory and adaptive control, volume 184 of Lecture Notes on Control and Inform. Sci., pages 185-197. Springer, 1992.

[FM92b]

W.H. FLEMING AND W.~i. MCENEANEY. Risk sensitive control with ergodic cost criteria. In Proc. 31st IEEE Conference on Decision and Control, pages 2048-2052. 1992.

[FM95]

W.H. FLEMINGAND W.M. McENEANEY. Risk-sensitive control on an infinite time horizon. SIAM .1. Control Optim., 33:1881-1915, 1995.

[FR75]

W.H. FLEMING AND R.W. RISHEL. Deterministic and stochastic optimal control. Springer, New York, 1975.

[FS93]

W.H. FLEMING AND H.M. SONER. Controlled Markov processes and viscosity solutions. Springer, New York, 1993.

SIAM

BIBLIOGRAPHY

549

[FSo86]

W.H. FLEMING AND P.E. SOUGANIDIS. Asymptotic series and the method of vanishing viscosity. Indiana Univ. Math. J., 35:425-448 and 925F, 1986.

[FSo89]

W . H . FLEMING AND P.E. SOUGANIDIS. On the existence of value functions

of two-players; zero-sum stochastic differential games. Indiana Univ. Math. g., 38:293-314, 1989.

[Fra87]

B.A. FRANCIS. A course in Hzr theory, volume 89 of Lecture Notes Control Inform. Sci. Springer, 1987.

[Fr87]

H. FRANKOWSKA. Equations d~Hamilton-Jacobi contingentes. C. R. Aead. Sci. Paris Sgr. I Math., 304:295-298, 1987.

[~89a]

H. FRANKOWSKA. Hamilton-Jacobi equations: viscosity solutions and generalized gradients. J. Math. Anal. Appl., 141:21-26, 1989.

[FrS9b]

H. FRANKOWSKA.Optimal trajectories associated with a solution of the contingent Hamilton-Jacobi equation. Appl. Math. Optim., 19:291-311, 1989.

[Fr93]

H. FRANKOWSKA. Lower semicontinuous solutions of Hamilton-JacobiBellman equations. SIAM J. Control Optim., 31:257-272, 1993.

[FPR94]

H. FRANKOWSKA, S. PLASKACZ, AND T. RZEZUCHOWSKI. Measurable viability theorems and Hamilton-Jacobi-Bellmanequation. Y. Differential Equations, 116:265-305, 1995.

[Fri71] [Fri74]

A. FRIEDMAN. Differential games. Wiley, New York, 1971. A. FRIEDMAN. Differential games, volume 18 of CBMS Regional Conf. Series in Math. American Math. Soc., Providence, R.I., 1974.

[FrSo86]

i.

[Gh91]

K.H. GHASSEMI. Differential games of fixed duration with state constraints. Y. Optim. Theory Appl., 68:513-537, 1991.

[GT83]

D. GILBARGAND N.S. TRUDINGER. Elliptic partial differential equations of second order. Springer, Berlin, 2nd edition, 1983.

[Ci85]

F. GIMBERT. Problemes de Neumann quasilineaires. Y. Funct. Anal., 62:6572, 1985.

[GD88]

K. GLOVER AND J. DOYLE. State-space formulas for all stabilizing controllers that satisfy an H~r norm bound and relations to risk-sensitivity. Systems Control Lett., 11:167-172, 1988.

[Go76]

R. GONZALEZ. Sur l'existence d'une solution maximale de l'6quation de Hamilton-Jacobi. C. R. Acad. Sci. Paris Sdr. I Math., 282:1287-1290, 1976.

[CR85]

R. GONZALEZ AND n . ROFMAN. On deterministic control problems:an approximation procedure for the optimal cost, I and II. SIAM J. Control Optim.,

FRIEDMAN AND P.E. SOUGANIDIS. Blow-up of solutions of HamiltonJacobi equations. Comm. Partial Differential Equations, 11:397-443, 1986.

23:242-285, 1985.

ITS901

R. GONZALEZ AND C. SAGASTIZABAL. Un algorithme pour la r6solution

rapide d'6quations discretes de Hamilton-Jacobi-Bellman. C. R. Acad. Sci. Paris Sdr. I Math., 311:45-50, 1990. [GTi90]

R. GONZALEZ AND M.M. TIDBALL. Fast solutions of discrete Isaacs' inequalities. Technical Report 1167, INRIA, 1990.

55O

BIBLIOGRAPHY

[GWi92]

R.L.V. GONZALEZ AND ~I.~,:[. TIDBALL. Sur F'ordre de convergence des solutions discr~tis~es en temps et en espace de l'6quation de Hamilton-Jacobi. C. R. Acad. Sci. Paris Sdr. I Math., 314:479-482, 1992.

[Gru97]

L. GRUNE. An adaptive grid scheme for the discrete Hamilton-Jacobi-Bellman equation. Numer. Math., 75:319-337, 1997.

[Gy84]

E. C-YURKOVICS. H51der continuity for the minimum time function of linear systems. In Thoft-Chistensen, editor, System modelling and optimization, Proc. 11th IFIP Conference, pages 383-392. Springer, 1984.

[Ha75]

O. H~JEK. Pursuit games. Academic Press, New York, 1975.

[Ha91]

O. H)~JEK. Viscosity solutions of Bellman's equation. In S. Elaydi, editor, Differential equations, pages 191-200. Marcel Dekker, New York, 1991.

[HH70]

G.W. HAYNES AND H. HERMES. Nonlinear controllability via Lie theory. SIAM J. Control Optim., 8:450-460, 1970.

[He88]

H. HERMES. Feedback synthesis and positive, local solutions to HamiltonJacobi-Bellman equations. In C.I. Byrnes, C.F. Martin, and R.E. Saeks, editors, Analysis and Control of Nonlinear Systems, pages 155-164. North Holland, Amsterdam, 1988.

[He91]

H. HERMES. Nilpotent and high-order approximations of vector field systems.

SIAM Review, 33:238-264, 1991. [HL69]

H. HERMES AND J.P. LASALLE. Functional Analysis and time optimal control. Academic Press, New York, 1969.

[Hes66]

M.R. HESTENES. Calculus of Variations and Optimal Control Theory. Wiley, New York, 1966.

[Hi91]

O. HIJAB. Control of degenerate diffusions in ~d. Trans. Amer. Math. Soc., 327:427-448, 1991.

[H65]

E. HOPF. Generalized solutions of non-linear equations of first order. 9". Math. Mech., 14:951-973, 1965.

[Hop86]

R. HOPPE. Multigrid methods for Hamilton-Jacobi-Bellman equations. Numer. Math., 49:235-254, 1986.

[HI97]

K. HORIE AND n . ISHII. Homogenization of Hamilton-Jacobi equations on domains with small scale periodic structure. Preprint 1, Tokyo Metropolitan University, 1997.

[Hor68]

L. HORMANDER. Hypoeliiptic second order differential equations. Acta Math.,

119:147-171, 1968. [Ho601

R.A. HOWARD. Dynamic Programming and Markov processes. Wiley, New York, 1960.

[Hr78]

M.M. HRUSTALEV. Necessary and sufficient conditions in the form of Bellman's equation. Soviet Math. Dokl., 19:1262-1266, 1978.

[Is65]

R. ISAACS. Differential games. Wiley, New York, 1965.

[I83]

H. ISHII. Remarks on existence of viscosity solutions of Hamilton-Jacobi equations. Bull. Facul. Sci. Eng. Chuo Univ., 26:5-24, 1983.

BIBLIOGRAPHY

551

[I84]

H. ISHII. Uniqueness of unbounded viscosity solutions of Hamilton-Jacobi equations. Indiana Univ. Math. J., 33:721-748, 1984.

[I85a]

H. ISHII. Hamilton-Jacobi equations with discontinuous Hamiltonians on arbitrary open sets. Bull. Facul. Sci. Eng. Chuo Univ., 28:33-77, 1985.

[I85b]

H. ISHII. On representation of solutions of Hamilton-Jacobi equations with convex Hamiltonians. In K. Masuda and hi. Mimura, editors, Recent topics in nonlinear PDE II. volume 8 of Lecture Notes in Numerical and Applied Analysis, pages 15-52. Kinokuniya-North Holland, Tokyo, 1985.

[I86]

H. ISHII. Existence and uniqueness of solutions of Hamilton-Jacobi equations. Funkcial Ekvac., 29:167-188, 1986.

[I87a]

H. ISHII. Perron's method for Hamilton-Jacobi equations. Duke Math. J., 55:369-384, 1987.

[I87b]

H. ISHn. A simple, direct proof of uniqueness for solutions of Hamilton-Jacobi equations of eikonal type. Proc. Amer. Math. Soc., 100:247-251, 1987.

[ISSa]

H. ISHII. Representation of solutions of Hamilton-Jacobi equations. Nonlinear Anal., 12:121-146, 1988.

[I88b]

H. ISHII. Lecture notes on viscosity solutions. Brown University, Providence.

RI, 1988.

[IS9]

H. IsslI. A boundary value problem of the Dirichlet type for Hamilton-Jacobi equations. Ann. Sc. Norm. Sup. Pisa, (IV) 16:105-135, 1989.

[I91]

H. ISHII. Fully nonlinear oblique derivative problems for nonlinear secondorder elliptic PDEs. Duke Math. J., 62:633-661, 1991.

[I92a]

H. ISHII. Viscosity solutions for a class of Hamilton-Jacobi equations in Hilbert spaces. J. Funct. Anal., 105:301-341, 1992.

[I92b]

H. ISHII. Perron's method for monotone systems of second-order elliptic PDEs. Differential Integral Equations, 5:1-24, 1992.

[I97]

H. ISHII. A comparison result for Hamilton-Jacobi equations without growth condition on solutions from above. Preprint 2, Tokyo Metropolitan University, 1997.

[IK91a]

H. ISHII AND S. KOIKE. Remarks on elliptic singular perturbation problems. Appl. Math. Optim., 23:1-15, 1991.

[IK91b]

H. ISHII AND S. KOIKE. Viscosity solutions of a system of nonlinear secondorder elliptic PDEs arising in switching games. Funkcial. Ekvac., 34:143-155, 1991.

[IK96]

H. ISHII AND S. KOIKE. A new formulation of state constraints problems for first order PDE's. SIAM J. Control Optim., 36:554-571, 1996.

[IMZ91]

H. ISHII, J.L. ~[ENALDI~ AND L. ZAREMBA. Viscosity solutions of the Bellman equation on an attainable set. Problems Control Inform. Theory, 20:317328, 1991.

[IR95]

H. ISHII AND hi. RAMASWAMY. Uniqueness results for a class of HamiltonJacobi equations with singular coefficients. Comm. Partial Differential Equations, 20:2187-2213, 1995.

552

BIBLIOGRAPHY

[IA92a]

A. ISlDORI AND A. ASTOLFI. Nonlinear Hor control via measurement feedback. J. Math. Systems Estimation Control, 2:31-44, 1992.

[IA92b]

A. ISIDORI AND A. ASTOLFI. Disturbance attenuation and H~r control via measurement feedback in nonlinear systems. IEEE Trans. Automat. Control, 37:1283-1293, 1992.

[IzK95]

S. ]ZUMIYAAND C. T. KOSSIORIS. Semi-local classification of geometric singularities for Hamilton-Jacobi equations. J. Differential Equations, 118:166193, 1995.

[Jac73]

D.H. JACOBSON. Optimal stochastic linear systems with exponential criteria and their relation to deterministic differential games. IEEE Trans. Automat. Control, 18:121-134, 1973.

[Jam92]

M.R. JAMES. Asymptotic analysis of nonlinear stochastic risk-sensitive control and differential games. Math. Control Signals Systems, 5:401-417, 1992.

[Jam93]

M.R. JAMES. A partial differential inequality for dissipative nonlinear systems. Systems Control Lett., 21:315-320, 1993.

[JB96]

M.R. JAMES AND J.S. BARAS. Partially observed differential games, infinitedimensional Hamilton-Jacobi-Isaacs equations, and nonlinear H ~ control. SIAM J. Control Optim., 34:1342-1364, 1996.

[JBE94]

M.R. JAMES, J.S. BARAS, AND R.J. ELLIOTT. Risk-sensitive control and

dynamic games for partially observed discrete-time nonlinear systems. IEEE Trans. Automat. Control, 39:780-792, 1994.

[JY95]

.~{.Pt. JAMES AND S. YULIAR. Numerical approximation of the H ~ norm for nonlinear systems. Automatica J. IFAC, 31:1075-1086, 1995.

[Ja77]

R. JANIN. On sensitivity in an optimal control problem. J. Math. Anal. Appl., 60:631-657, 1977.

[J88]

R. JENSEN. The maximum principle for viscosity solutions of fully nonlinear 2nd order partial differential equations. Arch. Rat. Mech. Anal., 101:1-27, 1988.

[J89]

R. JENSEN. Uniqueness criteria for viscosity solutions of fully nonlinear elliptic partial differential equations. Indiana Univ. Math. J., 38:629-667, 1989.

[Kal63]

R.E. KALMAN. The theory of optimal control and the calculus of variations. In R. Bellman, editor, Mathematical optimization techniques, pages 309-331. University of California Press, Berkeley, CA, 1963.

[Kam84]

S. KAMIN. Exponential descent of solutions of elliptic singular perturbation problems. Comm. Partial Differential Equations, 9:197-213, 1984.

[Kam86]

S. KAMIN. Singular perturbation problems and the Hamilton-Jacobi equation. Integral Equations Operator Theory, 9:95-105, 1986.

[Kam88]

S. KAMIN. Elliptic perturbation for linear and nonlinear equations with a singular point. J. Analyse Math., 50:241-257, 1988.

[Ka95]

M.A. KATSOULAKIS. A representation formula and regularizing properties for viscosity solutions of 2nd order fully nonlinear degenerate parabolic equations. Nonlinear Anal., 24:147-158, 1995.

BIBLIOGRAPHY

553

[Kaw90]

~,I. KAWSKI. High-order small-time controllability. In H.J. Sussmann, editor, Nonlinear controllability and optimal control, chapter 14, pages 431-467. Marcel Dekker, New York, 1990.

[Kle93]

A.F. KLEIMENOV. Nonantagonistic positional differential games. Nauka, Ekaterinburg, 1993. (in Russian).

[KSor96]

l~[. KOCAN AND P. SORAVIA. Differential games and nonlinear H~r control in infinite dimensions. SIAM J. Control Optim, to appear.

[KSS971

M. KOCAN, P. SORAVIA, AND A. SWIECH. On differential games for infinite dimensional systems with nonlinear, unbounded operators. J. Math. Anal. Appl., 211:395-423, 1997.

[KSw95]

M. KOCAN AND A. SWIECH. Second order unbounded parabolic equations in separated form. Studia Math., 115:291-310, 1995.

[K911

S. KOIKE. On the rate of convergence of solutions in singular perturbation problems. J. Math. Anal. Appl., 157:243-253, 1991.

[K95]

S. KOIKE. On the state constraint problem for differential games. Indiana Univ. Math. J., 44:467-487, 1995.

[K96a]

S. KOIKE. On the Bellman equations with varying controls. Bull. Australian Math. Soc., 53:51-62, 1996.

[K96b]

S. KOIKE. Semicontinuous viscosity solutions for Hamilton-Jacobi equations with a degenerate coefficient. Differential Integral Equations, to appear.

[KN971

S. KOIKE AND I. NAKAYAMA. Uniqueness of lower semicontinuous viscosity solutions for the minimum time problem. Preprint, Saitama University, 1997.

[Kok84]

P.V. KOKOTOVIC. Applications of singular perturbation techniques to control problems. SIAM Review, 277:1-42, 1984.

[Ko93a]

G.T. KOSSIORIS. Propagation of singularities for viscosity solutions of Hamilton-Jacobi equations in one space variable. Comm. Partial Differential Equations, 18:747-770, 1993.

[Ko93b]

G.T. KossIoRIs. Propagation of singularities for viscosity solutions of Hamilton-Jacobi equations in higher dimensions. Comm. Partial Differential Equations, 18:1085-1108, 1993.

[KK95]

A.N. KRASOVSKIIAND N.N. KRASOVSKn. Control under lack of information. Birkh~iuser, Boston, 1995.

[KS74]

N.N. KRASOVSKn AND A.I. SUBBOTIN. Positional differential games. Nauka, Moscow, 1974. (In Russian.) French translation: Jeux differentiels, Mir, Moscou, 1979. Revised English edition: Game-theoretical control problems, Springer, New York, 1988.

[Kr60]

S.N. KRU2KOV. The Cauchy problem in the large for certain nonlinear first order differential equations. Sov. Math. Dokl., 1:474-477, 1960.

[Kr64]

S.N. KRU2KOV. The Cauchy problem in the large for nonlinear equations and for certain quasilinear systems of the first order with several variables. Soviet Math. Dokl., 5:493-496, 1964.

[Kr66a]

S.N. KRU2KOV. On solutions of first-order nonlinear equations. Soviet Math. Dokl., 7:376-379, 1966.

554

BIBLIOGRAPHY

[Kr66b]

S.N. KRU2KOV. The method of finite differences for a first-order non-linear equation with many independent variables. USSR Comput. Math. Math. Phys., 6(5):136-151, 1966.

[Kr67]

S.N. KRU2KOV. Generalized solutions of nonlinear first order equations with several independent variables II. Math. USSR Sbornik, 1:93-116, 1967.

[Kr70]

S.N. KRU2KOV. First order quasilinear equations in several independent variables. Math. USSR Sbornik, 10:217-243, 1970.

[Kr75]

S.N. KRU2KOV. Generalized solutions of the Hamilton-Jacobi equations of eikonal type I. Math. USSR Sbornik, 27:406-445, 1975.

[Kry80]

N.V. KRYLOV. Controlled diffusion processes. Springer, New York, 1980.

[Kry87]

N.V. KRYLOV. Nonlinear elliptic and parabolic equations of the second order. Reidel, Dordrecht, 1987.

[KP72]

L.A. KUN AND Yu.F. PRONOZlN. Sufficient conditions for smoothness of the Bellman function in linear time-optimal problems. Engrg. Cybernetics, 10:6--11, 1972.

[Ku86]

J. KURZWEIL. Ordinary Differential Equations. Elsevier, Amsterdam, 1986.

[Kus77]

H.J. KUSltNER. Probability methods for approximations in stochastic control and for elliptic equations. Academic Press, New York, 1977.

[KD92]

H.J. KUSHNER AND P.G. DuPuIS. Numerical methods for stochastic control problems in continuous time. Springer, New York, 1992.

[La67]

R.E. LARSON. State increment Dynamic Programming. American Elsevier, New York, 1967.

[LL86]

J.M. LASRY AND P.L. LIONS. A remark on regularization in Hilbert spaces. Israel J. Math., 55:257-266, 1986.

[LhI67]

E.B. LEE AND L. MARKUS. Foundations of optimal control theory. Wiley, New York, 1967.

[Lei66]

G. LEITMANN. An introduction to optimal control. McGraw-Hill, New York, 1966.

[LeS7]

S.~I. LENHART. Viscosity solutions associated with switching control problems for piecewise-deterministic processes. Houston J. Math., 13:405-426, 1987.

[Le89]

S.M. LENHART. Viscosity solutions associated with impulse control problems for piecewise-deterministic processes. Internat. J. Math. Math. Sci., 12:145157, 1989.

[LeY91]

S.M. LENHART AND N. YAMADA. Perron's method for viscosity solutions associated with piecewise-deterministic processes. Funkcial. Ekvac., 34:173186, 1991.

[LeY92]

S.hl. LENHART AND N. YAMADA. Viscosity solutions associated with switching game for piecewise-deterministic processes. Stochastics Stochastics Rep., 38:27-47, 1992.

[Lew94]

J. LEWlN. Differential games. Springer, London, 1994.

[LiY95]

X. LI AND J. YONG. Optimal control theory for infinite dimensional systems. Birkh/iuser, Boston, 1995.

BIBLIOGRAPHY

555

[L82]

P.L. LIONS. Generalized solutions of Hamilton-Jacobi equations. Boston, 1982.

[L83a]

P.L. LIONS. Optimal control of diffusion processes and Hamilton-JacobiBellman equations. Part 1: The dynamic programming principle and applications; Part 2: Viscosity solutions and uniqueness. Comm. Partial Differential Equations, 8:1101-1174 and 1229-1276, 1983.

[L83b]

P.L. LIONS. On the Hamilton-Jacobi-Bellman equations. Acta Appl. Math., 1:17-41, 1983.

[L83c]

P.L. LIONS. Optimal control of diffusion processes and Hamilton-JacobiBellman equations, Part III: Regularity of the optimal cost function. In Nonlinear PDE's and Applications, volume V of College de France Seminar, pages 95-205. Pitman, Boston, 1983.

[L83d]

P.L. LIONS. Existence results for first-order Hamilton-Jacobi equations. Ricerehe Mat., 32:1-23, 1983.

[L85a]

P.L. LIONS. Optimal control and viscosity solutions. In I. Capuzzo Dolcetta, W.H. Fleming, and T. Zolezzi, editors, Recent Mathematical Methods in Dynamic Programming, volume 1119 of Lecture Notes in Mathematics, pages 94-112. Springer, 1985.

[LS5b]

P.L. LIONS. Neumann type boundary condition for Hamilton-Jacobi equations. Duke Math. J., 52:793-820, 1985.

[L85c]

P.L. LIONS. Equation de Hamilton-Jacobi et solutions de viscositY. In Ennio De Giorgi colloquium (Paris 1983), volume 125 of Res. Notes in Math., pages 83-97. Pitman, Boston, 1985.

[L85d]

P.L. LIONS. Regularizing effects for first-order Hamilton-Jacobi equations. Applicable Anal., 20:283-307, 1985.

[L86]

P.L. LIONS. Optimal control of reflected diffusion processes: an example of state constraints. In Stochastic differential systems (Bad Honnef, 1985), volume 78 of Lecture Notes in Control and Inform. Sci., pages 269-276. Springer, Berlin, 1986.

[L88]

P.L. LIONS. Viscosity solutions of fully nonlinear second-order equations and optimal stochastic control in infinite dimensions. Part I: the case of bounded stochastic evolutions. Acta Math., 161:243-278, 1988.

[L89]

P.L. LIONS. Viscosity solutions of fully nonlinear second-order equations and optimal stochastic control in infinite dimensions. Part III: Uniqueness of viscosity solutions for general second-order equations. J. Funct. Anal., 86:1-18, 1989.

[L92]

P.L. LIONS. Viscosity solutions and optimal control. In Proceedings of ICIAM 91, pages 182-195. SIAM, Philadelphia, 1992.

[LMr80]

P.L. LIONS AND B. h[ERCIER. Approximation numerique des equations de Hamilton-Jacobi-Bellman. RAIRO Anal. Numer., 14:369-393, 1980.

[LPV86]

P.L. LIONS, G. PAPANICOLAOU, AND S. VARADHAN. Homogenization of Hamilton-Jacobi equations, unpublished paper.

[LP86]

P.L. LIONS AND S . PERTHAME. Quasi-variational inequalities and ergodic impulse control. SIAM J. Control Optim., 24:604-615, 1986.

Pitman,

556

BIBLIOGRAPHY

[LP87]

P.L. LIONS AND B. PERTHAME. Remarks on Hamilton-Jacobi equations with measurable time-dependent Hamiltonians. Nonlinear Anal., 11:613-621, 1987.

[LR86]

P.L. LIONS AND J.-C. ROCHET. Hopf formula and multi-time HamiltonJacobi equations. Proc. Amer. Math. Soc., 96:79-84, 1986.

[LRT93]

P.L. LIONS, E. RouY, AND A. TOURIN. Shape-from-shading, viscosity solutions and edges. Numer. Math., 64:323-353, 1993.

[LS85]

P.L. LIONS AND P.E. SOUGANIDIS. Differential games and directional derivatives of viscosity solutions of Bellman's and Isaacs' equations. SIAM J. Control Optim., 23:566-583, 1985.

[LS86]

P.L. LIONS AND P.E. SOUGANIDIS. Differential games and directional derivatives of viscosity solutions of and Isaacs' equations II. SIAM J. Control Optim., 24:1086-1089, 1986.

[LS93]

P.L. LIONS AND P.E. SOUGANIDIS. Fully nonlinear 2nd order degenerate elliptic equations with large zeroth-order coefficients. Indiana Univ. Math. J., 42:1525-1543, 1993.

[LS951

P.L. LIONS AND P.E. SOUGANIDIS. Convergence of MUSCL and filtered schemes for scalar conservation laws and Hamilton-Jacobi equations. Numer. Math., 69:441-470, 1995.

[LSV87]

P.L. LIONS, P.E. SOUCANIDIS, AND J.L. VAZQUEZ. The relation between the porous medium and the eikonal equation in several space dimensions. Rev. Mat. Iberoamericana, 3:275-310, 1987.

[Liv80]

A.A. LIVEROVSKII. Some properties of Bellman's functions for linear and symmetric polysystems. Differential Equations, 16:255-261, 1980.

[Lo86]

P. LORETI. Approssimazione di soluzioni viscosit~ dell'equazione di Bellman. Boll. Un. Mat. Ital., (6) 5-B:141-163, 1986.

[Lo87]

P. LORETI. Some properties of viscosity solutions of Hamilton-Jacobi-Betlman equations. SIAM J. Control Optim., 25(5):1244-1252, 1987.

[LOS93]

P. LORETI AND A. SICONOLFI. A semigroup approach for the approximation

of a control problem with unbounded dynamics. J. Optim. Theory Appl., 79:599-610, 1993. [LOT94]

P. LORETI AND M.E. TESSITORE. Approximation and regularity results on constrained viscosity solution of Hamilton-Jacobi-Bellmanequations. J. Math. Systems Estimation Control, 4:467-483, 1994.

[~Is82]

J. MACKI AND A. STRAUSS. Introduction to optimal control theory. Springer, New York, 1982.

[MSou94]

A.J. ~IAJDA AND P.E. SOUGANIDIS.Large-scale front dynamics for turbulent reaction-diffusion equations with separated velocity scales. Nonlinearity, 7:130, 1994.

[M79]

K. ~IALANOWSKI. On convergence of finite difference approximation to optimal control problems for systems with control appearing linearly. Archivum Automatyki Telemechaniki, pages 155-170, 1979.

[ME95]

W.M. McENEANEY. Uniqueness for viscosity solutions of nonstationary HJB equations under some a priori conditions (with applications). SIAM J. Control Optim., 33:1560-1576, 1995.

BIBLIOGRAPHY

557

[Me1941

A.A. lk.IELIKYAN.Singular paths in differential games with simple motion. In T. Basar and A. Haurie, editors, Advances in Dynamic Games and Applications, pages 125-135. Birkhiiuser, Boston, 1994.

[Me80a]

J.L. ~,IENALDI. On the optimal stopping time problem for degenerate diffusions. SIAM J. Control Optim., 18:697-721, 1980.

[Me80b]

J.L. ~IENALDI. On the optimal impulse control problem for degenerate diffusions. SIAM J. Control Optim., 18:722-739, 1980.

[Me82]

J.L. MENALDI. Le probleme de temps d'arret optimal deterministe et l'inequation variationelle du premier ordre associee. Appl. Math. Optim., 8:131-158, 1982.

[Me89]

J.L. ~IENALDI. Some estimates for finite difference approximations. SIAM J. Control Optim., 27:579-607, 1989.

[MeR86]

J.L. [kIENALDI AND E. ROFMAN. An algorithm to compute the viscosity solution of the Hamilton-Jacobi-Bellmanequation. In C.I. Byrnes and A. Lindquist, editors, Theory and Applications of Nonlinear Control Systems, pages 461-468. North-Holland, 1986.

IMPS6}

F. MIGNANEGO AND G. PIERI. On the sufficiency of the Hamilton-JacobiBellman equation for optimality of the controls in a linear optimal-time problem. Systems Control Lett., 6:357-363, 1986.

[Mi88]

S. MIRICA. In6galit6s diff6rentielles impliquant l'6quation de Bellman-Isaacs et ses g6n6ralisations dans la th6orie du controle optimal. Anal. Univ. Bucuresti Mat., 37:25-35, 1988.

[Mi90a]

S. MIRICA. Control optimal. Editura Stiintifica, Bucharest, 1990. In rumanian, english translation to appear.

[Mi90b]

S. ~IIRICA. Some generalizations of the Bellman-Isaacs equation in deterministic optimal control. Studii Cercetari Mat., 42:437-447, 1990.

[Mi92a]

S. ~IIRICA. A proof of Pontryagin's minimum principle using dynamic programming. J. Math. Anal. Appl., 170:501-512, 1992.

[Mi92b]

S. MIRICA. Differential inequalities and viscosity solutions in optimal control. Anal. Univ. Ia~i, 38:89-102, 1992.

[Mi93a]

S. MIRICA. Nonsmooth fields of extremals and constructive Dynamic Programming in optimal control. Preprint 18, Dipartimento di Matematica, Universit~ di Padova, 1993.

[Mi9Zb]

S. MIRICA. Optimal feedback control in closed form via Dynamic Programming. Preprint, 1993.

[Mo66]

J.J. MOREAU. Fonctionelles convexes. Technical report, Seminaire LerayCollege de France, 1966.

[Mos76]

U. Mosco. Implicit variational problems and quasi variational inequalities. In Nonlinear operators and the calculus of variations, volume 543 of Lect. Notes Math., pages 83-156. Springer, 1976.

[Mo95]

M. MOTTA. On nonlinear optimal control problems with state constraints. SIAM J. Control Optim., 33:1411-1424, 1995.

558

BIBLIOGRAPHY

[hIR96a]

hi. I~.IOTTAAND F. RAMPAZZO.Dynamic Programming for nonlinear systems driven by ordinary and impulsive controls. SIAM J. Control Optim., 34:199225, 1996.

[MR96b]

M. hIOTTA AND F. RA~IPAZZO. The value function of a slow growth control problem with state constraints. J. Math. Systems Estim. Control, to appear.

[hlR96c]

M. ~IOTTA AND F. RAMPAZZO. State-constrained control problems with neither coercivity nor L 1 bounds on the controls. Appl. Math. Optim., to appear.

[O1951

G.J. OLSDER, editor. New Trends in Dynamic games and Applications, volume 3 of Ann. Internat. Soc. Dynam. Games, Boston, 1995. Birkhs

[OSe88]

S. OSHER AND J.A. SETHIAN. Fronts propagating with curvature dependent speed: Algorithms on Hamilton-Jacobi formulation. J. Comp. Phys., 79:12-49, 1988.

[OS91]

S. OSHER AND C.W. SHC. High-order essentially nonoscillatory schemes for Hamilton-Jacobi equations. SIAM ,l. Numer. Anal., 28:907-922, 1991.

[PBKTZ94] V.S. PATSKO, N.D. BOTKIN, V.M. KEIN, V.L. TUROVA, AND hI.A. ZARKtt. Control of an aircraft landing in windshear. J. Optim. Theory Appl., 83:237-267, 1994. [PS88]

B. PERTHAMEAND R. SANDERS. The Neumann problem for nonlinear second order singular perturbation problems. SIAM J. Math. Anal., 19:295-311, 1988.

[Pes94a]

H.J. PESCH. A practical guide to the solution of real-life optimal control problems. Control Cybernetics, 23:7-60, 1994.

[Pes94b]

H.J. PESCH. Solving optimal control and pursuit-evasion game problems of high complexity. In R. Bulirsch and D. Kraft, editors, Computational Optimal Control, pages 43-64. Birkh~iuser, Basel, 1994.

[PB94]

H.J. PESCH AND R. BULIRSCH.The Maximum Principle, Bellman's equation, and Carath~odory's work. J. Optim. Theory Appl., 80:199-225, 1994.

[Pe68]

N.N. PETROV. Controllability of autonomous systems. Differential Equations, 4:311-317, 1968.

[Pe70a]

N.N. PETROV. On the Bellman function for the time-optimal process problem. J. Appl. Math. Mech., 34:785-791, 1970.

[Pe70b]

N.N. PETROV. The continuity of Bellman's generalized function. Differential Equations, 6:290-292, 1970.

[PET9]

N.N. PETROV. On the continuity of the Bellman function with respect to a parameter. Vestnik Leningrad Univ. Math., 7:169-176, 197"9.

[PSus96]

B. PICCOLIAND H.J. SUSSMANN.Regular synthesis and sufficiency conditions for optimality. Preprint 68, SISSA-ISAS, Trieste, 1996.

[P66]

L.S. PONTRYAGIN. On the theory of differential games. Russian Math. Surveys, 21:193-246, 1966.

[P90]

L.S. PONTRYAGIN,editor. Optimal control and differential games, volume 185 of Proc. Steklov Inst. Math. Amer. Math. Soc., 1990.

BIBLIOGRAPHY

559

[PBGM62] L.S. PONTRYAGIN,V.G. BOLTYANSKII,R.V. GAMKRELIDZE,AND E . F . MISHCHENKO. The mathematical theory of optimal processes. Interscience, New York, 1962. [Q80]

J . P . QUADRAT. Existence de solutions et algorithme de resolution numerique des problemes stochastiques degenerees ou non. SIAM J. Control Optim., 18:199-226, 1980.

[QSP95]

1~[. QUINCAMPOIX AND P. SAINT-PIERRE. An algorithm for viability kernels in HSlderian case: approximation by discrete viability kernels. J. Math. Systems Estimation Control, 5:115-118, 1995. (Summary).

[Ra95]

F. RAMPAZZO. Differential games with unbounded versus bounded controls. SIAM J. Control Optim., to appear.

[RS961

F. RAMPAZZOAND C. SARTORI.The minimum time function with unbounded controls. J. Math. Systems Estim. Control, to appear.

[a~2]

3,I. RANGUIN. Propriete H61derienne de la fonction temps minimal d'un systerne lineaire autonome. RAIRO Automatique Syst. Anal. Control, 16:329340, 1982.

[Ro83]

3,[. ROBIN. Long run average control of continuous time Markov processes: a survey. Acta Appl. Math., 1:281-299, 1983.

[RT0]

R . T . ROCKAFELLAR. Convex analysis. Princeton University Press, Princeton, N J, 1970.

[R81]

R.T. ROCKAFELLAR. The theory of subgradients and its applications to problems of optimization. Heldermann, Berlin, 1981.

[aou92]

E. RouY. Numerical approximation of viscosity solutions of first order Hamilton-Jacobi equations with Neumann type boundary conditions. Math. Models Methods Appl. Sci., 2:357-374, 1992.

[RT92]

E. RouY AND A. TOURIN. A viscosity solutions approach to shape-fromshading. SIAM J. Numer. Anal., 29:867-884, 1992.

[RV91]

J.D.L. ROWLANDAND R.B. VINTER. Construction of optimal feedback controls. Systems Control Lett., 16:357-367, 1991.

[Ro69]

E. ROXIN. Axiomatic approach in differential games. J. Optim. Theory Appl., 3:153-163, 1969.

[RS88]

I. ROZYEV AND A.I. SUBBOTIN. Semicontinuous solutions of Hamilton-Jacobi equations. J. Appl. Math. Mech., 52:141-146, 1988.

[Ru86]

W. RUDIN. Real and complex analysis. McGraw-Hill, New York, 3 edition, 1986.

[SP94]

P. SAINT-PIERRE. Approximation of the viability kernel. Appl. Math. Optim., 29:187-209, 1994.

[Sa90]

C. SARTORI. On the continuity with respect to a parameter of the value function in some optimal control problems. Rend. Sem. Mat. Univ. Pol. Torino; 48:383-399, 1990.

[Sch89]

C. SCHERER. H~c control by state feedback: an alternative algorithm and characterization of high-gain occurrence. Systems Control Lett., 12:383-391, 1989.

560

BIBLIOGRAPHY

[Si951

A. SICONOLFI. A first order Hamilton-Jacobi equation with singularity and the evolution of level sets. Comm. Partial Differential Equations, 20:277-308, 1995.

[Sin95]

C. SINESTRARI. Semiconcavity of solutions of stationary Hamilton-Jacobi equations. Nonlinear Anal., 24:1321-1326, 1995.

[s861

H.M. SONER. Optimal control problems with state-space constraints I and II. SIAM J. Control Optim., 24:551-561 and 1110-1122, 1986.

[s88]

H.h][. SONER. Optimal control of jump-Markov processes and viscosity solutions. In W. Fleming and P.L. Lions, editors, Stochastic differential systems, stochastic control theory and applications, volume 10 of IMA , pages 501-511. Springer, New York, 1988.

[$93]

H.M. SONER. Singular perturbations in manufacturing. SIAM J. Control Optim., 31:132-146, 1993.

[$95]

H.M. SONER. Controlled Markov processes, viscosity solutions and applications to mathematical finance. In I. Capuzzo Dolcetta and Lions P.L., editors, Viscosity solutions and applications (Montecatini 1995), volume 1660 of Lecture Notes in Mathematics, pages 134-185. Springer, Berlin, 1997.

[Son90]

E.D. SONTAG. Mathematical control theory: Deterministic finite dimensional systems. Springer, New York, 1990.

[Sor92a]

P. SORAVIA.H61der continuity of the minimum time function for C 1 manifold targets. Y. Optim. Theory Appl., 75:401-421, 1992.

[Sor92b]

P. SORAVIA. The concept of value in differential games of survival and viscosiW solutions of Hamilton-Jacobi equations. Differential Integral Equations, 5:1049-1068, 1992.

[Sor93a]

P. SORAVIA. Pursuit-evasion problems and viscosity solutions of Isaacs equations. SIAM J. Control Optim., 31:604-623, 1993.

[Sor93b]

P. SORAVIA. Discontinuous viscosity solutions to Dirichlet problems for Hamilton-Jacobi equations with convex Hamiltonians. Comm. Partial Differential Equations, 18:1493-1514, 1993.

[Sor94a]

P. SORAVIA. Generalized motion of a front along its normal direction: a differential games approach. Nonlinear Anal., 22:1247-1262, 1994.

[Sor94b]

P. SORAVIA. Differential games and viscosity solutions to study the H~r control of nonlinear, partially observed systems. In Breton M. and G. Zaccour, editors, Preprint volume of the 6th International Symposium on Dynamic Garues and Applications, Montreal, 1994.

[Sor95]

P. SORAVIA. Stability of dynamical systems with competitive controls: the degenerate case. J. Math. Anal. Appl., 191:428-449, 1995.

[Sor96a]

P. SORAVIA.H~ control of nonlinear systems: Differential games and viscosity solutions. SIAM J. Control Optim., 34:1071-1097, 1996.

[Sor96b]

P. SORAVIA. Optimality principles and representation formulas for viscosity solutions of Hamilton-Jacobi equations: I. Equations of unbounded and degenerate control problems without uniqueness; II. Equations of control problems with state constraints. Advances Differential Equations, to appear.

BIBLIOGRAPHY

561

[Sor97a]

P. SORAVIA. Equivalence between nonlinear H~r control problems and existence of viscosity solutions of Hamilton-Jacobi-Isaacs equations. Appl. Math. Optim, to appear.

[Sor97b]

P. SORAVIA. Estimates of convergence of fully discrete schemes for the Isaacs equation of pursuit-evasion differential games via maximum principle. SIAM J. Control Optim., to appear.

[SSou96]

P. SORAVIAAND P.E. SOUGANIDIS.Phase-field theory for FitzHugh-Nagumotype systems. SIAM J. Math. Anal., 27:1341-1359, 1996.

[SouS5a]

P.E. SOUGANIDIS.Approximation schemes for viscosity solutions of HamiltonJacobi equations. J. Differential Equations, 57:1-43, 1985.

[SouS5b]

P.E. SOUGANIDIS. Max-rain representations and product formulas for the viscosity solutions of Hamilton-Jacobi equations with applications to differential games. Nonlinear Anal., 9:217-257, 1985.

[Sou86]

P.E. SOUGANIDIS. A remark about viscosity solutions of Hamilton-Jacobi equations at the boundary. Proc. Amer. Math. Soc., 96:323-329, 1986.

[Sou95]

P.E. SOUGANIDIS. Front propagation: theory and applications. In I. Capuzzo Dolcetta and P.L. Lions, editors, Viscosity solutions and applications (Montecatini, 1995), Lecture Notes in Mathematics, pages 186-242. Springer, 1997.

[Sta89]

V. STAICU. l~Iinimal time function and viscosity solutions. J. Optim. Theory Appl., 60:81-91, 1989.

[st91]

G. STEFANI. Regularity properties of the minimum-time map. In C.I. Byrnes and A. Kurzhansky, editors, Nonlinear systems, pages 270-282. Birkhs Boston, 1991.

[Sto92]

A.A. STOORVOGEL. The H~r control problem. Prentice Hall, New York, 1992.

[Su8O]

A.I. SUBBOTIN. A generalization of the basic equation of the theory of differential games. Soviet Math. Dokl., 22:358-362, 1980.

[Su84)

A.I. SUBBOTIN. Generalization of the main equation of differential game theory. 3. Optim. Theory Appl., 43:103-133, 1984.

[Su91a]

A.I. SUBBOTIN. Minimax inequalities and Hamilton-.lacobi equations. Nauka, Moscow, 1991. In Russian.

[Su91b]

A.I. SUBBOTIN. Existence and uniqueness results for Hamilton-Jacobi equations. Nonlinear Anal., 16:683-699, 1991.

[Su93a]

A.I. SUBBOTIN. Discontinuous solutions of a Dirichlet type boundary value problem for first order partial differential equations. Russian J. Numer. Anal. Math. Modelling, 8:145-164, 1993.

[Su93b]

A.I. SUBBOTIN. On a property of the subdifferential. Math. USSR Sbornik, 74:63-78, 1993.

[Su95]

A.I. SUBBOTIN. Generalized solutions of first order PDEs: The Dynamic Optimization Perspective. Birkh~user, Boston, 1995.

562

BIBLIOGRAPHY

[Su971

A.I. SUBBOTIN.Constructive theory of positional differential games and generalized solutions to Hamilton-Jacobi equations. In M. Bardi, T. Parthasarathy, and T.E.S. Ragha~an, editors, Stochastic and differential games: Theory and numerical methods. Birkhs Boston, to appear.

[ss78]

A.I. SUBBOTINAND N.N. SUBBOTINA.Necessary and sufficient conditions for a piecewise smooth value of a differential game. Soviet Math. Dokl., 19:14471451, 1978.

[ss83]

A.I. SUBBOTINAND N.N. SUBBOTINA. On justification of dynamic programming method in an optimal control problem, hr. Acad. Nauk SSSR Tehn. Kibernet., 2:24-32, 1983. (In Russian).

[ST861

A.I. SUBBOTINAND A.M. TARASYEV. Stability properties of the value function of a differential game and viscosity solutions of Hamilton-Jacobi equations. Problems Control Inform. Theory, 15:451-463, 1986.

[Sua89]

N.N. SUBBOTINA. The maximum principle and the superdifferential of the value function. Problems Control Inform. Theory, 18:151-160, 1989.

[Sua91]

N.N. SUBBOTINA. The method of Cauchy characteristics and generalized solutions of the Hamilton-Jacobi-Bellmanequation. Soviet Math. Dokl., 44:501506, 1992.

[Sun93]

M. SUN. Domain decomposition algorithms for solving Hamilton-JacobiBellman equations. Numer. Funct. Anal. Appl., 14:143-166~ 1993.

[Sus83]

H.J. SUSSMANN. Lie brackets, real analiticity and geometric control. In R.W. Brockett, R.S. Millman, and H.J. Sussmann, editors, Differential geometric control theory, pages 1-116. Birkh{iuser, Boston, 1983.

[Sus87]

H.J. SUSSMANN. A general theorem on local controllability. SIAM J. Control Optim., 25:158-194, 1987.

[SusS9a]

H.J. SUSSMANN. Optimal control. In H. Nijmeijer and J.M. Schumacher, editors, Three decades of mathematical system theory, volume 135 of Lect. Notes Control Inform. Sci., pages 409-425. Springer, 1989.

[SusS9b]

H.J. SUSSMANN.Lecture notes on topics in system theory. Rutgers University, 1989.

[Sus90]

H.J. SUSSMANN. Synthesis, presynthesis, sufficient conditions for optimality and subanalytic sets. In H.J. Sussmann, editor, Nonlinear controllability and optimal control, chapter 1, pages 1-19. Marcel Dekker, New York~ 1990.

[Sw96a]

A. SWIECI-I. Sub- and superoptimality principles of Dynamic Programming revisited. Nonlinear Anal., 26:1429-1436, 1996.

[Sw96b]

A. SWIECH. Another approach to the existence of value functions of stochastic differential games. J. Math. Anal. Appl., 204:884-897, 1996.

[Ta941

A.M. TARAS'YEV.Approximation schemes for constructing minimax solutions of Hamilton-Jacobi equations. J. Appl. Math. Mech., 58:207-221, 1994.

[Ta97]

A.M. TARASYEV. Optimal control synthesis in grid approximation schemes. Technical Report 12, I.I.A.S.A., Laxenburg, Austria, 1997.

[T92]

D. TATARU. Boundary value problems for first order HamiltomJacobi equations. Nonlinear Anal., 19:1091-1110, 1992.

BIBLIOGRAPHY

563

[T94]

D. TATARU. Viscosity solutions for Hamilton-Jacobi equations with unbounded nonlinear term: a simplified approach. J. Differential Equations, 111:123146, 1994.

[Te95]

M.E. TESSITORE. Optimality conditions for infinite horizon control problems. Boll. Un. Mat. hal., (7) 9-B:785-814, 1995.

[Ti91]

I~L~[. TIDBALL. Sobre la resolucion numerica de la ecuacion de HamiltonJacobi-BeUman. PhD thesis, Universidad Nacional de Rosario, Argentina, 1991.

[Ti951

M.M. TIDBALL. Undiscounted zero sum differential games with stopping times. In G.J. Olsder, editor, New Trends in Dynamic games and Applications, pages 305-322. Birkh/iuser, Boston, 1995.

[TiG94]

M.M. TIDBALL AND R.L.V. CONZALEZ. Zero sum differential games with stopping times: some results about their numerical resolution. In T. Basar and A. Haurie, editors, Advances in dynamic games and applications, pages 106-124. Birkh~user, Boston, 1994.

[Tou92]

A. TOURIN. A comparison theorem for a piecewise Lipschitz continuous Hamiltonian and application to Shape-from-Shading problems. Numer. Math., 62:75-85, 1992.

[Val64]

F.A. VALENTINE. Convex sets. McGraw-Hill, New York, 1964.

[vdS921

A.J. VAN DER SHAFT. L2 gain analysis for nonlinear systems and nonlinear H~r control. IEEE Trans. Automat. Control, 37:770-784, 1982.

[vdS931

A.J. VAN DER SHAFT. Nonlinear state space Hor control theory. In H.L. Trentelman and J.C. Willems, editors, Perspectives in Control. Birkhs Boston, 1993.

[vK93]

B. VAN KEULEN. H~-control with measurement-feedback for linear infinitedimensional systems. J. Math. Systems Estim. Control, 3:373-411, 1993.

[vKPC93]

B. VAN KEULEN, 1~[. PETERS, AND R. CURTAIN. H~c-control with statefeedback: the infinite-dimensional case. J. Math. Systems Estim. Control, 3:1-39, 1993.

[Va67]

P.P. VARAIYA. On the existence of solutions to a differential game. SIAM J. Control, 5:153-162, 1967.

[Ve961

V.M. VELIOV. On the Lipschitz continuity of the value function in optimal control. J. Optim. Theory Appl., to appear.

[VW90a]

R . B . VINTER AND P. WOLENSKI. Hamilton-Jarobi theory for optimal control problems with data measurable in time. SIAM J. Control Optim., 90:14041419, 1990.

[VW9Ob]

R.B. VINTER AND P.R. WOLENSKI. Coextremals and the value function for control problems with data measurable in time. J. Math. Anal. Appl., 153:37-51, 1990.

[Wa72]

J. WARCA. Optimal control of differential and functional equations. Academic Press, New York, 1972.

[Wh90]

P. WHITTLE. Risk-sensitive optimal control. Wiley, New York, 1990.

564

BIBLIOGRAPHY

[w25]

N. WIENER. Note on a paper of O. Perron. J. Math. Physics (M.I.T.), 4:2132, 1925.

[Wi96]

S.A. WILLIAMS. Local behavior of solutions of Hamilton-Jacobi equations. Nonlinear Anal., 26:323-328, 1996.

[WZ97]

P.R. WOLENSKI AND Yu ZHUANG. Proximal analysis and the minimal time function. SIAM J. Control Optim., to appear.

[Y89]

J. YONG. Systems governed by ordinary differential equations with continuous, switching and impulse controls. Appl. Math. Optim., 20:223-235, 1989.

[v9o]

J. YONG. A zero-sum differential game in a finite duration with switching strategies. SIAM J. Control Optim., 28:1234-1250, 1990.

[Y94]

J. YONG. Zero-sum differential games involving impulse controls. Appl. Math. Optim., 29:243-261, 1994.

[Yo6S]

J.A. YORKE. Extending Liapunov's second method to non-Lipschitz Liapunov functions. Bull. Amer. Math. Soc., 74:322-325, 1968.

[You69]

L.C. YOUNG. Lectures on the Calculus of Variations. Chelsea, New York, 2 edition, 1980.

[Za81]

G. ZAMES. Feedback optimal sensitivity: model reference transformations, multiplicative seminorms, and approximate inverses. IEEE Trans. Automat. Control, 26:301-320, 1981.

[z9o]

X.Y. Znou. Maximum principle, dynamic programming, and their connection in deterministic control. J. Optim. Th. Appl., 65:363-373, 1990.

[z93]

X.Y. Znou. Verification theorems within the framework of viscosity solutions. J. Math. Anal. Appl., 177:208-225, 1993.

[ZYL97]

X.Y. ZHOU, J. YONG, AND X. LI. Stochastic verification theorems within the framework of viscosity solutions. SIAM J. Control Optim., 35:243-253, 1997.

[Zo92]

W. ZOLEZZI.Wellposedness and the Lavrentiev phenomenon. SIAM J. Control Optim., 30:787-789, 1992.

Index

A acceleration algorithm, 500 adjoint system, 14, 138 adjoint vector, 13, 168, 174 admissible controls, 272 approximation, 17, 20, 118, 154 by discrete time games, 457 by vanishing viscosity, 20, 383 scheme, 472 semidiscrete, 359 Ascoli-Arzel~ theorem, 219 attainable set, 282 averaging, 430 B

bang-bang principle, 113 Barles-Perthame procedure, 357 barrier, 302 Barron-Jensen discontinuous solution, 285 Bellman functional equation, 389 Bellman's function, 99 bilateral solution, 120, 382 subsolution, 120 supersolution, 79, 120, 349 non-continuous, 342 Bolza problem, 147 boundary condition, 227, 347 at infinity, 245 Dirichlet, 256 in the viscosity sense, 317 for state constrained problems, 277 for the minimal time function, 241 Neumann, 281 nonlinear oblique derivative, 281

boundary layer, 323 Bump Lemma, 303 C certainty equivalence principle, 467, 523 chain rule, 135 characteristic system, 86, 181 characteristics, 77, 96, 181 chattering controls, 113 Chow theorem, 233 Clarke's gradient, 63, 85 comparison principle, 7, 9, 27, 50, 107, 109, 191, 215, 256, 262, 278, 281,286, 326, 328, 329, 389 for Cauchy problems, 56, 152, 156 for Cauchy-Dirichlet problems, 295 for convex Hamiltonian, 82 for time-dependent HJB equations, 182 local comparison for Dirichlet problems, 266 for initial value problems, 156, 295 compatibility of the boundary data, 284 compatible sets of strategies, 454 compatible terminal cost, 251 complementarity system, 196 complete solution, 107, 151, 244, 256, 287, 334 computational methods, 17, 471 conditions of optimality, 133 necessary, 134, 171 necessary and sufficient, 1_38, 172 suff• 136, 172 cone of dependence, 156 constrained viscosity solution, 277, 399

566 constraints on the endpoint of the trajectories, 348 contingent derivatives, 125 continuity of the value function, 99, 146, 185, 194, 200, 211, 248, 251, 260, 268, 274 control, 1 closed loop, 111 law, 13, 110 multivalued, 143 open loop, 98 optimal, 103, 368 piecewise constant, 99 space, 1 suboptimal, 17 system, 1, 97 controllability, 227, 398 matrix, 231 small-time, 228, 268 local, 228, 331 convexified problem, 112 cost functional, 1, 99 costate, 13, 168, 174 D Danskin theorem, 95 detectability, 507 differentiability of solutions of HJ equations, 80 of solutions of ODEs with respect to the initial data, 222 of the value function, 167 differential games, 431,512 N-person, 470 positional, 456 stochastic, 470 two-person zero-sum, 431 differential inequalities, 86 directional derivatives Dini, 64, 162 lower, 125 upper, 125 generalized, 63 regularized, 63 Dini solution, 125, 163 generalized, 127 lower, 131 subsolution, 162 generalized, 127 supersolution, 127, 162

INDEX Dirichlet problem, 51, 84, 254, 260, 277, 305, 308, 316, 342 subsolution, 287 supersolution, 287 v-subsolution, 318, 347 v-supersolution, 318 discontinuous equations, 319 discount factor, 2 discrete time Dynamic Programming, 17, 388 game, 457 fair, 458 infinite horizon problem, 388 system~ 388 dissipativity, 397, 403 distance function, 34, 42, 47 disturbances, 506 domain decomposition, 502 Dynamic Programming, 1 equation, 389, 458,473, 515 Principle, 2, 102~ 149, 155, 182, 187, 194, 201, 211, 239, 254, 276. 388, 437, 451,465 Backward, 119, 160, 344 discrete time, 18~ 389 dynamics, 1 E

e-solution, see envelope solution eikonal equation, 47, 82, 84, 241 entry time, 228 envelope solution, 309, 347 subsolution, 309, 319 supersolution, 309, 319 ergodic problems, 397 escape controllability domain, 237, 245, 265 escape time, 237 estimate of the error, 464, 475 existence of an optimal control, 368 of an optimal strategy, 520 of a value, 442 of solutions of HJ equations, 302 of trajectories global, 220 local, 219 exit time, 123, 228 exterior normal, 48 generalized, 48, 269

INDEX F falsification function, 112, 151 fast components, 420 feedback map, 111,387 admissible, 111, 144 memoryless, 111 multivalued admissible, 141 fully optimal, 142 weakly optima], 142 feedback strategies, 431,435 finite horizon problem, 147, 348 free boundary, 199, 241, 246, 263, 265 fundamental matrix, 222 G game differential, 431,512 discrete time, 457 majorant, 458 minorant, 458 pursuit-evasion, 469, 494 generalized gradient, 63 generalized solution, 10, 32, 77 geometric optics, 430 geometric properties of solutions, 96 geometric theory of controllability, 283 Gronwall inequality, 218 guaranteed outcome, 99 H

7-/~r control, 505 7-/~r norm, 508 Hamilton-Jacobi equation, 25, 516 Hamilton-Jacobi-Bellman equation, 2, 3, 104, 150, 188, 196, 203, 212, 240, 255, 414 backward, 154 evolutive, 156, 307 Hamilton-Jacobi-Isaacs equation, 438 lower, 439, 453 upper, 439, 453 Hamiltonian system, 181 Hausdorff distance, 116, 376 high-order methods, 501 homogenization, 430 Hopf's formulas, 96 I

imperfect state measurement, 467 implicit obstacle, 212

567 impulse, 183, 200 control, 200 cost, 200 inf-convolution, 43, 72, 345 infinite horizon discounted regulator, 1 infinite horizon problem, 99, 271, 376, 397, 435, 472 with constraints, 227 information pattern, 431,448, 458 input, 506 integrodifferential equations, 226 interest rate, 99 intrinsically classical solution, 128, 132, 163, 170 Isaacs' condition, 444 Ishii discontinuous solution, 285 Ito stochastic differential equation, 384 J

Jacobi bracket, 233 K

K-strategy, 449 Kru2kov transform, 243, 263 L gain, 508 Lagrange problem, 147 large deviations, 396 Lavrentiev phenomenon, 223 Lewy-Stampacchia type inequality, 216 Lie bracket, 233, 276 linear system, 224, 231,506, 510 linearized system, 231 Lipschitz continuity, 62 of subsolutions, 294 Lipschitz control, 425,427 local grid refinements, 501 long run averaged cost, 397, 400 lower semicontinuous envelope, 296 LQ problem, 224 Lyapunov stable, 507 asymptotically, 507 Lp

M marginal function, 34, 42, 65, 163 maximum principle, 27 Mayer problem, 147, 160 minimal interior time function, 228, 240 minimal time function, 228, 313 continuity, 230 HSlder continuity, 230, 233, 235

568 minimal time function (continued) Lipschitz continuity, 230, 234, 269 regularization, 379 minimax solution, 125, 127, 358 minimizing sequence, 363 Minimum Principle, 135, 170, 173 modulus, 98 monotone acceleration procedure, 477 monotone controls, 99 problem, 183, 426 value function, 184 motions of the game, 456 N Nash equilibrium, 447 nonanticipating strategy, 431 nonautonomous problems, 182 nonautonomous systems, 147, 154, 159 nonconvex problems in the Calculus of Variations, 96 nonlinear H~r control, 505 nonlinear systems, 98, 431 affine, 112, 444, 506 numerical tests, 481,496 O observed output, 506, 521 observed state, 467 optimal control, 2, 103 approximate, 362 existence, 368 in a differential game, 446 law, 110 optimal feedback control, 393 map, 2, 11,199 multivalued, 142 optimal point, 119, 160 optimal relaxed control, 366 feedback strategies, 524 strategy, 446, 520 existence, 520 optimal result function, 99 optimal stopping, 193, 402 optimal strategy, 520 optimal switching, 210, 411 optimal trajectory, 169 optimizing sequence of strategies, 466 output, 467, 506 state feedback strategy, 466

INDEX P parallel algorithms, 502 partial information, 521 partial observation, 467 penalization, 414 of state constraints, 417 of stopping costs, 414 of unilateral constraints, 414 perfect memory, 449 perfect state measurement, 449 Perron's method, 96, 304, 311,357 perturbation, 118, 154, 244, 279, 291, 320, 333, 336 of the domain, 323, 333 regular, 333, 376 singular, 21,322, 333, 384, 420, 421 Petrov theorem, 231,269 pointwise monotone limits, 306 Pontryagin Maximum Principle, 13, 135, 138, 168, 173, 355 extended, 179 for relaxed controls, 175 positive basis, 238 condition, 232 presynthesis, 110 multivalued, 143 maximal optimal, 143 of strategies, 441 problems with exit times, 247, 342 problems with state constraints, 271,484 projections, 47 propagation of fronts, 96, 430 proximal normal, 48, 50 pursuit-evasion games, 469, 494

Q quasivariational inequality, 183, 203 queuing systems, 430 R

Rademacher theorem, 63 Radon measure, 114 rate of convergence, 369, 372, 385, 408, 464, 476 reachable set, 228, 263 reaction-diffusion equations, 430 reduced order modeling, 421 regularity, 80, 163, 224 regularizing effect, 96 relaxation, 110, 154, 244, 279

INDEX relaxed controls, 113, 175, 354, 444 representation formula, 96, 260 Riccati equation, 394 risk-sensitive control, 505 robust attenuation of disturbances, 505 rocket railroad car, 242, 490 running cost, 1, 99 S saddle, 444 point, 454 Schauder fixed point theorem, 219 semiconcave, 11, 65, 77, 355 -convex, 164 locally, 163 semiconcavity, 372, 387, 391 semidifferentials, 29 semigroup property, 155 separation property, 444 shape-from-shading, 96 signed distance, 234, 272 simplex method, 477 singular perturbation, 21, 322, 333, 384, 420, 421 singular surfaces, 468 singularities of solutions, 96 solution in the extended sense, 85 solution operator, 175 solvability of the small game, 444 stability, 9, 35, 116, 289, 291, 306, 333, 336, 354 of v-solutions, 320 with respect to the first player, 456 with respect to the second player, 456 stable bridge, 457 state constraints, 271,397, 417, 484 state equation, 1 stochastic control, 20, 383 stochastic differential equation, 20 stopping cost, 193 stopping time, 183, 193, 200 strategy causal, 433 closed-loop, 449 feedback, 448 memoryless, 449 nonanticipating, 433 discrete, 458 optimal, 446, 454, 520 approximate, 462

569 output feedback, 521 playable, 449 sampled-data, 449 slightly delayed, 442 with finite memory, 452 subdifferential, 5, 29, 95 suboptimality principle global, 124, 161 local, 122, 161 sup-convolution, 72 superdifferential, 5, 29 superoptimality principle global, 124, 257 local, 122, 161 switching, 183, 425, 426 cost, 211 symmetric system, 233, 276 symplectic mechanics, 96 synthesis, 2, 10, 19, 111, 140, 209, 215, 382, 390 multivalued, 143 maximal optimal, 143 of feedback controls, 478, 502 of optimal strategies, 462 system of quasivariational inequalities, 212, 411 systems of HJB equations, 226 T target set, 227 smooth, 233 with piecewise smooth boundary, 236 tenet of transition, 102 terminal cost, 147, 247 compatible, 251 lower semicontinuous, 349 test functions, 5 time optimal control, 227, 313, 333, 345, 488 trajectory, 98 backward, 119 U unbounded costs, 156 undiseounted value, 406 unique evolution property, 155 uniqueness result, 7, 50, 107, 152, 159, 197, 206, 213, 241, 244, 246, 256, 260, 278, 330 in the sense of the graph, 336

570 upper semicontinuous envelope, 296 usable part of the boundarz. 331 V value function, 2, 99 approximate, 17 continuity, 99, 146, 185, 194; 200, 211,248, 251,260, 268, 274 differentiability, 167 discontinuous, 298 vector valued, 212 value (of a differential game), 434 Berkovitz, 456 existence, 442 feedback, 450 Fleming, 461 Friedman, 465 Krasovskii and Subbotin, 456 lower, 434 relaxed, 446 upper, 434 Varaiya, Roxin, Elliott and Kalton, 434, 468 vanishing discount, 397, 402 vanishing switching costs, 411 vanishing viscosity, 20, 322, 337, 383

Printed in the United States of America

INDEX variational inequality, 183, 185, 188 of obstacle type, 193, 196 verification function, 12, 151, 171, 316, 447 generalized, 112 verification theorems, 110, 154, 244, 279, 287, 316, 446 viability, 275, 358, 470 viscosity solution, 1, 5, 25 non-continuous, 296, 318 subsolution, 5, 25, 31 lower semicontinuous, 286 of Dirichlet boundary condition, 318 supersolution, 5, 25, 31,286, 318 upper semicontinuous, 286 of Dirichlet boundary condition, 318 W weak limits, 10, 321,359, 360 lower, 288 upper, 288 Z Zermelo Navigation Problem, 300, 307, 357