Lagrange Multiplier Approach to Variational Problems and Applications
Advances in Design and Control SIAM’s Advances i...
58 downloads
1463 Views
3MB Size
Report
This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form
Lagrange Multiplier Approach to Variational Problems and Applications
Advances in Design and Control SIAM’s Advances in Design and Control series consists of texts and monographs dealing with all areas of design and control and their applications. Topics of interest include shape optimization, multidisciplinary design, trajectory optimization, feedback, and optimal control. The series focuses on the mathematical and computational aspects of engineering design and control that are usable in a wide variety of scientific and engineering disciplines.
Editor-in-Chief Ralph C. Smith, North Carolina State University
Editorial Board Athanasios C. Antoulas, Rice University Siva Banda, Air Force Research Laboratory Belinda A. Batten, Oregon State University John Betts, The Boeing Company Stephen L. Campbell, North Carolina State University Eugene M. Cliff, Virginia Polytechnic Institute and State University Michel C. Delfour, University of Montreal Max D. Gunzburger, Florida State University J. William Helton, University of California, San Diego Arthur J. Krener, University of California, Davis Kirsten Morris, University of Waterloo Richard Murray, California Institute of Technology Ekkehard Sachs, University of Trier
Series Volumes Ito, Kazufumi and Kunisch, Karl, Lagrange Multiplier Approach to Variational Problems and Applications Xue, Dingyü, Chen, YangQuan, and Atherton, Derek P., Linear Feedback Control: Analysis and Design with MATLAB Hanson, Floyd B., Applied Stochastic Processes and Control for Jump-Diffusions: Modeling, Analysis, and Computation Michiels, Wim and Niculescu, Silviu-Iulian, Stability and Stabilization of Time-Delay Systems: An Eigenvalue-Based Approach ¸ Adaptive Control Tutorial Ioannou, Petros and Fidan, Baris, Bhaya, Amit and Kaszkurewicz, Eugenius, Control Perspectives on Numerical Algorithms and Matrix Problems Robinett III, Rush D., Wilson, David G., Eisler, G. Richard, and Hurtado, John E., Applied Dynamic Programming for Optimization of Dynamical Systems Huang, J., Nonlinear Output Regulation: Theory and Applications Haslinger, J. and Mäkinen, R. A. E., Introduction to Shape Optimization: Theory, Approximation, and Computation Antoulas, Athanasios C., Approximation of Large-Scale Dynamical Systems Gunzburger, Max D., Perspectives in Flow Control and Optimization Delfour, M. C. and Zolésio, J.-P., Shapes and Geometries: Analysis, Differential Calculus, and Optimization Betts, John T., Practical Methods for Optimal Control Using Nonlinear Programming El Ghaoui, Laurent and Niculescu, Silviu-Iulian, eds., Advances in Linear Matrix Inequality Methods in Control Helton, J. William and James, Matthew R., Extending H∞ Control to Nonlinear Systems: Control of Nonlinear Systems to Achieve Performance Objectives
Lagrange Multiplier Approach to Variational Problems and Applications
Kazufumi Ito North Carolina State University Raleigh, North Carolina
Karl Kunisch University of Graz Graz, Austria
Society for Industrial and Applied Mathematics Philadelphia
Copyright © 2008 by the Society for Industrial and Applied Mathematics. 10 9 8 7 6 5 4 3 2 1 All rights reserved. Printed in the United States of America. No part of this book may be reproduced, stored, or transmitted in any manner without the written permission of the publisher. For information, write to the Society for Industrial and Applied Mathematics, 3600 Market Street, 6th Floor, Philadelphia, PA 19104-2688 USA. Trademarked names may be used in this book without the inclusion of a trademark symbol. These names are used in an editorial context only; no infringement of trademark is intended. Library of Congress Cataloging-in-Publication Data Ito, Kazufumi. Lagrange multiplier approach to variational problems and applications / Kazufumi Ito, Karl Kunisch. p. cm. -- (Advances in design and control ; 15) Includes bibliographical references and index. ISBN 978-0-898716-49-8 (pbk. : alk. paper) 1. Linear complementarity problem. 2. Variational inequalities (Mathematics). 3. Multipliers (Mathematical analysis). 4. Lagrangian functions. 5. Mathematical optimization. I. Kunisch, K. (Karl), 1952– II. Title. QA402.5.I89 2008 519.3--dc22 2008061103
is a registered trademark.
To our families: Junko, Yuka, and Satoru Brigitte, Katharina, Elisabeth, and Anna
i
i
i
ItoKunisc 2008/6/12 page vii i
Contents Preface 1
xi
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
1 1 3 5 8 17 23 23 24
Sensitivity Analysis 2.1 Generalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Implicit function theorem . . . . . . . . . . . . . . . . . . . . . . 2.3 Stability results . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Lipschitz continuity . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Differentiability . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Application to optimal control of an ordinary differential equation
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
27 27 31 34 45 53 62
First Order Augmented Lagrangians for Equality and Finite Rank Inequality Constraints 3.1 Generalities . . . . . . . . . . . . . . . . . . . . . 3.2 Augmentability and sufficient optimality . . . . . . 3.3 The first order augmented Lagrangian algorithm . . 3.4 Convergence of Algorithm ALM . . . . . . . . . . 3.5 Application to a parameter estimation problem . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
65 65 67 75 78 82
4 Augmented Lagrangian Methods for Nonsmooth, Convex Optimization 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Convex analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.1 Conjugate and biconjugate functionals . . . . . . . . . . . . . 4.2.2 Subdifferential . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Fenchel duality theory . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . .
. . . . .
. . . . .
87 87 89 92 95 98
2
3
Existence of Lagrange Multipliers 1.1 Problem statement and generalities . . . . . . . . . . . 1.2 A generalized open mapping theorem . . . . . . . . . . 1.3 Regularity and existence of Lagrange multipliers . . . 1.4 Applications . . . . . . . . . . . . . . . . . . . . . . . 1.5 Weakly singular problems . . . . . . . . . . . . . . . . 1.6 Approximation, penalty, and adapted penalty techniques 1.6.1 Approximation techniques . . . . . . . . . . . . 1.6.2 Penalty techniques . . . . . . . . . . . . . . . .
. . . . .
. . . . .
. . . . . . . .
. . . . .
. . . . . . . .
. . . . .
. . . . . . . .
. . . . .
. . . . . . . .
. . . . .
. . . . . . . .
. . . . .
. . . . .
vii
i
i i
i
i
i
i
viii
Contents 4.4 4.5 4.6 4.7
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
104 109 114 119 120 121 122 122 124 125 126 126
Newton and SQP Methods 5.1 Preliminaries . . . . . . . . . . . . . . . . . . 5.2 Newton method . . . . . . . . . . . . . . . . . 5.3 SQP and reduced SQP methods . . . . . . . . . 5.4 Optimal control of the Navier–Stokes equations 5.4.1 Necessary optimality condition . . . . . 5.4.2 Sufficient optimality condition . . . . . 5.4.3 Newton’s method for (5.4.1) . . . . . . 5.5 Newton method for the weakly singular case . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
129 129 133 137 143 145 147 147 148
6 Augmented Lagrangian-SQP Methods 6.1 Generalities . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Equality-constrained problems . . . . . . . . . . . . . . . . 6.3 Partial elimination of constraints . . . . . . . . . . . . . . . 6.4 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.1 An introductory example . . . . . . . . . . . . . . . 6.4.2 A class of nonlinear elliptic optimal control problems 6.5 Approximation and mesh-independence . . . . . . . . . . . 6.6 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
155 155 156 165 172 172 174 183 186
7 The Primal-Dual Active Set Method 7.1 Introduction and basic properties . . . . . . . . . . . 7.2 Monotone class . . . . . . . . . . . . . . . . . . . . 7.3 Cone sum preserving class . . . . . . . . . . . . . . 7.4 Diagonally dominated class . . . . . . . . . . . . . . 7.5 Bilateral constraints, diagonally dominated class . . 7.6 Nonlinear control problems with bilateral constraints
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
189 189 196 197 200 202 206
Semismooth Newton Methods I 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Semismooth functions in finite dimensions . . . . . . . . . . 8.2.1 Basic concepts and the semismooth Newton algorithm 8.2.2 Globalization . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
215 215 217 217 222
5
8
ItoKunisc 2008/6/12 page viii i
Generalized Yosida–Moreau approximation Optimality systems . . . . . . . . . . . . . Augmented Lagrangian method . . . . . . . Applications . . . . . . . . . . . . . . . . . 4.7.1 Bingham flow . . . . . . . . . . . . 4.7.2 Image restoration . . . . . . . . . . 4.7.3 Elastoplastic problem . . . . . . . . 4.7.4 Obstacle problem . . . . . . . . . . 4.7.5 Signorini problem . . . . . . . . . . 4.7.6 Friction problem . . . . . . . . . . . 4.7.7 L1 -fitting . . . . . . . . . . . . . . . 4.7.8 Control problem . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . .
. . . . . .
. . . . . .
i
i i
i
i
i
i
Contents
8.3 8.4 8.5 8.6 9
ix
8.2.3 Descent directions . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.4 A Gauss–Newton algorithm . . . . . . . . . . . . . . . . . . . . . 8.2.5 A nonlinear complementarity problem . . . . . . . . . . . . . . . Semismooth functions in infinite-dimensional spaces . . . . . . . . . . . The primal-dual active set method as a semismooth Newton method . . . Semismooth Newton methods for a class of nonlinear complementarity problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Semismooth Newton methods and regularization . . . . . . . . . . . . .
Semismooth Newton Methods II: Applications 9.1 BV-based image restoration problems . . . . . 9.2 Friction and contact problems in elasticity . . . 9.2.1 Generalities . . . . . . . . . . . . . . . 9.2.2 Contact problem with Tresca friction . . 9.2.3 Contact problem with Coulomb friction .
. . . . .
225 228 231 234 240
. 243 . 246
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
253 254 263 263 265 272
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
277 281 291 292 297 302
11 Shape Optimization 11.1 Problem statement and generalities . . . . . . . . 11.2 Shape derivative . . . . . . . . . . . . . . . . . . 11.3 Examples . . . . . . . . . . . . . . . . . . . . . 11.3.1 Elliptic Dirichlet boundary value problem 11.3.2 Inverse interface problem . . . . . . . . . 11.3.3 Elliptic systems . . . . . . . . . . . . . . 11.3.4 Navier–Stokes system . . . . . . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
305 305 308 314 314 316 321 323
10 Parabolic Variational Inequalities 10.1 Strong solutions . . . . . . . . . . . . . 10.2 Regularity . . . . . . . . . . . . . . . . 10.3 Continuity of q → y(q) ∈ L∞ () . . . 10.4 Difference schemes and weak solutions 10.5 Monotone property . . . . . . . . . . .
ItoKunisc 2008/6/12 page ix i
. . . . .
. . . . .
. . . . .
. . . . .
Bibliography
327
Index
339
i
i i
i
i
ItoKunisc 2008/6/12 page x i
i
i
i
i
i
i
i
i
i
ItoKunisc 2008/6/12 page xi i
Preface The objective of this monograph is the treatment of a general class of nonlinear variational problems of the form min
y∈Y, u∈U
f (y, u)
subject to e(y, u) = 0,
g(y, u) ∈ K,
(0.0.1)
where f : Y ×U → R denotes the cost functional, and e : Y ×U → W and g : Y ×U → Z are functionals used to describe equality and inequality constraints. Here Y , U , W , and Z are Banach spaces and K is a closed convex set in Z. A special choice for K is simple box constraints K = {z ∈ Z : φ ≤ z ≤ ψ}, (0.0.2) where Z is a lattice with ordering ≤, and φ, ψ are elements in Z. Theoretical issues which will be treated include the existence of minimizers, optimality conditions, Lagrange multiplier theory, sufficient optimality considerations, and sensitivity analysis of the solutions to (0.0.1) with respect to perturbations in the problem data. These topics will be covered mainly in the first part of this monograph. The second part focuses on selected computational methods for solving the constrained minimization problem (0.0.1). The final chapter is devoted to the characterization of shape gradients for optimization problems constrained by partial differential equations. Problems which fit into the framework of (0.0.1) are quite general and arise in application areas that were intensively investigated during the last half century. They include optimal control problems, structural optimization, inverse and parameter estimation problems, contact and friction problems, problems in image reconstruction and mathematical finance, and others. The variable y is often referred to as the state variable and u as the control or design parameter. The relationship between the variables y and u is described by e. It typically represents a differential or a functional equation. If e can be used to express the variable y as a function of u, i.e., y = (u), then (0.0.1) reduces to min J (u) = f ((u), u) subject to g((u), u) ∈ K. u∈U
(0.0.3)
This is referred to as the reduced formulation of (0.0.1). In the more general case when y and u are both independent variables linked by the equation constraint e(y, u) = 0 it will be convenient at times to introduce x = (y, u) in X = Y × U and to express (0.0.1) as min f (x) x∈X
subject to e(x) = 0, g(x) ∈ K.
(0.0.4)
xi
i
i i
i
i
i
i
xii
ItoKunisc 2008/6/12 page xii i
Preface
From a computational perspective it can be advantageous to treat y and u as independent variables even though a representation of y in terms of u is available. As an example consider the inverse problem of determining the diffusion parameter a in −∇ · (a ∇y) = f in , y|∂ = 0 (0.0.5) from the measurement yobs , where is a bounded open set in R2 and f ∈ H −1 (). A nonlinear least squares approach to this problem can be formulated as (0.0.1) by choosing y ∈ Y = H01 (), a ∈ U = H 1 () ∩ L∞ (), W = H −1 (), Z = L2 () and considering min (|y − yobs |2 + β |∇a|2 ) dx (0.0.6) subject to (0.0.5) and a ≤ a ≤ a, ¯ where 0 < a < a¯ < ∞ are lower and upper bounds for the “control” variable a. In this example, given a ∈ U , the state y ∈ Y can be uniquely determined by solving the elliptic boundary value problem (0.0.5) for y, and we obtain a problem of type (0.0.3). The general approach that we follow in this monograph for the analytical as well as the numerical treatment of (0.0.1) is based on Lagrange multiplier theory. Let us subsequently suppose that Y, U , and W are Hilbert spaces and discuss the case Z = U and g(y, u) = u, i.e., the constraint u ∈ K. We assume that f and e are C 1 and denote by fy , fu the Fréchet derivatives of f with respect to y and u, respectively. Let W ∗ be the dual space of W and let the duality product be denoted by ·, · W ∗ ,W . The analogous notation is used for Z and Z ∗ . We form the Lagrange functional L(y, u, λ) = f (y, u) + e(y, u), λ W,W ∗ ,
(0.0.7)
where λ ∈ W ∗ is the Lagrange multiplier associated with the equality constraint e(y, u) = 0 which, for the present discussion, is supposed to exist. It will be shown that a minimizing pair (y, u) satisfies Ly (y, u) = fy (y, u) + ey (y, u)∗ λ = 0, fu (y, u) + eu (y, u)∗ λ , v − u Z∗ ,Z ≥ 0 for all v ∈ K,
(0.0.8)
e(y, u) = 0 and u ∈ K. In the case K = Z the second equation of (0.0.8) results in the equality fy (y, u) + eu (y, u) = 0. In this case a first possibility for solving the system (0.0.8) for the unknowns (y, u, λ) is the use of a direct equation solver of Newton type, for example. Here the Lagrange multiplier λ is treated as an independent variable just like y and u. Alternatively, for (y, u) satisfying e(y, u) = 0 and λ satisfying fy (y, u) + eu (y, u)∗ λ = 0 the gradient of J (u) of (0.0.4) can be evaluated as Ju = fu (y, u) + eu (y, u)∗ λ.
(0.0.9)
Thus the combined step of determining y ∈ Y for given u ∈ K such that e(y, u) = 0 and finding λ ∈ W ∗ satisfying fy (y, u) + eu (y, u)∗ λ = 0 for (y, u) ∈ Y × U provides a
i
i i
i
i
i
i
Preface
ItoKunisc 2008/6/12 page xiii i
xiii
possibility for evaluating the gradient of J at u. If in addition u has to satisfy a constraint of the form u ∈ K, the projected gradient is obtained by projecting fu (y, u) + eu (y, u)∗ λ onto K, and (projected) gradient-based iterative methods can be employed to solve (0.0.1). In optimal control of differential equations, the multiplier λ is called the adjoint state and the second equation in (0.0.8) is called the optimality condition. Further the system (0.0.8) coincides with the celebrated Pontryagin maximum principle. The steps in the procedure explained above for obtaining the gradient are referred to as the forward equation step for the state equation (the third equation in (0.0.8)) and the backward equation step for the adjoint equation (the first equation). This terminology is motivated by time-dependent problems, where the third equation is an initial value problem and the first one is a problem with terminal time boundary condition. Lastly, if K is of type (0.0.2), then the second equation of (0.0.8) can be written as the complementarity condition fu (y, u) + eu (y, u)∗ λ + η = 0, η = max(0, η + (u − ψ)) + min(0, η + (u − φ)).
(0.0.10)
Due to the nondifferentiability of the max and min operations classical Newton methods are not directly applicable to system (0.0.10). Active set methods provide a very efficient alternative. They turn out to be equivalent to semismooth Newton methods in function spaces. If appropriate structural conditions are met, including that f be quadratic and e affine, then such techniques are globally convergent. Moreover, under more general conditions they exhibit local superlinear convergence. Due to the practical relevance of problems with box constraints, a significant part of this monograph is devoted to the analysis of these methods. A frequently employed alternative to the multiplier approach is given by the penalty method. To explain the procedure consider the sequence of minimization problems ck min f (y, u) + | e(y, u)|2W (0.0.11) y∈Y,u∈K 2 for an increasing sequence of penalty parameter ck . That is, the equality constraint is eliminated through the quadratic penalty function. This requires to solve the unconstrained minimization problem (0.0.11) over Y × K for a sequence of {ck } tending to infinity. Under mild assumptions it can be shown that the sequence of minimizers (yk , uk ) determined by (0.0.11) converges to a minimizer of (0.0.1) as ck → ∞. Moreover, (yk , uk ) satisfies fy (yk , uk ) + ey (yk , uk )∗ (ck e(yk , uk )) = 0, fu (yk , uk ) + eu (yk , uk )∗ (ck e(yk , uk )), v − uk Z∗ ,Z ≥ 0.
(0.0.12)
Comparing (0.0.12) with (0.0.8) it is natural to ask whether ck e(yk , uk ) tends to a Lagrange multiplier associated to e(y, u) = 0 as ck → ∞. Indeed this can be shown under suitable conditions. Despite disadvantages due to slow convergence and possible ill-conditioning for solving (0.0.12) with large values of ck , the penalty method is widely accepted in practice. This is due, in part, to the simplicity of the approach and the availability of powerful algorithms for solving (0.0.12) if K = Z, when the inequality in (0.0.12) becomes an equality. Athird methodology is given by duality techniques. They are based on the introduction of the dual functional d(λ) = inf L(y, u, λ) (0.0.13) y∈Y, u∈K
i
i i
i
i
i
i
xiv
ItoKunisc 2008/6/12 page xiv i
Preface
and the duality property sup d(λ) = λ∈Y
inf
y∈Y, u∈K
f (y, u)
subject to e(y, u) = 0.
(0.0.14)
The duality method for solving (0.0.1) requires minimizing L(y, u, λk ) over (y, u) ∈ Y ×K and updating λ by means of λk+1 = λk + αk J e(yk , uk ),
(0.0.15)
where (yk , uk ) = argmin L(y, u, λk ), y∈Y, u∈K
αk > 0 is an appropriately chosen step size and J denotes the Riesz mapping from W onto W ∗ . It can be argued that e(yk , uk ) is the gradient of d(λ) at λk and (0.0.15) is in turn a steepest ascent method for the maximization of d(λ) under appropriate conditions. Such methods are called primal-dual methods. Despite the fact that the method can be justified only under fairly restrictive convexity assumptions on L(y, u, λ) with respect to (y, u), it provides an elegant use of Lagrange multipliers and is a basis for so-called augmented Lagrangian methods. Augmented Lagrangian methods with K = Z are based on the following problem which is equivalent to (0.0.1): c min f (y, u) + |e(y, u)|2W 2 subject to e(y, u) = 0. y∈Y,u∈K
(0.0.16)
Under rather mild conditions the quadratic term enhances the local convexity of L(y, u, λ) in the variables (y, u) for sufficiently large c > 0. It helps the convergence of direct solvers based on the necessary optimality conditions (0.0.8). To carry this a step further we introduce the augmented Lagrangian functional c Lc (y, u, λ) = f (x, u) + e(y, u), λ + |e(y, u)|2W . 2
(0.0.17)
The first order augmented Lagrangian method is the primal-dual method applied to (0.0.17), i.e., (yk , uk ) = argmin Lc (y, u, λk ), y∈Y, u∈K (0.0.18) λk+1 = λk + c J e(yk , uk ). Its advantage over the penalty method is attributed to the fact that local convergence of (yk , uk ) to a minimizer (y, u) of (0.0.1) holds for all sufficiently large and fixed c > 0, without requiring that c → ∞. As we noted, Lc (y, u, λ) has local convexity properties under well-studied assumptions, and the primal-dual viewpoint is applicable to the multiplier update. The iterates (yk , uk , λk ) converge linearly to the triple (y, u, λ) satisfying the first order necessary optimality conditions, and convergence can improve as c > 0 increases. Due to these attractive characteristics and properties, the method of multipliers and its subsequent Newton-like variants have been recognized as a powerful method for minimization problems with equality constraints. They constitute an important part of this book.
i
i i
i
i
i
i
Preface
ItoKunisc 2008/6/12 page xv i
xv
In (0.0.16) and (0.0.18) the constraint u ∈ K remained as explicit constraint and was not augmented. To describe a possibility for augmenting inequalities we return to the general form g(y, u) ∈ K and consider inequality constraints with finite rank, i.e., Z = Rp and K = {z ∈ Rp : zi ≤ 0, 1 ≤ i ≤ p}. Then under appropriate conditions the formulation min
y∈Y, u∈U,
q∈Rp
Lc (y, u, λ) + (μ, g(y, u) − q) +
c |g(y, u) − q|2Rp 2
subject to q ≤ 0
(0.0.19)
is equivalent to (0.0.1). Here μ ∈ Rp is the Lagrange variable associated with the inequality constraint g(y, u) ≤ 0. Minimizing the functional in (0.0.19) over q ≤ 0 results in the augmented Lagrangian functional Lc (y, u, λ, μ) = Lc (y, u, λ) +
1 c | max(0, μ + c g(y, u))|2Rp − |μ|2Rp , 2 2
(0.0.20)
where equality and finite rank inequality constraints are augmented. The corresponding augmented Lagrangian method is (yk , uk ) = argmin Lc (y, u, λk , μk ), y∈Y, u∈K
λk+1 = λk + c J e(yk , uk ),
(0.0.21)
μk+1 = max(0, μk + c g(yk , uk )). In the discussion of box-constrained problems we already pointed out the relevance of numerical methods for nonsmooth problems; see (0.0.10). In many applied variational problems, for example in mechanics, fluid flow, or image analysis, nonsmooth cost functionals arise. Consider as a special case the simplified friction problem 1 min f (y) = (|∇y|2 + |y|2 ) − f˜ y dx + g |y| ds over y ∈ H 1 (), (0.0.22) 2 where is a bounded domain with boundary . This is an unconstrained minimization problem with X = H 1 () and no control variables. Since the functional is not continuously differentiable, the necessary optimality condition fy (y) = 0 is not applicable. However, with the aid of a generalized Lagrange multiplier theory a necessary optimality condition can be written as ∂y = g λ on , ∂ν |λ(x)| ≤ 1 and λ(x)y(x) = | y(x)| a.e. in .
− y + y = f˜,
(0.0.23)
From a Lagrange multiplier perspective, problems with L1 -type cost functionals as in (0.0.22) and box-constrained problems are dual to each other. So it comes as no surprise that again semismooth Newton methods provide an efficient technique for solving (0.0.22) or (0.0.23). We shall analyze them and provide a theoretical basis for their numerical efficiency. This monograph also contains the analysis of optimization problems constrained by partial differential equations which are singular in the sense that the state variable cannot be differentiated with respect to the control. This, however, does not preclude that the cost
i
i i
i
i
i
i
xvi
ItoKunisc 2008/6/12 page xvi i
Preface
functional is differentiable with respect to the control and that an optimality principle can be derived. In terms of shape optimization problems this means that the cost functional is shape differentiable while the state is not differentiable with respect to the shape. In summary, Lagrange multiplier theory provides a tool for the analysis of general constrained optimization problems with cost functionals which are not necessarily C 1 and with state equations which are in some sense singular. It also leads to a theoretical basis for developing efficient and powerful iterative methods for solving such problems. The purpose of this monograph is to provide a rather thorough analysis of Lagrange multiplier theory and to show its impact on the development of numerical algorithms for problems which are posed in a function space setting. Let us give a short description of the book for those readers who do not intend to read it by consecutive chapters. Chapter 1 provides a variety of tools to establish existence of Lagrange multipliers and is called upon in all the following chapters. Here, as in other chapters, we do not attempt to give the most general results, nor do we strive for covering the complete literature. Chapter 2 is devoted to the sensitivity analysis of abstract constrained nonlinear programming problems and it essentially stands for itself. This chapter is of great importance, addressing continuity, Lipschitz continuity, and differentiability of the solutions to optimization and optimal control problems with respect to parameters that appear in the problem formulation. Such results are not only of theoretical but also of practical importance. The sensitivity equations have been a starting point for the development of algorithmic concepts for decades. Nevertheless, readers who are not interested in this topic at first may skip this chapter without missing technical results which might be needed for later chapters. Chapters 3, 5, and 6 form a unit which is devoted to smooth optimization problems. Chapter 3 covers first order augmented Lagrangian methods for optimization problems with equality and inequality constraints. Here as in the remainder of the book, the inequality constraints that we have in mind typically represent partial differential equations. In fact, during the period in which this monograph was written, the terminology “PDE-constrained optimization” emerged. Inverse problems formulated as regularized least squares problems and optimal control problems for (partial) differential equations are primary examples for the theories that are discussed here. Chapters 5 and 6 are devoted to second order iterative solution methods for equality-constrained problems. Again the equality constraints represent partial differential equations. This naturally gives rise to the following situation: The variables with respect to which the optimization is carried out can be classified into two groups. One group contains the state variables of the differential equations and the other group consists of variables which represent the control or input variables for optimal control problems, or coefficients in parameter estimation problems. If the state variables are considered as functions of the independent controls, inputs, or coefficients, and the cost functional in the optimization problem is only considered as a functional of the latter, then this is referred to as the reduced formulation. Applying a second order method to the reduced functional we arrive at the Newton method for optimization problems with partial differential equations as constraints. If both state and control variables are kept as independent variables and the optimality system involving primal and adjoint variables, which are the Lagrange multipliers corresponding to the PDE-constraints, is derived, we arrive at the sequential quadratic programming (SQP) technique: It essentially consists of applying a Newton algorithm to the first order necessary optimality conditions. The Newton method for the reduced formulation and the SQP technique are the focus of Chapter 5. Chapter 6
i
i i
i
i
i
i
Preface
ItoKunisc 2008/6/12 page xvii i
xvii
is devoted to second order augmented Lagrangian techniques which are closely related, as we shall see, to SQP methods. Here the equation constraint is augmented in a penalty term, which has the effect of locally convexifying the optimization problem. Since augmented Lagrangians also involve Lagrange multipliers, there is, however, no necessity to let the penalty parameter tend to infinity and, in fact, we do not suggest doing so. A second larger unit is formed by Chapters 4, 7, 8, and 9. Nonsmoothness, primal-dual active set strategy, and semismooth Newton methods are the keywords which characterize the contents of these chapters. Chapter 4 is essentially a recapture of concepts from convex analysis in a format that is used in the remaining chapters. A key result is the formulation of differential inclusions which arise in optimality systems by means of nondifferentiable equations which are derived from Yosida–Moreau approximations and which will serve as the basis for the primal-dual active set strategy. Chapter 7 is devoted to the primaldual active set strategy and its global convergence properties for unilaterally and bilaterally constrained problems. The local analysis of the primal-dual active set strategy is achieved in the framework of semismooth Newton methods in Chapter 8. It contains the notion of Newton derivative and establishes local superlinear convergence of the Newton method for problems which do not satisfy the classical sufficient conditions for local quadratic convergence. Two important classes of applications of semismooth Newton methods are considered in Chapter 9: image restoration and deconvolution problems regularized by the bounded variation (BV) functional and friction and contact problems in elasticity. Chapter 10 is devoted to a Lagrangian treatment of parabolic variational inequalities in unbounded domains as they arise in the Black–Scholes equation, for example. It contains the use of monotone techniques for analyzing parabolic systems without relying on compactness assumptions in a Gelfand-triple framework. In Chapter 11 we provide a calculus for obtaining the shape derivative of the cost functional in shape optimization problems which bypasses the need for using the shape derivative of the state variables of the partial differential equations. It makes use of the expansion technique that is proposed in Chapters 1 and 5 for weakly singular optimal control problems, and of the assumption that an appropriately defined adjoint equation admits a solution. This provides a versatile technique for evaluating the shape derivative of the cost functional using Lagrange multiplier techniques. There are many additional topics which would fit under the title of this monograph which, however, we chose not to include. In particular, issues of discretization, convergence, and rate of convergence are not discussed. Here the issue of proper discretization of adjoint equations consistent with the discretization of the primal equation and the consistent time integration of the adjoint equations must be mentioned. We do not enter into the discussion of whether to discretize an infinite-dimensional nonlinear programming problem first and then to decide on an iterative algorithm to solve the finite-dimensional problems, or the other way around, consisting of devising an optimization algorithm for the infinite-dimensional problem which is subsequently discretized. It is desirable to choose a discretization and an iterative optimization strategy in such a manner that these two approaches commute. Discontinuous Galerkin methods are well suited for this purpose; see, e.g., [BeMeVe]. Another important area which is not in the focus of this monograph is the efficient solution of those large scale linear systems which arise in optimization algorithms. We refer the reader to, e.g., [BGHW, BGHKW], and the literature cited there. The solution of large scale time-dependent optimal control problems involving the coupled system of primal and
i
i i
i
i
i
i
xviii
ItoKunisc 2008/6/12 page xviii i
Preface
adjoint equations, which need to be solved in opposite directions with respect to time, still offers a significant challenge, despite the advances that were made with multiple shooting, receding horizon, and time-domain decomposition techniques. From the point of view of optimization theory there are several topics as well into which one could expand. These include globalization strategies, like trust region methods, exact penalty methods, quasiNewton methods, and a more abstract Lagrange multiplier theory than that presented in Chapter 1. As a final comment we stress that for a full treatment of a variational problem in function spaces, both its infinite-dimensional analysis as well as its proper discretization and the relation between the two are indispensable. Proceeding from an infinite-dimensional problem directly to its disretization without such a treatment, important issues can be missed. For instance discretization without a well-posed analysis may result in the use of inappropriate inner products, which may lead to unnecessary ill-conditioning, which entails unnecessary preconditioning. Inconsiderate discretization may also result in the loss of structural properties, as for instance symmetry properties.
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 1 i
Chapter 1
Existence of Lagrange Multipliers
1.1
Problem statement and generalities
We consider the constrained optimization problem min f (x) subject to x ∈ C and g(x) ∈ K,
(1.1.1)
where f is a real-valued functional defined on a real Banach space X; further C is a closed convex subset of X, g is a mapping from X into the real Banach space Z, and K is a closed convex cone in Z. Throughout the monograph it is understood that the vertex of a cone coincides with the origin. Further, in this chapter it is assumed that (1.1.1) admits a solution x ∗ . Here we refer to x ∗ as solution if it is feasible, i.e., x ∗ satisfies the constraints in (1.1.1), and f has a local minimum at x ∗ . It is assumed that f is Fréchet differentiable at x ∗ , and (1.1.2) g is continuously Fréchet differentiable at x ∗ with Fréchet derivatives denoted by f (x ∗ ) and g (x ∗ ), respectively. The set of feasible points for (1.1.1) will be denoted by M = {x ∈ C : g(x) ∈ K}. Moreover, the following notation will be used. The topological duals of X and Z are denoted by X ∗ and Z ∗ . For the subset A of X the polar or dual cone is defined by A+ = {x ∗ ∈ X ∗ : x ∗ , a ≤ 0 for all a ∈ A}, where ·, · (= ·, · X∗ ,X ) denotes the duality pairing between X ∗ and X. Further for x ∈ X the conical hull of C \ {x} is given by C(x) = {λ(c − x) : c ∈ C, λ ≥ 0}, and K(z) with z ∈ Z is defined analogously. The Lagrange functional L : X × Z ∗ → R associated to (1.1.1) is defined by L(x, λ) = f (x) + λ, g(x) Z∗ ,Z . 1
i
i i
i
i
i
i
2
ItoKunisc 2008/6/12 page 2 i
Chapter 1. Existence of Lagrange Multipliers
Definition 1.1. An element λ∗ ∈ Z ∗ is called a Lagrange multiplier for (1.1.1) at the solution x ∗ if λ∗ ∈ K + , λ∗ , g(x ∗ ) = 0, and f (x ∗ ) + λ∗ ◦ g (x ∗ ) ∈ −C(x ∗ )+ . Lagrange multipliers are of fundamental importance for expressing necessary as well as sufficient optimality conditions for (1.1.1) and for the development of algorithms to solve (1.1.1). If C = X, then the inclusion in Definition 1.1 results in the equation f (x ∗ ) + λ∗ ◦ g (x ∗ ) = 0. For the existence analysis of Lagrange multipliers one makes use of the following tangent cones which approximate the feasible set M at x ∗ : 1 ∗ ∗ + T (M, x ) = x ∈ X : x = lim (xn − x ), tn → 0 , xn ∈ M , n→∞ tn ∗ L(M, x ) = {x ∈ X : x ∈ C(x ∗ ) and g (x ∗ )x ∈ K(g(x ∗ ))}. The cone T (M, x ∗ ) is called the sequential tangent cone (or Bouligand cone) and L(M, x ∗ ) the linearizing cone of M at x ∗ . Proposition 1.2. If x ∗ is a solution to (1.1.1), then f (x ∗ )x ≥ 0 for all x ∈ T (M, x ∗ ). Proof. Let x ∈ T (M, x ∗ ). Then there exist sequences {xn }∞ n=1 in M and {tn } with limn→∞ tn = 0 such that x = limn→∞ t1n (xn − x ∗ ). By the definition of Fréchet differentiability there exists a sequence {rn } in Z such that f (x ∗ )x = lim
1
n→∞ tn
and limn→∞
1 r |xn −x ∗ | n
f (x ∗ )(xn − x ∗ ) = lim
1
n→∞ tn
(f (xn ) − f (x ∗ ) + rn )
= 0. By optimality of x ∗ we have f (x ∗ )x ≥ lim |x| n→∞
1 rn = 0 |xn − x ∗ |
and the result follows. We now briefly outline the main steps involved in the abstract approach to prove the existence of Lagrange multipliers. For this purpose we assume that C = X, K = {0} so that the set of feasible points is described by M = {x : g(x) = 0}. We assume that g (x ∗ ) : X → Z is surjective. Then by Lyusternik’s theorem [Ja, We] we have L(M, x ∗ ) ⊂ T (M, x ∗ ).
(1.1.3)
Consider the convex cone B = {(f (x ∗ )x + r, g (x ∗ )x) : r ≥ 0, x ∈ X} ⊂ R × Z.
i
i i
i
i
i
i
1.2. A generalized open mapping theorem
ItoKunisc 2008/6/12 page 3 i
3
Observe that (0, 0) ∈ B and that due to Proposition 1.2 and (1.1.3) the origin (0, 0) is a boundary point of B. Since g (x ∗ ) is surjective, B has nonempty interior and hence by the Eidelheit separation theorem that we shall recall below there exists a closed hyperplane in R × Z which supports B at (0, 0), i.e., there exists 0 = (α, λ∗ ) ∈ R × Z ∗ such that α(f (x ∗ )x + r) + λ∗ , g (x ∗ )x ≥ 0 for all (r, x) ∈ R+ × X. Setting x = 0 we have α ≥ 0. If α = 0, then λ∗ g (x ∗ ) = 0, which would imply λ∗ = 0, which is impossible. Consequently α > 0 and without loss of generality we can assume α = 1. Hence λ∗ is a Lagrange multiplier. For the general case of equality and inequality constraints the application of Lyusternik’s theorem is replaced by stability results for solutions of systems of inequalities. The existence of a separating hyperplane will follow from a generalized open mapping theorem, which is covered in the following section. The separation theorem announced above is presented next. Theorem 1.3. Let K1 and K2 be nontrivial, convex, and disjoint subsets of a normed linear space X . If K1 is closed and K2 is compact, then there exists a closed hyperplane strictly separating K1 and K2 , i.e., there exist x ∗ ∈ X , β ∈ R, and > 0 such that x ∗ , x ≤ β − for all x ∈ K1 and x ∗ , x ≥ β + for all x ∈ K2 . If K1 is open and K2 is arbitrary, then these inequalities still hold with = 0. Let us give a brief description of the sections of this chapter. Section 1.2 contains the open mapping theorem already alluded to above. Regularity properties which guarantee the existence of a Lagrange multiplier in general nonlinear optimal programming problems in infinite dimensions are analyzed in section 1.3 and applications to parameter estimation and optimal control problems are described in section 1.4. Section 1.5 is devoted to the derivation of a first order optimality system for a class of weakly singular optimal control problems and to several applications which are covered by this class. Finally in section 1.6 several further alternatives for deriving optimality systems are presented in an informal manner.
1.2 A generalized open mapping theorem Throughout this section T denotes a bounded linear operator from X to Z. As in the previous section C stands for a closed convex set in X, and K stands for a closed convex cone in Z. For Theorem 1.4 below it is sufficient that K is a closed convex set. For ρ > 0 we set Xρ = {x : |x| ≤ ρ} and Zρ is defined analogously. Recall that by the open mapping theorem, surjectivity of T implies that T maps open sets of X onto open sets of Z. As a consequence T X1 contains Zρ for some ρ > 0. The following theorem generalizes this result. Theorem 1.4. Let x¯ ∈ C and y¯ ∈ K, where K is a closed convex set. Then the following statements are equivalent: (a) Z = T C(x) ¯ − K(y), ¯
i
i i
i
i
i
i
4
ItoKunisc 2008/6/12 page 4 i
Chapter 1. Existence of Lagrange Multipliers (b) there exists ρ > 0 such that Zρ ⊂ T ((C − {x}) ¯ ∩ X1 ) − (K − {y}) ¯ ∩ Z1 ,
where C(x) ¯ = {λ(x − x) ¯ : λ ≥ 0, x ∈ C} and K(y) ¯ = {k − λy¯ : λ ≥ 0, k ∈ K}. Proof. Clearly (b) implies (a). To prove the converse we first show that n((C − x) ¯ ∩ X1 ). C(x) ¯ =
(1.2.1)
n∈N
The inclusion ⊃ is obvious and hence it suffices to prove that C(x) ¯ is contained in the set on the right-hand side of (1.2.1). Let x ∈ C(x). ¯ Then x = λy with λ ≥ 0 and y ∈ C − x. ¯ Without loss of generality we can assume that λ > 0 and |y| > 1. Convexity of C implies 1 1 that |y| y ∈ (C − x) ¯ ∩ X1 . Let n ∈ N be such that n ≥ λ|y|. Then x = (λ|y|) |y| y ∈ n((C − x) ¯ ∩ X1 ), and (1.2.1) is verified. For α > 0 let Aα = αT ((C − x) ¯ ∩ X1 − (K − y) ¯ ∩ Z1 ). We will show that 0 ∈ int A1 . In fact, from (1.2.1) and the analogous equality with C replaced by K it follows from (a) that nT ((C − x) ¯ ∩ X1 ) ∪ (−n)((K − y) ¯ ∩ Z1 ). Z= n∈N
n∈N
This implies hat Z=
n∈N
An =
An .
(1.2.2)
n∈N
Thus the complete space Z is the countable union of the closed sets An . By the Baire category theorem there exists some m ∈ N such that int Am = ∅. Let a ∈ int Am . By (1.2.2) there exists k ∈ N with −a ∈ Ak . Using Aα = αA1 this implies that − mk a ∈ Am . It follows that the half-open interval {λ(− mk a)+(1−λ)a : 0 ≤ λ < 1} belongs to int Am and consequently 0 ∈ int Am = m int A1 . Thus we have 0 ∈ int A1 .
(1.2.3)
Hence for some ρ > 0 Zρ ⊂
1 A1 = A 1 ⊂ A 1 + Z ρ2 , 2 2 2
and consequently for all i = 0, 1, . . . i i 1 1 1 Z( )i ρ = Zρ ⊂ (A 1 + Z ρ2 ) = A( 1 )i+1 + Z( 1 )i+1 ρ . 2 2 2 2 2 2
(1.2.4)
(1.2.5)
Now choose y ∈ Zρ . By (1.2.4) there exist x1 ∈ (C − x) ¯ ∩ X1 , y1 ∈ (K − y) ¯ ∩ Z1 , and r1 ∈ Z ρ2 such that y = T ( 12 x1 ) − 12 y1 + r1 . Applying (1.2.5) with i = 1 to r1 implies the existence of x2 ∈ (C − x) ¯ ∩ X1 , ¯ ∩ Z1 , and r2 ∈ Z( 1 )2 ρ such that y2 ∈ (K − y) 2
y=T
1 x1 + 2
2 1 1 1 y1 + y2 + r 2 . x2 − 2 2 2
i
i i
i
i
i
i
1.3. Regularity and existence of Lagrange multipliers
ItoKunisc 2008/6/12 page 5 i
5
Repeated application of this argument implies that y = T un − vn + rn , where n i 1 un = xi with xi ∈ (C − x) ¯ ∩ X1 , 2 i=1 n i 1 vn = yi with yi ∈ (K − y) ¯ ∩ Y1 , 2 i=1 and rn ∈ Y( 1 )n ρ . Observe that 2
1 un = x1 + · · · + 2
n n i 1 1 0 ∈ (C − x) ¯ ∩ X1 xn + 1 − 2 2 i=1
and i n 1 1 |un − un+m | ≤ ≤ . 2 2 i=n+1 n+m
Consequently {un }∞ ¯ ∩ X1 such n=1 is a Cauchy sequence and there exists some x ∈ (C − x) ¯ ∩ Z1 such that limn→∞ vn = v. that limn→∞ un = x. Similarly there exists v ∈ (K − y) Moreover limn→∞ rn = 0 and continuity of T implies that y = T x − v.
1.3
Regularity and existence of Lagrange multipliers
In this section existence of a Lagrange multiplier for the optimization problem (1.1.1) will be established using a regular point condition which was proposed in the publications of Robinson for sensitivity analysis of generalized equations and later by Kurcyusz, Zowe, and Maurer for the existence of Lagrange multipliers. Definition 1.5. The element x¯ ∈ M is called regular (or alternatively x¯ satisfies the regular point condition) if 0 ∈ int {g (x)(C ¯ − x) ¯ − K + g(x)}. ¯
(1.3.1)
If (1.3.1) holds, then clearly 0 ∈ int {g (x)C( ¯ x) ¯ − K(g(x))}. ¯
(1.3.2)
¯ x) ¯ − K(g(x)) ¯ is a cone. Consequently (1.3.2) implies that Note that g (x)C( g (x)C( ¯ x) ¯ − K(g(x)) ¯ = Z.
(1.3.3)
Finally (1.3.3) implies (1.3.1) by Theorem 1.4. Thus, the three conditions (1.3.1)–(1.3.3) are equivalent. Condition (1.3.1) is used in the work of Robinson (see, e.g., [Ro2]) on the stability of solutions to systems of inequalities, and (1.3.3) is the regularity condition employed in [MaZo, ZoKu] to guarantee the existence of Lagrange multipliers.
i
i i
i
i
i
i
6
ItoKunisc 2008/6/12 page 6 i
Chapter 1. Existence of Lagrange Multipliers
Remark 1.3.1. If C = X, then the so-called Slater condition g (x)h ¯ ∈ int K(g(x)) ¯ for some h ∈ X implies (1.3.1). In case C = X = Rn , Z = Rm , K = {y ∈ Rm : yi = 0 for i = 1, . . . , k, and yi ≤ 0 for i = k + 1, . . . , m}, g = (g1 , . . . , gk , gk+1 , . . . , gm ), the constraint g(x) ∈ K amounts to k equality and m − k inequality constraints, and the regularity condition (1.3.3) becomes ⎧ ¯ ki=1 are linearly independent and the gradients {gi (x)} ⎪ ⎪ ⎨ there exists x ∈ X such that (1.3.4) g (x)x ¯ = 0 for i = 1, . . . , k, ⎪ ⎪ ⎩ i ¯ < 0 for i = k + 1, . . . , m with gi (x) ¯ = 0. gi (x)x This regularity condition is referred to as the Mangasarian–Fromowitz constraint qualification. Other variants of conditions which imply existence of Lagrange multipliers are named after F. John, Karush–Kuhn–Tucker (KKT), and Arrow, Hurwicz, Uzawa. Theorem 1.6. If the solution x ∗ of (1.1.1) is regular, then there exists an associated Lagrange multiplier λ∗ ∈ Z ∗ . Proof. By Proposition 1.2 we have f (x ∗ )x ≥ 0 for all x ∈ T (M, x ∗ ). From Theorem 1 and Corollary 2 in [We], which can be viewed as generalizations of Lyusternik’s theorem, it follows that L(M, x ∗ ) ⊂ T (M, x ∗ ) and hence f (x ∗ )x ≥ 0 for all x ∈ L(M, x ∗ ).
(1.3.5)
We define the set B = {(f (x ∗ )x + r, g (x ∗ )x − y) : r ≥ 0, x ∈ C(x ∗ ), y ∈ K(g(x ∗ ))} ⊂ R × Z. This is a convex cone containing the origin (0, 0) ∈ R × Z. Due to (1.3.5) the origin is a boundary point of B. By the regular point condition and Theorem 1.4 with T = g (x ∗ ) and y¯ = g(x ∗ ) there exists ρ > 0 such that {(α, y) : α ≥ max{f (x ∗ )x : x ∈ (C − x ∗ ) ∩ X1 } and y ∈ Zρ } ⊂ B, and hence int B = ∅. Consequently there exists a hyperplane in R × Z which supports B at (0, 0), i.e., there exists (α, λ∗ ) = (0, 0) ∈ R × Z ∗ such that α(f (x ∗ )x + r) + λ∗ (g (x ∗ )x − y) ≥ 0 for all x ∈ C(x ∗ ), y ∈ K(g(x ∗ )), r ≥ 0.
(1.3.6)
Setting (r, x) = (0, 0) this implies that λ∗ , y ≤ 0 for all y ∈ K(g(x ∗ )) and consequently λ∗ ∈ K ∗
and
λ∗ g(x ∗ ) = 0.
(1.3.7)
i
i i
i
i
i
i
1.3. Regularity and existence of Lagrange multipliers
ItoKunisc 2008/6/12 page 7 i
7
The choice (x, y) = (0, 0) in (1.3.6) implies α ≥ 0. If α = 0, then due to regularity of x ∗ we have λ∗ = 0, which is impossible. Consequently α > 0, and without loss of generality we can assume that α = 1. Finally with (r, y) = (0, 0), inequality (1.3.6) implies f (x ∗ ) + λ∗ g (x ∗ ) ∈ −C(x ∗ )+ . This concludes the proof. We close this subsection with two archetypical problems. Equality-constrained problem. We consider min f (x) subject to g(x) = 0.
(1.3.8)
Let x ∗ denote a local solution at which (1.1.2) is satisfied and such that g (x ∗ ) : X → Z is surjective. Then there exists λ∗ ∈ Z ∗ such that the KKT condition f (x ∗ ) + λ∗ g (x ∗ ) = 0 in X∗ holds. Surjectivity of g (x ∗ ) is of course only a sufficient, not a necessary, condition for existence of a Lagrangian. This topic is taken up in Example 1.12 and below. Inequality-constrained problem. This is the inequality-constrained problem with the so-called unilateral box, or simple constraints. We suppose that Z = L2 () with a domain in Rn and consider min f (x) (1.3.9) subject to g(x(s)) ≤ 0 for a.e. s ∈ . Setting C = X and K = {v ∈ L2 () : v ≤ 0}, this problem becomes a special case of (1.1.1). Let x ∗ denote a local solution at which (1.1.2) is satisfied and such that g (x ∗ ) : X → Z is surjective. Then there exists λ∗ ∈ L2 () such that the KKT conditions f (x ∗ ) + λ∗ g (x ∗ ) = 0 in X∗ , λ∗ (s) ≥ 0, g(x ∗ (s)) ≤ 0, λ∗ (s)g(x ∗ (s)) = 0 for a.e. s ∈
(1.3.10)
hold. Again surjectivity of g (x ∗ ) is a sufficient, but not a necessary, condition for (1.3.10) to hold. For later use let us consider the case where the functionals f (x ∗ ) ∈ X∗ and λ∗ ∈ Z ∗ are represented by duality pairings, i.e., f (x ∗ )(x) = JX f (x ∗ ), x X∗ ,X ,
λ∗ (z) = JZ λ∗ , z Z∗ ,Z
for x ∈ X and z ∈ Z. Here JX is the canonical isomorphism from L(X, R) into the space of representations of continuous linear functionals on X with respect to the ·, · X∗ ,X duality pairing, and JZ is defined analogously. In finite dimensions these isomorphisms are transpositions. With this notation f (x ∗ ) + λ∗ g (x ∗ ) = 0 can be expressed as JX f (x ∗ ) + g (x ∗ )∗ JZ λ∗ = 0. Henceforth the notation of the isomorphisms will be omitted.
i
i i
i
i
i
i
8
ItoKunisc 2008/6/12 page 8 i
Chapter 1. Existence of Lagrange Multipliers
1.4 Applications This section is devoted to a discussion of examples in which the general existence result of the previous section is applicable, as well as to other examples in which it fails. Throughout denotes a bounded domain in Rn with Lipschitz continuous boundary ∂. We use standard Hilbert space notation as introduced in [Ad], for example. Differently from the previous sections we shall use J to denote the cost functional and f for inhomogeneities in equality constraints representing partial differential equations. Moreover the constraints g(x) ∈ K from the previous sections will be split into equality constraints, denoted by e(x) = 0, and inequality constraints, g(x) ∈ K, with K a nontrivial cone. Example 1.7. Consider the unilateral obstacle problem ⎧
2 1 ⎪
⎪ ∇y dx − fydx min J (y) = ⎨ 2 ⎪ ⎪ ⎩ over y ∈ H01 () and y(x) ≤ ψ(x) for a.e. x ∈ ,
(1.4.1)
where f ∈ L2 () and ψ ∈ H01 (). This is a special case of (1.1.1) with X = Z = H01 (), K = {ϕ ∈ H01 () : ϕ(x) ≤ 0 for a.e. x ∈ }, C = X, and g(y) = y − ψ. It is well known and simple to argue that (1.4.1) admits a unique solution y ∗ ∈ H01 (). Moreover, since g is the identity, every element of X satisfies the regular point condition. Hence by Theorem 1.6 there exists λ∗ ∈ H01 ()∗ such that ∗ λ , ϕ H −1 ,H 1 ≤ 0 for all ϕ ∈ K, ∗ ∗ 0 0 λ , y − ψ H −1 ,H 1 = 0, 0
0
− y ∗ + λ∗ = f in H −1 , where H −1 = H01 ()∗ . Under well-known regularity assumptions on ψ and ∂, cf. [Fr, IK1], for example, λ∗ ∈ L2 (). This extra smoothness of the Lagrange multiplier does not follow from the general multiplier theory of the previous section. The following result is useful to establish the regular point condition in diverse optimal control and parameter estimation problems. We require some notation. Let L be a closed convex cone in the real Hilbert space U which induces an ordering denoted according to u ≥ 0 if u ∈ L. Let q ∗ ∈ U ∗ be a nontrivial bounded linear functional on U , with π : U → ker q ∗ the orthogonal projection. For ϕ ∈ U, μ ∈ R, and γ ∈ R+ define g : U → Z := U × R × R by
2 g(u) = u − ϕ, u − γ 2 , q ∗ (u) − μ ,
where | · | denotes the norm in U , and put K = L × R− × {0} ⊂ Z. Proposition 1.8. Assume that [ker q ∗ ]⊥ ∩ L = {0} and let h0 ∈ [ker q ∗ ]⊥ ∩ L with
2
2 q ∗ (h0 ) = 1. If q ∗ (L) ⊂ R+ , q ∗ (ϕ) < μ, and π(ϕ) + μ2 h0 < γ 2 ; then the set M = {u ∈ U : g(u) ∈ K}
i
i i
i
i
i
i
1.4. Applications
ItoKunisc 2008/6/12 page 9 i
9
is nonempty and every element u ∈ M is regular, i.e., 0 ∈ int g(u) + g (u)U − K for each u ∈ M.
(1.4.2)
Proof. Note that U = ker q ∗ ⊕ span {h0 } and that the orthogonal projection onto [ker q ∗ ]⊥ is given by q ∗ (u)h0 for u ∈ U . To show that M is nonempty, we define uˆ = ϕ + μ − q ∗ (ϕ) h0 . Observe that
uˆ − ϕ = μ − q ∗ (ϕ) h0 ∈ L,
2
uˆ = π(ϕ) 2 + μ2 h0 2 < γ ,
ˆ = μ. Thus uˆ ∈ M. Next let u ∈ M be arbitrary. We have to verify that and q ∗ (u) (1.4.3) 0 ∈ int u − ϕ + h − L, |u|2 − γ 2 + 2(u, h) − R− , q ∗ (h) : h ∈ U ⊂ Z, where (·, ·) denotes the inner product in U . Put
2
2 γ 2 − π(ϕ) − μ2 h0 μ − q ∗ (ϕ)
, , δ = min q ∗ +1 1 + 2γ + 2γ h0
and define B = (u, ˜ r˜ , s˜ ) ∈ Z : (u, ˜ r˜ , s˜ ) Z ≤ δ . Without loss of generality we endow the product space with the supremum norm in this proof. We shall show that
2 (1.4.4) B ⊂ u − ϕ + h1 + σ h0 − L, u − γ 2 + 2(u, h1 + σ h0 ) − R− , ∗ ∗ σ q (h0 ) : h1 ∈ ker q , σ ∈ R , which implies (1.4.3). Let (u, ˜ r˜ , s˜ ) ∈ B be chosen arbitrarily. We put σ = s˜ and verify that there exists (h1 , l, r − ) ∈ ker q ∗ × L × R− such that
2 ˜ r˜ ). (1.4.5) u − ϕ + h1 + s˜ h0 − l, u − γ 2 + 2(u, h1 + s˜ h0 ) − r − = (u, This will imply the claim. Observe that by the choice of δ we find μ − q ∗ (ϕ) − q (x) ˜ + s˜ ≥ 0. Hence l is defined by ˜ + s˜ h0 ∈ L. l = μ − q ∗ (ϕ) − q ∗ (u) ∗
˜ we obtain equality in the first coordinate of (1.4.5). For Choosing h1 = π(φ − u + u) the second coordinate in (1.4.5) we observe that
2
u − γ 2 + 2(u, h1 + s˜ h0 )
2
2 = πu + μ2 h0 − γ 2 + 2(u, π(ϕ − u + u) ˜ + s˜ h0 )
2
2 2
2
2 2 2 2 ˜ + s˜ h0 ≤ πu + μ h0 − γ + π u + π ϕ − 2 π u + 2γ |u|
2
2
≤ μ2 h0 − γ 2 + πϕ + 2δγ 1 + h0 ≤ −δ by the definition of δ. Hence there exists r − ∈ R− such that equality holds in the second coordinate of (1.4.5).
i
i i
i
i
i
i
10
ItoKunisc 2008/6/12 page 10 i
Chapter 1. Existence of Lagrange Multipliers
Remark 1.4.1. If the constraint q ∗ (u) = μ is not present in the definition of M, then the
conclusion of Proposition 1.8 holds provided that ϕ < γ . Example 1.9. We consider a least squares formulation for the estimation of the potential c in − y + cy = f in , (1.4.6) y = 0 on ∂ from an observation z ∈ L2 (), where f ∈ L2 () and ⊂ Rn , with n ≤ 4. For this purpose we set
M = c ∈ L2 () : c(x) ≥ 0, c ≤ γ for some γ > 0 and consider ⎧
2 α 2 1 ⎪ ⎨min J (c) = y(c) − z + c 2 2 ⎪ ⎩ subject to c ∈ M and y(c) solution to (4.6).
(1.4.7)
The Lax–Milgram theorem implies the existence of a variational solution y(c) ∈ H01 () for every c ∈ M. Here we use the fact that H 1 () ⊂ L4 () and cy ∈ L4/3 () for c ∈ L2 (), y ∈ H 1 (), for n ≤ 4. Proposition 1.8 with L = {c ∈ L2 () : c(x) ≥ 0} and
2 g(c) = c, c − γ 2 is applicable and implies that every element of M is a regular point. Using subsequential weak limit arguments one can argue the existence of a solution c∗ to (1.4.7). To apply Theorem 1.6 Fréchet differentiability of c → J (c) at c∗ must be verified. This will follow from Fréchet differentiability of c → y(c) at every c ∈ M, which in turn can be argued by means of the implicit function theorem, for example. The Fréchet derivative of c → y(c) at c ∈ M in direction h ∈ L2 () denoted by y (c)h satisfies − y (c)h + cy (c)h = −hy(c) in , (1.4.8) on ∂. y (c)h = 0 With these preliminaries we obtain a Lagrange multiplier (μ1 , μ2 ) ∈ −L × R+ such that the following first order necessary optimality system holds: ⎧ ∗ y(c ) − z, y (c∗ )h L2 + α (c∗ , h) − (μ1 , h) + 2μ2 (c∗ , h) = 0 ⎪ ⎪ ⎪ ⎨ for all h ∈ L2 (), (1.4.9) 2 ∗ ∗ −(μ1 , c ) + μ2 c − γ = 0, ⎪ ⎪ ⎪ ⎩ (μ1 , μ2 ) ∈ L × R+ . The first equation in (1.4.9) can be simplified by introducing p(c∗ ) as the variational solution to the adjoint equation given by − p + c∗ p = − (y(c∗ ) − z) in , (1.4.10) p=0 on ∂.
i
i i
i
i
i
i
1.4. Applications
ItoKunisc 2008/6/12 page 11 i
11
Using (1.4.8) and (1.4.10) in (1.4.9) we obtain the optimality system ⎧ ∗ p(c ) y(c∗ ) + αc∗ − μ1 + 2μ2 c∗ = 0, ⎪ ⎪ ⎨
2 (μ1 , μ2 ) ∈ −L × R+ , c∗ , c∗ − γ 2 ∈ L × R− , ⎪ ⎪ ⎩(μ1 , c∗ ) = 0, μ2 ( c∗ 2 − γ 2 = 0.
(1.4.11)
Example 1.10. We revisit Example 1.9 but this time the state variable is considered as an independent variable. This results in the following problem with equality and inequality constraints: ⎧ α 1 ⎪ ⎨min J (y, c) = |y − z|2 + |c|2 2 2 (1.4.12) ⎪ ⎩ − subject to e(y, c) = 0 and g(c) ∈ L × R , where g is as in Example 1.9 and e : H01 () × L2 () → H −1 () is defined by e(y, c) = − y + cy − f. Note that cy ∈ L () ⊂ H −1 () for c ∈ L2 (), y ∈ H 1 () if n ≤ 4. Let (c∗ , y ∗ ) ∈ L2 () × H01 () denote a solution of (1.4.12). Fréchet differentiability of J, e, and g is obvious for this formulation with y as independent variable. The regular point condition now has to be established for the constraint e ∈ K = {0} × L × R− ⊂ H −1 () × L2 () × R. g 4 3
The second and third components are treated as in Example 1.9 by means of Proposition 1.8 and the first coordinate is incorporated by an existence argument for (1.4.6). The details are left to the reader. Consequently there exists a Lagrange multiplier (p, μ1 μ2 ) ∈ H01 () × L × R+ such that the derivative of L(y, c, p, μ1 , μ2 )
2 = J (y, c) + e(y, c), p H −1 ,H01 + (μ1 , −c) + μ2 c∗ − γ 2
with respect to (y, c) is zero at (y ∗ , c∗ , p, μ1 , μ2 ). This implies that p is the variational solution in H01 () to − p + c∗ p = −(y ∗ − z) in , p=0 on ∂ and py ∗ + αc∗ − μ1 + 2μ2 c∗ = 0. In addition, of course,
2 (μ1 , μ2 ) ∈ −L × R+ , c∗ , c∗ − γ 2 ∈ L × R− ,
2 (μ1 , c∗ ) = 0, μ2 c∗ − γ = 0.
i
i i
i
i
i
i
12
ItoKunisc 2008/6/12 page 12 i
Chapter 1. Existence of Lagrange Multipliers
Thus the first order optimality condition derived in Example 1.9 above and the present one coincide, and the adjoint variable of Example 1.9 becomes the Lagrange multiplier of the equality constraint e(y, c) = 0. Example 1.11. We consider a least squares formulation for the estimation of the diffusion coefficient a in −(ayx )x + cy = f in = (0, 1), (1.4.13) y(0) = y(1) = 0, where f ∈ L2 (0, 1) and c ∈ L2 (), c ≥ 0, are fixed. We assume that a is known outside of an interval I = (β, γ ) with 0 < β < γ < 1 and that it coincides there with the values of a known function ν ∈ H 1 (0, 1), which satisfies min {ν(x) : x ∈ (0, 1)} > 0. The set of admissible parameters is then defined by
M = a ∈ H 1 (I ) : a ≥ ν, a − aˆ H 1 (I ) ≤ γ , a(β) = ν(β), γ a dx = m . a(γ ) = ν(γ ), and β
Here aˆ ∈ H 1 (I ) is a fixed reference parameter with a(β) ˆ = ν(β), a(γ ˆ ) = ν(γ ), and γ m = β aˆ dx. Recall that H 1 (I ) ⊂ C(I¯) and hence a(β) and a(γ ) are well defined. The coefficient a appearing in (1.4.13) coincides with ν on the complement of I and with some element of M on I . Consider the least squares formulation ⎧
2
2 ⎨min 12 y(a) − z L2 (0,1) + α2 a H 1 (I ) (1.4.14) ⎩ subject to a ∈ M and y(a) solution to (1.4.13), where z ∈ L2 (0, 1) is given. It is simple to argue the existence of a solution a ∗ to (1.4.14). Note that minimizing over M implies that the mean of the coefficient a is known. This, together with some additional technical assumptions, implies that the solution a ∗ is (locally) stable with respect to perturbations of z [CoKu]. To argue the existence of Lagrange multipliers associated to the constraints defining M one considers the set of shifted parameters M˜ = {a − aˆ : a ∈ M}, i.e., γ
1
˜ M = a ∈ H (I ) : a ≥ ν − a, ˆ a H 1 (I ) ≤ γ , a dx = 0, a(β) = a(γ ) = 0 . β
We apply Proposition 1.8 with U = and (|u|2L2 (I ) + |ux |2L2 (I ) )1/2 as norm, γ L = {a ∈ U : a ≥ 0}, q ∗ (a) = β a dx, ϕ = ν − aˆ ∈ U , and μ = 0. The mapping g : U → U × R × R is given by
2 g(a) = a − ϕ, a H 1 (I ) − γ 2 , q ∗ (a) . H01 (I ),
γ Let us assume that β ν dx < m and ν − aˆ H 1 (I ) < γ . Then we have q ∗ (L) ⊂ γ R+ , q ∗ (ϕ) < m − β adx ˆ = μ, and
πϕ 1 = π(ν − a) ˆ H 1 (I ) ≤ ν − aˆ H 1 (I ) < γ . H (I )
i
i i
i
i
i
i
1.4. Applications
ItoKunisc 2008/6/12 page 13 i
13
It remains to ascertain that [ker q ∗ ]⊥ ∩ L is nonempty. Let ψ be the unique solution of −ψxx + ψ = 1 on (β, γ ), ψ(β) = ψ(γ ) = 0. By the maximum principle ψ ∈ L. A short calculation shows that ψ ∈ [ker q ∗ ]⊥ and hence h0 := q ∗ (ψ)−1 ψ satisfies h0 ∈ [ker q ∗ ]⊥ ∩ L and q ∗ (h0 ) = 1. Next we present examples where the general existence result of Section 1.3 is not applicable. Different techniques to be developed in the following sections will guarantee, however, the existence of a Lagrange multiplier. In these examples we shall again use e for equality and g for inequality constraints. Example 1.12. Consider the problem min x12 + x32 subject to e(x1 , x2 , x3 ) = 0, where e(x1 , x2 , x3 ) =
(1.4.15)
x1 − x22 − x3 . x22 − x32
Here X = R3 , Z = R2 , e of Section 1.3 is e here, and K = {0}. Note that x ∗ = (0, 0, 0) is a solution of (1.4.15) and that ∇e(x ∗ ) is not surjective. Hence x ∗ is not a regular point for the constraint e(x) = 0. Nevertheless (0, λ2 ) with λ2 ⊂ R arbitrary is a Lagrange multiplier for the constraint in (1.4.15). Note that in case C = X and K = {0}, the last requirement in Definition 1.1 becomes f (x ∗ ) + e (x ∗ )∗ λ∗ = 0 in X ∗ , where e (x ∗ )∗ : Z ∗ → X ∗ is the adjoint of e (x ∗ ). Hence if e (x ∗ ) is not surjective and if it has closed range, then ker e (x ∗ )∗ is not empty and the Lagrange multiplier is not unique. Example 1.13. Here we consider optimal control of the equation with elliptic nonlinearity (Bratu problem) − y + exp(y) = u in , (1.4.16) ∂y = 0 on and y = 0 on ∂ \ , ∂n where is a bounded domain in Rn , n ≤ 3, with smooth boundary ∂ and is a connected, strict subset of ∂. Let H 1 () = {y ∈ H 1 () : y = 0 on ∂ \ }. The following lemma establishes the concept of solution to (1.4.16). Its proof is given at the end of this section. Lemma 1.14. For every u ∈ L2 () the variational problem (∇y, ∇v) + exp y, v H 1 ()∗ ,H 1 () = (u, v) for v ∈ H 1 () has a unique solution y = y(u) ∈ H 1 () and there exists a constant C such that |y|H 1 ∩L∞ ≤ C(|u|L2 () + C) for all u ∈ L2 ()
(1.4.17)
i
i i
i
i
i
i
14
ItoKunisc 2008/6/12 page 14 i
Chapter 1. Existence of Lagrange Multipliers
and |y(u1 ) − y(u2 )|H 1 ∩L∞ ≤ C|u1 − u2 |L2 () for all ui ∈ L2 (), i = 1, 2. The optimal control problem is given by ⎧
2
2 min J (y, u) = 21 |y − z L2 () + α2 |u L2 () ⎪ ⎪ ⎪ ⎪ ⎨ subject to (y, u) ∈ H 1 () ∩ L∞ () × L2 () ⎪ ⎪ ⎪ ⎪ ⎩ and (y, u) solution to (1.4.17),
(1.4.18)
(1.4.19)
where z ∈ L2 (). To consider this problem as a special case of (1.1.1) we set X = Y × L2 (), where Y = H 1 () ∩ L∞ (), Z = H 1 ()∗ , K = {0}, and define e : X → Z ∗ as the mapping which assigns to (y, u) ∈ Y × U the functional v → (∇y, ∇v) + (exp(y), v) − (u, v). The control component of any minimizing sequence {(yn , un )} for J is bounded. Together with Lemma 1.14 it is therefore simple to argue that (1.4.19) admits a solution (y ∗ , u∗ ) ∈ Y × U . Clearly J and e are continuously Fréchet differentiable. If (y ∗ , u∗ ) was a regular point, then this would require surjectivity of e (y ∗ , u∗ ) : X → H 1 ()∗ . For (δy, δu) ∈ X, e (y ∗ , u∗ )(δy, δu) is the functional defined by v → (∇δy, ∇v) + (exp(y ∗ )δy − δu, v),
v ∈ H 1 ().
Since − +exp(y ∗ ) is an isomorphism from H 1 () to H 1 ()∗ and H 1 () is not contained in L∞ (), if n > 1, it follows that e (y ∗ , u∗ ) : X → H 1 ()∗ is not surjective if n > 1. Example 1.15. We consider the least squares problem for the estimation of the vectorvalued convection coefficient u in − y + u · ∇y = f in , (1.4.20) y = 0 on ∂, div u = 0, from data z ∈ L2 (). Here is a bounded domain in Rn , n ≤ 3, with smooth boundary and f ∈ L 2 (). To cast the problem in abstract form we choose U = u ∈ L2n () : div u = 0 , X = H01 () ∩ L∞ () ×U, Z = H −1 (), K = {0} and define e : X → Z as the mapping which assigns to (y, u) ∈ X the functional v → (∇y, ∇v) − (uy, ∇v) − (f, v)
for v ∈ H01 ().
Note that e(y, u) is not well defined for (y, u) ∈ H01 () × U , since u · ∇y ∈ L1 () only. The regularized least squares problem is given by ⎧ 1 α 2 2 ⎨min J (y, u) = 2 |y − z|L2 () + 2 |u|L2n () (1.4.21) ⎩ subject to e(y, u) = 0, div u = 0, (y, u) ∈ X.
i
i i
i
i
i
i
1.4. Applications
ItoKunisc 2008/6/12 page 15 i
15
Observe that (u · ∇y, y) = (∇ · (uy), y) = 0. Using this fact and techniques similar to those of the proof of Lemma 1.14 (compare [Tr, Section 2.3] and [IK14, IK15]) it can be shown that for every u ∈ U there exists a unique solution y = y(u) ∈ X of (1.4.20) and for every bounded subset B of U there exists a constant k(B) such that |y(u)|X ≤ k(B)|u|L2n for all u ∈ B.
(1.4.22)
Extracting a subsequence of a minimizing sequence to (1.4.21) it is simple to argue the existence of a solution (y ∗ , u∗ ) ∈ X to (1.4.21). Clearly e is Fréchet differentiable at (y ∗ , u∗ ) and for δy ∈ H01 () ∩ L∞ (), ey (y ∗ , u∗ )δy ∈ H −1 is the functional defined by ey (y ∗ , u∗ )δy, v H −1 ,H 1 = (∇δy, ∇v) + (u∗ · ∇δy, v) with v ∈ H01 (). 0
Note that ey (y ∗ , u∗ ) is well defined on H01 () ∩ L∞ (). But as a consequence of the bilinear term u∗ · ∇δy which is only in L1 (), ey (y ∗ , u∗ ) is not defined on H01 (). The operator e (y ∗ , u∗ ) from X to H −1 () is not surjective if n > 1, and hence (y ∗ , u∗ ) does not satisfy the regular point condition. We now turn to the proof of Lemma 1.14 for which we require the following result from [Tr]. Lemma 1.16. Let ϕ : (k1 , h1 ) → R be a nonnegative, nonincreasing function and suppose that there are positive constants r, K, and β, with β > 1, such that ϕ(h) ≤ K(h − k)−r ϕ(k)β for k1 < k < h < h1 . If kˆ := K r 2 β−1 ϕ(k1 ) 1
β
β−1 r
ˆ = 0. satisfies k1 + kˆ < h1 , then ϕ(k1 + k)
Proof of Lemma 1.14. Let us first argue the existence of a solution y = y(u) ∈ of (∇y, ∇v) + (ey , v)H 1 ()∗ ,H 1 () = (u, v) for all v ∈ H 1 ().
(1.4.23)
Since (exp (s1 ) − exp (s2 ))(s1 − s2 ) ≥ 0 for s1 , s2 ∈ R, it follows that − φ + exp (φ) − γ I defines a maximal monotone operator from a subset of H 1 () to H 1 ()∗ for some γ > 0 [Ba], and hence (1.4.23) has a unique variational solution y = y(u) ∈ H 1 () for every u ∈ L2 (); see [IK15] for details. Since |∇(y(u1 ) − y(u2 ))|2 ≤ (u1 − u2 , y(u1 ) − y(u2 )) and ∂ \ is nonempty, there exists an embedding constant C such that |y(u1 ) − y(u2 )|H 1 () ≤ C |u1 − u2 |L2 ()
(1.4.24)
for u1 , u2 in L2 (), and |y(u)|H 1 ≤ C(|u|L2 () + C) for all u ∈ L2 (). Throughout the remainder of the proof C will denote a generic constant, independent of u ∈ L2 (). To verify (1.4.17) it remains to obtain an L∞ () bound for y = y(u). The
i
i i
i
i
i
i
16
ItoKunisc 2008/6/12 page 16 i
Chapter 1. Existence of Lagrange Multipliers
proof is based on a generalization of well-known L∞ () estimates due to Stampacchia and Miranda [Tr] for linear variational problems to the nonlinear problem (1.4.23). Let us aim first for a pointwise (a.e.) upper bound for y. For k ∈ (0, ∞) we set yk = (y − k)+ and k = {x ∈ : yk > 0}. Note that yk ∈ H 1 () and yk ≥ 0. Using (1.4.23) we find (∇yk , ∇yk ) = (∇y, ∇yk ) = (u, yk ) − (ey , yk ) ≤ (u, yk ), and hence |∇yk |2L2 ≤ |u|L2 |yk |L2 .
(1.4.25)
By Hölder’s inequality and a well-known embedding result 12 1 1 yk2 ≤ |yk |L6 |k | 3 ≤ C|∇yk |L2 |k | 3 . |yk |L2 = k
Here we used the assumption that n ≤ 3. Employing this estimate in (1.4.25) implies that 1
|∇yk |L2 ≤ C|k | 3 |u|L2 .
(1.4.26)
We denote by h and k arbitrary real numbers satisfying 0 < k < h < ∞, and we find 4 4 (y − k) > (y − k)4 ≥ |h |(h − k)4 , |yk |L4 = k
h
which, combined with (1.4.26), gives ˆ − k)−4 |k | 43 |u|4 2 , |h | ≤ C(h L
(1.4.27)
where the constant Cˆ is independent of h, k, and u. It will be shown that Lemma 1.16 is ˆ 4 2 . The conditions on k1 and applicable to (1.4.27) with ϕ(k) = |k |, β = 43 and K = C|u| L β β−1 1 h1 can easily be satisfied. In fact, in our case k1 = 0, h1 = ∞, and kˆ = Cˆ 4 |u|L2 2 β−1 |0 | 4 . The condition k1 + kˆ < h1 is satisfied since 1 kˆ = Cˆ 4 |u|L2 2 β−1 |0 | β
β−1 4
1 < Cˆ 4 |u|L2 2 β−1 || β
β−1 4
< ∞.
We conclude that |kˆ | = 0 and hence y ≤ kˆ a.e. in . A uniform lower bound on y can be obtained in an analogous manner by considering yk = (−(k + y))+ . We leave the details to the reader. This concludes the proof of (1.4.17). To verify (1.4.18) the H 1 estimate for y(u1 ) − y(u2 ) is already clear from (1.4.24) and it remains to verify the L∞ () estimate. Let us set yi = y(ui ), z = y1 − y2 , zk = (z − k)+ , and k = {x ∈ : zk > 0} for k ∈ (0, ∞). We obtain |∇zk |2L2 = (∇z, ∇zk ) = (u1 − u2 , zk ) − (ey1 − ey2 , zk ) ≤ (u1 − u2 , zk ). Proceeding as above with y and yk replaced by z and zk the desired pointwise upper bound for y1 − y2 is obtained. For the lower bound we define zk = (−(k + z))+ for k ∈ (0, ∞) and k = {x ∈ : zk > 0} = {x : k + y1 (x) < y2 (x)}. It follows that |∇zk |2L2 = −(∇(y1 − y2 ), ∇zk ) = (ey1 − ey2 , zk ) − (u1 − u2 , zk ) ≤ −(u1 − u2 , zk ). From this inequality we obtain the desired uniform pointwise lower bound on y1 − y2 .
i
i i
i
i
i
i
1.5. Weakly singular problems
ItoKunisc 2008/6/12 page 17 i
17
1.5 Weakly singular problems We consider the optimization problem with equality and inequality constraints of the type min J (y, u) (1.5.1) subject to e(y, u) = 0, u ∈ C ⊂ U, with J : Y × U → R, e : Y1 × U → W , where Y, U, W are Hilbert spaces and Y1 is a Banach space that is densely embedded in Y . Further C is a closed convex subset of U . Below W ∗ will denote the dual of W . Let (y ∗ , u∗ ) denote a local solution to (1.5.1) and let V (y ∗ ) × V (u∗ ) ⊂ Y1 × U denote a neighborhood of (y ∗ , u∗ ) such that J (y ∗ , u∗ ) ≤ J (y, u) for all (y, u) ∈ V (y ∗ ) × (V (u∗ ) ∩ C) satisfying e(y, u) = 0. It is assumed throughout that J is Fréchet differentiable in a neighborhood in the Y × U topology of (y ∗ , u∗ ) and that the Fréchet derivative is locally Lipschitz continuous. Further e is assumed to be Fréchet differentiable at (y ∗ , u∗ ) with Fréchet derivative e (y ∗ , u∗ )(δy, δu) = ey (y ∗ , u∗ )δy + eu (y ∗ , u∗ )δu. In particular ey (y ∗ , u∗ ) ∈ L(Y1 , W ). Since Y1 is dense in Y , one may consider ey (y ∗ , u∗ ) as densely defined linear operator with domain in Y . To distinguish this operator from ey (y ∗ , u∗ ) defined on Y1 we shall denote it in this section by G : D(G) ⊂ Y → W and we assume that (H1) G∗ : D(G∗ ) ⊂ W ∗ → Y ∗ is densely defined. Then necessarily G is closable [Ka, p. 168]. Its closure will be denoted by the same symbol. In addition the following assumptions will be required: (H2) Jy (y ∗ , u∗ ) ∈ Rg G∗ . Condition (H2) is a regularity assumption. It implies the existence of a solution λ∗ ∈ D(G∗ ) to the adjoint equation G∗ λ + Jy (y ∗ , u∗ ) = 0, which is a Lagrange multiplier associated to the constraint e(y, u) = 0. (H3) There exists a dense subset D of C with the following property: For every u ∈ D there exists tu > 0 such that for all t ∈ [0, tu ] there exists y(t) ∈ Y1 satisfying e(y(t), u∗ + t (u − u∗ )) = 0 and 1 lim+ |y(t) − y ∗ |2Y = 0. (1.5.2) t→0 t (H4) For every u ∈ D and y(·) as in (H3), e is directionally differentiable at every element of {(y ∗ + s(y(t) − y ∗ ), u∗ + st (u − u∗ )) : s ∈ [0, 1], t ∈ [0, tu ]} in all directions (y, ˜ u) ˜ ∈ Y1 × U and 1 1 ∗ ∗ ∗ ∗ ∗ ∗ ∗ [e (y +s(y(t)−y ), u +stv)−e (y , u )](y(t)−y , tv)ds, λ = 0, lim t→0+ t 0 W,W ∗ where v = u − u∗ .
i
i i
i
i
i
i
18
ItoKunisc 2008/6/12 page 18 i
Chapter 1. Existence of Lagrange Multipliers
Note that (H4) is satisfied if (H3) holds with Y replaced by Y1 and if e : Y1 → W is Fréchet differentiable with locally Lipschitzian derivative. Our assumptions do not require surjectivity of e (y ∗ , u∗ ) : Y1 × U → W which is required by (1.3.1) nor that e (y ∗ , u∗ ) is well defined on all of Y × U . Below we shall give several examples which illustrate the applicability of the assumptions. For now, we argue that if they are satisfied, then a Lagrange multiplier with respect to the equality constraint exists. Theorem 1.17. Let (y ∗ , u∗ ) be a local solution of (1.5.1) and assume that (H1)–(H4) hold. Then ⎧ ∗ ∗ ,u ) = 0 in W (primal equation), ⎪ ⎨e(y ∗ ∗ ∗ ∗ ∗ G λ + J (y , u ) = 0 in Y (adjoint equation), y ⎪ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ⎩ eu (y , u ) λ + Ju (y , u ), u − u ≥ 0 for all u ∈ C (optimality). ∗ U ,U
(1.5.3) The system of equations (1.5.3) is referred to as an optimality system. Proof. Let u ∈ D, set v = u − u∗ , choose tu according to (H3), and assume that t ∈ (0, tu ]. Due to (H3) and (H4) 0 = e(y(t), u∗ + tv) − e(y ∗ , u∗ ) = G(y(t) − y ∗ ) + t eu (y ∗ , u∗ ) v 1 + [e (y ∗ + s(y(t) − y ∗ ), u∗ + stv) − e (y ∗ , u∗ )](y(t) − y ∗ , tv)ds.
(1.5.4)
0
(H2) implies the existence of a solution λ∗ to the adjoint equation. Observe that by (1.5.4) and the fact that u∗ is a local solution to (1.5.1) 0 ≤ J (y(t), u∗ + tv) − J (y ∗ , u∗ ) = J (y ∗ , u∗ )(y(t) − y ∗ , tv) 1 ∗ + J (y + s(y(t) − y ∗ ), u∗ + stv) − J (y ∗ , u∗ ) (y(t) − y ∗ , tv)ds 0
+ G∗ λ∗ , y(t) − y ∗ Y ∗ ,Y + t eu (y ∗ , u∗ )v, λ∗ W,W ∗ 1 ∗ + e (y + s(y(t) − y ∗ ), u∗ + stv) − e (y ∗ , u∗ ) (y(t) − y ∗ , tv), λ∗
W,W ∗
0
ds.
By the second equation in (1.5.3), local Lipschitz continuous differentiability of J , (H3), (H4), and the fact that J (y ∗ , u∗ )(y(t) − y ∗ , tv) = Jy (y ∗ , u∗ )(y(t) − y ∗ ) + tJu (y ∗ , u∗ )v we obtain 1 0 ≤ lim+ J (y(t), u∗ + tv) − J (y ∗ , u∗ )) = Ju (y ∗ , u∗ )(u∗ ) + eu (y ∗ , u∗ )∗ λ∗ , u − u∗ U ∗ ,U . t→0 t Since u is arbitrary in D and D is dense in C it follows that Ju (y ∗ , u∗ )(u∗ ) + eu (y ∗ , u∗ )∗ λ∗ , u − u∗ U ∗ ,U ≥ 0 for all u ∈ C. This ends the proof.
i
i i
i
i
i
i
1.5. Weakly singular problems
ItoKunisc 2008/6/12 page 19 i
19
We next give several examples which demonstrate the applicability of hypotheses (H1)–(H4) and the necessity to allow for two spaces Y1 and Y with Y1 Y . The typical situation that we have in mind is Y1 = Y ∩ L∞ () with Y a Hilbertian function space over . Example 1.18. Consider first the finite-dimensional equality-constrained optimization problem ⎧ ⎨min y12 + u2 , y1 − y22 = u, (1.5.5) ⎩ 3 y2 = u2 for which (y ∗ , u∗ ) = (0, 0, 0) is the solution. Here Y = R2 , U = R1 , W = R1 , and y − y2 − u . e(y, u) = 1 3 2 2 y2 − u Note that with (y1 , y2 , u) = (x1 , x2 , x3 ) this problem coincides with Example 1.12. We recall that e (y ∗ , u∗ ) is not surjective and the theory of Section 1.3 assuring the existence of a Lagrange multiplier is therefore not applicable. However, G∗ = 10 00 and the adjoint equation λ 0 G∗ 1 = λ2 0 has infinitely many solutions. Thus (H1), (H2) are satisfied. As for (H3) note that yy12 (u) = (u) u+u 34 defines a solution branch to e(y, u) = 0 for which (H3) is satisfied. It is simple to 2 u3
verify (H4). Hence (1.5.5) is an example for an optimization problem where all hypotheses (H1)–(H4) are satisfied and Theorem 1.17 is applicable. Example 1.19. We consider the optimal control problems with distributed control min J (y, u) =
α 1 |y − z|2L2 () + |u|2L2 () 2 2
(1.5.6)
subject to ⎧ ⎨− y + exp(y) = u in , ∂y = 0 on , ⎩ ∂n y = 0 on ∂ \ ,
(1.5.7)
where α > 0, z ∈ L2 (), is a bounded domain in Rn , n ≤ 3, with smooth boundary ∂, and is a connected, strict subset of ∂. The concept of solution to (1.5.7) is the variational one of Lemma 1.14. To consider this problem in the general setting of this section the control variable u is chosen in U = L2 (). We set Y = H 1 () = {y ∈ H 1 () : y = 0 on ∂\ }, Y1 = H 1 () ∩ L∞ (), and W = (H 1 ())∗ . Moreover e : Y1 × U → W is defined by assigning to (y, u) ∈ Y1 × U the functional on W ∗ given by v → (∇y, ∇v) + (exp y, v) − (u, v) for v ∈ H 1 ().
i
i i
i
i
i
i
20
ItoKunisc 2008/6/12 page 20 i
Chapter 1. Existence of Lagrange Multipliers
Let (y ∗ , u∗ ) ∈ Y1 denote a solution to (1.5.6), (1.5.7). For (y, u) ∈ Y1 × L2 () the Fréchet derivative ey (y, u)δy of e with respect to y in direction δy ∈ H 1 () is given by the functional v → (∇δy, ∇v) + ((exp y)δy, v). Clearly ey (y, u) : Y → W is symmetric and (H1), (H2) are satisfied. (H3) is a direct consequence of Lemma 1.14. To verify (H4) note that e (y, u)(δy, δu) is the functional defined by v → (∇δy, ∇v) + (exp(y)δy, v) − (δu, v) for v ∈ H 1 (). If (y, u) ∈ Y1 × U , then e (y, u) is well defined on Y × U . (H4) requires us to consider 1 lim t→0+ t
1
0
(exp(y ∗ + s(y(t) − y ∗ )) − exp y ∗ )(y(t) − y ∗ )λ∗ dxds,
(1.5.8)
where y(t) is the solution to (1.5.7) with u = u∗ + tv, v ∈ U . Note that λ∗ ∈ L∞ and that {|y(t)|L∞ : t ∈ [0, 1]} is bounded by Lemma 1.14. Moreover |y(t) − y ∗ |Y1 ≤ c t|v|L2 for a constant c independent of t ∈ [0, 1], and thus the pointwise local Lipschitz property of the exponential function implies that the limit in (1.5.8) is zero. (H4) now easily follows. The considerations of this example remain correct for cost functionals J that are much more general than the one in (1.5.6). In fact, it suffices that J is weakly lower semicontinuous from Y ×U to R and radially unbounded with respect to u, i.e., limn→∞ |un |L2 = ∞ implies that lim supn→∞ inf y∈Y J (y, un ) = ∞. This guarantees existence of a solution (y ∗ , u∗ ). The general regularity assumptions and (H1)–(H4) are satisfied if J : Y × U → R is continuous and Fréchet differentiable in a neighborhood of (y ∗ , u∗ ) with locally Lipschitz continuous derivative. Example 1.20. Consider the nonlinear optimal boundary control problem min J (y, u) =
1 α |y − z|2L2 () + |u|2H s ( ) 2 2
(1.5.9)
subject to ⎧ ⎨− y + exp y = f in , ∂y = u on , ⎩ ∂n y = 0 on ∂ \ ,
(1.5.10)
where α > 0, z ∈ L2 (), f ∈ L∞ are fixed, is a bounded domain in Rn , with smooth boundary ∂, and is a nonempty, connected, strict subset of ∂. Further s is a real number strictly larger than n−3 if n ≥ 3, and s = 0 if n < 3. Differently from Example 1.19 2 the dimension n of is now allowed to be arbitrary. This example can be treated within the general framework of this book by setting Y, Y1 , W as in Example 1.19. The control space U is chosen to be H s ( ). To verify (H1)–(H4) one proceeds as in Example 1.19 by utilizing the following lemma. Its proof is given in [IK15] and is similar to that of Lemma 1.14. Lemma 1.21. The variational problem (∇y, ∇v) + (exp y, v) = (f, v) + (u, v) ,
v ∈ H 1 (),
i
i i
i
i
i
i
1.5. Weakly singular problems
ItoKunisc 2008/6/12 page 21 i
21
has a unique solution y = y(u) ∈ H 1 () ∩ L∞ () for every u ∈ H s ( ), and there exists a constant c such that |y|H 1 ∩L∞ ≤ c(|u|H s ( ) + c) for all u ∈ H s ( ).
(1.5.11)
Moreover, c can be chosen such that |y(u1 ) − y(u2 )|H 1 ()∩L∞ ≤ c|u1 − u2 |H s for all ui ∈ H s ( ), i = 1, 2.
(1.5.12)
Example 1.22. We reconsider (1.4.21) from Example 1.15 as a special case of (1.5.1). For this purpose we set Y = H01 (), Y1 = H01 () ∩ L∞ (), W = H −1 (), and e as in Example 1.15. Let (y ∗ , u∗ ) ∈ Y1 × U denote a solution of (1.4.21). Clearly e is Fréchet differentiable at (y ∗ , u∗ ) and the partial derivative G = ey (y ∗ , u∗ ) ∈ L(Y1 , W ) is the functional characterized by ey (y ∗ , u∗ )δy, v W,W ∗ = (∇δy, ∇v) + (u∗ · ∇δy, v), v ∈ W ∗ = H01 (). As a consequence of the quadratic term u∗ · ∇δy which is only in L1 (), G is not defined on all of Y = H01 (). As an operator from Y1 to W , the operator G is not surjective. Considered as an operator with domain in Y , its adjoint is given by G∗ w = − w − ∇ · (u∗ w). The domain of G∗ contains Y1 and hence G∗ is densely defined. Moreover its range contains L2 () and thus (H1) as well as (H2) are satisfied. Let U (u∗ ) ⊂ U be a bounded neighborhood of u∗ . Since for every u ∈ U (u∗ ) (∇(y(u) − y ∗ ), ∇v) − (u(y(u) − y ∗ ), ∇v) = ((u − u∗ )y ∗ , ∇v) for all v ∈ H01 (), it follows that there exists a constant k > 0 such that |y(u) − y ∗ |H 1 ≤ k|u − u∗ |L2n for all u ∈ U (u∗ ),
(1.5.13)
and (H3) follows. The validity of (H4) is a consequence of (1.5.13) and the fact that λ∗ is the unique variational solution in H01 () to − λ∗ − ∇ · (u∗ λ∗ ) = −(y ∗ − z) and hence an element of L∞ (). Remark 1.5.1. Comparing Examples 1.19 and 1.20 with Example 1.22 we observe that the linearization e (y, u), with (y, u) ∈ Y1 × U , is well defined on Y × U for Examples 1.19 and 1.20 but it is only defined with domain strictly contained in Y × U for Example 1.22. For none of these examples is e defined on all of Y × U . Example 1.23. Here we consider the nonlinear optimal control problem with nonlinearity of blowup type: ⎧ min 12 |∇(y − z)|2L2 () + α2 |u|2L2 ( ) subject to ⎪ ⎪ 2 ⎨ − y − exp y = f in , (1.5.14) ∂y ⎪ ⎪ ⎩ ∂n = u on , y = 0 on ∂ \ ,
i
i i
i
i
i
i
22
ItoKunisc 2008/6/12 page 22 i
Chapter 1. Existence of Lagrange Multipliers
where α > 0, z ∈ H 1 (), f ∈ L2 (), is a smooth bounded domain in R2 , and ⊂ ∂ is a connected strict subset of ∂. Since is assumed to be a two-dimensional domain we have the following property of the exponential function: for every p ∈ [1, ∞), {| exp y|Lp : y ∈ B} is bounded
(1.5.15)
provided that B is a bounded subset of H01 () [GT, p. 155]. The variational form of the boundary value problem in (1.5.14) is given by (∇y, ∇v) − (exp y, v) = (f, v) + (u, v) for all v ∈ H 1 (),
(1.5.16)
where H 1 () is defined in Example 1.19. To argue existence of a solution to (1.5.14) let {(yn , un )} be a minimizing sequence with weak limit (y ∗ , u∗ ) ∈ H01 () × L2 (). Due to (1.5.15) and the radial unboundedness of the cost functional with respect to the H 1 () × L2 ( )-norm the set {| exp yn |Lp : n ∈ N} is bounded for every p ∈ [1, ∞) and {| exp yn |W 1,p : n ∈ N} is bounded for every p ∈ [1, 2). Since W 1,p () is compactly embedded in L2 () for every p ∈ (1, 2) it follows that for a subsequence of {yn }, denoted by the same symbol, lim exp(yn ) = exp y ∗ in L2 (). It is now simple to argue that (y ∗ , u∗ ) is a solution to (1.5.14). Let us discuss then the validity of (H1)–(H4) with Y = Y1 = H 1 (), W = (H 1 ())∗ , with the obvious choice for J , and with e : Y × U → W the mapping assigning to (y, u) ∈ Y × U the functional v → (∇y, ∇v) − (exp y, v) − (f, v) − (u, v) for v ∈ H 1 (). From (1.5.15) it follows that e is well defined and its Fréchet derivative at (y, u) in direction (δy, δu) is characterized by (e (y, u)(δy, δu), v) = (∇δy, ∇v) − (exp(y)δy, v) − (δu, v) for v ∈ H 1 (). The operator G = ey (y ∗ , u∗ ) can be expressed as G(δy) = − δy − exp(y ∗ )δy. Note that G ∈ L(Y, W ∗ ), and G is self-adjoint with compact resolvent. In particular, (H1) is satisfied. The spectrum of G consists of eigenvalues only. It will be assumed that 0 is not an eigenvalue of G.
(1.5.17)
Due to the regularity assumption for z (note that it would suffice to have z ∈ (H 1 )∗ ), (1.5.17) implies that (H2) holds. To argue the validity of (H3) and (H4) one can rely on the implicit function theorem. Let B be a bounded open neighborhood of y ∗ in H 1 (). Using (1.5.15) one argues the existence of a constant κ > 0 such that | exp y − exp y| ¯ L4 ≤ κ|y − y| ¯ H 1 for all y, y¯ ∈ B. It follows that e is continuous on B × U and its partial derivative ey (y, u) is Lipschitz continuous with respect to (y, u) ∈ B × U . The implicit function theorem implies the existence of a neighborhood U (u∗ ) of u∗ such that for every u ∈ U (u∗ ) there exists a solution y(u∗ ) of (1.5.16) depending continuously on u. Since e(y, u) is Lipschitz continuous with respect to u it follows, moreover, that there exists L > 0 such that |y(u) − y ∗ |H 1 ≤ L|u − u∗ |L2 ( ) for all u ∈ U (u∗ ). (H3) and (H4) are a direct consequence of this estimate.
i
i i
i
i
i
i
1.6. Approximation, penalty, and adapted penalty techniques
ItoKunisc 2008/6/12 page 23 i
23
The methodology utilized to consider this example can also be applied to Examples 1.19 and 1.20 provided that is restricted to being two-dimensional. This is essential for (1.5.15) to hold. For Example 1.23 it is essential for the cost functional to be radially unbounded with respect to the H 1 ()-norm for the y-component to guarantee that minimizing sequences are bounded. For Examples 1.19 and 1.20 the a priori bound on the y-component of minimizing sequences can be obtained through the state equation.
1.6 Approximation, penalty, and adapted penalty techniques As in the previous section we consider problems of type min J (y, u) subject to e(y, u) = 0, u ∈ C ⊂ U.
(1.6.1)
We describe techniques that in specific situations can be more powerful than the general theory of Section 1.4 to obtain optimality systems of type ⎧ ∗ ∗ e(y , u ) = 0, ⎪ ⎪ ⎪ ⎪ ⎨ ey∗ (y ∗ , u∗ )λ∗ + Jy (y ∗ , u∗ ) = 0, (1.6.2) ⎪ ⎪ ⎪ ⎪ ∗ ∗ ∗ ∗ ⎩ eu (y , u )λ + Ju (y ∗ , u∗ ), u − u∗ U ∗ ,U ≥ 0 for all u ∈ C, where (y ∗ , u∗ ) denotes a solution to (1.6.1). We shall proceed formally here, referring the reader to [Ba, Chapter 3.2.2] and [Lio1] for details.
1.6.1 Approximation techniques If J and/or e are not smooth, then one can aim for obtaining an optimality system of the form (1.6.2) by introducing families of smooth operators eε and smooth functionals Jε , ε > 0, and replacing (1.6.1) by min Jε (y, u) (1.6.3) subject to eε (y, u) = 0, and u ∈ C. Let (yε , uε ), ε > 0, denote solutions to (1.6.3). Under relatively mild conditions it can be argued that {(yε , uε )}ε>0 contains accumulation points and that they are solutions of (1.6.2). But they need not coincide with (y ∗ , u∗ ). To overcome this difficulty a penalty term is added to the cost in (1.6.3), resulting in the adapted penalty formulation ⎧ 1 ⎪ ⎨min Jε (y, u) + | u − u∗ |2 2 (1.6.4) ⎪ ⎩ subject to eε (y, u) = 0 and u ∈ C. It can be shown that under appropriate assumptions (yε , uε ) converges to (y ∗ , u∗ ) and that there exists λ∗ such that (1.6.2) holds. This approach can be used for optimal control of semilinear elliptic equations, variational inequalities [Ba, BaK1, BaK2], and state-dependent parameter estimation problems [BKR], for example.
i
i i
i
i
i
i
24
1.6.2
ItoKunisc 2008/6/12 page 24 i
Chapter 1. Existence of Lagrange Multipliers
Penalty techniques
This technique is well suited for systems with multiple steady states or finite explosion property. The procedure is explained by means of an example. We consider the nonlinear parabolic control system ⎧ yt − y − y 3 = u in Q = (0, T ) × , ⎪ ⎪ ⎪ ⎪ ⎨ y = 0 on = (0, T ) × ∂, (1.6.5) ⎪ ⎪ ⎪ ⎪ ⎩ y(0, ·) = y0 in , where y = y(t, x), T > 0, and y0 is a given initial condition. With (1.6.5) we associate the optimal control problem ⎧ 1 α ⎪ ⎨min J (y, u) = |y − z|6L6 (Q) + |u|2L2 (Q) 6 2 (1.6.6) ⎪ ⎩ 1,2 subject to u ∈ C, y ∈ H (Q) and (1.6.5), where α > 0, z ∈ L6 (Q), C is a closed convex set in L2 (Q), and H 1,2 (Q) = ϕ ∈ L2 (Q) : ϕt , ϕxi , ϕxi xj ∈ L2 (Q) . Realizing the state equation by a penalty term leads to ⎧
2 1 ⎪ ⎨min Jε (y, u) = J (y, u) + yt − y − y 3 − u L2 (Q) ε ⎪ ⎩ subject to u ∈ C, y ∈ H 1,2 (Q), y| = 0, y(0, ·) = y0 .
(1.6.7)
It can be shown that for every ε > 0 there exists a solution (yε , uε ) to (1.6.7). Considering the behavior of the pair (yε , uε ) as ε → 0+ the question of convergence to a particular solution (y ∗ , u∗ ) of (1.6.6) again arises. As before this can be realized by means of adaptation terms: ⎧
2
2 1 1 ⎪ ⎨min Jε (y, u) + y − y ∗ L2 (Q) + u − u∗ L2 (Q) 2 2 (1.6.8) ⎪ ⎩ 1,2 subject to u ∈ C, y ∈ H (Q), y| = 0, y(0, ·) = y0 . Let us denote solutions to (1.6.8) by (y ε , uε ). It is fairly straightforward to derive optimality conditions for (1.6.8): Setting λε = − we find
1 ε yt − y ε − (y ε )3 − uε ∈ L2 (Q) ε
⎧ ε ε ε 2 ε 5 ε ∗ ⎪ ⎪λt − λ − 3(y ) λε = −(y − z) − (y − y ) ⎪ ⎪ ⎨ λε = 0 on , ⎪ ⎪ ⎪ ⎪ ⎩ ε λ (T , ·) = 0 in Q
(1.6.9)
in Q, (1.6.10)
i
i i
i
i
i
i
1.6. Approximation, penalty, and adapted penalty techniques and
(uε − u∗ )(u − uε )dQ ≥ 0
(αuε − λε )(u − uε )dQ + Q
for all u ∈ C,
ItoKunisc 2008/6/12 page 25 i
25
(1.6.11)
Q
where λε must be interpreted as a weak solution in L2 (Q) of (1.6.10) (i.e., the inner product with smooth test function is taken, and partial integration bringing all differentiations on the test function is carried out). It can be verified that uε → u∗ in L2 (Q), y ε y ∗ weakly in H 1,2 (Q) and that there exists λ∗ ∈ W 2,1;6/5 , the weak limit of λε , such that the following optimality system for (1.6.6) is satisfied: ⎧ 3 ⎪ ⎪yt − y − y = u in Q, ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨−λt − λ − 3y 2 λ = −(y − z)5 in Q, (1.6.12) ! ⎪ ⎪ y = λ = 0 on , ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ y(0, ·) = y 0 , λ(T , ·) = 0 in , where W 2,1;6/5 = ϕ ∈ L6/5 (Q) : ϕt , ϕxi , ϕxi xj ∈ L6/5 (Q) ; see [Lio1, Chapter I.3].
i
i i
i
i
ItoKunisc 2008/6/12 page 26 i
i
i
i
i
i
i
i
i
i
ItoKunisc 2008/6/12 page 27 i
Chapter 2
Sensitivity Analysis
2.1
Generalities
In this chapter we discuss the sensitivity of solutions to parameterized mathematical programming problems of the form min x∈C
f (x, p) subject to (2.1.1)
e(x, p) = 0, g(x, p) ≤ 0, and (x, p) ∈ K, where C is a closed convex set in X, and further e : X × P → W represents an equality constraint, g : X × P → Rm a finite-dimensional inequality constraint, and : X × P → Z an infinite-dimensional affine constraint. The cost f as well as the equality and inequality constraints are allowed to depend on a parameter p ∈ P . Unless otherwise specified P is a normed linear space, X, W, Z are real Hilbert spaces, and K is a closed convex cone with vertex at 0 in Z. Such problems arise in inverse, control problems, and variational inequalities, and some examples will be discussed in Section 2.6. The cone K induces a natural ordering ≤ by means of z1 ≤ z2 if z1 − z2 ∈ K. We denote by K + the dual cone given by K + = {z ∈ Z ∗ : z, z˜ ≤ 0 for all z˜ ∈ K}, where ·, · denotes the duality pairing between Z and Z ∗ . Suppose that x0 is the solution to (2.1.1) for the nominal parameter p = p0 . The objective of this section is to investigate • continuity and Lipschitz continuity of the solution mapping p ∈ P → (x(p), λ(p), μ(p), η(p)) ∈ X × W ∗ × Rm,+ × K + , where for p in a neighborhood of p0 , x(p) ∈ X denotes the local solution of (2.1.1) in a neighborhood of x0 , and λ, μ, η are the Lagrange multipliers associated with constraints described by e, g, in (2.1.1); • differentiability of the minimum value function F (p) = f (x(p), p) at p0 ; and 27
i
i i
i
i
i
i
28
ItoKunisc 2008/6/12 page 28 i
Chapter 2. Sensitivity Analysis • directional differentiability of the solution mapping.
Due to the local nature of the analysis, f , e, and g need only be defined in a neighborhood of (x0 , p0 ). We assume throughout this chapter that they are twice continuously Fréchet differentiable with respect to x and that their first and second derivatives are continuous in a neighborhood of (x0 , p0 ). It is assumed that is affine in x for every p ∈ P and that the first derivative (p0 ) with respect to x is continuous in a neighborhood of p0 . In order to ensure the existence of a Lagrange multiplier at x0 we assume that (H1) x0 is a regular point, i.e., ⎧⎛ ⎛ ⎞ ⎞ ⎛ ⎞⎫ 0 0 ⎨ e (x0 , p0 ) ⎬ ⎠ + ⎝ g(x0 , p0 ) ⎠ , 0 ∈ int ⎝ g (x0 , p0 ) ⎠ (C − x0 ) + ⎝ Rm + ⎩ ⎭ −K (p0 ) (x0 , p0 )
(2.1.2)
where the interior is taken in W × Rm × Z and primes denote the Fréchet derivatives with respect to x. We define the Lagrange functional L : X × P × W ∗ × Rm × Z ∗ → R by L(x, p, λ, μ, η) = f (x, p) + λ, e(x, p) + μ, g(x, p) Rm + η, (x, p) . With (2.1.2) holding, it follows from Theorem 1.6 that there exists a Lagrange multiplier (λ0 , μ0 , η0 ) ∈ W ∗ × Rm,+ × K + such that L (x0 , p0 , λ0 , μ0 , η0 ), c − x0 ≥ 0
for all c ∈ C,
e(x0 , p0 ) = 0, (2.1.3) μ0 , g(x0 , p0 ) = 0,
g(x0 , p0 ) ≤ 0, μ0 ≥ 0,
η0 , (x0 , p0 ) = 0,
(x0 , p0 ) ∈ K, η0 ∈ K + .
To express (2.1.3) in a more compact form we recall that the subdifferential ∂ψC of the indicator function 0 if x ∈ C, ψC (x) = ∞ if x ∈ /C of a closed convex set C in a Banach space X is given by {y ∈ X∗ : y, c − x X∗ ,X ≤ 0 for all c ∈ C} ∂ψC (x) = {}
if x ∈ C, if x ∈ / C.
The set ∂ψC (x) is also referred to as the normal cone to C at x. For convenience we also specify ∂ψK + (η) =
{z ∈ Z : z∗ − η, z X∗ ,X ≤ 0 for all z∗ ∈ K + } {}
if η ∈ K + , if η ∈ / K +.
i
i i
i
i
i
i
2.1. Generalities
ItoKunisc 2008/6/12 page 29 i
29
Note that (2.1.3) is equivalent to ⎧ L (x0 , p0 , λ0 , μ0 , η0 ) + ∂ψC (x0 ), ⎪ ⎪ ⎨ e(x0 , p0 ), 0∈ −g(x ⎪ 0 , p0 ) + ∂ψRm,+ (μ0 ), ⎪ ⎩ −(x0 , p0 ) + ∂ψK + (η0 ).
(2.1.4)
In fact this follows from the following characterization of the normal cone to K + . Lemma 2.1. Suppose that K is a closed convex cone in Z. Then z ∈ ∂ψK + (η) if and only if z ∈ K, η ∈ K + , and η, z = 0. Proof. Assume that z ∈ ∂ψK + (η). Then ∂ψK + (η) is nonempty, η ∈ K + , and z∗ − η, z ≤ 0
for all z∗ ∈ K + .
(2.1.5)
Hence with z∗ = 2η this implies η, z ≤ 0 and for z∗ = η2 we have η, z ≥ 0, and therefore η, z = 0. From (2.1.5) it follows that z∗ , z ≤ 0 for all z∗ ∈ K + , and hence the geometric form of the Hahn–Banach theorem implies that z ∈ K. Conversely, if z ∈ K, η ∈ K + , and η, z = 0, then z∗ − η, z ≤ 0 for all z∗ ∈ Z ∗ and thus z ∈ ∂ψK + (η). Let A : X → X∗ be the operator representation of the Hessian L (x0 , p0 , λ0 , μ0 , η0 ) of L such that Ax1 , x2 = L (x0 , p0 , λ0 , μ0 , η0 )(x1 , x2 ) and define operators E : X → W , G : X → Rm , and L : X → Z by E = e (x0 , p0 ),
G = g (x0 , p0 ),
L = (p0 ).
Without loss of generality one can assume that the coordinates of the inequality constraints g(x, p0 ) ≤ 0 and the associated Lagrange multiplier μ0 are arranged so that − 0 + 0 − + : X → R m 1 , g 0 : X → R m2 , μ0 = (μ+ 0 , μ0 , μ0 ) and g = (g , g , g ), with g − m3 g : X → R , m = m1 + m2 + m3 , and g + (x0 , p0 ) = 0, μ+ 0 > 0, g 0 (x0 , p0 ) = 0, μ00 = 0, g − (x0 , p0 ) < 0, μ− 0 = 0.
(2.1.6)
We further define G+ = g + (x0 , p0 ) ,
and E+ =
E G+
G0 = g 0 (x0 , p0 ) ,
: X → W × R m1 .
i
i i
i
i
i
i
30
ItoKunisc 2008/6/12 page 30 i
Chapter 2. Sensitivity Analysis
Note that the coordinates denoted by superscripts + behave locally like equality constraints. Define E(z) : X × R → (W × Rm1 ) × Rm2 × Z for z ∈ Z by ⎛
E+ E(z) = ⎝ G0 L
⎞ 0 0 ⎠. z.
The adjoint E ∗ (z) is given by E ∗ (z) =
E+∗ 0
G∗0 0
L∗ ·, z Z
.
The following conditions will be used. (H2) There exists a κ > 0 such that Ax, x X∗ ,X ≥ κ |x|2X for all x ∈ ker(E+ ). (H3) E(z) is surjective at z = (x0 , p0 ). (H4) There exists a neighborhood U˜ × N˜ of (x0 , p0 ) and a positive constant ν such that |f (x, p) − f (x, q)| + |e(x, p) − e(x, q)|W + |g(x, p) − g(x, q)|Rm + |(x, p) − (x, q)|Z ≤ ν |p − q|P for all (x, p) and (x, q) in U˜ × N˜ . Condition (H2) is a second order sufficient optimality condition for (2.1.1) at x0 and (H3) implies that there exists a neighborhood O of (x0 , p0 ) in Z such that E(z) is surjective for all z ∈ O. The chapter is organized as follows. In Sections 2.2 and 2.3 we discuss the basic results for establishing Lipschitz continuity of the solution mapping. The implicit function theory for the generalized equation of the form (2.1.4) is discussed in Section 2.2. In Section 2.3 stability results for solutions to mathematical programming problems including (2.1.1) are addressed. Sufficient conditions for stability of the solutions are closely related to sufficient optimality conditions for local minimizers. Section 2.3 therefore also contains first and second order sufficient conditions for local optimality. Sections 2.4 and 2.5 are devoted to establishing Lipschitz continuity and directional differentiability of the solution mapping. For the latter we employ the assumption of polyhedricity of a closed convex cone K which, together with appropriate additional conditions, implies that the directional ˙ μ, derivative (x, ˙ λ, ˙ η) ˙ in direction q satisfies ⎧ L p (x0 , p0 , λ0 , μ0 , η0 )q + Ax˙ + E ∗ λ˙ + G∗+ μ˙ + + G∗0 μ˙ 0 + L∗ η, ˙ ⎪ ⎪ ⎨ ˙ −ep+ (x0 , p0 )q − E+ x, 0∈ −gp0 (x0 , p0 )q − G0 x˙ + ∂ψRm2 ,+ (μ˙ 0 ), ⎪ ⎪ ⎩ ˙ −p (x0 , p0 )q − Lx˙ + ∂ψKˆ + (η),
(2.1.7)
where e+ = ge+ and Kˆ + is the dual cone of Kˆ = ∪λ>0 λ(K − (x0 , p0 ))∩[η0 ]⊥ . Section 2.6 contains some applications to optimal control of ordinary differential equations. This chapter is strongly influenced by the work of S. M. Robinson and also by that of W. Alt.
i
i i
i
i
i
i
2.2. Implicit function theorem
2.2
ItoKunisc 2008/6/12 page 31 i
31
Implicit function theorem
In this section we discuss an implicit function theorem for parameterized generalized equations of the form 0 ∈ F (x, p) + ∂ψC (x),
(2.2.1)
where F : X × P → X∗ , with X ∗ the strong dual space of X, and C is a closed convex set in X. We assume that X and P are normed linear spaces unless specified otherwise. Suppose x0 ∈ X is a solution to (2.2.1) for a reference parameter p = p0 ∈ P . Let F (x, p0 ) be Fréchet differentiable at x0 and define the linearized form by T x = F (x0 , p0 ) + F (x0 , p0 )(x − x0 ) + ∂ψC (x).
(2.2.2)
Definition 2.2 (Strong Regularity). The generalized equation (2.2.1) is called strongly regular at x0 with associated Lipschitz constant ρ if there exist neighborhoods of V of 0 and U of x0 in X such that (T −1 V ) ∩ U , the intersection of U with the restriction of T −1 to V , is single valued and Lipschitz continuous from V to U with modulus ρ. Note that if C = X, the strong regularity assumption coincides with F (x0 , p0 )−1 being a bounded linear operator, which is the common condition for the implicit function theorem. The following result is taken from Robinson’s work [Ro3]. Theorem 2.3 (Generalized Implicit Function Theorem). Let P be a topological space, and assume that F is Fréchet differentiable with respect to x and both F and F are continuous at (x0 , p0 ). If (2.2.1) is strongly regular at x0 with associated Lipschitz constant ρ, then for every > 0 there exists neighborhoods N of p0 and U of x0 , and a singlevalued function x : N → U such that for every p ∈ N , x(p) is the unique solution in U of (2.2.1). Furthermore, for each p, q ∈ N |x(p) − x(q)|X ≤ (ρ + ) |F (p, x(q)) − F (q, x(q))|X∗ .
(2.2.3)
Proof. For > 0 we choose a δ > 0 such that ρδ < /(ρ + ). By strong regularity, there exists neighborhoods V of 0 and U of x0 in X such that (T −1 V ) ∩ U is single valued and Lipschitz continuous from V to U with modulus ρ. Let h(x, p) = F (x0 , p0 ) + F (x0 , p0 )(x − x0 ) − F (x, p) and choose a neighborhood N of p0 and a closed ball U of radius r about x0 contained in U so that for each p ∈ N and x ∈ U we have for h(x, p) ∈ V |F (x, p) − F (x0 , p0 )| ≤ δ and ρ |F (x0 , p) − F (x0 , p0 )| ≤ (1 − ρδ)r.
i
i i
i
i
i
i
32
ItoKunisc 2008/6/12 page 32 i
Chapter 2. Sensitivity Analysis
For any p ∈ N define an operator p from U → U by p (x) = T −1 (h(x, p)) ∩ U.
(2.2.4)
Note that x ∈ U ∩ p (x) if and only if x ∈ U and (2.2.1) holds. We show that p is a strict contraction and p maps U into itself. For x1 , x2 ∈ U we find, using h (x, p) = F (x0 , p0 ) − F (x, p), that |p (x1 ) − p (x2 )| ≤ ρ |h(x1 , p) − h(x2 , p)| = ρ|
1 0
h (x1 + t (x1 − x2 ), p)(x1 − x2 ) dt| ≤ ρδ |x1 − x2 |,
where ρ δ < 1. Since x0 = T −1 (0) ∩ U we have |p (x0 ) − x0 | ≤ ρ |h(x0 , p)| = ρ |F (x0 , p) − F (x0 , p0 )| ≤ (1 − ρδ)r, and thus for x ∈ U |p (x) − x0 | ≤ |p (x) − p (x0 )| + |p (x0 ) − x0 | ≤ ρδ |x − x0 | + (1 − ρδ)r ≤ r. By the Banach fixed point theorem p has a unique fixed point x(p) in U and for each x ∈ U we have |x(p) − x| ≤ (1 − ρδ)−1 |p (x) − x|. (2.2.5) It follows from our earlier observation that x(p) is the unique solution to (2.2.1) in U . To verify (2.2.3) with p, q ∈ N we use (2.2.5) with x = x(q) and obtain |x(p) − x(q)| ≤ (1 − ρδ)−1 |p (x(q)) − x(q)|. Since x(q) = q (x(q)) we find |p (x(q)) − x(q)| ≤ ρ |h(x(q), p) − h(x(q), q)| = ρ |F (x(q), p) − F (x(q), q)|, and hence
|x(p) − x(q)| ≤ ρ(1 − ρδ)−1 |F (x(q), p) − F (x(q), q)|.
Observing that ρ(1 − ρδ)−1 ≤ ρ + the desired estimate (2.2.3) follows. As a consequence of the previous theorem, Lipschitz continuity of F (x, p) in p implies local Lipschitz continuity of x(p) at p0 . Corollary 2.4. Assume in addition to the conditions of Theorem 2.3 that P is a normed linear space and that for some ν > 0 |F (x, p) − F (x, q)| ≤ ν |p − q| for p, q ∈ N and x ∈ U . Then x(p) is Lipschitz on N with modulus ν (ρ + ). It should be remarked that the condition of strong regularity is the weakest possible condition which can be imposed on the values of a function F and its derivative at a point x0 ,
i
i i
i
i
i
i
2.2. Implicit function theorem
ItoKunisc 2008/6/12 page 33 i
33
so that for each perturbation satisfying the hypothesis of Theorem 2.3, a function x(·) will exist having the properties stated in Theorem 2.3. To see this, one has only to consider a function F : X → X ∗ which is Fréchet differentiable at x0 and satisfies 0 ∈ F (x0 ) + ∂ψC (x0 ). Let P be a neighborhood of the origin in X ∗ , and let F (x, p) = F (x0 , p0 ) + F (x0 , p0 )(x − x0 ) − p with p0 = 0. Choose > 0. If there exist neighborhoods N and V and a function x(·) satisfying the properties stated in Theorem 2.3, then with T x = F (x0 , p0 ) + F (x0 , p0 )(x − x0 ) + ∂ψC (x) we see that the restriction to N of T −1 V is a singled-valued, Lipschitz continuous function. Therefore the generalized equation 0 ∈ F (x, p0 ) + ∂ψC (x) is strongly regular at x0 . One of the important consequences of Theorem 2.3 is the following theorem on parametric sensitivity analysis. Theorem 2.5. Under the conditions of Theorem 2.3 and Corollary 2.4 there exists for each > 0 a function α (p) : N → R+ satisfying limp→p0 α (p) = 0, such that |x(p) − p (x0 )| ≤ α (p) |p − p0 |, where p (x0 ) is the unique solution in U of the linear generalized equation 0 ∈ F (x0 , p) + F (x0 , p0 )(x − x0 ) + ∂ψC (x).
(2.2.6)
Proof. From the proof of Theorem 2.3 it follows that x(p) = p (x(p)) and thus by strong regularity |x(p) − p (x0 )| ≤ ρ |h(x(p), p) − h(x0 , p)|
1
≤ ρ h (x0 + θ (x(p) − x0 )) dθ
|x(p) − x0 | 0
1
≤ ρν(ρ + )|p − p0 | h (x0 + θ (x(p) − x0 )) dθ
. 0
Since h (x, p) = F (x0 , p0 ) − F (x, p) and F is continuous,
1
e (x0 + θ (x(p) − x0 )) dθ
→ 0
0
as p → p0 , which completes the proof. In the case C = X the generalized equation (2.2.1) reduces to the nonlinear equation F (x, p) = 0 and p (x0 ) = F (x0 , p0 )−1 (F (x0 , p0 ) − F (x0 , p)).
i
i i
i
i
i
i
34
ItoKunisc 2008/6/12 page 34 i
Chapter 2. Sensitivity Analysis
Thus, Theorem 2.5 shows that if F (x0 , ·) is Fréchet differentiable at p0 , then so is x(p) and x (p0 ) = −F (x0 , p0 )−1
∂F (x0 , p0 ). ∂p
(2.2.7)
In many applications, one might find (2.2.6) significantly easier to solve than (2.2.1). In fact, the necessary optimality condition (2.1.3) is of the form (2.2.1), and (2.2.6) corresponds to a quadratic programming problem, as we will discuss in Section 2.4. Thus Theorem 2.5 can provide a relatively cheap way to find a good approximation to x(p) for p near p0 .
2.3
Stability results
Throughout this section, let X, Y be real Banach spaces and let C be closed convex subsets of X. Concerning K it suffices in this section to assume that it is a closed convex set in Y , unless it is specifically assumed that it is a closed convex cone in Y . For p in the metric space P with metric δ(·, ·) consider the parameterized minimization problem min f (x, p)
subject to x ∈ C, g(x, p) ∈ K.
(2.3.1)
This class of problems contains (2.1.1) as a special case. To facilitate the analysis of (2.3.1) let (p) denote the set of feasible points, i.e., (p) = {x ∈ C : g(x, p) ∈ K}. For a given p0 ∈ P and x0 ∈ (p0 ) a local minimizer of (2.3.1), define the set r (p) by ¯ 0 , r), r (p) = (p) ∩ B(x ¯ 0 , r) is the closure of the open ball with radius r at x0 in X. We further where r > 0 and B(x define the value function μr (p) by μr (p) = inf {f (x, p)|x ∈ r (p)}.
(2.3.2)
Throughout this section it is assumed that f : X × P → R and g : X × P → Y are continuously Fréchet differentiable with respect to x and that their first derivatives are continuous in a neighborhood of (x0 , p0 ). Recall from Definition 1.5 that x0 ∈ (p0 ) is a regular point if 0 ∈ int {g(x0 , p0 ) + gx (x0 , p0 )(C − x0 ) − K}. (2.3.3) We use a generalized inverse function theorem for set-valued functions due to Robinson [Ro3] for the stability analysis of (2.3.1). For a nonempty set S in X and x ∈ X we define d(x, S) = inf {|y − x| : y ∈ S}, and d(x, S) = ∞ if S = ∅. For nonempty sets S and S¯ in X the Hausdorff distance is defined by ¯ = max sup{d(x, S) ¯ : x ∈ S}, sup{d(y, S) : y ∈ S} ¯ . dH (S, S)
i
i i
i
i
i
i
2.3. Stability results
ItoKunisc 2008/6/12 page 35 i
35
We further set BX = {x ∈ X : |x| < 1}. For a set-valued function F : X → Y we define the preimage of F by F −1 (y) = {x ∈ X : y ∈ F (x)}. A set-valued function is called closed and convex if its graph is closed and convex. Theorem 2.6 (Generalized Inverse Function Theorem). Let X and Y be normed linear spaces and F a closed convex set-valued function from X to Y . Let y0 ∈ F (x0 ) and suppose that for some η > 0, y0 + η BY ⊂ F (x0 + BX ). Then for x ∈ x0 + BX and any y ∈ y0 + int (η BY ), d(x, F −1 (y) ∩ (x0 + BX )) ≤ (η − |y − y0 |)−1 (1 + |x − x0 |)d(y, F (x)). Proof. Let x ∈ x0 + BX and y − y0 ∈ int (η BY ). For y ∈ F (x) the claim is clearly satisfied and therefore we assume y ∈ / F (x). Let δ > 0 be arbitrary and choose yδ ∈ F (x) such that 0 < |yδ − y| < d(y, F (x)) + δ.
(2.3.4)
Let α = η − |y − y0 | > 0. Choose any ∈ (0, α) and set y = y + (α − )|y − yδ |−1 (y − yδ ). Then |y − y0 | ≤ |y − y0 | + (α − ) < η, and y ∈ y0 + int (ηBY ). Thus by assumption there exists x ∈ x0 + BX with y ∈ F (x ). For
1 1+(α−)|y−yδ |−1
∈ (0, 1), we find, using convexity of F and the fact that yδ ∈ F (x),
y = (1 − λ)yδ + λ y ∈ (1 − λ)F (x) + λ F (x ) ⊂ F ((1 − λ)x + λ x ).
(2.3.5)
Since x and x are contained in x0 + BX we have (1 − λ)x + λ x ∈ x0 + BX . Together with (2.3.5) this implies d(x, F −1 (y) ∩ (x0 + BX )) ≤ |x − (1 − λ)x + λ x | = λ |x − x |. However, |x − x | ≤ |x − x0 | + |x0 − x | ≤ |x − x0 | + 1 and λ=
|y − yδ | |y − yδ | < . |y − yδ | + α − α−
Hence by (2.3.4) d(x, F −1 (y) ∩ (x0 + BX )) ≤ (η − |y − y0 | − )−1 (1 + |x − x0 |) (d(y, F (x)) + δ), and letting → 0+ , δ → 0+ the claim follows.
i
i i
i
i
i
i
36
ItoKunisc 2008/6/12 page 36 i
Chapter 2. Sensitivity Analysis
The proof of the stability result below relies on the following set-valued fixed point lemma. For a metric space (X , d) let F(X ) denote the collection of all nonempty closed subsets of X . Lemma 2.7. Let (X , d) be a complete metric space and let : X → F(X ) be a set-valued mapping. Suppose that there exists x0 ∈ X and constants r > 0, α ∈ (0, 1), ∈ (0, r) such ¯ 0 , r) the sets (x1 ), (x2 ) are nonempty, that for all x1 , x2 in the closed ball B(x dH ((x1 ), (x2 )) ≤ α d(x1 , x2 ), and d(x0 , (x0 )) ≤ (1 − d)(r − ). ¯ 0 , r) such that x¯ ∈ (x) Then there exists x¯ ∈ B(x ¯ and d(x0 , x) ¯ ≤ (1−α)−1 d(x0 , (x0 ))+. If moreover dH ((S1 ), (S2 )) ≤ α dH (S1 , S2 )
(2.3.6)
¯ 0 , r), then the sequence Sk = (Sk−1 ), for all nonempty closed subsets S1 and S2 of B(x ¯ with S0 = {x0 }, converges in the Hausdorff metric to a set S¯ satisfying S¯ = (S). Proof. Let δ = α(x0 , (x0 )) and set γ = α −1 (1 − α). By induction one proves the ¯ 0 , r) and existence of a sequence xk+1 ∈ (xk ), k = 0, 1, . . . , such that xk ∈ B(x d(xk+1 , xk ) ≤ α k δ + (1 − 2−(k+1) )γ α k+1 , d(xk+ü1 , x0 ) ≤ (1 − α k+1 )(r − ) + γ
k+1
(1 − 2−i )α i .
(2.3.7)
i=1
In fact, choose x1 ∈ (x0 ) with d(x0 , x1 ) ≤ δ + γ2 α ≤ (1 − α)(r − ) + γ2 α ≤ r. ¯ 0 , r) and (2.3.7) holds for k = 0. Now suppose that (2.3.7) holds with k Hence x1 ∈ B(x replaced by k − 1. Then d(x0 , xk ) < r and hence (xk ) is nonempty and there exists xk+1 ∈ (xk ) satisfying d(xk+1 , xk ) ≤ d((xk ), xk )+2−(k+1) γ α k+1 . Consequently d(xk+1 , xk ) ≤ k+1 dH ((xk ), (xk−1 )) + 2−(k+1) γ α k+1 ≤ α d(xk , xk−1 ) + 2−(k+1) ≤ α k δ + (1 − !γk+1α −(k+1) k+1 2 )γ α , and d(xk+1 , x0 ) ≤ d(xk+1 , xk ) + d(xk , x0 ) ≤ γ i=1 (1 − 2−i )α i + (1 − k+1 α )(r − ), and (2.3.7) holds for all k. For k ≥ 0 and for every m ≥ 1 we have d(xk+m , xk ) ≤
m−1
d(xk+i+1 , xk+i )
i=0
≤
m−1
α k+i δ + (1 − 2−(k+i+1) )γ α k+i+1
i=0
≤ α k (1 − α)−1 (δ + γ α),
i
i i
i
i
i
i
2.3. Stability results
ItoKunisc 2008/6/12 page 37 i
37
¯ o , r). Since X is complete there exists and consequently {xk } is a Cauchy sequence in B(x δ ¯ x¯ ∈ B(x0 , r) with limk→∞ xk = x¯ and d(x, ¯ x0 ) ≤ 1−α + . To verify the fixed point property, note that for every k we have d(x, ¯ (x)) ¯ ≤ d(x, ¯ xk+1 ) + d(xk+1 , (x)) ¯ ≤ d(x, ¯ xk+1 ) + dH ((xk ), (x)) ¯
(2.3.8)
≤ d(x, ¯ xk+1 ) + α d(xk , x). ¯ Passing to the limit with respect to k implies that x¯ ∈ (x). ¯ ¯ o , r) We now assume that also satisfies (2.3.6). Since the set of closed subsets of B(x ¯ is complete with respect to the Hausdorff metric, the existence of a set S with lim Sk = S¯ in the Hausdorff metric follows. To argue invariance of S¯ under note that ¯ (S)) ¯ = lim dH (Sk , (S)) ¯ dH (S, k→∞
¯ ≤ lim dH (Sk−1 , S) ¯ =0 = lim dH ((Sk−1 ), (S)) k→∞
k→∞
¯ and hence S¯ = (S). The following result is an important consequence of Theorem 2.6. Theorem 2.8 (Stability). Let x0 ∈ (p0 ) be a regular point. Then for every > 0 there exist neighborhoods U of x0 , N of p0 such that for p ∈ N the set (p) is nonempty and for each x ∈ U ∩ C d(x, (p)) ≤ (M + ) d(0, {g(x, p) − K : x ∈ C}),
(2.3.9)
where a constant M > 0 is independent of > 0. Proof. Let F be the closed convex set-valued mapping from X into Y given by g (x0 , p0 )(x − x0 ) + g(x0 , p0 ) − K for x ∈ C, F (x) = {} for x ∈ / C. The regular point condition implies that there exists an η > 0 such that η BY ⊂ F (x0 + BX ). Here we use Theorem 1.4 and the discussion below Definition 1.5. For δ > 0 and r > 0 let ρ = (η − 2δ r)−1 (1 + r) where we assume that 2δ r < η Let
and
2ρδ < 1.
h(x, p) = g(x0 , p0 ) + g (x0 , p0 )(x − x0 ) − g(x, p)
i
i i
i
i
i
i
38
ItoKunisc 2008/6/12 page 38 i
Chapter 2. Sensitivity Analysis
¯ 0 , r) of radius r about x0 such and choose a neighborhood of N of p0 and a closed ball B(x ¯ that for x ∈ U = C ∩ B(x0 , r) and p ∈ N |g (x, p) − g (x0 , p0 )| ≤ δ and |g(x0 , p) − g(x0 , p0 )| ≤ δ r. As in the proof of Theorem 2.3 it can be shown that |h(x1 , p) − h(x2 , p)| ≤ δ|x1 − x2 | for all x1 , x2 ∈ U and p ∈ N . In particular, if x ∈ U and p ∈ N , then |h(x, p)| ≤ |h(x0 , p0 )| + |h(x, p) − h(x0 , p0 )| ≤ δ r + δ r < η. For any p ∈ N define the set-valued closed convex mapping p from U → C by p (x) = F −1 (h(x, p)).
(2.3.10)
Note that x ∈ p (x) implies that g(x, p) ∈ K. We argue that Lemma 2.7 is applicable with = p and X = C endowed with the metric induced by X. For every x ∈ U the set p (x) is a closed convex subset of X. (Hence, if X is a reflexive Banach space, then the metric projection is well defined and Lemma 2.7 will be applicable with = 0.) For every x1 , x2 ∈ U we have by Theorem 2.6 dH (p (x1 ), p (x2 ))| ≤ ρ |h(x1 , p) − h(x2 , p)| < ρδ |x1 − x2 |, where we used that h(xi , p) ∈ F (xi ) for i = 1, 2. Moreover |p (x0 ) − x0 | ≤ ρ |g(x0 , p) − g(x0 , p0 )| ≤ ρδr, and for every pair of sets S1 and S2 in U we have dH (p (S1 ), p (S2 )) ≤ ρ δ dH (S1 , S2 ). Hence all conditions of Lemma 2.7 are satisfied with α = 2ρδ and = r(1−2α) . Con1−α sequently there exists a Cauchy sequence {xk } in C such that xk+1 ∈ p (xk ) and x¯ = ¯ Moreover, if Sk = p (Sk−1 ) with S0 = {x0 }, then limk→∞ xk ∈ C satisfies x¯ ∈ p (x). ¯ and dH (Sm , Sk ) → 0 as m ≥ k → ∞ and thus S¯ = limk→∞ Sk satisfies S¯ = p (S) S¯ ⊂ p . Let (x, y) ˜ ∈ M = {(x, y) ˜ ∈ U × Y : g(x0 , p0 ) + g (x0 , p0 )(x − x0 ) + y˜ ∈ K}. For all x¯ ∈ S¯ we have by Theorem 2.6 d(x, F −1 (h(x, ¯ p)) ≤ ρ d(h(x, ¯ p), F (x)) ≤ ρ |h(x, ¯ p) + y| ˜ ≤ ρ (|g(x, p) − g(x0 , p0 ) − g (x0 , p0 )(x − x0 ) − y| ˜ + |h(x, ¯ p) − h(x, p)| ).
i
i i
i
i
i
i
2.3. Stability results
ItoKunisc 2008/6/12 page 39 i
39
Since |h(x, p) − h(x, ¯ p)| ≤ δ |x − x|, ¯ ¯ ≤ d(x, S)
ρ ˜ |g(x, p) − g(x0 , p0 ) − g (x0 , p0 )(x − x0 ) − y| 1 − ρδ
for all (x, y) ˜ ∈ M. Now, for x ∈ C and y ∈ g(x, p) − K we let y˜ = g(x, p) − g(x0 , p0 ) − g (x0 , p0 )(x − x0 ) − y. Then (x, y) ˜ ∈ M and hence ¯ ≤ d(x, (p)) ≤ d(x, S)
ρ |y|. 1 − ρδ
ρ 1+r This holds for all y ∈ g(x, p) − K. Since 1−ρδ = η−δ(1+3r) , we can select for every > 0 ρ −1 neighborhoods N = N and U = U such that 1−ρδ ≤ η + . This implies the claim with M = η−1 .
Theorem 2.9 (Lipschitz Continuity of Value Function). Let x0 ∈ (p0 ) be a local minimizer. Assume moreover that x0 is regular and that there exists a neighborhood U × N of (x0 , p0 ) such that |f (x, p) − f (x, ˜ p0 )| ≤ Lf (|x − x| ˜ + δ(p, p0 ))
(2.3.11)
for x, x˜ ∈ U and p ∈ N and |g(x, p) − g(x, p0 )| ≤ Lg δ(p, p0 )
(2.3.12)
for (x, p) ∈ U × N. Then there exist constants r > 0 and Lr and a neighborhood N˜ of p0 such that |μr (p) − μr (p0 )| ≤ Lr δ(p, p0 ) ˜ for all p ∈ N. Proof. Since x0 is a local minimizer of (2.3.1) at p0 there exists s > 0 such that f (x0 , p0 ) ≤ f (x, p0 )
for all x ∈ s (p0 ).
¯ 0 , s). By Theorem 1.4 and the discussion following Definition 1.5 the Let Cs = C ∩ B(x regular point condition (2.3.3) also holds with C replaced by Cs . Applying Theorem 2.8 with C = Cs and = 1 there exist neighborhoods U1 of x0 and N1 of p0 such that s (p) is nonempty and d(x, s (p)) ≤ (M + 1) d(0, {g(x, p) − K : x ∈ Cs })
(2.3.13)
for each x ∈ U1 and p ∈ N1 . We now choose r > 0 and a neighborhood N˜ of p0 such that ¯ 0 , r) ⊂ U1 ∩ U, N˜ ⊂ N1 ∩ N, and 2(M + 1)Lg δ(p, p0 ) < r for all p ∈ N˜ . r ≤ s, B(x (2.3.14) By (2.3.13) and (2.3.12) we obtain for all p ∈ N˜ d(x0 , s (p)) ≤ (M + 1) |g(x0 , p) − g(x0 , p0 )| ≤ (M + 1)Lg δ(p, p0 ).
i
i i
i
i
i
i
40
ItoKunisc 2008/6/12 page 40 i
Chapter 2. Sensitivity Analysis
Thus there exists a x(p) ∈ s (p) such that |x(p) − x0 | ≤ 2(M + 1)Lg δ(p, p0 ) < r and therefore x(p) ∈ r (p) and f (x(p), p) ≥ μr (p). From (2.3.11) |f (x(p), p) − f (x0 , p0 )| ≤ Lr δ(p, p0 ), where Lr = Lf (2(M + 1)Lg + 1). Combining these inequalities, we have μr (p) ≤ f (x(p), p) ≤ f (x0 , p0 ) + Lr δ(p, p0 ) = μr (p0 ) + Lr δ(p, p0 )
(2.3.15)
˜ for all p ∈ N. Conversely, let p ∈ N˜ and x ∈ r (p). From (2.3.13) and (2.3.12) d(x, s (p0 )) ≤ (M + 1) |g(x, p) − g(x0 , p0 )|. Hence there exists x¯ ∈ s (p0 ) such that |x − x| ¯ ≤ 2(M + 1)Lg δ(p, p0 ) < r,
(2.3.16)
¯ p0 ). From (2.3.11) we deduce that |f (x, p) − f (x, ¯ p0 )| ≤ and thus f (x0 , p0 ) ≤ f (x, Lr δ(p, p0 ), and hence ¯ p0 ) ≤ f (x, p) + Lr δ(p, p0 ) μr (p0 ) = f (x0 , p0 ) ≤ f (x,
(2.3.17)
for each x ∈ r (p). The desired estimate follows from (2.3.15) and (2.3.17). In the above proof we followed the work of Alt [Alt3]. Next, we establish Hölder continuity of the local minimizers x(p) of (2.3.1). Theorem 2.9 provides Lipschitz continuity of the value function under the regular point condition. The following example shows that the local solutions to the perturbed problems may behave rather irregularly. Example 2.10. Let X = Y = R 2 , P = R, C = R 2 , K = {(y1 , y2 ) : y1 ≥ 0, y2 ≥ 0}, f (x1 , x2 , p) = x1 + p x2 , and g(x1 , x2 , p) = (x1 , x2 ). For p0 = 0 a local solution to (2.3.1) is given by x0 = (0, 1) and μr (p0 ) = 0. Now let r > 0 be arbitrary. Then for p = p0 the perturbed problem min f (x, p),
x ∈ r (p)
(2.3.18)
has a unique solution x(p) = and
(0, max(1 − r, 0)) (0, 1 + r)
μr (p) =
p max(1 − r, 0) p(1 + r)
if p > 0, if p < 0
if p > 0, if p < 0.
This shows that μr (p) is Lipschitz continuous but the distance |x(p) − x0 | is not continuous with respect to p at p0 = 0. The reason for this behavior of x(p) is that the unperturbed
i
i i
i
i
i
i
2.3. Stability results
ItoKunisc 2008/6/12 page 41 i
41
problem does not depend on x2 and all (0, x2 ) with x2 ≥ 0 are optimal for p = p0 . On the other hand if we let f (x1 , x2 , p) = x1 + px2 + 12 (x2 − 1)2 , then for r ≥ 1 and |p| ≤ 1 the solution to (2.3.18) is given by x(p) = (0, 1 − p). Thus the distance |x(p) − x0 | is continuous in p. This example suggests that some kind of local invertibility should be assumed on f in order to ensure continuity of the solutions. We define a condition closely related to sufficient optimality conditions: There exist κ > 0, β ≥ 1, γ > 0, and neighborhoods Uˆ of x0 and Nˆ of p0 such that f (x, p) ≥ f (x0 , p0 ) + κ |x − x0 |β − γ δ(p, p0 )
(2.3.19)
for all p ∈ Nˆ and x ∈ (p) ∩ Uˆ . The following result shows that this condition is implied by sufficient optimality conditions at x0 . Theorem 2.11. Assume that x0 is regular, that (2.3.11)–(2.3.12) hold, and that there exist constants κ > 0, β ≥ 1 and neighborhood U˜ of x0 such that f (x, p0 ) ≥ f (x0 , p0 ) + κ |x − x0 |β for all x ∈ (p0 ) ∩ U˜ .
(2.3.20)
Then condition (2.3.19) holds. Proof. We select r, s > 0 and a neighborhood N1 of p0 as in (2.3.14) in the proof of Theorem ¯ 0 , s) ∈ U˜ . Set Uˆ = B(x ¯ 0 , r) and choose x ∈ r (p), p ∈ N . 2.9. We can assume that B(x From (2.3.16) there exists an x¯ ∈ s (p0 ) such that |x − x| ¯ ≤ 2(M + 1)Lg δ(p, p0 ), and due to (2.3.17) f (x, p) ≥ f (x, ¯ p0 ) − Lr δ(p, p0 ). By (2.3.20) f (x, ¯ p0 ) ≥ f (x0 , p0 ) + κ˜ |x¯ − x0 |β . Note that |x − x0 |β ≤ (|x¯ − x0 | + |x¯ − x|)β ≤ |x¯ − x0 |β + β(|x¯ − x0 | + |x¯ − x|)β−1 |x − x| ¯ ≤ |x¯ − x0 |β + β(3s)β−1 |x − x|. ¯ Thus, f (x, p) ≥ f (x0 , p0 ) + κ |x − x0 |β − Lr δ(p, p0 ) ≥ f (x0 , p0 ) + κ |x − x0 |β − γ δ(p, p0 ), where γ = Lr + κβ(3s)β−1 2(M + 1)Lg and (2.3.19) follows with Nˆ = N˜ .
i
i i
i
i
i
i
42
ItoKunisc 2008/6/12 page 42 i
Chapter 2. Sensitivity Analysis
Condition (2.3.20) can be verified under the following second order sufficient optimality condition due to Maurer and Zowe [MaZo] . Theorem 2.12. Assume that K is a closed convex cone in Y and that x0 ∈ (p0 ) is a regular point, and let L(x, λ) = f (x, p0 ) + λ, g(x, p0 ) Y ∗ ,Y be the Lagrangian for (2.3.1) at x0 with Lagrange multiplier λ0 . If there exist constants ω > 0, β¯ > 0 such that L (x0 , λ0 , p0 )(h, h) ≥ ω |h|2 for all h ∈ S,
(2.3.21)
where S = L((p0 ), x0 ) ∩ {h ∈ X : λ0 , g (x0 , p0 )h Y ∗ ,Y ≥ −β¯ |h|}, then there exist κ > 0 and a neighborhood U˜ of x0 such that (2.3.20) holds with β = 2. For convenience we recall the definition of the linearizing cone: L((p0 ), x0 ) = {x ∈ C(x0 ) : g (x0 )x ∈ K(g(x0 ))}, where K(g(x0 )) = {λ(y − g(x0 )) : y ∈ K, λ ≥ 0}. Proof. All quantities are evaluated at p0 and this dependence will therefore be suppressed. First we show that every x ∈ = (p0 ) can be represented as x − x0 = h(x) + z(x) with h(x) ∈ L(, x0 ) and |z(x)| = o(|x − x0 |), where
o(s) s
(2.3.22)
→ 0 as s → 0+ . Let x ∈ and expand g at x0 :
g(x) − g(x0 ) = g (x0 )(x − x0 ) + r(x, x0 )
with |r(x, x0 )| = o(|x − x0 |).
By the generalized open mapping theorem (see Theorem 1.4), there exist α > 0 and k(x) ∈ K(g(x0 )) such that for some z(x) ∈ α |r(x, x0 )| (BX ∩ C(x0 )) r(x, x0 ) = g (x0 )z(x) − k(x). If we put h(x) = x − x0 + z(x), then h(x) ∈ C(x0 ), |z(x)| = o(|x − x0 |), and g (x0 )h(x) = g (x0 )(x − x0 ) + r(x, x0 ) + k(x) = g(x) − g(x0 ) + k(x) ∈ K − g(x0 ) + k(x) ⊂ K(g(x0 )), i.e., h(x) ∈ L(, x0 ). Next, for x ∈ 1 f (x) ≥ f (x) + λ0 , g(x) = L(x, λ0 ) = L(x0 , λ0 ) + L (x − x0 , x − x0 ) + r(x − x0 ), 2 (2.3.23) where |r(x −x0 )| = o(|x −x0 |2 ), and we used λ0 , g(x) ≤ 0 for x ∈ , and L (x0 , λ0 ) = 0. Now put B = L (x0 , λ0 ) and S = L() ∩ {h ∈ X : λ0 , g (x0 )h ≥ −β¯ |h|} in Lemma 2.13. It implies the existence of δ0 > 0 and γ > 0 such that L (x0 , λ0 )(x − x0 , x − x0 ) ≥ δ0 |x − x0 |2 for all x − x0 = h(x) + z(x) ∈ − x0 with |z(x)| ≤ γ |h(x)| and λ0 , g (x0 )h ≥ −β¯ |h|.
i
i i
i
i
i
i
2.3. Stability results
ItoKunisc 2008/6/12 page 43 i
43
Choose δ ∈ (0, 1) satisfying δ/(1 − δ) < γ and ρ > 0 such that |z(x)| ≤ δ|x − x0 | for |x − x0 | ≤ ρ. Then for |x − x0 | ≤ ρ |h(x)| ≥ |x − x0 | − |z(x)| ≥ (1 − δ)|x − x0 | and thus |z(x)| ≤
δ |h(x)| < γ |h(x)|. 1−δ
(2.3.24)
Hence L (x0 , λ0 )(x − x0 , x − x0 ) ≥ δ0 |x − x0 |2 for all x − x0 = h(x) + z(x) ∈ − x0 with |x − x0 | ≤ ρ and λ0 , g (x0 )h(x) ≥ −β¯ |h(x)|. By (2.3.23) there exists ρ¯ ∈ (0, ρ] such that f (x) ≥ f (x0 ) + δ40 |x − x0 |2 for all x − x0 = h(x) + z(x) ∈ − x0 with |x − x0 | ≤ ρ and λ0 , g (x0 )h(x) ≥ −β¯ |h(x)|.
(2.3.25)
For the case x − x0 = h(x) + z(x) ∈ − x0 with |x − x0 | ≤ ρ¯ and λ0 , g (x0 )h(x) < −β¯ |h(x)| we find f (x) − f (x0 ) = f (x0 )(x − x0 ) + r¯ (x − x0 ) = − λ0 , g (x0 )h(x) − λ0 , g (x0 )z(x) + r¯ (x − x0 ) ¯ ≥ β|h(x)| + r1 (x − x0 ), where r1 (x − x0 ) = − λ0 , g (x0 )z(x) + r¯ (x − x0 ) and r1 (x − x0 ) = o(|x − x0 |). Together with (2.3.24) this implies ¯ − δ)|x − x0 | + r1 (x − x0 ) for x − x0 = h(x) + z(x) ∈ − x0 f (x) ≥ f (x0 ) + β(1 with |x − x0 | ≤ ρ¯ and λ0 , g (x0 )h(x) < −β¯ |h(x)|. Combined with (2.3.25) we arrive at the desired conclusion. Lemma 2.13. Let S be a subset of a normed linear space X and let B : X × X → R be a bounded symmetric quadratic form satisfying for some ω > 0 B(h, h) ≥ ω |h|2 for all h ∈ S. Then there exist δ0 > 0 and γ > 0 such that B(h + z, h + z) ≥ δ0 |h + z| 2 for all h ∈ S, z ∈ X with |z| ≤ γ |h|. Proof. Let b = B and choose γ > 0 such that δ = ω − 2bγ − bγ 2 > 0. Then for all z ∈ X and h ∈ S satisfying |z| ≤ γ |h| B(h + z, h + z) ≥ ω |h|2 − 2b|h||z| − b|z|2 ≥ δ |h|2 .
i
i i
i
i
i
i
44
ItoKunisc 2008/6/12 page 44 i
Chapter 2. Sensitivity Analysis
Since |h + z| ≤ |h| + |z| ≤ (1 + γ ) |h| we get B(h + z, h + z) ≥ δ0 |h + z|2 with δ0 = δ/(1 + γ )2 . Remark 2.3.1. An example which illustrates the benefit of including the set {h ∈ X : λ0 , g (x0 , p0 )h Y ∗ ,Y ≥ −β¯ |h|} into the definition of the set S on which the Hessian must be positive definite is given by the one-dimensional example with f (x) = −x 3 − x, g(x) = x, K = R− , and C = R. In this case = R − , x0 = 0, f (x0 ) = 0, f (x0 ) = −1, and f (x0 ) = 0. We further find that L(, x0 ) = K and that the Lagrange multiplier is given by λ0 = 1. Consequently for β¯ ∈ (0, 1) we have that S = L(, x0 ) ∩ {h ∈ R : λ0 g (x0 )h ≥ −β¯ |h|} = {0}. Hence (2.3.21) is trivially satisfied. Let us note that f is bounded below as f (x) ≥ f (x0 ) + max(|x|, 2|x|2 ) for x ∈ K. A similar argument does not hold if f is replaced by f (x) = −x 3 . In this case x0 = 0, λ0 = 0, f (x0 ) = 0, S = K, and (2.3.21) is not satisfied. Note that in this case we have lack of strict complementarity; i.e., the constraint is active and simultaneously λ0 = 0. Theorem 2.14. Suppose that x0 ∈ (p0 ) is a regular point and that for some β > 0 f (x0 )h ≥ β|h| for all h ∈ L((p0 ), x0 ). ) of x0 such that Then there exists α > 0 and a neighborhood U ). f (x) ≥ f (x0 ) + α|x − x0 | for all x ∈ (p0 ) ∩ U Proof. For any x ∈ (p0 ) we use the representation x − x0 = h(x) + z(x) from (2.3.22) in the proof of Theorem 2.12. By expansion we have f (x) − f (x0 ) = f (x0 )h(x) + f (x0 )z(x) + r(x, x0 ) with |r(x, x0 )| = o(|x − xo |). The first order condition implies f (x) − f (x0 ) ≥ β|h(x)| + r1 (x, x0 ), where r1 (x, x0 ) = f (xo )z(x) + r(x, xo ) and |r1 (x, x0 )| = o(|x − x0 |). Choose a neighbor) of x0 such that |z(x)| ≤ 1 |x − x0 | and |r1 (x, x0 )| ≤ β |x − x0 | for x ∈ (p0 ) ∩ U ). hood U 2 4 β 1 ) Then |h(x)| ≤ |x − x0 | and f (x) − f (x0 ) ≥ |x − x0 | for x ∈ U . 2
4
Note that for C = X we have f (x0 ) + g(x0 , p0 )∗ λ0 = 0 and the second order condition of Theorem 2.12 involves the set of directions h ∈ L((p0 ), x0 ) for which the first order condition is violated. Following [Alt3] we now show that (2.3.19) implies stability of local minimizers.
i
i i
i
i
i
i
2.4. Lipschitz continuity
ItoKunisc 2008/6/12 page 45 i
45
Theorem 2.15. Suppose that the local solution x0 ∈ (p0 ) is a regular point and that (2.3.11), (2.3.12), and (2.3.19) are satisfied. Then there exist real numbers r > 0 and L > 0 and a neighborhood N of p0 such that the value function μr is Lipschitz continuous at p0 and for each p ∈ N the following statements hold: (a) For every sequence {xn } in r (p) with the property that limn→∞ f (xn , p) = μr (p) it follows that xn ∈ B(x0 , r) for all sufficiently large n. (b) If there exists a point x(p) ∈ r (p) with μr (p) = f (x(p), p), then x(p) ∈ B(x0 , r), i.e., x(p) is a local minimizer of (2.3.1) and 1
|x(p) − x0 | ≤ L δ(p, p0 ) β . Proof. We choose r > 0 and a neighborhood N˜ of p0 as in (2.3.14) in the proof of Theorem ¯ 0 , r) ⊂ Nˆ × Uˆ with Nˆ , Uˆ 2.9. Without loss of generality we can assume that N˜ × B(x determined from (2.3.19) and that κ r β > (Lr + γ )δ(p, p0 )
(2.3.26)
˜ where Lr is determined in Theorem 2.9. Thus μr is Lipschitz continuous at for all p ∈ N, p0 by Theorem 2.9. Let p ∈ N˜ be arbitrary and let {xn } in (p) be a sequence satisfying limn→∞ f (xn , p) = μr (p). By (2.3.19) and Lipschitz continuity of μr κ |xn − x0 |β ≤ |f (xn , p) − f (x0 , p0 )| + γ δ(p, p0 ) ≤ |f (xn , p) − μr (p)| + |μr (p) − μr (p0 )| + γ δ(p, p0 ) ≤ |f (xn , p) − μr (p)| + (Lr + γ ) δ(p, p0 ). This estimate together with (2.3.26) imply |xn − x0 | < r for all sufficiently large n. This proves (a). Similarly, for x(p) ∈ r (p) satisfying μr (p) = f (x(p), p) we have κ |x(p) − x0 |β ≤ (Lr + γ ) δ(p, p0 ). Thus, |x(p) − x0 | < r, |x(p) − x0 | ≤
β1
Lr + γ δ(p, p0 ) κ
and (b) follows.
2.4
Lipschitz continuity
In this section we investigate Lipschitz continuous dependence of the solution and the Lagrange multipliers with respect to the parameter p in (2.1.1). Throughout this section and the next we assume that C = X and that the requirements on the function spaces specified below (2.1.1) are met.
i
i i
i
i
i
i
46
ItoKunisc 2008/6/12 page 46 i
Chapter 2. Sensitivity Analysis
Theorem 2.16. Assume (H1)–(H4) hold at the local solution x0 of (2.1.1). Then there exist neighborhoods N = N (p0 ) of p0 and U = U (x0 , λ0 , μ0 , η0 ) of (x0 , λ0 , μ0 , η0 ) and a constant M such that for all p ∈ N there exists a unique (xp , λp , μp , ηp ) ∈ U satisfying ⎧ L (x, p, λ, μ, η), ⎪ ⎪ ⎨ −e(x, p), 0∈ (2.4.1) −g(x, p) + ∂ψRm,+ (μ), ⎪ ⎪ ⎩ −(x, p) + ∂ψK + (η), and |(x(p), λ(p), μ(p), η(p)) − (x(q), λ(q), μ(q), η(q))| ≤ M |p − q|
(2.4.2)
for all p, q ∈ N. Moreover, there exists a neighborhood N˜ ⊂ N of p0 such that x(p) is a local solution of (2.1.1) if p ∈ N˜ . For the proof we shall employ the following lemma. Lemma 2.17. Consider the quadratic programming problem in a Hilbert space H min J (x) =
1 Ax, x + a, ˜ x
2
(2.4.3)
˜ subject to Ex = b˜ in Y˜ and Gx − c˜ ∈ K˜ ⊂ Z, ˜ and K˜ is a closed convex where Y˜ , Z˜ are Banach spaces, E ∈ L(H, Y˜ ), G ∈ L(H, Z), ˜ If cone in Z. (a) A·, · is a symmetric bounded quadratic form on H ×H and there exists an ω > 0 such that Ax, x ≥ ω |x|2H for all x ∈ ker (E), and ˜ c) (b) for all (b, ˜ ∈ Y˜ × Z˜ ˜ b, ˜ c) ˜ Gx − c˜ ∈ K} is nonempty, S( ˜ = {x ∈ H : Ex = b, then there exists a unique solution to (2.4.3) satisfying ˜ b, ˜ c). Ax + a, ˜ v − x ≥ 0 for all v ∈ S( ˜ ˜ η) Moreover, if x is a regular point, then there exists (λ, ˜ ∈ Y˜ ∗ × Z˜ ∗ such that ⎛
⎞ ⎛ a˜ A 0 ∈ ⎝ b˜ ⎠ + ⎝ −E −G c˜
E∗ 0 0
⎞⎛ G∗ 0 ⎠⎝ 0
⎞ ⎛ ⎞ x 0 ⎠. 0 λ˜ ⎠ + ⎝ ∂ψK˜ + (η) ˜ η˜
˜ b, ˜ c) ˜ b, ˜ c) Proof. Since K˜ is closed and convex, S( ˜ is closed and convex as well. Since S( ˜ ˜ Moreover, every is nonempty, there exists a unique w ∈ range E ∗ such that Ew = b. ˜ b, ˜ c) x ∈ S( ˜ can be expressed as x = w + y, where y ∈ ker (E). From (a) it follows that ˜ b, ˜ c). J is coercive and bounded below on S( ˜ Hence there exists a bounded minimizing ˜ b, ˜ c) sequence {xn } in S( ˜ such that limn→∞ J (xn ) → inf x∈S( ˜ b, ˜ c) ˜ J (x). Boundedness of
i
i i
i
i
i
i
2.4. Lipschitz continuity
ItoKunisc 2008/6/12 page 47 i
47
˜ b, ˜ c) xn and weak sequential closedness of S( ˜ imply the existence of a subsequence of xn ˜ ˜ that converges weakly to some x in S(b, c). ˜ Since J is weakly lower semicontinuous, ˜ b, ˜ c). J (x) ≤ lim inf J (xn )n→∞ . Hence x minimizes J over S( ˜ ˜ ˜ ˜ b, ˜ c). For v ∈ S(b, c) ˜ and t ∈ (0, 1) we have x + t (v − x) ∈ S( ˜ Thus, 0 ≤ J (x + t (v − x)) − J (x) = t Ax + a, ˜ v − x +
t2 A(v − x), v − x
2
and letting t → 0+ we have Ax + a, ˜ v − x ≥ 0
˜ b, ˜ c). for all v ∈ S( ˜
The last assertion follows from Theorem 1.6. Proof of Theorem 2.16. The proof of the first assertion of the theorem is based on the implicit function theorem of Robinson for generalized equations; see Theorem 2.3. It requires us to verify the strong regularity condition for the linearized form of (2.4.1) which is given by ⎧ L (x0 , p0 , λ0 , μ0 , η0 ) + A(x − x0 ) + E ∗ (λ − λ0 ) + G∗ (μ − μ0 ) + L∗ (η − η0 ), ⎪ ⎪ ⎨ −e(x0 , p0 ) − E(x − x0 ), 0∈ −g(x ⎪ 0 , p0 ) − G(x − x0 ) + ∂ψR m,+ (μ), ⎪ ⎩ −(x0 , p0 ) − L(x − x0 ) + ∂ψK + (η), where the operators A, E, G, and L are defined in the introduction to this chapter. If we define the multivalued operator T from X × W × Rm × Z to itself by ⎛ ⎞ ⎛ ⎞⎛ ⎞ x A E ∗ G ∗ L∗ x ⎜ λ ⎟ ⎜ −E 0 ⎜ ⎟ 0 0 ⎟ ⎟ ⎜ ⎟⎜ λ ⎟ T ⎜ ⎝ μ ⎠ = ⎝ −G 0 ⎠ ⎝ 0 0 μ ⎠ η −L 0 0 0 η ⎛ ⎞ ⎛ ⎞ f (x0 , p0 ) − Ax0 0 ⎜ ⎟ ⎜ ⎟ 0 Ex0 ⎟ ⎜ ⎟ +⎜ ⎝ −g(x0 , p0 ) + Gx0 ⎠ + ⎝ ∂ψRm,+ (μ) ⎠ , −(x0 , p0 ) + Lx0 ∂ψK + (η) then the linearization can be expressed as 0 ∈ T (x, λ, μ, η). Since the constraint associated with g − (x, p0 ) ≤ 0 is inactive in a neighborhood of x0 , strong regularity of T at (x0 , λ0 , μ0 , η0 ) easily follows from strong regularity of the multivalued 0 mapping T from X × (W × Rm1 ) × Rm2 × Z into itself at (x0 , (λ˜ 0 , μ+ 0 ), μ0 , η0 ) defined by ⎛ ⎞⎛ ⎞ ⎛ ⎞ x x A E+∗ G∗0 L∗ ⎜ λ˜ ⎟ ⎜ −E+ 0 ⎜ ˜ ⎟ 0 0 ⎟ ⎟⎜ λ ⎟ ⎟ ⎜ T⎜ ⎝ μ0 ⎠ = ⎝ −G0 ⎠ ⎝ μ0 ⎠ 0 0 0 −L 0 0 0 η η ⎛ ⎞ ⎛ ⎞ f (x0 , p0 ) − Ax0 0 ⎜ ⎟ ⎜ ⎟ 0 E+ x0 ⎟+⎜ ⎟ +⎜ 0 ⎠. ⎝ ⎠ ⎝ m ,+ ∂ψR 2 (μ ) G 0 x0 ∂ψK + (η) −(x0 , p0 ) + Lx0
i
i i
i
i
i
i
48
ItoKunisc 2008/6/12 page 48 i
Chapter 2. Sensitivity Analysis
Strong regularity of T requires us to show that there exist neighborhoods of Vˆ of 0 and Uˆ of 0 m1 m2 (x0 , (λ˜ 0 , μ+ × Z such that T −1 (Vˆ ) ∩ Uˆ is single valued 0 ), μ0 , η0 ) in X × (W × R ) × R and Lipschitz continuous from Vˆ to Uˆ . We now divide the proof in several steps. Existence. We show the existence of a solution to ˜ μ0 , η) (α, β, γ , δ) ∈ T (x, λ, for (α, β, γ , δ) ∈ Vˆ . Observe that this is equivalent to ⎛ ⎞ ⎛ ⎞⎛ ⎞ ⎛ x a A E+∗ G∗0 L∗ 0 ⎜ b ⎟ ⎜ −E+ 0 ⎟ ⎜ λ˜ ⎟ ⎜ 0 0 0 ⎟ ⎜ ⎟⎜ ⎟+⎜ 0∈⎜ ⎝ c ⎠ + ⎝ −G0 0 0 0 ⎠ ⎝ μ0 ⎠ ⎝ ∂ψRm2 ,+ (μ0 ) d −L 0 0 0 ∂ψK + (η) η
⎞ ⎟ ⎟, ⎠
(2.4.4)
where (a, b, c, d) = f (x0 , p0 ) − Ax0 − α, E+ x0 − β, G0 x0 − γ , −(x0 , p0 ) + Lx0 − δ). (2.4.5) To solve (2.4.4) we introduce the quadratic optimization problem with linear constraints: 1 min Ax, x + a, x
2
(2.4.6)
subject to E+ x = b, G0 x ≤ c, and Lx − d ∈ K. We verify the conditions in Lemma 2.17 for (2.4.6) with Y˜ = W × Rm1 , Z˜ = Rm2 × Z, E = E+ , G = (G0 , L), and K˜ = Rm2 ,− × K. Let S be the feasible set of (2.4.6): S(α, β, γ , δ) = {x ∈ X : E+ x = b, G0 x ≤ c, Lx − d ∈ K}, where the relationship between (α, β, γ , δ) and (a, b, c, d) is given by (2.4.5). Clearly, x0 ∈ S(0) and moreover x0 is regular point. From Theorem 2.8 it follows that there exists a neighborhood V of 0 such that for all (α, β, δ, γ ) ∈ V the set S(α, β, γ , δ) is nonempty. Lemma 2.17 implies the existence of a unique solution x to (2.4.6) for every (α, β, δ, γ ) ∈ V . By (H1)–(H2) and Theorems 2.12, 2.11, and 2.14 the solution x is Hölder continuous with exponent 12 with respect to (α, β, γ , δ) ∈ V . Next we show that x = x(α, β, γ , δ) is a regular point for (2.4.6), i.e., ⎧⎛ ⎞ ⎛ ⎞ ⎛ ⎞⎫ 0 E+ 0 ⎨ ⎬ 0 ∈ int ⎝ G0 x − c ⎠ + ⎝ G0 ⎠ (X − x) + ⎝ Rm2 ,+ ⎠ . ⎩ ⎭ Lx − d L −K This is equivalent to ⎧⎛ ⎨ 0 ∈ int ⎝ ⎩
⎛ ⎞ ⎛ ⎞ ⎞⎫ β 0 E+ ⎬ ⎠ + ⎝ G0 ⎠ (X − x0 ) + ⎝ Rm2 ,+ ⎠ . γ ⎭ (x0 , p0 ) + δ L −K
i
i i
i
i
i
i
2.4. Lipschitz continuity
ItoKunisc 2008/6/12 page 49 i
49
Since x0 is a regular point, there exists a neighborhood V˜ ⊂ V of 0 such that this inclusion holds for all (α, β, γ , δ) ∈ V˜ . Hence the existence of a solutions to (2.4.4) follows from Lemma 2.17. Uniqueness. To argue uniqueness, assume that (xi , λ˜ i , μ0i , ηi ), i = 1, 2, are solutions to (2.4.4). It follows that A(x1 − x2 ) + E+∗ (λ˜ 1 − λ˜ 2 ) + G∗0 (μ01 − μ02 ) + L∗ (η1 − η2 ), x1 − x2 ≤ 0, E+ (x1 − x2 ) = 0, G0 (x1 − x2 ), μ01 − μ02 ≥ 0, L(x1 − x2 ), η1 − η2 ≥ 0. Using (H2) we find κ |x1 −x2 |2 ≤ A(x1 −x2 ), x1 −x2 ≤ − G∗0 (μ01 −μ02 ), x1 −x2 −(L∗ (η1 −η2 ), x1 −x2 ) ≤ 0, which implies that x1 = x2 = x. To show uniqueness of the remaining components, we observe that E+∗ (λ˜ 1 − λ˜ 2 ) + G∗0 (μ01 − μ02 ) + L∗ (η1 − η2 ) = 0, η1 − η2 , L(x − x0 ) + (x0 , p0 ) + δ = 0, or equivalently
E ∗ (z)(λ˜ 1 − λ˜ 2 , μ01 − μ02 + η1 − η2 ) = 0,
where z = L(x − x0 ) + (x0 , p0 ) + δ and E is defined below (2.1.6). Due to continuous dependence of x on (α, β, γ , δ) we can assure that z ∈ O for all (α, β, γ , δ) ∈ V˜ . Hence (H3) implies that (λ˜ 1 − λ˜ 2 , μ01 − μ02 , η1 − η2 ) = 0. Continuity. By (H3) the operator E ∗ ((x0 , p0 )) has closed range and injective. Hence there exists > 0 such that |E ∗ ((x0 , p0 )) | ≥ | | for all = (λ˜ , μ0 , η). Since E ∗ ((x0 , p0 )) − E ∗ (z) ≤ |z − (x0 , p0 )|, there exists a neighborhood of 0 in X × (W × Rm1 ) × Rm2 × Z, again denoted by V˜ , such that |E ∗ (zδ ) | ≥
| | 2
(2.4.7)
˜ μ0 , η) for all (α, β, γ , δ) ∈ V˜ , where zδ = (x0 , p0 ) + Lx − Lx0 + δ. Any solution (x, λ, satisfies ˜ μ0 , η) = (−Ax − a, 0). E ∗ (zδ )(λ, It thus follows from (2.4.7) that there exists a constant k1 such that for all (α, β, γ , δ) ∈ V˜ the solution (x, λ˜ , μ, ˜ η) satisfies ˜ μ0 , η)|X×(W ×Rm1 )×Rm2 ×Z ≤ k1 . |(x, λ,
i
i i
i
i
i
i
50
ItoKunisc 2008/6/12 page 50 i
Chapter 2. Sensitivity Analysis
Henceforth let (αi , βi , γi , δi ) ∈ V˜ and let (xi , λ˜ i , μ0i , ηi ) denote the corresponding solution to (2.4.4) for i = 1, 2. Then ⎞ ⎛ λ˜ 1 − λ˜ 2 α1 − α2 + A(x2 − x1 ) . E ∗ (zδ1 ) ⎝ μ01 − μ02 ⎠ = η2 , L(x2 − x1 ) + δ2 − δ1
η1 − η 2 By (2.4.7) this implies that there exists a constant k2 such that |(λ˜ 1 − λ˜ 2 , μ01 − μ02 , η1 − η2 )|(W ×Rm1 )×Rm2 ×Z ≤ k2 (|α1 − α2 | + |δ1 − δ2 | + |x1 − x2 |). (2.4.8) Now, from the first equation in (2.4.4) we have A(x1 − x2 ) + E+∗ (λ˜ 1 − λ˜ 2 ) + G∗0 (μ01 − μ02 ) + L∗ (η1 − η2 ) + a1 − a2 , x1 − x2 ≤ 0. (2.4.9) We also find μ01 − μ02 , G0 (x1 − x2 ) = μ01 , c1 − c2 + μ01 , c2 − G0 x2
(2.4.10) − μ02 , G0 x1 − c1 − μ02 , c1 − c2 ≥ μ01 − μ02 , c1 − c2
and similarly η1 − η2 , L(x1 − x2 ) ≥ η1 − η2 , d1 − d2 .
(2.4.11)
Representing x1 − x2 = v + w with v ∈ ker (R+ ) and w ∈ range (E+∗ ), one obtains E+ (x1 − x2 ) = E+ w = b1 − b2 . Thus |w| ≤ k3 |b1 − b2 |
(2.4.12)
for some k3 , and from (H2) and (2.4.9)–(2.4.11) κ |v|2 ≤ Av, v = A(x1 − x2 ), x1 − x2 − 2 Av, w − Aw, w
≤ − λ˜ 1 − λ˜ 2 , b1 − b2 − μ01 − μ02 , c1 − c2 − η1 − η2 , d1 − d2
− a1 − a2 , v + w − 2 Av, w − Aw, w . Let = (λ˜ 1 − λ˜ 2 , μ01 − μ02 , η1 − η2 ). Then κ |v|2 ≤ | ||(β1 − β2 , γ1 − γ2 , δ1 − δ2 )| + |α1 − α2 |(|v| + |w|) + A(2|v||w| + |w|2 ). It thus follows from (2.4.8), (2.4.12) that there exists a constant k4 such that |x1 − x2 | ≤ k4 |(α1 − α2 , β1 − β2 , γ1 − γ2 , δ1 − δ2 )| for all (αi , βi , γi , δi ) ∈ V˜ . We apply (2.4.8) once again to obtain Lipschitz continuity of (x, λ˜ , μ0 , η) with respect to (α, β, γ , δ) in a neighborhood of the origin. Consequently T
i
i i
i
i
i
i
2.4. Lipschitz continuity
ItoKunisc 2008/6/12 page 51 i
51
is strongly regular at (x0 , λ0 , μ0 , η0 ), Robinson’s theorem is applicable, and (2.4.1), (2.4.2) follow. Local solution. We show that there exists a neighborhood N˜ of p0 such that for p ∈ N˜ the second order sufficient optimality (2.3.21) is satisfied at x(p) so that x(p) is a local solution of (2.1.1) by Theorem 2.12. Due to (H2) and the smoothness properties of f, e, g we can assume that L (x(p), p, λ(p), η(p))(h, h) ≥
κ 2 |h| 2
for all h ∈ ker E+
(2.4.13)
if p ∈ N (p0 ). Let us define Ep = (e (x(p), p), g+ (x(p), p)) for p ∈ N (p0 ). Due to surjectivity of Ep0 and smoothness properties of e, g there exists a neighborhood N˜ ⊂ N (p0 ) of p0 such that Ep is surjective for all p ∈ V˜ . Lemma 2.13 implies the existence of δ0 > and γ > 0 such that
L (x(p), p, λ(p), η(p))(h + z, h + z) ≥ δ0 |h + z|2 for all h ∈ ker E+ and z ∈ X satisfying |z| ≤ γ |h|. The orthogonal projection onto ker Ep is given by Pker Ep = I − Ep∗ (Ep Ep∗ )−1 Ep . We can select N˜ so that |Ep∗ (Ep Ep∗ )−1 Ep − Ep∗0 (Ep0 Ep∗0 )−1 Ep0 | ≤
γ 1+γ
for all p ∈ V˜ . For x ∈ ker Ep , we have x = h + z with h ∈ ker E+ and z ∈ (ker E+ )⊥ and |x|2 = |h|2 + |z|2 . Thus, |z| ≤ |Ep∗ (Ep Ep∗ )−1 Ep x − Ep∗0 (Ep0 Ep∗0 )−1 Ep0 x| ≤
γ (|h| + |z|) 1+γ
and hence |z| ≤ γ |h|. From (2.4.13) this implies L (x(p), p, λ(p), η(p)) ≥ δ0 |x|2
for all x ∈ ker Ep
and, since L((p), p) ⊂ ker Ep , the second order sufficient optimality (2.3.21) is satisfied at x(p). The first part of the proof of Theorem 2.16 contains a Lipschitz continuity result for linear complementarity problems. In the following corollary we reconsider this result in a modified form which will be used for the convergence analysis of sequential quadratic programming problems. Throughout the following discussion p0 in (2.1.1) (with C = X) is fixed and therefore its notation is suppressed. For (x, λ, μ) ∈ X × W ∗ × Rm let L(x, λ, μ) = f (x) + λ, e(x) + μ, g(x) ; i.e., the equality and finite rank inequality constraints are realized in the Lagrangian term. ¯ μ) For (x, λ, μ) ∈ X×W ∗ ×Rm , (x, ¯ λ, ¯ ∈ X×W ∗ ×Rm , and (a, b, c, d) ∈ X×W ×Rn ×Z
i
i i
i
i
i
i
52
ItoKunisc 2008/6/12 page 52 i
Chapter 2. Sensitivity Analysis
consider
⎧ ¯ μ)(x min 12 L (x, ¯ λ, ¯ − x, ¯ x − x) ¯ + f (x) ¯ + a, x − x , ¯ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ¯ − x) ¯ = b, ¯ + e (x)(x ⎨ e(x) (2.4.14)
⎪ ⎪ g(x) ¯ + g (x)(x ¯ − x) ¯ ≤ c, ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ (x) ¯ + L(x − x) ¯ − d ∈ K.
¯ μ) Let (x, ¯ λ, ¯ ∈ X × W ∗ × Rm and define the operator G from X × W ∗ × Rm × Z ∗ to X × W × Rm × Z by G(x, ¯ λ¯ , μ)(x, ¯ λ, μ, η) ⎛ ⎞ ⎛ ⎞ ¯ μ)(x f (x) ¯ L (x, ¯ λ, ¯ − x) ¯ + e (x) ¯ ∗ λ + g (x) ¯ ∗ μ + L∗ η ⎜ −e(x) ⎜ ⎟ ¯ ⎟ −e (x)(x ¯ − x) ¯ ⎟+⎜ ⎟. =⎜ ⎝−g(x) ⎠ ¯ ⎠ ⎝ −g (x)(x ¯ − x) ¯ −(x) ¯ −L(x − x) ¯ Note that
⎛
⎞ ⎛ a ⎜ b ⎟ ⎜ ⎟ ¯ μ)(x, 0∈⎜ ¯ λ, ¯ λ, μ, η) + ⎜ ⎝ c ⎠ + G(x, ⎝ d
⎞ 0 ⎟ 0 ⎟ ∂ψRm,+ (μ) ⎠ ∂ψK + (η)
(2.4.15)
is the first order optimality system for (2.4.14). The following modification of (H2) will be used
(H2) there exists κ > 0 such that L (x0 , λ0 , μ0 ) x, x X ≥ κ|x|2X for all x ∈ ker(E+ ). , (H3) hold at a local solution of (2.1.1) and that Corollary 2.18. Assume that (H1), (H2), the second derivatives of f , e, and g are Lipschitz continuous in a neighborhood of x0 . Then there exist neighborhoods U (ξ0 ) of ξ0 = (x0 , λ0 , μ0 , η0 ) in X × W ∗ × Rm × Z ∗ , ˜ Uˆ (x0 , λ0 , μ0 ) of (x0 , λ0 , μ0 ), and V of the origin in X × W × Rm × Z and a constant K, ¯ μ) such that for all (x, ¯ λ, ¯ ∈ Uˆ (x0 , λ0 , μ¯ 0 )) and q = (a, b, c, d) ∈ V , there exists a unique ¯ μ, solution ξ = ξ(x, ¯ λ, ¯ q) = (x, λ, μ, η) ∈ U (ξ0 ) of (2.4.15), and x is a local solution of (2.4.14). Moreover, for every pair q1 , q2 ∈ V and (x¯1 , λ¯ 1 , μ¯ 1 ), (x¯2 , λ¯ 2 , μ¯ 2 ) ∈ Uˆ (x0 , λ0 , μ¯ 0 ) we have ˜ x¯1 , λ¯ 1 , μ¯ 1 ) − (x¯2 , λ¯ 2 , μ¯ 2 )| + |q1 − q2 |). |ξ(x¯1 , λ¯ 1 , μ¯ 1 , q1 ) − ξ(x¯2 , λ¯ 2 , μ¯ 2 , q2 )| ≤ K(|( Proof. From the first part of the proof of Theorem 2.16 it follows that ⎛ ⎞ ⎛ a 0 ⎜ b ⎟ ⎜ 0 ⎟ ⎜ 0∈⎜ ⎝ c ⎠ + G(x0 , λ0 , μ0 )(x, λ, μ, η) + ⎝ ∂ψRm,+ (μ) d ∂ψK + (η)
⎞ ⎟ ⎟ ⎠
i
i i
i
i
i
i
2.5. Differentiability
ItoKunisc 2008/6/12 page 53 i
53
admits a unique solution ξ = (x, λ, μ, η) which depends on (a, b, c, d) Lipschitz continuously provided that (a, b, c, d) is sufficiently small. The proof of the corollary can be completed by observing that the estimates in the proof of Theorem 2.16 also hold uniformly ¯ μ) if (x0 , λ0 , μ0 ) is replaced by (x, ¯ λ, ¯ in a sufficiently small neighborhood Uˆ (x0 , λ0 , μ0 ) of (x0 , λ0 , μ0 ), and (a, b, c, d) is in a sufficiently small neighborhood V of the origin in X × W × Rm × Z. As an alternative to reconsidering the proof of Theorem 2.16 one can apply Theorem 2.16 to (2.1.1) with P = (X × W ∗ × Rm ) × (X × W × Rm × Z), p = (x, λ, μ, a, b, c, d), p0 = (x0 , λ0 , μ0 , 0), and f, e, g, replaced by ¯ μ, ¯ μ)(x f˜(x, x, ¯ λ, ¯ a) = 12 L (x, ¯ λ, ¯ − x, ¯ x − x) ¯ + f (x) ¯ + a, x − x , ¯
e(x; ˜ x, ¯ b) = e(x) ¯ + e (x)(x ¯ − x) ¯ − b, g(x; ˜ x, ¯ c) = g(x) ¯ + g (x)(x ¯ − x) ¯ − c, ˜ x, (x; ¯ d) = (x) ¯ + L(x − x) ¯ − d. Clearly (H1)–(H3) hold for e, ˜ g, ˜ ˜ at (x0 , p0 ) with p0 = (x0 , λ0 , μ0 , 0). Lipschitz continuity ¯ μ, of f˜, e, ˜ g, ˜ ˜ with respect to (x, ¯ λ, ¯ a, b, c, d) is a consequence of the general regularity requirements on f, e, g, made at the beginning of the chapter and the assumption that the second derivatives of f and e are Lipschitz continuous in a neighborhood of x0 . This ˜ implies (H4) for e, ˜ f˜, g, ˜ .
2.5
Differentiability
In this section we discuss the differentiability properties of the solution ξ(p) = (x(p), λ(p), μ(p), η(p)) of the optimality system (2.1.3) with respect to p. Throughout it is assumed that p ∈ N so that (2.1.3) has the unique solutions ξ(p) in U , where N and U are specified in Theorem 2.16. We assume that C = X. Definition 2.19. A function φ from P to a normed linear space X˜ is said to have a directional derivative at p0 ∈ P if φ(p0 + t q) − φ(p0 ) lim t→0+ t exists for all q ∈ P . Consider the linear generalized equation ⎧ ∗ ∗ ∗ ⎪ ⎪ L (x0 , p, λ0 , μ0 , η0 ) + A(x − x0 ) + E (λ − λ0 ) + G (μ − μ0 ) + L (η − η0 ), ⎨ −e(x0 , p) − E(x − x0 ), 0∈ −g(x ⎪ 0 , p) − G(x − x0 ) + ∂ψRm,+ (μ), ⎪ ⎩ −(x0 , p) − L(x − x0 ) + ∂ψK + (η). (2.5.1) In the proof of Theorem 2.16 strong regularity of (2.1.4) at ξ(0) = (x0 , λ0 , μ0 , η0 ) was established. Therefore it follows from Theorem 2.5 that there exist neighborhoods of p0
i
i i
i
i
i
i
54
ItoKunisc 2008/6/12 page 54 i
Chapter 2. Sensitivity Analysis
and ξ(p0 ) which, without loss of generality, we again denote by N and U , such that (2.5.1) ˆ admits a unique solution ξˆ (p) = (x(p), ˆ λ(p), μ(p), ˆ η(p)) ˆ in U for every p ∈ N . Moreover we have |ξ(p) − ξˆ (p)| ≤ α(p) |p − p0 |,
(2.5.2)
where α(p) : N → R+ satisfies α(p) → 0 as p → p0 . Thus if ξˆ has a directional derivative at p0 , then so does ξ(p) and the directional derivatives coincide. We henceforth concentrate on the analysis of directional differentiability of ξˆ (p). From (2.5.2) and Lipschitz continuity if p → ξ(p) at p0 , it follows that for every q ∈ P
ξˆ (p + t q) − ξˆ (p ) 0 0
lim sup
< ∞.
t t>0 ˆ
ˆ
ξ (p0 ) Hence there exist weak cluster points of ξ (p0 +t q)− as t → 0+ . Note that these weak t ξ(p0 +t q)−ξ(p0 ) cluster points coincide with those of as t → 0+ . We will show that under t appropriate conditions, these weak cluster points are strong limit points and we derive equations that they satisfy. The following definition and additional hypotheses will be used.
Definition 2.20. A closed convex set C of a Hilbert space H is called polyhedric at z ∈ H if ∪λ>0 λ(C − P z) ∩ [z − P z]⊥ = ∪λ>0 λ(C − P z) ∩ [z − P z]⊥ , where P denotes the metric projection onto C and [z − P z]⊥ stands for the orthogonal complement of the subspace spanned by x − P z ∈ H . Moreover C is called polyhedric if C is polyhedric at every z ∈ H . (H5) e(x0 , ·), (x0 , ·), f (x0 , ·), e (x0 , ·), g (x0 , ·), and (x0 , ·) are directionally differentiable at p0 . (H6) K is polyhedric at (x0 , p0 ) + η0 . (H7) There exists ν > 0 such that Ax, x ≥ ν |x|2 for all x ∈ ker E. (H8) EL : X → W × Z is surjective. Since every element z ∈ Z can be decomposed uniquely as z = z1 + z2 with z1 = PK z and z2 = PK + z and z1 , z2 = 0 (see [Zar]), (H.6) is equivalent to ∪λ>0 λ(K − (x0 , p0 )) ∩ [η0 ]⊥ = ∪λ>0 λ(K − (x0 , p0 )) ∩ [η0 ]⊥ . Recall the decomposition of the inequality constraint with finite rank and the notation that was introduced in (2.1.6). Due to the complementarity condition and continuous dependency of ξ(p) on p one can assume that g + (x(p), p) = 0, g − (x(p), p) < 0, μ+ (p) > 0, μ− (p) = 0
(2.5.3)
for p ∈ N. This also holds with (x(p), μ(p)) replaced by (x(p), ˆ μ(p)). ˆ
i
i i
i
i
i
i
2.5. Differentiability
ItoKunisc 2008/6/12 page 55 i
55
˙ μ, Theorem 2.21. Let (H1)–(H5) hold and let (x, ˙ λ, ˙ η) ˙ denote a weak cluster point of ξ(p0 +t q)−ξ(p0 ) + ˙ as t → 0 . Then ( x, ˙ λ, μ, ˙ η) ˙ satisfies t ⎧ Lp (x0 , po , λ0 , μ0 , η0 )q + Ax˙ + E ∗ λ˙ + G∗ μ˙ + L∗ η, ˙ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ −ep+ (x0 , p0 )q − E+ x, ˙ ⎪ ⎪ ⎪ ⎪ ⎨ 0∈ −gp0 (x0 , p0 ) − G0 x˙ + ∂ψRm2 ,+ (μ˙ 0 ), ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ μ˙ − , ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ η, ˙ (x0 , p0 ) + η0 , Lx˙ + p (x0 , p0 )q .
(2.5.4)
ˆ
ˆ
ξ (p0 ) Proof. As described above, we can restrict ourselves to a weak cluster point of ξ (p0 +t q))− . t We put pn = p0 + tn q. By (2.5.1)
0 = L (x0 , pn , λ0 , μ0 , η0 ) − L (x0 , p0 , λ0 , μ0 , η0 ) ˆ n ) − λ0 ) + G∗ (μ(p +A(x(p ˆ n ) − x0 ) + E ∗ (λ(p ˆ n ) − μ0 ) + L∗ (η(p ˆ n ) − η0 ) = 0, 0 = −e(x0 , pn ) + e(x0 , p0 ) − E(x(p ˆ n ) − x0 ) = 0. Dividing these two equations by tn > 0 and letting tn → 0+ we obtain the first two equations in (2.5.4). For the third equation note that μ˙ 0 ≥ 0 since μˆ 0 (pn ) ≥ 0 and μ0 (p0 ) = 0. Since g 0 (x0 , p0 ) = 0 we have from (2.5.1) g 0 (x0 , pn ) − g 0 (x0 , p0 ) + G0 (x(p ˆ n ) − x0 ), z − (μˆ 0 (pn ) − μ0 (p0 ) ≥ 0 for all z ∈ Rm2 ,+ . Dividing this inequality by tn > 0 and letting tn → 0+ we obtain the third inclusion. The fourth equation is obvious. For the last equation we recall that η(p ˆ n ), (x0 , pn ) + L(x(p ˆ n ) − x0 ) = η0 , (x0 , p0 ) = 0. Thus, η(p ˆ n ) − η0 , (x0 , pn ) + η(p ˆ n ), L(x(p ˆ n ) − x0 ) + η0 , (x0 , pn ) − (x0 , p0 ) = 0, which implies the last equality. Define the value function V (p) V (p) = inf {f (x, p) : e(x, p) = 0, g(x, p) ≤ 0, (x, p) ∈ K}. x∈c
Then we have the following corollary. Corollary 2.22 (Sensitivity of Cost). Let (H.1)–(H.4) hold and assume that e, g, are continuously differentiable in the sense of Fréchet at (x0 , p0 ). Then the Gâteaux derivative
i
i i
i
i
i
i
56
ItoKunisc 2008/6/12 page 56 i
Chapter 2. Sensitivity Analysis
of the value function μ at p0 exists and is given by V (p0 ) = Lp (x0 , p0 , λ0 , μ0 , η0 ) = fp (x0 , p0 ) + λ0 , ep (x0 , p0 ) + μ0 , gp (x0 , p0 ) + η0 , p (x0 , p0 ) . Proof. Note that V (p) = f (x(p), p) = L(x(p), p, λ(p), μ(p), η(p)). For q ∈ P let ξ(t) = t −1 (x(p0 + t q) − x(p0 )), t > 0. Then there exists a subsequence tn ˙ μ, such that limn→∞ tn = 0 and ξ(tn ) → (x, ˙ λ, ˙ η). ˙ Now, let pn = p0 + tn q and V (pn ) − V (p0 ) = L(x(pn ), pn , λ(pn ), μ(pn ), η(pn )) −L(x(pn ), p0 , λ(pn ), μ(pn ), η(pn )) + L(x(pn ), p0 , λ(pn ), μ(pn ), η(pn )) −L(x(pn ), p0 , λ(p0 ), μ(p0 ), η(p0 )) + L(x(pn ), p0 , λ(p0 ), μ(p0 ), η(p0 )) −L(x(p0 ), p0 , λ(p0 ), μ(p0 ), η(p0 )). Thus, V (pn )−V (p0 ) tn
= L (x0 , p0 , λ0 , μ0 , η0 ), x
˙ + L(x0 , p0 , λ0 , μ0 , η0 )p , q
˙ e(x0 , p0 ) + μ, ˙ g(x0 , p0 ) + η, ˙ (x0 , p0 ) + O(tn ). + λ, ˙ e(x0 , p0 ) = 0 and considering separately the cases according to (2.5.3) Clearly λ, we find μ, ˙ g(x0 , p0 ) = 0 as well. Below we verify that η0 , Lx˙ + p (x0 , p0 )q = 0.
(2.5.5)
From Theorem 2.21 this implies that η, ˙ (x0 , p0 ) = 0. To verify (2.5.5) note that from (2.5.1) we have ˆ n ) − x0 ) ≤ 0 η(t ˆ n ), (x0 , p0 ) − (x0 , pn ) − L(x(t and consequently η0 , p (x0 , p0 )q + L x
˙ ≥ 0. Moreover ˆ n ) − x0 ) ≤ 0 η0 , (x0 , pn ) − (x0 , p0 ) + L(x(t ˙ ≤ 0 and (2.5.5) holds. It follows that and therefore η0 , p (x0 , p0 )q + L x
lim
n→∞
V (pn ) − V (p0 ) = Lp (x0 , p0 , λ0 , μ0 , η0 ), q . tn
Since the limit does not depend on the sequence {tn } the desired result follows.
i
i i
i
i
i
i
2.5. Differentiability
ItoKunisc 2008/6/12 page 57 i
57
In order to prove strong differentiability of x(p) we use a result due to Haraux [Har] on directional differentiability of the metric projection onto a closed convex set. Theorem 2.23. Let C be a closed convex set in a Hilbert space H with metric projection P from H onto C. Let ψ be an H -valued function and assume that C is polyhedric at ψ(0). Set ˆ K(ψ(0)) = ∪λ>0 λ(C − P ψ(0)) ∩ [ψ(0) − P ψ(0)]⊥ , ˆ and denote by PK(ψ(0)) the projection onto K(ψ(0)). If there exists a sequence tn such that ˆ ψ(tn )−ψ(0) + ˙ exists in H , then =: ψ limn→∞ tn → 0 and limn→∞ tn lim
tn →0+
P ψ(tn ) − P ψ(0) ˙ = PK(ψ(0)) ψ. ˆ tn
ψ(0) Proof. Let γ (t) = P ψ(t)−P . Since the metric projection onto a closed convex set in a t Hilbert space is a contraction, γ (tn ) has a weak limit which is denoted by γ . Since
ψ(tn ) − P ψ(tn ), P ψ(0) − P ψ(tn ) ≤ 0 and P ψ(tn ) = tn γ (tn ) + P ψ(0), we obtain ψ(tn ) − tn γ (tn ) − P ψ(0), −tn γ (tn ) ≤ 0. This implies tn and hence
ψ(tn ) − ψ(0) + ψ(0) − tn γ (tn ) − P ψ(0), −tn γ (tn ) ≤ 0 tn
tn2
ψ(tn ) − ψ(0) γ (tn ), γ (tn ) − ≤ tn γ (tn ), ψ(0) − P ψ(0)
tn
(2.5.6)
= P ψ(tn ) − P ψ(0), ψ(0) − P ψ(0) ≤ 0 for all n. Since the norm is weakly lower semicontinuous, (2.5.6) implies that ˙ ≤ 0. γ , γ − ψ
(2.5.7)
Employing (2.5.6) again we have 0 ≥ γ tn ), ψ(0) − P ψ(0) ≥ tn
ψ(tn ) − ψ(0) γ (tn ), γ (tn ) − tn
for tn > 0. Since {γ (tn )} is bounded, this implies γ , ψ(0) − P ψ(0) = 0, ˆ and thus γ ∈ K(ψ(0)).
i
i i
i
i
i
i
58
ItoKunisc 2008/6/12 page 58 i
Chapter 2. Sensitivity Analysis
Let w ∈ ∪λ>0 λ(C − P ψ(0)) ∩ [ψ(0) − P ψ(0)]⊥ be arbitrary. Then there exist λ > 0 and u ∈ C such that w = λ (u − P ψ(0)) and ψ(0) − P ψ(0), u − P ψ(0) = 0. Since ψ(tn ) − tn γ (tn ) − P ψ(0), u − P ψ(0) − tn γ (tn ) ≤ 0, we find for δn = γ (tn ) − γ that ψ(tn ) − ψ(0) + ψ(0) − P ψ(0) − tn γ − tn δn , u − P ψ(0) − tn γ − tn δn ≤ 0. tn tn Thus, ψ(tn ) − ψ(0) tn − tn γ − tn δn , u − P ψ(0) − tn γ − tn δn ≤ ψ(0) − P ψ(0), tn δn
tn and therefore ψ(tn ) − ψ(0) − γ , u − P ψ(0) ≤ δn , u − P ψ(0) + ψ(0) − P ψ(0), δn + tn M tn for some constant M. Letting tn → 0+ we have ψ˙ − γ , u − P ψ(0) ≤ 0. Since C is polyhedric at ψ(0) it follows that ψ˙ − γ , w ≤ 0
ˆ for all w ∈ K(ψ(0)).
(2.5.8)
Combining (2.5.7) and (2.5.8) one obtains ˙ γ − w ≤ 0 γ − ψ,
ˆ for all w ∈ K(ψ(0)),
ψ˙ and thus γ = PK(ψ(0)) ˆ To show strong convergence of γ (tn ) observe that from (2.5.7) and (2.5.8) ψ˙ − γ , γ = 0. Hence ˙ γ = |γ |2 ≤ lim inf |γ (tn )|2 ≤ lim sup ψ,
ψ(tn ) − ψ(0) ˙ γ , , γ (tn ) = ψ, tn
which implies that lim |γ (tn )|2 = |γ |2 and completes the proof. Theorem 2.24 (Sensitivity Equation). Let (H1)–(H8) hold. Then the solution mapping p → ξ(p) = (x(p), λ(p), μ(p), η(p)) is directionally differentiable at p0 , and the directional derivative (x, ˙ λ˙ , μ, ˙ η) ˙ at p0 in direction q ∈ P satisfies ⎧ Lp (x0 , p0 , λ0 , μ0 , η0 )q + Ax˙ + E ∗ λ˙ + G∗+ μ˙ + G∗0 μ˙ 0 + L∗ η, ˙ ⎪ ⎪ ⎨ ˙ −ep+ (x0 , p0 )q − E+ x, 0∈ (2.5.9) −gp0 (x0 , p0 )q − G0 x˙ + ∂ψRm,+ (μ˙ 0 ), ⎪ ⎪ ⎩ −p (x0 , p0 )q − Lx˙ + ∂ψKˆ + (η). ˙
i
i i
i
i
i
i
2.5. Differentiability
ItoKunisc 2008/6/12 page 59 i
59
Proof. Let {tn } be a sequence of real numbers with limn→∞ tn = 0+ and w−lim tn−1 (ξ(p0 + ˙ μ, ˙ μ, tn q) − ξ(p0 )) = (x, ˙ λ, ˙ η). ˙ Then (x, ˙ λ, ˙ η) ˙ is also a weak cluster point of w − −1 ˆ lim tn (ξ (p0 + tn q) − ξ(p0 )). The proof is now given in several steps. ˆ n )−x(0) converges strongly in X as n → ∞ and (2.5.9) holds for We first show that x(t tn all weak cluster points. Let pn = p0 + tn q and define φ(tn ) = L (x0 , pn , λ0 , μ0 , η0 ) + f (x0 , p0 ) − Ax0 . From (2.5.1) we have ⎧ ˆ n )) ˆ n ) − x0 ) + E ∗ (λ(t L (x0 , pn , λ0 , μ0 , η0 ) + f (x0 , p0 ) + A(x(t ⎪ ⎪ ⎪ ∗ ∗ ⎪ ˆ n )) + L (η(t ˆ n )), +G (μ(t ⎨ 0∈ −e(x0 , pn ) − E(x(t ˆ n ) − x0 ), ⎪ ⎪ ˆ n ) − x0 ) + ∂ψRm,+ (μ(t ˆ n )), ⎪ −g(x0 , pn ) − G(x(t ⎪ ⎩ −(x0 , pn ) − L(x(t ˆ n ) − x0 ) + ∂ψK + (η(t ˆ n )).
(2.5.10)
By (H8) there exists a unique w(tn ) ∈ range (E ∗ , L∗ ) such that ⎛ ⎝
E
⎞
⎛
⎠ w(tn ) = ⎝
L
⎞
−e(x0 , pn ) + Ex0
⎠.
−(x0 , pn ) + Lx0
˙ If we Due to (H5) and (H8) there exists w˙ ∈ X such that lim tn−1 (w(tn ) − w(0)) = w. ˆ n ) − w(tn ), then using Lemma 2.7 one verifies that y(tn ) ∈ C, where define y(tn ) = x(t C = {x ∈ X : Ex = 0 and Lx ∈ K}. Observe that ˆ n ) + L∗ η(t ˆ n ), c − yn (tn ) ≤ 0 E ∗ λ(t
for all c ∈ C
and thus Ayn (t) + φ(tn ) + Aw(tn ) + G∗ μ(t ˆ n ), c − y(tn ) ≥ 0
for all c ∈ C.
(2.5.11)
If we equip ker E with the inner product ((x, y)) = Ax, y
for x, y ∈ ker E,
then ker E is a Hilbert space by (H7). Let AP = Pker E A Pker E and define ∗ ψ(tn ) = −A−1 P Pker E (φ(tn ) + Aw(tn ) + G μ(tn )) ∈ ker E.
Then limtn →0+
ψ(tn )−ψ(0) tn
=: ψ˙ exists, and (2.5.11) is equivalent to
((y(tn ) − ψ(tn ), c − y(tn ))) ≥ 0
for all c ∈ C.
i
i i
i
i
i
i
60
ItoKunisc 2008/6/12 page 60 i
Chapter 2. Sensitivity Analysis
Thus y(tn ) = PC ψ(tn ), where PC denotes the metric projection in ker E onto PC with respect to the inner product ((·, ·)). For any h ∈ ker E we find ((h, ψ(0) − PC ψ(0) )) = h, −f (x0 , p0 ) + A xo − A w(0) − G∗ μ0 − A y(0)
= h, −f (x0 , p0 ) − G∗ μ0 − E ∗ λ0 = h, L∗ η0 , and therefore [ψ(0) − PC ψ(0)]⊥ = {h : η0 , L h = 0}.
(2.5.12)
Here the orthogonal complement is taken with respect to the ((·, ·))—the inner product on ker E. Moreover we have L(y(0) = (x0 , p0 ). Hence from Lemma 2.25 below we have ∪λ>0 λ(C − PC ψ(0)) ∩ [ψ(0) − PC ψ(0)]⊥ = ∪λ>0 λ(C − PC ψ(0)) ∩ [ψ(0) − PC ψ(0)]⊥ , i.e., C is polyhedric with respect to ψ(0). Theorem 2.23 therefore implies that limtn →o+ (tn−1 )(y(tn ) − y(0)) = : y˙ exists and satisfies ˆ ((ψ˙ − y, ˙ v − y)) ˙ ≥ 0 for all v ∈ C(ψ(0)).
(2.5.13)
˙ Moreover limtn →0 (tn−1 )(x(tn ) − x(0)) exists and will be denoted by x. ˙ μ, To verify that (x, ˙ λ, ˙ η) ˙ satisfies (2.5.9) it suffices to prove the last inclusion. From (2.5.13) we have −L p (x0 , p0 , λ0 , μ0 , η0 )q − A x˙ − G∗ μ, ˙ v − y
˙ ≤0 and from the first equation of (2.5.4) L∗ η, ˙ v − y
˙ = η, ˙ L v − p (x0 , p0 )q ≤ 0
(2.5.14)
ˆ for all v ∈ C(ψ(0)) = ∪λ>0 λ(C − PC ψ(0)) ∩ [ψ(0) − PC ψ(0)]⊥ . Here we used the facts that φ˙ = Lp (x0 , λ0 , μ0 , ηo )q and L w˙ = −p (x0 , p0 )q. From (2.5.12), (H8), and L y(0) = (x0 , p0 ) we have ˆ L C(ψ(0)) = ∪λ>0 λ(K − (x0 , p0 ) ∩ [η0 ]⊥ = : Kˆ and therefore by (2.5.14) ˆ η, ˙ v − L x˙ − p(x0 , p0 )q ≤ 0 for all v ∈ K.
(2.5.15)
Recall from (2.5.5) that η0 , p (xo , p0 )q + L x
˙ = 0. ˆ and from (2.5.15) we deduce η˙ ∈ Kˆ + . ConseThis further implies p (x0 , p0 )q + L x˙ ∈ K, ˙ which is equivalent to p(xo , p0 )q +L x˙ ∈ ∂ ψKˆ + (η) ˙ quently η˙ ∈ ∂ ψKˆ (p (x0 , p0 )q +L x), and implies the last inclusion in (2.5.9).
i
i i
i
i
i
i
2.5. Differentiability
ItoKunisc 2008/6/12 page 61 i
61 ˆ
Next, we show that the weak cluster points of limt→0+ ξ (t)−ξ(0) are unique. Let t ˙ (x˙i , λi , μ˙ i , η˙ i ), i = 1, 2, be two weak cluster points. Then by (2.5.9) 0 = A(x˙1 − x˙2 ), x˙1 − x˙2 + μ˙ 1 − μ˙ 2 , G(x˙1 − x˙2 ) + η˙ 1 − η˙ 2 , L(x˙1 − x˙2 )
= A(x˙1 − x˙2 ), x˙1 − x˙2 + μ˙ 01 , −gp0 (x0 , p0 )q − G0 x˙2 + μ˙ 02 , −gp0 (x0 , p0 )q − G0 x˙1
+ η˙ 1 , −p (x0 , p0 )q − Lx˙2 + η˙ 2 , −p (x0 , p0 )q − Lx˙1
≥ A(x˙1 − x˙2 ), x˙1 − x˙2 , and by (H2) this implies x˙1 = x˙2 . From (2.5.9) we have ⎛ ˙ ⎞ λ˜ 1 − λ˙˜ 2 E ∗ ((x0 , p0 )) ⎝ μ˙ 01 − μ˙ 02 ⎠ = 0 η˙ 1 − η˙ 2 ˆ and hence (H3) implies (λ˙ 1 , μ˙ 1 , η˙ 1 ) = (λ˙ 2 , μ˙ 2 , η˙ 2 ). Consequently ξ (t)−ξ(0) has a unique t weak limit as t → 0+ . Finally we show that the unique weak limit is also strong. For the x-component this was already verified. Note that from (2.5.1) ⎞ ⎛ ⎛˜ ⎞ ˆ − x0 ) −L (x0 , p0 + t q, λ0 , μ0 , η0 ) − A(x(t) ˆ − λ˜ 0 λ(t) ⎠. E ∗ ((x0 , p0 )) ⎝ μˆ 0 − μ00 ⎠ = ⎝ ˆ − x0 )
η(t), ˆ −(x0 , p + t q) + (x0 , p0 ) − L(x(t) η(t) ˜ − η0
Dividing this equation by t and noticing that the right-hand side converges strongly as ˜ − λ0 , μ(t) t → 0+ , it follows from (H3) that limt→0+ t −1 (λ(t) ˜ − μ0 , η(t) ˜ − η0 ) converges strongly as well. Lemma 2.25. Assume that (H6) and (H8) hold and let y(0) ∈ C be such that L y(0) = (x0 , p0 ). Then we have ∪λ>0 λ(C − y(0)) ∩ {h ∈ ker E : η0 , L h = 0} (2.5.16) = ∪λ>0 λ(C − y(0)) ∩ {h ∈ ker E : η0 , L h = 0}. Proof. It suffices to verify that the set on the right-hand side of (2.5.16) is contained in the set on the left. For this purpose let y ∈ ∪λ>0 λ(C − y(0)) ∩ {h ∈ ker E : η0 , L h = 0} and decompose y = y1 + y2 with y1 ∈ ker( EL ) and y2 ∈ (ker( EL ))⊥ . We need only consider y2 . Since L y2 ∈ ∪λ→0 λ(K − (x0 , p0 )) ∩ [η0 ]⊥ and therefore L y2 ∈ ∪λ>0 λ(K − (x0 , p0 )) ∩ [η0 ]⊥ by (H6), hence there exist sequences {λn } and {kn } with λn > 0, kn ∈ K, kn , η0 = 0, and lim λn (kn − (x0 , p0 )) = L y2 . Let us define cn = c˜n +y(0), where c˜n is the unique element in ( EL )∗ satisfying ( EL )c˜n = ( kn −L0 y(0) ). Then we have E(cn −y(0)) = 0, η0 , L cn = η0 , kn = 0 and hence the sequence λn (cn −y(0)) is contained in the set on the left-hand side of (2.5.16). Moreover limn→∞ ( EL )(λn (cn −y(0)) = ( EL )y2 , and since both λn (cn − y(0)) and y2 are contained in the range of ( EL )∗ we find that limn→∞ λn (cn − y(0)) = y2 .
i
i i
i
i
i
i
62
ItoKunisc 2008/6/12 page 62 i
Chapter 2. Sensitivity Analysis
2.6 Application to optimal control of an ordinary differential equation We consider optimal control of the linear ordinary differential equation with a terminal constraint T min fˆ(y(t), u(t), α)dt 0
subject to y(t) ˙ = A(α)y(t) + B(α)u(t) + h(t) on (0, T ),
(2.6.1)
y(0) = 0, |y(T ) − y d | ≤ δ, u≤z
a.e. on (0, T ),
where T > 0, y(t) ∈ Rn , u(t) ∈ Rn , α ∈ Rr , y d ∈ Rn , δ > 0, z ∈ L2 , A(α) ∈ Rn×n , B(α) ∈ Rn×m , fˆ : Rn ×Rm ×Rr → R, and h ∈ L2 (0, T ). All function spaces of this section are considered on the interval (0, T ) with Euclidean image space of appropriate dimension. The perturbation parameter is given by p = (α, y d , h, z) ∈ P = Rr × Rn × L2 × L2 with reference parameter p0 = (α0 , y0d , h0 , z0 ). To identify (Pp ) as a special case of (2.1.1) we set X = HL1 × L2 , W = L2 , Z = L2 , K = {φ ∈ L2 : φ ≤ 0 a.e.}, where HL1 = {φ ∈ H 1 : φ(0) = 0}, x = (y, u), and T f (y, u, p) = fˆ(y, u, α)dt, 0
e(y, u, p) = y˙ − A(α)y − B(α)u − h, g(y, u, p) =
1 |y(T ) − y d |2 − δ 2 , 2
(y, u, p) = u − z. The linearizations E, G, and L are E(y0 , u0 , p0 )(h, v) = h˙ − A(α0 )h − B(α0 )v, G(y0 , u0 , p0 )(h, v) = y0 (T ) − y0d , h(T ) , L(y0 , u0 , p0 )(h, v) = v. The following assumptions will imply conditions (H1)–(H8) of this chapter. (A1) There exists a solution (y0 , u0 ) of (2.6.1).
i
i i
i
i
i
i
2.6. Application to optimal control of an ODE
ItoKunisc 2008/6/12 page 63 i
63
(A2) A(·) and B(·) are Lipschitz continuous and directionally differentiable at α0 . (A3) (y, u) → f (y, u, α) is twice continuously differentiable in a neighborhood of (y0 , u0 , α0 ) ∈ HL1 × L2 × Rr . (A4) There exists > 0 such that for the Hessian with respect to (y, u) (h, v) fˆ (y0 , u0 , α0 ) (h, v)T ≥ |v|2
for all (h, v) ∈ Rn × Rm . (A5) There exist a neighborhood Vˆ of (y0 , u0 , α0 ) ∈ HL1 × L2 × Rr and a constant κ such that |f (y, u, α1 ) − f (y, u, α2 )| ≤ κ|α1 − α2 | for all (y, u, αi ) ∈ Vˆ , i = 1, 2, and f (y0 , u0 , ·) is directionally differentiable at α0 , where the prime denotes the derivative with respect to (y, u). T (A6) y0 (T ) − y0d , 0 eA(α0 )(T −s) B(α0 ) (z0 − u0 )ds = 0. With (A3) holding, the regularity requirements made at the beginning of the chapter are satisfied. The regular point condition (H1) will be satisfied if for every (φ, ρ, ψ) ∈ L2 × R × L2 there exist (h, v) ∈ HL1 × L2 and (r + , r, k) ∈ R+ × R × K such that ⎧ h˙ − A(α0 )h − B(α0 )v = φ, ⎪ ⎪ ⎪ ⎪ ⎨ (2.6.2) y0 (T ) − y0d , h(T ) + r + + r g(y0 , u0 , p0 ) = ρ, ⎪ ⎪ ⎪ ⎪ ⎩ v − k + r(u0 − z0 ) = ψ. From the first and third equations we have v = ψ + k + r(z0 − u0 ),
t
h(t) = ρ1 (t) + t
eA(α0 )(t−s) B(α0 )(k + r(z0 − u0 ))dt,
0
where ρ1 (t) = 0 e y0 (T ) − y0α ,
(B(α0 )ψ +φ)ds. The second equation in (2.6.2) is equivalent to eA(α0 )(T −s) B(α0 )(k + r(z0 − u0 ))ds + r + + r g(y0 , u0 , p0 )
A(α0 )(t−s) T
0
= ρ − ρ1 (T ). If g(y0 , u0 , p0 ) = 0, then, using (A6), the desired solution for (2.6.2) is obtained by setting r + = k = 0 and −1 T d A(α0 )(T −s) e B(α0 )(z0 − u0 )ds (ρ − ρ1 (T )). r = y0 (t) − yo , 0
If g(y0 , u0 , p0 ) < 0 and ρ−ρ1 ≥ 0, then the choice r + = ρ−ρ1 , r = k = 0 gives the desired solution. Finally if g(y0 , u0 , p0 ) < 0 and ρ − ρ1 < 0, then r = (ρ − ρ1 )g(y0 , u0 , p0 ) > 0,
i
i i
i
i
i
i
64
ItoKunisc 2008/6/12 page 64 i
Chapter 2. Sensitivity Analysis
r + = 0, and k = r(u0 − u) ∈ K give a solution for (2.6.2). Thus the regular point condition holds and implies the existence of a Lagrange multiplier (λ0 , μ0 , η0 ) ∈ L2 × R × L2 . For the Lagrangian functional T L(y, u, p0 , λ0 , μ0 , η0 ) = fˆ(y, u, α0 )dt + λ0 , y˙ − A(α0 )y − B(α0 )u − h0
0
+
μ0 (|y(T ) − y0d | − δ 2 |) − η0 , u − z0 , 2
the Hessian with respect to (y, u) at (y0 , u0 ) in directions (h, v) ∈ HL1 × L2 satisfies
L (y0 , u0 , p0 , λ0 , μ0 , η0 )((h, v), (h, v))
T
= 0
(h(t), v(t))fˆ (y0 , u0 , α0 )(h(t), v(t))T dt + μ0 |h(T )|2 ≥ |v|2L2
by (A4). Let k1 > 0 be chosen such that |h|HL1 ≤ k1 |v|L2 for all (h, v) ∈ HL1 × L2 in ker E. Then there exists a constant k2 > 0 such that
L (y0 , u0 , p0 , λ0 , μ0 , η0 )((h, v), (h, v)) ≥ k2 (|h|2H 1 + |v|1L2 ) L
for all (h, v) ∈ ker E and (H2), (H7) hold. We turn to the verification of (H3). The case g(y0 , u0 , α0 ) < 0 is simple and we therefore consider the case g(y0 , u0 , α0 ) = 0. For arbitrary (φ, ρ, ψ) ∈ L2 × R × L2 existence of a solution (h, v, r) ∈ H11 × L2 × R to ⎧ h˙ − A(α0 )h − B(α0 )v = φ, ⎪ ⎪ ⎪ ⎪ ⎨ (2.6.3) y0 (T ) − y0d , h(T ) = ρ, ⎪ ⎪ ⎪ ⎪ ⎩ v + r(z0 − u0 ) = ψ must be shown. From the first and third equations v = ψ − r(z0 − u0 ) and
t
h(t) = ρ1 (t) − r
eA(α0 )(t−s) B(α0 )(z0 − u0 )ds.
0
From the second equation of (2.6.3) and (H7) we find −1 T A(α0 )(T −s) d r = y0 (T ) − y0 , e B(α0 )(z0 − u0 )ds 0
( y0 (T ) − y0d , ρ1 (T ) − ρ), and (H3) holds. Conditions (H4) and (H5) follow from (A2) and (A5). The cone of a.e. nonpositive functions in L2 is polyhedric [Har] and hence (H6) holds. Finally (H8) is simple to check. We note that (A5) and (A6) are not needed if (2.6.1) is considered without terminal constraint.
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 65 i
Chapter 3
First Order Augmented Lagrangians for Equality and Finite Rank Inequality Constraints 3.1
Generalities
This chapter is devoted to first order augmented Lagrangian methods for constrained problems of the type min f (x) over x ∈ X (3.1.1) subject to e(x) = 0, g(x) ≤ 0, (x) ∈ K, where f : X → R, e : X → W, g : X → Rm , : X → Z with X, W , and Z real Hilbert spaces, K a closed convex cone with vertex at 0 in Z, and Rm endowed with the natural ordering x ≤ 0 if xi ≤ 0 for i = 1, . . . , m. A sequence of unconstrained problems with the property that their solutions converge to the solution of (3.1.1) will be formulated. As we shall see, first order augmented Lagrangian techniques are related to Lagrangian or duality methods, as well as to the penalty technique. They arise from considering the Lagrangian functional for (3.1.1) and adding a proper penalty term. Differently from pure penalty methods, however, the penalty parameter is not taken to infinity. Rather the penalty term with fixed weight is used to enhance convexity of the Lagrangian functional in the neighborhood of a local minimum and to speed up convergence of pure Lagrangian methods. Affine inequality constraints with infinite-dimensional image space can be added to the problem formulation (3.1.1). Such constraints, however, are not augmented in this chapter. Central to the analysis is the augmentability property, which is also of interest in its own right. It has received a considerable amount of attention for finite-dimensional optimization problems; see, e.g., [Hes1]. Augmentability of (3.1.1) means that a properly defined penalty term involving the constraints in (3.1.1) can be added to the Lagrangian functional for (3.1.1) such that the resulting augmented Lagrangian functional has the property that its Hessian is positive definite on all of X under minimal assumptions on (3.1.1). We shall present a convergence theory for the first order augmented Lagrangian method applied to (3.1.1) without strict complementarity assumption. This means that we do not assume that μ∗i = 0 implies gi (x ∗ ) < 0, where x ∗ denotes a local solution of (3.1.1) and μ∗i is the ith coordinate of the Lagrange multiplier associated to the inequality constraint g(x) ≤ 0. As an application we shall discuss the fact that in the context of parameter estimation problems the first order 65
i
i i
i
i
i
i
66
ItoKunisc 2008/6/12 page 66 i
Chapter 3. First Order Augmented Lagrangians
augmented Lagrangian method can be considered as a hybrid method combining the outputmatching and equation error methods. In the context of parameter estimation or, equally well, of optimal control problems, the first order augmented Lagrangian method lends itself to vectorization and parallelization in a natural way; see [KuTa]. It will be convenient throughout this section to identify the Hilbert spaces X, W , and Z with their duals. As a consequence the Lagrange multiplier associated to the equality constraint e(x) = 0 is sought in W and that for (x) ∈ K in Z. Let us now fix those assumptions which are assumed throughout this chapter: There exists x ∗ ∈ X and (λ∗ , μ∗ , η∗ ) ∈ W × Rp × Z such that f, e, and g are twice continuously Fréchet differentiable (3.1.2) in a neighborhood of x ∗ , x ∗ is a stationary point of (3.1.1) with Lagrange multiplier (λ∗ , μ∗ ), i.e., (f (x ∗ ), h)X + λ∗ , e (x ∗ )h W + μ∗ , g (x ∗ )h Rm + η∗ , (x ∗ )h Z = 0 for all h ∈ X, and
(3.1.3)
-
e(x ∗ ) = 0, μ∗ , g(x ∗ ) Rm = 0, μ∗ ≥ 0, g(x ∗ ) ≤ 0, ∗ η , (x ∗ ) Z = 0, η∗ ∈ K + , (x ∗ ) ∈ K.
(3.1.4)
Moreover we assume that e (x ∗ ) : X → W is surjective. (3.1.5) Above ·, · W and ·, · Z denote the inner products in W and Z, respectively, and ·, · Rm is the usual inner product in Rm . If x ∗ is a local solution of (3.1.1), if further (3.1.2) holds and x ∗ is a regular point in the sense of Definition 1.5, then there exists a Lagrange multiplier (λ∗ , μ∗ , η∗ ) such that (3.1.3), (3.1.4) hold. Introducing the Lagrange functional L : X × W × Rm × Z → R by L(x, λ, μ, η) = f (x) + λ, e(x) W + μ, g(x) Rm + η, (x) Z ,
(3.1.3) can be expressed as L (x ∗ , λ∗ , μ∗ , η∗ )h = 0 for all h ∈ X. Here and below the prime denotes differentiation with respect to x. With (3.1.2) holding, L (x ∗ , λ∗ , μ∗ , η∗ ) : X × X → R exists as symmetric bilinear form given by L (x ∗ , λ∗ , μ∗ , η∗ )(h, k) = f (x ∗ )(h, k) + λ∗ , e (x ∗ )(h, k) W + μ∗ , g (x ∗ )(h, k) Rm for all (h, k) ∈ X×X. The index set of the coordinates of the finite rank inequality constraints is decomposed as I1 = {i : gi (x ∗ ) = 0, μ∗i = 0}, I2 = {i : gi (x ∗ ) = 0, μ∗i < 0}, I3 = {i : gi (x ∗ ) < 0}.
i
i i
i
i
i
i
3.2. Augmentability and sufficient optimality
ItoKunisc 2008/6/12 page 67 i
67
Clearly I1 ∪ I2 ∪ I3 = {1, . . . , m}. If I1 is empty, then we say that strict complementarity holds. Let us denote the cardinality of Ii by mi . Without loss of generality we assume that mi ≥ 1 for i = 1, 2, 3 and that I1 = 1, . . . , m1 , I2 = m1 + 1, . . . , m1 + m2 , and I3 = m1 + m2 + 1, . . . , m. The following second order sufficient optimality condition will be of central importance: There exists a constant γ > 0 such that ⎧ ⎪ L (x ∗ , λ∗ , μ∗ , η∗ )(h, h) ≥ γ |h|2X for all h ∈ C, ⎪ ⎨ where C = {h ∈ X : e (x ∗ )h = 0, gi (x ∗ )h ≤ 0 ⎪ ⎪ ⎩ for i ∈ I1 , gi (x ∗ )h = 0 for i ∈ I2 }.
(3.1.6)
In case X is finite-dimensional, (3.1.6) is well known to imply that x ∗ is a strict local solution to (3.1.1); see, e.g., [Be, p. 71]. For the infinite-dimensional case this will be proved in Theorem 3.4 below. Section 3.2 is devoted to augmentability of problem (3.1.1). This means that a functional is associated to (3.1.1) with the property that if x ∗ is a local minimizer of (3.1.1), then it is also a minimizer of that functional where the essential constraints are eliminated and at most simple affine constraints remain. Here this is achieved on the basis of Lagrangian functionals and augmentability is obtained without the use of a strict complementarity assumption. Section 3.3 contains the foundations for the first order augmented Lagrangian algorithm based on a duality framework. The convergence analysis for the first order algorithm is given in Section 3.4. Section 3.5 contains an application to parameter estimation problems.
3.2 Augmentability and sufficient optimality We consider a modification of (3.1.1) where the equality and finite rank inequality constraints of (3.1.1) are augmented: c c min fc (x, u) = f (x) + |e(x)|2W + |g(x) + u|2Rm over (x, u) ∈ X × Rm 2 2 subject to e(x) = 0, g(x) + u = 0, u ≥ 0, (x) ∈ K,
(3.2.1)
where c > 0 will be appropriately chosen. Observe that x ∗ is a local solution to (3.1.1) if and only if (x ∗ , u∗ ) = (x ∗ , −g(x ∗ )) is a (local) solution to (3.2.1). Moreover, x ∗ is stationary for (3.1.1) with Lagrange multiplier (λ∗ , μ∗ , η∗ ) if and only if (x ∗ , −g(x ∗ )) is stationary for (3.2.1) with Lagrange multiplier (λ∗ , μ∗ , μ∗ , η∗ ), i.e., (3.1.3)–(3.1.4) hold with g(x ∗ ) = −u∗ . Associated to (3.2.1) we introduce the functional c c Lc (x, u, λ, μ, η) = L(x, λ, μ, η) + |e(x)|2W + |g(x) + u|2Rm . 2 2
(3.2.2)
i
i i
i
i
i
i
68
ItoKunisc 2008/6/12 page 68 i
Chapter 3. First Order Augmented Lagrangians
Its Hessian L c at (x ∗ , u∗ ) in direction ((h, k), (h, k)) ∈ (X × Rm ) is given by L c (x ∗ , u∗ , λ∗ , μ∗ , η∗ )((h, k), (h, k)) = L (x ∗ , λ∗ , μ∗ , η∗ )(h, h) + c|e (x ∗ )h|2 + c|g (x ∗ )h + k|2 = L (x ∗ , λ∗ , μ∗ , η∗ )(h, h) + c|e (x ∗ )h|2 + c |g (x ∗ )h|2 +c
|gi (x ∗ )h + ki |2 + c
i∈I1 ∪I3
(3.2.3)
i∈I2
(|ki |2 + 2ki gi (x ∗ )h).
i∈I2
By the Riesz representation theorem there exists for every i ∈ {1, . . . , m} a unique li ∈ X such that gi (x ∗ )h = li , h X for every h ∈ X, where ·, · X denotes the inner product in X. The operator E : X → W × Rm2 , defined by Eh = e (x ∗ )h, lm1 +1 , h X , . . . , lm1 +m2 , h X , combines the linearized equality constraint and the active inequality constraints, which act like equalities in the sense that the associated Lagrange multipliers are positive. The essential technical result which will imply augmentability of the constraints in (3.1.1) is given next. Proposition 3.1. If (3.1.2)–(3.1.6) hold, then there exist constants c¯ > 0 and τ ∈ (0, γ ] such that ˆ (h, k)) ˆ := L (x ∗ , λ∗ , μ∗ , η∗ )(h, h) + c|Eh|2 + c H ((h, k),
m1 | li , h X + kˆi |2 i=1
≥ τ (|h|2 + |kˆR2 m1 |) 1 for all c ≥ c, ¯ h ∈ X, and kˆ ∈ Rm + .
Proof. Step 1. An equivalent characterization of C is derived by eliminating linear dependencies in the definition of C. The cone in (3.1.6) can be equivalently expressed as C = {h ∈ X : Eh = 0 and li , h X ≤ 0 for all i ∈ I1 }. (3.2.4) We next show that ⎧ there exist m ˜ 2 ≤ m2 and vectors ni , i = m1 + 1, . . . , m1 + m ˜ 2, ⎪ ⎪ ⎪ ⎨ m ˜2 ˜ such that E : X → Y × R defined by ˜ = e (x ∗ )h, nm1 +1 , h , . . . , nm1 +m˜ 2 , h ⎪ Eh ⎪ X X ⎪ ⎩ ˜ is surjective and ker E = ker E.
(3.2.5)
To verify (3.2.5) let ni = Pker e li , where Pker e denotes the projection onto ker e (x ∗ ). Choose I˜2 ⊂ I2 such that {ni }i∈I˜2 are linearly independent and span{ni }i∈I˜2 = span{ni }i∈I2 . Possibly ˜2 1 +m ˜ after reindexing we have {ni }i∈I˜2 = {ni }m m1 +1 . We show that ker E = ker E. Let h ∈ ker E. Then e (x ∗ )h = 0. For every i ∈ I˜2 let li = ni + li⊥ , where li⊥ ∈ (ker e (x ∗ ))⊥ . Then
i
i i
i
i
i
i
3.2. Augmentability and sufficient optimality
ItoKunisc 2008/6/12 page 69 i
69
˜ The converse, ker E˜ ⊂ ker E, is ni , h X = li , h X = 0, i ∈ I˜2 , and therefore h ∈ ker E. ˜ observe that the adjoint E˜ ∗ : W ×Rm˜ 2 → X of proved similarly. To verify surjectivity! of E, ! E˜ is given by E˜ ∗ (w, r) = e (x ∗ )∗ w + i∈I˜2 ri ni . If E˜ ∗ (w, r) = 0, then, using i∈I˜2 ri ni ∈ ! ker e (x ∗ ) and e (x ∗ )∗ w ∈ (ker e (x ∗ ))⊥ , we have i∈I˜2 ri ni = 0 and e (x ∗ )∗ w = 0. Therefore (w, r) = 0, ker E˜ ∗ = 0, and range E˜ = W × Rm2 , by the closed range theorem. Thus (3.2.5) holds. Finally we consider the set of vectors {ni }i∈I1 with ni = PE˜ li . After possible rear˜ 1 } ⊂ I1 such that {ni }i∈I˜1 are rangement of indices there exists a subset I˜1 = {1, . . . , m linearly independent in ker E. The cone C can therefore equivalently be expressed as C = h ∈ X : h ∈ ker E and ni , h X ≤ 0 for all i ∈ I˜1 . To exclude trivial cases we assume throughout that I˜1 and I˜2 are nonempty. Step 2. Here we characterize the polar cone C1∗ = h ∈ ker E : h, h˜ X ≤ 0 for all h˜ ∈ C1 of the closed convex cone C1 = h ∈ ker E : ni , h X ≤ 0 for all i ∈ I˜1 . Denoting by P and P ∗ the canonical projections in ker E onto C1 and C1∗ , respectively, every element h ∈ ker E can be uniquely expressed as h = h1 + h2 with h1 , h2 X = 0 and h1 = P h ∈ C1 , h2 = P ∗ h ∈ C1∗ (see [Zar, p. 256]). Moreover . C1 = Ci with Ci = {h ∈ ker E : ni , h X ≤ 0} i∈I˜1
and C1∗ =
.
Ci
∗
= co
Ci∗ = co
Ci∗
(see [Zar]), with co denoting convex closure. It follows that C1∗
=
m ˜1
αi ni : (α1 , . . . , αm˜ 1 ) ∈
m ˜1 R+
.
(3.2.6)
i=1
Step 3. Let K1 be chosen such that K1
m ˜1 i=1
2
m˜ 1
|αi | ≤ αi n i
2
i=1
for every (α1 , . . . , αm˜ 1 ) ∈ Rm˜ 1 . For arbitrary h = |h| = 2
m ˜1 i=1
!m˜ 1 i=1
αi ni ∈ C1∗ we have
αi n i , h X ≤ αi ni , h X , I1+
i
i i
i
i
i
i
70
ItoKunisc 2008/6/12 page 70 i
Chapter 3. First Order Augmented Lagrangians
where I1+ = {i ∈ I˜1 : ni , h X > 0 and αi > 0}. Consequently ⎞⎛ ⎞1/2 ⎞1/2 ⎛ ⎛ 2 2 −1/2 ni , h ⎠ ≤ K1 |h| ⎝ ni , h ⎠ αi2 ⎠ ⎝ |h|2 ≤ ⎝ X
I1+
X
Ii+
and K1 |h|2 ≤
I1+
2
ni , h
X
for every h ∈ C1∗ .
(3.2.7)
I1+
Step 4. Every h ∈ X can be uniquely decomposed into mutually orthogonal elements as h = h1 + h2 + h3 , where h1 ∈ C1 ⊂ ker E, h2 ∈ C1∗ ⊂ ker E, and h3 ∈ ker E ⊥ . !m˜ 1 α n . Note By Step 2 there exists a vector (α1 , . . . , αm˜ 1 ) ∈ Rm˜ 1 such that h2 = i=1 !m˜i1 i that ni , h1 X ≤ 0 for all i ∈ I˜1 . Therefore, using the fact that h1 , h2 X = i=1 αi h 1 , ni X = 0, it follows that ˜ , then if αi > 0 for some i = 1, . . . , m 1 (3.2.8) ni , h1 X = PE˜ li , h1 X = li , h1 X = 0. 1 We shall use Step 3 with I1+ = {i ∈ I˜1 : ni , h2 X > 0 and αi > 0}. Let kˆ ∈ Rm + and c > 0. Then by (3.1.6) and (3.2.8) we have ˆ (h, k)) ˆ = L (x ∗ , λ∗ , μ∗ , η∗ )(h, h) + c|Eh3 |2 H ((h, k),
li , h3 + li , h2 + kˆi 2 + c
li , h + kˆi 2 +c X X X I1+
I1 \I1+
∗
∗
∗
∗
≥ γ |h1 | + 2L (x , λ , μ , η )(h1 , h2 + h3 ) 2
+ L (x ∗ , λ∗ , μ∗ , η∗ )(h2 + h3 , h2 + h3 ) 2 2 c1 ni , h2 X + kˆi − c1 + c|Eh3 |2 + li , h3 X 2 + + I1
2 c2 ˆ 2 + |ki | − c2 li , h X 2 + + I1 \I1
I1
I1 \I1
for all 0 < c1 , c2 ≤ c to be chosen below. Here we used (a + b)2 ≥ 12 a 2 − b2 and the fact that li , h2 = ni , h2 for i ∈ I1 . To estimate Eh3 from below we make use of the fact that E˜ is an isomorphism from (ker E)⊥ to W × Rm˜ 2 . Hence there exists K2 > / / 0 such ˜ 3 | ≤ |Eh3 | for all h3 ∈ (ker E)⊥ . Setting L = /L (x ∗ , λ∗ , μ∗ )/ , c3 = that K2 |h3 | ≤ |Eh min(c1 , c2 ), and c4 = max(c1 , c2 ) we obtain ˆ (h, k)) ˆ = γ |h1 |2 − 2L|h1 |(|h2 | + |h3 |) − L(|h2 | + |h3 |)2 H ((h, k), 2 c1 c3 ˆ 2 + cK22 |h3 |2 + ni , h2 X + k 2 + 2 I i 1 I1 2 2 2 − 3c4 li , h3 X − 3c2 li , h1 X + li , h2 X , I1
where we used the fact that ni , h2
X
I1 −I1+
≥ 0 for i ∈ I1+ .
i
i i
i
i
i
i
3.2. Augmentability and sufficient optimality Setting |l|2 =
! I1
ItoKunisc 2008/6/12 page 71 i
71
|li |2 and using (3.2.7) we find
ˆ (h, k)) ˆ ≥ γ |h1 |2 + c1 K1 |h2 |2 + cK22 |h3 |2 + c3 kˆ 2 H ((h, k), 2 2 I i 1
− 2L|h1 |(|h2 | + |h3 |) − 2L(|h2 | + |h3 |2 ) 2
− 3c2 |l|2 |h1 |2 − 3c2 |l|2 |h2 |2 − 3c4 |l|2 |h3 |2 c1 c3 ˆ 2 ≥ γ |h1 |2 + K1 |h2 |2 + cK22 |h3 |2 + k 2 2 I i 1 γ 1/2 γ −1/2 γ 1/2 γ −1/2 − 2L |h1 ||h2 | − 2L |h1 ||h3 | 4 4 4 4 − 2L|h2 |2 − 2L|h3 |2 − 3c2 |l|2 |h1 |2 − 3c2 |l|2 |h2 |2 − 3c4 |l|2 |h3 |2 γ 2 2 ≥ |h1 | γ − − 3c2 |l| 2 c 4 2 1 2 2 + |h2 | K1 − L − 2L − 3c2 |l| 2 γ c3 ˆ 2 4 2 2 2 2 + |h3 | cK2 − L − 2L − 3c4 |l| + k . γ 2 I i 1
We now choose the constants in the order c2 , c1 , and c¯ such that the coefficients of |h1 |2 , |h2 |2 , and |h3 |2 , with c replaced by c, ¯ are positive. This implies the claim for every c ≥ c. ¯ It is worthwhile to point out the following corollary to Proposition 3.1 which is well known in finite-dimensional spaces. Corollary 3.2. Let E ∈ L(X, W ) be surjective and let A ∈ L(X) be self-adjoint and coercive on ker E; i.e., there exists γ > 0 such that Ax, x ≥ γ |x|2 for all x ∈ ker E. Then there exist constants τ > 0 and c¯ > 0 such that (A + cE ∗ E)x, x X ≥ τ |x|2 for all x ∈ X and c ≥ c. ¯ This follows from Proposition 3.1 with A = L (x ∗ ). Proposition 3.3. Let (3.1.2)–(3.1.6) hold. Then there exists σ > 0 such that μ∗i ki ≥ σ (|h|2 + |k 2 |) L c¯ (x ∗ , u∗ , λ∗ , μ∗ , η∗ )((h, k), (h, k)) + i∈I2
for all h ∈ X with |h| ≤ μ/(2 ˜ c¯ supi∈I2 |li |), μ˜ = minI2 μ∗i and all k ∈ Rm with ki ≥ 0 for i ∈ I1 ∪ I2 , and c¯ is as introduced in Proposition 3.1. ! Proof. Let A = L c¯ (x ∗ , u∗ , λ∗ , μ∗ , η∗ )((h, k), (h, k)) + i∈I2 μ∗i ki . Then from (3.2.3) and the definition of H
∗
li , h + ki 2 + c¯ ˆ (h, k)) ˆ + c¯ |ki |2 + 2 li , h X ki + μi ki , A = H ((h, k), X I3
I2
I2
i
i i
i
i
i
i
72
ItoKunisc 2008/6/12 page 72 i
Chapter 3. First Order Augmented Lagrangians
where kˆ denotes the first m1 coordinates of the vector k. By Proposition 3.1 we have for every ε > 1 2 ˆ 2Rm1 ) + c¯ li , h X (1 − ε 2 ) + |ki |2 (1 − ε −2 ) A ≥ τ (|h|2 + |k| + c¯
I3
li , h ki + |ki | − 2c¯ μ∗i ki X 2
I2
≥ τ (|h| + 2
I2
ˆ 2Rm1 ) |k|
+ c(1 ¯ − ε )|h| 2
2
−2
+ c(1 ¯ −ε )
A≥
τ 2
I2
|ki |2
I2 ∪I3
|li | − 2c¯ 2
I3
Now choose ε > 1 such that find that
= (ε 2 − 1)τ
li , h ki + μ∗i ki . X I2
! I3
I2
|li |2 . Then using the constraint for h we
1 ˆ 2Rm1 ) + c(1 ¯ − ε −2 ) ki2 + μ˜ − 2c¯ sup |li ||h| ki τ (|h|2 + |k| 2 I2 I ∪I I 2
3
2
3
1 ˆ 2Rm1 ) + c(1 ≥ τ (|h|2 + |k| ¯ − ε −2 ) ki2 . 2 I ∪I
2
This proves the claim for c = c. ¯ For arbitrary c ≥ c¯ the assertion follows from the form of L c (x ∗ , u∗ , λ∗ , μ∗ ). Theorem 3.4. Assume that (3.1.2)–(3.1.6) hold. Then there exist constants σ¯ > 0, c¯ > 0 and a neighborhood U (x ∗ , −g(x ∗ )) = U (x ∗ , u∗ ) of (x ∗ , u∗ ) such that Lc (x, u, λ∗ , μ∗ , η∗ ) + μ∗ , u Rm c c = f (x) + λ∗ , e(x) W + μ∗ , g(x) + u Rm + η∗ , (x) Z + |e(x)|2W + |g(x) + u|2Rm 2 2 ≥ f (x ∗ ) + σ¯ (|x − x ∗ |2 + |u − u∗ |2 ) (3.2.9) for all c ≥ c¯ and (x, u) ∈ U (x ∗ , u∗ ) with ui ≥ 0 for i ∈ I1 ∪ I2 . Proof. By (3.1.4) we have μ∗ , u∗ = − μ∗ , g(x ∗ ) = 0. Moreover Lc (x ∗ , u∗ , λ∗ , μ∗ , η∗ ) + μ∗ , u∗ Rm = f (x ∗ ), (Lc )x (x ∗ , u∗ , λ∗ , μ∗ , η∗ ) = (Lc )u (x ∗ , u∗ , λ∗ , μ∗ , η∗ ) = 0, where (Lc )x and (Lc )u denote the partial derivatives of Lc with respect to x and u. Consequently Lc (x ∗ , u∗ , λ∗ , μ∗ , η∗ ) + μ∗ , u∗ Rm = f (x ∗ ) + μ∗ , u − u∗ Rm
+ L c (x ∗ , u∗ , λ∗ , μ∗ , η∗ )((x − x ∗ , u − u∗ ), (x − x ∗ , u − u∗ )) + o(|x − x ∗ |2 + |u − u∗ |2 ).
i
i i
i
i
i
i
3.2. Augmentability and sufficient optimality
ItoKunisc 2008/6/12 page 73 i
73
The definitions of I1 and I3 imply that μ∗i = 0 for i ∈ I1 ∪ I3 . Thus Proposition 3.3 implies the existence of a neighborhood U (x ∗ , u∗ ) of (x ∗ , u∗ ) and a constant σ¯ > 0 such that (3.2.9) holds for all (x, u) ∈ U (x ∗ , u∗ ) with ui ≥ 0 for i ∈ I1 ∪ I2 . The conclusion of Theorem 3.4 is referred to as augmentability of problem (3.1.1). This means that a functional, which in our case is Lc (x, u, λ∗ , μ∗ , η∗ ) + μ∗ , u∗ Rm , can be found with the property that it has (x ∗ , u∗ ) as a strict local minimizer under the simple constraint u ≥ 0, while the explicit nonlinear constraints are eliminated. Let us note that if (3.1.6) holds with C replaced by the larger set Cˆ = {h ∈ X : e (x ∗ )h = 0, gi (x ∗ )h ≤ 0 for i ∈ I1 ∪ I2 }, then the conclusion of Theorem 3.4 holds with Lc (x, u, λ∗ , μ∗ , η∗ ) + μ∗ , u∗ Rm replaced by Lc (x, u, λ∗ , μ∗ , η∗ ). In case X is finite-dimensional, augmentability is analyzed in detail in [Hes1], for example. The proof of augmentability in [Hes1] depends in essential manner on compactness of the unit ball. In [Be, p. 161 ff ], the augmentability analysis relies on the strict complementarity assumption, i.e., I1 = ∅ is assumed. We obtain, as immediate consequence of Theorem 3.4, that (3.1.6) provides a second order sufficient optimality condition. Corollary 3.5. Assume that (3.1.2)–(3.1.6) hold. Then there exists a neighborhood U (x ∗ ) of x ∗ such that f (x) ≥ f (x ∗ ) + σ¯ |x − x ∗ |2 for all x ∈ U (x ∗ ) satisfying e(x) = 0, g(x) ≤ 0, and (x) ∈ K. Theorem 3.4 can be used as the basis to define augmented cost functionals in terms of x only with the property that x ∗ is a uniform strict local unconstrained minimum. The two choices we present differ in the way in which the inequality constraints are treated. The first choice uses the classical cutoff penalty functional g(x) ˜ = max(g(x), 0), and the second is the Bertsekas penalty functional μ g(x, ˆ μ, c) = max g(x), − , c
(3.2.10)
(3.2.11)
where μ ∈ Rm and c > 0. In each of the two cases the max operation acts coordinatewise. To motivate the choice (3.2.11) we use the Lagrangian Lc (x, u, λ, μ, η) introduced in (3.2.2) for (3.2.1) and consider ⎧ ⎨min Lc (x, u, λ, μ, η) = fc (x, u) + λ, e(x) W (3.2.12) ⎩ + μ, g(x) + u Rm + η, (x) Z over x ∈ X, u ≥ 0. Carrying out the constraint minimization with respect to u with x and μ fixed results in μ , u = u(x, μ) = max 0, − g(x) + c
i
i i
i
i
i
i
74
ItoKunisc 2008/6/12 page 74 i
Chapter 3. First Order Augmented Lagrangians
and consequently
μ g(x) + u(x, μ) = max g(x), − c
= g(x, ˆ μ, c).
(3.2.13)
Corollary 3.6. Let (3.1.2)–(3.1.6) hold. Then there exists a neighborhood U (x ∗ ) of x ∗ such that c c 2 f (x)+ λ∗ , e(x) W + μ∗ , g(x) ˜ + η∗ , (x) Z + |e(x)|2W + |g(x)| ˜ Rm 2 2 ≥ f (x ∗ ) + σ¯ |x − x ∗ |2 for all x ∈ U (x ∗ ), and c ≥ c, ¯ where g(x) ˜ = max(g(x), 0). Proof. Setting u = max(−g(x), 0)
(3.2.14)
we have g(x) + u = g(x) ˜ and u ≥ 0. Recall the definition of U = U (x ∗ , −g(x ∗ )) from Theorem 3.4. Determine a neighborhood U (x ∗ ) such that x ∈ U (x ∗ ) implies (x, −g(x)) ∈ U and gi (x) ≤ 0 if gi (x ∗ ) < 0. It is simple to argue that x ∈ U (x ∗ ) implies (x, u) ∈ U, where u is defined in (3.2.14). The claim now follows from Theorem 3.4. Corollary 3.7. Let (3.1.2)–(3.1.6) hold and let r > 0. Then there exist constants δ > 0 and c˜ = c(r) ˜ ≥ c¯ such that c c f (x)+ λ∗ , e(x) X + μ∗ , g(x, ˆ μ, c) Rm + η∗ , (x) Z + |e(x)|2W + |g(x, ˆ μ, c)|2 2 2 ≥ f (x ∗ ) + σ¯ |x − x ∗ |2 for all c ≥ c, ˜ x ∈ Bδ = {x ∈ X : |x − x ∗ | ≤ δ}, and μ ∈ Br+ = {μ ∈ Rm + : |μ| ≤ r}, where g(x, ˆ μ, c) = max(g(x), − μc ). Proof. Let ε > 0 be such that gi (x ∗ ) ≤ −ε for all i ∈ I3 and such that |x − x ∗ | < ε and |u − u∗ | < ε implies (x, u) ∈ U (x ∗ , u∗ ), where U (x ∗ , u∗ ) is given in Theorem 3.4. Determine δ ∈ (0, ε) such that |x − x ∗ | ≤ δ implies |g(x) − g(x ∗ )| < 2ε and choose c˜ ≥ 2rε . For c ≥ c˜ and μ ∈ Br+ we define μ u = max 0, − + g(x) . (3.2.15) c Then g(x) + u = g(x) ˆ and u ≥ 0. To verify the claim it suffices to show that x ∈ Bδ and μ ∈ Br+ imply |u − u∗ | < ε, where u is defined in (3.2.15). If i ∈ I1 ∪ I2 , then |ui − u∗i | ≤ |gi (x ∗ ) − gi (x)| + μci . For i ∈ I3 we have μci + gi (x) ≤ μci + |gi (x ∗ ) − gi (x)| + gi (x ∗ ) < rc + 2ε − ε < 0 and consequently |ui − u∗i | = |gi (x ∗ ) − gi (x) − μci |. Summing over i = 1, . . . , m we find |u − u∗ | ≤ 2|g(x ∗ ) − g(x)|2 + c22 |μ|2 < ε2 . Remark 3.2.1. Note that c x → μ, g(x, ˆ μ, c) + |g(x, ˆ μ, c)|2 2
i
i i
i
i
i
i
3.3. The first order augmented Lagrangian algorithm
ItoKunisc 2008/6/12 page 75 i
75
is C 1 if g is C 1 . This follows from the identity
μ, g(x, ˆ μ, c)
Rm
c 1 + |g(x, ˆ μ, c)|2Rm = | max(0, μ + cg(x))|2Rm − |μ|2Rm . (3.2.16) 2 2c
Here, as throughout, the norm on Rm is the Euclidean one. On the other hand, c x → μ, g(x, ˜ μ, c) + |g(x, ˜ μ, c)|2 2 is not C 1 .
3.3 The first order augmented Lagrangian algorithm Here we describe the first order augmented Lagrangian algorithm which is a hybrid method combining the penalty technique and a Lagrangian method. Considering for a moment equality constraints only, the penalty method for (3.1.1) consists in minimizing f (x) +
ck |e(x)|2 2
for a sequence of penalty parameters ck tending to ∞. The Lagrangian method relies on minimizing f (x) + (λk , e(x)) and updating λk as a maximizer of the dual problem associated to (3.1.1), which will be defined below. The first order augmented Lagrangian method provides a systematic technique for the multiplier update. Its convergence analysis will be given in the next section. Let x ∗ be a local solution of (3.1.1) with (3.1.2)–(3.1.6) holding. The algorithm will ∗ require startup values (λ0 , μ0 ) ∈ W × Rm + for the Lagrange multipliers. We set r = |μ | + ∗ 2 ∗ 2 1/2 ˜ ε) ≥ c¯ as in the proof of Corollary 3.6. (|λ0 −λ | +|μ0 −μ | ) and choose ε and c˜ = c(r, Recall that ε was chosen such that gi (x ∗ ) ≤ −ε for all indices i in the set of inactive indices I3 and such that |x −x ∗ | < ε, |u−u∗ | < ε imply (x, u) ∈ U (x ∗ , −g(x ∗ )) with U (x ∗ , −g(x ∗ )) given in Theorem 3.4. Finally, let {cn }∞ n=1 be a monotonically nondecreasing sequence of penalty parameters with c1 > c, ˜ and set σn = cn − c. ¯ It is not required that limn→∞ cn = ∞, as would be the case for penalty methods. Algorithm ALM. 1. Choose (λ0 , μ0 ) ∈ W × Rm +. 2. Let xn be a solution to min Ln (x) over x ∈ X with (x) ∈ K.
(Pn )
3. Update the Lagrange multipliers λn = λn−1 + σn e(xn ), μn = μn−1 + σn g(x ˆ n , μn−1 , cn ),
i
i i
i
i
i
i
76
ItoKunisc 2008/6/12 page 76 i
Chapter 3. First Order Augmented Lagrangians where Ln (x) = f (x) + λn−1 , e(x) W + μn−1 , g(x, ˆ μn−1 , cn ) Rm +
cn cn |e(x)|2W + |g(x, ˆ μn−1 , cn )|2 . 2 2
Observe that μn = max(μn−1 + σn g(xn ), μn−1 (1 − σcnn )) and consequently μn ∈ Rn+ for each n = 1, 2, . . . . Existence of solutions to (Pn ) will be discussed in Section 3.4. It will be shown that (Pn ) has a solution in the interior of Bδ (see Corollary 3.6) for all n sufficiently large, or for all n if c1 is sufficiently large, and that these solutions converge to x ∗ , while (λn , μn ) converges to (λ∗ , μ∗ ). To motivate the Lagrange multiplier update in step 3 of the algorithm and to justify calling this a first order algorithm, we consider problem (3.2.1) without the infinite rank inequality constraint. Introducing Lagrange multipliers for the equality constraints e(x) = 0 and g(x) + u = 0 we obtained the augmented Lagrange functional Lc in (3.2.12). Carrying out the minimization with respect to u and utilizing (3.2.13) suggests introducing the modified augmented Lagrangian functional c c Lˆ c (x, λ, μ) = f (x) + λ, e(x) W + μ, g(x, ˆ μ, c) Rm + |e(x)|2W + |g(x, ˆ μ, c)|2Rm . 2 2 (3.3.1) Since Lˆ c (x ∗ , λ∗ , μ∗ ) = f (x ∗ ) we find Lˆ c (x ∗ , λ, μ) ≤ Lˆ c (x ∗ , λ∗ , μ∗ ) ≤ Lˆ c (x, λ∗ , μ∗ ) for all x ∈ Bδ and μ ≥ 0, ∗
(3.3.2)
∗
ˆ , μ) = 0 for whenever c ≥ c. ˜ The first inequality follows from the fact that e(x ) = g(x μ ≥ 0 and the second one from Corollary 3.6. From the saddle point property (3.3.2) for Lˆ c (x, λ, μ) we deduce that sup inf Lˆ c (x, λ, μ) ≤ Lˆ c (x ∗ , λ∗ , μ∗ ) ≤ inf sup Lˆ c (x, λ, μ).
λ,μ≥0 x
x
(3.3.3)
λ,μ
For the following discussion we assume strict complementarity, i.e., I1 = ∅. Then x → Lˆ c (x, λ, μ) is twice continuously differentiable for (x, λ, μ) in a neighborhood of (x ∗ , λ∗ , μ∗ ) and Lˆ c (x ∗ , λ∗ , μ∗ )(h, h) = Lˆ (x ∗ , λ∗ , μ∗ )(h, h) + c|Eh|2W for h ∈ X. Thus by (3.1.6) and Corollary 3.2, Lˆ c (x ∗ , λ∗ , μ∗ ) is positive definite on X for all c sufficiently large. Moreover (x, λ, μ) → Lˆ c (x, λ, μ) is continuous in a neighborhood U (x ∗ ) × U (λ∗ , μ∗ ) of (x ∗ , λ∗ , μ∗ ) and Lˆ c (x, λ, μ) ≥ τ¯ |x|2 for τ¯ > 0 independent of x ∈ X and (λ, μ) ∈ W × Rm . It follows that min Lˆ c (x, λ, μ)
x∈U (x ∗ )
admits a unique solution x(λ, μ) for every (λ, μ) ∈ U (λ∗ , μ∗ ) and as a consequence of Theorem 2.24 (λ, μ) → x(λ, μ) is differentiable. Consequently the locally defined dual functional d(λ, μ) = min∗ Lˆ c (x, λ, μ) x∈U (x )
i
i i
i
i
i
i
3.3. The first order augmented Lagrangian algorithm
ItoKunisc 2008/6/12 page 77 i
77
is differentiable for (λ, μ) in a neighborhood of (λ∗ , μ∗ ). For the gradient ∇d at (λ, μ) in direction (δλ , δμ ) we obtain using (3.2.16) ∇d(λ, μ)(δλ , δμ ) = f (x) + e (x)∗ λ + g (x)∗ max(0, μ + cg(x)), xλ (δλ ) + xμ (δμ ) + δλ , e(x) + δμ , g(x, ˆ μ, c) , where x = x(λ, μ). Utilizing the first order optimality condition this implies that the gradient of d(λ, μ) is given by ∇d(λ, μ) = (e(x(λ, μ)), g(x(λ, ˆ μ), μ, c)) ∈ W × Rm .
(3.3.4)
Thus the multiplier updates in step 3 of Algorithm ALM are steepest ascent directions for the dual problem sup d(λ, μ). λ,μ≥0
In view of (3.3.3) the first order augmented Lagrangian algorithm is an iterative algorithm combining minimization in the primal direction in step 2 and maximization in the dual direction on every level of the iteration. Remark 3.3.1. In this chapter we identify X and W with their dual spaces and consider e (x)∗ as an operator from W to X. If, alternatively, e (x)∗ is considered as an operator from W ∗ to X ∗ , then the Lagrange multiplier is taken as an element of W ∗ and the Lagrange ˜ = f (x) + λ, ˜ e(x) W ∗ ,W . The relation between λ and λ˜ functional is defined as L(x, λ) ˜ is given by I λ = λ, with I the canonical isomorphism from W to W ∗ . If, for example, W = H −1 (), then W ∗ = H01 () and I = (− )−1 . The Lagrange multiplier update in step 3 of Algorithm ALM is then given by λn = λn−1 + σn I e(xn ). As already mentioned the augmented Lagrangian algorithm ALM is also a hybrid method combining Lagrange multiplier and penalty methods. Solving (Pn ) without the penalty terms we obtain the former, for λn = 0 and cn satisfying limn→∞ cn = ∞ we obtain the latter. The choice of cn for Algorithm ALM in practice is an important one. While no general purpose techniques appear to be available, guidelines for the choice include choosing them large enough such that the augmented Lagrangian has positive definite Hessian in the sense of Proposition 3.1. Large cn will improve the convergence estimates, as we shall see in the following section, and at the same time they can be the cause for ill-conditioning of the auxiliary problems (Pn ). In fact, for large cn the Hessian of the cost functional Ln (x) can have eigenvalues on significantly different scales. For further discussion on the choice of the parameter c we refer the reader to [Be]. Let us make a few historical comments on the first order augmented Lagrangian algorithm. It was originated by Hestenes [Hes2] and Powell [Pow] and extended among others by Bertsekas [Be] and in [PoTi, Roc2]. The infinite-dimensional case has received less attention. In this respect we refer the reader to the work of Polyak and Tret’yakov [PoTr] and Fortin and Glowinski [FoGl]. Both consider the case of augmenting equality constraints only. In [FoGl] augmented Lagrangian methods are also developed systematically for solving nonlinear partial differential equations.
i
i i
i
i
i
i
78
3.4
ItoKunisc 2008/6/12 page 78 i
Chapter 3. First Order Augmented Lagrangians
Convergence of Algorithm ALM
We shall use the following notation: c¯ c¯ Lˆc (x, μ) = f (x) + λ∗ , e(x) + μ∗ , g(x, ˆ μ, c)|2 . ˆ μ, c) + η∗ , (x) + |e(x)|2 + |g(x, 2 2 Further we set r = |μ∗ | + (|λ0 − λ∗ |2 + |μ0 − μ∗ |2 )1/2 . Corollary 3.7 then guarantees the existence of δ > 0 and c˜ = c(r) ˜ such that Lˆ c (x, μ) − f (x ∗ ) ≥ σ¯ |x − x ∗ |2
(3.4.1)
for all x ∈ Bδ , μ ∈ Br+ , and c ≥ c. ˜ We also assume that Bδ is contained in the region of applicability of (3.4.1) and that {cn }∞ n=1 is a monotonically nondecreasing sequence with c˜ < c1 . Then we have the following convergence properties of Algorithm ALM from arbitrary initializations (λ0 , μ0 ) ∈ W × Rm + in the case that suboptimal solutions are chosen in step 2. Theorem 3.8. Assume that (3.1.2)–(3.1.5), (3.4.1) hold and that x˜n ∈ Bδ satisfy Ln (x˜n ) ≤ Ln (x ∗ ) for each n = 1, . . . .
(3.4.2)
If (λn , μn ) are defined as in step 3 of Algorithm ALM with xn replaced by x˜n , then for n ≥ 1 we have with σn = cn − c¯ σ¯ |x˜n − x ∗ |2 +
1 (|λn − λ∗ |2 + |μn − μ∗ |2 ) 2σn 1 ≤ (|λn−1 − λ∗ |2 + |μn−1 − μ∗ |2 ). 2σn
(3.4.3)
This implies that μn ∈ Br+ for all n ≥ 1 and |x˜n − x ∗ |2 ≤
1 (|λ0 − λ∗ |2 + |μ0 − μ∗ |2 ) 2σ¯ σn
(3.4.4)
and ∞ n=1
σn |x˜n − x ∗ |2 ≤
1 (|λ0 − λ∗ |2 + |μ0 − μ∗ |2 ). 2σ¯
(3.4.5)
Proof. We proceed by induction and assume that the claim has been verified up to n−1. For n = 1 the result can be obtained by the general arguments given below. For the induction step we observe that (3.4.3), which is assumed to hold up to n − 1, implies |μn−1 | ≤ |μ∗ | + (|λ0 − λ∗ |2 + |μ0 − μ∗ |2 )1/2 = r. Consequently μn−1 ∈ Br+ and (3.4.1) with μ = μn−1 is applicable. Using the fact that [max(0, μn−1,i + cn gi (x ∗ ))]2 − μ2n−1,i ≤ 0
i
i i
i
i
i
i
3.4. Convergence of Algorithm ALM
ItoKunisc 2008/6/12 page 79 i
79
and (3.2.16) we find m 1 Ln (x ) = f (x ) + {[max(0, μn−1,i + cn gi (x ∗ ))]2 − μ2n−1,i } ≤ f (x ∗ ). 2cn i=1 ∗
∗
(3.4.6)
Next we rearrange terms in Ln (x˜n ) and obtain Ln (x˜n ) = f (x˜n ) + λ∗ , e(x˜n ) + λn−1 − λ∗ , e(x˜n ) + μ∗ , g( ˆ x˜n , μn−1 , cn ) c¯ 1 ˆ x˜n , μn−1 , cn ) + |e(x˜n )|2 + (cn − c)|e( ¯ x˜n )|2 + μn−1 − μ∗ , g( 2 2 c¯ 1 ˆ x˜n , μn−1 , cn )|2 + (cn − c)| + |g( ¯ g( ˆ x˜n , μn−1 , cn )|2 2 2 ˆ x˜n , μn−1 ) + 1 (|λn − λ∗ |2 − |λn−1 − λ∗ |2 ) = L( 2σn 1 + (|μn − μ∗ |2 − |μn−1 − μ∗ |2 ) − η∗ , (x˜n ) Z . 2σn Since η∗ ∈ K + and (x˜n ) ∈ K we have η∗ , (x˜n ) ≤ 0. This fact together with the above equality, (3.4.2), and (3.4.6) implies ˆ x˜n , μn−1 ) + 1 (|λn − λ∗ |2 − |λn−1 − λ∗ |2 ) L( 2σn 1 (|μn − μ∗ |2 − |μn−1 − μ∗ |2 ) ≤ Ln (x˜n ) ≤ f (x ∗ ). + 2σn Finally (3.4.1) implies that σ¯ |x˜n − x ∗ |2 +
1 1 1 1 |λn − λ∗ |2 + |μn − μ∗ |2 ≤ |λn−1 − λ∗ |2 + |μn−1 − μ∗ |2 , 2σn 2σn 2σn 2σn
which establishes (3.4.3). Estimates (3.4.4) and (3.4.5) follow from (3.4.3). The following conditions will be utilized to guarantee existence of local solutions to the auxiliary problems (Pn ): ⎧ f : X → R and gi : X → R, i = 1, . . . , m, are weakly lower ⎪ ⎪ ⎪ ⎨semicontinuous, ⎪e : X → W maps weakly convergent sequences to ⎪ ⎪ ⎩ weakly convergent sequences.
(3.4.7)
Further (PnC ) denotes problem (Pn ) with the additional constraint that x is contained in the closed ball Bδ . We refer to xn as the solution to (PnC ) if Ln (xn ) ≤ Ln (x) for all x ∈ Bδ . Proposition 3.9. If (3.1.2)–(3.1.5), (3.4.1), and (3.4.7) hold, then (PnC ) admits a solution xn for every n = 1, 2, . . . . Moreover, there exists n0 such that every solution xn of (PnC )
i
i i
i
i
i
i
80
ItoKunisc 2008/6/12 page 80 i
Chapter 3. First Order Augmented Lagrangians
satisfies xn ∈ int Bδ if n ≥ n0 . If c11−c¯ (|λ0 − λ∗ |2 + |μ0 − μ∗ |2 ) is sufficiently small, then every solution xn of (PnC ) is in int Bδ for every n = 1, 2, . . . . Thus for n0 or c sufficiently large the solutions to (PnC ) are local solutions of the unconstrained problems (Pn ). Proof. Let {xni }i∈N be a minimizing sequence for (PnC ). There exists a weakly convergent subsequence {xnik }k∈N with weak limit xn ∈ Bδ . Condition (3.4.7) implies weak lower semicontinuity of Ln and hence Ln (xn ) ≤ lim inf ik Ln (xnik ) and xn is a solution of (PnC ). In particular (3.4.2) holds with x˜n replaced by xn for n = 1, 2, . . . . Consequently limn xn = x ∗ and xn ∈ int Bδ for all n sufficiently large. Alternatively, if c11−c¯ (|λ0 − λ∗ |2 + |μ0 − μ∗ |2 ) is sufficiently small, then xn ∈ int Bδ for all n = 1, 2, . . . by (3.4.4). For the solutions xn obtained in Proposition 3.9 the conclusions of Theorem 3.8 hold, in particular limn xn = x ∗ and the associated Lagrange multipliers {λn } and {μn } are bounded. To investigate convergence of the Lagrange multipliers we introduce M : X → W × Rm1 +m2 × Z defined by M = (e (x ∗ ), gac (x ∗ ), (x ∗ )),
where gac denotes the first m1 + m2 coordinates of g. The following additional hypothesis will be needed: There exists a constant κ > 0 such that / ∗ / /f (x ) − f (x)/ ≤ κ|x ∗ − x|, / ∗ / /e (x ) − e (x)/ ≤ κ|x ∗ − x|, / ∗ / /g (x ) − g (x)/ ≤ κ|x ∗ − x|
(3.4.8)
for all x ∈ Bδ , and M is surjective.
(3.4.9)
We point out that (3.4.9) is more restrictive than the regular point condition in Definition 1.5 of Chapter 1. Below we set μn,ac = col(μn,1 , . . . , μn,m1 +m2 ) and μ∗ac = col(μ∗1 , . . . , μ∗m1 +m2 ). Without loss of generality we shall assume that σ¯ ≤ 1. Theorem 3.10. Let (3.1.2)–(3.1.5), (3.4.1), and (3.4.8)–(3.4.9) hold and let xn be a solution of (Pn ) in int Bδ , n˜ ≥ 1. Then there exists a constant K independent of n such that K |λn − λ∗ | + |μn − μ∗ | + |ηn − η∗ | ≤ √ (|λn−1 − λ∗ | + |μn−1 − μ∗ |) for all n ≥ 1. σ¯ σn Here ηn denotes the Lagrange multiplier associated with the constraint ∈ K in (Pn ). Sufficient conditions for xn ∈ Bδ were given in Proposition 3.9 above.
i
i i
i
i
i
i
3.4. Convergence of Algorithm ALM
ItoKunisc 2008/6/12 page 81 i
81
Proof. Optimality of xn and x ∗ for (Pn ) and (3.1.1) (see (3.1.3)) imply f (xn ), h + λn−1 + cn e(xn ), e (xn )h + max(0, μn−1 + cn g(xn )), g (xn )h + ηn , (xn )h = 0 and ∗ ∗ ∗ ∗ ∗ ∗ ∗ f (x ), h + λ , e (x )h + μ , g (x )h + η , (x )h = 0 for all h ∈ X. If we set λ˜ n = λn−1 + cn e(xn ) and μ˜ n = max(0, μn−1 + cn g(xn )), we obtain the following equation in X: e (x ∗ )∗ (λ˜ n − λ∗ ) + gac (x ∗ )∗ (μ˜ n,ac − μ∗ac ) + (x ∗ )∗ (ηn − η∗ ) = f (x ∗ ) − f (xn ) + (e (x ∗ )∗ − e (xn )∗ )λ˜ n + (gac (x ∗ )∗ − gac (xn )∗ )μn,ac .
Here we used the fact that μ˜ n,i = 0 for i ∈ I3 by the choice of δ and c˜ (see the proof of Corollary 3.7), and we set μn,ac = col (μn,1 , . . . , μn,m1 +m2 ). Note that M ∗ : W ×Rm1 +m2 → X is given by M ∗ = col (e (x ∗ )∗ , gac (x ∗ )∗ ). Hence we find MM ∗ (λ˜ n − λ∗ , μ˜ n,ac − μ∗ac , ηn − η∗ ) = M f˜ (x ∗ ) − f˜ (xn ) + (e (x ∗ )∗ − e (xn )∗ )λ˜ n + (gac (x ∗ )∗ − gac (xn )∗ )μ˜ n,ac .
(3.4.10)
Since λ˜ n − λ∗ = ce(x ¯ n ) and μ˜ n − μn = c¯g(x ˆ n , μn−1 , cn ) we have ∗ ¯ |λ˜ n − λn | = c|e(x n ) − e(x )|, |μ˜ n,ac − μn,ac | ≤ c|g ¯ ac (xn ) − gac (x ∗ )|.
(3.4.11)
Thus by (3.4.8) and (3.4.3), (3.4.4) of Theorem 3.8 the sequences {λn } and {μn } are uniformly bounded. Consequently by (3.4.9) and (3.4.10) there exists a constant K¯ such that |λ˜ n − ¯ n − x ∗ |, and further by (3.4.11) a constant K can be chosen λ∗ | + |μ˜ n,ac − μ∗ac | ≤ K|x such that ¯ n − x ∗ | for all n = 1, 2, . . . . |λn − λ∗ | + |μnac − μ∗ac | + |ηn − η∗ | ≤ K|x The choice of δ implies that gi (xn ) ≤ μni =
μn−1,i cn
(3.4.12)
for all i ∈ I3 and therefore
c¯ n−1 μ for i ∈ I3 , n ≥ 1. cn i
(3.4.13)
From (3.4.3), with x˜n replaced by xn , (3.4.12), and (3.4.13), and using σ¯ ≤ 1 the theorem follows. Corollary 3.11. Under the assumptions of Theorem 3.10 we have n K n0 1 ∗ ∗ ∗ |λn − λ | + |μn − μ | + |ηn − η | ≤ √ √ (|λ0 − λ∗ | + |μ0 − μ∗ |) σi σ¯ i=1 for all n ≥ 1.
i
i i
i
i
i
i
82
ItoKunisc 2008/6/12 page 82 i
Chapter 3. First Order Augmented Lagrangians
Remark 3.4.1. If the surjectivity requirement for M is replaced by assuming that M ∗ is surjective, then the proof of Theorem 3.10 implies the existence of a constant K such that |PM (λn − λ∗ , μn,ac − μ∗ac , ηn − η∗ )|W ×Rm1 +m2 ×Z ≤ K|xn − x ∗ |, where PM = M(M ∗ M)−1 M ∗ denotes the orthogonal projection of W × Rm1 +m2 × Z onto the range of M which is closed, since M ∗ is surjective. Since xn → x ∗ in X this implies convergence of PM (λn , μn,ac , ηn ) to PM (λ∗ , μ∗ac , η∗ ). Remark 3.4.2. If a constraint of the type x ∈ C, with C a closed convex set in X, appears in (3.1.1), then Theorem 3.8 and Proposition 3.9 remain valid if Algorithm ALM is modified such that in (Pn ) the functional Ln (x) is minimized over x ∈ C and x˜n appearing in (3.4.2) satisfies x˜n ∈ C. The stationarity condition (3.1.3) has to be replaced by f (x ∗ )(x − x ∗ ) + λ∗ , e (x ∗ )(x − x ∗ ) W + μ∗ , g (x ∗ )(x − x ∗ ) Rm ≥ 0 (3.1.3 ) for all x ∈ C, in this case.
3.5 Application to a parameter estimation problem We consider a least squares formulation for the estimation of the coefficient a in −div (a grad y) = f in , y = 0 in ∂,
(3.5.1)
where is a bounded domain in Rn , with Lipschitz continuous boundary ∂ if n ≥ 2, and f ∈ H −1 (), f = 0. Let Q be a Hilbert space that embeds compactly in L∞ () and let N : Q → R denote a seminorm on Q with ker(N ) ⊂ span{1} and such that N (a) defines a norm on Q/R = {a ∈ Q : 1, a Q = 0} that is equivalent to the norm induced by Q. For z ∈ H01 (), α > 0, and β > 0 we consider min 12 |y − z| 2H 1 () + α2 N 2 (a) 0 (3.5.2) subject to e(a, y) = 0, a(x) ≥ β, where e : Q × H01 () → H −1 () is given by e(a, y) = div (a grad y) + f. This is a special case of (3.1.1) with X = Q × H01 (), W = H −1 (), g = 0, and f the cost functional in (3.5.2). The affine constraint a(x) ≥ β can be considered as an explicit constraint as discussed in Remark 3.4.2 by setting C = {a ∈ Q : a(x) ≥ β}. Denote the solution to (3.5.1) as a function of a by y(a). To guarantee the existence of a solution for (3.5.2) we require the following assumption: either N is a norm on Q (3.5.3) or inf {|y(a) − z|H 1 : a is constant } < |z|H 1 .
i
i i
i
i
i
i
3.5. Application to a parameter estimation problem
ItoKunisc 2008/6/12 page 83 i
83
Lemma 3.12. If (3.5.3) holds, then there exists a solution (a ∗ , y ∗ ) of (3.5.2). Proof. Let {(an , yn )}∞ n=1 denote a minimizing sequence for (3.5.2). If N is a norm, then a subsequence argument implies the existence of a solution to (3.5.2). Otherwise we decompose an as an = an1 + an2 with an2 ∈ Q/R and an1 ∈ (Q/R)⊥ . Since {(an , yn )}∞ n=1 is a 2 ∞ ∞ minimizing sequence, {N (an2 )}∞ and consequently {|a | } are bounded. We have L n n=1 n=1 an1 |∇yn |2 dx = − an2 |∇yn |2 dx + fyn dx.
{an1 |∇yn |2L2 }∞ n=1 ,
Since a ≥ β implies a ≥ β, it follows that |∇yn | = |∇y(an )| → 0 for n → ∞. In this case 1
is bounded. If an1 → ∞, then
|z| ≤ lim |y(an ) − z|2H 1 () + βN 2 (an ) ≤ inf {|y(a) − z|H 1 : a = constant} < |z|H 1 , n→∞
∞ which is impossible. Therefore {an1 }∞ n=1 and as a consequence {an }n=1 are bounded, and a subsequence of {(an , yn )} converges weakly to a solution of (3.5.2).
H
−1
Using Theorem 1.6 one argues the existence of Lagrange multipliers (λ∗ , η∗ ) ∈ () × Q∗ such that the Lagrangian
L(a, y, λ∗ , η∗ ) =
1 α |y − z|2H 1 () + N 2 (a) + λ∗ , e(a, y) H01 ,H −1 + η∗ , β − a Q∗ ,Q 0 2 2
is stationary at (a ∗ , y ∗ ), and η∗ , β − a ∗ Q∗ ,Q = 0, η∗ , h Q∗ ,Q ≥ 0 for all h ≥ 0. Hence (3.1.2)–(3.1.5) hold. Here (3.1.3)–(3.1.4) must be modified to include the affine inequality constraint with infinite-dimensional image space. We turn to the augmentability condition (3.4.1) and introduce the augmented Lagrangian c Lc (a, y, λ∗ , η∗ ) = L(a, y, λ∗ , η∗ ) + |e(a, y)|2H −1 2 for c ≥ 0. Henceforth we assume that (3.5.2) admits a solution (a0 , y0 ) for α = 0. Such a solution exists if z is attainable, i.e., if there exists a0 ∈ Q, a0 ≥ β, such that y(a0 ) = y and in this case y0 = y(a0 ). Alternatively, if the set of coefficients is further constrained by a norm bound |a|Q ≤ γ , for example, then existence of a solution to (3.5.2) with α = 0 is also guaranteed. Proposition 3.13. Let (a0 , y0 ) denote a solution to (3.5.2) with α = 0, let (a ∗ , y ∗ ) be a solution for α > 0, and choose γ ≥ |a ∗ |Q . Then, if |y0 − z|H 1 is sufficiently small, there exist positive constants c0 and σ such that Lc (a, u; λ∗ , η∗ ) ≥
1 ∗ α |y − z|2H 1 + N 2 (a ∗ ) + σ (|a − a ∗ |2Q + |y − y ∗ |2H 1 ) 0 2 2
for all c ≥ c0 and (a, y) ∈ C × H01 () with |a|Q ≤ γ . This proposition implies that (3.4.1) is satisfied and that Theorem 3.8 and Proposition 3.9 are applicable with Bδ = {a ∈ Q : a(x) ≥ β, |a|Q ≤ γ } × H01 ().
i
i i
i
i
i
i
84
ItoKunisc 2008/6/12 page 84 i
Chapter 3. First Order Augmented Lagrangians
Proof. For (a, y) ∈ C × H01 () we have Lc (a, y, λ∗ , η∗ ) − Lc (a ∗ , y ∗ , λ∗ , η∗ ) = ∇L(a ∗ , y ∗ , λ∗ , η∗ ) 1 α c − h∇λ∗ , ∇v L2 + |v|2H 1 + N 2 (h) + |e(a, h)|2H −1 , 0 2 2 2 where h = a − a ∗ , v = y − y ∗ , and ∇L denotes the gradient of L with respect to (a, y). First order optimality implies that Lc (a, y, λ∗ , η∗ ) − Lc (a ∗ , y ∗ , λ∗ , η∗ ) ≥ − h∇λ∗ , ∇v L2 1 α c + |v|2H 1 + N 2 (h) + |e(a, h)|2H −1 =: D. 0 2 2 2 Introduce P = ∇ · (− )−1 ∇ and note that P can be extended to an orthogonal projection on L2 ()n . We find that |e(a, y)|2H −1 = P (a∇y − a ∗ ∇y ∗ ), a∇y − a ∗ ∇y ∗ L2 = P (a∇v + h∇y ∗ ), h∇y ∗ + a∇v L2 1 ≥ P (h∇y ∗ ), h∇y ∗ L2 − P (a∇v), a∇v L2 2 1 ≥ P (h∇y ∗ ), h∇y ∗ L2 − |a|2L∞ |v|2H 1 0 2 1 ≥ P (h∇y ∗ ), h∇y ∗ L2 − γ 2 k12 |v|2H 1 , 0 2 ∞ where k1 denotes the embedding constant of Q into L (). Henceforth we consider the case that ker N = span{1} and use the decomposition introduced in the proof of Lemma 3.12. We have 1 |e(a, y)|2 ≥ |h1 |2 |y ∗ |2H 1 − |h1 | |h2 |L∞ |y ∗ |2H 1 − γ 2 k12 |v|H01 . 0 0 2 ∞ Since Q embeds continuously into L () and N is a norm equivalent to the norm on Q/R, there exists a constant k2 such that |h2 |L∞ ≤ k2 N (h) for all h = h1 + h2 ∈ Q. Consequently |e(a, y)|2 ≥
1 |h1 |2 |y ∗ |2H 1 − k2 |h1 | N (h2 ) |y ∗ |2H 1 − γ 2 k12 |v|H01 0 0 2
(3.5.4)
for all (a, y) ∈ C×H01 (). Next note that Ly (a ∗ , u∗ ; λ∗ , η∗ ) = 0 implies that −∇(a ∗ ∇λ∗ ) = − (y ∗ − y) in H −1 (), and hence
h∇λ∗ , ∇v 2 ≤ 1 (|h1 | + k2 N (h2 ))|y ∗ − z|H 1 |v|H 1 . (3.5.5) L 0 0 β Setting c = δα with δ > 0 and observing that |f |H −1 ≤ k3 |y ∗ |H 1 for a constant k3 depending only on β, D=
1 1 (1 − δαγ 2 k12 )|v|2H 1 − (|h1 | + k2 N (h2 ))|y ∗ − z||v|H01 0 2 β α δ k2 2 ∗ 2 2 2 N (h2 ) + |y |H 1 |h1 | − 2 δ|h1 |N (h2 ) |f |H −1 . + 0 2 2 k3
i
i i
i
i
i
i
3.5. Application to a parameter estimation problem
ItoKunisc 2008/6/12 page 85 i
85
There exist δ > 0 and k4 > 0 such that D ≥ k4 (|v|2H 1 + |h1 |2 + N 2 (h2 )) − 0
1 (|h1 | + k2 N (h2 )) |y ∗ − z|H01 |v|H01 . β
Since |h1 | + N (h2 ) defines an equivalent norm on Q, the claim follows with c0 = δα. It is worthwhile to comment on the auxiliary problems of Algorithm ALM. These involve the minimization of Ln (a, y) =
1 α cn |y −z|2H 1 + N 2 (a)+ λn−1 , A(a)y +f H01 ,H −1 + |A(a)y +f |2H −1 (3.5.6) 0 2 2 2
over (a, y) ∈ C × H01 (), where A(a)y = div (a grad y). The resulting optimality conditions are given by α 2 (N (an )) (a − an ) 2 + (−∇λn−1 ∇ yn + cn ∇ −1 (A(an )yn + f ) ∇yn , a − an )Ln ≥ 0
(3.5.7)
for all a ≥ an and − (yn − z) + A(an )λn−1 + A(an )(− )−1 (A(an )yn + f ) = 0.
(3.5.8)
The Lagrange multiplier is updated according to λn = λn−1 + σn (− )−1 (A(an )yn + f ).
(3.5.9)
To simplify (3.5.7), (3.5.8) for numerical realization one can proceed iteratively solving (3.5.7) with yn replaced by yn−1 for an and then (3.5.8) for yn . A good initialization for the variable λ is given by λ0 = 0. In fact, for small residue problems |y ∗ − z|H01 is small ∂ L(a ∗ , y ∗ , λ∗ , η∗ ) = 0 implies that λ∗ is small. Setting λ0 = 0 and y0 = z the and then ∂y coefficient a1 is determined from min a≥β
α 2 C1 N (a) + |A(a)z + f |2H −1 . 2 2
(3.5.10)
Thus the first step of the augmented Lagrangian algorithm for estimating a in (3.5.1) involves a regularized equation error step. Recall that the equation error method for parameter estimation problems consists in solving the hyperbolic equation −div (a grad z) = f for a. This estimate is improved during the subsequent iterations. The first order augmented Lagrangian method can therefore be considered as a hybrid method combining the output least squares and the equation error approach to parameter estimation. Let us close this section with some brief comments. If the H 1 least squares criterion is replaced by an L2 -criterion, then the first negative Laplacian in (3.5.8) must be replaced by the identity operator. For the H 1 -criterion the same number of differentiations are applied to y in the least squares and the equation error term in (3.5.8). If one would approach the problem of estimating a in (3.5.1) by first discretizing and then applying an optimization strategy, the discrete analogue of (− )−1 can be interpreted as preconditioning. We assumed that Q embeds compactly in L∞ (). This is the case, for example, if Q = H 1 () when n = 1, or Q = H 2 () when n = 2 or 3, or if Q ⊂ L∞ () is finite-dimensional.
i
i i
i
i
ItoKunisc 2008/6/12 page 86 i
i
i
i
i
i
i
i
i
i
ItoKunisc 2008/6/12 page 87 i
Chapter 4
Augmented Lagrangian Methods for Nonsmooth, Convex Optimization
4.1
Introduction
The class of optimization problems that motivates this chapter is given by min f (x) + ϕ( x) over x ∈ C,
(4.1.1)
where X, H are real Hilbert spaces, C is a closed convex subset of X, and ∈ L(X, H ). In this chapter we identify H with its dual space and we distinguish between X and its dual X ∗ . Further f : X → R is a continuously differentiable, convex function, and ϕ : H → (−∞, ∞] is a proper, lower semicontinuous, convex but not necessarily differentiable function. This problem class encompasses a wide variety of optimization problems including variational inequalities of the first and second kind [Glo]. Our formulation here is slightly more general than the one in [Glo, EkTe], since we allow the additional constraint x ∈ C. For example, one can formulate regularized inverse scattering problems in the form (4.1.1):
2
μ 1
2 min |∇u| + g |∇u| dx + k(x, y)u(y) dy − z(x)
dx (4.1.2)
2 2 over u ∈ H 1 () and u ≥ 0, where k(x, y) denotes a scattering kernel. The problem consists in recovering the original image u defined on a domain from scattered and noisy data z ∈ L2 (). Here μ > 0 and g > 0 are fixed and should be adjusted to the statistics of the noise. If μ = 0, then this problem is equivalentto the image enhancement algorithm in [ROF] based on minimization of the BV-seminorm |∇u| ds. In this example ϕ(v) = g |v| ds, which is nondifferentiable. Several additional applications are treated at the end of this chapter. We develop a Lagrange multiplier theory to deal with the nonsmoothness of ϕ. To briefly explain the approach let x, λ ∈ H and c > 0, and define the family of generalized Yosida–Moreau approximations ϕc (x, λ) by c ϕc (x, λ) = inf ϕ(x − u) + (λ, u)H + |u|2H . (4.1.3) u∈H 2 87
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 88 i
88 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization This is equivalent to an augmented Lagrangian approach. In fact, note that (4.1.1) is equivalent to min f (x) + ϕ( x − u) (4.1.4) subject to x ∈ C and u = 0 in H. Treating the equality constraint u = 0 in (4.1.4) by the augmented Lagrangian method results in the minimization problem min f (x) + ϕ( x − u) + (λ, u)H +
x∈C,u∈H
c 2 |u| , 2 H
(4.1.5)
where λ ∈ H is a multiplier and c is a positive scalar penalty parameter. Equivalently, problem (4.1.5) is written as min Lc (x, λ) = f (x) + ϕc ( x, λ). x∈C
(4.1.6)
It will be shown that ϕc (u, λ) is continuously Fréchet differentiable with respect to u ∈ H . Moreover, if xc ∈ C denotes the solution to (4.1.6), then it satisfies f (xc ) + ∗ λc , x − xc X∗ ,X ≥ 0
for all x ∈ C,
λc = ϕc ( xc , λc ). It will further be shown that under appropriate conditions the pair (xc , λc ) ∈ C × H has a ¯ as c → ∞ such that x¯ ∈ C is the minimizer of (4.1.1) (strong-weak) cluster point (x, ¯ λ) and that λ¯ ∈ H is a Lagrange multiplier in the sense that ¯ x − x
f (x) ¯ + ∗ λ, ¯ X∗ ,X ≥ 0
for all x ∈ C
(4.1.7)
with the complementarity condition ¯ ¯ λ) λ¯ = ϕc ( x,
for each c > 0.
(4.1.8)
System (4.1.7)–(4.1.8) for the pair (x, ¯ λ¯ ) is a necessary and sufficient optimality condition for problem (4.1.1). We analyze iterative algorithms of Uzawa and augmented Lagrangian ¯ and present a convergence analysis. It will be shown type for finding the optimal pair (x, ¯ λ) that condition (4.1.8) is equivalent to the complementarity condition λ¯ ∈ ∂ϕ( x). ¯ Thus the frequently employed differential inclusion λ¯ ∈ ∂ϕ( x) ¯ is replaced by the nonlinear equation (4.1.8). This chapter is organized as follows. In Sections 4.2–4.3 we present the basic convex analysis and duality theory in Banach spaces. Section 4.4 is devoted to the generalized Yosida–Moreau approximation which is the basis for the remainder of the chapter. Conditions for the existence of Lagrange multiplier for (4.1.1) and optimality systems are derived in Section 4.5. Section 4.6 is devoted to the augmented Lagrangian algorithm. Convergence of both the augmented Lagrangian and the Uzawa methods are proved. Section 4.7 contains a large number of concrete applications.
i
i i
i
i
i
i
4.2. Convex analysis
4.2
ItoKunisc 2008/6/12 page 89 i
89
Convex analysis
In this section we present standard results from convex analysis and duality theory in Banach spaces following, in part, [BaPe, EkTu]. Throughout X denotes a real Banach space. Definition 4.1. (1) A functional F : X → (−∞, ∞] is called convex if F ((1 − λ) x1 + λ x2 ) ≤ (1 − λ) F (x1 ) + λ F (x2 ) for all x1 , x2 ∈ X and 0 ≤ λ ≤ 1. It is called proper if it is not identically ∞. (2) A functional F : X → (−∞, ∞] is said to be lower semicontinuous (l.s.c.) at x ∈ X if F (x) ≤ lim inf F (y). y→x
A functional F is l.s.c. if it is l.s.c. at all x ∈ X. (3) A functional F : X → (−∞, ∞] is called weakly lower semicontinuous (w.l.s.c.) at x if F (x) ≤ lim inf F (xn ) n→∞
for all sequences {xn } converging weakly to x. Further F is w.l.s.c. if it is w.l.s.c. at all x ∈ X. (4) The subset D(F ) = {x ∈ X : F (x) < ∞} of X is called the effective domain of F . (5) The epigraph of F is defined by epi(F ) = {(x, c) ∈ X × R : F (x) ≤ c}. Lemma 4.2. A functional F : X → (−∞, ∞] is l.s.c. if and only if its epigraph is closed. Proof. The lemma follows from the fact that epi (F ) is closed if and only if (xn , cn ) ∈ epi(F ) → (x, c) in X × R implies F (x) ≤ c. Lemma 4.3. A functional F : X → (−∞, ∞] is l.s.c. if and only if its level sets Sc = {x ∈ X : F (x) ≤ c} are closed for all c ∈ R. Proof. Assume that F is l.s.c. Let c > 0 and let {xn } be a sequence in Sc with limit x. Then F (x) ≤ lim inf x→∞ F (xn ) ≤ c and hence x ∈ Sc and Sc is closed. Conversely, assume that Sc is closed for all c > 0, and let {xn } be a sequence converging to x in X. Choose a subsequence {xnk } with lim F (xnk ) = lim inf F (xn ), k→∞
n→∞
and suppose that F (x) > lim inf n→∞ F (xn ). Then there exists c¯ ∈ R such that lim inf F (xn ) < c¯ < F (x), n→∞
and there exists an index m such that F (xnk ) < c¯ for all k ≥ m. Since Sc¯ is closed x ∈ Sc¯ , which is a contradiction to c¯ < F (x). Lemma 4.4. Assume that all level sets of the functional F : X → (−∞, ∞] are convex. Then F is l.s.c. if and only if it is w.l.s.c. on X.
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 90 i
90 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization Proof. Assume that F is l.s.c. Suppose that {xn } converges weakly to x¯ in X, and let {xnk } be a subsequence such that d = lim inf F (xn ) = limk→∞ F (xnk ). By Lemma 4.3 the sets {x : F (x) ≤ d + } are closed for every > 0. By assumption they are also convex. Hence by Mazur’s lemma they are also weakly (sequentially) closed. Hence F (x) ¯ ≤ d + for every > 0. Since is arbitrary, we have that F (x) ¯ ≤ d and F is w.l.s.c. The converse implication is obvious. Note that for a convex function all associated level sets are convex, but not vice versa. Theorem 4.5. Let F be a proper, l.s.c., convex functional on X. Then F is bounded below by an affine functional; i.e., there exist x ∗ ∈ X∗ and c ∈ R such that F (x) ≥ x ∗ , x X∗ ,X + c
for all x ∈ X.
Moreover, F is the pointwise supremum of a family of such continuous affine functionals. Proof. Let x0 ∈ X and choose β ∈ R such that F (x0 ) > β. Since epi(F ) is a closed convex subset of the product space X × R, it follows from the separation theorem for convex sets [EkTu] that there exists a closed hyperplane H ⊂ X × R given by H = {(x, r) ∈ X × R : x0∗ , x + ar = α} with x0∗ ∈ X∗ , a ∈ R, α ∈ R, such that
x0∗ , x0 + aβ < α < x0∗ , x + ar
for all (x, r) ∈ epi(F ).
Setting x = x0 and r = F (x0 ), we have x0∗ , x0 + aβ < α < x0∗ , x0 + aF (x0 ) and thus a (F (x0 ) − β) > 0. If F (x0 ) < ∞, then a > 0 and thus α 1 − x0∗ , x ≤ r a a β<
for all (x, r) ∈ epi(F ),
α 1 − x0∗ , x0 < F (x0 ). a a
Hence, b(x) = αa − a1 x0∗ , x is a continuous affine function on X such that b ≤ F and −x ∗ the first claim is established with c = αa and x ∗ = a 0 . Moreover β < b(x0 ) < F (x0 ). Therefore F (x0 ) = sup b(x0 ) = x ∗ , x0 + c : (4.2.1) x ∗ ∈ X∗ , c ∈ R, b(x) ≤ F (x) for all x ∈ X . If F (x0 ) = ∞, either a > 0 (thus we proceed as above) or a = 0. In the latter case α − x0∗ , x0 > 0 and α − x0∗ , x < 0 on D(F ). Since F is proper there exists an affine function b(x) = x ∗ , x + c such that b ≤ F . Thus x ∗ , x + c + θ (α − x0∗ , x ) < F (x)
i
i i
i
i
i
i
4.2. Convex analysis
ItoKunisc 2008/6/12 page 91 i
91
for all x and θ > 0. Choosing θ > 0 large enough so that x ∗ , x + c + θ (α − x0∗ , x ) > β, we have that b(x) = x ∗ , x + c + θ (α − x0∗ , x ) is a continuous affine function on X with b ≤ F and β < b(x0 ). Therefore (4.2.1) holds at x0 with F (x0 ) = ∞ as well. Theorem 4.6. If F : X → (−∞, ∞] is convex and bounded above on an open set U , then F is continuous on U . Proof. We choose M ∈ R such that F (x) ≤ M − 1 for all x ∈ U . Let xˆ be any element in U . Since U is open there exists a δ > 0 such that the open ball {x ∈ X : |x − x| ˆ < δ} is contained in U . For any ∈ (0, 1), let θ = M−F . Then for x ∈ X satisfying |x − x| ˆ <θδ (x) ˆ we have
x − xˆ
|x − x| ˆ
< δ.
θ + xˆ − xˆ = θ Hence
x−xˆ θ
+ xˆ ∈ U . By convexity of F x − xˆ F (x) ≤ (1 − θ )F (x) ˆ +θF + xˆ < (1 − θ)F (x) ˆ + θ M, θ
and thus F (x) − F (x) ˆ < θ (M − F (x)) ˆ = . x−x ˆ θ
+ xˆ ∈ U and θ 1 θM 1 xˆ − x F (x) ˆ ≤ F + xˆ + F (x) < + F (x), 1+θ θ 1+θ 1+θ 1+θ
Similarly,
which implies F (x) − F (x) ˆ > −θ(M − F (x)) ˆ = −. Therefore |F (x) − F (x)| ˆ < if |x − x| ˆ < θ δ and F is continuous in U . Theorem 4.7. If F : X → (−∞, ∞] is a proper, l.s.c., convex functional on X, and F is bounded above on a convex, open δ-neighborhood U of a bounded, convex set C, then F is Lipschitz continuous on C. Proof. By Theorem 4.5 and by assumption there exist constants M and m such that m ≤ F (x) ≤ M
for all x ∈ U.
x| ˆ δ Let x and xˆ be in C with |x − x| ˆ ≤ M−m , x = x, ˆ and set θ = 2|x− . Without loss of θ 2 x−xˆ generality we assume that M−m < 1 so that θ ∈ (0, 1). Then y = θ + xˆ ∈ U since xˆ ∈ U and |y − x| ˆ = x−θ x|ˆ ≤ 2δ . Due to convexity of F we have x − xˆ F (x) ≤ (1 − θ )F (x) ˆ +θ F + xˆ ≤ (1 − θ)F (x) ˆ +θM θ
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 92 i
92 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization and hence F (x) − F (x) ˆ ≤ θ(M − F (x)) ˆ ≤ Similarly,
x−x ˆ θ
2 (M − m)|x − x|. ˆ δ
+ xˆ ∈ U and
F (x) ˆ ≤
θ xˆ − x 1 θM 1 F + xˆ + F (x) ≤ + F (x), 1+θ θ 1+θ 1+θ 1+θ
which implies −θ (M − m) ≤ −θ(M − F (x)) ˆ ≤ F (x) − F (x) ˆ and therefore |F (x) − F (x)| ˆ ≤
2 δ (M − m)|x − x| ˆ for |x − x| ˆ ≤ . θ M −m
Since C is bounded and convex, Lipschitz continuity for all x, xˆ ∈ C follows.
4.2.1
Conjugate and biconjugate functionals
Definition 4.8. The functional F ∗ : X∗ → [−∞, ∞] defined by F ∗ (x ∗ ) = sup { x ∗ , x X∗ ,X − F (x) : x ∈ X} is called the conjugate of F . If F is bounded below by an affine functional x ∗ , · − c, then F ∗ (x ∗ ) = sup { x ∗ , x − F (x)} ≤ sup { x ∗ , x − x ∗ , x + c} = c, x∈X
x∈X
and hence F ∗ is not identically ∞. Note also that D(F ) = ∅ implies that F ∗ ≡ −∞ and conversely, if F ∗ (x ∗ ) = −∞ for some x ∗ ∈ X, then D(F ) is empty. Example 4.9. If F (x) = 1 + q1 = 1. p
1 |x|p , p
x ∈ R, then F ∗ (x ∗ ) =
1 ∗ q |x | q
for 1 < p < ∞ and
Example 4.10. If F (x) is the indicator function of the closed unit ball of X, i.e., F (x) = 0 for |x| ≤ 1 and F (x) = ∞ otherwise, then F ∗ (x ∗ ) = |x ∗ |. In fact, F ∗ (x ∗ ) = sup{ x ∗ , x : |x| ≤ 1} = |x ∗ |. If F is a proper, l.s.c., convex functional on X, then by Theorem 4.5 F (x0 ) = sup { x ∗ , x0 − c : x ∗ ∈ X∗ , c ∈ R, x ∗ , x − c ≤ F (x) for all x ∈ X} for every x0 ∈ X. For x ∗ ∈ X define c(x ∗ ) = inf {c ∈ R : c ≥ x ∗ , x − F (x) for all x ∈ X}
i
i i
i
i
i
i
4.2. Convex analysis
ItoKunisc 2008/6/12 page 93 i
93
and observe that
F (x0 ) = sup { x ∗ , x0 − c(x ∗ )}. x ∗ ∈X ∗
∗
∗
But c(x ) = supx∈X { x , x − F (x)} = F ∗ (x ∗ ) and hence F (x0 ) = sup { x ∗ , x0 − F ∗ (x ∗ )}. x ∗ ∈X ∗
This suggests the following definition. Definition 4.11. For F : X → (−∞, ∞] the biconjugate functional F ∗∗ : X → (−∞, ∞] is defined by F ∗∗ (x) = sup { x ∗ , x − F ∗ (x ∗ )}. x ∗ ∈X ∗
Above we proved the following result. Theorem 4.12. If F is a proper, l.s.c., convex functional, then F = F ∗∗ . It is simple to argue that (F ∗ )∗ = F ∗∗ if X is reflexive. Next we prove some important results on conjugate functionals. Theorem 4.13. For every functional F : X → (−∞, ∞] the conjugate F ∗ is convex and l.s.c. on X∗ . Proof. If F is identically ∞, then F ∗ is identically −∞. Otherwise D(F ) is nonempty and F (x ∗ ) > −∞ for all x ∗ ∈ X∗ , and F ∗ (x ∗ ) = sup { x ∗ , x − F (x) : x ∈ D(F )}. Thus F ∗ is the pointwise supremum of the family of continuous affine functionals x ∗ → x ∗ , x − F (x) for x ∈ D(F ). This implies that F ∗ is convex. Moreover {x ∗ ∈ X∗ : x ∗ , x − F (x) ≤ c} is closed for each c ∈ R and x ∈ D(F ). Consequently . {x ∗ ∈ X∗ : F ∗ (x ∗ ) ≤ c} = {x ∗ ∈ X∗ : x ∗ , x − F (x) ≤ c} x∈D(F )
is closed for each c ∈ R. Hence F ∗ is l.s.c. by Lemma 4.3. Theorem 4.14. For every F : X → (−∞, ∞] the biconjugate F ∗∗ is convex, l.s.c., and F ∗∗ ≤ F . Moreover for each x¯ ∈ X F ∗∗ (x) ¯ = sup { x ∗ , x −c ¯ : x ∗ ∈ X∗ , c ∈ R, x ∗ , x −c ≤ F (x) for all x ∈ X}. (4.2.2) Proof. If there is no continuous affine functional which is everywhere less than F , then F ∗ ≡ ∞ and hence F ∗∗ ≡ −∞. In fact, if there exist c ∈ R and x ∗ ∈ X∗ with c ≥ F ∗ (x ∗ ), then x ∗ , · − c is everywhere less than F , which is impossible. The claims readily follow in this case.
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 94 i
94 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization Otherwise, assume that there exists a continuous affine functional everywhere less than F . Then D(F ∗ ) is nonempty and F ∗∗ (x) > −∞ for all x ∈ X. If F ∗ (x ∗ ) = −∞ for some x ∗ ∈ X∗ , then F ∗∗ ≡ ∞, F ≡ ∞, and the claims follow. In the remaining case F ∗ (x ∗ ) is finite for all x ∗ and we have F ∗∗ (x) = sup{ x ∗ , x − F ∗ (x ∗ ) : x ∗ ∈ X∗ , F ∗ (x ∗ ) finite}. For every x ∗ ∈ D(F ∗ ) we have x ∗ , x − F ∗ (x ∗ ) ≤ F (x) for all x ∈ X, hence F ∗∗ ≤ F and F ∗∗ (x) ¯ ≤ sup{ x ∗ , x
¯ − c : x ∗ ∈ X∗ , c ∈ R, x ∗ , x − c ≤ F (x) for all x ∈ X}. Let x ∗ and c be such that x ∗ , x − c ≤ F (x) for all x ∈ X. Then x ∗ , x − c ≤ x ∗ , x − F (x ∗ ) for all x ∈ X. Therefore sup{ x ∗ , x
¯ − c : x ∗ ∈ X∗ , c ∈ R, x ∗ , x − c ≤ F (x) for all x ∈ X} ≤ F ∗∗ (x), ¯ and (4.2.2) follows. Moreover F ∗∗ is the pointwise supremum of a family of continuous affine functionals, and it follows as in the proof of Theorem 4.13 that F ∗ is convex and l.s.c. Theorem 4.15. If F : X → (−∞, ∞] is a convex functional which is finite and l.s.c. at x, then F (x) = F ∗∗ (x). For the proof of this result we refer the reader to [EkTu, p. 104], for example. Theorem 4.16. For every F : X → (−∞, ∞], we have F ∗ = F ∗∗∗ . Proof. Since F ∗∗ ≤ F due to Theorem 4.14, F ∗ ≤ F ∗∗∗ by the definition of conjugate functions. The definition of F ∗∗ implies that x ∗ , x − F ∗∗ (x) ≤ F (x ∗ ) ∗∗
∗
∗
∗
∗
(4.2.3)
∗∗
if F (x) and F (x ) are finite. If F (x ) = ∞ or F (x) = ∞, inequality (4.2.3) holds as well. If F ∗ (x ∗ ) = −∞, we have that F and F ∗ are identically ∞. If F ∗∗ (x) = −∞, then F ∗ is identically ∞, and (4.2.3) holds for all (x, x ∗ ) ∈ X × X ∗ . Thus, F ∗∗∗ (x) = sup{ x ∗ , x − F ∗∗ (x)} ≤ F (x ∗ ) x∈X ∗
∗
for all x ∈ X . Hence F
∗∗∗
∗
=F .
Theorem 4.17. Let F : X → (−∞, ∞] and assume that ∂F (x) = ∅ for some x ∈ X. Then F (x) = F ∗∗ (x). Proof. Since ∂F (x) = ∅ there exists a continuous affine functional ≤ F with (x) = F (x). Due to Theorem 4.14 we have ≤ F ∗∗ ≤ F . It follows that (x) = F ∗∗ (x) = F (x), as desired.
i
i i
i
i
i
i
4.2. Convex analysis
4.2.2
ItoKunisc 2008/6/12 page 95 i
95
Subdifferential
In this short section we summarize some important results on subdifferentials of functionals F : X → (−∞, ∞]. While F is not assumed to be convex, the applications that we have in mind are to convex functionals. Definition 4.18. Let F : X → (−∞, ∞]. The subdifferential of F at x is the (possibly empty) set ∂F (x) = {x ∗ ∈ X∗ : F (y) − F (x) ≥ x ∗ , y − x for all y ∈ X}. If ∂F (x) is nonempty, then F is called subdifferentiable at x and the set of all points where F is subdifferentiable is denoted by D(∂F ). As a first observation, we note that x0 = argminx∈X F (x) if and only if 0 ∈ ∂F (x0 ). In fact, these two statements are equivalent to F (x) ≥ F (x0 ) + (0, x − x0 )
for all x ∈ X.
Example 4.19. Let F be Gâteaux differentiable at x, i.e., there exists w ∗ ∈ X∗ such that lim
t→0+
F (x + t v) − F (x) = w ∗ , v
t
for all v ∈ X,
and w∗ ∈ X∗ is called the Gâteaux derivative of F at x. It is denoted by F (x). If in addition F is convex, then F is subdifferentiable at x and ∂F (x) = {F (x)}. Indeed, for v = y − x F (x + t (y − x)) − F (x) ≤ F (y) − F (x), t
where 0 < t < 1.
As t → 0+ we obtain F (x), y − x ≤ F (y) − F (x)
for all y ∈ X,
and thus F (x) ∈ ∂F (x). On the other hand, if w ∗ ∈ ∂F (x), we find for y ∈ X and t > 0 F (x + t y) − F (x) ≥ w ∗ , y . t Taking the limit t → 0+ , we obtain F (x) − w ∗ , y ≥ 0
for all y ∈ X.
This implies that w∗ = F (x). Example 4.20. For F (x) = 12 |x|2 , x ∈ X, we will show that ∂F (x) = F(x), where F : X → X∗ denotes the duality mapping. In fact, if x ∗ ∈ F(x), then x ∗ , x − y = |x|2 − x ∗ , y ≥
1 (|x|2 − |y|2 ) 2
for all y ∈ X.
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 96 i
96 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization Thus x ∗ ∈ ∂ϕ(x). Conversely, if x ∗ ∈ ∂ϕ(x), then 1 (|y|2 − |x|2 ) ≥ x ∗ , y − x
2
for all y ∈ X.
(4.2.4)
We let y = t x, 0 < t < 1, and obtain 1+t |x|2 ≤ x ∗ , x
2 and thus |x|2 ≤ x ∗ , x . Similarly, if t > 1, then |x|2 ≥ x ∗ , x and therefore |x|2 = x ∗ , x
and |x ∗ | ≥ |x|. On the other hand, setting y = x + t u, t > 0, in (4.2.4), we have t x ∗ , u ≤
1 t2 (|x + t u|2 − |x|2 ) ≤ t |u||x| + |u|2 , 2 2
which implies x ∗ , u ≤ |u||x|. Hence |x ∗ | ≤ |x| and we obtain |x|2 = |x ∗ |2 = x ∗ , x
Example 4.21. Let K be a closed convex subset of X and let ψK be the indicator function of K, i.e., 0 if x ∈ K, ψK (x) = ∞ otherwise. Obviously, ψK is convex and l.s.c. on X. By definition we have for x ∈ K ∂ψK (x) = {x ∗ ∈ X∗ : x ∗ , y − x ≤ 0 for all y ∈ K} and ∂ψK (x) = ∅ if x ∈ / X. Thus D(ψK ) = D(∂ψK ) = K and ∂ψK (x) = {0} for each interior point of K. Moreover, for x ∈ K, ∂ψK (x) coincides with the definition of the normal cone to K at x. Theorem 4.22. Let F : X → (−∞, ∞]. (1) We have x ∗ ∈ ∂F (x) ¯ if and only if F (x) ¯ + F ∗ (x ∗ ) = x ∗ , x . ¯ (2) Assume that X is reflexive. Then x ∗ ∈ ∂F (x) ¯ implies that x¯ ∈ ∂F ∗ (x ∗ ). If ∗ moreover F is convex and l.s.c., then x ∈ ∂F (x) ¯ if and only if x¯ ∈ ∂F ∗ (x ∗ ). Proof. (1) Note that x ∗ ∈ ∂F (x) ¯ if and only if x ∗ , x − F (x) ≤ x ∗ , x
¯ − F (x) ¯
(4.2.5)
for all x ∈ X. By the definition of F ∗ this implies F ∗ (x ∗ ) = x ∗ , x
¯ − F (x). ¯ Conversely, if F (x ∗ ) + F (x) ¯ = x ∗ , x , ¯ then (4.2.5) holds for all x ∈ X. (2) Since F ∗∗ ≤ F by Theorem 4.14, it follows from (1) that F ∗∗ (x) ¯ ≤ x ∗ , x
¯ − F ∗ (x ∗ ).
i
i i
i
i
i
i
4.2. Convex analysis
ItoKunisc 2008/6/12 page 97 i
97
By the definition of F ∗∗ , F ∗∗ (x) ¯ ≥ x ∗ , x
¯ − F ∗ (x ∗ ). Hence F ∗ (x ∗ ) + F ∗∗ (x) ¯ = x ∗ , x . ¯ Since X is reflexive, F ∗∗ = (F ∗ )∗ . Applying (1) to F ∗ we have x¯ ∈ ∂F ∗ (x ∗ ). If in addition F is convex and l.s.c., it follows from Theorem 4.12 that F ∗∗ = F . Thus if x¯ ∈ ∂F ∗ (x ∗ ), then F (x) ¯ + F ∗ (x ∗ ) = F ∗ (x ∗ ) + F ∗∗ (x) ¯ = x ∗ , x
¯ by applying (1) to F ∗ . Therefore x ∗ ∈ ∂F (x) ¯ again by (1). Proposition 4.23. For F : X → (−∞, ∞] the set ∂F (x) is closed and convex for every x ∈ X. Proof. If F (x) = ∞, then ∂F (x) = ∅. Henceforth let F (x) < ∞. For every x ∗ ∈ X ∗ we have F ∗ (x ∗ ) ≥ x ∗ , x − F (x) and hence by Theorem 4.22 ∂F (x) = {x ∗ ∈ X∗ : F ∗ (x ∗ ) − x ∗ , x ≤ −F (x)}. By Theorem 4.13 the functional x ∗ → F ∗ (x ∗ ) − x ∗ , x is convex and l.s.c. The claim follows from Lemma 4.3. Theorem 4.24. If the convex function F is continuous at x, ¯ then ∂F (x) ¯ is not empty. Proof. Since F is continuous at x, there exists for every > 0 an open neighborhood U of x¯ such that F (x) ≤ F (x) ¯ + , x ∈ U . Then U × (F (x) ¯ + , ∞) is an open set in X × R, contained in epi F . Hence (epi F )o , the relative interior of epi F , is nonempty. Since F is convex, epi F is convex and (epi F )o is convex. Note that (x, ¯ F (x)) ¯ is a boundary point of epi F . Hence by the Hahn–Banach separation theorem, there exists a closed hyperplane S = {(x, a) ∈ X × R : x ∗ , x + α a = β} for nontrivial (x ∗ , α) ∈ X ∗ × R and β ∈ R such that x ∗ , x + α a > β
for all (x, a) ∈ (epi F )o ,
x ∗ , x
¯ + α F (x) ¯ = β.
(4.2.6)
Since (epi F )o = epi F , every neighborhood of (x, a) ∈ epi F contains an element of (epi ϕ)o . Suppose x ∗ , x + α a < β. Then {(x, ˜ a) ˜ ∈ X × R : x ∗ , x
˜ + α a˜ < β}
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 98 i
98 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization is a neighborhood of (x, a) and contains an element of (epi ϕ)o , which contradicts (4.2.6). Therefore x ∗ , x + α a ≥ β
for all (x, a) ∈ epi F.
(4.2.7)
Suppose α = 0. For any u ∈ U there is an a ∈ R such that F (u) ≤ a. Then from (4.2.7) x ∗ , u = x ∗ , u + α a ≥ β and thus
x ∗ , u − x
¯ ≥0
for all u ∈ U .
Choose δ > 0 such that |u − x| ¯ ≤ δ implies u ∈ U . For any nonzero element x ∈ X let δ t = |x| . Then |(tx + x) ¯ − x| ¯ = |tx| = δ so that tx + x¯ ∈ U . Hence ¯ − x /t ¯ ≥ 0. x ∗ , x = x ∗ , (tx + x) Similarly, −t x + x¯ ∈ U and x ∗ , x = x ∗ , (−tx + x) ¯ − x /(−t) ¯ ≤ 0. Thus, x ∗ , x , x ∗ = 0, which is a contradiction. Therefore α is nonzero. From (4.2.6), (4.2.7) we have for a > F (x) ¯ that α(a − F (x)) ¯ > 0 and hence α > 0. Employing (4.2.6), (4.2.7) again implies that ∗ x − , x − x¯ + F (x) ¯ ≤ F (x) α ∗
for all x ∈ X and therefore − xα ∈ ∂F (x). ¯
4.3
Fenchel duality theory
In this section we discuss some elements of duality theory. We call inf F (x)
x∈X
(P )
the primal problem, where X is a real Banach space and F : X → (−∞, ∞] is a proper l.s.c. convex function. We have the following result for the existence of a minimizer. Theorem 4.25. Let X be reflexive and let F be a l.s.c. proper convex functional defined on X satisfying lim F (x) = ∞.
|x|→∞
(4.3.1)
Then there exists an x¯ ∈ X such that F (x) ¯ = inf {F (x) : x ∈ X}. Proof. Let η = inf {F (x) : x ∈ X} and let {xn } be a minimizing sequence such that limn→∞ F (xn ) = η. Condition (4.3.1) implies that {xn } is bounded in X. Since X is
i
i i
i
i
i
i
4.3. Fenchel duality theory
ItoKunisc 2008/6/12 page 99 i
99
reflexive, there exists a subsequence that converges weakly to some x¯ in X and it follows from Lemma 4.4 that F (x) ¯ = η. We embed (P ) into a family of perturbed problems inf (x, y),
(Py )
x∈X
where y ∈ Y is an embedding variable, Y is a Banach space, and : X × Y → (−∞, ∞] is a proper l.s.c. convex function with (x, 0) = F (x). Thus (P0 ) = (P ). For example, in terms of (4.1.1) we let F (x) = f (x) + ϕ( x) (4.3.2) and with Y = H (x, y) = f (x) + ϕ( x + y).
(4.3.3)
Definition 4.26. The dual problem of (P ) with respect to is defined by sup (−∗ (0, y ∗ )).
(P ∗ )
y ∗ ∈Y ∗
The value function of (Py ) is defined by h(y) = inf (x, y), x∈X
y ∈ Y.
In this section we analyze the relationship between the primal problem (P ) and its dual (P ∗ ). Throughout we assume that h(y) > −∞ for all y ∈ Y . Theorem 4.27. sup (P ∗ ) ≤ inf (P ). Proof. For any (x, y) ∈ X × Y and (x ∗ , y ∗ ) ∈ X∗ × Y ∗ we have x ∗ , x + y ∗ , y − (x, y) ≤ ∗ (x ∗ , y ∗ ). Thus 0 = 0, x + y ∗ , 0 ≤ F (x) + ∗ (0, y ∗ ) for all x ∈ X and y ∗ ∈ Y ∗ . Therefore sup (−∗ (0, y ∗ )) = sup (P ∗ ) ≤ inf (P ) = inf F (x). x∈X
y ∗ ∈Y ∗
Lemma 4.28. h is convex. Proof. The proof is established by contradiction. Suppose there exist y1 , y2 ∈ Y and θ ∈ (0, 1) such that θ h(y1 ) + (1 − θ ) h(y2 ) < h(θ y1 + (1 − θ) y2 ).
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 100 i
100 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization Then there exist c and > 0 such that θ h(y1 ) + (1 − θ ) h(y2 ) < c − < c < h(θ y1 + (1 − θ) y2 ). Set a1 = h(y1 ) +
θ
and a2 =
c − θ a1 c − − θ h(y1 ) = > h(y2 ). 1−θ 1−θ
By definition of h there exist x1 , x2 ∈ X such that h(y1 ) ≤ (x1 , y1 ) ≤ a1
and
h(y2 ) ≤ (x2 , y2 ) ≤ a2 .
Thus h(θ y1 + (1 − θ ) y2 ) ≤ (θ x1 + (1 − θ) x2 , θ y1 + (1 − θ) y2 ) ≤ θ (x1 , y1 ) + (1 − θ) (x2 , y2 ) ≤ θ a1 + (1 − θ) a2 = c, which is a contradiction. Hence h is convex. Lemma 4.29. For all y ∗ ∈ Y ∗ , h∗ (y ∗ ) = ∗ (0, y ∗ ). Proof. h∗ (y ∗ ) = sup ( y ∗ , y − h(y)) = sup ( y ∗ , y − inf (x, y)) y∈Y
x∈X
y∈Y
= sup sup ( y ∗ , y − (x, y)) = sup sup ( 0, x + y ∗ , y − (x, y)) y∈Y x∈X
=
y∈Y x∈X
( (0, y ∗ ), (x, y) − (x, y)) = ∗ (0, y ∗ ).
sup (x,y)∈X×Y
Theorem 4.30. If h is l.s.c. at 0, then inf (P ) = sup (P ∗ ). Proof. Since F is proper, h(0) = inf x∈X F (x) < ∞. Since h is convex by Lemma 4.28, it follows from Theorem 4.15 that h(0) = h∗∗ (0). Thus by Lemma 4.29 sup (P ∗ ) = sup (−∗ (0, y ∗ )) = sup ( y ∗ , 0 − h∗ (y ∗ )) y ∗ ∈Y ∗
y ∗ ∈Y ∗
= h∗∗ (0) = h(0) = inf (P ). Theorem 4.31. If h is subdifferentiable at 0, then inf (P ) = sup (P ∗ ) and ∂h(0) is the set of solutions of (P ∗ ). Proof. By Lemma 4.29 we have that y¯ ∗ solves (P ∗ ) if and only if −h∗ (y¯ ∗ ) = −∗ (0, y¯ ∗ ) = sup (−∗ (0, y ∗ )) y ∗ ∈Y ∗
= sup ( y ∗ , 0 − h∗ (y ∗ )) = h∗∗ (0). y ∗ ∈Y ∗
i
i i
i
i
i
i
4.3. Fenchel duality theory
ItoKunisc 2008/6/12 page 101 i
101
By Theorem 4.22 h∗∗ (0) + h∗∗∗ (y¯ ∗ ) = y¯ ∗ , 0 = 0 if and only if y¯ ∗ ∈ ∂h∗∗ (0). Since h∗∗∗ = h∗ by Theorem 4.16, we have y¯ ∗ ∈ ∂h∗∗ (y ∗ ) if and only if −h∗ (y¯ ∗ ) = h∗∗ (0). Consequently y¯ ∗ solves (P ∗ ) if and only if y ∗ ∈ ∂h∗∗ (0). Since ∂h(0) is not empty, ∂h(0) = ∂h∗∗ (0) by Theorem 4.17. Therefore ∂h(0) is the set of all solutions of (P ∗ ) and (P ∗ ) has at least one solution. Let y ∗ ∈ ∂h(0). Then y ∗ , x + h(0) ≤ h(x) for all x ∈ X. If {xn } is a sequence in X such that xn → 0, then lim inf h(xn ) ≥ lim y ∗ , xn + h(0) = h(0) n→∞
and h is l.s.c. at 0. By Theorem 4.30 inf (P ) = sup (P ∗ ). Corollary 4.32. If there exists an x¯ ∈ X such that (x, ¯ ·) is finite and continuous at 0, then h is continuous on an open neighborhood U of 0 and h = h∗∗ . Moreover, inf (P ) = sup (P ∗ ) and ∂h(0) is the set of solutions of (P ∗ ). Proof. First show that h is continuous. Clearly, (x, ¯ ·) is bounded above on an open neighborhood U of 0. Since for all y ∈ Y h(y) ≤ (x, ¯ y), h is bounded above on U . Since h is convex by Lemma 4.28 it is continuous by Theorem 4.6. Hence h = h∗∗ by Theorem 4.15. Moreover h is subdifferentiable at 0 by Theorem 4.24. The conclusion then follows from Theorem 4.31. Example 4.33. Consider the case (4.3.2)–(4.3.3), i.e., (x, y) = f (x) + ϕ( x + y), where f : X → (−∞, ∞], ϕ : Y → (−∞, ∞] are l.s.c. and convex and : X → Y is a continuous linear operator. Let us calculate the conjugate of : ∗ (x ∗ , y ∗ ) = sup sup{ x ∗ , x + y ∗ , y − (x, y)} x∈X y∈Y
= sup{ x ∗ , x − f (x) + sup[ y ∗ , y − ϕ( x + y)]}, x∈X
y∈Y
where sup[ y ∗ , y − ϕ( x + y)] = sup[ y ∗ , x + y − ϕ( x + y) − y ∗ , x ] y∈Y
y∈Y
= sup[ y ∗ , z − ϕ(z)] − y ∗ , x = ϕ ∗ (y ∗ ) − y ∗ , x . z∈Y
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 102 i
102 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization Thus ∗ (x ∗ , y ∗ ) = sup{ x ∗ , x − y ∗ , x − f (x) + ϕ ∗ (y ∗ )} x∈X
= sup{ x ∗ − ∗ y ∗ , x − f (x) + ϕ ∗ (y ∗ )} = f ∗ (x ∗ − y ∗ ) + ϕ ∗ (y ∗ ). x∈X
Theorem 4.34. For any x¯ ∈ X and y¯ ∗ ∈ Y ∗ , the following statements are equivalent. (1) x¯ solves (P ), y¯ ∗ solves (P ∗ ), and min (P ) = max (P ∗ ). (2) (x, ¯ 0) + ∗ (0, y¯ ∗ ) = 0. (3) (0, y¯ ∗ ) ∈ ∂(x, ¯ 0). Proof. Clearly (1) implies (2). If (2) holds, then (x, ¯ 0) = F (x) ¯ ≥ inf (P ) ≥ sup (P ∗ ) ≥ −∗ (0, y¯ ∗ ) by Theorem 4.27. Thus (x, ¯ 0) = min (P ) = max (P ∗ ) = −∗ (0, y¯ ∗ ). Therefore (2) implies (1). Since (0, y¯ ∗ ), (x, ¯ 0) = 0, equivalence of (2) and (3) follows by Theorem 4.22. Any solution y ∗ of (P ∗ ) is called a Lagrange multiplier associated with . For Example 4.33 the optimality condition implies 0 = (x, ¯ 0) + ∗ (0, y¯ ∗ ) = f (x) ¯ + f ∗ (− ∗ y¯ ∗ ) + ϕ( x) ¯ + ϕ(y¯ ∗ ) = [f (x) ¯ + f ∗ (− ∗ y¯ ∗ ) − − ∗ y¯ ∗ , x ] ¯ + [ϕ( x) ¯ + ϕ ∗ (y¯ ∗ ) − y¯ ∗ , x)]. ¯
(4.3.4)
Since each expression of (4.3.4) in square brackets is nonnegative, it follows that ¯ = 0, f (x) ¯ + f ∗ (− ∗ y¯ ∗ ) − − ∗ y¯ ∗ , x
ϕ( x) ¯ + ϕ ∗ (y¯ ∗ ) − y¯ ∗ , x) ¯ = 0. By Theorem 4.22 ¯ − ∗ y¯ ∗ ∈ ∂f (x), y¯ ∗ ∈ ∂ϕ( x). ¯
(4.3.5)
The functional L : X × Y ∗ → (−∞, ∞] defined by −L(x, y ∗ ) = sup { y ∗ , y − (x, y)}
(4.3.6)
y∈Y
i
i i
i
i
i
i
4.3. Fenchel duality theory
ItoKunisc 2008/6/12 page 103 i
103
is called the Lagrangian. Note that ∗ (x ∗ , y ∗ ) =
{ x ∗ , x + y ∗ , y − (x, y)}
sup x∈X, y∈Y
= sup x ∗ , x + sup { y ∗ , y − (x, y)}} = sup ( x ∗ , x − L(x, y ∗ )). x∈X
y∈Y
Thus
x∈X
−∗ (0, y ∗ ) = inf L(x, y ∗ ) x∈X
(4.3.7)
and therefore the dual problem (P ∗ ) is equivalent to sup inf L(x, y ∗ ).
y ∗ ∈Y ∗ x∈X
If is a convex l.s.c. function that is finite at (x, y), then for the biconjugate of x : y → (x, y) in y we have x (y)∗∗ = (x, y) and ∗ ∗ ∗ (x, y) = ∗∗ x (x, y) = sup { y , y − x (y )} y ∗ ∈Y ∗
= sup { y ∗ , y + L(x, y ∗ )}. y ∗ ∈Y ∗
Hence
(x, 0) = sup L(x, y ∗ ) y ∗ ∈Y ∗
(4.3.8)
and the primal problem (P ) is equivalently written as inf sup L(x, y ∗ ).
x∈X y ∗ ∈Y
Thus by means of the Lagrangian L, the problems (P ) and (P ∗ ) can be formulated as min-max problems, and by Theorem 4.27 sup inf L(x, y ∗ ) ≤ inf sup L(x, y ∗ ). y∗
x
x
y∗
Theorem 4.35 (Saddle Point). Assume that is a convex l.s.c. function that is finite at (x, ¯ y¯ ∗ ). Then the following are equivalent. (1) (x, ¯ y¯ ∗ ) ∈ X × Y ∗ is a saddle point of L, i.e., L(x, ¯ y ∗ ) ≤ L(x, ¯ y¯ ∗ ) ≤ L(x, y¯ ∗ ) for all x ∈ X, y ∗ ∈ Y ∗ .
(4.3.9)
(2) x¯ solves (P ), y¯ ∗ solves (P ∗ ), and min (P ) = max (P ∗ ). Proof. Suppose (1) holds. From (4.3.7) and (4.3.9) L(x, ¯ y¯ ∗ ) = inf L(x, y¯ ∗ ) = −∗ (0, y¯ ∗ ) x∈X
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 104 i
104 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization and from (4.3.8) and (4.3.9) L(x, ¯ y¯ ∗ ) = sup L(x, ¯ y ∗ ) = (x, ¯ 0). y ∗ ∈Y ∗
Thus, ∗ (x, ¯ 0) + (0, y¯ ∗ ) = 0 and (2) follows from Theorem 4.34. Conversely, if (2) holds, then from (4.3.7) and (4.3.8) −∗ (0, y¯ ∗ ) = inf L(x, y¯ ∗ ) ≤ L(x, ¯ y¯ ∗ ), x∈X
¯ y ∗ ) ≥ L(x, ¯ y¯ ∗ ). (x, ¯ 0) = sup L(x, y ∗ ∈Y ∗
Consequently −∗ (0, y¯ ∗ ) = (x, ¯ 0) by Theorem 4.34 and (4.3.9) holds. Theorem 4.35 implies that no duality gap between (P ) and (P ∗ ) is equivalent to the saddle point property of the pair (x, ¯ y¯ ∗ ). For Example 4.33 we have L(x, y ∗ ) = f (x) + y ∗ , x − ϕ(y ∗ ).
(4.3.10)
If (x, ¯ y¯ ∗ ) is a saddle point, then from (4.3.9) − ∗ y¯ ∗ ∈ ∂f (x), ¯ x¯ ∈ ∂ϕ ∗ (x). ¯
(4.3.11)
It follows from Theorem 4.22 that the second equation is equivalent to y¯ ∗ ∈ ∂ϕ( x) ¯ and (4.3.11) is equivalent to (4.3.5), if X is reflexive. Thus the necessary optimality system for min F (x) = f (x) + ϕ( x) x∈X
is given by
¯ − ∗ y¯ ∗ ∈ ∂f (x), y¯ ∗ ∈ ∂ϕ ∗ ( x). ¯
4.4
(4.3.12)
Generalized Yosida–Moreau approximation
Definition 4.36 (Monotone Operator). Let A be a graph in X × X ∗ . (1) A is monotone if y1 − y2 , x1 − x2 ≥ 0 for all x1 , x2 ∈ D(A) and y1 ∈ Ax1 , y2 ∈ Ax2 (2) A is maximal monotone if any monotone extension of A coincides with A.
i
i i
i
i
i
i
4.4. Generalized Yosida–Moreau approximation
ItoKunisc 2008/6/12 page 105 i
105
Let ϕ be an l.s.c. convex function on X. For x1∗ ∈ ∂ϕ(x1 ) and x2∗ ∈ ∂ϕ(x2 ), ϕ(x1 ) − ϕ(x2 ) ≤ x2∗ , x1 − x2 , ϕ(x2 ) − ϕ(x1 ) ≤ x1∗ , x2 − x1 . It follows that x1∗ − x2∗ , x1 − x2 ≥ 0. Hence ∂ϕ is a monotone in X × X∗ . The following theorem characterizes maximal monotone operators by a range condition [ItKa]. Theorem 4.37 (Minty–Browder). Assume that X and X∗ are reflexive and strictly convex Banach spaces and let F : X → X∗ denote the duality mapping. Then a monotone operator A is maximal monotone if and only if R(λ F + A) = X∗ for all λ > 0 (or, equivalently, for some λ > 0). Theorem 4.38 (Rockafeller). Let X be a real Banach space. If ϕ is an l.s.c. proper convex functional on X, then ∂ϕ is a maximal monotone operator from X into X∗ . Proof. We prove the theorem for the case that X is reflexive . The general case is considered in [Roc2]. ByAsplund’s renorming theorem we can assume that after choosing an equivalent norm, X and X∗ are strictly convex. Using the Minty–Browder theorem it suffices to prove that R(F + ∂ϕ) = X∗ . For x0∗ ∈ X∗ we must show that the equation x0∗ ∈ F x + ∂ϕ(x) has at least a solution x0 . Note that F x is single valued due to the fact that X ∗ is strictly convex. Define the proper convex functional f : X → (−∞, ∞] by f (x) =
1 2 |x| + ϕ(x) − x0∗ , x . 2 X
Since f is l.s.c. and f (x) → ∞ as |x| → ∞, there exists x0 ∈ D(f ) such that f (x0 ) ≤ f (x) for all x ∈ X. The subdifferential of the mapping x → 12 |x| 2 is given by the monotone operator F . Hence we find ϕ(x) − ϕ(x0 ) ≥ x0∗ , x − x0 − F (x), x − x0 for all x ∈ X. Setting x(t) = x0 + t (u − x0 ) with u ∈ X and using convexity of ϕ we have ϕ(u) − ϕ(x0 ) ≥ x0∗ , u − x0 − F (x(t)), u − x0 . Taking the limit t → 0+ and using the fact that x → F (x) is continuous from X endowed with the norm topology to X ∗ endowed with the weak∗ topology, we obtain ϕ(u) − ϕ(x0 ) ≥ x0∗ , u − x0 − F (x0 ), u − x0 , which implies that x0∗ − F (x0 ) ∈ ∂ϕ(x0 ). Throughout the remainder of this section let H denote a real Hilbert space which is identified with its dual H ∗ . Further let A denote a maximal monotone operator in H × H . Recall that A is necessarily densely defined and closed. Moreover the resolvent
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 106 i
106 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization Jμ = (I + μ A)−1 , with μ > 0, is a contraction defined on all of H ; see [Bre, ItKa, Paz]. Moreover |Jμ x − x| → 0 as μ → 0+ for each x ∈ H. (4.4.1) The Yosida approximation Aμ of A is defined by Aμ x =
1 (x − Jμ x). μ
The operator Aμ is single valued, monotone, everywhere defined, Lipschitz continuous with Lipschitz constant μ1 and Aμ x ∈ AJμ x for all x ∈ H . Let ϕ be an l.s.c., proper, and convex function on H . Throughout the remainder of this chapter let A denote the maximal monotone operator ∂ϕ on the Hilbert space H = H ∗ . For x, λ ∈ H and c > 0 define the functional ϕc (x, λ) by c ϕc (x, λ) = inf ϕ(x − u) + (λ, u)H + |u|2H . (4.4.2) u∈H 2 Then ϕc (x, λ) is a smooth approximation of ϕ in the following sense. Theorem 4.39. For x, λ ∈ H the infimum in (4.4.2) is attained at a unique point λ uc (x, λ) = x − J 1 x + . c c ϕc (x, λ) is convex, Lipschitz continuously Fréchet differentiable in x and λ ϕc (x, λ) = λ + c uc (x, λ) = A 1 x + . c c Moreover, limc→∞ ϕc (x, λ) = ϕ(x) and λ 1 ϕ J1 x + − |λ|2 ≤ ϕc (x, λ) ≤ ϕ(x) for every x, λ ∈ H. c c 2c H
Proof. First we observe that (4.4.2) is equivalent to
2
c
1 λ ϕc (x, λ) = inf ϕ(v) + x + − v
− |λ|2 , v∈H 2 c 2c
(4.4.3)
where v = x − u. Note that v → ψ(v) = ϕ(v) + 2c |v − y|2 , with y = x + λc , is convex and l.s.c. for every y ∈ H . Since ϕ is proper, D(A) = ∅, and thus for x0 ∈ D(A), and x0∗ ∈ Ax0 we have ϕ(x) ≥ ϕ(x0 ) + (x0∗ , x − x0 )
for all x ∈ H.
Thus, lim ψ(v) = ∞ as |v| → ∞. Hence there exists a unique minimizer v0 ∈ H of ψ. Let ξ ∈ H and define η = (1 − t)v0 + tξ for 0 < t < 1. Since ϕ is convex we have ϕ(η) − ϕ(v0 ) ≤ t (ϕ(ξ ) − ϕ(v0 )).
(4.4.4)
i
i i
i
i
i
i
4.4. Generalized Yosida–Moreau approximation
ItoKunisc 2008/6/12 page 107 i
107
Moreover, since ψ(v0 ) ≤ ψ(η), ϕ(η) − ϕ(v0 ) ≥
c (|v0 − y|2 − |(1 − t)v0 + tξ − y|2 ) 2 t 2c = tc (v0 − ξ, v0 − y) − |v0 − ξ |2 . 2
(4.4.5)
From (4.4.4) and (4.4.5) we obtain, taking the limit t → 0+ , ϕ(ξ ) − ϕ(v0 ) ≥ c (y − v0 , ξ − v0 ). Since ξ ∈ H is arbitrary this implies that y − v0 ∈
1 c
Av0 . Thus v0 = J 1 y and c
uc (x, λ) = x − v0 = x − J 1 c
x+
λ c
attains the minimum in (4.4.2). Note that this argument also implies that A is maximal. For x1 , x2 ∈ X and 0 < t < 1 ϕc ((1 − t)x1 + tx2 , λ) = ψ((1 − t)v1 + tv2 ) −
1 2 |λ| , 2c
where yi = xi + λc and vi = J 1 yi for i = 1, 2. Hence the convexity of x → ϕc (x, λ) c follows from the one of ψ. Next, we show that ∂ϕc (x, λ) = A 1 (x + λc ). For xˆ ∈ H , let yˆ = xˆ + λc ∈ H and c vˆ = J 1 y. ˆ Then, we have c
ϕ(v) ˆ +
c c |vˆ − y|2 ≥ ϕ(v0 ) + |v0 − y|2 2 2
ϕ(v0 ) +
c c ˆ 2 ≥ ϕ(v) ˆ + |vˆ − y| |v0 − y| ˆ 2. 2 2
and
Thus, c c ˆ 2 − |v0 − y|2 ) ≥ ϕc (x, ˆ λ) − ϕc (x, λ) ≥ (|vˆ − y| (|v0 − y| ˆ 2 − |vˆ − y|2 ). 2 2
(4.4.6)
Since |vˆ − v0 | → 0 as |xˆ − x| → 0, it follows from (4.4.6) that ˆ λ) − ϕc (x, λ) − c(y − v0 ), xˆ − x | |ϕc (x, →0 |xˆ − x| as |xˆ − x| → 0. Hence x → ϕc (x, λ) is Fréchet differentiable with F -derivative λ c(y − v0 ) = λ + c uc (x, λ) = A 1 x + . c c
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 108 i
108 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization The dual representation of ϕc (x, λ) is derived next. Define the functional (v, y) on H × H by c (v, y) = ϕ(v) + |v − (yˆ + y)|2 , 2 where we set yˆ = x + λc . Consider the family of primal problems inf (v, y)
v∈H
and the dual problem
for each y ∈ H
(Py )
sup {−∗ (0, y ∗ )}.
(P ∗ )
y ∗ ∈H
If h(y) is the value functional of (Py ), i.e., h(y) = inf v∈H (v, y), then from (4.4.3) we have ϕc (x, λ) = h(0) −
1 |λ|2 . 2c
(4.4.7)
From the proof of Theorem 4.39 it follows that h(y) is continuously Fréchet differentiable with h (0) = ϕc (x, λ). It thus follows from Theorem 4.31 that inf (P0 ) = max (P ∗ ) and h (0) is the solution of (P ∗ ). This leads to the following theorem. Theorem 4.40. For x, λ ∈ H
ϕc (x, λ) = sup
y ∗ ∈H
1 ∗ 2 (x, y )H − ϕ (y ) − |y − λ|H , 2c ∗
∗
∗
where the supremum is attained at a unique point λc (x, λ) and we have λ λc (x, λ) = λ + c uc (x, λ) = ϕc (x, λ) = A 1 x + . c c
(4.4.8)
(4.4.9)
Proof. For (v ∗ , y ∗ ) ∈ H × H we have by definition c ∗ (v ∗ , y ∗ ) = sup sup (v ∗ , v) + (y ∗ , y) − ϕ(v) − |v − (yˆ + y)|2 2 v∈H y∈H
= sup (v ∗ , v) + (y ∗ , v − y) ˆ − ϕ(v) v∈H
1
c + sup − y ∗ , v − (yˆ + y) − |v − (yˆ + y)|2 2 y∈H
2
1 ∗2 |y | ˆ + = sup (y ∗ + v ∗ , v) − ϕ(v) − (y ∗ , y) 2c v∈H ˆ + = ϕ ∗ (y ∗ + v ∗ ) − (y ∗ , y)
1 ∗2 |y | . 2c
i
i i
i
i
i
i
4.5. Optimality systems
ItoKunisc 2008/6/12 page 109 i
109
Hence, ∗
∗
h(0) = sup {− (0, y )} = sup y ∗ ∈H
y ∗ ∈H
1 ∗2 (y , y) ˆ − ϕ (y ) − |y | 2c ∗
∗
∗
which implies (4.4.8) from (4.4.7) and since yˆ = x + λc . By Theorem 4.31 the maximum of y ∗ → y ∗ , x − ϕ ∗ (y ∗ ) − 2c1 |y ∗ − λ|2 is attained at the unique point h (0) = λc (x, λ) that is given by (4.4.9). The following theorem provides an equivalent characterization of λ ∈ ∂ϕ(x). This results will be used in the following section to replace the differential inclusion, which relates the primal and adjoint variable by means of a nonlinear equation. Theorem 4.41. (1) If λ ∈ ∂ϕ(x) for x, λ ∈ H , then λ = ϕc (x, λ) for all c > 0. (2) Conversely, if λ = ϕc (x, λ) for some c > 0, then λ ∈ ∂ϕ(x). Proof. If λ ∈ ∂ϕ(x), then by Theorems 4.39, 4.40, and 4.22 ϕ(x) ≥ ϕc (x, λ) ≥ λ, x − ϕ ∗ (λ) = ϕ(x) for every c > 0. Thus, λ ∈ H attains the supremum in (4.4.8) and by Theorem 4.40 we have λ = ϕc (x, λ). Conversely, if λ ∈ H satisfies λ = ϕc (x, λ) for some c > 0, then uc (x, λ) = 0 by Theorem 4.40. Hence it follows from Theorem 4.39, (4.4.2), and Theorem 4.40 that ϕ(x) = ϕc (x, λ) = λ, x − ϕ ∗ (λ), and thus λ ∈ ∂ϕ(x) by Theorem 4.22.
4.5
Optimality systems
In this section we derive first order optimality systems for (4.1.1) based on Lagrange multipliers. In what follows we assume that f : X → R and ϕ : H → (−∞, ∞] are l.s.c., convex functions, that f is continuously differentiable, and that ϕ is proper. Further ∈ L(X, H ) and C is a closed convex set in X. In addition we require that (A1) f, ϕ are bounded below by zero on K, (A2) f (x1 ) − f (x2 ), x1 − x2 X∗ ,X ≥ σ |x1 − x2 |2X for some σ > 0 independent of x1 , x2 ∈ C, and (A3) ϕ( x0 ) < ∞ and ϕ is continuous at x0 for some x0 ∈ C.
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 110 i
110 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization As a consequence of (A2) we have f (x) − f (x0 ) − f (x0 ), x − x0 X∗ ,X 1 σ = f (x0 + t (x − x0 )) − f (x0 ), x − x0 X∗ ,X dt ≥ |x − x0 |2X 2 0
(4.5.1)
for all x ∈ X. Due to Theorem 4.24 there exists y0∗ ∈ D(A(y0 )) = ∂ϕ(y0 ) such that ϕ( x) − ϕ( x0 ) ≥ (y0∗ , x − x0 )H
for x0 ∈ H.
(4.5.2)
Hence, lim f (x) + ϕ( x) → ∞ as |x|X → ∞ and it follows from Theorem 4.25 and (A2) that there exists a unique minimizer x¯ ∈ C of (4.1.1). Theorem 4.42 (Optimality). A necessary and sufficient condition for x¯ ∈ C to be the minimizer of (4.1.1) is given by f (x), ¯ x − x
¯ X∗ ,X + ϕ( x) − ϕ( x) ¯ ≥ 0 for all x ∈ C.
(4.5.3)
Proof. Assume that x¯ is the minimizer of (4.1.1). Then for x ∈ C and 0 < t < 1 we have x¯ + t (x − x) ¯ ∈ C and f (x¯ + t (x − x)) ¯ + ϕ( (x¯ + t (x − x)) ¯ ≥ f (x) ¯ + ϕ( x). ¯ Since ϕ( ((1 − t)x¯ + tx)) − ϕ( x) ¯ ≤ t (ϕ( x) − ϕ( x)), ¯ we obtain ¯ − f (x)) ¯ + ϕ( x) − ϕ( x) ¯ ≥ 0. t −1 (f (x¯ + t (x − x)) Taking the limit t → 0+ , we have (4.5.3) for all x ∈ C. Conversely, assume that x¯ ∈ C satisfies (4.5.3). Then, from (4.5.1), f (x) + ϕ( x) − (f (x) ¯ + ϕ( x)) ¯ ¯ x − x
¯ + f (x), ¯ x − x
¯ + ϕ( x) − ϕ( x) ¯ = f (x) − f (x) ¯ − f (x), ≥
σ |x − x| ¯ 2X 2
for all x ∈ C. Thus, x¯ is a minimizer of (4.1.1). Next, ϕ is split from the optimality system in (4.5.3) by means of an appropriately defined Lagrange multiplier. For this purpose we consider the regularized minimization problems: min f (x) + ϕc ( x, λ) over x ∈ C
(4.5.4)
i
i i
i
i
i
i
4.5. Optimality systems
ItoKunisc 2008/6/12 page 111 i
111
for c > 0 and λ ∈ H . From Theorem 4.39 it follows that x → ϕc ( x, λ) is convex, continuously differentiable, and bounded below by − 2c1 |λ|2H . Thus, for λ ∈ H , (4.5.4) has a unique solution xc ∈ C and xc ∈ C is the solution of (4.5.4) if and only if xc satisfies f (xc ), x − xc + ϕc ( xc , λ), (x − xc ) H ≥ 0 for all x ∈ C. (4.5.5) It follows from Theorems 4.39 and 4.40 that 1 ϕc ( xc , λ) = A 1 xc + λ = λc ∈ H. c c
(4.5.6)
We have the following result. Theorem 4.43 (Lagrange Multipliers). (1) Suppose that xc converges strongly to x¯ in X as c → ∞ and that {λc }c≥1 has a weak cluster point. Then for each weak cluster point λ¯ ∈ H of {λc }c≥1 λ¯ ∈ ∂ϕ( x) ¯ and
¯ (x − x) f (x), ¯ x − x
¯ X∗ ,X + λ, ¯ H ≥ 0 for all x ∈ C.
(4.5.7)
¯ ∈ C × H satisfies (4.5.7), then x¯ minimizes (4.1.1). Conversely, if (x, ¯ λ) (2) Assume that there exists λ˜ c ∈ ∂ϕ( xc ) for c ≥ 1 such that {|λ˜ c |H }, c ≥ 1, is bounded. Then, xc converges strongly to x¯ in X as c → ∞. Proof. To verify (1), assume that xc → x¯ and let λ¯ be a weak cluster point of λc in H . ¯ ∈ C × H satisfies Then, from (4.5.5)–(4.5.6), we can conclude that (x, ¯ λ) ¯ (x − x) f (x), ¯ x − x
¯ + λ, ¯ H ≥0 for all x ∈ C. It follows from Theorems 4.39 and 4.40 that 1 λc , vc H − |λc − λ|2 = ϕc ( xc , λ) + ϕ ∗ (λc ) 2c 1 λ ≥ ϕ J 1 vc + − |λ|2 + ϕ ∗ (λc ). c c 2c H Since J 1 (vc + λc ) → v¯ = x¯ and ϕ and ϕ ∗ are l.s.c., letting c → ∞, we obtain c ¯ v¯ ≥ ϕ(v) ¯ λ, ¯ + ϕ ∗ (λ), H ¯ ∈ C × H satisfies (4.5.7). which implies that λ¯ ∈ ∂ϕ(v) ¯ by Theorem 4.22. Hence, (x, ¯ λ) ¯ ∈ C × H satisfies (4.5.7). Then ϕ( x) − ϕ( x) Conversely, suppose that ( x, ¯ λ) ¯ ≥ ¯ (x − x) λ, ¯ H for all x ∈ C. Thus the inequality in (4.5.7) implies (4.5.3). It then follows from Theorem 4.42 that x¯ minimizes (4.1.1). For (2) we note that from (4.5.5) we have f (xc ), x − xc + ϕc ( x, λ) − ϕc ( xc , λ) ≥ 0
for all x ∈ C.
Thus f (xc ), x¯ − xc + ϕc ( x, ¯ λ) − ϕc ( xc , λ) ≥ 0.
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 112 i
112 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization Also, from (4.5.3) f (x), ¯ xc − x
¯ + ϕ( xc ) − ϕ( x) ¯ ≥ 0. Adding these inequalities, we obtain f (x) ¯ − f (xc ), x¯ − xc + ϕc ( xc , λ) − ϕ( xc ) (4.5.8) ≤ ϕc ( x, ¯ λ) − ϕ( x) ¯ ≤ 0. By (4.4.8) ϕ(vc ) −
1 1 |λ˜ c − λ|2H = vc , λ˜ c H − ϕ ∗ (λ˜ c ) − |λ˜ c − λ|2H ≤ ϕc (vc , λ), 2c 2c
(4.5.9)
where vc = xc and hence ϕ(vc ) − ϕc (vc , λ) ≤
1 |λ˜ c − λ|2H . 2c
Thus, from (4.5.8) and (A2) σ 1 |xc − x ∗ |2X ≤ |λ˜ c − λ|2H 2 2c and since {|λ˜ c |H }c≥1 is bounded, |xc − x ∗ |X → 0 as c → ∞. The following lemma addresses the condition in Theorem 4.43 (2). Lemma 4.44. (1) If D(ϕ) = H and bounded above, on bounded sets, then ∂ϕ( xc ) is nonempty and |∂ϕ( xc )|H is uniformly bounded for c ≥ 1. (2) If ϕ = χK with K a closed convex set in H and xc ∈ K for all c > 0, then λ˜ c can be chosen to be 0 for all c > 0. Proof. (1) By definition of xc and by Theorem 4.39 f (xc ) + ϕc (vc , λ) ≤ f (x) ¯ + ϕ( x) ¯ and
f (xc ) + ϕ J 1 c
λ vc + c
≤ f (x) ¯ + ϕ( x) ¯ +
1 |λ|2 , 2c H
where vc = xc . Since ϕ is bounded below by 0, there exists a constant K1 > 0 independent of c ≥ 1 such that f (xc ) − f (x) ¯ − f (x), ¯ xc − x
¯ ≤ K1 + f (x)|x ¯ ¯ X. c − x| By (4.5.1) σ |xc − x| ¯ 2X ≤ K1 + f (x)|x ¯ ¯ X. c − x| 2 Thus there exists a constant M such that |xc − x| ¯ X ≤ M for c ≥ 1. Since ϕ is everywhere defined, convex, l.s.c., and assumed to be bounded above on bounded sets, it follows from
i
i i
i
i
i
i
4.5. Optimality systems
ItoKunisc 2008/6/12 page 113 i
113
Theorem 4.7 that ϕ is Lipschitz continuous in the open set B = {v ∈ H : |v − v| ¯ < (M + 1) }, where v¯ = x. ¯ Let L denote the Lipschitz constant. By Theorem 4.24 ∂ϕ(vc ) is nonempty for c ≥ 1. Let λ˜ c ∈ ∂ϕ(vc ) for c ≥ 1. Hence L |v − vc | ≥ ϕ(v) − ϕ(vc ) ≥ λ˜ c , v − vc for all v ∈ B. Since vc ∈ B we have |λ˜ c | ≤ L, and the claim follows. (2) In this case ∂ϕ(vc ) is given by the normal cone NC (vc ) = {w ∈ H : (w, vc −v)H ≥ 0 for all v ∈ C} and thus 0 ∈ ∂ϕ(vc ) for all c > 0. ¯ ∈ C × H that Theorem 4.45 (Complementarity). Assume that there exists a pair (x, ¯ λ) ¯ satisfies (4.5.7). Then the complementarity condition λ ∈ ∂ϕ( x) ¯ can equivalently be expressed as ¯ λ¯ = ϕc ( x, ¯ λ)
(4.5.10)
¯ min f (x) + ϕc ( x, λ)
(4.5.11)
and x¯ is the unique solution of x∈C
for every c > 0. Proof. The first claim follows directly from Theorem 4.41. From (4.5.5) we conclude that xˆ is a minimizer of (4.5.11) if and only if xˆ ∈ C satisfies ¯ (x − x) f (x), ˆ x − x
ˆ X∗ ,X + ϕc ( x, ˆ λ), ˆ H ≥ 0 for all x ∈ C. From (4.5.7) and (4.5.10) it follows that x¯ ∈ C satisfies this inequality as well and hence xˆ = x. ¯ ¯ ∈ C × H satisfies Theorems 4.43 and 4.45 imply that if a pair (x, ¯ λ) ⎧ ¯ (x − x) ¯ x − x
¯ + λ, ¯ ≥ 0 for all x ∈ C, ⎨ f (x), ⎩ ¯ ¯ λ = ϕc ( x, ¯ λ)
(4.5.12)
for some c > 0, then x¯ is the minimizer of (4.1.1). Conversely, if x¯ is a minimizer of (4.1.1) and ∂(ϕ ◦ + ψC )(x) ¯ = ∗ ∂ϕ( x) ¯ + ∂ψC (x), ¯
(4.5.13)
¯ satisfies (4.5.12) for all c > 0. Here ψC then there exists a λ¯ ∈ H such that the pair (x, ¯ λ) denotes the indicator function of the set C. In fact it follows from (4.5.7) that −f (x) ¯ ∈ ∂(ϕ ◦ + ψC )(x) ¯ and by (4.5.13) −f (x) ¯ ∈ ∗ ∂ϕ( x) ¯ + ∂ψC (x) ¯ = ∗ ∂ϕ( x) ¯ + NC (x), ¯ where NC (x) ¯ = {z ∈ X ∗ : z, x − x
¯ ≤ 0 for all x ∈ C}. This implies that there exists ¯ Condition some λ¯ ∈ ∂ϕ( x) ¯ such that (4.5.7) holds and also (4.5.12) for the pair (x, ¯ λ). (4.5.13) holds, for example, if there exists x ∈ int (C) and ϕ is continuous and finite at x (see, e.g., Propositions 12 and 13 in Section 3.2 of [EkTu]).
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 114 i
114 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization The following theorem discusses the equivalence between the existence of a Lagrange multiplier λ¯ and uniform boundedness of λc . Theorem 4.46. Suppose is compact and let λ ∈ H be fixed. Then λc = ϕc (vc , λ) is uniformly bounded in c ≥ 1 if and only if there exists λ∗ ∈ ∂ϕ( x) ¯ such that (4.5.7) holds. In either case there exists a subsequence (xcˆ , λcˆ ) ∈ X × H such that xcˆ → x¯ strongly in X ¯ ∈ C × H satisfies (4.5.7). and λcˆ → λ∗ weakly in H where (x, ¯ λ) Proof. Assume that |λc |H is uniformly bounded. From (4.5.5) f (xc ) + ∗ λc , x − xc ≥ 0
for all x ∈ C,
where λc = ϕc ( xc , λ). From (A2) it follows that ¯ 2X ≤ f (xc ) − f (x), ¯ xc − x
¯ ≤ f (x) ¯ − ∗ λc , xc − x
¯ σ |xc − x| ≤ ( |λc | + f (x)) ¯ |xc − x|, ¯ which by assumption implies that |xc |X is uniformly bounded. Since is compact it follows that any weakly convergent sequence λc → λ∗ in H satisfies ∗ λc → ∗ λ¯ strongly in X ∗ . Again, from (A2) we have σ |xc − xcˆ |X ≤ f (xc ) − f (xcˆ ), xc − xcˆ ≤ | ∗ λc − ∗ λcˆ |X∗ for any c, cˆ > 0. Hence {xc } is a Cauchy sequence in X and thus there exists x¯ ∈ X such ¯ X → 0. The rest of the proof for the “only if” part is identical to the one of the that |xc − x| first part of Theorem 4.43. It will be shown in Theorem 4.49 that 1 1 σ ¯ 2H ≤ ¯ 2H ¯ 2X + |xc − x| |λc − λ| |λ − λ| 2 2c 2c if (4.5.7) holds. This implies the “if” part.
4.6 Augmented Lagrangian method In this section we discuss the augmented Lagrangian method for (4.1.1). Throughout we assume that (A1)–(A3) of Section 4.5 hold and we set Lc (x, λ) = f (x) + φc ( x, λ). Augmented Lagrangian Method. Step 1: Choose a starting value λ1 ∈ H a positive number c and set k = 1. Step 2: Given λk ∈ H find xk ∈ C by Lc (xk , λk ) = min Lc (x, λk )
over x ∈ C.
Step 3: Update λk by λk+1 = ϕc ( xk , λk ). Step 4: If the convergence criterion is not satisfied, then set k = k + 1 and go to Step 2.
i
i i
i
i
i
i
4.6. Augmented Lagrangian method
ItoKunisc 2008/6/12 page 115 i
115
Before proving convergence of the augmented Lagrangian method we motivate the update of the Lagrange multiplier in Step 3. First, it follows from (4.5.12) that it is a fixed point iteration for the necessary optimality condition in the dual variable λ. To give the second motivation we define the dual functional dc : H → R by dc (λ) = inf f (x) + ϕc ( x, λ) over x ∈ C
(4.6.1)
for λ ∈ H . The update is then a steepest ascent step for dc (λ). In fact it will be shown in the following lemma that (4.6.1) attains the minimum at a unique minimizer x(λ) ∈ C and that dc is continuously Fréchet differentiable with F -derivative dc (λ) = u( x(λ), λ), where uc (x, λ) is defined in Theorem 4.39. Thus the steepest ascent step is given by λk+1 = λk + c u( x(λk ), λk ), which by Theorem 4.40 coincides with the update given in Step 3. Lemma 4.47. For λ ∈ H and c > 0 the infimum in (4.6.1) is attained at a unique minimizer x(λ) ∈ C and the mapping λ ∈ H → x(λ) ∈ X is Lipschitz continuous with Lipschitz constant σ −1 . Moreover, the dual functional dc is continuously Fréchet differentiable with F-derivative dc (λ) = u( x(λ), λ), where uc (v, λ) is defined in Theorem 4.39. Proof. The proof is given in three steps. Step 1. Since f (x) + ϕc ( x, λ) is convex and l.s.c. and since (4.5.1) holds, there exists a unique x(λ) that attains its minimum over K. To establish Lipschitz continuity of λ → x(λ) we note that x(λ) satisfies the necessary optimality condition f (x(λ)), x − x(λ) + ϕc ( x(λ), λ), (x − x(λ)) ≥ 0 for all x ∈ C. (4.6.2) Using this inequality at λ, μ ∈ H , we obtain f (x(μ)) − f (x(λ)), x(μ) − x(λ)
+ ϕc ( x(μ), μ) − ϕc ( x(λ), λ), (x(μ) − x(λ)) ≤ 0. By (A2) and (4.5.6) this implies that μ λ 2 σ |x(μ) − x(λ)|X + A 1 x(μ) + − A 1 x(λ) + , (x(μ) − x(λ)) ≤ 0, c c c c and thus
μ μ σ |x(μ) − x(λ)|2X + A 1 x(μ) + − A 1 x(λ) + , (x(μ) − x(λ)) c c c c λ μ ≤ A 1 x(λ) + − A 1 x(λ) + , (x(μ) − x(λ)) . c c c c
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 116 i
116 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization Since A 1 is monotone and Lipschitz continuous with Lipschitz constant c, this inequality c yields σ |x(μ) − x(λ)|X ≤ |μ − λ|H , which shows the first assertion. Step 2. We show that for every v ∈ H the functional λ ∈ H → ϕc (v, λ) is Fréchet ∂ differentiable with F -derivative ∂λ ϕc (v, λ) given by u(v, λ). Since (4.4.2) can be equivalently written as (4.4.3), it follows from the proof of Theorem 4.39 that λ ∈ H → ϕc (v, λ) is Fréchet differentiable and ∂ λ 1 ϕc (v, λ) = (λ + c u(v, λ)) − = u(v, λ). ∂λ c c Step 3. To argue differentiability of λ → dc (λ) note that it follows from (4.6.2) that for λ, μ ∈ H dc (μ) − dc (λ) = f (x(μ)) − f (x(λ)) + ϕc ( x(μ), λ) − ϕc ( x(λ), λ) +ϕc ( x(μ), μ) − ϕc ( x(μ), λ) ≥ f (x(λ)), x(μ) − x(λ) + ϕc ( x(λ), λ), (x(μ) − x(λ))
(4.6.3)
+ϕc ( x(μ), μ) − ϕc ( x(μ), λ) ≥ ϕc ( x(μ), μ) − ϕc ( x(μ), λ). Similarly, we have dc (μ) − dc (λ) = f (x(μ)) − f (x(λ)) + ϕc ( x(μ), μ) − ϕc ( x(λ), μ) +ϕc ( x(λ), μ) − ϕc ( x(λ), λ) ≤ − f (x(μ)), x(λ) − x(μ) + ϕc ( x(μ), μ), (x(λ) − x(μ))
(4.6.4)
+ϕc ( x(λ), μ) − ϕc ( x(λ), λ) ≤ ϕc ( x(λ), μ) − ϕc ( x(λ), λ). It follows from Step 2 and Theorem 4.39 that |ϕc ( x(μ), μ) − ϕc ( x(μ), λ) − (u( x(λ)λ), μ − λ)|
1
≤ (u( x(μ), λ + t (μ − λ)) − u( x(λ), λ), μ − λ) dt
0
1 ≤ |μ − λ| c ≤
1 |μ − λ| c
1
A 1 x(μ) + 1 (λ + t (μ − λ)) − A 1 x(λ) + λ + t (λ − μ) dt
c
c c c 0 0
1
(c |x(μ) − x(λ)| + 2t |μ − λ|) dt ≤
1 + |μ − λ|2 . σ c
i
i i
i
i
i
i
4.6. Augmented Lagrangian method
ItoKunisc 2008/6/12 page 117 i
117
Hence (4.6.3)–(4.6.4) and Step 2 imply that λ ∈ H → dc (λ) is Fréchet differentiable with F -derivative given by u( x(λ), λ), where u( x(λ), λ) is Lipschitz continuous in λ. The following theorem asserts convergence of the augmented Lagrangian method. Theorem 4.48. Assume that (A1)–(A3) hold and that there exists λ¯ ∈ ∂ϕ( x) ¯ such that (4.5.7) is satisfied. Then the sequence (xk , λk ) is well defined and satisfies 1 1 σ ¯ 2H ≤ ¯ 2H |xk − x| |λk+1 − λ| |λk − λ| ¯ 2X + 2 2c 2c
(4.6.5)
∞ σ 1 ¯ 2H , |xk − x| ¯ 2X ≤ |λ1 − λ| 2 2c k=1
(4.6.6)
and
¯ X → 0 as k → ∞. which implies that |xk − x| ¯ where v¯ = x. Proof. It follows from Theorem 4.45 that λ¯ = ϕc (v, ¯ λ), ¯ Next, we establish (4.6.5). From (4.5.5) and Step 3 f (xk ), x¯ − xk + λk+1 , (x¯ − xk ) ≥ 0 and from (4.5.7)
¯ (xk − x) ¯ xk − x
¯ + λ, ¯ ≥ 0. f (x),
Adding these two inequalities, we obtain ¯ (xk − x) f (xk ) − f (x), ¯ xk − x
¯ + λk+1 − λ, ¯ ≤ 0. From Theorems 4.39 and 4.40 λk+1 − λ¯ = A 1 c
vk +
λk c
− A1 c
v¯ +
(4.6.7)
λ¯ , c
¯ From the definition of Aμ we have where vk = xk and v¯ = x. ˆ v − v
ˆ = μ |Aμ v − Aμ v| ˆ 2 + Aμ v − Aμ v, ˆ Jμ v − Jμ v
ˆ ≥ μ |Aμ v − Aμ v| ˆ2 Aμ v − Aμ v, for μ > 0 and v, vˆ ∈ H , since Aμ v ∈ AJμ v and A is monotone. Thus, λ λ¯ ¯ vk − v) ¯ vk + k − v¯ + (λk+1 − λ, ¯ = λk+1 − λ, c c 1 1 1 ¯ λk − λ) ¯ ≥ |λk+1 − λ| ¯ 2 − (λk+1 − λ, ¯ λk − λ) ¯ − (λk+1 − λ, c c c ≥
1 1 ¯ 2− ¯ 2. |λk+1 − λ| |λk − λ| 2c 2c
Hence, (4.6.5) follows from (A2) and (4.6.7). Summing up (4.6.5) with respect to k, we obtain (4.6.6).
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 118 i
118 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization The duality (Uzawa) method is an iterative method for λk ∈ H . It is given by λk+1 = ϕc ( xk , λk ),
(4.6.8)
where xk ∈ C solves f (xk ) + ∗ λk , x − xk ≥ 0
for all x ∈ C.
(4.6.9)
It will be shown that the Uzawa method is conditionally convergent in the sense that there exists 0 < c < c¯ such that it converges for c ∈ [c, c]. ¯ On the other hand, the augmented Lagrangian method can be written as λk+1 = ϕc ( xk , λk ), where xk ∈ C satisfies f (xk ) + ∗ λk+1 , x − xk ≥ 0
for all x ∈ C.
Note that the Uzawa method is explicit with respect to λ while the augmented Lagrangian method is implicit and converges unconditionally (see Theorem 4.48) with respect to c. Theorem 4.49 (Uzawa Algorithm). Assume that (A1)–(A3) hold and that there exists λ¯ ∈ ∂ϕ( x) ¯ such that (4.5.7) is satisfied. Then there exists c¯ such that for the sequence (xk , λk ) generated by (4.6.8)–(4.6.9) we have |xk − x| ¯ X → 0 as k → ∞ if 0 < c ≤ c. ¯ Proof. As shown in the proof of Theorem 4.48 it follows from (4.5.7) and (4.6.9) that ¯ (xk − x)
f (xk ) − f (x), ¯ xk − x
¯ + λk − λ, ¯ ≤ 0. Since λk+1 − λ¯ = A 1
c
λk vk + c
(4.6.10)
−A
1 c
λ¯ v¯ + , c
where vk = xk , and since v¯ = x¯ and A 1 is Lipschitz continuous with Lipschitz conc stant c, 2 ¯ ≤ |λk − λ¯ + c (vk − v)| ¯ 2 |λk+1 − λ| ¯ 2 + 2c λk − λ, ¯ vk − v)
= |λk − λ| ¯ + c2 |vk − v| ¯2 ¯ 2 − (2cσ − c2 2 )|xk − x| ¯ 2, ≤ |λk − λ| where (4.6.10) was used for the last inequality. Choose c¯ < 2σ ( )−1 . Then β = ¯ and 2cσ − c2 2 > 0 if c ∈ (0, c] β
∞
¯ 2. |xk − x| ¯ 2 ≤ |λ0 − λ|
k=0
Consequently |xk − x| ¯ → 0 as k → ∞.
i
i i
i
i
i
i
4.7. Applications
ItoKunisc 2008/6/12 page 119 i
119
4.7 Applications In this section we discuss applications of the results in Sections 4.5 and 4.6. In many cases the conjugate functional ϕ ∗ is given by ϕ ∗ (v) = ψK ∗ (v), where K ∗ is a closed convex set in H and ψS is the indicator function of a set S, i.e., ⎧ if x ∈ S, ⎨ 0 ψS (x) = ⎩ ∞ if x ∈ S. Then it follows from Theorem 4.40 that for v, λ ∈ H 1 1 ϕc (v, λ) = sup − |y ∗ − (λ + c v)|2H + (|λ + c v|2H − |λ|2H ). 2c 2c y ∗ ∈K ∗
(4.7.1)
Hence the supremum is attained at λc (v, λ) = ProjK ∗ (λ + c v), where ProjK ∗ (φ) denotes the projection of φ ∈ H onto K ∗ . This implies that the update in Step 3 of the augmented Lagrangian algorithm is given by λk+1 = ProjK ∗ (λk + c xk ).
(4.7.2)
Example 4.50. For the equality constraint ϕ(v) = ψK (v) with K = {0} and ϕ ∗ (w) = 0 for w ∈ H . Thus ϕc (v, λ) =
1 (|λ + c v|2H − |λ|2H ) 2c
and λk+1 = λk + c xk , which coincides with the first order augmented Lagrangian update for equality constraints discussed in Chapter 3. Example 4.51. If ϕ(v) = ψK (v), where K is a closed convex cone with vertex at the origin in H , then ϕ ∗ = ψK + , where K + = {w ∈ H : w, v H ≤ 0 for all v ∈ K} is the dual cone of K. In particular, if K = {v ∈ L2 () : v ≤ 0 a.e.}, then K + = {w ∈ L2 () : w ≥ 0 a.e.}. Thus for the inequality constraint in H = L2 () the update (4.7.2) becomes λk+1 = max(0, λk + c xk ), where the max operation is defined pointwise in . Here L2 () denotes the space of scalarvalued square integrable functions over a domain in Rn . For K = {v ∈ Rn : vi ≤ 0 for all i} the update (4.7.2) is given by λk+1 = max(0, λk + c xk ), where the maximum is taken coordinatewise. It coincides with the first order augmented Lagrangian update for finite rank inequality constraints in Chapter 3.
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 120 i
120 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization Example 4.52. If ϕ(v) = |v|H , then ϕ ∗ = ψB , where B is the closed unit ball in H . Thus the update (4.7.2) becomes λk+1 =
λ k + c k xk . max(1, |λ + c xk |)
Example 4.53. We consider the bilateral inequality constraint φ ≤ v ≤ ψ in L2 (). We set ψ −φ φ+ψ ψ −φ 2 K = v ∈ L () : − ≤v− ≤ 2 2 2 and ϕ = ψK . It follows that
φ+ψ ϕ (w) = w, 2 ∗
L2
ψ −φ + , |w| 2
. L2
From Theorem 4.40 we conclude that the expression x, w − φ ∗ (w) − maximized with respect to w at the element y ∗ satisfying x−
1 |w 2c
− λ|2 is
φ+ψ y∗ − λ ψ − φ − ∈ ∂(| · |)(y ∗ ) 2 c 2
a.e. in . Thus it follows from Theorem 4.40 that the complementarity condition (4.1.8) is given by λ¯ = max(0, λ¯ + c ( x¯ − ψ)) + min(0, λ¯ + c ( x¯ − φ)), and the Lagrange multiplier update in Step 3 of the augmented Lagrangian method is λk+1 = max(0, λk + c ( xk − ψ)) + min(0, λk + c ( xk − φ)), where the max and min operations are defined pointwise a.e. in . Note that with obvious modifications, x in Sections 4.5 and 4.6 can be replaced by the affine function of the form x + a with a ∈ H .
4.7.1
Bingham flow
We consider the problem μ 2 ˜ min |∇u| − f u dx + g |∇u| dx 2
over u ∈ H01 (),
(4.7.3)
where is a bounded open set in R2 with Lipschitz boundary, g and μ are positive constants, and f˜ ∈ L2 (). For a discussion of (4.7.3) we refer the reader to [Glo, GLT] and the references therein. In the context of the general theory of Section 4.5 we choose X = H01 (), H = L2 () × L2 (),
and
= g grad,
i
i i
i
i
i
i
4.7. Applications
ItoKunisc 2008/6/12 page 121 i
121
K = X, and define f : X → R and ϕ : H → R by μ 2 ˜ f (u) = |∇u| − f u dx 2 and ϕ(v1 , v2 ) =
3
v12 + v22 dx.
Conditions (A1)–(A3) are clearly satisfied. Since dom(ϕ) = H it follows from Theorem 4.45 and Lemma 4.44 that there exists λ¯ such that (4.5.7) holds. Moreover, it is not difficult to show that ϕ ∗ (v) = ψK ∗ (v), where K ∗ is given by K ∗ = {v ∈ H : |v(x)|R2 ≤ 1 a.e. in }. Hence it follows from (4.7.2) that Steps 2 and 3 in the augmented Lagrangian method are given by −μ uk − g div λk+1 = f˜, where
λk+1 =
4.7.2
⎧ ⎪ λ + c ∇uk ⎪ ⎨ k ⎪ ⎪ ⎩
λk + c ∇uk |λk + c ∇uk |R2
(4.7.4)
on Ak = {x : |λk (x) + c ∇uk (x)|R2 ≤ 1}, (4.7.5) on \ Ak .
Image restoration
For the image restoration problem introduced in (4.1.2) the analysis is similar to the one for the Bingham flow and Steps 2 and 3 in the augmented Lagrangian method are given by −μ uk + K∗ (Kuk − z) + g div λk+1 = 0, where
λk+1 =
⎧ ⎪ λ + c ∇uk ⎪ ⎨ k ⎪ ⎪ ⎩
λk + c ∇uk |λk + c ∇uk |R2
μ
∂u + g λk+1 = 0 on , ∂n
on Ak = {x : |λk (x) + c ∇uk (x)|R2 ≤ 1}, on \ Ak
in the strong form. Here K : L2 () → L2 () denotes the convolution operator defined by (Ku)(x) = k(x, s)u(s) ds.
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 122 i
122 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization
4.7.3
Elastoplastic problem
We consider the problem min
β 2 ˜ |∇u| − f u dx 2
subject to
over u ∈ H01 () (4.7.6)
|∇u| ≤ 1 a.e. in ,
where β is a positive constant, is a bounded domain in R2 , and f˜ ∈ L2 (). In the context of the general theory we choose X = H01 (), H = L2 () × L2 (), K = X, and define f : X → R and ϕ : H → R by β f (u) = |∇u|2 − f˜ u dx 2
and
and
= ∇,
ϕ = ψCˆ ,
where Cˆ is the closed convex set defined by C = {v ∈ H : |v|R2 ≤ 1 a.e. in }. Then |w|R2 dx. ϕ ∗ (w) =
The maximum of (4.4.8), i.e., ∇u, w − φ ∗ (w) − L2 (), is attained at y ∗ such that
1 |w 2c
− λ| with respect to w ∈ L2 () ×
λ + c ∇u = y ∗ + cμ, where |μ| ≤ 1 and μ · y ∗ = |y ∗ | a.e. in . It thus follows from Theorem 4.40 that the Lagrange multiplier update is given by k λk + c ∇uk λ + c ∇uk |−1 λk+1 = c max 0, | c |λk + c ∇uk | a.e. in . The existence of Lagrange multiplier λ¯ ∈ L∞ () for the inequality constraint in (4.7.6) is shown in [Bre2] for f˜ = 1. In general, existence is still an open problem.
4.7.4
Obstacle problem
We consider the problem min
1 |∇u|2 − f˜ u dx 2
subject to
over u ∈ H01 () (4.7.7)
φ ≤ u ≤ ψ a.e. in .
In the context of the general theory we choose X = H01 (), H = L2 (),
and
= the natural injection,
i
i i
i
i
i
i
4.7. Applications
ItoKunisc 2008/6/12 page 123 i
123
C = X, and define f : X → R and ϕ : H → R by 1 2 ˜ f (u) = |∇u| − f u dx and 2
ϕ(v) = ψK ,
where K is the closed convex set defined by K = {v ∈ H : φ ≤ v ≤ ψ a.e. in }. For the one-sided constraint u ≤ ψ (i.e., φ = −∞) it is shown in [IK4], for example, that there exists a unique λ¯ ∈ ∂ϕ(u) ¯ such that (4.5.7) is satisfied provided that ψ ∈ H 1 (), ψ| ≥ 0, ˜ ¯ ∈ (H 2 ∩ H01 ) × L2 satisfies and sup(0, f + ψ) ∈ L2 (). In fact, the optimal pair (u, ¯ λ) − u¯ + λ¯ = f˜, λ¯ = max(0, λ¯ + c (u¯ − ψ)).
(4.7.8)
In this case Steps 2 and 3 in the augmented Lagrangian method is given by − uk + λk+1 = f˜, λk+1 = max(0, λk + c (uk − ψ)). For the two-sided constraint case we assume that φ, ψ ∈ H 1 () satisfy on , max(0, ψ + f˜), min(0, φ + f˜) ∈ L2 (),
(4.7.9)
S1 = {x ∈ : ψ + f˜ > 0} ∩ S2 = {x ∈ : φ + f˜ < 0} = ∅,
(4.7.10)
φ≤0≤ψ
and that there exists a c0 > 0 such that − (ψ − φ) + c0 (ψ − φ) ≥ 0
a.e. in .
(4.7.11)
Let λˆ ∈ H be defined by
⎧ ⎨ ψ(x) + f˜(x), x ∈ S1 , ˆ λ(x) = φ(x) + f˜(x), x ∈ S2 , ⎩ 0 otherwise.
(4.7.12)
Employing the regularization procedure of (4.5.4) we find the following theorem. Theorem 4.54. Assume that φ, ψ ∈ H 1 () satisfy (4.7.9)–(4.7.12) where λˆ is defined in (4.7.12). Then (4.5.5) is given by ⎧ ⎨ λˆ + c (uc − ψ) if λˆ + c (uc − ψ) > 0, ˜ − uc + λc = f , λc = (4.7.13) λˆ + c (uc − φ) if λˆ + c (uc − φ) < 0, ⎩ 0 otherwise, ˆ a.e. in for c ≥ c0 . Moreover, as c → ∞, uc u∗ weakly and φ ≤ uc ≤ ψ, |λc | ≤ |λ| 2 ¯ satisfies in H () and λc λ¯ weakly in L2 () where u¯ is the solution of (4.7.7) and (u, ¯ λ) the necessary and sufficient optimality condition ⎧ ⎨ λ¯ + c (u¯ − ψ) if λ¯ + c (u¯ − ψ) > 0, ˜ ¯ ¯ − u¯ + λ = f , λ = λ¯ + c (u¯ − φ) if λ¯ + c (u¯ − φ) < 0, ⎩ 0 otherwise.
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 124 i
124 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization Proof. From Theorem 4.40 it follows that ⎧ ⎨ λˆ + c (u − ψ) if λˆ + c (u − ψ) > 0, ˆ λc (u, λ) = λˆ + c (u − φ) if λˆ + c (u − φ) < 0, ⎩ 0 otherwise.
(4.7.14)
We note that (4.5.4) has a unique solution uc ∈ H01 (). From (4.5.5)–(4.5.6) we deduce ˆ ∈ L2 () and thus that uc satisfies (4.7.13). Since λˆ ∈ L2 () it follows that λc (uc , λ) 1 2 uc ∈ H (). Let η = sup(0, uc − ψ). Then η ∈ H0 (). Hence, we have (∇(uc − ψ), ∇η) + (−( ψ + f˜) + λc , η) = 0. ˆ For x ∈ assume that uc (x) ≥ ψ(x). If λ(x) > 0, then −( ψ + f˜)+λc = c (uc −ψ) ≥ 0. ˆ ˆ If λ(x) = 0, then −( ψ + f˜) + λc ≥ c (uc − ψ) ≥ 0. If λ(x) < 0, then −( ψ + f˜) + λc ≥ − (ψ − φ) + c (ψ − φ) ≥ 0 for c ≥ c0 . Thus, we have (−( ψ + f˜) + λc , η) ≥ 0 and |∇η|2 = 0. This implies that η = 0 and uc ≤ ψ a.e. in . Similarly, we can prove that uc ≥ φ a.e. in by choosing the test function η = inf (0, uc − φ) ∈ H01 (). Moreover, it ˆ ≤ |λ| ˆ a.e. in and |λc |L2 is uniformly bounded. follows from (4.7.14) that |λc | = |λc (uc , λ)| ¯ in H 2 () × L2 (). Thus, there exists a weakly convergent subsequence (ucˆ , λcˆ ) (u, ¯ λ) 1 Moreover the subsequence ucˆ converges strongly to u¯ in H0 () and, as shown in the proof of Theorem 4.46, − u¯ + λ¯ = f˜ and λ¯ ∈ ∂ϕ(u). ¯ (4.7.15) Hence, it follows from Theorem 4.43 that u¯ minimizes (4.7.7). Since the solution to (4.7.7) is unique it follows from (4.7.15) that λ¯ ∈ L2 () is unique. The theorem now follows from Theorem 4.45. From Theorem 4.54 it follows that Steps 2 and 3 in the augmented Lagrangian method for the two-sided constraint are given by ⎧ ⎨ λk + c (uk − ψ) if λk + c (uk − ψ) > 0, λk + c (uk − φ) if λk + c (uk − φ) < 0, − uk + λk+1 = f˜, λk+1 = ⎩ 0 otherwise. Note that the last equality can be equivalently expressed as λk+1 = max(0, λk + c (uk − ψ)) + min(0, λk + c (uk − φ)).
4.7.5
Signorini problem
Consider the Signorini problem 1 min (|∇u|2 + |u|2 ) − f˜ u dx 2 subject to u ≥ 0
over u ∈ H 1 () (4.7.16)
on the boundary ,
i
i i
i
i
i
i
4.7. Applications
ItoKunisc 2008/6/12 page 125 i
125
which is a simplified version of a contact problem arising in elasticity theory. In this case we choose X = H 1 (), H = L2 ( ),
and
= the trace operator on boundary ,
C = X, and define f : X → R and ϕ : H → R by 1 f (u) = (|∇u|2 + |u|2 ) − f˜ u dx 2
and
ϕ(v) = ψK ,
where K is the closed convex set defined by K = {v ∈ L2 () : v ≥ 0 a.e. in }. If f˜ ∈ L2 (), then the unique minimizer u¯ ∈ H 2 () [Bre2] and it is shown in [Glo] that ∂ ¯ satisfies (4.5.7). Note that range( ) is dense in H and is for λ¯ = ∂n u, ¯ the pair (u, ¯ λ) compact. It thus follows from Theorems 4.43 and 4.46 that (4.5.7) has a unique solution and that uc → u¯ strongly in X and λc → λ¯ weakly in H . Steps 2 and 3 in the augmented Lagrangian method are given by − uk + uk = f˜,
∂ uk − λk+1 = 0 ∂n
λk+1 = max(0, λk − c uk )
4.7.6
on ,
on .
Friction problem
Consider a simplified friction problem from elasticity theory 1 min (|∇u|2 + |u|2 ) − f˜ u dx + g |u| ds over u ∈ H 1 (), 2
(4.7.17)
where is a bounded open domain of R2 with sufficiently smooth boundary . In this case we choose X = H 1 (), H = L2 ( ),
and
= the trace operator on boundary ,
C = X, and define f : X → R and ϕ : H → R by 1 f (u) = (|∇u|2 + |u|2 ) − f˜ u dx and 2
ϕ(v) = g
|v| ds.
Note that ϕ = ψK ∗ , where K ∗ = {v ∈ H : |v(x)| ≤ 1 a.e. in }. Since dom(ϕ) = H it follows from Theorem 4.43 and Lemma 4.44 that there exists λ¯ such that (4.5.7) holds. From (4.7.2) Steps 2 and 3 in the augmented Lagrangian method are given by − uk + uk = f˜,
λk+1
∂ uk + λk+1 = 0 ∂n
⎧ ⎨ λk + c u k λk + c uk = ⎩ |λk + c uk |
on ,
on k = {x ∈ : |λk + c uk | ≤ 1}, on \ k .
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 126 i
126 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization
L1 -fitting
4.7.7
Consider the minimization problem μ min |u − z| dx + |∇u|2 ds 2
over u ∈ H01 ()
(4.7.18)
for interpolation of noisy data z ∈ L2 () by minimizing the L1 -norm of the error u − z over u ∈ H01 (). Again μ > 0 is fixed but should be adjusted to the statistics of noise. The analysis of this problem is very similar to that of a friction problem and Steps 2 and 3 in the augmented Lagrangian method are given by −μ uk + λk+1 = 0,
λk+1
4.7.8
⎧ ⎨ λk + c (uk − z) λk + c (uk − z) = ⎩ |λk + c (uk − z)|
on k = {x : |λk (x) + c (uk (x) − z)| ≤ 1}, on \ k .
Control problem
Consider the optimal control problem min
1 2
T
|x(t)|2 + |u(t)|2 dt
over (x, u) ∈ L2 (0, T ; Rn ) × L2 (0, T ; Rm )
0
(4.7.19)
t
subject to x(t) − x0 −
Ax(s) + Bu(s) ds = 0
and
|u(t)| ≤ 1 a.e.,
0
where A ∈ Rn×n , B ∈ Rn×m , and x0 ∈ Rn are fixed. In this case we formulate the problem as in the form (4.1.1) by choosing X = L2 (0, T ; Rn )×L2 (0, T ; Rm ), H = L2 (0, T ; Rm ),
and
(x, u) = u for (x, u) ∈ X,
and define f : X → R and ϕ : H → by f (x, u) =
1 2
T
(|x(t)|2 + |u(t)|2 ) dt
and
ϕ(v) = ψK (v),
0
where K = {v ∈ H : |v(t)| ≤ 1 a.e. in (0, T )}. C is the closed affine space defined by t C = (x, u) ∈ X : x(t) − x0 − Ax(s) + Bu(s) ds = 0 . 0
It follows from Theorems 4.39 and 4.40 that
v + λc λ λ+cv λ
λc = c v + − = c max 0, v + − 1 λ c c |λ + c v| max(1, |v + c |)
(4.7.20)
i
i i
i
i
i
i
4.7. Applications
ItoKunisc 2008/6/12 page 127 i
127
for v, λ ∈ H . For c > 0 the regularized problem fc (x, u) = f (x, u) + ϕc (u, 0)
over (x, u) ∈ C
has a unique solution (xc , uc ) ∈ C. Define the Lagrangian T t x(t) − x0 − L(x, u, μ) = fc (x, u) + Ax(s) + Bu(s) ds, μ(t) 0
0
Since the mapping F : X → L2 (0, T ; Rn ) defined by t Ax(s) + Bu(s) ds, F (x, u) = x(t) −
dt. Rn
t ∈ (0, T ),
0
is surjective it follows from the Lagrange multiplier theory that there exists a unique Lagrange multiplier μc ∈ L2 (0, T ; Rn ) such that the Fréchet derivative of L with respect to (x, u) satisfies L (xc , uc , μc )(h, v) = 0 for all (h, v) ∈ X. Hence we obtain 2 t T 1 h(t) − Ah(s) ds, μc (t) + (x(t), h(t)) dt = 0 0
0
for all h ∈ L2 (0, T ; Rn ) and t 2 T 1 uc u(t) + c max(0, |uc | − 1) , v(t) − Bv(s) ds, μc (t) dt = 0 |uc | 0 0 for all v ∈ L2 (0, T ; Rm ). It is not difficult to show that if we define pc (t) = − then (xc , uc , pc ) satisfies d pc (t) + AT pc (t) + xc (t) = 0, dt
T t
μc (s) ds,
pc (T ) = 0, (4.7.21)
uc (t) uc (t) + c max(0, |uc (t)| − 1) = −B T pc (t). |uc (t)| Let (x, ¯ u) ¯ be a feasible point of (4.7.19); i.e., (x, ˜ u) ˜ ∈ C and u˜ ∈ K. Since fc (xc , uc ) ≤ fc (x, ˜ u) ˜ = f (x, ˜ u) ˜ and ϕc (uc , 0) ≥ 0 it follows that |(xc , uc )|X is uniformly bounded in c > 0. Thus, there exists a subsequence of (xc , uc ) that converges weakly to (x, ¯ u) ¯ ∈ X as c → ∞. We shall show that (x, ¯ u) ¯ is the solution of (4.7.19). First, since F (xc , uc ) = x0 , xc converges strongly to x¯ in L2 (0, T ; Rn ) and weakly in H 1 (0, T ; Rn ) and (x, ¯ u) ¯ ∈ C. From the first equation of (4.7.21) it follows that pc converges strongly to p¯ in H 1 (0, T ; Rn ), where p¯ satisfies d p(t) ¯ + AT p(t) ¯ + x(t) ¯ = 0, dt
p(T ¯ ) = 0.
Next, from the second equation of (4.7.21) we have |uc (t)| + c max(0, |uc (t)| − 1) = |B T pc (t)|
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 128 i
128 Chapter 4. Augmented Lagrangian Methods for Nonsmooth, Convex Optimization and thus
|uc (t)| =
⎧ |B T pc (t)| ⎪ ⎪ ⎨
if |B T pc (t)| ≤ 1,
|B T pc (t)| − 1 ⎪ ⎪ ⎩ 1+ c+1
if |B pc (t)| ≥ 1,
t ∈ [0, T ].
(4.7.22)
T
Note that uc (t) =
−B T pc (t) , 1 + ηc (t)|uc (t)|−1
(4.7.23)
c where ηc (t) = c max(0, |uc (t)| − 1) = max(0, |B T pc (t)| − 1). 1+c Define the functions η, ¯ uˆ ∈ H 1 (0, T ) by η(t) ¯ = max(0, |B T p(t)| ¯ − 1)
and
for t ∈ [0, T ]. Then |u(t)| ˆ =
u(t) ˆ =
¯ ¯ −B T p(t) −B T p(t) = 1 + η(t) ¯ max(1, |B T p(t)|) ¯
(4.7.24)
¯ |B T p(t)| . max(1, |B T p(t)|) ¯
ˆ in C(0, T ) by (4.7.22) and it follows Since B T pc → B T p¯ in C(0, T ) we have |uc | → |u| from (4.7.23)–(4.7.24) that ηc → η¯ and uc → uˆ in C(0, T ). Since uc → u¯ weakly in L2 (0, T ; Rm ) we have u¯ = u. ˆ Therefore, we conclude that uc → u¯ in C(0, T ) and if ¯ ¯ satisfies (x, λ(t) = η(t) ¯ u(t), ¯ then (x, ¯ u, ¯ p, ¯ λ) ¯ u) ¯ ∈ C, u¯ ∈ C and d p(t) ¯ + AT p(t) ¯ + x¯ = 0, dt ¯ u(t) ¯ + λ¯ = −B T p(t),
p(T ¯ ) = 0, (4.7.25)
¯ ¯ ¯ λ(t) = λc (u(t), ¯ λ(t)) = ϕc (u, ¯ λ),
which is equivalent to (4.5.7). Now, from Theorem 4.43 we deduce that (x, ¯ u) ¯ is the solution to (4.7.19). The last equality in (4.7.25) can be verified by separately considering the cases |u(t)| ¯ < 1 and |u(t)| ¯ = 1. It follows from (4.7.20) that Steps 2 and 3 in the augmented Lagrangian method are given by d xk (t) = Axk (t) + Buk (t), dt
xk (0) = x0 ,
d pk (t) + AT pk (t) + xk = 0, dt
pk (T ) = 0,
uk (t) + λk+1 (t) = −B T pk (t),
λ k + c uk λk where λk+1 = c max 0,
uk +
− 1 c |λk + c uk |
for t ∈ [0, T ].
i
i i
i
i
i
i
ItoKunisc 2008/6/12 page 129 i
Chapter 5
Newton and SQP Methods
In this chapter we discuss the Newton method for the equality-constrained optimal control problem min J (y, u) subject to e(y, u) = 0, (P ) where J : Y × U → R and e : Y × U → W , and Y, U , and W are Hilbert spaces. The focus is on problems where for given u there exists a solution y(u) to e(y, u) = 0, which is typical for optimal control problems. We shall refer to Jˆ(u) = J (y(u), u)
(5.0.1)
as the reduced cost functional. In the first section we give necessary and sufficient optimality conditions based on Taylor series expansion arguments. Section 5.2 is devoted to Newton’s method to solve (P ) and sufficient conditions for quadratic convergence are given. Section 5.3 contains a discussion of SQP (sequential quadratic programming) and reduced SQP techniques. We do not provide a convergence analysis here, since this can be obtained as a special case from the results on second order augmented Lagrangians in Chapter 6. The results are specialized to a class of optimal control problems for the Navier–Stokes equation in Section 5.4. Section 5.5 is devoted to the Newton method for weakly singular problems as introduced in Chapter 1.
5.1
Preliminaries
(I) Necessary Optimality Condition The first objective is to characterize the derivative of Jˆ without recourse to the derivative of the state y with respect to the control u. This is in the same spirit as Theorem 1.17 but obtained under the less restrictive assumption of this chapter. We assume that (C1) (y, u) ∈ Y × U satisfies e(y, u) = 0 and there exists a neighborhood V (y) × V (u) of (y, u) on which J and e are C 1 with Lipschitz continuous first derivatives such that for every v ∈ V (u) there exists a unique y(v) ∈ V (y) satisfying e(y(v), v) = 0. 129
i
i i
i
i
i
i
130
ItoKunisc 2008/6/12 page 130 i
Chapter 5. Newton and SQP Methods
Theorem 5.1. Assume that the pair (y, u) satisfies (C1) and that there exists λ ∈ W ∗ such that (C2) ey (y, u)∗ λ + Jy (y, u) = 0, and (C3) limt→0+ 1t |y(u + td) − y|2Y = 0 for all d ∈ U . Then the Gâteaux derivative Jˆ (u) exists and Jˆ (u) = eu (y, u)∗ λ + Ju (y, u).
(5.1.1)
Proof. For d ∈ U and t sufficiently small let v = y(u + td) − y. Then 1 ˆ ˆ J (u+td)− J (u) = J (y +sv, u+std)(v, td)−J (y, u)(v, td) ds +J (y, u)(v, td). 0
(5.1.2)
Similarly, 0 = λ, e(v, u + td) − e(y, u) W ∗ ,W
1
= λ,
e (y + sv, u + std)(v, td) − e (y, u)(v, td) ds + λ, e (y, u)(v, td) ,
0
and hence by Lipschitz continuity of e in V (y) × V (u) and (C3) we have
(5.1.3)
1 lim+ λ, ey (y, u)v = − λ, eu (y, u)d . t→0 t Since J is Lipschitz continuous in V (y) × V (u), it follows from (5.1.2) and conditions (C2)–(C3) that lim+
t→0
1 Jˆ(u + td) − Jˆ(u) = − lim+ λ, ey (y, u)v + Ju (y, u)d t→0 t t
= (eu (y, u)∗ λ + Ju (y, u))d. Defining the Lagrangian L : Y × U × W ∗ → R by L(y, u, λ) = J (y, u) + λ, e(y, u) , (5.1.1) can be written as Jˆ (u) = Lu (y, u, λ). If ey (y, u) : Y → W is bijective, then (5.1.1) also follows from the implicit function theory. In fact, by the implicit function theorem there exists a neighborhood V (y) × V (u) of (y, u) such that e(y, v) = 0 has a unique solution y(v) ∈ V (y) for v ∈ V (u) and y(·) : V (u) ⊂ U → Y is C 1 with ey (y, v)y (v) + eu (y, v) = 0,
v ∈ V (u).
i
i i
i
i
i
i
5.1. Preliminaries
ItoKunisc 2008/6/12 page 131 i
131
Thus Jˆ (u)v = Jy (y, u)y (u)v + Ju (y, u)v = eu (y, u)∗ λ, v + Ju (y, u)v. If conditions (C1)–(C3) hold at a local minimizer (y ∗ , u∗ ) of (P ), it follows from Theorem 5.1 that the first order necessary optimality condition for (P ) is given by ⎧ ey (y ∗ , u∗ )∗ λ∗ + Jy (y ∗ , u∗ ) = 0, ⎪ ⎪ ⎨ eu (y ∗ , u∗ )∗ λ∗ + Ju (y ∗ , u∗ ) = 0, (5.1.4) ⎪ ⎪ ⎩ e(y ∗ , u∗ ) = 0, or equivalently L (y ∗ , u∗ , λ∗ ) = 0,
e(y ∗ , u∗ ) = 0,
where, as in the previous chapter, prime with L denotes the derivative with respect to (y, u).
(II) Sufficient Optimality Condition Assume that (y ∗ , u∗ , λ∗ ) ∈ Y1 × U × W ∗ satisfies (C1) and that the necessary optimality condition (5.1.4) holds. For (y, u) ∈ V (y ∗ ) × V (u∗ ) define the quadratic approximation to J (y, u) and λ, e(y, u) by E1 (y −y ∗ , u−u∗ ) = J (y, u)−J (y ∗ , u∗ )−Ju (y ∗ , u∗ )(u−u∗ )−Jy (y ∗ , u∗ )(y −y ∗ ) (5.1.5) and E2 (y − y ∗ , u − u∗ ) = λ∗ , e(y, u) − e(y ∗ , u∗ ) − ey (y ∗ , u∗ )(y − y ∗ ) − eu (y ∗ , u∗ )(u − u∗ ) . (5.1.6) Since for y = y(u) with u ∈ V (u∗ ) E2 (y(u) − y ∗ , u − u∗ ) = − λ∗ , ey (y ∗ , u∗ )(y − y ∗ ) + eu (y ∗ , u∗ )(u − u∗ ) , we find by summing (5.1.5) and (5.1.6) with y = y(u) and using (5.1.4) Jˆ(u) − Jˆ(u∗ ) = J (y(u), u) − J (y ∗ , u∗ ) = E1 (y(u) − y ∗ , u − u∗ ) + E2 (y(u) − y ∗ , u − u∗ ). Based on this identity, we have the following local sufficient optimality conditions. • If E(u) := E1 (y(u) − y ∗ , u − y ∗ ) + E2 (y(u) − y ∗ , u − u∗ ) ≥ 0 for all u ∈ V (u∗ ), then u∗ is a local minimizer of (P ). If E(u) > 0 for all u ∈ V (u∗ ) with u = u∗ , then u∗ is a strict local minimizer. • Assume that J is locally uniformly convex in the sense that for some α > 0 E1 (y(u) − y ∗ , u − u∗ ) ≥ α(|y(u) − y ∗ |2Y + |u − u∗ |2U ) for all u ∈ V (u∗ ), and let β ≥ 0 be the size of the nonlinearity of e, i.e., |e(y, u) − e(y ∗ , u∗ ) − ey (y ∗ , u∗ )(y − y ∗ ) − eu (y ∗ , u∗ )(u − u∗ )| ≤ β(|y − y ∗ |2Y + |u − u∗ |2U )
i
i i
i
i
i
i
132
ItoKunisc 2008/6/12 page 132 i
Chapter 5. Newton and SQP Methods
for all (y, u) ∈ V (y ∗ ) × V (u∗ ). Then E(u) ≥ (α − β|λ∗ |W ∗ ) (|y(u) − y ∗ |2Y + |u − u∗ |2U ) for all u ∈ V (u∗ ). Thus, u∗ is a strict local minimizer of (P ) if α − β|λ∗ |W ∗ > 0. • Note that E(u) = L(x, λ∗ ) − L(x ∗ , λ∗ ) − L (x ∗ , λ∗ )(x − x ∗ ), where x = (y, u), x ∗ = (y ∗ , u∗ ), and y ∗ = y(u∗ ). If L is C 2 with respect to x = (y, u) in V (y ∗ ) × V (u∗ ) with second derivative with respect to x denoted by L , we have E(u) = L (x ∗ , λ∗ )(x − x ∗ , x − x ∗ ) + r(x − x ∗ ),
(5.1.7)
where |r(x − x ∗ )| = o(|x − x ∗ |2 ). Thus, if there exists σ > 0 such that L (x ∗ , λ∗ )(δx, δx) ≥ σ |δx|2 for all δx ∈ ker e (x ∗ ),
(5.1.8)
and e (x ∗ ) : Y × U → W is surjective, then there exist σ0 > 0 and ρ > 0 such that Jˆ(u) − Jˆ(u∗ ) = E(u) ≥ σ0 (|y(u) − y(u∗ ))|2 + |u − u∗ |2 )
(5.1.9)
for all y(u) ∈ V (y ∗ )×V (u∗ ) with |(y(u), u)−(y(u∗ ), u∗ )| < ρ. In fact, from Lemma 2.13, Chapter 2, with S = ker e (x ∗ ) there exist γ > 0, σ0 > 0 such that L (x ∗ , λ∗ )(x − x ∗ , x − x ∗ ) ≥ σ0 |x − x ∗ |2 for all x ∈ V (y ∗ ) × V (u∗ ) with |xker⊥ | ≤ γ |xker |,
(5.1.10)
where we decomposed x − x ∗ as x − x ∗ = xker + xker⊥ ∈ ker e (x ∗ ) + (ker e (x ∗ ))⊥ . Since 0 = e(x) − e(x ∗ ) = e (x ∗ )(x − x ∗ ) + η(x − x ∗ ), where |η(x − x ∗ )|W = o(|x − x ∗ |), γ ) there exists ρ > 0 such that it follows that for δ ∈ (0, 1+γ |xker⊥ | ≤ δ|x − x ∗ |
if |x − x ∗ | < ρ.
Consequently |xker⊥ | ≤
δ |xker | < γ |xker | 1−δ
if
|x − x ∗ | < ρ,
and hence (5.1.9) follows from (5.1.7) and (5.1.10). We refer the reader to [CaTr, RoTr] and the literature cited therein for detailed investigations on the topic of second order sufficient optimality. The aim of these results is to establish second order sufficient optimality conditions which are close to second order necessary conditions.
i
i i
i
i
i
i
5.2. Newton method
5.2
ItoKunisc 2008/6/12 page 133 i
133
Newton method
In this section we describe the Newton method applied to the reduced form (5.0.1) of (P ). That is, let y(u) denote a solution to e(y, u) = 0. Then the constrained optimization problem is transformed to the unconstrained problem for u in U : min Jˆ(u) = J (y(u), u).
(5.2.1)
u∈U
Let (y ∗ , u∗ ) denote a solution to (P ), and assume that (C1) holds for (y ∗ , u∗ ) and that (C2), (C3) hold for all (y(u), u) ∈ V (y ∗ ) × V (u∗ ). In addition it is assumed that J and e are C 2 in V (y ∗ ) × V (u∗ ) with Lipschitz continuous second derivatives. From Theorem 5.1 the first derivative of Jˆ(u) is given by Jˆ (u) = eu (y, u)∗ λ + Ju (y, u),
(5.2.2)
where u ∈ V (u∗ ), y = y(u), and λ = λ(u) satisfy ey (y, u)∗ λ = −Jy (y, u).
(5.2.3)
From (5.
