Iterative Methods for Fixed Point Problems in Hilbert Spaces

Lecture Notes in Mathematics Editors: J.-M. Morel, Cachan B. Teissier, Paris

For further volumes: http://www.springer.com/series/304

2057

•

Andrzej Cegielski

Iterative Methods for Fixed Point Problems in Hilbert Spaces

123

Andrzej Cegielski University of Zielona G´ora Faculty of Mathematics, Computer Science and Econometrics Zielona G´ora, Poland

ISBN 978-3-642-30900-7 ISBN 978-3-642-30901-4 (eBook) DOI 10.1007/978-3-642-30901-4 Springer Heidelberg New York Dordrecht London Lecture Notes in Mathematics ISSN print edition: 0075-8434 ISSN electronic edition: 1617-9692 Library of Congress Control Number: 2012945521 Mathematics Subject Classification (2010): 47-02, 49-02, 65-02, 90-02, 47H09, 47J25, 37C25, 65F10, 65K15, 90C25 c Springer-Verlag Berlin Heidelberg 2012  This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher’s location, in its current version, and permission for use must always be obtained from Springer. Permissions for use may be obtained through RightsLink at the Copyright Clearance Center. Violations are liable to prosecution under the respective Copyright Law. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)

To my family: El˙zbieta, Joanna, Gustaw, Szymon and Katarzyna

•

Motto: If you want to find the source, you have to go up, against the current [John Paul II]

•

Preface

In this monograph we deal with iteration methods for finding fixed points (if they exist) of nonexpansive operators defined on a Hilbert space, i.e., operators T having the property d.T x; T y/  cd.x; y/ for all x; y 2 X (1) and for some constant c 2 Œ0; 1. The origin of these methods dates back to 1920, when Stefan Banach (1892–1945) formulated his famous contraction mapping principle. Banach proved that if T W X ! X is a contraction (an operator satisfying (1) with c < 1) defined on a complete metric space, then T has a unique fixed point x  , i.e., a point for which T x  D x  . Furthermore, for any x 2 X a sequence  fT k xg1 kD0 converges geometrically to x . Many practical problems can be reduced to finding a fixed point of a nonexpansive operator or a common fixed point of a family of nonexpansive operators. A simple example is a system of linear equations Ax D b. Any solution of this system can be identified with a common fixed point of (nonexpansive) operators of orthogonal projections onto hyperplanes corresponding to particular equations of the system. Under some additional conditions a nonexpansive operator has a fixed point (see, e.g., the Browder–G¨ohde–Kirk fixed point theorem), but the Banach theorem does not guarantee the convergence of the sequence fT k xg1 kD0 . Therefore, it is of great interest to develop methods for finding fixed points of nonexpansive operators. The first iterative method for solving a linear system was proposed in 1937 by a Polish mathematician, Stefan Kaczmarz (1890–1945), in a very short paper (3 pages) “Angen¨aherte Aufl¨osung von Systemen linearer Gleichungen” (“Approximate solution of systems of linear equations”) published in Bulletin International de l’Acad´emie Polonaise des Sciences et des Lettres. A year later, an Italian mathematician, Gianfranco Cimmino (1908–1989), proposed another iterative method for a linear system. He published his result in a paper “Calcolo approssimato per le soluzioni dei sistemi di equazioni lineari” (“Approximate computation of the solutions of linear systems”) in a La Ricerca Scientifica. As opposed to earlier methods for solving systems of linear equations, both methods are motivated rather by geometrical operations than algebraic ones. The main operation in both methods ix

x

Preface

is a cyclic (in the Kaczmarz method) or simultaneous (in the Cimmino method) orthogonal projection onto hyperplanes described by particular equations of the linear system. The results of Kaczmarz and Cimmino were disregarded for several decades except few references. A great interest in these results started in the 1970s, when it suddenly turned out that the Kaczmarz and Cimmmino methods can be efficiently applied in computed tomography (CT), because the mathematical model of CT can be reduced to a solution of large systems of linear equations with sparse and unstructured matrices, and the methods behave well for such systems. At the end of the twentieth century the interest in the Kaczmarz and Cimmino results was still increasing. Both results have become fundamental to modern iterative methods for fixed point problems for nonexpansive operators. The third result which influenced the development of this area concerns alternative projections onto two subspaces of a Hilbert space. The result was published in 1950 by John von Neumann in a paper “Functional Operators—Vol. II. The Geometry of Orthogonal Spaces” in Annals of Mathematical Studies. Von Neumann proved that a sequence generated by his method converges to the projection of the starting point onto the intersection of the subspaces. The results of Kaczmarz, Cimmino, and von Neumann have been generalized several times in the last decades. Today it is known that the convergence holds for an essentially wider class of operators than orthogonal projections onto hyperplanes, e.g., for nonexpansive operators or for quasi-nonexpansive operators, satisfying some additional conditions. All three methods belong today to classical iterative methods for finding fixed points of nonexpansive operators defined on a Hilbert space. These methods served as the basis for several methods, e.g., for: the Landweber method, projected Landweber method, Douglas–Rachford method, sequential projection methods, methods of cyclic and simultaneous subgradient projections, Dos Santos method of extrapolated simultaneous subgradient projections, reflection-projection method, surrogate projection method, and many others. Irrespective of their theoretical value, they have found application in many areas of mathematics, physics, and technology. The most spectacular application of the methods is an intensity-modulated radiation therapy (IMRT). Iterative methods for finding fixed points of nonexpansive operators in Hilbert spaces have been described in many publications. In this monograph we try to present the methods in a consolidated way. We introduce some classes of operators, give their properties, define iterative methods generated by operators from these classes, and present general convergence theorems. On this basis we present the conditions under which particular methods converge. A large part of the results presented in this monograph can be found in various forms in the literature. We tried, however, to show that the convergence of a big class of iteration methods follows from general properties of some classes of operators and from some general convergence theorems, in particular from Opial’s theorem or from its modifications. This theorem was presented in 1967 by a Polish mathematician, Zdzisław Opial (1930–1974), in a paper “Weak convergence of the sequence of successive approximations for nonexpansive mappings” published in the Bulletin of the American Mathematical Society.

Preface

xi

In this monograph we work with operators defined on a real Hilbert space, although a part of the results presented herein holds for wider classes of spaces. The monograph is divided into five chapters. In Chap. 1, we recall basic definitions and facts from linear algebra, functional analysis, and convex analysis which we apply in the further part of the monograph. In Chap. 2, we introduce some classes of algorithmic operators (i.e., operators which generate some algorithms or iterative methods): nonexpansive operators, quasi-nonexpansive operators, relaxed quasinonexpansive operators, cutter operators, firmly nonexpansive operators, metric projection, relaxed firmly nonexpansive operators, averaged operators, strongly nonexpansive operators, and generalized relaxations of algorithmic operators. Then we present the properties of these classes, in particular the relationships among these classes and their closedness with respect to some algebraic operations on the operators from these classes. In Chap. 3, we analyze the convergence properties of the sequences generated by the operators introduced in Chap. 2. Opial’s theorem, its generalizations, and modifications for sequences generated by a subclass of the class of quasi-nonexpansive operators play the key role here. In Chap. 4, we apply the properties of classes of operators presented in Chap. 2 to constructions of operators used in many iterative methods for fixed point problems. In this chapter we also give the properties of the following: the alternating projection, simultaneous projection, cyclic projection, Landweber operator, projected Landweber operator, and some generalizations and extrapolations of these operators. In Chap. 5, we apply the results presented in the previous chapters in order to show the convergence of sequences generated by many iterative methods for fixed point problems, some of which are known in the literature, but several are new. The notions and facts presented in the book are illustrated with 61 figures. Each chapter is followed by several exercises. Many persons have looked through successive versions of the monograph. I am deeply grateful for their helpful remarks. In particular, I would like to express my thanks to Prof. Simeon Reich from the Technion (Israel), Prof. Yair Censor from the University of Haifa (Israel), Prof. Diethard Pallaschke from the University of Karlsruhe (Germany), and Prof. Heinz Bauschke from the University of British Columbia in Okanagan (Canada). Their valuable remarks have contributed to substantial improvements in successive versions of the monograph and have consolidated me in my aim to give the monograph its final shape. I am also very grateful to my colleagues from the University of Zielona G´ora, who looked through the final version of the monograph and also made some useful remarks: Prof. Michał Kisielewicz, Prof. Krzysztof Przesławski, and Prof. Jerzy Motyl. I would like to express my thanks to my Ph.D. student, Rafał Zalas, for his help in the preparation of figures which illustrate the notions and facts presented herein and to Danuta Michalak for the technical composition of the monograph. Last but not least, I would like to express my deep gratitude to my wife, El˙zbieta, for her understanding and assistance during the preparation of the monograph. Zielona G´ora, Poland April 2012

Andrzej Cegielski

•

Contents

1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 1.1.1 Hilbert Space .. . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 1.1.2 Notations and Basic Facts . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 1.2 Metric Projection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 1.2.1 Existence and Uniqueness of the Metric Projection .. . . . . . . 1.2.2 Characterization of the Metric Projection.. . . . . . . . . . . . . . . . . . 1.2.3 First Applications of the Characterization Theorem . . . . . . . 1.3 Convex Optimization Problems .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 1.3.1 Convex Minimization Problems .. . . . . . . .. . . . . . . . . . . . . . . . . . . . 1.3.2 Variational Inequality . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 1.3.3 Common Fixed Point Problem . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 1.3.4 Convex Feasibility Problem . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 1.3.5 Linear Feasibility Problem . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 1.3.6 General Convex Feasibility Problem .. . .. . . . . . . . . . . . . . . . . . . . 1.3.7 Split Feasibility Problem . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 1.3.8 Linear Split Feasibility Problem . . . . . . . .. . . . . . . . . . . . . . . . . . . . 1.3.9 Multiple-Sets Split Feasibility Problem.. . . . . . . . . . . . . . . . . . . . 1.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .

1 1 1 5 16 17 19 20 23 24 27 27 27 30 33 34 35 35 36

2 Algorithmic Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 2.1 Basic Definitions and Properties . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 2.1.1 Nonexpansive Operators .. . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 2.1.2 Quasi-nonexpansive Operators . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 2.1.3 Cutters and Strongly Quasi-nonexpansive Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 2.2 Firmly Nonexpansive Operators . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 2.2.1 Basic Properties of Firmly Nonexpansive Operators .. . . . . . 2.2.2 Relationships Between Firmly Nonexpansive and Nonexpansive Operators . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 2.2.3 Further Properties of the Metric Projection .. . . . . . . . . . . . . . . .

39 40 41 45 53 65 66 70 76 xiii

xiv

Contents

2.3 2.4 2.5

2.2.4 Metric Projection onto a Closed Subspace.. . . . . . . . . . . . . . . . . 80 2.2.5 Metric Projection onto a Closed Affine Subspace .. . . . . . . . . 82 2.2.6 Properties of Relaxed Firmly Nonexpansive Operators .. . . 84 2.2.7 Fixed Points of Firmly Nonexpansive Operators .. . . . . . . . . . 90 Strongly Nonexpansive Operators . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 91 Generalized Relaxations of Algorithmic Operators .. . . . . . . . . . . . . . . . . 96 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 102

3 Convergence of Iterative Methods . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 3.1 Iterative Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 3.2 Properties of the Weak Convergence . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 3.3 Properties of Fej´er Monotone Sequences .. . . . . . . .. . . . . . . . . . . . . . . . . . . . 3.4 Asymptotically Regular Operators .. . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 3.5 Opial’s Theorem and Its Consequences . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 3.6 Generalization of Opial’s Theorem . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 3.7 Opial-Type Theorems for Cutters . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 3.8 Strong Convergence of Fej´er Monotone Sequences . . . . . . . . . . . . . . . . . 3.9 Relationships Among Algorithmic Operators . . .. . . . . . . . . . . . . . . . . . . . 3.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .

105 105 106 108 111 114 116 118 123 126 127

4 Algorithmic Projection Operators . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.1 Examples of Metric Projections.. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.1.1 Metric Projection onto a Hyperplane . . .. . . . . . . . . . . . . . . . . . . . 4.1.2 Metric Projection onto a Finite Dimensional Affine Subspace .. . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.1.3 Metric Projection onto a Half-Space .. . .. . . . . . . . . . . . . . . . . . . . 4.1.4 Metric Projection onto a Band. . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.1.5 Metric Projection onto the Orthant .. . . . .. . . . . . . . . . . . . . . . . . . . 4.1.6 Metric Projection onto Box Constraints . . . . . . . . . . . . . . . . . . . . 4.1.7 Metric Projection onto a Ball . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.1.8 Metric Projection onto an Ellipsoid .. . . .. . . . . . . . . . . . . . . . . . . . 4.1.9 Metric Projection onto an Ice Cream Cone . . . . . . . . . . . . . . . . . 4.2 Cutters .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.2.1 Characterization of Cutters . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.2.2 Cutters with Subsets of Fixed Points Being Affine Subspaces .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.2.3 Subgradient Projection.. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.3 Alternating Projection . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.3.1 Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.3.2 Fixed Points of the Alternating Projection .. . . . . . . . . . . . . . . . . 4.3.3 Alternating Projection for a Closed Affine Subspace . . . . . . 4.3.4 Generalized Relaxation of the Alternating Projection.. . . . . 4.3.5 Averaged Alternating Reflection . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.4 Simultaneous Projection.. . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.4.1 Simultaneous Projection as an Alternating Projection in a Product Space . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .

129 129 129 132 133 133 134 135 137 137 140 142 142 143 144 147 147 148 151 152 160 162 163

Contents

xv

4.4.2 4.4.3

Properties of the Simultaneous Projection .. . . . . . . . . . . . . . . . . Simultaneous Projection for a System of Linear Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.4.4 Simultaneous Projection for the Linear Feasibility Problem . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.5 Cyclic Projection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.5.1 Cyclic Relaxed Projection . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.5.2 Cyclic-Simultaneous Projection .. . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.5.3 Projections with Reflection onto an Obtuse Cone .. . . . . . . . . 4.5.4 Cyclic Cutter . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.6 Landweber Operator .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.6.1 Main Properties . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.6.2 Landweber Operator for Linear Systems . . . . . . . . . . . . . . . . . . . 4.6.3 Extrapolated Landweber Operator for a System of Linear Equations . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.7 Projected Landweber Operator.. . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.8 Simultaneous Cutter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.9 Extrapolated Simultaneous Cutter . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.9.1 Properties of the Extrapolated Simultaneous Cutter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.9.2 Extrapolated Simultaneous Projection . .. . . . . . . . . . . . . . . . . . . . 4.9.3 Extrapolated Simultaneous Projection for LFP .. . . . . . . . . . . . 4.9.4 Surrogate Projection . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.9.5 Surrogate Projection with Residual Selection . . . . . . . . . . . . . . 4.9.6 Extrapolated Simultaneous Subgradient Projection . . . . . . . . 4.10 Extrapolated Cyclic Cutter . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.10.1 Useful Inequalities . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 4.10.2 Properties of the Extrapolated Cyclic Cutter . . . . . . . . . . . . . . . 4.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5 Projection Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.1 Alternating Projection Methods .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.1.1 General Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.1.2 Alternating Projection Method for Closed Linear Subspaces.. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.2 Extrapolated Alternating Projection Methods .. . .. . . . . . . . . . . . . . . . . . . . 5.2.1 Acceleration Techniques for Consistent Problems . . . . . . . . . 5.2.2 Acceleration Techniques for Inconsistent Problems . . . . . . . 5.2.3 Douglas–Rachford Algorithm .. . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.3 Projected Gradient Method.. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.4 Simultaneous Projection Method . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.4.1 Convergence of the SPM . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.4.2 Projected Simultaneous Projection Methods . . . . . . . . . . . . . . .

165 168 169 171 172 173 174 176 176 177 178 181 184 185 187 187 189 190 191 196 198 199 200 201 202 203 204 204 206 208 209 210 212 213 215 215 217

xvi

Contents

5.5

5.6

5.7 5.8

5.9

5.10

5.11

5.12

5.13

5.14

Cyclic Projection Methods . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.5.1 Convergence .. . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.5.2 Projection-Reflection Method . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . Successive Projection Methods . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.6.1 Convergence .. . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.6.2 Control Sequences . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.6.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . Landweber Method and Projected Landweber Method .. . . . . . . . . . . . . Simultaneous Cutter Methods .. . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.8.1 Assumptions on Weight Functions . . . . .. . . . . . . . . . . . . . . . . . . . 5.8.2 Convergence Theorem .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.8.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.8.4 Block Iterative Projection Methods . . . . .. . . . . . . . . . . . . . . . . . . . Sequential Cutter Methods . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.9.1 Convergence Theorem .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.9.2 Control Sequences for Sequential Cutter Methods .. . . . . . . . 5.9.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . Extrapolated Simultaneous Cutter Methods .. . . . .. . . . . . . . . . . . . . . . . . . . 5.10.1 Assumptions on Step Sizes . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.10.2 Convergence Theorem .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.10.3 Extrapolated Simultaneous Subgradient Projection Method . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . Extrapolated Cyclic Cutter Method .. . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.11.1 Convergence .. . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.11.2 Accelerated Kaczmarz Method for a System of Linear Equations . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . Surrogate Constraints Methods . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.12.1 Proper Control.. . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.12.2 Convergence Theorem .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.12.3 Examples of Proper Control .. . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . SCM with Residual Selection . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.13.1 General Properties . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.13.2 Description of the Method .. . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.13.3 Obtuse Cone Selection .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 5.13.4 Regular Obtuse Cone Selection . . . . . . . . .. . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .

218 219 220 222 222 223 227 228 230 231 242 245 249 250 251 251 252 253 254 255 258 259 260 262 263 264 265 266 268 268 271 272 273 274

References .. .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 275 Index . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 295

Chapter 1

Introduction

1.1 Background In this section we present the notation, definitions and basic facts of convex subsets and convex functions defined on a Hilbert space, convergence and differentiation properties, properties of matrices, etc., which will be used in further parts of the book.

1.1.1 Hilbert Space A linear space pH with an inner product h; i, which is complete with respect to the norm kk WD h; i induced by this inner product is called a Hilbert space. In what follows, we consider a real Hilbert space H, and denote by X a nonempty closed convex subset of H and by I a finite subset f1; 2; : : : mg  N, where m 2 N.

1.1.1.1 Properties of the Inner Product The following equalities hold for all x; y 2 H: kx C yk2 D kxk2 C kyk2 C 2hx; yi,

(1.1)

kx  yk2 D kxk2 C kyk2  2hx; yi

(1.2)

hx C y; x  yi D kxk2  kyk2 .

(1.3)

and Cauchy–Schwarz inequality. The following inequality holds for all x; y 2 H:  kxk kyk  hx; yi  kxk kyk . A. Cegielski, Iterative Methods for Fixed Point Problems in Hilbert Spaces, Lecture Notes in Mathematics 2057, DOI 10.1007/978-3-642-30901-4 1, © Springer-Verlag Berlin Heidelberg 2012

(1.4) 1

2

1 Introduction

Furthermore, (i) hx; yi D kxk kyk if and only if x and y are positive linearly dependent vectors, i.e., ˛x D ˇy for some ˛; ˇ  0, (ii) hx; yi D  kxk kyk if and only if x and y are negative linearly dependent vectors, i.e., ˛x D ˇy for some ˛; ˇ  0. Moreover, one can take ˛ D kyk and ˇ D kxk in (i) and in (ii). Triangle inequality. The following inequality holds for all x; y 2 H: j kxk  kyk j  kx C yk  kxk C kyk .

(1.5)

Furthermore, (i) kx C yk D kxk C kyk if and only if x and y are positive linearly dependent, (ii) kx C yk D j kxk  kyk j if and only if x and y are negative linearly dependent. Property (i) is known as the strict convexity of the norm and can also be formulated equivalently in the form: (iii) For any  2 .0; 1/ there holds kxk D kyk D k.1  /x C yk H) x D y.

(1.6)

Convexity of the norm. Let w D .!1 ; : : : ; !m / 2 Rm C (all coordinates of w are nonnegative). The following inequality holds for all xi 2 H, i D 1; 2; : : : ; m;   m m  X X    ! x !i kxi k .  i i   i D1

(1.7)

i D1

If, moreover, !i > 0, i D 1; 2; : : : ; m, then the equality holds in (1.7) if and only if all xi , i D 1; 2; : : : ; m, are pairwise positive linearly dependent. Let w D .!1 ; : : : ; !m / 2 Rm . The following equalities hold for all xi 2 H, i D 1; 2; : : : ; m;  m 2 m m X  X X   !i xi  D !i !j hxi ; xj i D !i2 kxi k2 C    i D1

i;j D1

i D1

m X

2!i !j hxi ; xj i.

i;j D1;i ¤j

Parallelogram law. The following equality holds for all x; y 2 H: kx C yk2 C kx  yk2 D 2.kxk2 C kyk2 /.

(1.8)

Theorem 1.1.1. A Banach space with a norm kk is a Hilbert space with an inner product defined by the equality

1.1 Background

3

hx; yi D

1 .kxk2 C kyk2  kx  yk2 /, 2

if and only if the parallelogram law holds.

1.1.1.2 Examples of Hilbert Spaces Example 1.1.2. The Euclidean space Rn WD fx D .1 ; 2 ; : : : ; n / W j 2 R; j D 1; 2; : : : ; ng. An element x D .1 ; 2 ; : : : ; n / of Rn can be identified with a matrix 2 3 1 6 2 7 6 7 of type n  1 (column vector), i.e., we write x D 6 : 7 or x D Œ1 ; 2 ; : : : ; n > . 4 :: 5 n The Euclidean space is equipped with the standard inner product defined by hx; yi WD x > y D

n X

j j ,

j D1 1

where x D .1 ; : : : ; n /, y D .1 ; : : : ; n /, and with the norm kxk WD .x > x/ 2 . Example 1.1.3. The space Rm with the inner product induced by a positive definite matrix G D Œgij mm , defined by hw; viG WD w> Gv D

m m X X

gij !i j ,

(1.9)

i D1 j D1 1

where w D .!1 ; : : : ; !m /, v D .1 ; : : : ; m /, and with the norm kukG WD .u> Gu/ 2 . Example 1.1.4. The space l2 of real sequences x D .1 ; 2 ; : : :/ which are squaresummable, i.e., l2 WD fx W k 2 R, k D 1; 2; : : : and

1 X

k2 < 1g,

kD1

with the inner product hx; yi WD

P1

kD1 k k , and with

the norm kxk WD.

P1

2 12 kD1 k / .

Example 1.1.5. L2 .Œa; b/, where a; b 2 Œ1; C1, a < b—the space of (equivalence classes of) Lebesgue measurable functions f W R ! R which are square-integrable, i.e., Z

b

L2 .a; b/ WD ff W a

f 2 .x/dx < 1g

4

1 Introduction

with the inner product

Z

b

hf; gi WD

f .x/g.x/dx, a

and with the norm

Z

b

kf k WD .

1

f 2 .x/dx/ 2 . a

Example 1.1.6. Let Hi be a Hilbert space with an inner product h; iHi and with the norm kkHi induced by this inner product, i 2 I WD f1; 2; : : : ; mg. Let H D H1  : : :  Hm . The function h; i W H  H ! R defined by hx; yiw WD

m X

!i hxi ; yi iHi ,

(1.10)

i D1

where x D .x1 ; : : : ; xm /, y D .y1 ; : : : ; ym / 2 H, w D .!1 ; : : : ; !m / 2 Rm CC (all coordinates of w are positive), is an inner product in H, and the function kk W H ! R defined by the equality kxkw WD .

m X

1

!i kxi k2Hi / 2

i D1

is the norm in H induced by this inner product. The space H with the inner product h; iw defined by (1.10) is a Hilbert space and is called a product Hilbert space.

1.1.1.3 Weak Convergence in Hilbert Spaces We say that a sequence fx k g1 kD0 of elements of a Hilbert space H converges weakly to x 2 H if for any y 2 H the sequence fhy; x k ig1 kD0 converges to hy; xi. We k call the point x a weak limit of the sequence fx k g1 * x. If a kD0 and write x nk 1 k 1 subsequence fx gkD0  fx gkD0 converges weakly to x, then x is called a weak k 1 cluster point of the sequence fx k g1 kD0 . We say that fx gkD0 converges (strongly) to  k   x if limk!1 x  x D 0. Weakly convergent sequences have the following properties: A weakly convergent sequence fx k g1 kD0  H has exactly one weak limit. A weakly convergent sequence fx k g1 kD0  H is bounded. A bounded sequence fx k g1  H includes a weakly convergent subsequence. kD0 If a sequence fx k g1  H is bounded and has exactly one weak cluster point kD0 x 2 H, then x k * x. 5. If a sequence fx k g1 kD0 converges to x 2 H, then it converges weakly to x 2 H. 6. A weakly convergent sequence fx k g1 kD0 of a finite dimensional Hilbert space H is convergent. 1. 2. 3. 4.

1.1 Background

5

Note that a bounded sequence fx k g1 kD0 of a Hilbert space H needs not to include a convergent subsequence.

1.1.2 Notations and Basic Facts 1.1.2.1 Euclidean Space Elements of the Euclidean space Rn can be represented as vectors with real coordinates, e.g., x D .1 ; : : : ; n /, y D .1 ; : : : ; n /, z D .1 ; 2 ; : : : ; n / or as n  1 matrices (column vectors), e.g., x D Œ1 ; : : : ; n > , y D Œ1 ; : : : ; n T , z D Œ1 ; 2 ; : : : ; n > , where j , j , i are coordinates of x, y and z respectively, j D 1; 2; : : : ; n. We usually denote the coordinates of a vector in the Euclidean space with Greek letters. Let x; y 2 Rn . Then, x  0 means that all coordinates of the vector x are nonnegative and x > 0 means that all coordinates of the vector x are positive (the symbols x  0 and x < 0 are defined similarly). Furthermore, x  y denotes that y  x  0 and x  y denotes that x  y  0. We use the following notation: maxfx; yg WD .maxf1 ; 1 g; : : : ; maxfn ; n g/; minfx; yg WD .minf1 ; 1 g; : : : ; minfn ; n g/; xC WD maxfx; 0g; x WD maxfx; 0g; x D xC  n m x ; Rm C WD fx 2 R W x  0g denotes the nonnegative orthant; R WD fx 2 n m n R W x  0g denotes the nonpositive orthant; RCC WD fx 2 R W x > 0g denotes the positive orthant; ej WD .0; : : : ; 0; 1; 0; : : : ; 0/ 2 Rm denotes the j th unit vector, i.e., a vector, for which the j th coordinate equals 1 and the others are zeros; e denotes a vector with all coordinates equal to 1, i.e., e D .1; : : : ; 1/; m WD fu 2 Rm W u  0; e > u D 1g denotes the standard simplex; N denotes the set of positive integers; For a finite subset J  N, the symbol jJ j denotes the number of elements of J .

1.1.2.2 Subsets of a Hilbert Space Let V  H. The subset V ? WD fy 2 H W hy; xi D 0 for all x 2 V g denotes the subspace orthogonal to V . Let V  H be a closed subspace. Then H D V ˚ V ? , i.e., for any x 2 H, there are unique v 2 V and w 2 V ? such that x D v C w. The subset B.x; / WD fy 2 H W ky  xk  g denotes a ball with a centre x 2 H and radius > 0. C 0 denotes the complement of a subset C  H, i.e., C 0 D HnC . The subset

6

1 Introduction

C C D WD fz 2 H W z D x C y; x 2 C; y 2 Dg denotes the Minkowski sum of C; D  H. In particular, C C a WD C C fag D fx C a W x 2 C g, where C  H and a 2 H. The subsets int C , cl C and bd C denote the interior, the closure and the boundary of a subset C  H, respectively. The subset S.x; / WD bd B.x; / D fy 2 H W ky  xk D g denotes a sphere with a centre x 2 H and radius > 0. Lin S and Lin a denote a linear subspace generated by the subset S  H and by the vector a 2 H, respectively. The subset Œa; b WD fz 2 H W z D .1  /a C b;  2 Œ0; 1g denotes a segment with endpoints a; b 2 H. The subset Argmin f .x/ WD fz 2 C W f .z/  f .x/ for all x 2 C g, x2C

where C  H and f W C ! R is called a subset of minimizers of a function f . An element of Argminx2C f .x/ is called a minimizer of f and is denoted by argminx2C f .x/. The subset H.a; ˇ/ WD fx 2 H W ha; xi D ˇg, where a 2 H, a ¤ 0 and ˇ 2 R is called a hyperplane in a Hilbert space H. The hyperplane H.a; ˇ/ is the boundary of two half-spaces HC .a; ˇ/ WD fx 2 H W ha; xi  ˇg and H .a; ˇ/ WD fx 2 H W ha; xi  ˇg. A subset K  H is called an affine subspace if for all x; y 2 K and for any  2 R we have .1  /x C y 2 K. The number ^.a; b/ WD arccos

ha; bi kak  kbk

denotes an angle between nonzero vectors a; b 2 H.

1.1 Background

7

1.1.2.3 Functions Let X  H, fi W X ! R, i 2 I . The function f WD maxi 2I fi is defined by f .x/ WD maxffi .x/ W i 2 I g In particular, fC WD maxff; 0g, f WD maxff; 0g. Let f W X ! R. The subset S.f; ˛/ D fx 2 X W f .x/  ˛g is called a sublevel set of f at a level ˛ 2 R. The subset epi f WD f.x; / 2 X  R W  f .x/g  X  R is called an epigraph of f . We say that f is lower semi-continuous at x 2 X if lim infy!x f .y/  f .x/. We say that f is weakly lower semi-continuous at x 2 X if lim infy*x f .y/  f .x/. We say that a function f W H ! R is coercive if limkxk!1 f .x/ D C1. We say that a function f W X ! R attains the global minimum at a point x  2 X if f .x  /  f .x/ for all x 2 X . We say that a function f W X ! R attains a local minimum at a point x  2 X if there exists ı > 0 such that f .x  /  f .x/ for all x 2 X \ B.x  ; ı/. Theorem 1.1.7. A function f W X ! R is (weakly) lower semi-continuous if and only if epi f is a (weakly) closed subset of X  R. Theorem 1.1.8. If a function f W X ! R is weakly lower semi-continuous on a weakly closed and bounded subset X  H, then f is bounded from below and attains its minimum. Theorem 1.1.9. If a function f W H ! R is weakly lower semi-continuous and coercive and X  H is weakly closed, then f attains the global minimum on X . f .x C ts/  f .x/ exists, then we call it a t directional derivative of a function f W H ! R at a point x 2 H and in a direction s 2 H and we denote it by f 0 .x; s/.

Definition 1.1.10.

If a limit limt #0

If a function f W H ! R attains minimum at x 2 H and f 0 .x; s/ exists for some s 2 H, then f 0 .x; s/  0. Definition 1.1.11. We say that a function f W H ! R is Fr´echet-differentiable or, shortly, differentiable at x 2 H if there exists y 2 H such that f .x C h/ D f .x/ C hy; hi C o.khk/, o.t/ D 0. The element y is called a derivative (or differential) of f t at x and is denoted by f 0 .x/. where limt !0

8

1 Introduction

The derivative of a differentiable function is uniquely determined. Definition 1.1.12. We say that a function f W H ! R is Gˆateaux-differentiable at a point x 2 H if it has directional derivatives f 0 .x; s/ for all s 2 H and f 0 .x; s/ D hg; si holds for some g 2 H. The element g is called a Gˆateaux-derivative or Gˆateauxdifferential of f at x and is denoted by Df .x/. The Gˆateaux-differential of a Gˆateaux-differentiable function is uniquely determined. Theorem 1.1.13. If a function f W H ! R is Fr´echet-differentiable at x 2 H, then it is Gˆateaux-differentiable at x and f 0 .x/ D Df .x/. Theorem 1.1.14. If a function f WH ! R is Gˆateaux-differentiable in a neighborhood of x and the Gˆateaux-differential Df is continuous at x, then f is Fr´echet-differentiable at x. If H is a Euclidean space (H D Rn and h; i is the standard inner product), then the derivative Df .x/ is called a gradient of f at the point x and is denoted by rf .x/. There holds the equality rf .x/ D . @f@.x/ ; : : : @f@.x/ /, where @f@.x/ denotes a partial 1 n j derivative of f at the point x D .1 ; : : : ; n / with respect to j . Let x 2 Rn and 2f .x/ for all assume that f W Rn ! R has partial derivatives of the second order @@i @ j i; j D 1; 2; : : : ; n. Then the matrix 2 6 r 2 f .x/ WD 6 4

@2 f @12

3

.x/ : : :

::: @2 f .x/ @n @1

::: :::

@2 f @1 @n .x/ 7

::: 7 5 .x/

@2 f @n2

is called the Hessian of f at . 2

f are continuous at a point Theorem 1.1.15 (Schwarz). Let f W Rn ! R. If @@i @ j x 2 Rn , i; j D 1; 2; : : : ; n, then the Hessian is a symmetric matrix.

1.1.2.4 Operators Let H; H1 ; H2 ; H3 be Hilbert spaces. The operator Id W H ! H denotes the identity, i.e., Id x D x for all x 2 H. Definition 1.1.16. Let A W H1 ! H2 be a bounded linear operator. The number kAk WD

kAxkH2 , xW0
1.1 Background

9

where kkH1 and kkH2 denote the norms in H1 and H2 , respectively, is called a norm of A in L.H1 ; H2 /. Definition 1.1.17. Let A W H1 ! H2 be a bounded linear operator. An operator A W H2 ! H1 with the property hAx; yiH2 D hx; A yiH1 for all x 2 H1 and y 2 H2 , where h; iH1 and h; iH2 are inner products in H1 and H2 , respectively, is called an adjoint operator. For every A an adjoint operator A exists and is uniquely determined. Futhermore, A is a bounded linear operator. Definition 1.1.18. We say that a bounded linear operator A W H ! H is unitary if A A D Id holds. Definition 1.1.19. We say that a bounded linear operator A W H ! H is selfadjoint, if A D A. Definition 1.1.20. We say that a self-adjoint operator A W H ! H is nonnegative if hAx; xi  m kxk2 for some constant m  0 and for all x 2 H. If m > 0, then we say that A is positive (or elliptic). Theorem 1.1.21. For any bounded linear operator A W H1 ! H2 it holds kAk2 D kA k D kA Ak D kAA k . 2

Theorem 1.1.22. If a bounded linear operator A W H ! H is self-adjoint, then kAk D supkxk1 hAx; xi. Theorem 1.1.23. Let A W H1 ! H2 be a bounded linear operator. Then the operators AA and A A are nonnegative. A number  2 R is called an eigenvalue of a bounded linear operator A W H ! H if Ax D x for some x ¤ 0. Theorem 1.1.24. If  is an eigenvalue of a bounded linear operator A W H ! H, then jj  kAk. In particular, if A is self-adjoint, then jj  supkxk1 hAx; xi. Definition 1.1.25. We say that a bounded linear operator A W H1 ! H2 is compact k 1 if for any bounded sequence fx k g1 kD0  H1 , the sequence fAx gkD0  H2 contains a convergent subsequence. If dim H1 < 1, then every bounded linear operator A is compact. Theorem 1.1.26. Let A W H ! H be a nonnegative, nonzero compact linear operator. Then: (i) There exists an eigenvalue  D max .A/ of the operator A such that

10

1 Introduction

 D kAk D sup hAx; xi;

(1.11)

kxk1

consequently, at least one eigenvalue of the operator A is positive. (ii) If A is positive, then all of its eigenvalues are positive. (iii) There exists z 2 H with kzk D 1 such that hAz; zi D sup hAx; xi. kxk1

Theorem 1.1.27. Let A W H1 ! H2 be a linear and compact operator. Then kAk D

p p max .A A/ D max .AA / D kA k :

Definition 1.1.28. We say that an operator T W H ! H is monotone if hT x  T y; x  yi   kx  yk2 for a constant   0 and for all x; y 2 X . If  > 0, then we say that T is strongly monotone or -strongly monotone. Definition 1.1.29. We say that an operator T W H ! H is Lipschitz continuous if kT x  T yk  kx  yk . for a positive constant . The constant is called a Lipschitz constant. We also say that T is -Lipschitz continuous. Definition 1.1.30. We say that an operator T W H1 ! H2 is differentiable at a point x 2 H1 if there exists a bounded linear operator G 2 L.H1 ; H2 / such that T .x C h/ D T .x/ C Gh C o.khk/. The operator G is uniquely determined and is called a derivative or a differential of the operator T at the point x and is denoted by DT .x/ or T 0 x. Theorem 1.1.31. Let T W H1 ! H2 be continuously differentiable in a neighborhood of a point x 2 H1 and U W H2 ! H3 be continuously differentiable in a neighborhood of T x 2 H2 . Then there exists a differential D.U ı T /.x/ and D.U ı T /.x/ D D.U.T x//DT .x/. Definition 1.1.32. Let T W X ! Y , where X  H1 and Y  H2 , and let C  X and D  Y . The subset T .C / WD fT x W x 2 C g  Y

1.1 Background

11

is called an image of C by the operator T . The subset T 1 .D/ WD fx 2 X W T x 2 Dg is called an inverse image of D by the operator T . The iteration T k of an operator T W X ! X , where X  H, is defined by the recurrence: T 1 D T , T kC1 D T T k , k  1. For any x 2 X the sequence fT k xg1 kD0 is called an orbit of T . A point z 2 X is called a fixed point of an operator T W X ! H, if T z D z. The subset of all fixed points of the operator T is denoted by Fix T . A subset C  X is called a retract of X if there exists a continuous operator T W X ! C (called a retraction) with Fix T D C . We say that an operator T W X ! X is idempotent or a projection if T T x D T x for all x 2 X , i.e., T .X /  Fix T . Since the converse implication holds for all operators, we have: T is idempotent if and only if T .X / D Fix T . A continuous projection is a retraction. Let V  H be a closed subspace. The operator P W H ! V with the property P x  x 2 V ? for all x 2 H is called the orthogonal projection onto V . The orthogonal projection P onto a closed subspace is defined uniquely and is a linear and bounded operator with the norm kP k D 1. Let a 2 H. The orthogonal projection onto Lin a is denoted by Pa . We say that an operator A W H1 ! H2 is affine if for all x; y 2 H1 and for any  2 R, A..1  /x C y/ D .1  /Ax C Ay holds. Let A W H1 ! H2 be a linear operator. The subset ker A WD fx 2 H1 W Ax D 0g denotes a kernel or null space of A and the subset Im A WD fy 2 H2 W y D Ax for some x 2 H1 g denotes an image or range of A. The subsets ker A  H1 and Im A  H2 are subspaces of H1 and H2 , respectively. It holds .ker A/? D Im A , consequently,

H1 D ker A ˚ Im A .

1.1.2.5 Matrices Definition 1.1.33. We say that a symmetric matrix A of type n  n is positive definite, if x > Ax > 0 for all nonzero vectors x 2 Rn . We say that A is positive semi-definite if x > Ax  0 for all x 2 Rn . Let v D .1 ; : : : ; m / 2 Rm . Then diag v denotes a diagonal matrix of type mm with 1 ; : : : ; m on its diagonal, i.e.,

12

1 Introduction

2

1 6 0 diag v D 6 4::: 0

0 2 ::: 0

::: ::: ::: :::

3 0 0 7 7. :::5 m

Theorem 1.1.34. For a symmetric matrix A the following conditions are equivalent: (i) A is positive definite (positive semi-definite), (ii) All eigenvalues of A are positive (nonnegative), (iii) There exists an orthogonal matrix U and a diagonal matrix D with positive (nonnegative) elements on the diagonal such that A D U > DU , (iv) There exists a positive definite (positive semi-definite) matrix B such that A D B 2. 1

We denote by A 2 the matrix B with property (iv) above. Definition 1.1.35. Let A be an m  n matrix. The matrix A> A is called the Gram matrix of the columns of A. The Gram matrix A> A is positive semi-definite. Furthermore, it is positive definite if and only if A has full column rank, i.e., the columns of A are linearly independent. Definition 1.1.36. The Moore–Penrose pseudoinverse AC of an m  n matrix A is a uniquely determined n  m matrix satisfying the following conditions: AAC A D A, AC AAC D AC , .AAC /> D AAC , .AC A/> D AC A: If A is a full column rank or a full row rank matrix, then AC D .A> A/1 A> or

AC D A> .AA> /1 ,

respectively.

1.1.2.6 Convex Subsets Let H, Hi , i D 1; 2; : : : ; m, be Hilbert spaces. Definition 1.1.37. A subset C  H is said to be convex, if .1  /x C y 2 C for all x; y 2 C and for all  2 Œ0; 1. An intersection of arbitrary family of convex subsets is convex. In particular, a polytope, i.e., the intersection of a finite number of half-spaces, is convex. The ball

1.1 Background

13

B.x; / is convex for any x 2 H and for any  0. Let A W H1 ! H2 be an affine operator. If C  H1 is a convex subset, then the image A.C /  H2 is convex. If D  H2 is convex, then the inverse image A1 .D/  H1 is convex. The product C1  : : :  Cm of convex subsets Ci  Hi , i D 1; 2; : : : ; m, is a convex subset of the product space H WD H1  : : :  Hm . Theorem 1.1.38. If C  H is convex, then its closure and its interior are convex subsets. Theorem 1.1.39. A subset C  H is closed and convex if and only if C is an intersection of half-spaces. The following theorem follows from the Mazur theorem (see, e.g., [158, Chap. I, Sect. 1.2], [180, Sect. 2.3, Corollary 2], [267, Theorem 21.4], [300, Theorem 3.12] or [214, Theorem 8.48]). Theorem 1.1.40. If C  H is a closed convex subset, then C is weakly closed. Definition 1.1.41. The smallest convex subset containing S  H is called a convex hull of S and is denoted by conv S . The smallest affine subspace containing S  H is called an affine hull of S and is denoted by aff S . The subset ri C WD fx 2 C W B.x; "/ \ aff C  C for some " > 0g is called a relative interior of a convex subset C  H . It holds

ri m D fu 2 Rm W u > 0; e > u D 1g,

where m denotes a standard simplex in Rn . Definition 1.1.42. A subset K  H is called a convex cone if ˛x C ˇy 2 K for all x; y 2 K and for all ˛; ˇ  0. Definition 1.1.43. For a convex cone C  H the subset C  WD fx 2 H W hy; xi  0 for all y 2 C g is called a polar cone to C  H. Definition 1.1.44. We say that a convex cone C  H is obtuse if C   C . Theorem 1.1.45 (Moreau’s decomposition). Let K  H be a nonempty closed convex cone and x; x C ; x  2 H. Then the following conditions are equivalent: (i) x D x C C x  , where x C 2 K, x  2 K  and hx C ; x  i D 0, (ii) x C D PK x and x  D PK  x. Definition 1.1.46. The subset NC .x/ WD fy 2 H W hy; z  xi  0 for all z 2 C g is called a normal cone to a convex subset C  H at x 2 C . The subset

14

1 Introduction

 TC .x/ WD s 2 H W 9fx k g1 9ftk g1 k kD0 ;tk #0 kD0 C;x !x

xk  x with s D lim k tk



is called the tangent cone to a convex subset C  H at x 2 C . Theorem 1.1.47. Let C  H be a closed convex subset and x 2 C . The following equalities hold .NC .x// D TC .x/ and .TC .x// D NC .x/.

1.1.2.7 Convex Functions Let X; Y  H be convex subsets of a Hilbert space H and let f W X ! R. Definition 1.1.48. We say that f is convex if f ..1  /x C y/  .1  /f .x/ C f .y/ holds for all x; y 2 X and for all  2 Œ0; 1. If the inequality is strict for all x; y 2 X , x ¤ y and for all  2 .0; 1/, then we say that f is strictly convex. We say that f is ˛-strongly convex, where ˛ > 0 or, shortly, strongly convex if 1 f ..1  /x C y/  .1  /f .x/ C f .y/  ˛.1  / kx  yk2 2 for all x; y 2 X and for all  2 Œ0; 1. A function f W X ! R is called concave if f is convex.

Properties of convex functions 1. If fi W X ! R, i 2 I , are convex, then the function f WD maxi 2I fi is convex. 2. If f W X ! R is convex and g W R ! R is convex and nondecreasing, then the composition g ı f is convex. 3. If fi W X ! R, i D 1; 2; : : : ; m, are convex and F W Rm ! R is a convex function which is nondecreasing with respect to any coordinate, then the function f WD F .f1 ; : : : ; fm / is convex. 4. If f W Y ! R is a convex function and A W X ! Y is an affine operator, then the function f ı A is convex. 5. If f W X  Y ! R is convex, then the function h W X ! R, h.x/ WD infy2Y f .x; y/ is convex. 6. Any sublevel set S.f; ˛/, ˛ 2 R, of a convex function f W X ! R is convex. 7. A function f W X ! R is convex if and only if its epigraph epi f is a convex subset of X  R. 8. A convex function f W Rn ! R is continuous.

1.1 Background

15

9. A strongly convex function f W H ! R is coercive. 10. A convex function f W H ! R has directional derivatives f 0 .; s/ for all s 2 H, and for any x; s 2 H it holds f 0 .x; s/ D inf

t >0

f .x C ts/  f .x/ . t

11. Let x 2 H. The directional derivative f 0 .x; / W H ! R of a convex function f W H ! R is convex and positively homogeneous of degree 1, i.e., for all s 2 H and for all ˛  0 the equality f 0 .x; ˛s/ D ˛f 0 .x; s/ holds. 12. If f W H ! R is a convex function and f 0 .x; s/  0 for all s 2 H, then f attains its minimum at x 2 H. 13. The norm kk W H ! R is convex and differentiable on Hnf0g and D kxk D x for any x ¤ 0. kxk 14. The function f W H ! R, f .x/ WD 12 kxk2 is differentiable and 1-strongly convex, and Df .x/ D x for all x 2 H. 15. Let C  H be nonempty. A function d.; C / W H ! R defined by d.x; C / WD infz2C kz  xk is called a distance function to the subset C . If C is convex, then d.; C / is a convex function. 16. Let a function f W Rn ! R have continuous partial derivatives of the second order. Then f is convex if and only if its Hessian r 2 f .x/ is a positive semidefinite matrix for all x 2 Rn . The function f is ˛-strongly convex if and only if s > r 2 f .x/s  ˛ ksk2 for all x; s 2 Rn . Theorem 1.1.49. If a convex function f W H ! R is bounded in a neighborhood of a point x 0 2 H, then f is locally Lipschitz continuous on H, i.e., for any x 2 H there is > 0 and  0 such that j f .u/  f .v/ j ku  vk for all u; v 2 B.x; /. In particular, a continuous convex function f W H ! R is locally Lipschitz continuous on H. If H is finite dimensional, then Theorem 1.1.49 can be strengthened. In this case the local Lipschitz continuity can be replaced by the global one on bounded subsets. Theorem 1.1.50. If a convex function f W H ! R is -Lipschitz continuous in a neighborhood of a point x 2 H, then for any s 2 H it holds f 0 .x; s/  ksk . Theorem 1.1.51. A lower semi-continuous convex function f W H ! R is weakly lower semi-continuous. Theorem 1.1.52. A continuous convex function f W X ! R defined on a closed bounded convex subset X  H attains its global minimum. Corollary 1.1.53. If a continuous convex function f W X ! R defined on a closed convex subset X  H is coercive, then f attains its global minimum on X . Theorem 1.1.54. A convex Gˆateaux-differentiable function f WRn ! R is differentiable.

16

1 Introduction

Subdifferential of a Convex Function and Its Properties Definition 1.1.55. Let f W H ! R be convex. The subset @f .x/ WD fg 2 H W hg; y  xi  f .y/  f .x/

for all y 2 Hg

is called a subdifferential of f at x 2 H. The function f is said to be subdifferentiable at x if @f .x/ ¤ ;. An element of the subdifferential @f .x/ is called a subgradient of f at x and is denoted by gf .x/. The affine functionfNx W H ! R defined by fNx .y/ WD hg; y  xi C f .x/, where g 2 @f .x/ is called a linearization of f at x. Theorem 1.1.56. For any x 2 H the subdifferential @f .x/ of a continuous convex function f W H ! R is a nonempty, weakly closed and bounded convex set. Theorem 1.1.57. A continuous convex function f W H ! R is Gˆateauxdifferentiable at x 2 H if and only if its subdifferential @f .x/ consists of one point. In this case @f .x/ D fDf .x/g. Theorem 1.1.58. Let f W H ! R be continuous and convex and x; s 2 H. The following equality holds f 0 .x; s/ D supfhy; si W y 2 @f .x/g. In particular, if f is differentiable, then f 0 .x; s/ D hDf .x/; si If, moreover, H is a Euclidean space, then f 0 .x; s/ D s > rf .x/. The subdifferential of the norm kk has the form ( @.kxk/ D

x g if x ¤ 0 f kxk B.0; 1/ if x D 0.

All the facts presented in this section can be found in the following references [33, 38, 40, 61, 133, 158, 180, 181, 209, 210, 214, 234, 262–264, 267, 298, 300, 317, 325].

1.2 Metric Projection In this section we define the metric projection in a Hilbert space H and present its basic properties. This operator plays an important role in further parts of the book. Other properties of the metric projection will be presented in Chap. 2. Definition 1.2.1. Let C  H be a nonempty subset and x 2 H. If there exists a point y 2 C such that

1.2 Metric Projection

17 x

Fig. 1.1 Metric projection

C z y = PC x

ky  xk  kz  xk for any z 2 C , then y is called a metric projection of x onto C and is denoted by PC x (Fig. 1.1). The vector s D PC x  x is called a projection vector of x onto C . If PC x exists and is uniquely determined for all x 2 H, then the operator PC W H ! C is called the metric projection (onto C ). Remark 1.2.2. Let C  H. If x 2 C , then it follows from the definition of the metric projection that PC x D x. If x … C and there exists a metric projection PC x, then PC x 2 bd C . Indeed, if PC x 2 int C , then B.PC x; "/  C for a sufficiently small " > 0. Then z WD x C .1 

" /.PC x  x/ 2 B.PC x; "/  C , kPC x  xk

because kz  PC xk D ", and kz  xk D .1 

" / kPC x  xk < kPC x  xk kPC x  xk

which contradicts the definition of PC x.

1.2.1 Existence and Uniqueness of the Metric Projection Definition 1.2.1 does not yield the existence and the uniqueness of a metric projection PC x for a point x 2 H and for a subset C  H. If C  H is compact, then the existence of a metric projection follows from the Weierstrass theorem but it needs not to be defined uniquely. For example, both points a; b 2 H are metric projections of the midpoint x WD aCb of the segment Œa; b onto the subset C D fa; bg. 2 Therefore, the convexity of C seems to be a natural assumption for the uniqueness of the metric projection PC x for any x 2 H. Furthermore, it follows from the definition of a metric projection and from the continuity of the norm that PC x (if it exists) belongs to the closure of C . Actually, PC x 2 bd C for x … C (see Remark 1.2.2). Therefore, the closedness of C is a natural assumption for the existence of the metric projection PC x for any x 2 H. It turns out that in a Hilbert space these two assumptions (the closedness and the convexity of C ) are sufficient for the existence

18

1 Introduction

and the uniqueness of the metric projection PC x for all x 2 C . The following theorem can be found, e.g., in [61, Sect. 1.2.2, Theorem 1.7], [140, Theorem 3.4(2)], [173, Theorem 7.43], [209, Chap. III, Sect. 3.1] or [267, Theorem 8.25]. Theorem 1.2.3. Let C  H be a nonempty closed convex subset. Then for any x 2 H there exists a metric projection PC x and it is uniquely determined. Proof. First we prove the assertion for a sequence fy k g1 kD0  C be such that parts.

x D 0. Let d WD inffkyk W y 2 C g and  y k  ! d . We split the proof onto three

(a) We show that fy k g1 kD0 is a Cauchy sequence. Let " > 0 and k0  0 be such  k 2 2    d C "=4 for k  k0 . Let k; that y  l  k0 . Since C is convex, we have 1 k 1 l 1 k l y  d . The parallelogram law yields y C y 2 C . Therefore, C y 2 2 2 now  k        y  y l 2 D 2 y k 2 C 2 y l 2  y k C y l 2  2.d 2 C "=4/ C 2.d 2 C "=4/  4d 2 D ", i.e., fy k g1 kD0 is a Cauchy sequence. (b) It follows from (a) that y k converges to an element y 2 H, because H is a complete space. Furthermore, y 2 C , because C is closed. Hence, by the continuity of the norm, it follows that kyk D d . Therefore, y D PC 0. (c) Now we show that the metric projection PC x is uniquely determined. Let y 0 2 C with ky 0 k D d . It follows from the convexity of C that 12 y C 12 y 0 2 C . Furthermore,   1 1 0 1  1  d  yC y  kyk C y 0  D d;  2 2 2 2 i.e., ky C y 0 k D 2d . Again, by the parallelogram law, we have       y  y 0 2 D 2 kyk2 C 2 y 0 2  y C y 0 2 D 2d 2 C 2d 2  4d 2 D 0; i.e., y D y 0 . We have proved that PC 0 exists and is uniquely defined. Of course, C  x is closed convex, because C is closed convex. Therefore, PC x 0 also exists and is uniquely defined. It follows easily from the definition of the metric projection that PC x D x C PC x 0, consequently, the assertion is true for every x 2 H.

t u

One can ask if the converse theorem to Theorem 1.2.3 is also true. The answer is positive in the Euclidean space and is known as the Motzkin theorem (see [336, Theorem 7.5.5]). In general Hilbert spaces the problem is still open.

1.2 Metric Projection

19

1.2.2 Characterization of the Metric Projection The theorem below gives a criterion for y 2 C to be the metric projection of a point x 2 H onto a convex set C  H and is often used in applications. The theorem can be found, e.g., in [140, Theorem 4.1], [173, Theorem 7.45], [209, Chap. III, Theorem 3.1.1] and in [185, Proposition 3.5] which contains several other equivalent conditions. Theorem 1.2.4 (Characterization Theorem). Let x 2 H, C  H be a convex subset and y 2 C . The following conditions are equivalent: (i) y D PC x, (ii) hx  y; z  yi  0 for all z 2 C . Proof. (i) )(ii). Let y D PC .x/, z 2 C and z D y C .z  y/ for  2 .0; 1/. Obviously, z 2 C because C is convex. By (i) and by the properties of the inner product, we have kx  yk2  kx  z k2 D kx  y  .z  y/k2 D kx  yk2  2hx  y; z  yi C 2 kz  yk . Since  > 0, the above inequalities yield hx  y; z  yi 

 kz  yk2 , 2

and for  ! 0 we obtain (ii) in the limit. (ii) )(i). By the properties of the inner product and by (ii) we obtain for any z 2 C kz  xk2 D kz  y C y  xk2 D kz  yk2 C ky  xk2 C 2hz  y; y  xi  ky  xk2 , which, by the definition of the metric projection, gives (i).

t u

Condition (ii) of Theorem 1.2.4 says that ^.x  y; z  y/  =2 if x  y and z  y are nonzero vectors (see Fig. 1.2). A simple proof of the following lemma is left to the reader. Lemma 1.2.5. Let x; y; z 2 H. The following conditions are equivalent: (a) (b) (c) (d)

hx  y; z  yi  0, hz  x; y  xi  ky  xk2 , kz  yk2  kz  xk2  ky  xk2 , hz  x; z  yi  0.

20

1 Introduction x

Fig. 1.2 Characterization of the metric projection

C z y

Fig. 1.3 Metric projection onto a translated subset

C x−u

−u

PC (x − u)

u

x

C+u PC+ u x

It follows from Lemma 1.2.5 that the inequality in condition (ii) of the Characterization Theorem 1.2.4 can be replaced by one of the conditions (a)–(d) of Lemma 1.2.5.

1.2.3 First Applications of the Characterization Theorem In this section we give some useful facts which follow from the Characterization Theorem 1.2.4. Lemma 1.2.6. Let x 2 H, C  H be a convex subset and u 2 H. If a metric projection PC .x  u/ exists, then a metric projection PC Cu x also exists and PC Cu x D PC .x  u/ C u

(1.12)

holds (Fig. 1.3). Proof. The proof of the lemma is a direct application of the Characterization Theorem 1.2.4 and is left to the reader. t u Lemma 1.2.7. Let u 2 H, C  H be convex and A W H ! H be a unitary operator. If a metric projection PC .A u/ exists, then a metric projection PA.C / u also exists and PA.C / u D A.PC .A u// holds (Fig. 1.4).

1.2 Metric Projection

21

Fig. 1.4 Metric projection onto a unitary transformation of a subset

A(C ) PA(C) u u

A∗ A

A∗ (u)

P C (A ∗ u)

C

Proof. Let y D PC .A u/ and v D Ay. We show that v D PA.C / u. Since y 2 C , we have v 2 A.C /. Since A is a linear operator and C is convex, the subset A.C / is convex. By the characterization of the metric projection (see Theorem 1.2.4), for all w 2 A.C / and for z 2 C with Az D w, we have 0  hA u  y; z  yi D hA u  A Ay; z  yi D hu  v; Az  Ayi D hu  v; w  vi. Again by the characterization of the metric projection (see Theorem 1.2.4), we u t obtain v D PA.C / u. Let Hi be Hilbert spaces with the inner products h; iHi , i 2 I WD f1; 2; : : : ; mg, and H WD H1  : : :  Hm be the product Hilbert space. Lemma 1.2.8. Let Ci  Hi be convex, i D 1; 2; : : : ; m, C WD C1  : : :  Cm 2 H and x D .x1 ; : : : ; xm / 2 H. If metric projections PCi xi , i D 1; 2; : : : ; m, exist, then a metric projection PC x also exists and PC x D .PC1 x1 ; : : : ; PCm xm /: Proof. The proof of the lemma is a direct application of the Characterization Theorem 1.2.4 and is left to the reader. t u Lemma 1.2.8 is illustrated in Fig. 1.5. Lemma 1.2.9. Let C  H be a nonempty closed convex set and PC W H ! H be a metric projection onto C . Then for any x 2 H the following conditions are equivalent: (i) y D PC x (ii) y 2 C and x  y 2 NC .y/.

22

1 Introduction

Fig. 1.5 Metric projection onto a product of subsets

Fig. 1.6 Metric projection and normal cone x

NC (y)

z

y = PC x

C

Proof. By the definition of the normal cone, we can write the second part of the right hand side of (ii) in the form hx  y; z  yi  0 for any z 2 C (Fig. 1.6). The lemma follows now from the characterization of the metric projection (see Theorem 1.2.4). t u For T W H ! H and  2 R denote T WD Id C.T  Id/. Corollary 1.2.10. Let T W H ! H, C  H be a nonempty closed convex subset. Then Fix.PC T / D Fix.PC T / for any  > 0. Proof. Let  > 0. It follows from Lemma 1.2.9 that x

2 Fix.PC T / ” PC .x C .T x  x// D x ” .T x  x/ 2 NC .x/ ” T x  x 2 NC .x/ ” PC T x D x ” x 2 Fix.PC T /

which completes the proof. In Sect. 2.2.3 we give further properties of the metric projection.

t u

1.3 Convex Optimization Problems

23

1.3 Convex Optimization Problems In this section we present several convex optimization problems: convex minimization, variational inequalities, convex feasibility problems and split feasibility problems. These problems as well as the methods for solving them have applications in various areas of mathematics, e.g., in solving linear equations, in probability and statistics, in conformal mappings, in frame design problems and in many other problems (for details, see, e.g., [139, 156, 182, 207, 329, 337]). Furthermore, the mentioned above abstract problems can be treated as mathematical models for many practical problems which arise in physical, medical, technical and information sciences. The problems and methods presented in this book found applications: • In signal processing [192,275,316], in particular in resolution enhancement [109] and in signal synthesis [117, 123], • In image processing [37,312,316], in particular in image recovery [118,309,315], in image restoration [354, 359], in image reconstruction [220, 221, 280, 289], in image reconstruction from projections [98, 100, 111, 224, 330], in demosaicking [248] and in color imaging [311], • In discrete tomography [82], • In medical imaging, in particular in computerized tomography [81,189,204,205, 270, 313], in proton computed tomography: [282] and in magnetic resonance imaging [301], • In radiation therapy treatment planning [81, 83–85, 93, 110, 199, 206, 206, 212, 241, 261, 324, 334, 335, 341], • In physics [149, 240], in astrophysics [360], in acoustic [39], in crystallography [328], in mechanics [265], in materials science [225], • In seismic tomography [60, 332], in meteorology [323], • In optics [316], in particular in phase retrieval problem [26, 27, 161] and in holography [219, 310], • In wavelet-based denoising [115], in watermarking [242], in adaptive filtering [314, 355], • In control problems [192], in control design problems [193, 194], • In antenna design [195], in antenna pattern synthesis problems [288], • In data compression [245], in video coding [281], • In graph matching [340], • In learning process [319], in supervised learning process [222] • And in finance [208]. In the next chapters we will show how to solve convex optimization problems by iterative methods constructed by application of some algorithmic operators. We will also show that the problem of finding a solution of a convex optimization problem is equivalent to finding a fixed point of an algorithmic operator.

24

1 Introduction

1.3.1 Convex Minimization Problems Let f W H ! R and C  H. The constrained minimization problem minimize f .x/ with respect to x 2 C

(1.13)

is to find at least one local minimizer of the function f on C , i.e., a point x  2 C such that f .x  /  f .x/ for all x 2 C \ B.x  ; / and for some > 0, if it exists. If f .x  /  f .x/ for all x 2 C , then x  is called a global minimizer of the function f on C . The point x  is also called a (local or global) optimal solution of problem (1.13). The value f  D f .x  / is called the minimum of the function f on the subset C . Theorem 1.3.1. Let C  H be a convex subset and f W H ! R be a convex function. If x  is a local minimizer of f on C , then it is also a global one. If, furthermore, f is strictly convex, then the minimizer is uniquely determined. If f is continuous and strongly convex and C is closed, then Argminx2C f .x/ ¤ ;. Proof. Let x  2 C be a local minimizer of f on C . Suppose that there is a point x 0 2 C with f .x 0 / < f .x  /. Let x D .1  /x  C x 0 , where  2 .0; 1/. It is clear that x 2 C , because C is convex. By the convexity of f , we have f .x  /  f .x /  .1  /f .x  / C f .x 0 / < f .x  / for a sufficiently small  > 0. The contradiction shows that f .x/  f .x  / for all x 2 C , i.e., x  is a global minimizer of f on C . Suppose now that f is strictly convex and that f .x 0 / D f .x  / holds for some x 0 ¤ x  . Then for x D 12 x  C 12 x 0 , we obtain 1 1 f .x/ < f .x  / C f .x 0 / D f .x  /  f .x/, 2 2 a contradiction which shows that the minimizer is uniquely determined. Now suppose that f is continuous and strongly convex and that C is closed. Recall that a strongly convex function is coercive. Therefore, Corollary 1.1.53 yields Argminx2C f .x/ ¤ ;. t u The subset C in (1.13) is often given in the form C WD fx 2 H W ci .x/  0; i 2 I g, where ci W H ! R, i 2 I WD f1; 2; : : : ; mg. In this case (1.13) can be written as the following constrained minimization problem:

1.3 Convex Optimization Problems

minimize f .x/ subject to ci .x/  0; i 2 I , x 2 H.

25

(1.14)

The functions ci are called the constraints of problem (1.14). If the functions f; ci ; i 2 I , are convex, then (1.14) is called a convex minimization problem (CMP). If f; ci ; i 2 I , are differentiable, then (1.14) is called a constrained differentiable minimization problem. In order to solve problem (1.13) or (1.14) one can apply suitable methods (algorithms) which generate sequences approximating a solution x  of the problem. These algorithms should be constructed in such a way that the sequences converge (in some sense) to a solution of the problem. Since an algorithm should terminate after a finite number of iterations, we cannot expect in general that the algorithm gives an exact solution. Let "  0. We say that x" 2 C is an "-optimal solution of (1.13) or (1.14) if f .x" /  f .x/ C " for all x 2 C . If f is a convex function and C is a closed convex subset (or ci ; i 2 I , are convex functions), then problem (1.13) (or (1.14)) is called a convex (constrained) minimization problem. For a constrained differentiable minimization problem (1.14) with H D Rn (equipped with the standard inner product) a point .x  ; y  / 2 Rn  Rm satisfying the following system of equalities and inequalities rf .x/ C y > rc.x/ D 0 c.x/  0 y0 > y c.x/ D 0,

(1.15)

where c.x/ D .c1 .x/; c2 .x/; : : : ; cm .x//, is called a Karush–Kuhn–Tucker point or, shortly, KKT-point. If f and ci ; i 2 I , in (1.14) are convex (not necessarily differentiable), then a point .x  ; y  / 2 Rn  Rm satisfying the conditions P 0 2 @f .x/ C i 2I i @ci .x/ c.x/  0 (1.16) y0 y > c.x/ D 0, where y D .1 ; : : : m / 2 Rm , is also called a KKT-point. For convex differentiable minimization problem, (1.15) is, of course, a special case of (1.16). Below we give necessary conditions and sufficient conditions for a point x to be an optimal solution of (1.13) or (1.14). Theorem 1.3.2. A continuous and convex function f W H ! R attains its minimum at a point x 2 H if and only if 0 2 @f .x/. Proof. The theorem follows directly from the definition of the subdifferential.

t u

26

1 Introduction

Corollary 1.3.3. A differentiable convex function f W H ! R attains its minimum at a point x 2 H if and only if Df .x/ D 0. Theorem 1.3.4. Let f W H ! R be a continuous and convex function and C  H be a closed convex subset. The function f jC attains its minimum at a point x 2 C if and only if @f .x/  NC .x/. In particular, a convex differentiable function f attains its minimum at a point x 2 C if and only if Df .x/ 2 NC .x/, i.e., f 0 .x; y  x/ D hDf .x/; y  xi  0 for all y 2 C . Proof. See [209, Chap. VII, Theorem 1.1.1] for finite dimensional case or [325, Sect. 7.1] for an infinite dimensional one. t u Corollary 1.3.5. Let C  H be a closed convex subset and f W H ! R be a differentiable convex function. Then for any > 0 it holds Argmin f .x/ D Fix PC .Id  Df /. x2C

Proof. Let > 0. It follows from Theorem 1.3.4 and from Lemma 1.2.9 that x

2 Argmin f .x/ x2C

” Df .x/ 2 NC .x/ ”  Df .x/ 2 NC .x/ ” x D PC .x  Df .x// ” x 2 Fix PC .Id  Df / which completes the proof.

t u

In the theorem below we assume that H D Rn and that the constraints of the minimization problem (1.14) are regular in some sense, i.e., they satisfy a certain condition called a constraints qualification. If the constraints ci are convex, i D 1; 2; : : : ; m, then one often supposes that they satisfy the Slater constraints qualification, i.e., there exists xN 2 Rn such that ci .x/ N < 0 for all non-affine functions ci . Other constraints qualifications are also considered in the literature. We omit the details and refer the reader to [169, Sect. 9.2], [209, Chap. VII, Sect. 2.2], [38, Sect. 3.3] or [179, Sect. 2.2]. Theorem 1.3.6 (Karush–Kuhn–Tucker). Suppose that the constraints qualification for the convex minimization problem (1.14) is satisfied. If a point x  2 Rm is an optimal solution of (1.14), then there exists y  2 Rm such that .x  ; y  / is a KKT-point. Proof. See [209, Chap. VII, Theorem 2.2.5].

t u

1.3 Convex Optimization Problems

27

Theorem 1.3.7. Let f; ci W Rn ! R, i 2 I be convex. If .x  ; y  / 2 Rn  Rm is a KKT-point of (1.14), then x  is an optimal solution of this problem. Proof. See [169, Theorem 9.1.1] for the differentiable minimization problem or [209, Chap. VII, Theorem 2.1.4] for the convex minimization problem. u t

1.3.2 Variational Inequality Let C  H be closed convex and F W H ! H. The variational inequality problem (VIP.F ; C /) is to find x 2 C such that hF x; y  xi  0

(1.17)

holds for all y 2 C . Note that, by Theorem 1.3.4, the differentiable convex minimization problem (1.13) is a special case of the variational inequality. The proof of the following theorem is similar to the proof of Corollary 1.3.5 and is left to the reader. Theorem 1.3.8. Let > 0. A point x 2 C is a solution of variational inequality (1.17) if and only if x 2 Fix PC .Id  F /. For a general discussion on the existence of solutions of variational inequality problems we refer the reader to [165, 226, 358]. Below we only recall sufficient conditions for the existence and uniqueness of a solution of VIP.F ; C /. Theorem 1.3.9. If F W H ! H is a strongly monotone and Lipschitz continuous operator, then the variational inequality (1.17) has a unique solution x  2 C . Proof. See, e.g., [358, Theorem 46.C].

t u

1.3.3 Common Fixed Point Problem Let a finite T family of operators Ui W H ! H with Fix Ui ¤ ;, i 2 I , be given. If F WD i 2ITFix Ui ¤ ;, then the common fixed point problem (CFPP) is to find a point x 2 i 2I Fix Ui . In order to ensure that Fix Ui are closed and convex, some additional assumptions on the operators Ui are usually made. The details will be explained in Chap. 2.

1.3.4 Convex Feasibility Problem Let a finite family ofTclosed convex subsets Ci , i 2 I , of a Hilbert space H be given. Denote C WD i 2I Ci . If C ¤ ;, then the convex feasibility problem (CFP)

28

1 Introduction

is to find a point x 2 C . We see that the CFP can be formulated as the CFPP with Ui WD PCi , i 2 I . On the other hand, the CFPP is a CFP with Ci D Fix Ui . In applications concerning the CFP the subsets Ci are often called the constraints sets or, shortly, constraints and are given in the form Ci WD fx 2 H W ci .x/  0g;

(1.18)

where ci W H ! R are convex functions, called the constraints functions or, shortly, constraints, i 2 I . The CFP can also be considered without the assumption C ¤ ;. In this case we should define an appropriate convex proximity function f W H ! RC which measures a “distance” to the constraints. The proximity function should have the property f .x/ D 0 if and only if x 2 C . (1.19) The convex feasibility problem can be formulated as a minimization of the proximity function f . The proximity function f W H ! RC for the CFP can be defined by the following general form f .x/ WD F .f1 .x/; : : : ; fm .x//,

(1.20)

where fi W H ! RC are convex functions with the property  fi .x/

D 0 for x 2 Ci > 0 for x … Ci ,

(1.21)

i 2 I WD f1; 2; : : : ; mg, and the function F W Rm C ! R is convex and increasing (or at least nondecreasing) with respect to any variable and such that F .v/ D 0 if and only if v D 0. In particular, if F .v/ D w> v for a vector w D .!1 ; : : : ; !m / 2 Rm CC , the proximity function f has the form of a weighted sum of functions fi , i 2 I , f .x/ WD

X

!i fi .x/.

(1.22)

i 2I

If F .v/ WD maxi 2I i , where v D .1 ; : : : ; m /, the proximity function f has the form f .x/ D max fi .x/. (1.23) i 2I

Two important examples of functions fi with properties (1.21) are fi WD d.; Ci / and fi WD d 2 .; Ci /. We can also write fi .x/ D kPCi x  xk in the first case or fi .x/ D 12 kPCi x  xk2 in the other one. Note that both functions are convex and the other one is differentiable, as well. These facts will be proved in Sect. 2.2.3 (see Lemmas 2.2.27 and 2.2.28 and Corollary 2.2.29). The computation of these two functions requires, however, a simple structure of Ci . If we take

1.3 Convex Optimization Problems

29

fi .x/D 12 kPCi x  xk2 , i 2 I , in (1.22), then we obtain the following important proximity function: 1X f .x/ WD !i kPCi x  xk2 . (1.24) 2 i 2I In Chap. 4 we show that determining a minimizer P of the above function is equivalent to finding a fixed point of the operator T D i 2I !i PCi (see Theorem 4.4.6). When the subset Ci is given by (1.18), the function ciC WD maxf0; ci g expresses a “distance” to the i th constraint, i 2 I . In this case we can define a function fi satisfying (1.21) in the form fi WD hi ı ciC (1.25) for a convex and increasing function hi W RC ! RC such that hi .0/ D 0, i 2 I . The function hi has often the form hi .u/ WD u or hi .u/ WD 12 u2 , i 2 I . In the first case the proximity function f defined by (1.22) is a weighted sum of “distances” to the constraints, i.e., X f .x/ D !i ciC .x/, i 2I

and, in the other one, f has the form f .x/ D

1X !i .ciC .x//2 , 2 i 2I

P where w D .!1 ; : : : ; !m / 2 Rm i 2I !i D 1 CC . One can additionally suppose that which leads to the assumption that w 2 ri m . This assumption does not change the minimization problem. Both proximity functions are convex and the other one is even differentiable. As mentioned above, the CFP is, in general, to minimize a convex proximity function f . Denote f  WD infx2H f .x/. It is clear that f   0. We emphasize that the proximity function f needs not to attain its minimum. If C ¤ ;, we have C D Argminx2H f .x/ for any proximity function with the above described properties. In this case, the problem of determining x 2 C and the problem of minimization of the proximity function f are equivalent. Furthermore, in this case, a significant information about f is available, namely f  WD minx2H f .x/ D 0. One can say that in the general case, i.e., without the assumption C ¤ ;, the CFP is to find an element x  2 H, for which “distances” to the constraints Ci , i 2 I , are minimal in some sense, if such x  exists. For x 2 H, the value f .x/ expresses this “distance”. If C D ; and the proximity function attains its minimum, then, of course, f  > 0. Note, however, that the nonemptiness of C does not follow from the fact that f  WD infx2H f .x/ D 0. In some applications, the CFP is to find an element x 2 C or to state that C D ;. We can also say that the CFP is to find a zero of the proximity function or to state that the proximity function is positive for all arguments.

30

1 Introduction

1.3.5 Linear Feasibility Problem In the case where all Ci are half-spaces, i.e., Ci WD H .ai ; ˇi / D fx 2 H W hai ; xi  ˇi g for ai 2 H, ai ¤ 0 and ˇi 2 R, i 2 I WD f1; 2; : : : ; mg, the CFP is called the linear feasibility problem (LFP). If H D Rn , the linear feasibility problem is to solve a system of linear inequalities ai> x  ˇi , i D 1; 2; : : : ; m,

(1.26)

where ai 2 Rn , ai ¤ 0, ˇi 2 R, i 2 I , which can also be written in matrix form as Ax  b, where A is a matrix with rows ai , i.e., A D Œa1 ; : : : ; am > and b D .ˇ1 ; : : : ; ˇm /. By duality theorems, the linear feasibility problem in Rn is equivalent to the linear programming problem (see, e.g., [4, Chap. 6]). Similarly to the CFP, the linear feasibility problem can also be considered without the assumption C ¤ ;. In this case, the problem is to minimize a proximity function. One can apply general forms of a proximity function given by equalities (1.20)– (1.25). Since the constraints for the LFP have the form ai> xˇi  0 or, equivalently, kai k1 .ai> x  ˇi /  0, as fi one often takes functions fi defined by fi .x/ WD hi ..ai> x  ˇi /C / or by

 fi .x/ WD hi

.ai> x  ˇi /C kai k

(1.27)

 (1.28)

for some convex increasing functions hi W RC ! RC with hi .0/ D 0, i 2 I . In most cases the functions hi are given by hi .u/ WD !i u or hi .u/ WD 12 !i u2 for !i > 0, i 2 I . If we apply these definitions of fi and hi to formulas (1.20)–(1.25) we obtain the following collection of proximity functions: (a)

f .x/ WD max.ai> x  ˇi /C , i 2I

(1.29)

in order to determine a point for which the value of the most violated constraint is minimal, (b) f .x/ WD max i 2I

.ai> x  ˇi /C , kai k

(1.30)

1.3 Convex Optimization Problems

31

in order to determine a point for which the distance to the furthest constraint is minimal, (c) f .x/ WD

X

!i .ai> x  ˇi /C ,

(1.31)

i 2I

in order to determine a point for which the weighted sum of the values of the constraints is minimal, (d) f .x/ WD

X

!i

i 2I

.ai> x  ˇi /C , kai k

(1.32)

in order to determine a point for which the weighted sum of distances to the constraints is minimal, (e) f .x/ WD

1X !i Œ.ai> x  ˇi /C 2 , 2 i 2I

(1.33)

in order to determine a point for which the weighted sum of squared values of the constraints is minimal, or (f) f .x/ WD

 > 2 1X .ai x  ˇi /C !i , 2 i 2I kai k

(1.34)

in order to determine a point for which the weighted sum of squared distances to the constraints is minimal. All of the proximity functions presented herein are convex. The last two functions are even differentiable. If we denote the residuum of the i th constraint at a point x by i .x/, i.e., i .x/ WD ai> x  ˇi , i 2 I , and by r.x/ the residual vector, i.e., r.x/ WD . 1 .x/; : : : ; m .x//, then the proximity function (1.31) has the form f .x/ D w> rC .x/, where w D .!1 ; : : : ; !m / 2 Rm CC , and the proximity function 2 1 (1.33) has the form f .x/ D 2 krC .x/kW for W WD diag w. A system of linear equations Ax D b is a special case of the LFP, because the system can be presented as     A b x . A b In this case the proximity function f given by (1.34) obtains the form 1X f .x/ WD !i 2 i 2I



ai> x  ˇi kai k

2 .

(1.35)

Remark 1.3.10. If we multiply the inequality ai> x  ˇi by a constant ˛i > 0, i 2 I , we obtain an equivalent linear feasibility problem, regardless of the proximity

32

1 Introduction

function under consideration, whenever C ¤ ;. In the case, where C D ;, the new LFP defined by linear inequalities ˛i ai> x  ˛i ˇi ; i 2 I;

(1.36)

is, in general, not equivalent to the original problem, even if we consider the same proximity function as for the original problem. However, there are some relationships between these two problems. The problems are equivalent if we use the proximity functions (1.30), (1.32) or (1.34). If we apply the proximity functions (1.31) or (1.33) with weights !i D ˛i1 i or !i D ˛i2 i , respectively, i 2 I , to problem (1.36), then we obtain a new LFP which is equivalent to the original one with the proximity function (1.31) or (1.33), respectively, and with weights !i D i , i 2 I . In particular, if we take ˛i D kai k1 , i 2 I , in (1.36), then the half-spaces in the new LFP are defined by normalized vectors. Therefore, if we apply the proximity function (1.33) with weights !i D kai k2 i , i 2 I , to the new LFP, then the problem is equivalent to the original one with the proximity function (1.33) and with weights !i D i , i 2 I . Note that we can suppose, without loss of generality, that the vectors ai , i 2 I , are normalized. Otherwise, an appropriate change of the weights leads to an equivalent problem, because the weights !i , i 2 I , can be multiplied by a constant ˛ > 0 without changing the minimizers of the proximity function. Therefore, we can suppose that w 2 ri m instead of w 2 Rm CC . Example 1.3.11. (Linear least squares problem) Given a linear system Ax D b, where A is a matrix of type m  n, x 2 Rn and b 2 Rm , the problem minimize 12 kAx  bk2 subject to x 2 Rn

(1.37)

is called a linear least squares problem (LLSP). This problem is equivalent to the following compatible system of linear equations A> Ax D A> b

(1.38)

O where x N D AC b, Actually, any solution of (1.38) has the form x D x N C x, C xO 2 ker A and A denotes the Moore–Penrose pseudoinverse of A. System (1.38) is called a normal equation. The equivalence of (1.37) and (1.38) follows from the necessary and sufficient condition for convex minimization (see Corollary 1.3.3). In particular, any solution of a compatible system Ax D b is a solution of the LLSP. In general, one can solve system (1.38) by so called SVD-decomposition of the matrix A> A (for details see, e.g., in [181, 234, 317]). We show that a solution of (1.37) (or, equivalently, (1.38)) with a minimal norm has the form x WD x N D AC b. This solution is called the normal solution of problem (1.37). Consider the following differentiable convex constrained minimization problem

1.3 Convex Optimization Problems

minimize 12 kxk2 subject to A> .Ax  b/ D 0 x 2 Rn .

33

(1.39)

It follows from the definition of the metric projection that a solution x  of (1.39) is equal to PL 0, where L WD fx 2 Rn W A> Ax D A> bg. The uniqueness of the metric projection yields the uniqueness of x  . The Lagrange function L W Rn  Rn ! R has the form 1 L.x; y/ D kxk2 C y > A> .Ax  b/ 2 and the KKT-system has the form rx L.x; y/ D x C A> Ay D 0 ry L.x; y/ D A> .Ax  b/ D 0.

(1.40)

By the KKT-theorem, this system has a solution .x; y/ 2 Rn  Rn . It follows from the definition of the Moore–Penrose pseudoinverse that A> AAC b D A> .AAC /> b D .AAC A/> b D A> b. Therefore, x N WD AC b satisfies the second equation of the KKT-system (1.40). By sufficient optimality conditions for the differentiable convex minimization problem (see Theorem 1.3.7), x N is a solution of (1.39). Therefore, x  D x N . Subsuming, x N WD AC b is the unique solution of (1.37). Note that the LLSP is a special case of the LFP with the proximity function defined by (1.35).

1.3.6 General Convex Feasibility Problem In many applications of the convex feasibility problems one seeks a point, for which some constraints are satisfied and a proximity function defined for the rest of constraints is minimal. The constraints which should be satisfied are called hard constraints and the rest of constraints are called soft constraints. We call the problem a general convex feasibility problem (GCFP). In other words, the GCFP is to find a point satisfying the hard constraints for which “distances” to soft constraints are minimal in some sense. Such model was described in details in [123]. Let fCi gi 2I be a family of closed convex subsets. Let Ci for i 2 Ih be the hard constraints and Ci for i 2 Is beTthe soft constraints,Twhere I D Ih [ Is and Ih \ Is D ;. Furthermore, let Ch WD i 2Ih Ci and Cs WD i 2Is Ci . If Ch \ Cs ¤ ;, then GCFP is to find x  2 Ch \Cs . If we do not suppose Ch \Cs ¤ ;, then, similarly as for the CFP, one defines a convex proximity function f W Ch ! RC , defined on the subset Ch of hard constraints, which measures “distances” to the soft constraints. Similarly as for the CFP, we require the proximity function f to have the property

34

1 Introduction

Fig. 1.7 Split feasibility problem

H1

A

H2

A−1 (Q)

Ax

x C

Q

f .x/ D 0 if and only if x 2 Ch \ Cs . As a proximity function one can use functions described in Sect. 1.3.4, e.g., the function f W H ! RC defined by (1.24) with I replaced by Is .

1.3.7 Split Feasibility Problem In some applications the general convex feasibility problem has the form: Given closed convex subsets C  H1 , Q  H2 of Hilbert spaces H1 ; H2 and a bounded linear operator A W H1 ! H2 , find a point x 2 C such that Ax 2 Q (if such a point exists) (see Fig. 1.7). If H1 ; H2 are Euclidean spaces, then the above problem has the form: Given closed convex subsets C  Rn , Q  Rm and a matrix A of type m  n, find a point x 2 C such that Ax 2 Q (if such a point exists). The subset C is a hard constraint and the subset D WD A1 .Q/ D fx 2 H1 W Ax 2 Qg is a soft constraint. It is clear that D is closed convex, because for a bounded linear operator A, the inverse image of a closed convex subset Q is closed and convex. This problem was introduced by Censor and Elfving [88] and was called a split feasibility problem (SFP). If we do not suppose that C \ D ¤ ;, then, similarly as for the GCFP, we introduce a convex proximity function f W C ! RC and we present the SFP as minimization of f . Byrne [55, Sect. 2] has proposed a proximity function defined by f .x/ WD

 1 PQ .Ax/  Ax 2 . 2

(1.41)

The function f is convex as a composition A and a convex  of a linear operator  function 12 d 2 .; Q/. Note that d.Ax; Q/ D PQ .Ax/  Ax . The function defined by (1.41) measures the square distance of the image of a point x 2 C by the operator A to the subset Q. The function f has the required property: x 2 C \ D , f .x/ D 0. Practical applications of the SFP require a simple structure of C and Q, which allows an easy computation of PC .x/ and PQ .y/ for any x 2 H1 and y 2 H2 .

1.3 Convex Optimization Problems

35

1.3.8 Linear Split Feasibility Problem Now consider the SFP in the finite dimensional case. If the subset Q  Rm has the form Q WD fy 2 Rm W y  bg, (1.42) where b 2 Rm , then the problem is called a linear split feasibility problem (LSFP). In other words, the problem can be written as follows: find x 2 C such that Ax  b (if such a point exists). In general, we do not suppose, however, that C \ fx 2 Rn W Ax  bg ¤ ;. Now we present a few examples of a proximity function for the LSFP which are often used in applications. In Sect. 4 we will show that for the subset Q defined by (1.42) we have PQ .Ax/  Ax D b  Ax. (see Example 4.1.4). Therefore, the proximity function f W C ! R defined by (1.41), which is applied to the LSFP can be written in the form 1X > 1 Œ.a x  ˇi /C 2 . f .x/ WD krC .x/k2 D 2 2 i D1 i m

Note that f is differentiable. Alternatively, one can also apply a proximity function given by m X f .x/ WD w> rC .x/ D !i .ai> x  ˇi /C i D1

or by

1X D !i Œ.ai> x  ˇi /C 2 , 2 i D1 m

f .x/ WD

krC .x/k2W

(1.43)

where W WD diag w for w D .!1 ; : : : ; !m / 2 Rm CC . The latter proximity function is used in many applications.

1.3.9 Multiple-Sets Split Feasibility Problem The following problem, called a multiple-sets split feasibility problem, has been studied in the literature (see, e.g., [93, 103, 257, 344]): Given closed convex subsets Ci  H1 , i D 1; 2; : : : ; p, Qj  H2 , j D 1; 2; : : : ; r, of Hilbert spaces H1 ; H2 , respectively, and a bounded linear T Tr operator A W H1 ! H2 , find a point x 2 C WD p C such that Ax 2 Q WD j D1 Qj . i D1 i This problem can be considered as a split feasibility problem with the proximity function given by (1.41). The application of this proximity function requires, however, a simple structure of C and Q, which allows an easy computation of PC x

36

1 Introduction

and PQ y for any x 2 Rn and y 2 Rm . If the structure of Ci , i D 1; 2; : : : ; p, and Qj , j D 1; 2; : : : ; r, is simple, one can also use the following proximity function f W Rn ! R proposed in [93]: f .x/ WD

p r 2 1X 1X  ˇj PQj Ax  Ax  . kPCi x  xk2 C 2 i D1 2 j D1

Note that in this case the problem is a special case of the general convex feasibility problem, where all subsets Ci , i D 1; 2; : : : ; p, and Qj , j D 1; 2; : : : ; r, are treated as soft constraints. If Ch has a simple structure allowing an easy computation of PCh x for any x 2 Rn , one can also use the proximity function f W Ch ! R given by r 2 1X  f .x/ WD ˇj PQj Ax  Ax  . 2 j D1 In this case the problem is a special case of the general convex feasibility problem, where Ci , i D 1; 2; : : : ; p, are hard constraints and Qj , j D 1; 2; : : : ; r, are soft constraints.

1.4 Exercises Exercise 1.4.1. Prove that in a Hilbert space the Cauchy–Schwarz inequality and the triangle inequality are equivalent. Exercise 1.4.2. Let w D .!1 ; !2 ; : : : ; !m / 2 m . Prove that  m 2 m m X  X  2 1 X   !i xi  D !i kxi k2  !i !j xi  xj  .    2 i;j D1 i D1 i D1

(1.44)

Exercise 1.4.3. Prove the parallelogram law (see equality (1.8)). Exercise 1.4.4. Prove Theorem 1.1.1. Exercise 1.4.5. Let G be a positive definite matrix of type m  m. Prove that the function h; iG defined by hw; viG WD w> Gv is an inner product in Rm . Exercise 1.4.6. Let H D l2 and ek D .ek1 ; ek2 ; : : :/ with  1 if j D k ekj D 0 if j ¤ k, j; k  0. Prove that fek g1 kD1 converges weakly in l2 but does not converge strongly.

1.4 Exercises

37

Exercise 1.4.7. Let f W H ! R be a convex function. Prove that: (a) The sublevel set S.f; ˛/ is convex for any ˛ 2 R. (b) The subset Argminx2H f .x/ is convex. Exercise 1.4.8. Prove that the epigraph of a function f W H ! R is a convex subset if and only if f is convex. Exercise 1.4.9. Let A W H1 ! H2 be an affine operator, C  H1 and D  H2 be convex subsets. Show that A.C / and A1 .D/  H1 are convex. Exercise 1.4.10. Evaluate derivatives of the following functions f W H ! R: (a) f .x/ D ha; xi, (b) f .x/ D 12 hAx; xi, where A W H ! H is a bounded linear operator. Exercise 1.4.11. Evaluate derivatives of the following functions f W H1 ! R: (a) f .x/ D 12 kAx  bk2 , where A W H1 ! H2 is a bounded linear operator and b 2 H2 ,  2 (b) f .x/ D 12 PQ .Ax/  Ax  , where A W H1 ! H2 is a bounded linear operator and Q  H2 is a closed convex subset. Exercise 1.4.12. Let A be an m  n matrix. Prove that the matrix AA> is positive semi-definite and that AA> is positive definite if and only if A has full row rank, i.e., the rows of A are linearly independent. Exercise 1.4.13. Let A be a positive definite matrix of type n  n and min .A/ denote the smallest eigenvalue of A. Prove that: (a) 12 x > Ax  min .A/ kxk2 for any x 2 Rn , (b) The function f W Rn ! R defined by f .x/ WD 12 x > Ax is convex. Exercise 1.4.14. Let C  H be a polytope, i.e., C D fx 2 H W hai ; xi  ˇi , i 2 I g and let x 2 C . Show that NC x D conefai W i 2 I.x/g, where I.x/ D fi 2 I W hai ; xi D ˇi g. Exercise 1.4.15. Let m D fu 2 Rm W u  0; e > u D 1g denote a standard simplex in Rn . Show that ri m D fu 2 Rm W u > 0; e > u D 1g. Exercise 1.4.16. Let C  H be a nonempty convex subset. Show that the distance function d.; C / W H ! R defined by d.x; C / WD infy2C kx  yk is convex. Exercise 1.4.17. Prove Lemma 1.2.5. Exercise 1.4.18. Let f W H ! R be a differentiable convex function. Show that the function h W H ! RC defined by h.x/ D ..f .x//C /2 is differentiable and convex.

38

1 Introduction

Exercise 1.4.19. Let fi W X ! R, i D 1; 2; : : : ; m, be convex functions and F W Rm ! R be a convex function which is nondecreasing with respect to any coordinate. Show that the function f WD F .f1 ; : : : ; fm / is convex. Exercise 1.4.20. Let A be a positive definite matrix of type n  n. Show that the function f W Rn ! R, f .x/ D 12 x > Ax is strongly convex. Exercise 1.4.21. Consider the following problem: Find a point x 2 RnC for which the sum of squares of distances to the balls Ci WD B.ai ; i /, where ai 2 Rn , i  0, i D 1; 2; : : : ; m, is minimal. Present the problem as a convex feasibility problem and as a generalized convex feasibility problem, where RnC is a subset of hard constraints. Write corresponding proximity functions. Exercise 1.4.22. Prove Lemma 1.2.6. Exercise 1.4.23. Prove Lemma 1.2.8. Exercise 1.4.24. Prove Theorem 1.3.8.

Chapter 2

Algorithmic Operators

In Chap. 5 we will present several methods for solving convex optimization problems. We will focus our study on iterative methods (we also call them iterative procedures or algorithms) which are given in the form of the following recurrence x kC1 D Tk x k

(2.1)

defined on a closed convex subset X  H, where Tk W X ! X is a sequence of operators. We suppose that the starting point x 0 is an element of a starting subset X0  X . Usually, one supposes that X0 D X . A sequence fx k g1 kD0 generated by the iterative method (2.1) is called an approximating sequence. If Tk D T for all k  0, then this sequence is called an orbit of T . Any iterative method for solving a convex optimization problem is constructed in such a way that the approximating sequences fx k g1 kD0 generated by this method converge (at least weakly) to a solution of the optimization problem. As we will see, the solution is a fixed point of an operator S W X ! H, which is usually a nonexpansive one. The form of this operator depends on the considered optimization problem. A sequence of operators Tk whichTdefines the iterative method is usually constructed in such a way that Fix S  1 kD0 Fix Tk . In this chapter we deal with general properties of operators which define algorithms for solving convex optimization problems. In one iteration of the algorithm an appropriate operator T W X ! X defines an actualization, also called an update x C of the current approximation x of a solution of the convex optimization problem. Usually, this actualization has the form x C D T x. We call T an algorithmic operator. One can also consider algorithms, where the actualization has the form x C 2 T x for a mapping (multifunction) T W X  X . In this case, T is called an algorithmic mapping. Operators defining iterations of an algorithm usually depend on some parameters which are constant or vary during the iteration process. The properties of approximating sequences depend on the properties of algorithmic operators defining the iterative method as well as on the choice of parameters defining these operators. A. Cegielski, Iterative Methods for Fixed Point Problems in Hilbert Spaces, Lecture Notes in Mathematics 2057, DOI 10.1007/978-3-642-30901-4 2, © Springer-Verlag Berlin Heidelberg 2012

39

40

2 Algorithmic Operators

2.1 Basic Definitions and Properties Let H be a Hilbert space. In what follows, we consider operators which are defined on a nonempty closed convex subset X  H. P Remark 2.1.1. Let Ui W X ! X , i 2 I WD f1; 2; : : : ; mg. If (i) U D i 2I !i Ui , where w D .!1 ; !2 ; : : : ; !m / 2 m , or (ii) U D Um Um1 : : : U1 , then the following obvious inclusion holds \ Fix Ui  Fix U i 2I

The converse inclusion needs not to be true even if all Ui , i 2 I , have a common fixed point (see Example 2.1.27). Definition 2.1.2. Let T W X ! H and  2 Œ0; 2. The operator T W X ! H defined by T WD .1  / Id CT is called a -relaxation or, shortly, relaxation of the operator T . Obviously, T D Id C.T  Id/. We call  a relaxation parameter. If  2 .0; 1/, then T is called an under-relaxation of T . If  2 .1; 2/, then T is called an over-relaxation of T and if  D 2, then T is called the reflection of T . If  2 .0; 2/, then T is called a strict relaxation of T . A relaxation T of an operator T can be defined for any  2 R. However, if we do not extend explicitly the range of , we assume that  2 Œ0; 2. Remark 2.1.3. Note that the equality .T / D T holds for all ;  2 R, consequently .T /1 D T for  ¤ 0. Remark 2.1.4. It is clear that Fix T D Fix T whenever  ¤ 0. Let Ui W X ! X , i 2 I WD f1; 2; : : : ; mg, U WD Um Um1 : : : U1 and Qi WD Ui Ui 1 : : : U1 Um : : : Ui C1 , i D 1; 2; : : : ; m. Denote Q0 WD Qm D U and U0 WD Um . There exists a relationship among the subsets of fixed points of operators Qi , which is expressed by the following theorem. Theorem 2.1.5. For i D 1; 2; : : : ; m there holds Fix Qi D Ui .Fix Qi 1 /.

(2.2)

Proof. Let i 2 I . First we prove the inclusion Fix Qi  Ui .Fix Qi 1 /. Let zi 1 2 Fix Qi 1 and zi D Ui zi 1 . Then we have zi D Ui zi 1 D Ui Qi 1 zi 1 D Ui Ui 1 : : : U1 Um : : : Ui C1 Ui zi 1 D Ui Ui 1 : : : U1 Um : : : Ui C1 zi D Qi zi ,

(2.3)

2.1 Basic Definitions and Properties

41

which proves that (2.3) holds for any i 2 I . Consequently, Fix Qi  Ui .Fix Qi 1 /  Ui Ui 1 .Fix Qi 2 /  : : :  Ui Ui 1 : : : U1 .Fix Q0 / D Ui Ui 1 : : : U1 .Fix Qm /  Ui Ui 1 : : : U1 Um .Fix Qm1 /  : : :  Ui Ui 1 : : : U1 Um Um1 : : : Ui C1 .Fix Qi / D Qi .Fix Qi / D .Fix Qi /, i 2 I , and all inclusions are, actually, equations. In particular, Fix Qi D Ui .Fix Qi 1 /, i.e., (2.2) is satisfied for all i 2 I . t u

2.1.1 Nonexpansive Operators Definition 2.1.6. We say that an operator T W X ! H is: (i) Nonexpansive (NE), if kT x  T yk  kx  yk for all x; y 2 X , (ii) Strictly nonexpansive if kT x  T yk < kx  yk or x  y D T x  T y for all x; y 2 X , (iii) An ˛-contraction, where ˛ 2 .0; 1/ or, shortly, a contraction if kT x  T yk  ˛ kx  yk for all x; y 2 X . The theorem below, called the Banach fixed point theorem or the Banach theorem on contractions, is widely applied in various areas of mathematics. The theorem holds for any complete metric space, and hence, in particular, for every closed subset of a Hilbert space. Theorem 2.1.7 (Banach, 1922). Let X be a complete metric space and T W X ! X be a contraction. Then T has exactly one fixed point x  2 X . Furthermore,  for any x 2 X , the orbit fT k xg1 kD0 converges to x with a rate of geometric progression. Proof. See, e.g., original paper of Banach [15], [185, Theorem 1.1], [267, Theorem 24.2], [184, Theorem 2.1], [183, Theorem 2.1] or [36, Theorem 2.1]. t u

42

2 Algorithmic Operators

The Banach fixed point theorem is a widely applied tool for an iterative approximation of fixed points. Unfortunately, its application is restricted to contractions. We will need, however, appropriate tools for an iterative approximation of fixed points of nonexpansive operators T with Fix T ¤ ;. Below, we present several classical fixed points theorems. Theorem 2.1.8 (Brouwer, 1912). Let X  Rn be nonempty compact and convex and T W X ! X be continuous. Then T has a fixed point. Proof. See, e.g., original paper of Brouwer [43] or [191, Chap. II, 5, Theorem 7.2] or [183, Theorem 7.6]. t u The Brouwer fixed point theorem was generalized by Juliusz Schauder. Theorem 2.1.9 (Schauder, 1930). Let X be a nonempty compact and convex subset of a Banach space and T W X ! X be continuous. Then T has a fixed point. Proof. See, e.g., original paper of Schauder [302] or [191, Chap. II, 6, Theorem 3.2] or [183, Theorem 8.1]. t u For nonexpansive operators in a Hilbert space H the compactness of X  H in the Schauder theorem can be replaced by the boundedness of X . The following theorem was proved independently by Browder [45, Theorem 1], G¨ohde [188] and by Kirk [227]. The proof can also be found, e.g., in [185, Theorem 5.1], [191, Chap. I, 4, Theorem 1.3], [183, Theorem 4.1] or [36, Theorem 3.1]. Theorem 2.1.10 (Browder–G¨ohde–Kirk, 1965). Let X be a nonempty closed, convex and bounded subset of a uniformly convex Banach space (e.g., of a Hilbert space H) and U W X ! X be nonexpansive. Then U has a fixed point. Contrary to the Banach fixed point theorem, the theorems of Brouwer, Schauder and of Browder–G¨ohde–Kirk are only of existential nature. In Chap. 3 we present theorems which can be applied to iterative methods for determining fixed points of nonexpansive operators. Below, we present some properties of the subset of fixed points of a nonexpansive operator. The following result can be found in [185, Proposition 5.3]. Proposition 2.1.11. The subset of fixed points of a nonexpansive operator T W X ! H is closed and convex. Proof. (cf. [185, Proposition 5.3]) Let x k 2 Fix T and x k ! x. We have x 2 X because X is closed. By the continuity of T , x D lim x k D lim T x k D T x, k

i.e., Fix T is a closed subset. Now we show the convexity of Fix T . Let x; y 2 Fix T , x ¤ y and z D .1  /x C y for  2 .0; 1/. By the nonexpansivity of T and by the positive homogeneity of the norm we have kx  T zk D kT x  T zk  kx  zk D  kx  yk

(2.4)

2.1 Basic Definitions and Properties

43

and kT z  yk D kT z  T yk  kz  yk D .1  / kx  yk .

(2.5)

Now, the triangle inequality yields kx  yk  kx  T zk C kT z  yk   kx  yk C .1  / kx  yk D kx  yk . Consequently, kx  yk D kx  T zk C kT z  yk . By the strict convexity of the norm, the vectors x  T z and T z  y are positive linearly dependent. Therefore, ˛.x  T z/ C ˇ.y  T z/ D 0 for some ˛; ˇ  0. ˇ ˛ Since x ¤ y, it follows that ˛ C ˇ > 0, and hence, T z D ˛Cˇ x C ˛Cˇ y. Now, the nonexpansivity of T and inequalities (2.4) and (2.5) yield ˇ kx  yk D kx  T zk D kT x  T zk  kx  zk D  kx  yk ˛Cˇ

(2.6)

and ˛ kx  yk D kT z  yk D kT z  T yk  kz  yk D .1  / kx  yk . (2.7) ˛Cˇ If at least one inequality in (2.6) and (2.7) is strict, then by summing up (2.6) ˇ ˛ D  and ˛Cˇ D .1  /, and (2.7) we would obtain a contradiction. Therefore, ˛Cˇ consequently T z D .1  /x C y D z. t u The closedness and convexity of the subset of fixed points of a nonexpansive operator follows also from a property presented in Sect. 2.2 (see Corollary 2.2.48). Lemma 2.1.12. Let Si W X ! X , i 2 I WD f1; 2; : : : ; mg, be nonexpansive. Then: P (i) A convex combination S WD i 2I !i Si , where w D .!1 ; : : : ; !m / 2 m , is nonexpansive. If, furthermore, at least one operator Si is a contraction and the corresponding weight !i > 0, then S is a contraction; (ii) A composition S WD Sm Sm1 : : : S1 is nonexpansive. If, furthermore, at least one operator Si is a contraction, then S is a contraction. Proof. Let x; y 2 X and Si be nonexpansive, i.e., kSi x  Si yk  ˛i kx  yk, where ˛i 2 .0; 1, i 2 I . P P (i) Let w 2 m , S WD i 2I !i Si and ˛ D j 2I !j ˛j . It is clear that ˛ 2 .0; 1. By the convexity of the norm and the nonexpansivity of Si , i 2 I , we have

44

2 Algorithmic Operators

   X   !i .Si x  Si y/ kS x  Syk D    i 2I X  !i kSi x  Si yk i 2I



X

!i ˛i kx  yk

i 2I

D

X

P

i 2I

!i ˛i ˛ kx  yk j 2I !j ˛j

D ˛ kx  yk , i.e., S is a nonexpansive operator. Now suppose that Si0 is a contraction, i.e., ˛i0 < 1 and that !i0 > 0, for some i0 2 I . Then ˛ 2 .0; 1/, i.e., S is a contraction. (ii) We have kS x  Syk D kSm Sm1 : : : S1 x  Sm Sm1 : : : S1 yk  ˛ kx  yk , where ˛ D ˛m ˛m1 : : : ˛1 2 .0; 1. If Si0 is a contraction for some i0 2 I , i.e., ˛i0 2 .0; 1/, then, of course, ˛ 2 .0; 1/ and S is a contraction. t u Theorem 2.1.13. Let Ui W X ! X be nonexpansive for all i 2 I WD f1; 2; : : : ; mg, and U WD Um Um1 : : : U1 . If Uj .X / is bounded for at least one j 2 I , then Fix U ¤ ;. Proof. Let Uj .X / be bounded for some j 2 I . Since Ui are nonexpansive, i 2 I; the boundedness of Uj .X / yields the boundedness of U.X /. Therefore, Y WD cl conv U.X / is closed, convex and bounded. Since U.X /  X and X is closed and convex, we have Y  X . The operator U jY maps a closed, convex and bounded subset Y into itself. By the Browder–G¨ohde–Kirk Fixed Point Theorem, the operator U jY has a fixed point z 2 Y . Of course, U z D U jY .z/ D z. t u Theorem 2.1.14. Let Ui W X ! H, i 2 I WD P f1; 2; : : : ; mg, be nonexpansive operators with a common fixed point and U WD i 2I !i Ui with w 2 ri m . Then Fix U D

\

Fix Ui .

i 2I

T 2.1.1). Now Proof. The inclusion i 2I Fix Ui  Fix U is always true (see RemarkT we show that the converse inclusion also holds. Let z 2 Fix U and u 2 i 2I Fix Ui . T If z D u, then, of course, z 2 i 2I Fix Ui . Otherwise, for z ¤ u, by the convexity of the norm and by the nonexpansivity of Ui , i 2 I , we have

2.1 Basic Definitions and Properties

45

kz  uk D kU z  uk     X  X      D !i Ui z  u D  !i .Ui z  u/     i 2I i 2I X X  !i kUi z  uk D !i kUi z  Ui uk i 2I



X

i 2I

!i kz  uk D kz  uk .

i 2I

Consequently,   X  X X   !i .Ui z  u/ D !i kUi z  uk D !i kz  uk .    i 2I

i 2I

(2.8)

i 2I

Since !i > 0 for all i 2 I , the first equality in (2.8) yields a positive linear dependence of all pairs of vectors Ui z  u and Uj z  u, i; j 2 I , i ¤ j , i.e.,   kUi z  uk .Uj z  u/ D Uj z  u .Ui z  u/.

(2.9)

The second equality in (2.8), together with the inequality kUi z  uk  kz  uk, i 2 I , and the assumption !i > 0, i 2 I , yield kUi z  uk D kz  uk

(2.10)

for all i 2 I . Since z ¤ u, we have Ui z ¤ u, i 2 I . Now, it follows from (2.9) and (2.10) that Ui z D v for all i 2 I and for some v 2 H. Consequently, z D Uz D

X

!j Uj z D

j 2I

for all i 2 I , i.e., z 2

T i 2I

X

!j v D v D Ui z,

j 2I

Fix Ui .

2.1.2 Quasi-nonexpansive Operators Definition 2.1.15. We say that an operator T W X ! H is: (i) Fej´er monotone (FM) with respect to a nonempty subset C  X if kT x  zk  kx  zk for all x 2 X and z 2 C ,

t u

46

2 Algorithmic Operators

Fig. 2.1 Equivalence (2.11) x

z

x+ y 2 y

(ii) Strictly Fej´er monotone with respect to a nonempty subset C  X if kT x  zk < kx  zk for all x … C and z 2 C . Remark 2.1.16. Because of the following obvious equivalence   yCx ;y  x  0 kz  yk  kz  xk ” z  2

(2.11)

for arbitrary x; y; z 2 H (see Fig. 2.1), an operator T W X ! H is Fej´er monotone with respect to C if and only if   Tx x z ; T x  x  0. (2.12) 2 Furthermore, T is strictly Fej´er monotone if and only if inequality (2.12) is strict for all x … C . We have not supposed that C is closed convex in Definition 2.1.15. Inequality (2.12) yields, however, that if T is (strictly) Fej´er monotone with respect to C , then T is (strictly) Fej´er monotone with respect to conv C . Furthermore, the continuity of the norm yields that if T is Fej´er monotone with respect to C , then T is Fej´er monotone with respect to cl C . Therefore, we can suppose, without loss of generality, that C is closed convex in Definition 2.1.15 (i) and that C is convex in Definition 2.1.15 (ii). There exists the largest subset, with respect to which an operator T is Fej´er monotone. This subset is closed and convex, as follows from the following lemma. Lemma 2.1.17. Let T W X ! H. If the subset    \ Tx Cx Fej T WD ;Tx  x  0 z2X W z 2 x2X

(2.13)

is nonempty, then Fej T is the largest subset, with respect to which T is Fej´er monotone.

2.1 Basic Definitions and Properties

47

Proof. The assertion follows directly from the equivalence (2.11).

t u

Remark 2.1.18. Because of frequent use we state some obvious properties of Fej´er monotone operators: (i) If T is (strictly) Fej´er monotone with respect to a nonempty subset C  H, then for an arbitrary  2 .0; 1/ its relaxation T is also (strictly) Fej´er monotone with respect to C . (ii) If T is (strictly) Fej´er monotone with respect to a nonempty subset C  H, then T is (strictly) Fej´er monotone with respect to any nonempty subset D  C. (iii) Every composition and every convex combination of operators which are Fej´er monotone with respect to a nonempty subset C  H is Fej´er monotone with respect C . Definition 2.1.19. We say that an operator T W X ! H having a fixed point is: (i) Quasi-nonexpansive (QNE) if T is Fej´er monotone with respect to Fix T , i.e., kT x  zk  kx  zk for all x 2 X and z 2 Fix T , (ii) Strictly quasi-nonexpansive (sQNE) if T is strictly Fej´er monotone with respect to Fix T , i.e., kT x  zk < kx  zk for all x … Fix T and z 2 Fix T , (iii) C -strictly quasi-nonexpansive (C -sQNE), where C ¤ ; and C  Fix T , if T is quasi-nonexpansive and kT x  zk < kx  zk for all x … Fix T and z 2 C . For an operator T having a fixed point the following relation is clear: T is sQNE H) T is C -sQNE where C  Fix T . Furthermore, by definition, T is Fix T -sQNE H) T is sQNE. The metric projection onto a closed convex subset is a typical example of a strictly quasi-nonexpansive operator. A nonexpansive and strictly Fej´er monotone operator is also called attracting (see [22, Definition 2.1]). Yamada and Ogura use the name an attracting quasinonexpansive operator for a strictly quasi-nonexpansive one (see [346, page 623]). Vasin and Ageev call these operators strongly Q-quasi-nonexpansive

48

2 Algorithmic Operators

(see [333, Definition 2.2]). Reich and Zaslavski define a more general operator than the strictly quasi-nonexpansive one and call it an F -attracting mapping, where F D Fix T (see [297, Sect. 1]). A continuous strictly quasi-nonexpansive operator is also called a paracontraction (see, [164, Definition 1]). The class of quasi-nonexpansive operators is denoted in [126, page 161] by F 0 . Properties of quasi-nonexpansive operators in metric spaces have been intensively studied since 1969 (see, e.g., [145, 148, 283], [50, Sect. 1], [113]), but the name quasinonexpansive was introduced by Dotson [147]. Lemma 2.1.20. A nonexpansive operator U W X ! H with a fixed point is quasinonexpansive. Proof. Let U be nonexpansive and z 2 Fix U . Then kUx  zk D kUx  U zk  kx  zk , t u

i.e., U is quasi-nonexpansive.

It is clear that the class of nonexpansive operators having a fixed point is an essential subclass of quasi-nonexpansive operators, because a quasi-nonexpansive operator needs not to be continuous. Moreover, a quasi-nonexpansive operator needs not to be nonexpansive even if it is continuous (see Exercise 2.5.2). In this section we present properties of the family of quasi-nonexpansive operators. In further parts of the book we show that these operators play an important role in iterative methods for fixed point problems. The following lemma gives a relation between the subset Fej T and the subset Fix T for an operator T W X ! H (cf. [24, Proposition 2.6 (ii)]). Lemma 2.1.21. For any operator T W X ! H the inclusion Fej T  Fix T holds. If Fix T ¤ ; and T is quasi-nonexpansive, then the converse inclusion also holds. Consequently, the subset of fixed points of a quasi-nonexpansive operator is closed and convex. Proof. If Fej T D ;, then the first part of the assertion is obvious. Now let Fej T ¤ ; and w 2 Fej T . Then, for z D x D w in (2.13), we obtain 0  hw 

TwCw ; T w  wi 2

1 D  kT w  wk2  0, 2 i.e., T w D w. Therefore, Fej T  Fix T . Now suppose that Fix T ¤ ; and that T is quasi-nonexpansive, i.e., T is Fej´er monotone with respect to Fix T . Then, Lemma 2.1.17 yields the inclusion Fix T  Fej T , which together with the first part of the lemma gives Fix T D Fej T . The convexity and the closedness of Fix T follows now from Lemma 2.1.17 and from the fact that the intersection of closed half-spaces is closed and convex. t u

2.1 Basic Definitions and Properties

49

Fig. 2.2 Nonconvex Fix T for a Fej´er monotone operator T

y h

x = Tx

»2 a

Ty

»1

b

Fix T

−h

Remark 2.1.22. It follows from Remark 2.1.18 (ii), Lemmas 2.1.17 and 2.1.21 that a quasi-nonexpansive operator T W X ! X is Fej´er monotone with respect to any nonempty subset of Fix T . Therefore, we will restrict our further consideration of Fej´er monotone operators to quasi-nonexpansive ones. Note, however, that without the quasi nonexpansivity of T the equality Fix T D Fej T needs not to be true. In this case, Fix T needs not to be convex, even if T is Fej´er monotone. Example 2.1.23. Let H D R2 , X WD Œa; b  R for 1  a  b  C1 and h W X ! RC be a function with infx2Œa;b h.x/ D 0. Define the operator T W X ! R2 by  x if j2 j  h.1 / T x WD .1 ; 0/ if j2 j > h.1 /, where x D .1 ; 2 / (see Fig. 2.2). The reader may check that Fej T D Œa; b  f0g and that Fix T D fx 2 X W j2 j  h.1 /g. If h is positive in at least one point, then Fej T ¤ Fix T . If, moreover, h is not concave, then Fix T is not convex. Let Ui W X ! X , i 2 I WD f1; 2; : : : ; mg, and: P (i) U WD i 2I !i Ui , where w D .!1 ; !2 ; : : : ; !m / 2 m or (ii) U WD Um Um1 : : : U1 . As we observed before, the following inclusion holds \

Fix Ui  Fix U

(2.14)

i 2I

(see Remark 2.1.1) and the converse inclusion holds in case (i) when all Ui , i 2 I , are nonexpansive operators with a common fixed point and w 2 ri m (see Theorem 2.1.14). It turns out that, in both cases (i) and (ii), the inclusion converse to (2.14)) is true for strictly quasi-nonexpansive operators (see [22, Proposition 2.12], where the property was formulated for attracting operators). P In case (i) this property is also true for a more general form of the operator U D i 2I !i U , where the weights !i , i 2 I , may depend on x.

50

2 Algorithmic Operators

Definition 2.1.24. A function w W X ! m , with w.x/ D .!1 .x/; : : : ; !m .x// is called a weight function. Definition 2.1.25. Let Ui W X ! H, i 2 I . We say that the weight function w W X ! m is appropriate with respect to the family fUi gi 2I or, shortly, appropriate, T if for any x … i 2I Fix Ui there exists an index j 2 I such that !j .x/ k Uj x  x k ¤ 0.

(2.15)

I.x/ WD fi 2 I W x … Fix Ui g

(2.16)

Denote for a family of operators Ui W X ! H, i 2 I . The subset I.x/ is called a subset of violated constraints. Note that w is appropriate if and only if wj .x/ > 0 for some j 2 I.x/

(2.17)

T and for any x … i 2I Fix Ui (or, equivalently, for any x 2 H such that I.x/ ¤ ;). A weight function w W X ! ri m is appropriate with respect to any family of operators fUi gi 2I if: (i) w 2 ri m is a vector of constant weights (this case was considered in [22, Proposition 2.12]), or if (ii) wi .x/ > 0 for all x … Fix Ui and for all i 2 I . It is clear that property (2.15) is weaker than conditions (i) and (ii) above. The following T theorem extends important results of [22, Proposition 2.12], where C D i 2I Fix Ui and only constant weights are considered (see also [25, Proposition 2.5] for a related result). These extended results will be applied in further parts of the book. T Theorem 2.1.26. Let the operators Ui W X ! TX , i 2 I , with i 2I Fix Ui ¤ ;, be C -strictly quasi-nonexpansive, where C  i 2I Fix Ui , C ¤ ;. If U has one of the following forms: P (i) U WD i 2I !i Ui and the weight function w W X ! m is appropriate, (ii) U WD Um Um1 : : : U1 , then Fix U D

\

Fix Ui

(2.18)

i 2I

and U is C -strictly quasi-nonexpansive. T Proof. The inclusion i 2I Fix Ui  FixTU is obvious. Now we show that Fix U  T is clear if i 2I Fix Ui D X . Now suppose that x … Ti 2I Fix Ui . This inclusion T Fix U . Let z 2 Fix Ui . If z 2 C , then the C -strict quasi nonexpansivity i i 2I i 2I of Ui , i 2 I , yields kUi x  zk < kx  zk for any i 2 I.x/:

(2.19)

2.1 Basic Definitions and Properties

51

P (i) Let Ux D i 2I !i .x/Ui x, where the weight function w W X ! m is appropriate. Then the convexity of the norm, (2.19) and (2.15) yield    X   !i .x/.Ui x  z/ kUx  zk D    i 2I X X !i .x/ kUi x  zk  !i .x/ kx  zk D kx  zk ,  i 2I

i 2I

where the second inequality is strict if z 2 C . (ii) Let j WD minfi 2 I W x … Fix Ui g. Then we have Uj Uj 1 : : : U1 x D Uj x and (2.19) yields kUx  zk D kUm : : : U1 x  zk   D Um : : : Uj x  z    Um1 : : : Uj x  z    : : :  Uj x  z  kx  zk , where the latter inequality is strict if z 2 C . Now it is clear that x … Fix U because, otherwise, for z 2 C , in both cases (i) and (ii) we would obtain kx  zk D kUx  zk < kx  zk , T a contradiction. We have proved that Fix U  i 2I Fix Ui . Hence, (2.18) holds and, in both cases (i) and (ii), U is C -strictly quasi-nonexpansive. t u Note that equality (2.18) needs not to be true for nonexpansive operators, even if they have a common fixed point. Example 2.1.27. (cf. [22, Remark 2.11]) Let X  H be a subspace with dim X > 0. Let Ui W X ! X , Ui WD  Id, i D 1; 2. We have U2 U1 D Id, consequently, Fix.U2 U1 / D X , but Fix U1 \ Fix U2 D f0g. The assumption on the C -strict quasi nonexpansivity in Theorem 2.1.26 (i) can be weakened. In this case it suffices to suppose that all Ui are quasi-nonexpansive, i 2 I , and at least one of them is C -strictly quasi-nonexpansive. The assumption that the weight function w is appropriate should be replaced in this case by a stronger one, namely: wj .x/ > 0 for all x such that I.x/ ¤ ; and for all j 2 I.x/. We leave the proof of this fact to the reader. A stronger version of the first part of Theorem 2.1.26 (ii) for two operators is stated below (cf. [346, Proposition 1(d) (i)]).

52

2 Algorithmic Operators

Theorem 2.1.28. Let S W X ! X be quasi-nonexpansive, T W X ! X be strictly quasi-nonexpansive and Fix S \ Fix T ¤ ;. Then Fix S T D Fix T S D Fix S \ Fix T . Furthermore, S T is quasi-nonexpansive and T S is strictly quasinonexpansive. Proof. The inclusions Fix S \ Fix T  Fix S T and Fix S \ Fix T  Fix T S are clear. (i) We prove that Fix S T  Fix S \Fix T . The inclusion is obvious if Fix S T D ;. Suppose that Fix S T ¤ ; and let x 2 Fix S T be such that x … Fix S \ Fix T . We consider two cases: (a) x 2 Fix T . Then x D S T x D S x, i.e., x 2 Fix S . Therefore, x 2 Fix S \ Fix T . (b) x … Fix T . Let z 2 Fix S \ Fix T . By the quasi nonexpansivity of S and by the strict quasi nonexpansivity of T , we have kx  zk D kS T x  zk  kT x  zk < kx  zk . In both cases we obtain a contradiction, which proves that Fix S T  Fix S \ Fix T . (ii) We prove that Fix T S  Fix T \Fix S . The inclusion is obvious if Fix T S D ;. Suppose that Fix T S ¤ ; and let x 2 Fix T S be such that x … Fix T \ Fix S . Consider two cases: (a) S x 2 Fix T . Then x D T S x D S x, consequently, x 2 Fix S . Now we have x D S x 2 Fix T , i.e., x 2 Fix T \ Fix S . (b) S x … Fix T . Let z 2 Fix T \ Fix S . By the strict quasi nonexpansivity of T and by the quasi nonexpansivity of S , we have kx  zk D kT S x  zk < kS x  zk  kx  zk . In both cases we obtain a contradiction, which proves that Fix T S  Fix T \ Fix S . Let now z 2 Fix T S D Fix T \ Fix S and x 2 X . We have kS T x  zk  kT x  zk  kx  zk , i.e., S T is quasi-nonexpansive. Furthermore, kT S x  zk  kS x  zk  kx  zk , where the second inequality is strict if x … Fix S and the first one is strict if x 2 Fix S and x … Fix T . Hence, T S is strictly quasi-nonexpansive. u t

2.1 Basic Definitions and Properties

53

Fig. 2.3 Pcl conv C is a separator of C

z x

y

cl conv C

C y = Pcl conv

C (x)

Corollary 2.1.29. Let U WD Um Um1 : : : U1 , where U1 ; U2 ; : : : ; Um1 W X ! X are quasi-nonexpansive, Um W T X ! X is strictly quasi-nonexpansive and T i 2I Fix Ui ¤ ;. Then Fix U D i 2I Fix Ui and U is strictly quasi-nonexpansive. Proof. The corollary follows from Theorem 2.1.28. We leave to the reader an easy proof by induction with respect to m. t u The assumption of Theorem 2.1.28 that T is strictly quasi-nonexpansive is essential. Note that the composition of quasi-nonexpansive operators needs not to be quasi-nonexpansive (see Example 2.1.52). Furthermore, the assumptions of Theorem 2.1.28 do not yield the strict quasi nonexpansivity of the operator S T (see Example 2.1.54).

2.1.3 Cutters and Strongly Quasi-nonexpansive Operators Definition 2.1.30. Let x 2 H. We say that y 2 H separates a subset C  H from x if hx  y; z  yi  0 for all z 2 C . We say that an operator T W X ! H is a separator of a subset C  X or T separates a subset C , if y WD T x separates C from x for all x 2 H. We say that T is an ˛-relaxed separator of C , where ˛ 2 Œ0; 2, if T is an ˛-relaxation of a separator of C . Let T have a fixed point. We say that T is a cutter if T is a separator of Fix T , i.e., hx  T x; z  T xi  0 (2.20) for all x 2 X and all z 2 Fix T . We say that T is an ˛-relaxed cutter, where ˛ 2 Œ0; 2, if T is an ˛-relaxed separator of Fix T . For any nonempty C  H the projection Pcl conv C is a separator of C (see Fig. 2.3). In general, a separator of C is not uniquely determined. The name cutter expresses the fact that, for any x … Fix T , the hyperplane H.x  T x; hT x; x  T xi/ cuts the space into two half-spaces, one of which contains the point x while the other one contains the subset Fix T (see Fig. 2.4). In the literature one can find different names for cutters. Bauschke and Combettes call the class of

54

2 Algorithmic Operators

Fig. 2.4 A cutter and an ˛-relaxed cutter

Fix T Tα x

z

Tx x − T x; u − T x

0

x

cutters a T -class (see [24, Definition 2.2] and [121, Definition 2.1]). Yamada and Ogura (see [346, Sect. B]) and M˘arus¸ter (see [254]) call the operators firmly quasinonexpansive. Zaknoon, Segal and Censor denoted cutters as directed operators (see [104–106, 307, 356]). In [69] these operators were called separating operators. The name cutter was proposed by Cegielski and Censor in [70]. Note that a separator and, in particular, a cutter need not to be continuous operators. Remark 2.1.31. Let T W X ! H have a fixed point. Then, by Lemma 1.2.5, the operator T is a cutter if and only if hT x  x; z  xi  kT x  xk2

(2.21)

holds for all x 2 X and for all z 2 Fix T (cf. [121, Proposition 2.3 (ii)]), and T is an ˛-relaxed cutter, where ˛ 2 Œ0; 2, if and only if ˛hT x  x; z  xi  kT x  xk2

(2.22)

holds for all x 2 X and for all z 2 Fix T . Furthermore, if T is a cutter, then T jFix T D Id. Therefore, T is a cutter (respectively, an ˛-relaxed cutter) if and only if inequality (2.21) (respectively, (2.22)) is satisfied for all x … Fix T and for all z 2 Fix T . Relaxed cutters were also studied in [253, 255, 346] and in [249], where they were called averaged quasi-nonexpansive mappings. Remark 2.1.32. Let T W X ! H be a separator of a subset C  X . Then the following obvious properties of T hold: (i) (ii) (iii) (iv)

T is a separator of the closed convex hull of C . T is a separator of any subset D  C . For an arbitrary  2 Œ0; 1, the relaxation T of T is a separator of C . C  Fix T .

Corollary 2.1.33. Let U W X ! H, T WD 12 .U C Id/ and C  X . Then: (i) U is Fej´er monotone with respect to C if and only if T is a separator of C .

2.1 Basic Definitions and Properties

55

(ii) If Fix U ¤ ;, then U is quasi-nonexpansive if and only if T is a cutter. Proof. The corollary follows easily from equivalence (2.11) (see [24, Proposition 2.3 (v),(vi)] for a different proof). t u Corollary 2.1.34. Let T W X ! H and C  X . If T is Fej´er monotone with respect to C , then T is Fej´er monotone with respect to the closed convex hull of C . Proof. The corollary follows directly from Corollary 2.1.33 (i) and from Remark 2.1.32 (i). t u By Remark 2.1.32 (iii), the right hand side of the equivalence in Corollary 2.1.33 (i) can be written in the form: U is a separator of C for all  2 Œ0; 12 . Similarly, the right hand side of the equivalence in Corollary 2.1.33 (ii) can be written in the form: U is a cutter for all  2 Œ0; 12 . Corollary 2.1.33 (ii) can also be written equivalently as follows: U is quasi-nonexpansive if and only if there is a cutter S W X ! H and  2 Œ0; 2 such that U D S . A subset C  X for which the operator T W X ! H is a separator needs not to be convex. However there exists the largest subset for which T is a separator, which is closed and convex. This fact follows from the following lemma. Lemma 2.1.35. Let T W X ! H. If the subset Sep T WD

\

fz 2 X W hz  T x; x  T xi  0g

x2X

is nonempty, then Sep T is the largest subset for which T is a separator. Furthermore, Sep T is a closed convex subset. Proof. The first part of the lemma follows directly from Definition 2.1.30. The second part follows from the fact that an intersection of closed convex subsets is closed and convex. t u If T is nonexpansive, then Fix T is a closed convex subset (see Proposition 2.1.11). It turns out that cutters have the same property. The second part of the following lemma was proved in [24, Proposition 2.6 (i)–(ii)]. Lemma 2.1.36. Let T W X ! H. The following inclusion holds Sep T  Fix T .

(2.23)

If T is a cutter, then a converse inclusion is also true. Hence, the subset of fixed points of a cutter is closed and convex. Proof. Let y 2 Sep T , i.e., hx  T x; y  T xi  0 for all x 2 X . If we take x D y, we get ky  T yk  0, and hence, T y D y, i.e., y 2 Fix T . Now suppose that T is a cutter and that y 2 Fix T . Then for any x 2 X we have hy  T x; x  T xi  0g, i.e.,

56

2 Algorithmic Operators

y 2 Sep T . Therefore, we have Sep T D Fix T . The subset Fix T is closed convex as an intersection of closed convex subsets. t u It follows from Remark 2.1.32 (ii) and from Lemmas 2.1.35 and 2.1.36 that a cutter T W X ! X is a separator of any nonempty subset of Fix T . Therefore, we will restrict our further considerations of separators to cutters. Note, however, that the converse inclusion of (2.23) is not true in general (see Example 2.2.7). Hence, there is a separator with a fixed point which is not a cutter. It is an immediate consequence of the characterization of the metric projection (see Theorem 1.2.4) that an operator T W H ! H is a metric projection onto a closed convex subset if and only if T 2 D T and T is a cutter (a more general fact will be presented in Theorem 2.2.5). In this case, we have T D PFix T . Even if a cutter T is not idempotent, T is closely related to the metric projection. The following corollary was proved in [121, Proposition 2.3 (iii)]. Corollary 2.1.37. Let T W X ! H be a cutter. Then, for any x 2 X , it holds kT x  xk  kPFix T x  xk .

(2.24)

Proof. If x 2 Fix T , then inequality (2.24) is obvious. Now let x … Fix T . Then it follows from inequality (2.21) for z WD PFix T x together with the Cauchy–Schwarz inequality that kT x  xk 

hT x  x; PFix T x  xi  kPFix T x  xk kT x  xk

which completes the proof.

t u

Definition 2.1.38. Let ˛  0 and assume that T W X ! H has a fixed point. We say that T is ˛-strongly quasi-nonexpansive (˛-SQNE), if kT x  zk2  kx  zk2  ˛ kT x  xk2

(2.25)

for all x 2 X and z 2 Fix T . If T satisfies (2.25) with ˛ > 0, then T is called strongly quasi-nonexpansive (SQNE). A property which is more general than the strong nonexpansivity was introduced by Halperin [198, Sect. 2] and was called '-property, where ' W Œ0; 1/ ! Œ0; 1/ is a nondecreasing function. If '.t/ D t 2 for all t 2 Œ0; 1/, then '-property is equivalent to the strong quasi-nonexpansivity. The notion strong quasi nonexpansivity was introduced by Bruck [50, Sect. 1] for operators defined on a metric space. Strongly quasi-nonexpansive operators are widely studied in the literature. Bauschke and Borwein use the name strongly attracting operators for operators which are NE and SQNE (see [22, Definition 2.1]). Reich and Zaslavski define a more general operator and call it a uniformly F -attracting mapping, where F D Fix T (see [297, Sect. 1]). Vasin and Ageev call the ˛-SQNE operators, where ˛ 2 .0; 1/, Q-pseudocontractive operators (see [333, Definition 2.3]). Yamada and Ogura

2.1 Basic Definitions and Properties

57

use the notation ˛-attracting quasi-nonexpansive for the ˛-SQNE operators [346]. Crombez denotes the class of ˛-SQNE operators by F ˛ (see [126, pages 160–161]) and gives several equivalent conditions for T 2 F ˛ (see [126, Theorem 2.1]). It follows easily from the equivalence (a),(c) of Lemma 1.2.5 that an operator T which has a fixed point is a cutter if and only if it is 1-strongly quasinonexpansive. The following theorem extends this property to relaxations of T (cf. [121, Proposition 2.3 (ii)]). Theorem 2.1.39. Assume that T W X ! H has a fixed point and let  2 .0; 2. Then T is a cutter if and only if its relaxation T is 2  -strongly quasi-nonexpansive, i.e., 2 (2.26) kT x  xk2 kT x  zk2  kx  zk2   for all x 2 X and for all z 2 Fix T . Proof. Since T x  x D .T x  x/, the properties of the inner product yield kT x  zk2  kx  zk2 C

2 kT x  xk2 

D kx  z C .T x  x/k2  kx  zk2 C .2  / kT x  xk2 D 2.kT x  xk2  hz  x; T x  xi/ D 2hz  T x; x  T xi for all x 2 X and for all z 2 C . The assertion follows directly from the equalities above. t u The following corollary is an equivalent formulation of Theorem 2.1.39. Corollary 2.1.40. Assume that U W X ! H has a fixed point and let ˛ 2 .0; 2. Then U is an ˛-relaxed cutter if and only if U is 2˛ ˛ -strongly quasi-nonexpansive. In general, a relaxation T of a cutter T with   2 needs not to be strongly quasi-nonexpansive. Nevertheless, the following proposition holds. Proposition 2.1.41. Let T W X ! H be a cutter with int Fix T ¤ ; and  > 0. Then for any z 2 int Fix T and x … Fix T it holds kT x  zk2  kx  zk2  .2 C

2ı  / kT x  xk2 , kT x  xk

(2.27)

where ı > 0 is such that B.z; ı/  Fix T . If X is bounded, then T is int Fix T strictly quasi-nonexpansive for any  2 .0; 2.

58

2 Algorithmic Operators

xx Proof. Let z 2 int Fix T and x … Fix T . Then w WD z  ı kTT xxk 2 Fix T  X and inequality (2.21) yields

kT x  zk2 D kx C .T x  x/  zk2 D kx  zk2 C 2 kT x  xk2  2hz  x; T x  xi D kx  zk2 C 2 kT x  xk2 2hz  w; T x  xi  2hw  x; T x  xi  kx  zk2 C 2 kT x  xk2 2ı kT x  xk  2 kT x  xk2 D kx  zk2  .2 C

2ı  / kT x  xk2 . kT x  xk

Let X be bounded and d > 0 be such that kT u  uk  d for any u 2 X . The existence of such d follows from Corollary 2.1.37. Denote " WD 2ı . Then (2.27) d yields kT x  zk2  kx  zk2  .2 C "  / kT x  xk2 . Consequently, T is int Fix T -strictly quasi-nonexpansive for any  2 .0; 2.

t u

The corollary below follows immediately from Proposition 2.1.41 and from Theorem 2.1.26. Corollary 2.1.42. Let Ui W X ! H, i 2 I be quasi-nonexpansive with C WD T Fix U i ¤ ; and let U WD Um Um1 : : : U1 . If int C ¤ ;, then Fix U D Ti 2I Fix U i and U is int C -strictly quasi-nonexpansive. i 2I An equivalent formulation of the following result appeared in [127, Theorem 3.2 (iii)]. Corollary 2.1.43. Assume that U W X ! H has a fixed point and let ˇ  0. Then 2 U is ˇ-strongly quasi-nonexpansive if and only if U is a ˇC1 -relaxed cutter. Proof. It suffices to take ˛ D

2 ˇC1

in Corollary 2.1.40.

t u

Remark 2.1.44. Assume that T W X ! H has a fixed point and is ˛-strongly quasi-nonexpansive, where ˛  0. (i) If ˛ D 0, then T is quasi-nonexpansive. (ii) T is  -strongly quasi-nonexpansive for all  2 Œ0; ˛. (iii) If ˛ > 0, then T is strictly quasi-nonexpansive. Therefore, all properties of strictly quasi-nonexpansive operators are also valid for strongly quasinonexpansive operators and for cutters.

2.1 Basic Definitions and Properties

59

Fig. 2.5 Solution of (2.28) as a function of ;  2 .0; 2/

γ = γ(λ, μ)

2

1

0 2 2 1

1

μ

0

λ

Cutters and strongly quasi-nonexpansive operators play an important role in methods presented in further parts of the book. Therefore, we focus our attention on the properties of these operators which enable us to construct new cutters or strongly quasi-nonexpansive operators. Below, we show that a family of relaxed cutters is closed under composition and under convex combination of operators having a common fixed point. The first property of relaxed cutters follows from the lemma below whose proof is left to the reader. Lemma 2.1.45. Let ;  2 .0; 2/. The unique solution  of the equation 1

2 

!2 D.

2

1 1 1 1  /.  /    

(2.28)

is D

2  . 2

C

 1 / 2

C1

D

4. C   / . 4  

(2.29)

Moreover, 0 < minf; g <

4 minf; g 4 maxf; g   < 2. minf; g C 2 maxf; g C 2

A solution of (2.28) is illustrated in Fig. 2.5. Theorem 2.1.46. Let T W X ! X be a -relaxed cutter, U W X ! X be a relaxed cutter, where ;  2 .0; 2, and let Fix T \ Fix U ¤ ;. If ;  2 .0; 2/, then U T is a  -relaxed cutter, where  is given by (2.29). If  D 2 and  < 2 or  D 2 and  < 2, then U T is a quasi-nonexpansive operator or, equivalently, U T is a 2-relaxed cutter.

60

2 Algorithmic Operators

Proof. Suppose that T is a -relaxed cutter and that U is a -relaxed cutter. Take a WD T x  x and b WD U T x  T x. Then it follows from inequality (2.22) that hz  x; ai  1 kak2 and hz  T x; bi  1 kbk2 for any z 2 Fix U \ Fix T . Let ;  2 .0; 2/ and  be defined by (2.29). Then Lemma 2.1.45 yields hz  x; UTx  xi  D hz  x; a C bi 

1 kU T x  xk2 

1 ka C bk2 

D hz  x; ai C hz  x; bi 

1 ka C bk2 

D hz  x; ai C hz  T x; bi C ha; bi  

1 ka C bk2 

1 1 1 kak2 C kbk2 C ha; bi  ka C bk2   

1 1 1 1 2  / kak2 C .  / kbk2 C .1  /ha; bi      s s 2  1 1 1  1   D  a  b   0.      

D.

Applying inequality (2.22) we obtain that U T is a  -relaxed cutter. If  D 2 and  < 2 or  D 2 and  < 2, then U T is quasi-nonexpansive by Theorem 2.1.28. u t The following result is due to Yamada and Ogura (see [346, Proposition 1(d)]). Corollary 2.1.47. Let T; U W X ! X have a common fixed point and ; > 0. If T is -SQNE and U is -SQNE, then U T is ı-SQNE, where ıD

1

1 C

1

.

(2.30)

Proof. Suppose that T is -SQNE and U is -SQNE. It follows from Corollary 2.1.43 that T is a -relaxed cutter and that U is a -relaxed cutter, where 2 2  D 1C and  D 1C

. By Theorem 2.1.46 the operator U T is a  -relaxed cutter, where D

2  . 2

C

 1 / 2

C1

D

2 . 1

C

1 1

/

C1

.

Corollary 2.1.40 yields now that U T is ı-SQNE, where ı is given by (2.30).

t u

2.1 Basic Definitions and Properties

61

Theorem 2.1.48. Let Ti W X ! X be an ˛i -relaxed cutter, where ˛i 2 .0; 2/, i 2 I WD f1; 2; : : : ; mg, or, equivalently,T Ti be ˇi -strongly quasi-nonexpansive, where i ˇi D 2˛ 2 .0; C1/, i 2 I . Let i 2I Fix Ti ¤ ; and Um WD Tm Tm1 : : : T1 . ˛i Then: (i) The operator Um is a m -relaxed cutter, with m D

˛1 . 2˛ C 1

˛2 2˛2

2 C ::: C

˛m 1 / 2˛m

C1

.

(2.31)

(ii) The operator Um is ım -strongly quasi-nonexpansive, with ım D

1 ˇ1

C

1 ˇ2

1 C ::: C

1 ˇm

.

(2.32)

Moreover, 0 < min ˛i < i 2I

2m mini 2I ˛i 2m maxi 2I ˛i  m  <2 .m  1/ mini 2I ˛i C 2 .m  1/ maxi 2I ˛i C 2 (2.33)

and 0<

mini 2I ˇi maxi 2I ˇi  ım  . m m

(2.34)

Proof. The assertion is obvious for m D 1. Note that m D ım2C1 and that Corollary 2.1.43 yields the equivalence of conditions (i) and (ii). We prove by induction with respect to m that these conditions hold for any m  2. 10 If m D 2, then conditions (i) and (ii) follow directly from Theorem 2.1.46 and from Corollary 2.1.47. 20 Suppose that (ii) is true for some m D k. Consequently, Uk is ık -SQNE. It follows now from Corollary 2.1.47 that the operator UkC1 D TkC1 Uk is ı-SQNE, where ıD

1 ık

1 D 1 C ˇkC1

1 ˇ1

C

1 ˇ2

1 C ::: C

1 ˇk

C

1 ˇkC1

D ıkC1 .

Now, for m D k C 1, equality (2.31) follows from the above mentioned equivalence of (i) and (ii). Hence, we have proved that conditions (i) and (ii) hold for all m  1. Both inequalities in (2.34) follow immediately from equality (2.32). Now we have m D

2  ım C 1

2 mini 2I ˇi m

C1

D

2 mini 2I m

2˛i ˛i

D C1

2m maxi 2I ˛i < 2. .m  1/ maxi 2I ˛i C 2

62

2 Algorithmic Operators

In a similar way one can prove that m 

2m mini 2I ˛i > min ˛i > 0 i 2I .m  1/ mini 2I ˛i C 2 t u

which completes the proof.

Bauschke and Borwein proved that a composition of ˇi -SQNE operators with a i 2I ˇi common fixed point is ˇ-SQNE for ˇ WD min2m1 (see [22, Theorem 2.10 (ii)]). It is clear that this result is weaker than Theorem 2.1.48 (ii), because ˇ  minim2I ˇi  ım . Note that the first inequality is strict for m > 2 and that the other one is strict if ˇi ¤ ˇj for at least one pair i; j 2 I . Corollary 2.1.49. Let Ui W X ! H be cutters with a common fixed point, i 2 I WD f1; 2; : : : ;P mg, and w W X ! m be an appropriate weight function. Then the operator U WD i 2I !i Ui is a cutter. P Proof. Let U WD i 2I !i Ui . It is clear that a cutter is strictly quasi-nonexpansive (see Remark T 2.1.44 (iii)). Therefore, it follows from Theorem 2.1.26 (i) that Fix U D i 2I Fix Ui . By Remark 2.1.31 and by the convexity of the function kk2 , we have X hUx  x; z  xi D !i .x/hUi x  x; z  xi i 2I



X

!i .x/ kUi x  xk2

i 2I

 2 X     !i .x/Ui x  x    i 2I

D kUx  xk2 for all x 2 X and all z 2 Fix U . Again, by Remark 2.1.31, U is a cutter.

t u

Theorem 2.1.50. Let Ti W X ! H be an ˛i -relaxed cutter, where ˛i 2 .0; 2/, i 2 I WD f1; 2; : : : ; mg, or, equivalently,TTi be ˇi -strongly quasi-nonexpansive, i where ˇi D 2˛ i 2I Fix Ti ¤ ; and w 2 m . Then the ˛iP 2 .0; C1/, i 2 I . Let operator T WD i 2I !i Ti is an ˛-relaxed cutter with ˛ WD

X

!i ˛i .

(2.35)

!i 1 /  1. ˇi C 1

(2.36)

i 2I

Consequently, T is ˇ-SQNE, with ˇ WD .

X i 2I

2.1 Basic Definitions and Properties

63

Moreover, 0 < min ˛i  ˛  max ˛i < 2

(2.37)

0 < min ˇi  ˇ  max ˇi .

(2.38)

i 2I

i 2I

and i 2I

i 2I

Proof. Without loss of generality we suppose that w 2 ri m . Let Ui WD .Ti /˛i1 , i.e., 1 Ui D Id C .Ti  Id/. ˛i It is clear that Ui are cutters, i 2 I . Let ˛ be defined by (2.35) and i WD !i˛˛i , i 2 I . Note P that v D .1 ; 2 ; : : : ; m / 2 ri m , consequently, v is appropriate. Define U WD i 2I i Ui . By Corollary 2.1.49, the operator U is a cutter. We have U D

X i 2I

i Ui D Id C

X i 1X 1 .Ti  Id/ D Id C !i .Ti  Id/ D Id C .T  Id/, ˛i ˛ i 2I ˛ i 2I

i.e., T D Id C˛.U Id/ and T is an ˛-relaxed cutter. The second part of the theorem follows now immediately from Corollaries 2.1.40 and 2.1.43. Inequalities in (2.37) are obvious and inequalities in (2.38) follow easily from (2.36). t u Bauschke and Borwein proved that a convex combination of ˇi -SQNE operators, i 2 I , with a common fixed point is ˇ-SQNE, where ˇ WD mini 2I ˇi (see [22, Proposition 2.12]). By inequality (2.38) this result is weaker than Theorem 2.1.50. Note that this inequality is strict if ˇi ¤ ˇj for at least one pair i; j 2 I for which !i and !j are nonzero. The second part of Theorem 2.1.50 for m D 2 was proved by Yamada and Ogura (see [346, Proposition 1(c)]). The following important result extends Theorem 2.1.39. Theorem 2.1.51. Let S W H ! X be nonexpansive, T W X ! H be a cutter and  2 .0; 2/. If Fix S \ Fix T ¤ ;, then, for any x 2 Fix S and z 2 Fix S \ Fix T , the following estimations hold kS T x  zk2  kx  zk2  .2  / kT x  xk2

(2.39)

2 kS T x  xk2 . 

(2.40)

and kS T x  zk2  kx  zk2  Consequently, the operator S T jFix S is

2 -strongly 

quasi-nonexpansive.

Proof. Let x 2 Fix S and z 2 Fix S \ Fix T . Then the assumptions that S is nonexpansive and T is a cutter yield

64

2 Algorithmic Operators

Fig. 2.6 Composition of cutters needs not to be a cutter B

1 1 2

U1x

Ux = U2 U1x

A∩B

z 1 2

1

A

x

kS T x  zk2 D kS T x  S zk2  kT x  zk2 D kx  zk2 C 2 kT x  xk2  2hz  x; T x  xi  kx  zk2  .2  / kT x  xk2 2 kT x  xk2  2  kx  zk2  kS T x  S xk2  2 D kx  zk2  kS T x  xk2  D kx  zk2 

t u

which completes the proof.

Below we give several examples which show that a composition of quasinonexpansive operators does not need to be quasi-nonexpansive, that a composition of a strictly quasi-nonexpansive operator and a quasi-nonexpansive one does not need to be strictly quasi-nonexpansive and that a composition of cutters does not need to be a cutter, even if they have a common fixed point. Example 2.1.52. Let X WD Œ1; 1  R, S; T W X ! X , S WD  Id and  T x WD

x if x D 1 1 2 x otherwise.

One can easily check that S; T are quasi-nonexpansive, Fix S D Fix T D f0g and Fix S T D f0; 1g. The operator S T is not quasi-nonexpansive, because a subset of fixed points of a quasi-nonexpansive operator is convex (see Lemma 2.1.21). Example 2.1.53. Let X D H WD R2 , A WD fx 2 R2 W he; xi  1g, B WD fx 2 R2 W 2  0g, U1 WD PB , U2 WD PA and U WD U2 U1 . Then U1 and U2 are cutters and it follows from Theorem 2.1.26 that Fix U D Fix U1 \ Fix U2 D A \ B ¤ ;. For x D .0; 1/ and z D .1; 0/ 2 A \ B, we have Ux D . 12 ; 12 / and hx  Ux; z  Uxi D 12 (see Fig. 2.6). Therefore, U is not a cutter.

2.2 Firmly Nonexpansive Operators

65

Example 2.1.54. Let A; B  H be nonempty closed convex subsets and A   B. Define S WD 2PA  Id and T WD PB . We have Fix S \ Fix T D A. It follows easily from the characterization of the metric projection that PA and PB are cutters. By Theorem 2.1.39, T is strictly quasi-nonexpansive and S is quasi-nonexpansive. By Theorem 2.1.28, the operator S T is quasi-nonexpansive. Unfortunately, S T is not strictly quasi-nonexpansive, because for any x 2 BnA and for z WD PA x it holds kS T x  zk D kS x  zk D kx  zk .

2.2 Firmly Nonexpansive Operators Definition 2.2.1. We say that an operator T W X ! H is firmly nonexpansive (FNE), if hT x  T y; x  yi  kT x  T yk2 (2.41) for all x; y 2 X . Let  2 Œ0; 2. We say that T W X ! H is -relaxed firmly nonexpansive (-RFNE) or, shortly, relaxed firmly nonexpansive (RFNE) if T is a -relaxation of a firmly nonexpansive operator U , i.e., T D U D .1  / Id CU . If, furthermore,  2 .0; 2/, then we say that T is strictly relaxed firmly nonexpansive. The definition of a firmly nonexpansive operator in a Hilbert space is due to Browder (see [46]), who called it a firmly contractive operator. Bruck introduced the name firmly nonexpansive for operators in a Banach space (see [49, Definition 6]). In Hilbert spaces both definitions coincide, as we will show in Theorem 2.2.10. Condition (vi) of this theorem is, actually, the definition of a firmly nonexpansive operator proposed by Bruck. The following lemma is obvious. Lemma 2.2.2. Let T W X ! H and x; y 2 X . The following inequalities are equivalent: (i) (ii) (iii) (iv)

hT x  T y; x  yi  kT x  T yk2 , hT x  T y; .x  T x/  .y  T y/i  0, hT y  T x; x  T xi C hT x  T y; y  T yi  0, hT y  x; T x  xi C hT x  y; T y  yi  kT x  xk2 C kT y  yk2 .

It follows from Lemma 2.2.2 that inequality (2.41) defining a firmly nonexpansive operator can be replaced by any inequality in (i)–(iv). Corollary 2.2.3. Let  > 0. An operator S W X ! H is -RFNE if and only if hy  x; S x  xi C hx  y; Sy  yi 

1 k.S x  x/  .Sy  y/k2 . 

(2.42)

66

2 Algorithmic Operators

Fig. 2.7 NE and monotone operator which is not FNE Ty

Tx

a

PLina (x − y) y '

x

Proof. Let S WD T D Id C.T  Id/ for a firmly nonexpansive operator T W X ! H. Let x; y 2 X . It follows from the equivalence (i),(iv) in Lemma 2.2.2 and from the equality T D S1 (see Remark 2.1.3) that S is -RFNE if and only if hT y  x; S x  xi C hT x  y; Sy  yi 

1 .kS x  xk2 C kSy  yk2 /. 

Since T y  x D y  x C 1 .Sy  y/ and T x  y D x  y C 1 .S x  x/, the last inequality is equivalent to 1 2 hyx; S xxiChxy; SyyiC hS xx; Syyi  .kS x  xk2 CkSy  yk2 /.   t u

The latter inequality is equivalent to (2.42).

2.2.1 Basic Properties of Firmly Nonexpansive Operators Theorem 2.2.4. A firmly nonexpansive operator T W X ! H is monotone and nonexpansive. Proof. Let T be firmly nonexpansive. By the Cauchy–Schwarz inequality, we have kT x  T yk  kx  yk  hT x  T y; x  yi  kT x  T yk2  0, for all x; y 2 X , which yields the monotonicity and the nonexpansivity of T .

t u

The converse of Theorem 2.2.4 is not true, e.g., the operator T W R ! R , 2

2

T x WD .1 cos '  2 sin '; 1 sin ' C 2 cos '/ is nonexpansive and monotone for ' 2 .0; =2/, but T is not firmly nonexpansive (see Fig. 2.7). Now we prove a property of firmly nonexpansive operators, which also appears in Theorem 1.2.4. In particular, the characterization of the metric projection is,

2.2 Firmly Nonexpansive Operators

67

actually, a corollary of the following theorem which is due to Goebel and Reich (see [185, pp. 43–44]). Theorem 2.2.5. Let T W X ! H be an operator with a fixed point. (i) If T is firmly nonexpansive, then T is a cutter, i.e., hz  T x; x  T xi  0

(2.43)

for all x 2 X and z 2 Fix T . (ii) If T is a projection, i.e., T .X / D Fix T , then the implication converse to .i/ is also true. In this case, T D PFix T . Proof. (i) Let T be firmly nonexpansive, x 2 X and z 2 Fix T . By the equivalence (i),(iii) in Lemma 2.2.2, we have hT y  T x; x  T xi C hT x  T y; y  T yi  0, and for y D z 2 Fix T we obtain (2.43). (ii) Suppose that T is a projection and that inequality (2.43) holds for all x 2 X and z 2 Fix T . Let u; v 2 X . Taking x D u and z D T v in (2.43) we get hT v  T u; u  T ui  0,

(2.44)

and, taking x D v and z D T u in (2.43), we get hT u  T v; v  T vi  0.

(2.45)

Note that, in both cases, z 2 Fix T because T .X / D Fix T . Therefore, the characterization of the metric projection yields that T D PFix T . Summing up inequalities (2.44) and (2.45) we get hT u  T v; .T u  T v/  .u  v/i  0; i.e., T is firmly nonexpansive (see equivalence (i),(ii) in Lemma 2.2.2).

t u

Suppose that Fix T ¤ ;. It follows from the equivalence of (i) and (iii) in Lemma 2.2.2 that inequality (2.41) for y D z 2 Fix T gives (2.43). Therefore, for T being a cutter, inequality (2.41) is required for all x 2 X and all y 2 Fix T , while for T being firmly nonexpansive this inequality should hold for all x; y 2 X . Remark 2.2.6. Neither a projection nor a separator of a nonempty subset C  H need to be nonexpansive (note that a separator can even be discontinuous). Furthermore, a nonexpansive separator and even a nonexpansive cutter need not to be firmly nonexpansive (see Examples 2.2.7 and 2.2.8 below).

68

2 Algorithmic Operators x

Fig. 2.8 NE separator which is not FNE

y a B

Tx

B1

Ty Tz w =Tw z

A B2

Example 2.2.7. (cf. [204] and [78, Sect. 4.10]) Let a 2 H, kak D 1 and ˛ > 0. Furthermore, let A WD fx 2 H W ha; xi D 0g, B1 WD fx 2 H W ha; xi D ˛g, B2 WD fx 2 H W ha; xi D ˛g and B WD fx 2 H W jha; xij  ˛g. The subset B is a band with a width of 2˛ and is bounded by two hyperplanes B1 and B2 . The hyperplane A cuts the band B into two bands bounded by A and B1 and by A and B2 . Define the operator T W H ! H as follows 8 PA x ˆ ˆ < 2PB1 x  x Tx D ˆ 2PB2 x  x ˆ : x

if jha; xij  2˛ if ˛ < ha; xi < 2˛ if  2˛ < ha; xi < ˛ if jha; xij  ˛

(2.46)

Note that T projects onto A all points with the distance to A equal at least 2˛, T reflects (with respect to the closest hyperplane B1 or B2 ) the points which do not belong to the band B with the distance to A less than 2˛ and T does not move the elements of the band B (see Fig. 2.8). The reader can easily check that T is nonexpansive and that T is a separator of A but T is not firmly nonexpansive. Note that Fix T D B and that T is not a cutter, i.e., it does not separate Fix T , but T separates A (see Fig. 2.8). Example 2.2.8. Let A WD R  f0g and B WD f0g  R be two subspaces of R2 and T W R2 ! R2 be defined by T x WD Œ1  .x/PA x C .x/PB x, where .x/ D

12 12 C22

for x D .1 ; 2 / 2 R2 (see Fig. 2.9). We have PA x D .1 ; 0/,

PB x D .0; 2 /. Consequently, Tx D .

1 22  2 2 ; 2 1 2/ 2 C 2 1 C 2

12

for x ¤ .0; 0/. Note that z WD .0; 0/ is the unique fixed point of T . The operator T is a cutter, because hz  T x; x  T xi D 

12 22 0 C 22

12

2.2 Firmly Nonexpansive Operators

69

Fig. 2.9 NE cutter which is not FNE

»2 = »1 PB x

x

Tx

FixT

PA x

for all x ¤ z. (Since the weight function w W R2 ! 2 , w.x/ WD .1  .x/; .x// is appropriate, this fact follows also from Corollary 2.1.49). Let x; y ¤ .0; 0/. A straightforward calculation shows that kT x  T yk2 kx  yk2

D

22 22 .1  1 /2 C 12 21 .2  2 /2 .12 C 22 /. 21 C 22 /Œ.1  1 /2 C .2  2 /2 

holds for all x D .1 ; 2 / 2 R2 and for all y D . 1 ; 2 / 2 R2 , x ¤ y. If 1 1 D 2 2 D 0, then, of course T x D T y D .0; 0/. Suppose that 0 < 12 21  22 22 . Then we have kT x  T yk2 kx  yk2

D

.1  1 /2 C .1 C

12 /.1 22

C

21 /Œ.1

22

12 21 . 22 22 2

 2 /2

 1 /2 C .2  2 /2 

 1.

If 0 < 22 22  12 21 , then we have kT x  T yk2 kx  yk2

D

22 22 . 12 21 1

.1 C

22 /.1 12

C

 1 /2 C .2  2 /2

22 /Œ.1

21

 1 /2 C .2  2 /2 

 1.

Therefore, T is nonexpansive. If we take x D . 12 ; 1/ and y D .1; 12 /, then T x D . 25 ; 15 /, T y D . 51 ; 25 / and hT x  T y; x  yi D 

1 2 < D kT x  T yk2 . 5 25

Therefore, T is not firmly nonexpansive. The following property of firmly nonexpansive operators (cf. [22, Lemma 2.4 (iv)]) is often used in applications.

70

2 Algorithmic Operators

Corollary 2.2.9. Let T W X ! H be an operator with a fixed point and  2 .0; 2. If T is firmly nonexpansive, then its relaxation T is 2 -strongly quasi nonexpansive, i.e., kT x  zk2  kx  zk2 

2 kT x  xk2 

(2.47)

for all x 2 X and z 2 Fix T . Proof. It follows from the first part of Theorem 2.2.5 that a firmly nonexpansive operator having a fixed point is a cutter. Therefore, T is 2 -strongly quasi nonexpansive (see Theorem 2.1.39). t u

2.2.2 Relationships Between Firmly Nonexpansive and Nonexpansive Operators One can find in the literature several equivalent definitions of firmly nonexpansive operators. The properties of these operators were studied by Zarantonello [357, Sect. 1], Bruck [49, Sects. 2 and 3], Rockafellar [299], Bruck and Reich [51, Sect. 1], Goebel and Reich [185, Chap. 1, Sect. 11], Reich and Shafrir [296], Goebel and Kirk [184, Chap. 12], Bauschke and Borwein [22, Sects. 2 and 3], Byrne [56, Sect. 2], and by Crombez [127, Sect. 2]. The class of firmly nonexpansive operators is included in the class of nonexpansive ones (see Theorem 2.2.4). Further important relationships between these two classes are also useful for the investigation of firmly nonexpansive operators. These relationships are given in the following theorem. Theorem 2.2.10. Let T W X ! H. Then the following conditions are equivalent: (i) (ii) (iii) (iv) (v)

T is firmly nonexpansive. T is nonexpansive for any  2 Œ0; 2. T has the form T D 12 .S C Id/; where S W X ! H is a nonexpansive operator. Id T is firmly nonexpansive. For all x; y 2 X it holds kT x  T yk2  kx  yk2  k.x  T x/  .y  T y/k2 .

(2.48)

(vi) For all x; y 2 X and for any ˛  0 it holds kT x  T yk  k˛.x  y/ C .1  ˛/.T x  T y/k . Proof. The equivalence (i),(iv) in Theorem 2.2.10 is obvious, because both conditions can be written in the form hT x  T y; .x  T x/  .y  T y/i  0 for all x; y 2 X . Nevertheless, we prove the following relations among (i)-(vi): (i) ) (ii) ) (iii) ) (iv) ) (v) ) (i) , (vi).

2.2 Firmly Nonexpansive Operators

71

(i))(ii) Let T be firmly nonexpansive and x; y 2 X . By the definition of a firmly nonexpansive operator, the Cauchy–Schwarz inequality and the nonexpansivity of T (see Theorem 2.2.4), we have kT x  T yk2 D kT x C .1  /x  T y  .1  /yk2 D k.T x  T y/ C .1  /.x  y/k2 D 2 .kT x  T yk2  hT x  T y; x  yi/ C.2  2 /hT x  T y; x  yi C .1  /2 kx  yk2  .2  2 /hT x  T y; x  yi C .1  /2 kx  yk2  .2  2 / kT x  T yk kx  yk C .1  /2 kx  yk2  .2  2 / kx  yk2 C .1  /2 kx  yk2 D kx  yk2 , i.e., T is nonexpansive. (ii))(iii) This implication is obvious. It suffices to take S D T for  D 2. (iii))(iv) Let S be nonexpansive, T WD 12 .S C Id/ and G WD Id T . Then we have G D 12 .Id S / and kGx  Gyk2 D hGx  Gy; x  yi C hGx  Gy; .Gx  Gy/  .x  y/i D hGx  Gy; x  yi 1 C h.S x  Sy/  .x  y/; .S x  Sy/ C .x  y/i 4 1 D hGx  Gy; x  yi C .kS x  Syk2  kx  yk2 / 4  hGx  Gy; x  yi, for all x; y 2 X . (iv))(v) Let G WD Id T be firmly nonexpansive. Then, for all x; y 2 X we have kT x  T yk2 C k.Id T /x  .Id T /yk2  kT x  T yk2 C h.Id T /x  .Id T /y; x  yi D kT x  T yk2  hT x  T y; x  yi C kx  yk2 D hT x  T y; .x  T x/  .y  T y/i C kx  yk2  kx  yk2 , i.e., (2.48) holds.

72

2 Algorithmic Operators

(v))(i) Let x; y 2 X . If (2.48) holds, then, by the properties of the inner product, we have kT x  T yk2  kx  yk2  k.x  y/  .T x  T y/k2 D  kT x  T yk2 C 2hT x  T y; x  yi, i.e., T is firmly nonexpansive. (i),(vi) Let x; y 2 X . The function h W RC ! RC defined by h.˛/ D

1 k˛.x  y/ C .1  ˛/.T x  T y/k2 2

is convex as a composition of the convex function f ./ D 12 kk2 and an affine function A W R ! H, A.˛/ D ˛.x  y/ C .1  ˛/.T x  T y/. Note that h.0/ D 1 x  T yk2 , h is differentiable and 2 kT h0 .0/ D hT x  T y; .x  y/  .T x  T y/i. Since h is convex, we have h.0/  h.˛/ ” h0 .0/  0 for all ˛  0, i.e., kT x  T yk2  k˛.x  y/ C .1  ˛/.T x  T y/k2 ” hT x  T y; x  yi  kT x  T yk2 which completes the proof.

t u

The same kind of correspondences between firmly nonexpansive operators and nonexpansive ones (the equivalence (i),(iii) in Theorem 2.2.10) and between cutters and quasi-nonexpansive operators (Corollary 2.1.33 (ii)) explain the name firmly quasi-nonexpansive operators for cutters (see [346, page 624]). Condition (ii) in Theorem 2.2.10 can be formulated equivalently as follows: (ii’) T2 WD 2T  Id is nonexpansive. The nonexpansivity of T2 and Lemma 2.1.12 (i) yield the nonexpansivity of T for all  2 Œ0; 2, because T D .1  2 / Id C 2 T2 . Moreover, the assumption that T2 is nonexpansive is sufficient in the implication (ii))(iii) as follows from the proof. Now we present a series of corollaries of Theorem 2.2.10. Corollary 2.2.11. Let T W X ! H. The operator T is firmly nonexpansive if and only if its relaxation T is firmly nonexpansive for all  2 Œ0; 1. Proof. Let T be firmly nonexpansive and  2 Œ0; 1. By the implication (i))(iii) in Theorem 2.2.10 we obtain

2.2 Firmly Nonexpansive Operators

73

1  T D .1  / Id C .Id CS / D ŒId C.1  / Id CS  2 2 for a nonexpansive operator S . Note that .1/ Id CS is nonexpansive as a convex combination of nonexpansive operators (see Lemma 2.1.12 (ii)). Therefore, T is firmly nonexpansive by the implication (iii))(i) in Theorem 2.2.10. The sufficiency of the condition is obvious. t u Corollary 2.2.12. Let U W X ! H and  2 Œ0; 2. Then U is -RFNE if and only if U is -RFNE for all  2 Œ; 2. Proof. Let U WD T D Id C.T  Id/, where T W X ! H is a firmly nonexpansive operator, and  2 Œ; 2. It is easy to see that U D Id C.T=  Id/: The corollary follows now from the fact that T= is firmly nonexpansive (see Corollary 2.2.11). t u Corollary 2.2.13. Let X  H be a closed convex subset and S W X ! H. The following conditions are equivalent: (i) S is nonexpansive, (ii) S D 2T  Id, where T W X ! H is a firmly nonexpansive operator. Proof. (ii))(i) Let S WD 2T  Id for a firmly nonexpansive operator T . It follows from the implication (i))(ii) in Theorem 2.2.10 that S is nonexpansive. (i))(ii) Let S be nonexpansive and T WD 12 .S C Id/. By the implication (iii))(i) in Theorem 2.2.10 the operator F is firmly nonexpansive. Furthermore, S D 2T  Id. t u Definition 2.2.14. (cf. [127, Definition 2.1]) We say that an operator U W X ! H is -firmly nonexpansive (-FNE), where  > 0, if kUx  Uyk2  kx  yk2   k.x  Ux/  .y  Uy/k2 . Vasin and Ageev call a -firmly nonexpansive operator for  2 .0; 1/, a pseudocontractive operator (see [333, Definition 2.5]). In [127, Theorem 2.3] several equivalent conditions for U to be -FNE are presented. By the equivalence (i),(v) of Theorem 2.2.10, an operator is firmly nonexpansive if and only if it is 1-firmly nonexpansive. Note, however, that there is a difference between a -RFNE operator and a -FNE operator. Below, we present the relationship between these two notions. Corollary 2.2.15. Let  2 .0; 2/. An operator U W X ! H is -relaxed firmly -firmly nonexpansive, i.e., nonexpansive if and only if U is 2 

74

2 Algorithmic Operators

kUx  Uyk2  kx  yk2 

2 k.x  Ux/  .y  Uy/k2 

for all x; y 2 X . If, furthermore, Fix U ¤ ;, then kUx  zk2  kx  zk2  for all x 2 X and z 2 Fix U , i.e., U is

2 kUx  xk2 

2  -strongly

quasi-nonexpansive.

Proof. Let U WD T for a firmly nonexpansive operator T and x; y 2 X . Applying the properties of the inner product we get for G WD Id T kUx  Uyk2 D k.1  /x C T x  .1  /y  T yk2 D kx  y  .Gx  Gy/k2 D kx  yk2  2hx  y; Gx  Gyi C 2 kGx  Gyk2 . Since x  Ux D x  T x D Gx, the equalities above yield kUx  Uyk2  kx  yk2 C

2 k.x  Ux/  .y  Uy/k2 

D kUx  Uyk2  kx  yk2 C .2  / kGx  Gyk2 D 2.hx  y; Gx  Gyi  kGx  Gyk2 /. The first part of the corollary follows now from the equivalence (i),(iv) in Theorem 2.2.10, and now the other part follows directly from the definition of an ˛-strongly quasi-nonexpansive operator. t u Definition 2.2.16. Let ˛ 2 .0; 1/. We say that an operator T W X ! H is ˛-averaged or, shortly, averaged (AV) if T D .1  ˛/ Id C ˛S holds for a nonexpansive operator S W X ! H. Averaged operators were studied, e.g., by Mann [252], Krasnosel’ski˘ı [238], Baillon et al. [14, Sect. 2]. In [56, Sect. 2], Byrne gives relationships between averaged operators and inverse strongly monotone operators, i.e., operators G W X ! H such that hGx  Gy; x  yi   kGx  Gyk2 for all x; y 2 X and for some constant  > 0. Definition 2.2.16 states that an operator is averaged if and only if it is an underrelaxation of a nonexpansive operator.

2.2 Firmly Nonexpansive Operators

75

Corollary 2.2.17. Let  2 .0; 2/ and ˛ D =2. An operator U W X ! H is -relaxed firmly nonexpansive if and only if U is ˛-averaged. Proof. ()) Let T W X ! H be firmly nonexpansive and U WD T D .1  / Id CT . By the implication (i))(iii) in Theorem 2.2.10 we have T D 12 .S C Id/ for a nonexpansive operator S W X ! H. Hence, U D .1  ˛/ Id C ˛S , i.e., U is ˛-averaged. (() Let U be ˛-averaged, i.e., U D .1˛/ Id C ˛S for a nonexpansive operator S and for ˛ D =2 2 .0; 1/. By Corollary 2.2.13 we have U D .1  ˛/ Id C ˛.2T  Id/ D .1  2˛/ Id C2˛T for a firmly nonexpansive operator T . Hence, U is the -relaxation of T W X ! H with  D 2˛ 2 .0; 2/. u t Corollary 2.2.18. Let G W X ! H. Then G is firmly nonexpansive if and only if Id G is averaged for any  2 .0; 2/. Proof. Necessity. Let G be firmly nonexpansive. We have Id G D .1  =2/ Id C.=2/Œ2.Id G/  Id. By the implications (i))(iv) and (i))(ii) in Theorem 2.2.10 the operator 2.Id G/  Id is nonexpansive. Consequently, the operator Id G is averaged. Sufficiency. Let Id G be averaged for any  2 .0; 2/. Then Id G is nonexpansive for any  2 .0; 2/ and Id 2G is nonexpansive as a limit of nonexpansive operators. Now, it follows from the implication (ii))(i) in Theorem 2.2.10 that G is firmly nonexpansive. t u Corollary 2.2.19. Let U W X ! H and  2 .0; 2. The operator U is -relaxed firmly nonexpansive if and only if its relaxation U is firmly nonexpansive for  2 Œ0; 1 . Proof. Take U WD T for a firmly nonexpansive operator T W X ! H. Then the claim follows from the equality U1 D T (see Remark 2.1.3) and Corollary 2.2.11. The converse implication is obvious. t u The following corollary shows that the family of firmly nonexpansive operators is closed under convex combination. Corollary 2.2.20. Let Ti W X ! H, i 2 I WD f1; 2; : : : ; mg, be Pfirmly nonexpansive and w D .!1 ; !2 ; : : : ; !m / 2 m . Then the operator T WD i 2I !i Ti is firmly nonexpansive. P Proof. Let T WD i 2I !i Ti . By the implication (i))(iii) in Theorem 2.2.10, we have Ti D 12 .Si C Id/ for a nonexpansive operator Si , i 2 I . Observe

76

2 Algorithmic Operators

Fig. 2.10 Basic relationships among algorithmic operators

Fix

2

Fix =(2- )/ =(2- )/

=2

P that T D 12 .S C Id/ for S WD i 2I !i Si . By Lemma 2.1.12 (i), the operator S is nonexpansive. The corollary follows now from the implication (iii))(i) in Theorem 2.2.10. t u In Fig. 2.10 we shortly present important relationships among the FNE operators, cutters, QNE operators SQNE operators and AV operators, which are proved in Sects. 2.1.3, 2.2.1 and 2.2.2. In Fig. 2.10, T W X ! H and U WD I D Id C.T  Id/ is its -relaxation, where  2 .0; 2/. We will extend this figure in Sect. 3.9.

2.2.3 Further Properties of the Metric Projection The basic facts concerning firmly nonexpansive operators presented in the previous section yield further properties of the metric projection. Theorem 2.2.21. Let C  H be a nonempty closed convex subset and PC W H ! H be the metric projection onto C . Then the operator PC is: (i) (ii) (iii) (iv) (v)

Idempotent, consequently Fix PC D C , A cutter, Firmly nonexpansive, Monotone and nonexpansive, Averaged.

Proof. (i) The property follows directly from the definition of the metric projection. (ii) It follows from (i) and from the characterization of the metric projection (see Theorem 1.2.4) that hz  PC x; x  PC xi  0 for all x 2 H and for all z 2 C D Fix PC , which means that PC is a cutter. (iii) The property follows directly from (i), (ii) and from Theorem 2.2.5 (ii).

2.2 Firmly Nonexpansive Operators

77

(iv) By Theorem 2.2.4, any firmly nonexpansive operator is monotone and nonexpansive. Therefore, the property follows from (iii). (v) By the firm nonexpansivity of PC and by the implication (i))(iii) in Theorem 2.2.10, we can write PC D 12 .S C Id/ for a nonexpansive operator S W X ! H. Hence, PC is averaged. t u Definition 2.2.22. Let C  H be a nonempty closed convex subset. We call a relaxation of the metric projection PC W H ! C a relaxed metric projection onto the subset C and we denote it by PC; or, shortly, by P . If  < 1, then P is called an under-projection. If  > 1, then P is called an over-projection. If  D 2, then P is called the reflection. We have PC; D P D Id C.PC  Id/. Corollary 2.2.23. Let C  H be a nonempty closed convex subset,   0 and P W H ! H be a relaxed metric projection. Then (i) P is a nonexpansive operator for all  2 Œ0; 2, (ii) Fix P D C for all  > 0, (iii) For all x 2 H, z 2 C and  2 .0; 2 the following inequality holds kP x  zk2  kx  zk2  Consequently, P is

2 -strongly 

2 kP x  xk2 . 

(2.49)

quasi-nonexpansive for all  2 .0; 2.

Proof. Part (i) follows from the equivalence (i),(ii) in Theorem 2.2.10, because PC is firmly nonexpansive (see Theorem 2.2.21 (iii)). Part (ii) is obvious, because Fix PC D C . Part (iii) follows now from Corollary 2.2.9. t u Corollary 2.2.24. Let C  H be a nonempty closed convex subset and x; y 2 H. Then kPC x  PC yk2  kx  yk2  k.PC x  x/  .PC y  y/k2  kx  yk  .kPC x  xk  kPC y  yk/ . 2

2

(2.50) (2.51)

In particular, kPC x  zk2  kx  zk2  kPC x  xk2

(2.52)

for all x 2 H and z 2 C . Consequently, the metric projection PC W H ! C is strongly quasi-nonexpansive. Proof. By Theorem 2.2.21 (iii), the metric projection is firmly nonexpansive. Therefore, inequalities (2.50) and (2.51) follow directly from the implication (i))(v) in Theorem 2.2.10 and from the Cauchy–Schwarz inequality. The second part follows directly from Theorem 2.2.21 (i). t u

78

2 Algorithmic Operators

Fig. 2.11 Function f .x/ D kPX .x C ˛u/  xk is nondecreasing

x + α2u x + α1u PX (x + α2u) PX (x + α1u) X

u x

Corollary 2.2.25. Let T W X ! H and  2 .0; 2/. If T is a cutter, then for any x 2 X and z 2 Fix T the following estimations hold kPX T x  zk2  kx  zk2  .2  / kT x  xk2 and kPX T x  zk2  kx  zk2  Consequently, the operator PXT W X ! X is

2 kPX T x  xk2 :  2 -strongly 

(2.53)

quasi-nonexpansive.

Proof. Note that PX is a nonexpansive operator and that Fix PX \ Fix T D X \ Fix T D Fix T ¤ ;. Therefore, the corollary follows from Theorem 2.1.51.

t u

The following corollary will be useful in further parts of the book (see also [327, Lemma 2] and [172, Lemma 1] for related results). Corollary 2.2.26. Let x 2 X; u 2 H and 0  ˛1 < ˛2 . Then the following inequality holds kPX .x C ˛2 u/  xk2  kPX .x C ˛1 u/  xk2 C kPX .x C ˛2 u/  PX .x C ˛1 u/k2 .

(2.54)

Consequently, the function f W RC ! RC , f .˛/ WD kPX .x C ˛u/  xk is nondecreasing. Corollary 2.2.26 is illustrated in Fig. 2.11. Proof. Inequality (2.54) is obvious for ˛1 D 0. Let now ˛1 > 0. Take y WD x C ˛2 u,  z WD x C ˛1 u and  D ˛˛12 . Then we have  2 .0; 1/ and .x  z/ D  1 .y  z/. Now inequality (2.54) can be written in the form kPX y  xk2  kPX z  xk2 C kPX y  PX zk2 .

(2.55)

2.2 Firmly Nonexpansive Operators

79

The characterization of the metric projection (see Theorem 1.2.4) and its monotonicity (see Theorem 2.2.21 (iv)) yield hx  PX z; PX y  PX zi D hx  z; PX y  PX zi C hz  PX z; PX y  PX zi 

 hy  z; PX y  PX zi  0, 1

t i.e., hx  PX z; PX y  PX zi  0, which is equivalent to (2.55), by Lemma 1.2.5. u Let C  H be convex. Define the distance function d.; C / W H ! R by d.x; C / D infy2C kx  yk. It follows from the continuity of the norm and from the definition of the metric projection that d.x; C / D d.x; cl C / D kx  Pcl C xk . Therefore, we suppose without loss of generality that C is closed. It turns out that the functions d.; C / and d 2 .; C / are convex and differentiable. Lemma 2.2.27. Let C  H be a closed convex subset. Then the function f W H ! R, f .x/ WD 12 d 2 .x; C / is differentiable and Df .x/ D x  PC x for all x 2 H. Proof. (cf. [167, Proposition 2.2] and [209, Chap. IV, Example 4.1.6]) Let x; h 2 H. It follows from the definition of the metric projection and from the properties of the inner product that f .x C h/  f .x/  hx  PC x; hi 1 1 kx C h  PC .x C h/k2  kx  PC xk2  hx  PC x; hi 2 2 1 1  kx C h  PC xk2  kx  PC xk2  hx  PC x; hi 2 2 1 D khk2 . 2

D

Similarly, by the definition of the metric projection, the Cauchy–Schwarz inequality and the nonexpansivity of the metric projection, we obtain f .x C h/  f .x/  hx  PC x; hi 1 1 kx C h  PC .x C h/k2  kx  PC xk2  hx  PC x; hi 2 2 1 1  kx C h  PC .x C h/k2  kx  PC .x C h/k2  hx  PC x; hi 2 2 1 D khk2 C hPC x  PC .x C h/; hi 2

D

80

2 Algorithmic Operators

1 khk2  kPC x  PC .x C h/k  khk 2 1 1  khk2  khk2 D  khk2 . 2 2



Now we see that 1 1  khk2  .f .x C h/  f .x/  hx  PC x; hi/  khk2 . 2 2 Consequently, f .x C h/ D f .x/ C hx  PC x; hi C o.khk/. Therefore, f is differentiable and Df .x/ D x  PC x.

t u

Lemma 2.2.28. Let C  H be a closed convex subset. The function h W H ! R, h.x/ WD d.x; C / is convex and differentiable for all x … C and Dh.x/ D

x  PC x . kx  PC xk

(2.56)

Proof. Since h.x/ D infy2C kx  yk, the convexity of h follows from the fact that the function p W H  H ! H, p.x; y/ WD kx  yk is convex (as a composition of a linear function .x; y/ ! x  y and a convex function z ! kzk) and from the fact pthat for a convex function p, the function infy2C p.; y/ is convex. Since h D d 2 .; C /, the differentiability of h as well as equality (2.56) for x … C z follow from Lemma 2.2.27 and from the formula D.kzk/ D kzk for z ¤ 0. t u Corollary 2.2.29. Let C  H be a closed convex subset. Then the function f W H ! R, f .x/ WD 12 d 2 .x; C / is convex. Proof. The function f is convex as a composition f D g ı h of a convex function h WD d.; C / and of a convex and increasing function g W RC ! R, g.t/ WD 12 t 2 . t u

2.2.4 Metric Projection onto a Closed Subspace Let V  H be a closed linear subspace. Since V is convex, the metric projection PV is well defined. The theorem below states some properties of PV . In particular, the first part of the theorem states that the metric projection onto V is equal to the orthogonal projection onto V . Theorem 2.2.30. Let V  H be a closed subspace and x 2 H, y 2 V . Then (i) y D PV x if and only if hx  y; zi D 0 for all z 2 V , (ii) PV is a bounded linear operator and kPV k D 1,

2.2 Firmly Nonexpansive Operators

81

(iii) PV is self-adjoint, (iv) Id D PV C PV ? . Proof. (i) Necessity. Let y WD PV x. By the characterization of the metric projection (see Theorem 1.2.4), hx  y; z  yi  0 for all z 2 V . Suppose that hx  y; w  yi < 0 for some w 2 V . Let u WD 2y  w. Then u 2 V because V is a linear subspace and we have hx  y; u  yi D hx  y; y  wi > 0. This contradiction shows that hx  y; z  yi D 0 for all z 2 V . If we take z WD 0 2 V in the latter equality, we obtain hx  y; yi D 0. Hence, hx  y; zi D 0 for all z 2 V . Sufficiency. Let hx  y; zi D 0 for all z 2 V . Taking z WD y 2 V we obtain in particular hx  y; yi D 0. Hence, hx  y; z  yi D 0 for all z 2 V . By the characterization of the metric projection (see Theorem 1.2.4), we have y D PV x. (ii) Let x1 ; x2 2 H, ˛1 ; ˛2 2 R, y1 WD PV x1 , y2 WD PV x2 and x WD ˛1 x1 C ˛2 x2 , y WD ˛1 y1 C ˛2 y2 . We show that y D PV x. By (i) we have hx  y; zi D h˛1 .x1  y1 / C ˛2 .x2  y2 /; zi D ˛1 hx1  y1 ; zi C ˛2 hx2  y2 ; zi D 0, for all z 2 V , i.e., y D PV x. Since PV is nonexpansive, it is bounded. Furthermore, kPV xk D kPV x  PV 0k  kx  0k D kxk for all x 2 H and kPV xk D kxk for x 2 V . Hence, kPV k D 1. (iii) Let x; u 2 H. It follows from (i) that hx; PV ui D hPV x; PV ui and hu; PV xi D hPV u; PV xi. By the symmetry of the inner product hPV x; ui D hx; PV ui; i.e., PV is self-adjoint. (iv) Let x 2 H. By (i), we have x  PV x 2 V ? and x  PV x D PV ? x. Since x D PV x C .x  PV x/, it holds x D PV x C PV ? x. t u

82

2 Algorithmic Operators

Corollary 2.2.31. Let V  H be a closed subspace and x 2 H. Then hPV x; xi D kPV xk2 . Proof. Since PV is self-adjoint (see Theorem 2.2.30 (iii)), we have hPV x; ui D hx; PV ui for all u 2 H. If we take u WD PV x we obtain the desired property. t u Corollary 2.2.32. A bounded linear operator is an orthogonal projection if and only if it is idempotent and self-adjoint. Proof. The necessity follows from Theorems 2.2.21 (i) and 2.2.30 (iii). Let now T W H ! H be idempotent and self-adjoint. Let V WD T .H/. It is clear that V D Fix T and that V is a closed subspace. Now we show that T D PV . Let x 2 H and z 2 V . Then hT x; zi D hx; T zi D hx; zi, t u

i.e., hT x  x; zi D 0. Theorem 2.2.30 (i) implies now that T D PV .

2.2.5 Metric Projection onto a Closed Affine Subspace Let A  H be a closed affine subspace and a 2 A. Then A  a is a closed linear subspace. In order to show some properties of the metric projection PA we apply Theorem 2.2.30 together with PA x D PAa .x  a/ C a

(2.57)

(see Lemma 1.2.6). Theorem 2.2.33. Let A  H be a closed affine subspace and x; u; v; w 2 H, a; y 2 A. Then (i) (ii) (iii) (iv) (v) (vi)

y D PA x if and only if hx  y; z  yi D 0 for all z 2 A, PA u  PA v D PAa .u  v/ D PA .u  v/  PA 0, hPA u  PA v; wi D hu  v; PAa wi D hu  v; PA w  PA 0i, hPA u  PA v; u  vi D kPA u  PA vk2 , ku  vk2 D kPA u  PA vk2 C k.PA u  u/  .PA v  v/k2 , PA is an affine operator.

Proof. (i) Since A  a is a linear subspace, v 2 A  a if and only if v D z  y, for some z 2 A: By (2.57) and Theorem 2.2.30 (i), we have y D PA x , y  a D PAa .x  a/ , hy  a  .x  a/; z  yi D 0 for any z 2 A.

2.2 Firmly Nonexpansive Operators

83

(ii) By (2.57) and the linearity of PAa , we have PA u  PA v D PAa .u  a/ C a  .PAa .v  a/ C a/ D PAa .u  v/ D PAa .u  v  a/  PAa .a/ D PA .u  v/  PA 0. (iii) Since PAa is self-adjoint, (2.57) and (ii) yield hPA u  PA v; wi D hPAa .u  a/  .PAa .v  a/; wi D hu  v; PAa wi D hu  v; PAa .w  a/  PAa .a/i D hu  v; PA w  PA 0i. (iv) Property (i) yields hPA u  u; PA u  PA vi D 0 and hPA v  v; PA u  PA vi D 0 Therefore, h.PA u  u/  .PA v  v/; PA u  PA vi D 0, i.e., hPA u  PA v; u  vi D kPA u  PA vk2 . (v) It follows from the properties of the inner product and from property (iv) that k.PA u  u/  .PA v  v/k2 D kPA u  PA vk2 C ku  vk2  2hPA u  PA v; u  vi D ku  vk2  kPA u  PA vk2 . (vi) Let  2 R. Since A  a is a closed subspace, (2.57) and Theorem 2.2.30 yield PA ..1  /u C y/ D PAa ..1  /.u  a/ C .y  a// C a D .1  /.PAa .u  a/ C a/ C .PAa .y  a/ C a/ D .1  /PA u C PA y which completes the proof. t u

84

2 Algorithmic Operators

2.2.6 Properties of Relaxed Firmly Nonexpansive Operators In this section we present relationships among families of relaxed firmly nonexpansive operators, contractions, averaged operators and strongly quasi-nonexpansive operators. Furthermore, we give properties of relaxed firmly nonexpansive operators which are used in many constructions of algorithmic operators. Theorem 2.2.34. An ˛-contraction is .1 C ˛/-relaxed firmly nonexpansive. Proof. Let T W X ! H be an ˛-contraction, i.e., kT x  T yk  ˛ kx  yk for all 1˛ x; y 2 X , where ˛ 2 .0; 1/. Let U WD 1C2 ˛ T  1C ˛ Id, or, equivalently, T D i.e.. T is

1C ˛ 2 -averaged.

1˛ 1C ˛ UC Id , 2 2

By the convexity of the norm and the nonexpansivity of T ,

    2 1˛  .T x  T y/  .x  y/ kUx  Uyk D   1C ˛ 1C ˛ 1˛ 2 kT x  T yk C kx  yk 1C ˛ 1C ˛ 2˛ 1˛  kx  yk C kx  yk 1C ˛ 1C ˛ 

D kx  yk , i.e., U is nonexpansive. Therefore, T is .1 C ˛/-relaxed firmly nonexpansive as a . 1C2 ˛ /-averaged operator (see Corollary 2.2.17). t u The next results show that a family of relaxed firmly nonexpansive operators is closed under convex combination and under composition. Theorem 2.2.35. Let i 2 Œ0; 2 and UP i W X ! H be i -relaxed firmly nonexpanm sive, i 2 I WD f1; 2; : : : ; mg, U WD !m / 2 m . i D1 !i Ui for w D .!1 ; : : : ;P m Then the operator U is -relaxed firmly nonexpansive, where  D j D1 !j j . Consequently, U is strictly relaxed firmly nonexpansive if i 2 .0; 2/ for some i 2 I and the corresponding weight !i > 0. Proof. Let Ui WD Id Ci .Ti  Id/, where Ti W X ! H are firmly P nonexpansive, i 2 Œ0; 2, i 2 I , and w D .!1 ; : : : ; !m / 2 m . It is clear that  WD m j D1 !j j 2 Œ0; 2. For  D 0 the claim is obvious, because U D Id in this case. Let now  2 .0; 2. Since m X i D1

!i i Pm D 1, j D1 !j j

2.2 Firmly Nonexpansive Operators

85

the operator T WD

m X i D1

!i i Pm Ti j D1 !j j

is firmly nonexpansive as a convex P combination of firmly nonexpansive operators Ti (see Corollary 2.2.20). Let U WD m i D1 !i Ui . Then we have U D

m X

!i ŒId Ci .Ti  Id/

i D1

D Id C

m X

!i i .Ti  Id/

i D1

0 D Id C @

m X

1 !j j A

j D1

m X i D1

! m X !i i !i i Pm Pm Ti  Id j D1 !j j j D1 !j j i D1

D Id C.T  Id/ and, consequently, U is -relaxed firmly nonexpansive. The second part of the theorem is obvious. u t Corollary 2.2.36. A convex combination of averaged operators is an averaged operator. Proof. It suffices to apply Corollary 2.2.17 to Theorem 2.2.35.

t u

Theorem 2.2.37. Let T; U W X ! X and ;  2 Œ0; 2. If T is -RFNE and U is -RFNE, then the composition V WD U T is  -RFNE, with 8 ˆ 0 if  D 0 and  D 0 ˆ ˆ ˆ ˆ if .2  /.2  / D 0 <2  D 4. C   / 2 ˆ otherwise. D ˆ ˆ  1  4   ˆ ˆ C / C1 . : 2 2 (2.58) Proof. If  D 0 or  D 0, then T D Id or U D Id, respectively, and the claim is obvious, because the operator Id is 0-RFNE. If  D 2 or  D 2, then T and U are nonexpansive (see Theorem 2.2.10 (ii)) and U T is nonexpansive as a composition of nonexpansive operators. Therefore, U T is 2-RFNE (see Corollary 2.2.13). Let now ;  2 .0; 2/ and x; y 2 H. Denote a1 WD T x  x, a2 WD T y  y, b1 WD U T x  T x and b2 WD U T y  T y. It is clear that y  x D T y  T x C a1  a2 .

(2.59)

86

2 Algorithmic Operators

By Corollary 2.2.3, we have hy  x; a1 i C hx  y; a2 i 

1 ka1  a2 k2 

and hT y  T x; b1 i C hT x  T y; b2 i 

1 kb1  b2 k2 . 

Therefore, the properties of the inner product, equality (2.59) and Lemma 2.1.45 yield hy  x; U T x  xi C hx  y; U T y  yi  D hy  x; a1 C b1 i C hx  y; a2 C b2 i 

1 .k.U T x  x/  .U T y  y/k2 / 

1 .k.a1 C b1 /  .a2 C b2 /k2 / 

D hy  x; a1 i C hx  y; a2 i C hT y  T x; b1 i C hT x  T y; b2 i Cha1  a2 ; b1  b2 i  

1 .k.a1 C b1 /  .a2 C b2 /k2 / 

1 1 ka1  a2 k2 C kb1  b2 k2 C ha1  a2 ; b1  b2 i  

1  .k.a1 C b1 /  .a2 C b2 /k2        1 1 1 1 2 2 2 ha1  a2 ; b1  b2 i   D ka1  a2 k C kb1  b2 k C 1       s s 2  1 1  1 1    .a1  a2 /   .b1  b2 /  0. D       Now it follows from Corollary 2.2.3 that U T is  -RFNE.

t u

Remark 2.2.38. Because of Corollary 2.2.17, Theorem 2.2.37 can be stated equivalently in terms of averaged operators: if T is ˛-averaged and U is ˇ-averaged, where ˛; ˇ 2 .0; 1/, then U T is ı-averaged, with ˛ C ˇ  2˛ˇ ı WD . (2.60) 1  ˛ˇ This result is due to Ogura and Yamada (see [273, Theorem 3 (b)]). The fact that a composition of averaged operators T WD .1  ˛/ Id C ˛R and U WD .1  ˇ/ Id CˇS is averaged follows also from the following identity (cf. [56, Lemma 2.2 and Proposition 2.1]) U T D .1  ˛/.1  ˇ/ Id C.˛ C ˇ  ˛ˇ/Œ

.1  ˇ/˛ ˇ RC S T , ˛ C ˇ  ˛ˇ ˛ C ˇ  ˛ˇ

2.2 Firmly Nonexpansive Operators

87

and from the fact that the family of nonexpansive operators is closed under compositions and convex combinations (see Lemma 2.1.12). Note, however, that [273, Theorem 3 (b)] is stronger than the result mentioned above, because ı < ˛ C ˇ  ˛ˇ for ˛; ˇ 2 .0; 1/ and ı given by (2.60). It follows from Corollary 2.2.17 that the result of Ogura and Yamada is equivalent to Theorem 2.2.37 with ;  2 .0; 2/. Moreover, the proof of this theorem differs from the proof of [273, Theorem 3 (b)]. Note that the property of composition of relaxed cutters with a common fixed point, expressed in Theorem 2.1.46 and the property of compositions of relaxed firmly nonexpansive operators presented in Theorem 2.2.37 are similar. Therefore, it is quite natural that the proofs of both theorems are similar. But Theorem 2.1.46 is no special case of Theorem 2.2.37 because a cutter needs not to be firmly nonexpansive, even if it is nonexpansive (see Example 2.2.8). An equivalent formulation of the following result can be found in [349, Lemma 1]. Corollary 2.2.39. Let T; U W H ! H be firmly nonexpansive. Then the composition V WD U T is 43 -relaxed firmly nonexpansive. Consequently, V is firmly nonexpansive for all  2 Œ0; 34  and nonexpansive for all  2 Œ0; 32 . If, furthermore, V has a fixed point, then V is strongly quasi-nonexpansive for all  2 .0; 32 /. Proof. If we take  D  D 1 in Theorem 2.2.37, we obtain that V is 43 -relaxed firmly nonexpansive. Recall that .V / D V (see Remark 2.1.3). Corollary 2.2.19 yields the firm nonexpansivity of V for all  2 Œ0; 34 . By the implication (i))(ii) in Theorem 2.2.10, V is nonexpansive for all  2 Œ0; 32 . Now let Fix V ¤ ; and  2 .0; 32 /. Then V is strongly quasi-nonexpansive, by Corollary 2.2.9. t u Yamada et al. also proved that, for any  > 32 , there exist firmly nonexpansive operators T; U such that V is not nonexpansive, where V WD U T (see [349, Remark 1 (b)]). This means that the constant 32 is optimal in Corollary 2.2.39. Remark 2.2.40. Let T; U W H ! H be firmly nonexpansive having a common fixed point. Then it follows from Corollaries 2.2.39 and 2.2.15 that U T is 12 -strongly quasi nonexpansive. A special case of this property was proved in [152, Proposition 1] for T; U being orthogonal projections onto subspaces of H. Corollary 2.2.41. Let T W H ! H be firmly nonexpansive and  2 Œ0; 2. If V is a 4 closed affine subspace, then the operator U WD .1  /PV C PV T is 4 -relaxed firmly nonexpansive. Proof. Let V be a closed affine subspace. By Theorem 2.2.33 (vi), the operator PV is affine, consequently, .1  /PV C PV T D PV T . 4 -relaxed firmly nonexpansive, Now it follows from Theorem 2.2.37 that U is 4 because the metric projection is firmly nonexpansive. t u

88

2 Algorithmic Operators

A weaker formulation of Corollary 2.2.41 can be found in [349, Lemma 2], where T WD PC for a closed convex subset C . Theorem 2.2.42. Let Ti W X ! X be i -relaxed firmly nonexpansive, where ˛i 2 Œ0; 2, i 2 I . Then the composition Sm WD Tm Tm1 : : : T1 is m -relaxed firmly nonexpansive, where m D 0 if i D 0 for all i 2 I , m D 2 if i D 2 for at least one i 2 I and m D

2 1 21

C

2 22

C ::: C

m 2m

1

,

(2.61)

C1

otherwise. Moreover, 2m maxi 2I i 2m mini 2I i  m  , .m  1/ mini 2I i C 2 .m  1/ maxi 2I i C 2

(2.62)

consequently, m < 2 if i < 2 for all i 2 I . Proof. Let i D 0 for all i 2 I . In this case, Sm D Id, i.e., Sm is 0-relaxed firmly nonexpansive. Let i D 2 for some i 2 I . Then Sm is nonexpansive as a composition of nonexpansive operators, i.e., Sm is 2-RFNE (see Corollary 2.2.13). Let now i 2 Œ0; 2/ for all i 2 I and j > 0 for at least one j 2 I . We prove by induction with respect to m that Sm is m -RFNE, where m is given by (2.61). Note that (2.61) is equivalent to m 1 2 m D C C ::: C . 2  m 2  1 2  2 2  m

(2.63)

10 For m D 2 the above fact follows directly from Theorem 2.2.37. 20 Suppose that, for some m D k, the operator Sm is m -RFNE. We prove that SkC1 is kC1 -RFNE. If kC1 D 0, then TkC1 D Id, SkC1 is a composition of k operators which are relaxed firmly nonexpansive and the claim follows from the induction assumption. Let now kC1 2 .0; 2/, then we have SkC1 D TkC1 Sk , where TkC1 is kC1 -RFNE and Sk is k -RFNE. It follows from Theorem 2.2.37 that SkC1 is  -RFNE, where D

2 k 2k

C

kC1 2kC1

1

, C1

and, together with (2.63), this gives for m D k k  kC1 D C 2 2  k 2  kC1 D

1 2 k kC1 C C ::: C C , 2  1 2  2 2  k 2  kC1

2.2 Firmly Nonexpansive Operators

89

consequently,  D kC1 . We have proved that, for any m 2 N, the operator Sm is m -RFNE, where m is given by (2.61). Now we prove (2.62). By (2.63), we have m

m maxi 2I i mini 2I i  m , 2  mini 2I i 2  m 2  maxi 2I i t u

which is equivalent to (2.62).

A part of the results presented in Theorem 2.2.42 can be found in [122, Lemma 2.2 (iii)], where it was proved that a composition of i -RFNE operators Ti , where 2m maxi 2I i i 2 Œ0; 2, i 2 I , is .m1/ maxi 2I i C2 -SQNE. Corollary 2.2.43. Let Ti W X ! X , i 2 I , be firmly nonexpansive. Then the oper2m ator Sm D Tm : : : T1 is m -relaxed firmly nonexpansive with m D mC1 . Conse1 quently, Sm is m -strongly quasi-nonexpansive. Proof. It suffices to take i D 1; i 2 I , in (2.61). The second part of the corollary follows from Corollary 2.2.9. t u Dye and Reich obtained a result which is a special case of the second part of Corollary 2.2.43 with Ti , i 2 I , being orthogonal projections onto one-dimensional subspace of a Hilbert space (see [152, Theorem on page 109]). Corollary 2.2.44. Let Ti W X ! X be firmly nonexpansive, PSmi WD Ti : : : T1 , i 2 I , and w D .!1 ; : : : ; !m / 2 m . Then the operator S WD i D1 !i Si is -relaxed firmly nonexpansive, where D

m X i D1

!i

2i . i C1

(2.64)

Proof. By Corollary 2.2.43, the operators Si are i -relaxed firmly nonexpansive 2i with i D i C1 . By Theorem 2.2.35, S is -relaxed firmly nonexpansive, where  is given by (2.64). t u The composition of firmly nonexpansive operators needs not to be firmly nonexpansive (see Exercise 2.5.10). Definition 2.2.45. Let T W X ! H,  2 Œ0; 2. The operator R W X ! H, R WD PX T is called a projected relaxation of T . The theorem below gives important properties of the projected relaxation of a firmly nonexpansive operator. Theorem 2.2.46. Let T W X ! H be firmly nonexpansive, R WD PX T , be the projected relaxation of T , where  2 .0; 2/. Then: 4 -relaxed firmly nonexpansive. (i) R is 4 (ii) Fix R D Fix.PX T /.

90

2 Algorithmic Operators

(iii) If Fix.PX T / ¤ ;, then the operator R is kR x  zk2  kx  zk2 

2 -SQNE, 2

i.e.,

2 kR x  xk2 2

(2.65)

for all x 2 X and for all z 2 Fix.PX T /. (iv) If Fix T ¤ ;, then the operator R is 2  -SQNE. Proof. (i) Since the metric projection PX is firmly nonexpansive, it is 1-relaxed 4 firmly nonexpansive. By Theorem 2.2.37, the operator R is 4 -RFNE. (ii) This property follows from Corollary 1.2.10. 4 (iii) Since R is -RFNE, where  D 4 (see (i)), Corollary 2.2.9 yields kR x  zk2  kx  zk2  D kx  zk2 

2 kR x  xk2  2 kR x  xk2 . 2

(iv) The claim follows from Corollary 2.2.25. t u If X D H, then R D T , nevertheless, estimation (2.65) is weaker than estimation (2.47). Furthermore, estimation (2.65) is weaker than estimation (2.53). Note, however, that we have supposed in Corollary 2.2.25 that the operator T W X ! H is a cutter, consequently Fix T ¤ ;, while in Theorem 2.2.46 (iii) we have supposed that Fix.PX T / ¤ ;, which is weaker than the assumption Fix T ¤ ;.

2.2.7 Fixed Points of Firmly Nonexpansive Operators A firmly nonexpansive operator is nonexpansive (see Theorem 2.2.4), therefore, the subset of its fixed points is closed and convex (see Proposition 2.1.11). In this section we show that the subsets Fix T for FNE- and for NE-operators are intersections of half-spaces, which also yields the closedness and convexity of Fix T . Equivalent formulations to the results below can be found in [185, Equalities (11.3) and (11.4)]. Theorem 2.2.47. Let X  H be closed convex and T W X ! H be firmly nonexpansive. Then Fix T D

\

fz 2 X W

hT x  x; T x  zi  0g.

x2X

Consequently, Fix T is a closed convex subset.

2.3 Strongly Nonexpansive Operators

91

Proof. Since a firmly nonexpansive operator with a fixed point is a cutter (see Theorem 2.2.5), the theorem follows from Lemmas 2.1.36 and 2.1.35. t u Corollary 2.2.48. Let X  H be closed and convex. The subset of fixed points of a nonexpansive operator S W X ! H has the form Fix S D

\

fz 2 X W

2hz  x; S x  xi  kS x  xk2 g,

(2.66)

x2X

consequently, Fix S is a closed convex subset. Proof. Let S W X ! H be nonexpansive. By Corollary 2.2.13, we have S D 2T Id for a firmly nonexpansive operator T . It is clear that Fix S D Fix T . Theorem 2.2.47 yields now Fix S D

\

fz 2 X W

x2X

1 1 h .S x C x/  x; .S x C x/  zi  0g 2 2 t u

which is equivalent to (2.66).

2.3 Strongly Nonexpansive Operators Definition 2.3.1. An operator T W X ! H is called strongly nonexpansive (SNE), k 1 if T is nonexpansive and for all sequences fx k g1 kD0 ; fy gkD0  X the following implication is true k k  k .x  y /is bounded and  x  y k   T x k  T y k  ! 0

 H) .x k  y k /  .T x k  T y k / ! 0,

The notion of strongly nonexpansive operators in Banach spaces was proposed by Bruck and Reich in [51, Sect. 1], where also properties of these operators are proved (see also [23, Sect. 4.3]). Remark 2.3.2. It is clear that a contraction is a strongly nonexpansive operator. Indeed, let T be a contraction, i.e., kT x  T yk  ˛ kx  yk for all x;  yk 2 Xk and k k x  y   for a constant ˛ 2 .0; 1/, and .x  y / be bounded and such that  k  T x  T y k  ! 0. Then we have       k x  y k   T x k  T y k   .1  ˛/ x k  y k  ! 0. Consequently, x k  y k ! 0 and T x k  T y k ! 0, i.e., T is strongly nonexpansive.

92

2 Algorithmic Operators

Remark 2.3.3. (S. Reich, A private communication (2009)) Let X  H be compact. Then a strictly nonexpansive operator defined on X is strongly nonexpansive. Indeed, let T W X ! H be strictly nonexpansive, i.e., kT x  T yk < kx  yk or x  y D T x  T y for all x; y 2 X , and X be compact. We show that T is strongly nonexpansive.  k  k 1 k   Suppose thatsequences fx k g1 kD0 and fy gkD0 are given such that x  y  k T x  T y k  ! 0 and that there exist subsequences fx nk g1  fx k g1 and kD0 kD0 k 1 fy nk g1 kD0  fy gkD0 and a constant " > 0 such that k.x nk  y nk /  .T x nk  T y nk /k  ". Since X is compact, we can suppose without loss of generality that x nk ! x and y nk ! y. Since T is continuous as a nonexpansive operator, we have T x nk ! T x and T y nk ! T y. Hence, we obtain in the limit kx  yk D kT x  T yk, which yields, due to strict nonexpansivity of T , that x  y D T x  T y. On the other hand, we have k.x  y/  .T x  T y/k D lim k.x nk  y nk /  .T x nk  T y nk /k  ", k

a contradiction, which shows that T is strongly nonexpansive. Theorem 2.3.4. Let T W X ! H be firmly nonexpansive and  2 .0; 2/. Then the relaxation T of T is strongly nonexpansive.  k  k 1 k  Proof. Let fx k g1 is bounded and kD0 ; fy gkD0  X be such that x  y  k    x  y k   T x k  T y k  ! 0: The firm nonexpansivity of T yields of T (see Theorem 2.2.10   1 ˚ the nonexpansivity (ii)), consequently, the sequence x k  y k  C T x k  T y k  kD0 is bounded. Therefore, by the obvious equality T x  x D .T x  x/ and by Corollary 2.2.15, we have  k  .x  y k /  .T x k  T y k /2  2 D .T x k  x k /  .T y k  y k /   

  x k  y k 2  T x k  T y k 2  2          x k  y k   T x k  T y k  x k  y k  C T x k  T y k  ! 0, D 2  k  i.e., .x  y k /  .T x k  T y k / ! 0 and T is strongly nonexpansive. t u

2.3 Strongly Nonexpansive Operators

93

In the previous sections we have proved that the following classes of operators are closed under composition and under convex combination: (a) The class of strictly relaxed cutters with a common fixed point (see Theorems 2.1.46 and 2.1.50), (b) The class of strongly quasi-nonexpansive operators with a common fixed point (see Corollary 2.1.47 and Theorem 2.1.50), (c) The class of strictly relaxed firmly nonexpansive operators (see Theorems 2.2.37 and 2.2.35) (d) The class of averaged operators (see Remark 2.2.38 and Corollary 2.2.36). It turns out that the class of strongly nonexpansive operators has the same properties. The first part of the theorem below was proved by Bruck and Reich in [51, Proposition 1.1] and the other one by Reich in [295, Lemma 1.3]. Theorem 2.3.5. Let T1 ; T2 W X ! X be strongly nonexpansive and T have one of the following forms: (i) T WD T2 T1 , (ii) T WD .1  /T1 C T2 , where  2 Œ0; 1. Then T is strongly nonexpansive. Proof. By Lemma 2.1.12, the operator T is nonexpansive. Let the  ksequences  k 1 k 1 k k x  y k   fx g ; fy g  X be such that .x  y / is bounded and kD0  kD0  T x k  T y k  ! 0. (i) By the nonexpansivity of T1 and T2 , we have  k        T x  T y k  D T2 .T1 x k /  T2 .T1 y k /  T1 x k  T1 y k   x k  y k  , k  0, consequently,  k    x  y k   T1 x k  T1 y k  ! 0 and     T1 x k  T1 y k   T2 .T1 x k /  T2 .T1 y k / ! 0. Since T1 and T2 are strongly nonexpansive, we have .x k  y k /  .T x k  T y k / D .x k  y k / .T1 x k  T1 y k / C .T1 x k  T1 y k /  .T2 .T1 x k /  T2 .T1 y k // ! 0; i.e., T is strongly nonexpansive. (ii) The assertion is clear when  D 0 or  D 1. Let  2 .0; 1/. By the convexity of the norm and the nonexpansivity of T1 and T2 , we have

94

2 Algorithmic Operators

Fig. 2.12 SNE operator which is not AV

Ux y

Uy

Tx

Ty

B(0, 1)

x

 k    T x  T y k  D .1  /T1 x k C T2 x k  .1  /T1 y k  T2 y k       .1  / T1 x k  T1 y k  C  T2 x k  T2 y k         .1  / x k  y k  C  x k  y k  D x k  y k  , consequently,  k    x  y k   T x k  T y k           .1  /.x k  y k   T1 x k  T1 y k / C .x k  y k   T2 x k  T2 y k /. Therefore,

 k    x  y k   T1 x k  T1 y k  ! 0

and

 k    x  y k   T2 x k  T2 y k  ! 0.

By the strong nonexpansivity of T1 and T2 , we have now .x k  y k /  .T x k  Ty k / D .1  /..x k  y k /  .T1 x k  T1 y k // C ..x k  y k /  .T2 x k  T2 y k // ! 0,

i.e., T is strongly nonexpansive. t u The following example shows that the class of averaged operators or, equivalently, the class of strictly relaxed firmly nonexpansive operators is a proper subclass of the class of strongly nonexpansive operators. Example 2.3.6. Let X WD B.0; 1/  H be a unit ball, U W H ! H be a unitary operator such that hUx; xi D 0 for all x 2 H (e.g., U W R2 ! R2 is defined by Ux WD .2 ; 1 / for x D .1 ; 2 / 2 R2 with the standard inner product) and the operator T W X ! X be defined by T x WD ˛.x/Ux. with ˛.x/ WD 1  12 kxk (see Fig. 2.12).

2.3 Strongly Nonexpansive Operators

95

It is clear that ˛.x/Ux D U.˛.x/x/, consequently, kT x  T yk D k˛.x/Ux  ˛.y/Uyk D kU.˛.x/x/  U.˛.y/y/k D k˛.x/x  ˛.y/yk . A straightforward calculation shows that kx  yk2  kT x  T yk2 D kx  yk2  k˛.x/x  ˛.y/yk2 1 1 D  kxk4  kyk4 C kxk3 C kyk3 4 4 1 hx; yi.kxk C kyk  kxk  kyk/ 2 1 D .kxk  kyk/2 .kxk C kyk/.1  .kxk C kyk// 4 1 C.kxk  kyk  hx; yi/.kxk C kyk  kxk  kyk/. 2 We have kxkCkyk  kxkkyk, since kxk ; kyk 2 Œ0; 1. This fact and the Cauchy– Schwarz inequality yield kxk C kyk 

1 1 1 1 kxk  kyk  kxk  kyk  kxk  kyk  hx; yi, 2 2 4 4

consequently, kx  yk2  kT x  T yk2 1 1  .kxk  kyk/2 .kxk C kyk/.1  .kxk C kyk// C .kxk  kyk  hx; yi/2 4 4 and T is nonexpansive. We apply the above tox D x k and y D y k .  inequalities  Suppose that x k ; y k 2 X and that x k  y k T x k  T y k  ! 0. Then, of course,     k x  y k 2  T x k  T y k 2 ! 0;     because x k  y k  C T x k  T y k  is bounded. Therefore,  k  k x   y  ! 0

96

2 Algorithmic Operators

    (note that 1  14 .x k  C y k /  12 ) and  k  k x   y   hx k ; y k i ! 0. Now we have  k         

x  y k 2 D x k   y k  2 C 2 x k   y k   hx k ; y k i ! 0, i.e., .x k  y k / ! 0. Furthermore, .T x k  T y k / ! 0, by the nonexpansivity of T , consequently, .x k  y k /  .T x k  T y k / ! 0, i.e., T is strongly nonexpansive. Note that z D 0 is the unique fixed point of T . Suppose that T is ˛-averaged, for a constant ˛ 2 .0; 1/. By Corollary 2.2.17, the operator T is .2˛/-relaxed firmly nonexpansive. Consequently, the operator V D T , where  D .2˛/1 2 . 12 ; C1/ is firmly nonexpansive (see Corollary 2.2.19) and V is a cutter (see Theorem 2.2.5), i.e., hx; x  T xi C 2 kx  T xk2 D hx C .T x  x/; x  T xi D hT x; x  T xi D h0  V x; x  V xi  0. Dividing the inequalities above by  > 0, we obtain, for all x ¤ z, 1 hx; x  T xi 1 kxk2 D D . < 2 2 2 2 2 1 C .1  12 kxk/2 kx  T xk kxk C ˛ .x/ kxk k Applying the inequalities above to a sequence fx k g1 kD0 with limk x D 0, we obtain

1 1 1   D , <   lim k 1 C .1  1 x k /2 2 2 2 a contradiction, which proves that T is not averaged.

2.4 Generalized Relaxations of Algorithmic Operators In the definition of a relaxation T of an operator T W X ! H we have supposed that the relaxation parameter  2 Œ0; 2 (see Definition 2.1.2). Furthermore, the assumption  2 .0; 2/ is necessary for the strong quasi nonexpansivity of the -relaxation of a firmly nonexpansive operator T with Fix ¤ ; (see proof of

2.4 Generalized Relaxations of Algorithmic Operators

97

Theorem 2.1.39). However, in some applications, relaxations of operators (e.g., of firmly nonexpansive ones) with the relaxation parameter which are greater than 2 are successfully used. In general, the convergence of sequences generated by such operators is not guaranteed. It turns out that, if we allow to vary the relaxation parameter in dependence on the current point, in such a way that the relaxed operator is a cutter, then we can apply the usual convergence analysis for sequences generated by such an operator. Below we define a generalization of a relaxation of an operator, which permits us to extend the convergence results to sequences generated by the generalized relaxation. Definition 2.4.1. Let T W X ! H,  2 Œ0; 2 and W X ! .0; C1/. The operator T ; W X ! H, T ; x WD x C  .x/.T x  x/ (2.67) is called the generalized relaxation of T , the value  is called the relaxation parameter and is called the step size function. If .x/  1 for all x 2 X , then the operator T ; is called an extrapolation of T . Some special cases of generalized relaxations of some classes of nonexpansive operators, presented in various forms and applied in most cases to the convex feasibility problems, were studied by Gurin et al. [196, Sect. 3], Pierra [284, Sect. 1], Cegielski [62, Sect. 4.3], Kiwiel [229, Sect. 3], Bauschke [17, Sects. 7.3 and 8.3], Combettes [118, Sects. 5.4–5.8], [120, Sect. IV], Bauschke et al. [30, Sect. 3] Bauschke et al. [25] and by Cegielski and Suchocka in [76]. In this section we present properties of generalized relaxations of cutters and give conditions for a generalized relaxation to be strongly quasi-nonexpansive. These properties will be applied in one of the next chapters in order to prove the convergence of sequences generated by such operators. Denote T D T ;1 . Remark 2.4.2. Let T W X ! H,  2 Œ0; 2 and W X ! .0; C1/. (a) If .x/ D 1 for all x 2 X , then T ; D T , i.e., the generalized relaxation of T is reduced to the classical relaxation of T . (b) The values of the step size function for x 2 Fix T have no influence on the form of an operator T ; because T ; jFix T D Id for any step size function and for any  2 .0; 2. Therefore, we can suppose without loss of generality that

.x/ D 1 for all x 2 Fix T . (c) For any x 2 X the following equalities hold T ; x  x D  .x/.T x  x/ D .T x  x/, i.e., T ; is a -relaxation of an operator T . (d) For any  ¤ 0 it holds Fix T ; D Fix T (cf. Remark 2.1.4). The corollary below is a version of Theorem 2.1.39.

(2.68)

98

2 Algorithmic Operators

Corollary 2.4.3. Let T W X ! H have a fixed point, W X ! .0; C1/ be a step size function and  2 .0; 2/. Then T is a cutter if and only if T ; is 2 -strongly  quasi-nonexpansive. In both cases kT ; x  zk2  kx  zk2  .2  / 2 .x/ kT x  xk2

(2.69)

for all x 2 X and z 2 Fix T . Proof. By Remark 2.4.2 (c), T ; is the -relaxation of T . The first part of the theorem follows now from Theorem 2.1.39. The 2 -strong quasi nonexpansivity  of T ; means kT ; x  zk2  kx  zk2 

2 kT ; x  xk2 . 

Applying now (2.68) to the inequality above we obtain (2.69).

t u

Let T W X ! H be an operator with a fixed point. Our aim is to give sufficient conditions for the step size function W X ! .0; C1/, at which T is a cutter. The following definition was proposed in [70, Definition 9.17]. Definition 2.4.4. We say that an operator T W X ! H with a fixed point is oriented if for all x … Fix T ı.x/ WD inf

z2Fix T

hz  x; T x  xi kT x  xk2

> 0.

(2.70)

If ı.x/  ı > 0 for all x … Fix T , then we say that T is strongly oriented. It follows from Remark 2.1.31 that T W X ! H is strongly oriented if and only if T is an ˛-relaxed cutter for some ˛ > 0. Corollary 2.4.5. Let T W X ! H be an oriented operator with Fix T ¤ ;. If a step size function W X ! .0; C1/ satisfies the inequality

.x/ 

hz  x; T x  xi kT x  xk2

(2.71)

for all x … Fix T and z 2 Fix T , then T is a cutter. Consequently, for any  2 .0; 2/, the generalized relaxation T ; of T is 2  -strongly quasi-nonexpansive. Proof. Let x … Fix T , z 2 Fix T and W X ! .0; C1/ be a step size function satisfying (2.71). The existence of follows from the assumption that T is oriented. Then (2.68) and inequality (2.71) yield hz  x; T x  xi D hz  x; .x/.T x  x/i  k .x/.T x  x/k2 D kT x  xk2 .

2.4 Generalized Relaxations of Algorithmic Operators

99

By the equivalence (a),(b) in Lemma 1.2.5, we have hz  T x; x  T xi  0, i.e., T is a cutter. The Corollary 2.4.3.

2 -strong 

quasi nonexpansivity of T ; follows now from t u

The convergence of sequences generated by generalized relaxations of an algorithmic operator U , which we present in the next chapter, requires a stronger condition than (2.71). As we will see, the convergence holds if we additionally suppose that U is strongly oriented, or, equivalently, that the step size .x/  ˛ for all x 2 X and for a constant ˛ > 0. This leads to ˛-relaxed cutters (see Remark 2.1.31). It is clear that if an operator T W X ! H with a fixed point is an ˛-relaxed cutter for some ˛ > 0, then there exists a step size function W X ! .0; C1/ satisfying inequality (2.71), e.g., .x/ D ˛ 1 for all x 2 X (cf. (2.22)). In practice, however, it is important to determine a step size .x/ for which the difference between the rightand the left-hand side of inequality (2.71) is as small as possible for all z 2 Fix T . Theoretically, the best possibility would be .x/ D ı.x/ for x … Fix U , where ı.x/ is defined by (2.70), but the computation of ı.x/ is, in most cases, impossible, because we usually do not know Fix T explicitly. Having an ˛-relaxed cutter T we can construct its generalized relaxation T ; with the range of the step size function contained in Œ˛; C1/ and satisfying assumptions of Corollary 2.4.5. The corollary below gives a collection of operators which are ˛-relaxed cutters. Corollary 2.4.6. Let U W X ! H have a fixed point. Then U is an ˛-relaxed cutter with: (a) (b) (c) (d) (e)

˛ ˛ ˛ ˛ ˛

D 1 if U is firmly nonexpansive, D  if U is -relaxed firmly nonexpansive, where  2 .0; 2, D 2 if U is nonexpansive, D 2 if U is -averaged, where  2 .0; 1/, 2 D 1Cˇ if U is ˇ-strongly quasi-nonexpansive, where ˇ > 0.

Proof. (a) Let U be firmly nonexpansive. Then it follows from the first part of Theorem 2.2.5 that T is a cutter, i.e., T is a 1-relaxed cutter. (b) Let  2 .0; 2 and U WD Id C.T  Id/ for a firmly nonexpansive operator T . Then, by (a), we have hz  x; Ux  xi D hz  x; T x  xi   kT x  xk2 D

1 kUx  xk2 . 

(c) Let U be nonexpansive. Then U D 2T  Id for a firmly nonexpansive operator T (see Corollary 2.2.13) and this case is covered by (b) for  D 2. (d) Let  2 .0; 1/ and U WD .1  / Id CS for a nonexpansive operator S . Then U is 2-relaxed firmly nonexpansive (see Corollary 2.2.17). The claim follows now from (b) with  D 2.

100

2 Algorithmic Operators

(e) Let ˇ > 0 and U be ˇ-strongly quasi-nonexpansive. It follows from Corol2 lary 2.1.43 that U is a 1Cˇ -relaxed cutter. t u In the lemma below we state some obvious properties of the generalized relaxation. Lemma 2.4.7. Let T W X ! H be an operator with a fixed point, and f j gj 2J W X ! .0; C1/ be a family of step size functions. (i) If T j , j 2 J , are cutters, then Tsupj 2J j is a cutter. (ii) If i  j for some i; j 2 J and T j is a cutter, then T i is a cutter. If T is a cutter, then there exists a step size function with .x/  1 for all x … Fix T , for which T is a cutter, e.g., a step size function defined by

.x/ D ı.x/, where ı.x/ is given by (2.70) for x … Fix T . Consequently, the generalized relaxation T ; is strongly quasi-nonexpansive for any  2 .0; 2/ (see Theorem 2.4.5). Note that .x/  1, by Remark 2.1.31. The following example shows, however, that there is a cutter T such that the generalized relaxation T ; is strongly quasi-nonexpansive for all  2 .0; 2/ if and only if .x/  1 for all x … Fix T . Example 2.4.8. Let T ; be a generalized relaxation of the metric projection PC W H ! H, where C  H is a nonempty closed convex subset, i.e., T ; .x/ D x C  .x/.PC x  x/ for a relaxation parameter  2 .0; 2/ and for some step size function W H ! .0; C1/. For any x 2 H we have kT ; x  PC xk2 D kx C  .x/.PC x  x/  PC xk2 D kx  PC xk2 C 2 2 .x/ kPC x  xk2  2 .x/ kPC x  xk2 D kx  PC xk2   .x/.2   .x// kPC x  xk2 , consequently, kT ; x  PC xk2 D kx  PC xk2 

2   .x/ kT ; x  xk2 .  .x/

(2.72)

Let  2 .0; 2/. Suppose that T ; is strongly quasi-nonexpansive, i.e., kT ; x  zk2  kx  zk2  ˛ kT ; .x/  xk2

(2.73)

for some ˛ > 0, for all x 2 H and z 2 C WD Fix PC . Note that ˛ can depend on . Let x … C and z D PC x. Then (2.72) and (2.73) yield 0<˛

2   .x/ .  .x/

There exists a constant ˛ satisfying the above inequalities for all  2 .0; 2/ if and only if .x/  1.

2.4 Generalized Relaxations of Algorithmic Operators

101

Fig. 2.13 Operators T and T from Example 2.4.9

x T¾x

z Tx C

√

1

2

2

Ty D

y

If T W X ! H is firmly nonexpansive with a fixed point, then T is oriented and for the function ı defined by (2.70) it holds ı.x/  1 for all x … Fix T . Therefore, Corollary 2.4.5 applied to a firmly nonexpansive operator T with the step size .x/ WD ı.x/ for x … Fix T is an extension of Theorem 2.2.5 (i) for generalized relaxations. Unfortunately, Theorem 2.2.5 (ii) cannot be analogously extended. The fact that T W X ! H is a projection and T is a cutter for some step size function W X ! .0; C1/ does not yield the firm nonexpansivity of T . Even if we additionally suppose that T is nonexpansive, T needs not to be firmly nonexpansive (see Example 2.2.7). Moreover, a projection T for which T is a cutter needs not to be continuous. p Example 2.4.9. Let H D R2 , C WD B.0; 1/, D WD bd B.0; 2/; a D .1; 0/. Define the operator T W R2 ! R2 by 8 < PC x for kxk  2 T x WD a for kxk > 2, 1  0 : a for kxk > 2, 1 < 0. It is clear that T is a projection with Fix T D C . For kxk > 2, let Ux be the unique common point of the segment Œx; T x and the circle D. Define the function

W R2 ! R by ( 1 if kxk  2

.x/ WD kUxxk kT xxk if kxk > 2. Observe that for kxk > 2 it holds T x D Ux. It follows from geometrical considerations (note that the p square circumscribed on the circle bd B.0; 1/ is inscribed in the circle bd B.0; 2/) that for all x 2 R2 and z 2 C D Fix T it holds hx  T .x/; z  T .x/i  0 (see Fig. 2.13). Therefore, T is a cutter. Note that T is not continuous, therefore, T cannot be firmly nonexpansive.

102

2 Algorithmic Operators FNE operators cutters with a common fixed point relaxed cutters with a common fixed point The family of

is closed under convex combination

RFNE operators strictly RFNE operators

is closed under composition

SQNE operators with a common fixed point SNE operators

Fig. 2.14 Closedness of families of algorithmic operators

SUMMARY In Fig. 2.14 we recall in a short form the properties of algorithmic operators which were presented in this chapter. These properties are useful in construction of projection methods. We will describe these constructions in Chaps. 4 and 5.

2.5 Exercises Exercise 2.5.1. Show that .T / D T for all ;  2 R. Exercise 2.5.2. Let T W R ! R,  Tx D

x2 jxj 

3 16

if jxj  34 if jxj > 34 .

Show that T is quasi-nonexpansive and continuous, but T is not a nonexpansive operator. Exercise 2.5.3. Let fUi gi 2I be a finite family of operators, Ui W X ! H, i 2 I . Let w W X ! m be a weight function satisfying !i .x/ > 0 for some i.x/ D argmaxi 2I kUi x  xk for all x 2 X . Prove that w is appropriate with respect to the family fUi gi 2I .

2.5 Exercises

103

Exercise 2.5.4. Prove that the assumption on the C -strict quasi nonexpansivity in Theorem 2.1.26 (i) can be weakened. In this case it suffices to suppose that all Ui are quasi-nonexpansive, i 2 I , and at least one of them is C -strictly quasinonexpansive. The assumption that the weight function w is appropriate should be replaced in this case by a stronger one, namely: wj .x/ > 0 for all x such that I.x/ ¤ ; and for all j 2 I.x/. Exercise 2.5.5. Prove Corollary 2.1.29. Exercise 2.5.6. Prove Lemma 2.1.45. Exercise 2.5.7. Show that the operator T W R2 ! R2 , T x WD .1 cos '  2 sin '; 1 sin ' C 2 cos '/ is nonexpansive and monotone for ' 2 .0; =2/, but T is not firmly nonexpansive. Exercise 2.5.8. Prove Lemma 2.2.2. Exercise 2.5.9. Show that the operator T presented in Example 2.2.7 is nonexpansive and that T is a separator of A, but T is not firmly nonexpansive. Exercise 2.5.10. Let H D R2 , A WD fx 2 R2 W 2 D 0g and B WD fx 2 R2 W 1 D 2 g. By Theorem 2.2.21 (iii) PA and PB are firmly nonexpansive. Check that T WD PB PA is not firmly nonexpansive.

•

Chapter 3

Convergence of Iterative Methods

3.1 Iterative Methods Convex minimization problems are usually presented as minimization of a continuous convex function on a Hilbert space H or on a closed convex subset X . The subset of solutions of this problem is a closed convex subset M  X . We recall as an example the split feasibility problem and a connected problem of minimization of a convex proximity function (see Sect. 1.3.7). The CMP is equivalent to finding a fixed point of an operator U W X ! H (usually nonexpansive) corresponding to the objective function. The operator U is constructed in such a way that M D Fix U . Iteration methods for solving these problems (finding an element x  2 M ) have the form of a recurrence x kC1 D Ux k or a more general recurrence x kC1 D Uk x k , k  0, where x 0 2 X is arbitrary, fUk g1 kD0 is a sequence of algorithmic operators Uk W X ! X such that 1 \ Fix Uk  Fix U D M: kD0

If Uk D U for all k  0, then we say that the iterative method is autonomous, otherwise, we say that the method is nonautonomous. In the first case we can also write x k D U k x 0 . The algorithmic operator or the sequence of algorithmic operators describing the method should be constructed in such a way that any sequence fx k g1 kD0 generated by the method converges (weakly or strongly) to a solution x  2 M .

A. Cegielski, Iterative Methods for Fixed Point Problems in Hilbert Spaces, Lecture Notes in Mathematics 2057, DOI 10.1007/978-3-642-30901-4 3, © Springer-Verlag Berlin Heidelberg 2012

105

106

3 Convergence of Iterative Methods

3.2 Properties of the Weak Convergence In this section we present properties of the weak convergence which will be applied in the sequel.   Lemma 3.2.1. If x k * x 2 H, then lim infk x k   kxk. Proof. Let x k * x. The lemma is clear for x D 0. Let now x ¤ 0. By the Cauchy– Schwarz inequality, we have   lim inf kxk  x k   lim infhx; x k i D kxk2 , k

k

  i.e., lim infk x k   kxk.

t u

Note that a sequence which is weakly convergent needs not to converge strongly. Example 3.2.2. (cf. [267, Example 2.13]). Let H WD l 2 and x k D ek D .ek1 ; ek2 ; : : :/, where  1 for i D k eki WD 0 for i ¤ k.  k Then x  D 1, although fx k g1 kD0 does not contain a convergent subsequence,  p  because x k  x l  D 2 for all k; l, k ¤ l, i.e., fx k g1 kD0 is not a Cauchy sequence. k Note, however, that x * 0. Under some additional assumptions the weak convergence yields the strong one.   Lemma 3.2.3. If x k * x 2 H and x k  ! kxk, then x k ! x. Proof. Let x k * x. It follows from the properties of the inner product that     x  x k 2 D kxk2 C x k 2  2hx; x k i ! 0 as k ! 1.

t u

A Hilbert space has an important property which is expressed in the following lemma. Lemma 3.2.4 ([279]). If x k * y 2 H, then, for any y 0 2 H, y 0 ¤ y, the following inequality holds     lim inf x k  y 0  > lim inf x k  y  . (3.1) k 0

k

Proof. Let x * y and y ¤ y. Denote ı WD ky  y 0 k2 . Since a weakly convergent sequence is bounded, both lower limits in (3.1) are finite. The properties of the inner product yield k

3.2 Properties of the Weak Convergence

107

  k   x  y 0 2 D x k  y C y  y 0 2 2  2  D x k  y  C y  y 0  C 2hx k  y; y  y 0 i  2 D x k  y  C ı C 2hx k  y; y  y 0 i. By assumption, hx k  y; y  y 0 i ! 0, consequently, 2 2  2   lim inf x k  y 0  D lim inf x k  y  C ı > lim inf x k  y  , k

k

k

    i.e., lim infk x k  y 0  > lim infk x k  y .

t u

Lemma 3.2.4 was proved by Zdzisław Opial in [279, Lemma 1]. The property described in this lemma is known under the name Opial property. A Banach space for which the property holds is called an Opial space. Therefore, Lemma 3.2.4 can be formulated as follows: any Hilbert space is an Opial space. The following lemma, also proved by Opial in [279, Lemma 2], is a consequence of the Opial property. Lemma 3.2.5. Let T W X ! and y 2 X be a weak cluster point  Hk be nonexpansive  T x  x k  ! 0, then y 2 Fix T . . If of a sequence fx k g1 kD0   k nk k 1 k  ! 0. Proof. Let x *y for a subsequence fx nk g1 kD0 of fx gkD0 and T x  x Suppose that T y ¤ y. Then, by the triangle inequality and by Lemma 3.2.4, we have lim inf kx nk  yk  lim inf kT x nk  T yk k!1

k!1

D lim inf kT x nk  x nk C x nk  T yk k!1

 lim inf.kx nk  T yk  kT x nk  x nk k/ k!1

D lim inf kx nk  T yk k!1

> lim inf kx nk  yk . k!1

A contradiction proves that y 2 Fix T .

t u

Definition 3.2.6. We say that an operator S W X ! H is demi-closed at 0 if for any sequence x k * y 2 X with S x k ! 0 we have Sy D 0. If we replace the weak convergence x k * y by the strong one in Definition 3.2.6, then we obtain the definition of the closedness of S at 0. If H is finite dimensional, then a weakly convergent sequence is convergent. Therefore, the notions of a demiclosed operator and a closed operator coincide in a finite dimensional Hilbert space. The property expressed in Lemma 3.2.5 is known under the name demiclosedness principle (see, e.g., [22, Fact 1.2]). Opial writes that the result is due

108

3 Convergence of Iterative Methods

to Browder [44] (see [278, Proposition 2.5]). The property says, actually, that for a nonexpansive operator T in a Hilbert space, the operator T  Id is demi-closed at 0.

3.3 Properties of Fej´er Monotone Sequences The notions of Fej´er monotone operators, quasi-nonexpansive operators and strongly quasi-nonexpansive operators (see Definitions 2.1.15, 2.1.38) are closely related to the Fej´er monotonicity of sequences which are generated by sequences of such operators. Definition 3.3.1. Let C  H be nonempty. We say that a sequence fx k g1 kD0  H is: (a) Fej´er monotone (FM) with respect to C if  kC1    x  z  x k  z

(3.2)

for all z 2 C and for all k  0, (b) Strictly Fej´er monotone with respect to C if     kC1 x  z < x k  z for all z 2 C and for all k  0 with x k … C , (c) Strongly Fej´er monotone (SFM) with respect to C if there exists a constant ˛ > 0 such that 2  2  2  kC1 x  z  x k  z  ˛ x kC1  x k  , for all z 2 C and for all k  0. Similarly as for the Fej´er monotone operators, we can suppose without loss of generality that C is a closed convex subset in Definition 3.3.1. Let a sequence kC1 D Uk x k , where Uk W X ! X . fx k g1 kD0  X be generated by the recurrence x If the operators Uk , k  0, are (strictly) Fej´er monotone with respect to C  X (see Definition 2.1.15), then fx k g1 er monotone with respect to C . kD0 is (strictly) Fej´ Similarly, if the operators Uk , k  0, with a common fixed point, are ˛-strongly k 1 quasi-nonexpansive T1 for some ˛ > 0, then fx gkD0 is strongly Fej´er monotone with respect to kD0 Fix Uk . It is clear that a Fej´er monotone sequence is bounded, consequently it has a weak cluster point. Below, we give other important properties of Fej´er monotone sequences. Theorem 3.3.2. Let fx k g1 er monotone with respect to a closed convex kD0 be Fej´ subset C  H. Then:

3.3 Properties of Fej´er Monotone Sequences

109

(i) fd.x k ; C /g1 kD0 is a nonincreasing sequence, consequently, it converges. (ii) PC x k converges in norm to a point z 2 C . (iii) For any weak cluster point xN of fx k g1 N D z . In particular, if kD0 we have PC x  xN 2 C , then xN D z . (iv) It holds     (3.3) lim x k  z  D inf lim x k  z . k

z2C

k

Proof. (i) By the definition of the metric projection and the Fej´er monotonicity of fx k g1 kD0 , we have     d.x kC1 ; C / D x kC1  PC x kC1   x kC1  PC x k     x k  PC x k  D d.x k ; C /. (ii) (cf. [22, Theorem 2.16 (iv)]) Let l  k  0. By the parallelogram law with x D PC x l  x l and y D PC x k  x l , the definition of the metric projection, the convexity of C and the Fej´er monotonicity of fx k g1 kD0 , we have       PC x l  PC x k 2 D 2 PC x l  x l 2 C 2 PC x k  x l 2 2   1 l k l .P 4  x C P x /  x C  2 C    2 2 2  2 PC x l  x l  C 2 PC x k  x l   4 PC x l  x l   2  2 D 2.PC x k  x l   PC x l  x l  / 2  2   2.PC x k  x k   PC x l  x l  / D 2.d.x k ; C /  d.x l ; C //. By (i), the right hand side of the latter equality goes to zero as k; l ! 1. Therefore, fPC x k g1 kD0 is a Cauchy sequence, consequently, it converges in norm to a point z 2 C . (iii) Because any subsequence of a Fej´er monotone sequence is Fej´er monotone, we can suppose, without loss of generality, that the whole sequence converges weakly to xN 2 H. Let x 2 C and " > 0. Denote zk D PC x k . Let k0  0 be such that       1 maxfzk  x k   zk  z  ; kz  xk  zk  z  ; hx k  x; N z  xig < " 3 for all k  k0 . Then, by the characterization of the metric projection and the Cauchy–Schwarz inequality, we have

110

3 Convergence of Iterative Methods

0  hzk  x k ; zk  xi D hzk  x k ; zk  z i C hzk  x k ; z  xi D hzk  x k ; zk  z i C hzk  z ; z  xi Chz  x; N z  xi C hxN  x k ; z  xi         zk  x k   zk  z   zk  z   kz  xk N z  xi C hxN  x k ; z  xi Chz  x; N z  xi  " C hz  x; for all k  k0 , i.e., hz  x; N z  xi  ". Since " > 0 is arbitrary, we obtain   hz  x; N z  xi  0, i.e., z D PC x. N (iv) The definition of the metric projection and the continuity of the norm yield       lim x k  z  lim x k  PC x k  D lim x k  z  k

k

k

for any z 2 C . Consequently,     inf lim x k  z  lim x k  z  .

z2C

k

k

But z 2 C , therefore, the equality in (3.4) holds.

(3.4) t u

er monotone (or even strictly Fej´er monotone) with A sequence fx k g1 kD0 which is Fej´ respect to a subset C  H needs not be weakly convergent (see Exercise 3.10.1). Furthermore, weak cluster points of a sequence which is Fej´er monotone with respect to a subset C  H need not to have equal distance to the set C (see Exercise 3.10.2). The following result which is due to Browder (see [47, Lemma 6]) follows immediately from Theorem 3.3.2 (iii). Nevertheless, we present below an independent simple proof of this result. Corollary 3.3.3. If a sequence fx k g1 er monotone with respect to a kD0  H is Fej´ nonempty subset C  H, then fx k g1 has at most one weak cluster point in C . kD0 Consequently, x k converges weakly to a point z 2 C if and only if all weak cluster points of fx k g1 kD0 belong to C .   Proof. (cf. [22, Theorem 2.16 (ii)]). By assumption, the sequence fx k  zg1 kD0 is monotone, consequently, it converges for any z 2 C . Denote   2 ˛.z/ WD lim x k  z  kzk2 . k

We have   2   2 lim x k   2hx k ; zi D lim x k  z  kzk2 D ˛.z/. k

k

3.4 Asymptotically Regular Operators

111

nk 1 k 1 mk Let fx mk g1 * z0 2 kD0 and fx gkD0 be two subsequences of fx gkD0 such that x nk 00 C and x * z 2 C as k ! 1. Then h 2   2 i 2 limhx k ; z0  z00 i D lim x k   2hxk ; z00 i  x k   2hxk ; z0 i k

k

D ˛.z00 /  ˛.z0 /. But

limhx mk ; z0  z00 i D hz0 ; z0  z00 i k

and

limhx nk ; z0  z00 i D hz00 ; z0  z00 i. k

0

0

00

Therefore, hz ; z  z i D hz00 ; z0  z00 i, i.e., kz0  z00 k D 0 and z0 D z00 .

t u

The following lemma gives a sufficient condition for the strong convergence of Fej´er monotone sequences and was proved in [22, Theorem 2.16 (v)] (see also [283, Theorem 1.1] for a stronger result). Lemma 3.3.4. Let fx k g1 er monotone with respect to a nonempty kD0  H be Fej´ subset C  H. If at least one cluster point x  of fx k g1 kD0 belongs to C , then xk ! x . Proof. Since a Fej´er monotone sequence is bounded, fx k g1 kD0 has a weak cluster k 1  point x  . Suppose that a subsequence fx nk g1 kD0  fx gkD0 converges to x 2 C . k 1  We prove that the entire sequence fx gkD0 converges to x . Suppose that x 0 2 H, mk 1 x 0 ¤ x  , is a cluster point of fx k g1 kD0 and that a subsequence fx gkD0 converges to 1 0 0  mk 0 x . Let " WD 2 kx  x k > 0, k0 be such that kx  x k < " and kx nk  x  k < ", for all k  k0 and let mk > nk0 . By the triangle inequality and by the Fej´er monotonicity of fx k g1 kD0 with respect to C , we obtain     2" D x 0  x    x 0  x mk  C kx mk  x  k < 2", a contradiction. Therefore, x k ! x  .

t u

3.4 Asymptotically Regular Operators Definition 3.4.1. An operator U W X ! X is called asymptotically regular (AR) if   lim U kC1 x  U k x  D 0

k!1

for all x 2 X .

112

3 Convergence of Iterative Methods

Asymptotically regular operators were studied, e.g., by Browder and Petryshyn [48], Opial [279], Baillon, Bruck and Reich [14], Bruck and Reich [51] and by Bauschke [19]. Remark 3.4.2. It is clear that any projection is asymptotically regular. In particular, the metric projection onto a nonempty closed convex subset C  H is asymptotically regular. Since the notion of asymptotically regular operators plays an important role in iterative methods for finding fixed points of operators, below we give several sufficient conditions for an operator to be asymptotically regular. Theorem 3.4.3. Let U W X ! X be an operator with a fixed point. If U is strongly quasi-nonexpansive, then U is asymptotically regular. Proof. Let U be strongly quasi-nonexpansive, x 2 X and z 2 Fix U . For x k D U k x and for some constant ˛ > 0, we have  kC1 2  2  2  2 x  z D Ux k  z  x k  z  ˛ Ux k  x k  .   Consequently, the sequence fx k  zg1 kD0 is monotone and, therefore, it converges. By setting k ! 1 in the above inequality, we obtain in the limit 2  2  kC1 U x  U k x  D Ux k  x k  ! 0, i.e., U is asymptotically regular.

t u

Corollary 3.4.4. Let U W X ! H be an operator with a fixed point. If U is a cutter, then its projected relaxation PX U W X ! X is asymptotically regular for any  2 .0; 2/. Proof. Let U be a cutter and  2 .0; 2/. It follows from Theorem 2.1.39 that U is 2 -strongly quasi-nonexpansive. Let x 2 X and z 2 Fix U . Note that z D PX z.  For x k D .PX U /k x we have  kC1 2  2  2 x  z D PX U x k  z D PX U x k  PX z  2  2 2     U x k  x k 2 .  U x k  z  x k  z     Similarly as in the proof of Theorem 3.4.3, we obtain limk U x k  x k  D 0. Since PX is nonexpansive and x k 2 X , we have       PX U x k  x k  D PX U x k  PX x k   U x k  x k  ,   i.e., limk PX U x k  x k  D 0 and PX U is asymptotically regular.

t u

3.4 Asymptotically Regular Operators

113

Corollary 3.4.5. Let T W X ! H be a firmly nonexpansive operator with Fix.PX T / ¤ ;. Then, for any  2 .0; 2/, the projected relaxation R D PX T of the operator T is asymptotically regular. Proof. Let  2 .0; 2/ and x 2 X . By Theorem 2.2.46 (iii), the operator R is strongly quasi-nonexpansive. The asymptotic regularity of R follows now from Theorem 3.4.3. t u There is an essential difference between Corollaries 3.4.4 and 3.4.5. Since Fix T  X , it is clear that Fix T \ Fix PX D Fix T ¤ ; in Corollary 3.4.4. Note that the assumption Fix.PX T / ¤ ; in Corollary 3.4.5 is weaker than the assumption Fix T ¤ ; in Corollary 3.4.4. On the other hand, for an operator T with a fixed point, the assumption that T is firmly nonexpansive in Corollary 3.4.5 is stronger than the assumption that T is a cutter in Corollary 3.4.4. Corollary 3.4.6. Let T W H ! H be a firmly nonexpansive operator with a fixed point. Then, for any  2 .0; 2/, the relaxation T of the operator T is asymptotically regular. Proof. It suffices to take X D H in Corollary 3.4.5.

t u

Corollary 3.4.7. Let Ti W X ! H, i 2 I , be firmly nonexpansive and T WD Pm ! T i D1 i i , where w D .!1 ; : : : ; !m / 2 m . If Fix.PX T / ¤ ;, then, for any  2 .0; 2/, the operator R defined by R WD PX T D PX .Id C 

m X

!i .Ti  Id//

i D1

is asymptotically regular.

Pm Proof. The operator T WD i D1 !i Ti is firmly nonexpansive (see Corollary 2.2.20). Therefore, the asymptotic regularity of R follows from Corollary 3.4.5. t u Corollary 3.4.8. Let Ti W X ! X , i 2 I , be strictly relaxed firmly nonexpansive and the composition T WD T1 T2 : : : Tm have a fixed point. Then T is asymptotically regular. Proof. By Theorem 2.2.42, the operator T is strictly relaxed firmly nonexpansive. Therefore, the asymptotic regularity of T follows from Corollary 3.4.6. t u The following result is due to Bruck and Reich (see [51, Corollary 1.1]). Theorem 3.4.9. Let T W X ! X be a strongly nonexpansive operator with a fixed point. Then T is asymptotically regular. Proof. The operator T is quasi-nonexpansive, because a strongly nonexpansive operator is nonexpansive and a nonexpansive operator with a fixed point is quasinonexpansive (see Lemma 2.1.20). Let x 0 D x 2 X , x k D T k x and z 2 Fix T be such that

114

3 Convergence of Iterative Methods

    lim x k  z  D inf lim x k  z . z2Fix T

k

k



The the sequence  and uniqueness of z follows from Theorem 3.3.2. Since  existence k 1 fx k  z g1 is decreasing, it converges and the sequence fx g kD0 kD0 is bounded. We have  k        x  z   T x k  z  D x k  z   x kC1  z  ! 0. The equality above, the strong nonexpansivity of T and the equality T z D z yield now  k    T x  x k  D .T x k  T z /  .x k  z / ! 0. t u

Therefore, T is asymptotically regular.

3.5 Opial’s Theorem and Its Consequences Below, we give a theorem, which is useful in proving the convergence of sequences generated by iterative methods for convex optimization problems. The first part of the theorem below was proved by Opial in [279, Theorem 1]. Theorem 3.5.1 (Opial, 1967). Let X  H be a nonempty closed convex subset of a Hilbert space H and U W X ! X be a nonexpansive and asymptotically regular operator with a fixed point. Then, for any x 2 X , the sequence fU k xg1 kD0 converges weakly to a point z 2 Fix U . Furthermore, z is the unique fixed point of U satisfying equality (3.3). Proof. (cf. [279, Theorem 1]) Let x 2 X , x k WD U k x and z 2 Fix U . By the nonexpansivity of U , we have           kC1 x  z D U kC1 x  z D U kC1 x  U z  U k x  z D x k  z . Therefore, the sequence fx k g1 kD0 is bounded, consequently, it contains a subsequence fx nk g1 kD0 which is weakly convergent to a point y 2 X . Since U is asymptotically regular,     lim U kC1 x  U k x  D lim Ux k  x k  D 0. k

k

Now, it follows from the demi-closedness principle (Lemma 3.2.5) that y 2 Fix U . Theorem 3.3.2 yields that y D y  ; where y  is the unique fixed point of U satisfying equality (3.3). t u If X  H is a subset of a finite dimensional Hilbert space, then theorem holds by weaker assumptions. In this case it suffices to suppose the closedness of U  Id at 0 and quasi nonexpansivity of U instead of its nonexpansivity. The following theorem holds.

3.5 Opial’s Theorem and Its Consequences

115

Theorem 3.5.2. Let X  H be a nonempty closed convex subset of a finite dimensional Hilbert space H and U W X ! X be an operator with a fixed point and such that U  Id is closed at 0. If U is quasi-nonexpansive and asymptotically regular, then, for an arbitrary x 2 X , the sequence fU k xg1 kD0 converges to a point z 2 Fix U . Proof. Let x 2 X , x k WD U k x and z 2 Fix U . By the quasi nonexpansivity of U , we have       kC1 x  z D Ux k  z  x k  z ,  consequently, fx k g1 2 X be an arbitrary cluster point kD0 is bounded. Let z k 1 nk 1 of fx gkD0 and fx gkD0 be a subsequence which converges to z . Since U is asymptotically regular,

  kUx nk  x nk k D U nk C1 x  U nk x  ! 0. By the closedness of U  Id at 0, we have U z D z , i.e., z 2 Fix U . By  Lemma 3.3.3, the whole sequence fx k g1 t u kD0 converges to z . Corollary 3.5.3. Let  2 .0; 2/ and T W H ! H be a firmly nonexpansive operator with a fixed point. Then, for any x 2 H, the sequence fTk xg1 kD0 converges weakly to a point z 2 Fix T . Proof. Let x 2 H and  2 .0; 2/. The operator T is nonexpansive (see Theorem 2.2.10) and strongly quasi-nonexpansive (see Corollary 2.2.9). Consequently, T is asymptotically regular (see Theorem 3.4.3). By Opial’s theorem, the sequence  fTk xg1 t u kD0 converges weakly to a point z 2 Fix T . By the equivalence of averaged operators and strictly relaxed firmly nonexpansive operators (see Corollary 2.2.17), Corollary 3.5.3 is equivalent to the following result, known in the literature as the Krasnosel’ski˘ı–Mann theorem (see, e.g., [56, Theorem 2.1], [361] and [57, Theorem 5.16]). Actually, Krasnosel’ski˘ı proved the strong convergence for compact operators (see [238, Theorem 1]). Theorem 3.5.4. Let X  H be a nonempty closed convex subset and U W X ! X be an averaged operator with Fix U ¤ ;. Then, for an arbitrary x 2 X , the sequence fU k xg1 kD0 converges weakly to a point z 2 Fix U . The following result follows from Corollary 3.5.3. Corollary 3.5.5. Let T W X ! H be firmly nonexpansive and such that Fix.PX T / ¤ ; and R D PX T be a projected relaxation of T , where  2 .0; 2/. Then, for an arbitrary x 2 X , the sequence fRk xg1 kD0 converges weakly to a point z 2 Fix.PX T /. Proof. The operator R is relaxed firmly nonexpansive (see Theorem 2.2.46 (i)). Therefore, the corollary follows from Corollary 3.5.3. t u

116

3 Convergence of Iterative Methods

3.6 Generalization of Opial’s Theorem Applying Opial’s theorem we can prove the convergence of sequences generated by iterating a nonexpansive and asymptotically regular operator. Below we present a generalization of Opial’s theorem which can be applied to sequences generated by a sequence of operators. First, we propose a definition of an asymptotically regular sequence of operators, which extends Definition 3.4.1. Let fx k g1 kD0  X be a sequence generated by the recurrence x kC1 D Uk x k ,

(3.5)

where x 0 2 X is arbitrary and Uk W X ! X , k  0. Definition 3.6.1. Let X  H be a nonempty closed convex subset. We say that a sequence of operators Uk W X ! X is asymptotically regular, if for any x 2 X lim kUk Uk1 : : : U0 x  Uk1 : : : U0 xk D 0 k

or, equivalently,

  lim Uk x k  x k  D 0, k

where the sequence

fx k g1 kD0

is generated by recurrence (3.5) with x 0 D x.

It is clear that an operator U W X ! X is asymptotically regular, if a constant sequence of operators Uk D U is asymptotically regular. Therefore, Definition 3.6.1 extends Definition 3.4.1 to a sequence of operators. A weaker version of the first part of the following theorem appeared in [66, Theorem 1]. Theorem 3.6.2. Let X  H be a nonempty closed convex subset, S W X ! H be an operator with a fixed point and such that S  Id is demi-closed at 0. Let fUk g1 kD0 be an asymptotically T regular sequence of quasi-nonexpansive operators k 1 Uk W X ! X such that 1 kD0 FixUk  Fix S . Let the sequence fx gkD0 be 0 generated by recurrence (3.5) with an arbitrary x 2 X . (i) If the sequence of operators fUk g1 kD0 has the property   lim Uk x k  x k  D 0 k

H)

  lim S x k  x k  D 0, k

(3.6)

 then fx k g1 kD0 converges weakly to a point z 2 Fix S . (ii) If H is finite dimensional and the sequence of operators fUk g1 kD0 has the property     lim Uk x k  x k  D 0 H) lim inf S x k  x k  D 0, (3.7) k

 then fx k g1 kD0 converges to a point z 2 FixS .

k

3.6 Generalization of Opial’s Theorem

117

Opial’s Theorem is, actually, a corollary of Theorem 3.6.2 (i). Indeed, suppose that the assumptions of Theorem 3.5.1 are satisfied. Then U is quasi-nonexpansive (see Lemma 2.1.20) and U  Id is demi-closed at 0 (see Lemma 3.2.5). We see that all assumptions of Theorem 3.6.2 are satisfied for a constant sequence of operators Uk D U , k  0. Therefore, for an arbitrary x 2 C , the sequence fU k xg1 kD0 converges weakly to a point x  2 Fix U . Proof of Theorem 3.6.2. (cf. [70, Theorem 9.9]) Let x 2 X , z 2 Fix S and the sequence fx k g1 kD0 be generated by recurrence (3.5). Since Uk is quasi-nonexpansive and Fix Uk  Fix S , we have  kC1      x  z D Uk x k  z  x k  z , k  0, Therefore, x k is Fej´er monotone with respect to Fix S , consequently, x k is bounded. (i) Suppose that condition (3.6) is satisfied. By the   asymptotic regularity 1 Uk x k  x k  ! 0, consequently, g , we have of the sequence fU k kD0  k  S x  x k  ! 0. Let x  2 X be a weak cluster point of fx k g1 and kD0 k 1  fx nk g1 kD0  fx gkD0 be a subsequence which converges weakly to x . Then, of course, kS x nk  x nk k ! 0 and x  2 Fix S , by the demi-closedness of S  Id k at 0. Since x  is an arbitrary weak cluster point of fx k g1 er kD0 and x is Fej´ monotone with respect to Fix S , the weak convergence of the whole sequence  fx k g1 kD0 to x follows from Lemma 3.3.3. (ii) Let H be finite dimensional and suppose that condition (3.7)  is satisfied.  By the k k  asymptotic regularity of the sequence fUk g1 ! 0, kD0 , we have Uk x  x k 1 consequently, limk kS x nk  x nk k D 0 for a subsequence fx nk g1  fx gkD0 . kD0 nk 1 Due to the boundedness of x nk , there is a subsequence fx mnk g1  fx gkD0 kD0 which converges to a point x  2 X . Since S  Id is demi-closed at 0, we have  x  2 Fix S . The convergence of the whole sequence fx k g1 kD0 to x follows now from Lemma 3.3.4. t u Remark 3.6.3. We do not suppose that the operators Uk are nonexpansive in Theorem 3.6.2. It allows application of this theorem to a sequence of relaxed cutters or, equivalently, to a sequence of strongly quasi-nonexpansive operators. We only suppose that the sequence of operators fUk g1 kD0 is asymptotically regular and that condition (3.6) (or (3.7) in the finite dimensional case) is satisfied for an operator S W X ! H such that S  Id is demi-closed at 0, in order to ensure the claims of Theorem 3.6.2. Condition (3.6) is satisfied if, e.g., the inequality kUk x  xk  ˛ kS x  xk

(3.8)

holds for all x 2 X , for all k and for some ˛ > 0. Condition (3.7) is satisfied if, e.g., inequality (3.8) holds for all x 2 X , for infinitely many k and for some ˛ > 0.

118

3 Convergence of Iterative Methods

Remark 3.6.4. It follows from the proof that Theorem 3.6.2 remains true if we replace the assumption that fUk g1 kD0 is asymptotically regular and the assumption (3.6) in case (i) or (3.7) in case (ii) assumption    by a weaker  limk!1 S x k  x k  D 0 in case (i) or lim infk!1 S x k  x k  D 0 in case (ii), respectively. The formulation presented in Theorem 3.6.2 is preferred, because in applications, the operators Uk are often relaxed cutters with relaxation parameters guaranteeing the asymptotic regularity of fUk g1 kD0 . Furthermore, various practical algorithms which apply relaxed cutters have properties which yield (3.6), (3.7) or some related conditions.

3.7 Opial-Type Theorems for Cutters In this section we present modifications and applications of Theorem 3.6.2 for sequences of firmly nonexpansive operators, sequences of cutters and for generalized relaxations of cutters. In all these three cases we adopt Opial’s theorem or its generalization (Theorem 3.6.2). Consider the following recurrence x kC1 D PX .x k C k .Tk x k  x k //,

(3.9)

where x 0 2 X , k 2 Œ0; 2 and Tk W X ! H. A stronger version of the following result was presented in [70, Proposition 9.12]. Corollary 3.7.1. Let S W X ! H be an operator with a fixed point and such that S  Id is demi-closed at 0, x 0 2 X and the sequence fx k g1 kD0  X be generated by recurrence (3.9), where lim inf  .2   / > 0 and T W k k k k X ! H is a sequence of T cutters with 1 Fix T  Fix S . Then: k kD0 T1 (i) For all z 2 kD0 Fix Tk 2  2   kC1 2 x  z  x k  z  k .2  k / Tk x k  x k  .

(3.10)

(ii) If the following implication holds   lim Tk x k  x k  D 0 k

H)

  lim S x k  x k  D 0, k

(3.11)

then x k converges weakly to a fixed point of S . (iii) If H is finite dimensional and the following implication holds   lim Tk x k  x k  D 0 k

H)

then x k converges to a fixed point of S .

  lim inf S x k  x k  D 0, k

(3.12)

3.7 Opial-Type Theorems for Cutters

119

T1 Proof. Let C D kD0 Fix Tk and z 2 C . Denote Uk WD PX .Id C k .Tk  Id//. Inequality (3.10) and the strong nonexpansivity of Uk follow from  k quasi  x  z converges as a bounded and monotone Corollary 2.2.25. Consequently,   Tk x k  x k  D 0. Since lim infk k .2  k / > 0, we sequence and lim  .2   / k k k   have limk Tk x k  x k  D 0. The rest part of the theorem can be proved by similar arguments as in the proof of Theorem 3.6.2. t u A special case of Corollary 3.7.1 (ii) with X D H and Tk D S for all k  0 can be found in [253, Theorem 1]. Note that condition (3.11) holds in this case automatically. Bauschke and Combettes proved a theorem [24, Theorem 2.9 (i)] which is related to Corollary 3.7.1 (ii). They supposed that X D H, k D  2 Œ1; 2/ and, instead T1 of (3.11), supposed that all cluster points of fx k g1 belong to kD0 Fix Tk (see kD0 also [124, Theorem 2.6 (iii)] and [352, Theorems 2.1–2.3] for related results). Remark 3.7.2. If int Fix S ¤ ;, then the range of the relaxation parameters in Corollary 3.7.1 (ii) and (iii) can be extended. By Proposition 2.1.41, for any z 2 int Fix S there is " > 0 such that 2  2  2  kC1 x  z  x k  z  k .2 C "  k / Tk x k  x k  .

(3.13)

If lim infk k > 0 and lim sup  k  2, then, similarly, as in in the proof of Corollary 3.7.1, we obtain limk Tk x k  x k  D 0, because lim inf k .2 C 2"  k /  lim inf k .2  k / C 2" lim inf k > 0. k

k

k

Therefore, if int Fix T ¤ ;, then the results of Corollary 3.7.1 (ii) and (iii) remain true if we suppose that lim infk k > 0 and lim sup k  2 instead of lim infk k .2  k / > 0. This explains the convergence of sequences generated by several projection methods which apply the extended range of relaxation parameters (see, e.g., [54, 266]). If we iterate inequality (3.10) k-times, we obtain for an arbitrary z 2 Fix S the following inequality k  kC1 2 2  2 X  x  z  x 0  z  l .2  l / Tl x l  x l  .

(3.14)

lD0

If we know an upper approximation R of d.x 0 ; Fix S /, then inequality (3.14) can be applied to the following estimation of d.x kC1 ; Fix S / d 2 .x kC1 ; Fix S /  R2 

k X lD0

where R  d.x 0 ; Fix S /.

 2 l .2  l / Tl x l  x l  ,

120

3 Convergence of Iterative Methods

In what follows, we will apply the following property of real sequences fk g1 kD0  Œ0; 2. lim inf k .2  k / > 0 ” .lim inf k > 0 and lim sup k < 2/. k

k

Corollary 3.7.3. Let T W X ! H be a cutter and such that T  Id is demi-closed at 0 (e.g., a firmly nonexpansive operator with Fix T ¤ ;). Further, let fx k g1 kD0 be generated by the recurrence x kC1 D PX .x k C k .T x k  x k //, where x 0 2 X and k 2 Œ0; 2. If lim inf k .2  k / > 0, then x k converges weakly to a fixed point of T . Proof. Denote Tk D T , for all k  0, and S D T . Then implication (3.11) is obvious. Therefore, the corollary follows from Corollary 3.7.1. u t A related result to Corollary 3.7.3 was obtained by Reich [294, Theorem 2] for a uniformly convex Banach space with a Frech´et differentiable norm, where it supposed (in an equivalent form) that T is firmly nonexpansive and that Pis 1 kD0 k .2  k / D C1 instead of lim inf k .2  k / > 0. If we take k D  2 .0; 2/ and X D H in Corollary 3.7.3, then we obtain Corollary 3.5.3. Denote Uk WD PX .Id C k .Sk  Id// for an operator Sk W X ! H and for k 2 .0; 2, k  0. Corollary 3.7.4. Let U W X ! H be an operator with a fixed point and such that U  Id is demi-closed at 0, x 0 2 X and a sequence fx k g1 kD0 be generated by the recurrence x kC1 D PX .x k C k .Sk x k  x k //, (3.15) where k 2 .0; 2, lim supk k < 2, and T fSk g1 kD0 is a sequence of firmly nonexpansive operators Sk W X ! H such that 1 kD0 Fix .PX Sk /  Fix U , k  0. (i) If the following implication holds   lim Uk x k  x k  D 0 k

H)

  lim Ux k  x k  D 0, k

(3.16)

then x k converges weakly to a fixed point of U . (ii) If H is finite dimensional and the following implication holds   lim Uk x k  x k  D 0 k

H)

then x k converges to a fixed point of U .

  lim inf Ux k  x k  D 0, k

(3.17)

3.7 Opial-Type Theorems for Cutters

121

Fig. 3.1 Operators Tk and Uk

x +¹k (Sk x − x)

Sk x

x

Tk x

Uk x

X

(iii) Let S W X ! H be firmly nonexpansive. If Sk D S and k 2 Œ"; 2  " for all k  0 and some " > 0, then implication (3.16) holds for U WD PX S" . Consequently, x k converges weakly to a point x  2 Fix U D Fix PX S . 4 Proof. (i) and (ii) The operator Uk W X ! X is 4 -relaxed firmly nonexpansive k and Fix Uk D Fix .PX Sk / ¤ ;, k  0 (see Theorem 2.2.46). Let Tk WD .Uk / 4k 4 (see Fig. 3.1). Then Tk is firmly nonexpansive (see Corollary 2.2.19), Tk is a cutter (see Theorem 2.2.5 (i)), Fix Tk D Fix Uk ¤ ; (see Remark 2.1.4) and Uk D .Tk / 4 (see Remark 2.1.3). It is clear that recurrence (3.15) is a special 4k

case of (3.9) with X D H and k D

4 . 4k

  Uk x k  x k  D

Note that Tk;k W X ! X and that

 4  Tk x k  x k  , 4  k

i.e., condition (3.16) is equivalent to (3.11) with S WD U and condition (3.17) is equivalent to (3.12) with S WD U . 4 Setting k D 4 we easily obtain lim infk k .2  k / > 0. Now the weak k convergence in (i) and the convergence in (ii) follows from Corollary 3.7.1 (i) and (ii), respectively. (iii) Suppose that Sk D S in (3.15) and k 2 Œ"; 2  " for all k  0 and for some " 2 .0; 1/. Let U WD PX S" D PX .Id C ".S  Id//. It is clear that Fix U D Fix PX S (see Corollary 1.2.10). Furthermore, U is nonexpansive as a composition of nonexpansive operators PX and S" . Consequently, U  Id is demi-closed at 0 (see Lemma 3.2.5). It follows from Corollary 2.2.26 that     Uk x k  x k  D PX .x k C k .S x k  x k //  x k       PX .x k C ".S x k  x k //  x k  D Ux k  x k  , i.e., implication (3.16) is satisfied. Consequently, x k converges weakly to a fixed point of PX S . t u Note the difference in assumptions of Corollaries 3.7.1 and 3.7.4. An iteration in recurrence (3.9) is defined by a projected relaxation of a cutter Tk with Fix Tk  Fix S , while iteration (3.15) is defined by a projected relaxations of a firmly nonexpansive operator Sk for which we do not suppose that Fix Sk ¤ ;. We only

122

3 Convergence of Iterative Methods

suppose that Fix.PX Sk /  Fix S ¤ ;. Furthermore, conditions (3.11) and (3.16) have a different nature. In the first one, we apply the cutters Tk , while in the other one—projected relaxations of firmly nonexpansive operators Sk . Remark 3.7.5. Similarly as in Theorem 3.6.2, we can replace the assumption on the demi-closedness of S Id at 0 and assumptions (3.11) and (3.16) in Corollaries 3.7.1 and 3.7.4 with a weaker one:     • If limk Tk x k  x k  D 0 or limk Uk x k  x k  D 0, respectively, then all weak cluster points of x k lie in Fix S or in Fix U , respectively. In these cases the claims remain true (cf. Remark 3.6.4). Conditions of a similar form were proposed by Schott [303, Theorem 3] and by Bauschke and Borwein [22, Definition 3.7] for slightly different recurrences. Now we present an Opial-type theorem for generalized relaxations of algorithmic operators (see Sect. 2.4 for definitions). Theorem 3.7.6. Let U W X ! H be a strongly oriented operator with a fixed point and such that U  Id is demi-closed at 0, and the sequence fx k g1 kD0  X be generated by the recurrence x kC1 D PX U;k .x k /,

(3.18)

where x 0 2 X , lim infk k .2 k / > 0 and the step size function  W X ! .0; C1/ satisfies the following condition ˛  .x/ 

hz  x; Ux  xi kUx  xk2

(3.19)

for all x 2 X , for all z 2 Fix U and for some ˛ > 0. Then 2  2  2  kC1 x  z  x k  z  k .2  k / 2 .x k / Ux k  x k 

(3.20)

and x k converges weakly to a fixed point of U . Proof. Let z 2 Fix U , " > 0 and k0  0 be such that k 2 Œ"; 2  " for all k  k0 . It follows from the second inequality in (3.19) that U is a cutter and that U;k k is 2 k -strongly quasi-nonexpansive (see Corollary 2.4.5). This, together with the nonexpansivity of the metric projection and with the first inequality in (3.19), gives  kC1 2  2 x  z D PX U;k x k  z  2 D PX U;k x k  PX z  2  U; x k  z k

3.8 Strong Convergence of Fej´er Monotone Sequences

123

 2 2  k   U; x k  x k 2  x k  z  k k  k 2  2 D x  z  k .2  k / 2 .x k / Ux k  x k   2  2  x k  z  "2 ˛ 2 Ux k  x k  k for all k  k0 . Therefore,  er monotone with respect to Fix U , consequently  kx iskFej´ k  x is bounded and Ux  x  ! 0. Let x  2 X be a weak cluster point of nk 1 k 1  fx k g1 kD0 and fx gkD0  fx gkD0 be a subsequence which converges weakly to x .  By the demi-closedness of U  Id, we have x 2 Fix U . This, together with the Fej´er monotonicity of fx k g1 kD0 with respect to Fix U , gives the weak convergence of x k to x  (see Lemma 3.3.3). t u

If we iterate inequality (3.20) k-times, we obtain k 2  2 X  2  kC1 x  z  x 0  z  l .2  l / 2 .x l / Ux l  x l 

(3.21)

lD0

for any z 2 Fix U . If we know an upper approximation R of d.x 0 ; Fix U /, then inequality (3.21) can be applied to the following estimation of d.x kC1 ; Fix U / d 2 .x kC1 ; Fix U /  R2 

k X

 2 l .2  l / 2 .x l / Ux l  x l  ,

(3.22)

lD0

where R  d.x 0 ; Fix U /.

3.8 Strong Convergence of Fej´er Monotone Sequences Theorem 3.6.2 as well as other results presented in the last two sections, yield only the weak convergence of a sequence fx k g1 kD0 generated by a sequence of asymptotically regular and quasi-nonexpansive operators to a fixed point of a nonexpansive operator. If H is finite dimensional, the weak convergence is equivalent to the strong convergence. Under some assumptions on the structure of the convex minimization problem we can obtain, however, the strong convergence in any Hilbert space. The following theorem is due to Bauschke and Borwein (see [22, Theorem 2.16 (iii)]). er monotone with respect to a subset Theorem 3.8.1. Let fx k g1 kD0  H be Fej´ C  H. If int C ¤ ;, then fx k g1 converges strongly. kD0

124

3 Convergence of Iterative Methods

Proof. Let z 2 int C and " > 0 be such that B.z; "/  C . Denote d k D " .x k  x kC1 /. Then, obviously, we have zk D z C d k 2 C . By the Fej´er kx k x kC1 k monotonicity of fx k g1 kD0 with respect to C , 2  2  kC1 x  z C d k   2hx kC1  z; d k i 2  D x kC1  z  d k   2  2  2 D x kC1  zk   x k  zk  D x k  z  d k   2  2 D x k  z C d k   2hx k  z; d k i, i.e., 2  2  kC1 x  z  x k  z  2hx k  x kC1 ; d k i  2   D x k  z  2" x k  x kC1  and

  k      x  x kC1   1 x k  z2  x kC1  z2 . 2"

Consequently,  X  k      m1  x  x kCm   x kCl  x kClC1   1 x k  z2  x kCm  z2 2" lD0  k(3.23)  x  z with respect to C , for all m  1. By the Fej´er monotonicity of fx k g1 kD0 converges and inequalities (3.23) yield that fx k g1 kD0 is a Cauchy sequence, therefore it converges strongly. t u Before we formulate our next result, we prove some auxiliary lemmas. T Lemma 3.8.2. Let C WD i 2J H .ai ; ˇi /, where ai 2 H and ˇi 2 R, i 2 J , and fx k g1 er monotone with respect to C . Then x k 2 x 0 C Linfai ; i 2 J g for kD0 be Fej´ all k  0. P Proof. Denote V WD Linfai ; i 2 J g and fix k  0. Since x kC1 D x 0 C klD0 .x lC1  x l /, it suffices to prove that x lC1  x l 2 V for all l 2 f0; 1; : : : ; kg. We leave it to the reader to check that the Fej´er monotonicity of fx k g1 kD0 with respect to C yields that   1  lC1 x (3.24) w  x l ; x lC1  x l  0 2 for any w 2 C (see equivalence (2.11)). Let l 2 f0; 1; : : : ; kg, z 2 C , and ul 2 V and vl 2 V ? be such that x lC1  x l D ul C vl . Suppose that x lC1  x l … V . Then vl ¤ 0. For any ˛ 2 R and i 2 J , we have

3.8 Strong Convergence of Fej´er Monotone Sequences

125

hai ; z C ˛vl i D hai ; zi C ˛hai ; vl i D hai ; zi  ˇi , i.e., z C ˛vl 2 C . Then, for ˛l <

1 2

 l 2 u   hz; x lC1  x l i 1 C  2 vl  2

we obtain 1 hz C ˛l vl  .x lC1  x l /; x lC1  x l i 2 1 D hz; x lC1  x l i C h˛l vl  .x lC1  x l /; x lC1  x l i 2 1 D hz; x lC1  x l i C h˛l vl  .ul C vl /; ul C vl i 2 1  2 1  2 D hz; x lC1  x l i C .˛l  / vl   ul  < 0, 2 2 a contradiction with (3.24). Therefore, x lC1  x l 2 V which completes the proof. t u Lemma 3.8.3. Let X be a finite dimensional affine subspace of H and k k fx k g1 kD0  X . If x converges weakly to a point z 2 X , then x converges strongly to z. Proof. Let x k * z 2 X . Denote y k WD x k  x 0 . It is clear that y k 2 V WD X  x 0 , y k * z  x 0 and V is a finite dimensional subspace of H. Therefore, y k ! z  x 0 and limk x k D limk y k C x 0 D z. t u A special case of the following Theorem was proved by Gurin et al. (see [196, Theorem 1 (d)]). er monotone with respect to a subset C Theorem 3.8.4. Let fx k g1 kD0  H be Fej´ being the intersection of finitely many half-spaces. If fx k g1 kD0 converges weakly, then it converges strongly. T Proof. Let C WD i 2I H .ai ; ˇi /, where ai 2 H and ˇi 2 R, i 2 I WD f1; 2; : : : ; mg. Denote X D x 0 C Linfai ; i 2 I g. Then X is a finite dimensional k 1 affine subspace. By Lemma 3.8.2, fx k g1 kD0  X . Suppose that fx gkD0 converges k 1 weakly. Then the strong convergence of fx gkD0 to a point of X follows from Lemma 3.8.3. t u Corollary 3.8.5. Suppose that a sequence fx k g1 kD0 generated by recurrence (3.5) satisfies the assumptions of Theorem 3.6.2. Furthermore, suppose that one of the following conditions is satisfied: (i) X is a finite dimensional affine subspace of H, T Fix Uk has a nonempty interior, (ii) The subset F WD T1 kD0 (iii) The subset F WD 1 Fix Uk is an intersection of finitely many half-spaces. kD0

126

3 Convergence of Iterative Methods

Then for an arbitrary x 0 2 X the sequence fx k g1 kD0 generated by recurrence (3.5) converges strongly to a fixed point of S . Proof. The assumptions of Theorem 3.6.2 guarantee the weak convergence of fx k g1 kD0 to a fixed point of S . The strong convergence in case (i) follows from Lemma 3.8.3, in case (ii)—from Theorem 3.8.1 and in case (iii)—follows from Theorem 3.8.4. t u Sufficient conditions for the strong convergence of Fej´er monotone sequences, in particular of sequences generated by several projection methods for the convex feasibility problems, were presented in [196], [283, Theorem 1.1], [16, 308], [22, Sect. 5], [255] and in [256]. The strong convergence also holds under additional assumptions on the operators which generate the sequences fx k g1 kD0 . For details see [36, Chaps. 3 and 4], [14], [114, Chap. 6], [272] and [295, Theorem 1.7]. The papers [32, 213, 258] contain examples of projection methods for the convex feasibility problems, which generate sequences converging weakly but not strongly. In the paper [24], a method is described which transforms algorithms of type (3.9) generating weakly convergent sequences to sequences which are strongly convergent. The method is based on the idea of Haugazeau [202].

3.9 Relationships Among Algorithmic Operators In this section we recall relationships among nonexpansive operators, firmly nonexpansive operators, relaxed firmly nonexpansive operators, cutters, quasinonexpansive operators, strongly quasi-nonexpansive operators, strongly nonexpansive and asymptotically regular operators which were presented in Chap. 2. These relationships are presented in the form of a diagram. In Fig. 3.2, T W X ! H and U WD T D Id C .T  Id/ is a -relaxation of T , where  2 .0; 2/. The figure shows an important role of firmly nonexpansive operators, cutters and strongly nonexpansive operators. It follows from Corollary 2.2.9, from Theorem 3.4.3 and from Opial’s Theorem that for  2 .0; 2/, sequences generated by the relaxation T of a firmly nonexpansive operator T converge weakly to its fixed point, if Fix T ¤ ;. We also see that the weak convergence is guaranteed if U is nonexpansive, Fix T ¤ ; and at least one of the conditions below is satisfied for U D T : (i) T is a cutter (Theorems 2.1.39 and 3.4.3), (ii) U is strongly quasi-nonexpansive (Theorem 3.4.3), (iii) U is strongly nonexpansive (Theorem 3.4.9). Note that a relaxed firmly nonexpansive operator is nonexpansive (see implication (i))(ii) in Theorem 2.2.10). A strongly nonexpansive operator is nonexpansive (see Definition 2.3.1). In cases (i) and (ii) we should additionally suppose that U WD T is nonexpansive if we want to apply Opial’s Theorem. Note, however,

3.10 Exercises

127

U

U

/2-AV

U

-RFNE

T - FNE

FixT

FixU =(2- )/

T 2=T

T

cutter

NE

=(2- )/

U

-SQNE

U

contraction

U

SNE

FixU

U - AR U NE FixU

=2

U

QNE

Ukx x* FixU

Fig. 3.2 Relationships among algorithmic operators

that the generalization of Opial’s Theorem (Theorem 3.6.2) applied to a sequence of relaxations Uk of operators Tk , which satisfy one of the conditions (i)–(ii) above, does not require the nonexpansivity of Uk . In this case it suffices to suppose that implication (3.6) holds for a nonexpansive operator S W X ! H in order to guarantee the weak convergence of sequences generated by the sequence of operators Uk . In one of the next chapters we will present examples which show how to construct sequences of such operators Uk for a given nonexpansive operator S , which guarantee implication (3.6). The basic tools in the proofs of convergence of sequences generated by such sequences of operators are the generalizations of Opial’s Theorem presented in Sects. 3.6 and 3.7.

3.10 Exercises Exercise 3.10.1. Let H WD R2 , C WD Œ0; 1 f0g, x k WD .1 C k1 ; .1/k /. Check that fx k g1 er monotone with respect to C , but fx k g1 kD0 is strictly Fej´ kD0 does not converge. Exercise 3.10.2. Let H WD l2 and x k WD ek (see Example 3.2.2). Show that the 2kC1 sequence fy k g1 D x k and y 2k D x 1 , k  0, is Fej´er monotone with kD0 with y respect to f0g and has two weak cluster points: 0 and x 1 (the latter one is even a strong cluster point of y k ).

•

Chapter 4

Algorithmic Projection Operators

As we have mentioned in Chap. 2, the algorithms (or methods) for solving convex optimization problems are defined by algorithmic operators. Usually, particular iterations of these algorithms have the form x C D Ux, where x is the current approximation of a solution and x C is a next approximation (also called an actualization or an update). If an operator U describing this actualization belongs to the class of strongly quasi-nonexpansive operators (or, equivalently, to the class of relaxed cutters), then we call them algorithmic projection operators. The name projection operator can be explained by the following property of a cutter T W H ! H: For any x … Fix T , T x is the metric projection of x onto a hyperplane H.x  T x; hT x; x  T xi/ D fy 2 H W hy  T x; x  T xi D 0g which separates the point x from the subset Fix T . In this chapter we give examples of algorithmic projection operators and we show their properties. These properties are, in most cases, corollaries of general properties of operators presented in Chap. 2. Since the metric projection plays an important role in the construction of algorithmic projection operators, in Sect. 4.1 we give the formulas for the metric projection onto simple closed convex subsets.

4.1 Examples of Metric Projections 4.1.1 Metric Projection onto a Hyperplane A hyperplane in a Hilbert space H has the form H.a; ˇ/ WD fz 2 H W ha; zi D ˇg, A. Cegielski, Iterative Methods for Fixed Point Problems in Hilbert Spaces, Lecture Notes in Mathematics 2057, DOI 10.1007/978-3-642-30901-4 4, © Springer-Verlag Berlin Heidelberg 2012

129

130

4 Algorithmic Projection Operators

Fig. 4.1 Metric projection onto a hyperplane

x a

H(a,β)

PH(a,β ) x

where a 2 H, a ¤ 0 and ˇ 2 R. It is clear that H.a; ˇ/ is a closed convex subset. We show that ha; xi  ˇ a. (4.1) PH.a;ˇ/ x D x  kak2 Let y WD x  kak2 .ha; xi  ˇ/a. A straightforward computation shows that ha; yi D ˇ, i.e., y 2 H.a; ˇ/. Furthermore, for any z 2 H.a; ˇ/, we have hx  y; z  yi D

ha; xi  ˇ kak2

.ha; zi  ha; yi/ D 0.

Now, the characterization of the metric projection (see Theorem 1.2.4) yields y D PH.a;ˇ/ x (Fig. 4.1). We can also write PH.a;ˇ/ x D x 

.x/ kak2

a;

where .x/ D ha; xi  ˇ is the residuum of the equality ha; xi D ˇ at the point x 2 H. Example 4.1.1. In a Euclidean space (Rn with the standard inner product) we have H.a; ˇ/ D fx 2 Rn W a> x D ˇg, where a 2 Rn and ˇ 2 R. In this case equality (4.1) can be written in the form PH.a;ˇ/ x D x 

a> x  ˇ

or PH.a;ˇ/ x D x 

kak2 .x/ kak2

a

a,

where .x/ D a> x  ˇ. Example 4.1.2. Denote by PCG x the metric projection of x 2 Rn onto a closed convex subset C of Rn equipped with the inner product h; iG induced by a positive definite matrix G of type n  n, defined by hx; yiG WD x > Gy. We call PCG x an G oblique projection. We calculate the oblique projection PH.a;ˇ/ x for x 2 Rn , where

4.1 Examples of Metric Projections

131 J G, x, ρ)

Fig. 4.2 Oblique projection onto a hyperplane

x

H a,β

G PH a,β x

a 2 Rn , a ¤ 0 and ˇ 2 R. It follows from the definition of the metric projection that P G .x/ is a solution of the following differentiable minimization problem minimize f .z/ D 12 kz  xk2G subject to ha; zi D ˇ z 2 Rn

(4.2)

(see Fig. 4.2). By the properties of the metric projection, this problem has a unique G solution. Since the minimization problem (4.2) is convex, the solution z D PH.a;ˇ/ x can be derived from the Karush–Kuhn–Tucker conditions (see Theorem 1.3.7). The Lagrange function L W Rn  R ! R has the form L.z; / D

1 kz  xk2G C .ha; zi  ˇ/ 2

and the KKT-point .z; / is the unique solution of the KKT-system G.z  x/ C a D 0 ha; zi D ˇ. Consequently,

hG 1 a; G.z  x/ C ai D 0

and D

ha; z  xi ha; xi  ˇ D . 1 ha; G ai kak2G 1

If we put the computed Lagrange multiplier  into the first equation of the KKTsystem, we obtain ha; xi  ˇ 1 P G .x/ D x  G a. (4.3) kak2G 1 Example 4.1.3. In H D L2 .Œ˛; ˇ/ with the inner product defined by Z

ˇ

hf; gi WD

f .x/g.x/dx ˛

132

4 Algorithmic Projection Operators x

Fig. 4.3 Metric projection onto an affine subspace H(a2; ¯2 )

PH x H (a1; ¯1 )

H

we have

Z

ˇ

H.g;  / D fh 2 L2 .Œ˛; ˇ/ W

g.x/h.x/dx D  g,

˛

where g 2 L2 .Œ˛; ˇ/, g ¤ 0, and  2 R. By applying equality (4.1), we obtain Rˇ PH.g; / f D f 

˛

g.x/f .x/dx   g. Rˇ 2 ˛ g .x/dx

4.1.2 Metric Projection onto a Finite Dimensional Affine Subspace n Let Tm H be an intersection of an finite number of hyperplanes in R , i.e., H D i D1 H.ai ; ˇi /, where ai 2 R , ai ¤ 0 and ˇi 2 R, i D 1; 2; : : : ; m. Suppose that H ¤ ;. Obviously, H is an affine subspace in Rn . It is clear that H is closed and convex as an intersection of closed convex subsets. Let A D Œa1 ; : : : ; am > be a matrix, with rows ai and b D .ˇ1 ; : : : ; ˇm /. Suppose that Rn is p equipped with a standard inner product hx; yi WD x > y and with the norm kk WD hx; xi. Then H D fy 2 Rn W Ay D bg. Let x 2 Rn be arbitrary. Note that

H  x D fy 2 Rn W A.y C x/ D bg. We have PH x 0 D AC .b  Ax/, where AC denotes the Moore–Penrose pseudoinverse of A (see Example 1.3.11). Lemma 1.2.6 yields PH x D x  AC .Ax  b/.

(4.4)

If A has full row rank, then AC D A> .AA> /1 and PH x D x  A> .AA> /1 .Ax  b/

(4.5)

(Fig. 4.3). If m D 1, we have A D a> 2 Rn and b D ˇ 2 R. In this case we have AA> D a> a D kak2 and (4.5) obtains the form (4.1).

4.1 Examples of Metric Projections

133 x

Fig. 4.4 Metric projection onto a half-space

a H−(a,β) PH − (a,β) x

4.1.3 Metric Projection onto a Half-Space A half-space in a Hilbert space H has the form H .a; ˇ/ WD fz 2 H W ha; zi  ˇg, where a 2 H, a ¤ 0 and ˇ 2 R. It is clear that H .a; ˇ/ is closed and convex. We show that ( a if ha; xi > ˇ x  ha;xiˇ kak2 (4.6) PH .a;ˇ/ x D x if ha; xi  ˇ. Equality (4.6) is clear if x 2 H .a; ˇ/, i.e., ha; xi  ˇ. Let now ha; xi > ˇ and y WD x  kak2 .ha; xi  ˇ/a. We easily see that ha; yi D ˇ, i.e., y 2 H.a; ˇ/  H .a; ˇ/. Furthermore, for an arbitrary z 2 H .a; ˇ/, we have hx  y; z  yi D

ha; xi  ˇ kak2

.ha; zi  ha; yi/  0.

Now, it follows from the characterization of the metric projection (see Theorem 1.2.4) that y D PH .a;ˇ/ x. We can also write PH .a;ˇ/ x D x 

.ha; xi  ˇ/C kak2

a

(Fig. 4.4) or PH .a;ˇ/ x D x 

C .x/ kak2

a,

where C .x/ D .ha; xi  ˇ/C .

4.1.4 Metric Projection onto a Band A band in a Hilbert space H has the form C WD fx 2 H W ˇ1  ha; xi  ˇ2 g,

(4.7)

134

4 Algorithmic Projection Operators x

Fig. 4.5 Metric projection onto a band

PC x

z = PC z

PC y

C y

where a 2 H, a ¤ 0, ˇ1 ; ˇ2 2 R and ˇ1 < ˇ2 . Obviously, C is closed and convex as an intersection of closed convex subsets H .a; ˇ2 / and H .a; ˇ1 /. By equality (4.6), we easily obtain 8 ˆ <x  PC x D x ˆ :x 

ha;xiˇ2 a kak2 ha;xiˇ1 a kak2

if ha; xi > ˇ2 if ˇ1  ha; xi  ˇ2 if ha; xi < ˇ1

(see Fig. 4.5).

4.1.5 Metric Projection onto the Orthant Let RnC be the nonnegative orthant in the Euclidean space Rn . Obviously, RnC is closed and convex. We show that PRnC .x/ D xC . Let y WD xC . It is clear that y 2 RnC . Let z 2 RnC be arbitrary. Since x D xC  x , hx ; zi  0 and hxC ; x i D 0, we have hx  y; z  yi D hx  xC ; z  xC i D hx ; z  xC i D hx ; zi C hx ; xC i  0. The characterization of the metric projection (see Theorem 1.2.4) yields PRnC x D xC (see Fig. 4.6). In a similar way we can show that PRn x D x :

(4.8)

We obtain similar results in H D L2 .Œ˛; ˇ/, where 1  ˛ < ˇ  C1, Rˇ equipped with the inner product hf; gi WD ˛ f .x/g.x/dx, for the subsets LC 2 WD

4.1 Examples of Metric Projections

135

Fig. 4.6 Metric projection onto RnC

R +n

PR +n x

x

w = P R +n w PR+n y

y

P R +n z

z

ff 2 L2 .Œ˛; ˇ/ W f  0g and L 2 WD ff 2 L2 W f  0g. Obviously, both subsets are closed and convex. By applying the same arguments as above, we obtain PLC f D fC 2

and PL f D f . 2 Example 4.1.4. Let Q WD fu 2 Rm W u  bg, where b 2 Rm . We have Q D Rm  Cb or, equivalently, Rm  D Q  b. Applying equalities (1.12) and (4.8), we obtain PQ .y/ D PRm .y  b/ C b D .y  b/ C b D y  .y  b/C . If we take y WD Ax and r.x/ WD Ax  b, we obtain .PQ  Id/Ax D .Ax  b/C D rC .x/.

(4.9)

This equality will be used in the sequel.

4.1.6 Metric Projection onto Box Constraints The box constraints in Rn have the form a  z  b, where a; b 2 Rn , a  b. Let C WD fz 2 Rn W a  z  bg. Obviously, the subset C is closed and convex as the Cartesian product of closed intervals, C D Œ˛1 ; ˇ1   : : :  Œ˛n ; ˇn .

(4.10)

In a one-dimensional case the metric projection of a point  2 R onto the interval Œ˛; ˇ has the form PŒ˛;ˇ  D medianf; ˇ; ˛g WD maxfminf; ˇg; ˛g.

136

4 Algorithmic Projection Operators

Fig. 4.7 Metric projection onto box constraints

y ¯2

x

PC x

®2

PC y z = PC z C

®1

¯1 f

Fig. 4.8 Metric projection onto C D ff 2 L2 W g  f  hg

h g PC f a

b

It follows from the above equality and and from Lemma 1.2.8 that PC x D maxfminfx; bg; ag (see Fig. 4.7). This formula can also be used when ˛j D 1 or ˇj D C1 for some j , j D 1; 2; : : : ; n. We obtain a similar formula in H D L2 WD L2 .Œa; b/ for the metric projection of f 2 L2 onto the subset C WD ff 2 L2 W g  f  hg, where g; h 2 L2 and g  h. Obviously, C is closed and convex. It follows from the characterization of the metric projection (see Theorem 1.2.4) that PC f D maxfminff; hg; gg, i.e.,

8 < g.x/ if f .x/ < g.x/ .PC f /.x/ D f .x/ if g.x/  f .x/  h.x/ : h.x/ if f .x/ > h.x/

(see Fig. 4.8). The formula above can be extended to the case C D ff 2 L2 W g  f  h on S g, where S  Œa; b is a measurable subset and g  h on S . We leave it to the reader to check that 8 < g.x/ if f .x/ < g.x/ and x 2 S .PC f /.x/ D f .x/ if g.x/  f .x/  h.x/ or x … S : h.x/ if f .x/ > h.x/ and x 2 S .

4.1 Examples of Metric Projections

137 x

Fig. 4.9 Metric projection onto a ball PB(z,ρ) x y = PB(z,ρ) y z ρ

B(z,ρ)

If we take g D h on S , then C D ff 2 L2 W f .x/ D g.x/ for x 2 S g and we obtain, in particular  g.x/ if x 2 S .PC f /.x/ D f .x/ if x … S . The subset C defined by (4.10) is a special case of a polytope. The metric projection onto the latter subset can be evaluated by several methods, e.g., by a finite procedure called an active set method (see [339] for details).

4.1.7 Metric Projection onto a Ball Obviously, a ball B.z; /  H, where z 2 H and  > 0, is a nonempty closed convex subset. It follows easily from the characterization of the metric projection (see Theorem 1.2.4) and from the Cauchy–Schwarz inequality that ( PB.z;/ .x/ D

x zC

 kxzk .x

if kx  zk    z/ if kx  zk > 

(see Fig. 4.9). The details are left to the reader.

4.1.8 Metric Projection onto an Ellipsoid An ellipsoid in Rn has the form C D J.D; z; / WD fy 2 Rn W .y  z/> D.y  z/  g, where D is a positive definite matrix, z 2 Rn and  > 0. An ellipsoid is a closed convex subset as a sublevel set of a convex function f .x/ D

1 .x  z/> D.x  z/ 2

138

4 Algorithmic Projection Operators x

Fig. 4.10 Metric projection onto an ellipsoid

PC x y = PC y z J (D; z ; ½)

(note that the Hessian r 2 f D D is positive definite). We present a method for calculating the metric projection of a point x 2 Rn onto the ellipsoid C (cf. [316, Sect. 3.4] and [228]). A different method was also presented in [128]. It follows from the definition of the metric projection that y D PC x if and only if y is a solution of the following convex minimization problem minimize f .y/ D 12 ky  xk2 subject to .y  z/> D.y  z/   y 2 Rn

(4.11)

(see Fig. 4.10). This problem has a unique solution, because the objective is strongly convex. The same also follows from Theorem 1.2.3. All assumptions of the Karush–Kuhn–Tucker theorem are satisfied. The Slater constraints qualification for problem (4.11) is satisfied, because int J.D; z; / ¤ ;. Therefore, y 2 Rn is a solution of this problem if and only if there exists  2 RC such that .y; / is a KKTpoint (see Theorems 1.3.6 and 1.3.7). Define the Lagrange function L W Rn R ! R by the equality L.y; / D

1 ky  xk2 C Œ.y  z/> D.y  z/  . 2

The Karush–Kuhn–Tucker system has the form y  x C 2D.y  z/ .y  z/> D.y  z/  Œ.y  z/> D.y  z/  

D0  0 D 0.

(4.12)

If .x  z/> D.x  z/  , then x 2 J.D; z; / and .y; / D .x; 0/ is the only KKT-point, i.e., PC x D x. If .x  z/> D.x  z/ > , then x … J.D; z; /, i.e., y ¤ x. It follows from the first and from the third equality in (4.12) that  > 0, consequently, .y  z/> D.y  z/   D 0;

(4.13)

4.1 Examples of Metric Projections

139

by the complementary condition (fourth equality in (4.12)). The first equality in the KKT-system gives y D .Id C 2D/1 .x C 2Dz/ (note that Id C 2D is positive definite, because D is positive definite). Setting it in equality (4.13), we obtain Œ.x C 2Dz/> .Id C 2D/1  z> DŒ.Id C 2D/1 .x C 2Dz/  z D . (4.14) By the existence and uniqueness of the metric projection PC x, equation (4.14) has a unique solution . Summarizing, we obtain  PC x D

x if .x  z/> D.x  z/   1 .Id C 2D/ .x C 2Dz/ if .x  z/> D.x  z/ > ;

where  > 0 is the unique solution of equality (4.14). If D D W D diag w for w 2 RnCC , z D 0 and  D 1, we have C D J.W; 0; 1/ D fy 2 Rn W y > Wy  1g and (4.14) obtains the form x > .Id C 2W /1 W .Id C 2W /1 x D 1.

(4.15)

Since 2

.Id C 2W /1

0 .1 C 2!1 /1 1 6 0 .1 C 2! 2/ D6 4 ::: ::: 0 0

3 ::: 0 7 ::: 0 7, 5 ::: ::: 1 : : : .1 C 2!n /

(4.15) can be written in the form n X

!j j2

j D1

.1 C 2!j /2

D1

(4.16)

(note that this equation is considered only in the case x … J.W; 0; 1/, i.e., P n 2 j D1 !j j > 1). As mentioned before, this equation has a unique solution  > 0, which can be computed, e.g., by the Newton method with the starting point 0 D 0. P !j  2 Note that the function h W RC ! R, h./ D nj D1 .1C2!j /2 is convex (h00 ./ > 0 j for all   0), h.0/ > 1 and lim!C1 h./ D 0. Therefore, the Newton method generates an increasing sequence k which converges to a unique solution  of (4.16). The details can be found in [316, Theorem 3.4-2].

140

4 Algorithmic Projection Operators

Fig. 4.11 Ice-cream cone and its polar

4.1.9 Metric Projection onto an Ice Cream Cone Let a 2 H a ¤ 0 and ˛ 2 Œ0; 2 . A subset C.a; ˛/ WD fz 2 H W ^.z; a/  ˛g [ f0g is called an ice cream cone with axis a and angle ˛. Without loss of generality we suppose that kak D 1. Then the ice cream cone has the form C.a; ˛/ D fz 2 H W ha; zi   kzkg, where  D cos ˛ 2 Œ0; 1. This subset is closed and convex as a sublevel set S.f; 0/ of a continuous convex function f W H ! R, f .x/ WD  kxk  ha; xi. It is clear that C.a; ˛/ is a cone. The polar cone to an ice cream cone is again an ice cream cone and has the form .C.a; ˛// D C.a;

  ˛/ 2

(4.17)

(see, e.g., [186, Sect. 3.1] or [187]) (Fig. 4.11). We will give a formula for PC.a;˛/ x, where x 2 H and ˛ 2 .0; 2 /. If x 2 C.a; ˛/, then, obviously, PC.a;˛/ x D x. Let x 2 C.a; 2  ˛/. Then it follows from equality (4.17) and from the characterization of the metric projection (see Theorem 1.2.4) that PC.a;˛/ x D 0. Now suppose that x 2 C.a; ˛/0 \ C.a; 2  ˛/0 . Let u WD a with  2 R, be such that ha; u  xi D sin ˛ ku  xk

(4.18)

4.1 Examples of Metric Projections

141

Fig. 4.12 Metric projection onto an ice cream cone

C ( a;

π 2

®) π 2

a

®

xy

u

®

y

a C (a; ®)

x

or, equivalently,

q

kxk2  ha; xi2 ku  xk

D cos ˛.

(4.19)

Obviously, x ¤ u, because ˛ > 0, and kxk2  ha; xi2  0, by the Cauchy–Schwarz inequality. Let y WD x C .u  x/ with  2 R, be such that hy; x  ui D 0 (see Fig. 4.12). A simple calculation shows that q  D ha; xi C

kxk2  ha; xi2 tan ˛

(4.20)

and D

hx; x  ui ku  xk

2

D

kxk2  ha; xi ka  xk2

.

(4.21)

Consequently, hy; ai D hx C .u  x/; ai D and

.kxk2  ha; xi2 / ka  xk2

(4.22)

q

kyk D kx C .u  x/k D

 kxk2  ha; xi2 ka  xk

.

(4.23)

By (4.19), (4.22) and (4.23), we obtain hy; ai D kyk cos ˛, i.e., y 2 C.a; ˛/. Furthermore, (4.17) and (4.18) yield x  y D .x  u/ 2 C  .a; ˛/, and we have hx  y; yi D hx  u; yi D 0, consequently, hx  y; z  yi D hx  y; zi  0

142

4 Algorithmic Projection Operators

for all z 2 C.a; ˛/. Now, the characterization of the metric projection yields that y D PC.a;˛/ x. Subsuming all cases, we obtain 8 if ha; xi  kxk cos ˛ <x PC.a;˛/ x D 0 if ha; xi   kxk sin ˛ : x C .a  x/ if  kxk sin ˛ < ha; xi < kxk cos ˛, where  and  are given by (4.20) and (4.21). Equivalent formulas can also be found in [17, Theorem 3.3.6] or in [316, Sect. 3.5].

4.2 Cutters Recall that an operator T W H ! H with a fixed point is a cutter if hT x  x; T x  zi  0 for all x 2 H and for all z 2 Fix T (see Definition 2.1.30).

4.2.1 Characterization of Cutters It follows from Lemma 2.1.36 that, for T being a cutter, Fix T is closed and convex, consequently, Fix T is an intersection of half-spaces, i.e., Fix T D T H .ai ; ˇi /, where ai 2 H, ai ¤ 0, ˇi 2 R, i 2 J . This fact is applied  i 2J in the theorem below which gives a necessary and sufficient condition for T to be a cutter. Denote V WD Linfai ; i 2 J g. Theorem 4.2.1. Let T W H ! H. For any y 2 H the following statements are equivalent: (i) T is a cutter. (ii) For all x 2 H and for all w 2 .V C y/ \ Fix T , it holds hT x  x; T x  wi  0

(4.24)

Tx x 2 V.

(4.25)

and Proof. Let z 2 Fix T , v? 2 V ? and ˛ 2 R. We prove that z C ˛v? 2 Fix T . Note that hai ; v? i D 0 and hai ; zi  ˇi , i 2 J , Therefore, we have hai ; z C ˛v? i D hai ; zi C ˛hai ; v? i  ˇi for all i 2 J , i.e., z C ˛v? 2

T i 2J

H .ai ; ˇi / D Fix T .

4.2 Cutters

143

Fig. 4.13 Cutter T with affine Fix T

u v Fix T

Tx x

(ii))(i) Let x; y 2 H and z 2 Fix T . Suppose that conditions (4.24) and (4.25) are satisfied for all w 2 .V C y/ \ Fix T . Let y; N zN 2 V and y ? ; z? 2 V ? be such ? ? that y D yN C y and z D zN C z . It follows from the first part of the proof that zN 2 Fix T and that zN C y ? 2 Fix T . It is clear that V C y D V C y ? , consequently, zN C y ? 2 V C y. By (4.24) and (4.25), we have hT x  x; T x  zi D hT x  x; T x  .Nz C y ? /i  hT x  x; z?  y ? i  0 and T is a cutter. (i))(ii) Suppose that T is a cutter. Let x 2 H and w 2 .V C y/ \ Fix T . Inequality (4.24) is obvious. Suppose that (4.25) does not hold. Then T x  x D vN C v? for some vN 2 V and v? 2 V ? , v? ¤ 0. It follows from the first part of the  2 proof that w C ˛v? 2 Fix T for any ˛ 2 R. For ˛ < hT x  x; T x  wi= v?  we obtain 0  hT x  x; T x  .w C ˛v? /i D hT x  x; T x  wi  ˛hNv C v? ; v? i  2 D hT x  x; T x  wi  ˛ v?  > 0, a contradiction which shows that T x  x 2 V .

t u

Corollary 4.2.2. T Let T W H ! H be a cutter. If Fix T is a polyhedral subset, i.e., Fix T D i 2I H .ai ; ˇi /, where ai 2 H, ai ¤ 0 and ˇi 2 R, i 2 I WD f1; 2; : : : ; mg, then T x 2 x C Linfai W i 2 I g.

4.2.2 Cutters with Subsets of Fixed Points Being Affine Subspaces In this section we apply Theorem 4.2.1 to an operator T W H ! H with the subset of fixed points being a nonempty and affine subspace (Fig. 4.13). Corollary 4.2.3. Let T W H ! H be an operator with Fix T being a nonempty closed and affine subspace. The operator T is a cutter if and only if hT x  x; T x  ui  0

(4.26)

144

4 Algorithmic Projection Operators

for all x 2 H and for some u 2 Fix T and hT x  x; v  ui D 0

(4.27)

for all x 2 H and for all v; u 2 Fix T . Proof. Since Fix T is a closed affine subspace, it has the form Fix T D W C u, where W is a closed linear subspace and u 2 Fix T . Denote V WD W ? . It is clear that .V C u/ \ .W C u/ D fug. The corollary follows now from Theorem 4.2.1. u t Corollary 4.2.3 can also be proved directly. 2nd Proof of Corollary 4.2.3. The sufficiency of the conditions and the necessity of (4.26) is obvious. Suppose that T is a cutter and that hT x  x; v  ui ¤ 0

(4.28)

for some x 2 H and for some v; u 2 Fix T . By the symmetry of (4.28) with respect to u and v, it suffices to consider only the case hT x  x; v  ui > 0. Let w D u C t.v  u/ for t 2 R. Of course, w 2 Fix T , because Fix T is an affine subspace. For hT x  x; T x  ui t< hT x  x; v  ui it holds hT x  x; T x  wi > 0. We have obtained a contradiction with the assumption that T is a cutter, which yields the necessity of (4.27). t u

4.2.3 Subgradient Projection Definition 4.2.4. Let f W H ! R be a continuous convex function. Let gf .x/ 2 @f .x/ be a subgradient of f at x, x 2 H (note that f is subdifferentiable, i e., for any x 2 H, there exists gf .x//. Let ˛ 2 R. The operator Pf;˛ W H ! H defined by ( Pf;˛ x WD

x x

.f .x/˛/C

kgf .x/k

2

gf .x/ if gf .x/ ¤ 0 if gf .x/ D 0

(4.29)

is called a subgradient projection relative to f by a level ˛. The operator Pf WD Pf;0 is called a subgradient projection relative to f or, shortly, a subgradient projection. If gf .x/ ¤ 0, then

Pf;˛ x D PS.fNx ;˛/ .x/,

(4.30)

4.2 Cutters

145

Fig. 4.14 Subgradient projection

f

x Pf x

Fig. 4.15 Pf;˛ x D PS.fNx ;˛/ x S (f,α) x

Pf ,αx

S(f x, α)

where fNx D hgf .x/;   xi C f .x/ denotes a linearization of f at x. Indeed, for gf .x/ ¤ 0 the sublevel set S.fNx ; ˛/ is a half-space S.fNx ; ˛/ D fy 2 H W hgf .x/; y  xi C f .x/  ˛g and, by (4.7), .f .x/  ˛/C PS.fNx ;˛/ x D x    gf .x/ D Pf;˛ x. gf .x/2 Equality (4.30) is illustrated in Figs. 4.14 and 4.15. If ˛  infx2H f .x/ it holds that Argminx2H f .x/  S.f; ˛/ and we can write ( Pf;˛ x D

x x

f .x/˛

kgf .x/k

2

gf .x/ if x … S.f; ˛/ if x 2 S.f; ˛/,

(4.31)

because for x … S.f; ˛/ we have .f .x/  ˛/C D f .x/  ˛ and gf .x/ ¤ 0 (see Theorem 1.3.2), and for x 2 S.f; ˛/ we have .f .x/  ˛/C D 0. Obviously, Pf;˛ x depends on the choice of gf .x/ 2 @f .x/, x 2 H. Nevertheless, the properties of the subgradient projection presented below hold for any choice of gf .x/ 2 @f .x/, x 2 H. Note that Pf;˛ is well defined even if S.f; ˛/ D ;, i.e., if ˛ < infx2H f .x/ or if ˛ D infx2H f .x/ and the function f does not attain its minimum. Lemma 4.2.5. Let Pf;˛ W H ! H be a subgradient projection relative to a convex subdifferentiable function f by a level ˛ 2 R. If S.f; ˛/ ¤ ;, then

146

4 Algorithmic Projection Operators

Fix Pf;˛ D S.f; ˛/. Proof. The inclusion S.f; ˛/  Fix Pf;˛ is obvious. Suppose that x … S.f; ˛/ ¤ ;. Then f .x/ > ˛  infx2H f .x/, consequently, gf .x/ ¤ 0 (see Theorem 1.3.2) and f .x/  ˛  gf .x/ ¤ 0,  gf .x/2 i.e., x … Fix Pf;˛ . Therefore, Fix Pf;˛ D S.f; ˛/.

t u

Let ˛ 2 R. It is clear that S.f; ˛/ D S.f  ˛; 0/. Furthermore, f is a convex and subdifferentiable function if and only if f  ˛ is convex and subdifferentiable. Moreover, for any x 2 H we have @.f  ˛/.x/ D @f .x/. Therefore, we can restrict our further analysis of Pf;˛ to the case ˛ D 0. Corollary 4.2.6. Let f be convex and subdifferentiable and S.f; 0/ ¤ ;. A subgradient projection Pf is a cutter. Proof. By Lemma 4.2.5, we have Fix Pf D S.f; 0/. Let x … S.f; 0/ and z 2 S.f; 0/, i.e., f .z/  0 < f .x/. By the definition of a subgradient of a convex function, we have  2 hPf x  x; Pf x  zi D Pf x  x  C hPf x  x; x  zi !2 f .x/ f .x/   D   hgf .x/; x  zi gf .x/ gf .x/2 

f .x/   gf .x/

!2

f .x/   .f .x/  f .z// gf .x/2

0 which completes the proof.

t u

Theorem 4.2.7. Let f W H ! R be a convex function which is globally Lipschitz continuous on bounded subsets (this holds if, e.g., H D Rn ). Then the operator Pf  Id is demi-closed at 0.   Proof. Let x k * z 2 H and Pf x k  x k  ! 0. Then we have   .f .x k //C  D Pf x k  x k  ! 0.  gf .x k /

(4.32)

Note that fx k g1 kD0 is bounded as a weakly convergent sequence. By the global Lipschitz continuity of f on bounded subsets, there exists a constant > 0 such that   f 0 .x k ; gf .x k //  gf .x k /

4.3 Alternating Projection

147

(see Theorem 1.1.50). By Theorem 1.1.58,     gf .x k /2  sup hg; gf .x k /i D f 0 .x k ; gf .x k //  gf .x k / , g2@f .x k /

  i.e., gf .x k /  and the sequence fgf .x k /g1 kD0 is bounded. Now (4.32) yields k .f .x //C ! 0. Recall that a continuous and convex function is weakly lower semicontinuous (see Theorem 1.1.51). Therefore, f .z/  lim inf f .x k /  lim inf.f .x k //C D 0 k

k

and Pf z D z.

t u

4.3 Alternating Projection 4.3.1 Basic Properties Definition 4.3.1. Let A; B  H be a nonempty closed convex subset. We call an operator T WD PA PB W H ! A an alternating projection (Fig. 4.16). The alternating projection was introduced by John von Neumann [271], who studied the convergence of sequences generated by this operator for closed convex subspaces A; B  H. The alternating projection or its modifications were studied by Aronszajn [11], Gurin et al. [196], Deutsch [138, 139, 141], Bauschke and Borwein [20, 21], Combettes [117], Bauschke et al. [30], Hundal [213], Kopeck´a and Reich [235, 236], Bauschke et al. [25], Scolnik et al. [306], Cegielski and Dylewski [74] and by Cegielski and Suchocka [75, 76]. First we give properties of an alternating projection T D PA PB in the case A \ B ¤ ;. It follows from Theorem 2.1.26 (ii) that in this case we have Fix T D A\B, because the metric projection is strictly quasi-nonexpansive (see Corollary 2.2.24). Lemma 4.3.2. Let A; B  H be nonempty closed convex subsets and A \ B ¤ ;. Then for all x 2 A and z 2 A \ B the following inequalities hold hz  T x; x  T xi  hPB x  T x; x  PB xi   kT x  PB xk2 , where T WD PA PB is an alternating projection. Consequently, the operator T jA is a cutter and its projected relaxation PA T is asymptotically regular for all  2 .0; 2/. Proof. Let x 2 A and z 2 Fix T D A \ B. It follows from the characterization of the metric projections PA .PB x/ and PB x and from the equivalence (a),(b) in Lemma 1.2.5 that hz  T x; x  T xi D hz  T x; x  PB xi C hz  T x; PB x  T xi  hz  T x; x  PB xi

148

4 Algorithmic Projection Operators

Fig. 4.16 Alternating projection

PA PB x

x

PB x

A

B

D hz  PB x; x  PB xi C hPB x  T x; x  PB xi  hPB x  T x; x  PB xi   kT x  PB xk2  0, i.e., the operator T jA is a cutter. The asymptotic regularity of of the projected relaxation PA T for  2 .0; 2/ follows now from Corollary 3.4.4. t u If A \ B ¤ ;, then Lemma 4.3.2 and Theorem 2.1.39 imply the strong quasi nonexpansivity of the relaxation T of the alternating projection T WD PA PB for any  2 .0; 2/. Note that Lemma 4.3.2 is not true without the assumption A \ B ¤ ;. In the case A \ B D ; the asymptotic regularity of the projected relaxation PA T holds only for a narrower range of the relaxation parameter . Corollary 4.3.3. Let A; B  H be nonempty closed convex subsets. Then the alternating projection T WD PA PB is 43 -relaxed firmly nonexpansive. Furthermore, for any  2 Œ0; 34 , the relaxation T of T is firmly nonexpansive. Consequently, for any  2 .0; 32 /, the operator T is relaxed firmly nonexpansive and T is asymptotically regular whenever Fix T ¤ ;. Proof. By Corollary 2.2.39, the operator T is 43 -relaxed firmly nonexpansive as a composition of FNE operators PA and PB . Moreover, by Corollary 2.2.19, its relaxation T is firmly nonexpansive for any  2 Œ0; 34 . Suppose that Fix T ¤ ;. Then Corollary 3.4.6 yields the asymptotic regularity of the operator T for any  2 .0; 32 /. t u If we take  D 1 in Corollary 4.3.3, then we obtain that the alternating projection T WD PA PB is relaxed firmly nonexpansive and asymptotically regular, whenever Fix T ¤ ;.

4.3.2 Fixed Points of the Alternating Projection Recall that the distance function d.; C / W H ! R, where C  H, is defined by d.x; C / WD infz2C kx  zk. Denote d.A; B/ WD infx2A;y2B kx  yk, where A; B  H.

4.3 Alternating Projection

149

If A \ B ¤ ;, then Fix PA PB D A \ B (see Theorem 2.1.26 (ii)). In a general case, the alternating projection does not need to have fixed points (e.g., for A WD f.x; y/ 2 R2 W y D 0g and B WD f.x; y/ 2 R2 W y  e x g we have Fix PA PB D ;). The theorem below gives relationships between fixed points of operators PA PB and PB PA and minimizers of the distance functions d.; B/ W A ! R and d.; A/ W B ! R. The theorem presents a part of the results of [21, Lemma 2.2]. Theorem 4.3.4. Let A; B  H be nonempty closed convex subsets, x  2 A and y  2 B. The following conditions are equivalent: (i) (ii) (iii) (iv) (v)

x  2 Fix PA PB and y  D PB x  , y  2 Fix PB PA and x  D PA y  , kx   y  k D d.x  ; B/  d.x; B/ for all x 2 A, kx   y  k D d.y  ; A/  d.y; A/ for all y 2 B. kx   y  k D d.A; B/.

Proof. The inequality in condition (iii) denotes that the function f WD 12 d 2 .; B/ W A ! R attains its minimum at x  2 A. The function f is convex (see Corollary 2.2.29) and differentiable and Df D Id  PB (see Lemma 2.2.27). Now, it follows from the necessary and sufficient optimality condition for the convex differentiable minimization (see Theorem 1.3.4) and from Lemma 1.2.9 that x

2 Argmin f .x/ x2A

” PB x   x  D Df .x  / 2 NA .x  / ” x  D PA PB x 

(4.33)



” x 2 Fix PA PB . (i),(ii) Let PA PB x  D x  and y  D PB x  . Then PA y  D PA PB x  D x  and, consequently, PB PA y  D PB x  D y  . The converse implication can be shown in a similar way. (i),(iii) The equivalence of the first part of (i) and of the inequality in (iii) was already proved in (4.33). If y  D PB x  , then the equality in (iii) follows from the definition of the metric projection PB . The converse implication follows from the uniqueness of the metric projection. In a similar way, one can show the equivalence (ii),(iv). (i),(v) The function h W A  B ! R defined by h.x; y/ WD 12 kx  yk2 is convex as a composition of a linear function .x; y/ 7! x  y and of a convex function u 7! 12 kuk2 . Moreover, h is differentiable and Dh.x; y/ D .x  y; y  x/ for all x 2 A and y 2 B. We leave it to the reader to check that NAB .x; y/ D NA .x/  NB .y/. Now, using similar arguments as in the first part of the proof, we

150

4 Algorithmic Projection Operators

Fig. 4.17 Fix.PA PB /, Fix.PB PA / and minimizers of d.; A/ and of d.; B/

x∗

A

Fix( PB PA )

Fix( PA PB )

y∗

B

obtain .x  ; y  / 2 Argmin h.x; y/ x2A;y2B

” .y   x  ; x   y  / D Dh.x  ; y  / 2 NAB .x  ; y  / ” y   x  2 NA .x  / and x   y  2 NB .y  / ” x  D PA y  and y  D PB x  ” x  D PA PB x  and y  D PB x  , i.e., conditions (i) and (v) are equivalent.

t u

Theorem 4.3.4 is illustrated in Fig. 4.17. Part (ii) of the following corollary was proved by Cheney and Goldstein (see [112, Theorem 2]). Corollary 4.3.5. Let A; B  H be nonempty closed convex subsets. Then: (i) (ii) (iii) (iv) (v)

Fix PA PB ¤ ; ” Fix PB PA ¤ ;, Fix PA PB D Argminx2A d.x; B/, Fix PB PA D Argminy2B d.y; B/, Fix PA PB  Fix PB PA D Argmin.x;y/2AB kx  yk, If A \ B ¤ ;, then Fix PA PB D A \ B.

The corollary below gives a sufficient condition for the nonemptiness of the subset Fix PA PB . Corollary 4.3.6. Let A; B  H be nonempty closed convex subsets. If at least one of the subsets A and B is bounded, then Fix PA PB ¤ ;. Proof. Suppose that A is bounded. It is clear that PA PB W A ! A is nonexpansive. It follows from the Browder–G¨ohde–Kirk theorem that Fix PA PB ¤ ;. Suppose now that B is bounded. Then, Fix PB PA ¤ ; and the corollary follows from Corollary 4.3.5. t u If we suppose that H D Rn and A; B  Rn are polytopes, then Corollary 4.3.6 holds without any assumption on the boundedness of A or B (see [112, Theorem 5]).

4.3 Alternating Projection

151

4.3.3 Alternating Projection for a Closed Affine Subspace The alternating projection PA PB does not need not to be firmly nonexpansive even if A an B are closed subspaces of H (see Exercise 2.5.9). Nevertheless, if we suppose that A is a closed affine subspace and if we restrict the operator PA PB to the subset A, then PA PB is firmly nonexpansive. This fact is formulated in the following theorem which is an extension of an important result of Combettes [117, Proposition 3]. Theorem 4.3.7. Let A  H be a closed affine subspace and B  H be a closed convex subset. Then, for all x; y 2 A, hT x  T y; x  yi  kT x  T yk2 C k.T x  PB x/  .T y  PB y/k2

(4.34)

and hT x  T y; x  yi  kT x  T yk2 C .kT x  PB xk  kT y  PB yk/2 .

(4.35)

Consequently, the alternating projection T WD PA PB W A ! A, is firmly nonexpansive. Proof. Let a 2 A. Applying the property hPA u  PA v; wi D hu  v; PAa wi for u; v; w 2 H (see Theorem 2.2.33 (iii)), the fact x  y 2 A  a for x; y 2 A, the firm nonexpansivity of PB and Corollary 2.2.24, we obtain hT x  T y; x  yi D hPA PB x  PA PB y; x  yi D hPB x  PB y; PAa .x  y/i D hPB x  PB y; x  yi  kPB x  PB yk2  kPA PB x  PA PB yk2 C k.PA PB x  PB x/  .PA PB y  PB y/k2 D kT x  T yk2 C k.T x  PB x/  .T y  PB y/k2  kT x  T yk2 C .kT x  PB xk  kT y  PB yk/2  kT x  T yk2 for all x; y 2 A. Consequently, T is firmly nonexpansive.

t u

152

4 Algorithmic Projection Operators

Corollary 4.3.8. Let A  H be a closed affine subspace and B  H a closed convex subset. Further, let T W A ! A be a relaxation of the alternating projection T WD PA PB , where  2 .0; 2/. Then kT x  T yk2  kx  yk2 

2 k.T x  x/  .T y  y/k2 

(4.36)

for arbitrary x; y 2 A. If, furthermore, Fix T ¤ ;, then, for any x 2 A and z 2 Fix T it holds kT x  zk2  kx  zk2  i.e., the operator T is

2 kT x  xk2 , 

2  -strongly quasi-nonexpansive and

asymptotically regular.

Proof. By Theorem 4.3.7, the operator T WD PA PB is firmly nonexpansive. Therefore, the corollary follows from Corollary 2.2.15 and Theorem 3.4.3. t u

4.3.4 Generalized Relaxation of the Alternating Projection Let A; B  H be nonempty closed and convex and T WD PA PB be the alternating projection with a fixed point. Consider a generalized relaxation T ; W A ! H of T , given by the formula T ; .x/ D x C  .x/.T x  x/,

(4.37)

where the relaxation parameter  2 Œ0; 2 and W A ! .0; C1/ is a step size function. It is clear that if .x/ D 1 for all x 2 A, then T ; coincides with the classical relaxation T of the alternating projection T WD PA PB . Corollaries 2.4.3 and 2.4.5 suggest that a step size function with values larger than 1, for which the operator T is a cutter, leads to an acceleration (at least local) of the corresponding iterative procedures of the form x kC1 D PA T ;k .x k /. Therefore, the ability to construct such step sizes is of big interest. In the remaining part of this section we will present various step sizes for which T is a cutter. Recall that Fix T ; D Fix T for all  > 0 (see Remark 2.4.2 (d)). Denote ı WD d.A; B/ D and

inf

x2A;y2B

kx  yk

N ı.x/ WD kT x  PB xk ,

4.3 Alternating Projection

153

where x 2 A. If A \ B ¤ ;, then, of course, ı D 0. Suppose that we know an upper Q N approximation ı.x/ 2 Œı; ı.x/ of ı, for any x 2 A. Recall that we still suppose that Fix T ¤ ;. Lemma 4.3.9. Let x 2 A be such that T x … Fix T . Then the vectors x  PB x and T x  PB x are linearly independent. Proof. The assumption yields that x  PB x and T x  PB x are nonzero vectors. Suppose that x  PB x and T x  PB x are linearly dependent, i.e., T x  PB x D .x  PB x/

(4.38)

for some  . By the characterization of the metric projection PA .PB x/, we have kT x  PB xk2 D  hx  PB x; T x  PB xi   kT x  PB xk2 > 0.

(4.39)

Therefore,  > 0. By Lemma 1.2.9, we have x  PB x 2 NB .PB x/, consequently, .x  PB x/ 2 NB .PB x/ and, again by Lemma 1.2.9, PB .PB x C .x  PB x// D PB x:

(4.40)

Now, by (4.38) and (4.40), we obtain T 2 x D PA PB T x D PA PB .PB x C .x  PB x// D PA PB PB x D PA PB x D T x, a contradiction with the assumption T x … Fix T . Therefore, the vectors x  PB x and T x  PB x are linearly independent. t u Define the step size function W A ! .0; C1/ by the formula 8 2 Q ˆ kPB x  xk C hPB x  x; T x  xi < kT x  PB xk  ı.x/ if x … Fix T;

.x/ WD kT x  xk2 ˆ :1 if x 2 Fix T; (4.41) Q where ı.x/ is a fixed element of the segment Œı; kT x  PB xk (see Fig. 4.18). This step size was proposed in [76, equality (13)]. Lemma 4.3.10. Let x 2 A and the step size .x/ be defined by (4.41). Then

.x/ 

1 2

(4.42)

and the inequality is strict if T x … Fix T . Proof. If x 2 Fix T , then .x/ D 1. Let now x … Fix T . Denote a WD PB x  x, Q b WD T x  x, c WD PB x  T x (see Fig. 4.18) and ıQ WD ı.x/.

154

4 Algorithmic Projection Operators

Fig. 4.18 Step size .x/ given by (4.41) with Q ı.x/ WD kT x  PB xk

x

Tσ x

A

Tx δ( x )

B PB x

Since b ¤ 0, b D a  c and ıQ  kck, the Cauchy–Schwarz inequality yields

.x/ D  D D D

kck2  ıQ kak C ha; bi kbk2 kck2  kak  kck C ha; bi kbk2 kck2  kak  kck C ha; a  ci ka  ck2 kak2 C kck2  kak  kck  ha; ci kak2 C kck2  2ha; ci .kak  kck/2 C kak  kck  ha; ci . .kak  kck/2 C 2.kak  kck  ha; ci/

˛Cˇ Let ˇ 2 R. Note that the function f W .2ˇ; C1/ ! R, f .˛/ WD ˛C2ˇ is 2 increasing. Therefore, if we take ˛ WD .kak  kck/ and ˇ WD kak  kck  ha; ci in the above inequalities, we obtain ˛  0  2ˇ and

.x/  f .˛/  f .0/ D

1 , 2

i.e., .x/  12 . If T x … Fix T , then a and c are linearly independent by Lemma 4.3.9. In this case, ˇ D kak  kck  ha; ci > 0 and .x/ > 12 . t u Inequality (4.42) can be strengthened. One can prove that

.x/ 

1 1  ; 1 C cos ˛.x/ 2

where ˛.x/ denotes the angle between the nonzero vectors x  PB x and T x  PB x (for details see [76, Lemmas 3 and 4]).

4.3 Alternating Projection

155

Fig. 4.19 Solution of system (4.44)–(4.45) with Q ı.x/ WD ı

x

Tx

A y

x∗ δ y∗

Tσ x B

δ PB x

Let x 2 A be such that T x … Fix T . One can prove that the step size .x/ given by (4.41) is characterized by the equality hx C .x/.T x  x/  y; T x  xi D 0,

(4.43)

where y 2 aff.x; PB x; T x/ is a unique solution of the system Q hy  PB x; x  PB xi D ı.x/ kPB x  xk hy  PB x; T x  PB xi D kT x  PB xk2

(4.44) (4.45)

Q and ı.x/ 2 Œı; kT x  PB xk (see Fig. 4.19). By Lemma 4.3.9, such a solution is defined uniquely. For details, see [76, Lemma 6]. Theorem 4.3.11. Let A; B  H be nonempty closed convex subsets, T WD PA PB , Fix T ¤ ; and the step size function W A ! .0; C1/ be defined by (4.41). Then, for any x … Fix T and z 2 Fix T ,

.x/ 

hz  x; T x  xi kT x  xk2

(4.46)

holds and the operator T W A ! H is a cutter. Consequently, for any  2 .0; 2/, the operator T ; is 2 -strongly quasi-nonexpansive and asymptotically regular.  Proof. Let z 2 Fix T , x … Fix T and  2 .0; 2/. It is clear that Fix T ; D Fix T D Fix T . Denote w WD PB z. Note that kz  wk D ı (see the implication (i))(v) in Theorem 4.3.4). By the characterization of the metric projection PB x and by the Cauchy–Schwarz inequality, we have

156

4 Algorithmic Projection Operators

hz  PB x; PB x  xi D hz  w; PB x  xi C hw  PB x; PB x  xi  hz  w; PB x  xi   kz  wk  kPB x  xk D ı kPB x  xk . Therefore, if we apply again the characterization of the metric projection PA .PB x/, we obtain hz  x; T x  xi D hz  PB x; T x  xi C hPB x  x; T x  xi D hz  PB x; T x  PB xi C hz  PB x; PB x  xi ChPB x  x; T x  xi  kT x  PB xk2  ı kPB x  xk C hPB x  x; T x  xi. Q Now, (4.46) follows from the inequality ı  ı.x/ and from the definition of the step size given by (4.41). We see that the step size .x/ satisfies the assumptions of Corollary 2.4.5. Therefore, the operator T is a cutter and the operator T ; is 2  -strongly quasi-nonexpansive. The asymptotic regularity of T ; follows from Theorem 3.4.3. t u Q If A \ B ¤ ;, then, of course, ı D 0. If we take ı.x/ WD ı D 0 in (4.41), then we obtain 8 2 < kT x  PB xk C hPB x  x; T x  xi if x … A \ B (4.47)

.x/ D kT x  xk2 : 1 if x 2 A \ B. We have the following result (cf. [76, Lemma 10]). Lemma 4.3.12. Let A \ B ¤ ; and x 2 AnB. Then kT x  PB xk2 C hPB x  x; T x  xi kT x  xk

2



kPB x  xk2 , hPB x  x; T x  xi

(4.48)

Furthermore, the equality holds in (4.48) if A is a closed affine subspace. Proof. Denote a WD PB x  x, b WD T x  x and c WD PB x  T x. We have a D b C c and b ¤ 0, because x … A \ B D Fix T . By the characterization of the metric projection PA .PB x/, we have ha; bi D hc; bi C kbk2  kbk2 > 0,

(4.49)

i.e., both sides of inequality (4.48) are well defined. Inequality (4.48) can be written in the form kb  ak2 C ha; bi kak2  ha; bi kbk2

4.3 Alternating Projection

157

Fig. 4.20 Step sizes .x/ given by (4.47) and by (4.52)

x

Tx A Tσ1 x PB x B

y

Tσ 2 x

which is equivalent to .kak2  ha; bi/.ha; bi  kbk2 /  0.

(4.50)

It follows from the characterization of the metric projection PA .PB x/ that hb; a  bi D hb; ci  0.

(4.51)

Therefore, the inequalities in (4.49), the Cauchy–Schwarz inequality and the nonexpansivity of the metric projection PA yield 0 < kbk2  ha; bi  kak  kbk  kak2 , consequently, (4.50) is true. Suppose now that A is a closed affine subspace. Then hc; bi D 0 (see Theorem 2.2.33 (i)), consequently, ha; bi D kbk2 and the equality in (4.50) holds. t u Suppose that A \ B ¤ ;. Define the step size function W A ! .0; C1/ by the formula 8 kPB x  xk2 < if x 2 AnB ;

.x/ D hPB x  x; T x  xi (4.52) : 1 if x 2 A \ B, where T WD PA PB is the alternating projection. The step size defined by (4.52) was proposed by Gurin et al. in [196, Sect. 3] and was applied in an accelerated alternating projection method of the form x kC1 D PA T x k . Note that Lemma 4.3.12 says that in the case A \ B ¤ ;, the step size given by (4.47) is equal to at least the step size given by (4.52). Therefore, the estimation (2.69) for T WD PA PB with the step size .x/ given by (4.47) is better than the one with the step size .x/ given by (4.52) (see Fig. 4.20). The following lemma shows that both step sizes lead to generalized relaxations T which are extrapolations of T . Lemma 4.3.13. Let A \ B ¤ ; and the step size function W A ! .0; C1/ be defined by (4.52). Then .x/  1 for all x 2 A and the equality holds if and only if PB x 2 A.

158

4 Algorithmic Projection Operators

Proof. Let x 2 A. If PB x 2 A, then .x/ D 1. Let now PB x … A. Then, of course, x … B. Similarly as in the proof of Lemma 4.3.12, one can prove that hPB x  x; T x  xi > 0, i.e., the step size .x/ is well defined. Furthermore, the characterization of the metric projection PA .PB x/ yields hPB x  x; T x  xi D kPB x  xk2 C hPB x  x; T x  PB xi  kPB x  xk2  kT x  PB xk2 < kPB x  xk2 . Therefore, .x/  1 and the equality holds if and only if PB x 2 A.

t u

Corollary 4.3.14. Let A; B  H be nonempty closed convex subsets, A \ B ¤ ; and the step size function W A ! .0; C1/ be defined by (4.52). Then the operator T W A ! H is a cutter. Consequently, for any  2 .0; 2/, the operator T ; is 2 -strongly quasi-nonexpansive.  Proof. Let x 2 A and  2 .0; 2/. It follows from Lemma 4.3.12 that .x/ is not greater than the step size given by (4.47). Theorem 4.3.11 yields now that .x/ satisfies conditions of Corollary 2.4.5. Therefore, T is a cutter and T ; is 2  strongly quasi-nonexpansive. t u Remark 4.3.15. Let x 2 AnB. Note that the step size .x/ defined by (4.52) is the unique solution of the equality hx C .x/.T x  x/  PB x; PB x  xi D 0 (see Fig. 4.20). Suppose that A is a closed affine subspace and that A \ B ¤ ;. Then Theorem 2.2.33 (i) yields hPB x  x; T x  xi D kT x  xk2 C hPB x  T x; T x  xi D kT x  xk2 , where x 2 A and T D PA PB . The step size .x/ defined by (4.52) can be now written in the form 8 2 ˆ < kPB x  xk if x 2 AnB;

.x/ D (4.53) kT x  xk2 ˆ :1 if x 2 A \ B, (see Fig. 4.21). This step size was proposed by Bauschke et al. in [25, Corollary 4.11] and was applied in an accelerated alternating projection method of the form x kC1 D T x k . Now suppose that A and B are closed subspaces of H. In this case, the metric projections PA , PB are, actually, orthogonal projections (see Theorem 2.2.30 (i)) and

4.3 Alternating Projection Fig. 4.21 Step size .x/ given by (4.53)

159 Tσ x

Tx

x

A

PB x

Fig. 4.22 Step size .x/ given by (4.54)

x

B

Tσ x

Tx

A

B

PB x

kPB x  xk2 D hPB x; PB x  xi C hx; x  PB xi D hx; x  PB xi D hx; x  T xi C hx; T x  PB xi D hx; x  T xi, where x 2 A and T WD PA PB . The step size .x/ defined by (4.53) can now be written in the form 8 < hx; x  T xi if x 2 AnB (4.54)

.x/ D kT x  xk2 : 1 if x 2 A \ B, (see Fig. 4.22). This step size was applied in [30, equality 3.1.2] in order to accelerate globally the alternating projection method (see [30, Theorem 3.23]). Now we consider the case in which A is a closed affine subspace and B is a closed convex subset. We will propose a step size function W A ! .0; C1/ which leads to an extrapolation T of T and we will show that the generalized relaxation T ; which employs this step size function is strongly quasi-nonexpansive. Define the step size function W A ! .0; C1/ by the formula 8 2 Q ˆ < 1 C .kT x  PB xk  ı.x// if x … Fix T (4.55)

.x/ WD kT x  xk2 ˆ :1 if x 2 Fix T , Q where ı.x/ 2 Œı; kT x  PB xk is an upper approximation of ı WD d.A; B/ D infx2A;y2B kx  yk. A geometrical interpretation of the steps size .x/ is presented kT xxk Q in Fig. 4.23, where .x/  ı.x/ < kT x  PB xk, tan ˛ D Q . We have

.x/ D 1 C cot2 ˛ D

1 sin2 ˛

kT xPB xkı.x/ kT xxk . sin2 ˛

and kT x  xk D .x/ kT x  xk D

160

4 Algorithmic Projection Operators

Fig. 4.23 Step size .x/ given by (4.55)

x

Tx

®

Tσ x

x∗

A

PB x ∗

®

δ B

δ(x)

PB x

Theorem 4.3.16. Let A  H be a closed affine subspace, B  H be a nonempty closed convex subset and the step size function W A ! .0; C1/ be defined by (4.55). Then the operator T W A ! A is a cutter. Consequently, for any  2 .0; 2/, the operator T ; is strongly quasi-nonexpansive and asymptotically regular. Proof. Let z 2 Fix T , x … Fix T and  2 .0; 2/. Note that Fix T ; D Fix T D Fix T . It is clear that .x/  1. We prove that hz  x; T x  xi  .x/ kT x  xk2 The inequality is obvious for x 2 Fix T . Let now x … Fix T . Since ı D kT z  PB zk (see equivalence (i),(v) in Theorem 4.3.4), inequality (4.35) yields hz  x; T x  xi D kT x  xk2 C hz  T x; T x  xi D kT x  xk2 C hT z  T x; z  xi  kT z  T xk2  kT x  xk2 C .kT x  PB xk  kT z  PB zk/2 D .1 C  .1 C

.kT x  PB xk  ı/2 kT x  xk2

/ kT x  xk2

2 Q .kT x  PB xk  ı.x//

kT x  xk2

/ kT x  xk2

D .x/ kT x  xk2 . We see that the step size .x/ satisfies the assumptions of Theorem 2.4.5. Therefore, the operator T is a cutter and the operator T ; is 2  -strongly quasinonexpansive. The asymptotic regularity of T ; follows now from Theorem 3.4.3. t u

4.3.5 Averaged Alternating Reflection Let A; B  H be closed convex subsets. An operator T W H ! H defined by T WD

1 .RA RB C Id/, 2

(4.56)

4.3 Alternating Projection

161

Fig. 4.24 Averaged alternating reflection

RB x

B

x Tx A

RA R B x

where RA WD 2PA  Id and RB WD 2PB  Id are reflection operators onto A and B, respectively, is called an averaged alternating reflection (AAR) (see Fig. 4.24). The properties of AAR were studied in [149, 246], [157, Sect. 4], [26, Sect. 5.D], [27, Sect. 3], [28, Sect. 3] and in [29]. AAR is a special case of Elser’s difference map (for details, see [161]). It is clear that a -relaxation of T has the form T D .1 

  /Id C RA RB 2 2

More general results than the following one can be found in [28, Sect. 3]. Corollary 4.3.17. Let A; B  H be closed convex, U WD RA RB W H ! H and T WD 12 .U C Id/ be the averaged alternating reflection. Then (i) A \ B  Fix T , (ii) If int.A \ B/ ¤ ;, then A \ B D Fix T , (iii) If A\B ¤ ; and x 2 Fix T , then PB x D PA RB x, consequently, PB .Fix T / D A \ B, (iv) T is firmly nonexpansive, consequently, T is nonexpansive and 2  -strongly quasi-nonexpansive for any  2 .0; 2/. Proof. (i) The inclusion is obvious (see Remark 2.1.1). (ii) Let C WD int.A \ B/ ¤ ;. It is clear that C D int A \ int B D int Fix RA \ int Fix RB  Fix RA \ Fix RB . By Proposition 2.1.41, RA and RB are int C -strictly quasi-nonexpansive. Now Theorem 2.1.26 (ii) and the facts that Fix RA D A and Fix RB D B yield Fix T D Fix U D Fix RA \ Fix RB D A \ B. (iii) Let x 2 Fix T (D Fix U ), b WD PB x, y WD RB x D 2b  x and a D PA y. It is clear that y D b C 12 .y  x/ and x D RA y D a C 12 .x  y/. Consequently, b D a, i.e., PB x D PA RB x. Therefore, PB x 2 A \ B, i.e., PB .Fix T / D PB .Fix U /  A \ B. The converse inclusion follows immediately from (i) and from the fact that PB .A \ B/ D A \ B. (iv) The metric projections PA and PB are firmly nonexpansive (see Theorem 2.2.21 (iii)). By the implication (i))(ii) in Theorem 2.2.10, the reflections RA and RB are nonexpansive. Therefore, U WD RA RB is nonexpansive. Now the implication (iii))(i) in Theorem 2.2.10 yields the

162

4 Algorithmic Projection Operators

firm nonexpansivity of T WD 12 .U C Id/. Let  2 .0; 2/. Applying again the implication (i))(ii) in Theorem 2.2.10, we obtain that T is nonexpansive. The 2 t u  -strong quasi nonexpansivity of T follows from Corollary 2.2.9.

4.4 Simultaneous Projection Definition 4.4.1. Let Ci  H be nonempty closed convex subsets, i 2 I WD f1; 2; : : : ; mg, and w D .!1 ; : : : ; !m / 2 m be a vector of weights. The operator X !i PCi T WD i 2I

is called a simultaneous projection. The operator X

!i PCi ;i ,

i 2I

where PCi ;i WD Id C i .PCi  Id/, i 2 Œ0; 2, i 2 I , is called a simultaneous relaxed projection (see Fig. 4.25). The simultaneous projection was introduced by Gianfranco Cimmino [116] and was investigated by many authors, e.g., by Auslender [12], Censor and Elfving [87], Reich [295], Pierra [284], De Pierro and Iusem [134–137,215], Butnariu and Censor [53], Combettes [117,118,120], Bauschke [17], Bauschke and Borwein [22], Censor and Zenios [108] and by Matouˇskov´a and Reich [258]. OnePcan also consider a generalization of the simultaneous projection of the form T WD i 2I !i PCi , where w W H ! m is a weight function (see Sect. 4.8). In this section, however, we restrict ourselves to the classical simultaneous projection with T constant weights. Denote C WD i 2I C P i . The metric projection PCi is a special case of a simultaneous projection T WD i 2I !i PCi , where w D ei , i 2 I . Corollary 4.4.2. LetP Ci  H be nonempty closed convex subsets, i 2 I , C ¤ ;, w 2 ri m and T WD i 2I !i PCi . Then Fix T D C . T Proof. T By the nonexpansivity of the metric projection and by the fact i 2I Fix PCi D i 2I Ci D C ¤ ;, the claim follows from Theorem 2.1.14. t u P P Remark 4.4.3. Let  WD i 2I !i i . If  D 0, then it is clear that i 2I !i PCi ;i D Id. Now let  > 0 and i D !ii , where w D .!1 ; !2 ; : : : ; !m / 2 ri m and i 2 Œ0; 2, i 2 I . Then X X !i PCi ;i D !i .Id C i .PCi  Id// i 2I

i 2I

D Id C

X i 2I

!i i .PCi  Id/

4.4 Simultaneous Projection

163

Fig. 4.25 Simultaneous projection and simultaneous relaxed projection

PC1 x x

C1 Tx

PC 2 x C2

D Id C 

X !i i i 2I

D Id C  D

X

X



Tλ x PC 3 x

C3

.PCi  Id/ !

i PCi  Id

i 2I

i .Id C .PCi  Id//

i 2I

D

X

i PCi ; .

i 2I

We see that in both cases ( D 0 and  > 0), the simultaneous relaxed projection P projection i PCi ;i can be presented as a relaxation U of a simultaneous i 2I !P P U WD i 2I i PCi or as a simultaneous relaxed projection i 2I i PCi ; with the same relaxation parameter  2 Œ0; 2 for all projections PCi . Therefore, we restrict P our further analysis of simultaneous relaxed projection operators i 2I !i PCi ;i to the case i D  2 Œ0; 2 for all P i 2 I , or, equivalently, to the relaxation T of a simultaneous projection T WD i 2I !i PCi with a relaxation parameter  2 Œ0; 2.

4.4.1 Simultaneous Projection as an Alternating Projection in a Product Space Let Ci  H be nonempty closed convex subsets, i 2 I , w D .!1 ; !2 ; : : : ; !m / 2 P ri m and T WD i 2I !i PCi be a simultaneous projection. In this subsection we present the operator T as an alternating projection in a product Hilbert space H D Hm with the inner product h; iH W H ! R defined by hu; viH WD

m X i D1

!i hui ; vi i,

164

4 Algorithmic Projection Operators

where u D .u1 ; u2 ; : : : ; um / 2 H, v D .v1 ; v2 ; : : : ; vm / 2 H, and with the norm kkH induced by this inner product. Define the subsets C  H and D  H by C WD C1  : : :  Cm and D WD fx D .x1 ; x2 ; : : : ; xm / 2 H W x1 D x2 D : : : D xm g. By Lemma 1.2.8, we have PC x D .PC1 x1 ; PC2 x2 ; : : : ; PCm xm /, where x D .x1 ; : : : ; xm / 2 H. One can easily show that PC; x D .PC1 ; x1 ; PC2 ; x2 ; : : : ; PCm ; xm /, where  2 Œ0; 2. Now, we calculate the metric projection PD y for y D .y1 ; y2 ; : : : ; ym / 2 H. By the definition of the metric projection, we have PD y D argmin x2D

1 kx  yk2H 2 1X !i kx  yi k2 . 2 i D1 m

D argmin x2H

P 2 Since the function f W H ! R defined by f .x/ WD 12 m i D1 !i kx  yi k is differentiable and convex, x is a minimizer of this function if and only if x is its stationary point. After differentiation we obtain the following condition m X

!i .x  yi / D 0.

i D1

By the equality

Pm

i D1 !i

D 1, we obtain x D PD y D .

m X

Pm

!i yi ; : : : ;

i D1

i D1 !i yi ,

m X

i.e.,

!i yi /.

i D1

Subsuming, for the operator T WD PD PC W D ! D, we obtain Tx D .

m X i D1

!i PCi x; : : : ;

m X

!i PCi x/ D .T x; : : : ; T x/.

i D1

Therefore, x 2Fix PD PC if and only if x 2 Fix T . Furthermore, for the operator T D Id C .PD PC  Id/ it holds

4.4 Simultaneous Projection

165

T x D x C .PD PC x  x/ D.

m X

!i PCi ; x; : : : ;

i D1

m X

!i PCi ; x/

i D1

D .T x; T x; : : : ; T x/ D PD PC; x, where x D .x; : : : ; x/ 2 D. We see that the simultaneous projection T WD P i 2I !i PCi W H ! H can be presented equivalently as an alternating projection T D PD PC W D ! D in the product space H D Hm . Note that D  H is a closed affine subspace, consequently, T is firmly nonexpansive (see TheoremP 4.3.7). This leads to the firm nonexpansivity of the simultaneous projection T WD i 2I !i PCi . This property also follows from Corollary 2.2.20. The idea of representing the simultaneous projection as an alternating projection is due to Pierra [284, Sect. 1], where !i D m1 , i 2 I . The idea was continued by Combettes in [117, Sect. III] and [120, Sect. IV].

4.4.2 Properties of the Simultaneous Projection Let Ci  H be nonempty closed convex subsets, i 2 I , and w D .!1 ; : : : ; !m / 2 m be a vector of weights.P In this section we give basic properties of the simultaneous projection T WD i 2I !i PCi and of the proximity function f W H ! R defined by 1X f .x/ WD !i kPCi x  xk2 . (4.57) 2 i 2I P Corollary 4.4.4. The simultaneous projection T WD i 2I !i PCi , where w 2 m , is firmly nonexpansive. If Fix T ¤ ;, then T is a cutter and, for any  2 .0; 2/, its relaxation T is 2 -strongly quasi-nonexpansive and asymptotically regular.  Furthermore,  2 X    kT x  zk  kx  zk  .2  /  .!i PCi x  x/   2

2

(4.58)

i 2I

for any z 2 Fix T . Proof. Since T is a convex combination of firmly nonexpansive operators PCi , i 2 I , the firm nonexpansivity of T follows from Corollary 2.2.20. Let Fix T ¤ ;. Then T is a cutter, by Theorem 2.2.5 (i). Corollary 2.2.9 yields the 2  -strong quasi nonexpansivity of T , i.e., kT x  zk2  kx  zk2 

2 kT x  xk2 . 

166

4 Algorithmic Projection Operators

Now inequality (4.58) follows from the obvious equality T x  x D .T x  x/. The asymptotic regularity of T follows from Theorem 3.4.3. t u T If i 2I Ci ¤ ;, then inequality (4.58) can be strengthened. In the following theorem we allow that the weights !i , i 2 I , can depend on x 2 H. Theorem 4.4.5. Let Ci  H be nonempty closed convex subsets, i 2 I , C WD T Pi 2I Ci ¤ ;, w W H ! m be an appropriate weight function and T WD i 2I !i PCi . Then kT x  zk2  kx  zk2  .2  /

X

!i .x/ kPCi x  xk2 ,

(4.59)

i 2I

for all x 2 H, z 2 C and  2 Œ0; 2.

T T Proof. Let x 2 H, z 2 C and  2 Œ0; 2. We have i 2I Fix Ti D i 2I Ci . By Corollary 2.2.24, PCi is strictly quasi-nonexpansive, i 2 I , consequently, Fix T D T i 2I Ci (see Theorem 2.1.26). Furthermore, PCi is a cutter (see Theorem 2.2.21 (ii)), i.e., hPCi x  x; z  xi  kPCi x  xk2 (see Remark 2.1.31). Now the convexity of the function kk2 yields  2   X   !i .x/.PCi x  x/  z kT x  zk D x C    2

i 2I

 2 X    D kx  zk C   !i .x/.PCi x  x/   i 2I X 2 !i .x/hz  x; PCi x  xi 2

2

i 2I

 kx  zk2 C 2 2

X

X

!i .x/ kPCi x  xk2

i 2I

!i .x/ kPCi x  xk2

i 2I

D kx  zk2  .2  /

X

!i .x/ kPCi x  xk2

i 2I

t u

which completes the proof. 2

Let C ¤ ;. The strong convexity of the function kk yields that estimation (4.59) with a constant weight function w is stronger than estimation (4.58). P Theorem 4.4.6. Let T WD i 2I !i PCi be a simultaneous projection, where w 2 m , and a proximity function f W H ! R be defined by (4.57) Then Fix T D Argmin f .x/. x2H

4.4 Simultaneous Projection

167

If at least one of the subsets Ci , i 2 I , is bounded and the corresponding weight !i > 0, then Fix T ¤ ;. Proof. The proximity function f is convex and differentiable, and X Df .x/ D !i .x  PCi x/ D x  T x i 2I

(see Lemma 2.2.27) Therefore, the sufficient and necessary optimality condition (see Corollary 1.3.3) yields that z 2 Argminx2H f .x/ if and only if Df .z/ D z  T z D 0, i.e., z 2 Fix T . Now suppose that a subset Ci is bounded for some i 2 I and that !i > 0. Then the function f is coercive. By Corollary 1.1.53, Argminx2H f .x/ ¤ ; and Fix T ¤ ;. t u The proximity function f W H ! R defined by (4.57) has the following nice property (cf. [215, Lemma 2]). P Lemma 4.4.7. Let T WD i 2I !i PCi , where w 2 m . Then, for any x 2 H, it holds 1 f .T x/  f .x/  kT x  xk2 . 2 Proof. Let x 2 H. It follows from the definition of the metric projection that kPCi T x  T xk  kPCi x  T xk. Therefore, the properties of the inner product yield f .T x/ D

1X 1X !i kPCi T x  T xk2  !i kPCi x  T xk2 2 i 2I 2 i 2I

D

X 1X 1X !i kPCi x  xk2 C !i kT x  xk2  !i hPCi x  x; T x  xi 2 i 2I 2 i 2I i 2I

D

1X 1X !i kPCi x  xk2 C !i kT x  xk2  kT x  xk2 2 i 2I 2 i 2I

D f .x/ 

1 kT x  xk2 2

which completes the proof.

t u

Let x 2 H. By the equality Df .x/ D x  T x, Lemma 4.4.7 can be written in the form f .x  Df .x//  f .x/  kDf .x/k2 . The equality above is related to the efficiency of a minimization method defined by the recurrence x kC1 D x k  Df .x k / (see [178, Definition 4.5]). It is known that in the case H D Rn the sequence fx k g1 kD0 generated by this recurrence converges to a minimizer of f (see [178, Theorem 4.6]). P It is clear that for aTsimultaneous projection T WD i 2I !i PCi the following inclusion is true C WD i 2I Ci  Fix T . If C ¤ ; and w 2 ri m , then C D Fix T

168

4 Algorithmic Projection Operators

(see Theorem 2.1.14) and, for the proximity function f W H ! R defined by (4.57) it holds that minx2H f .x/ D 0.

4.4.3 Simultaneous Projection for a System of Linear Equations Suppose that Ci  H are hyperplanes, Ci WD fy 2 H W hai ; yi D ˇi g, where ai 2 H, ai ¤ 0, ˇi 2 R, i 2 I . Then the convex feasibility problem reduces to the solution of a system of linear equations hai ; yi D ˇi , i 2 I . We have PCi x D ;xiˇi j . Now the simultaneous x C ˇi hai2;xi ai (see (4.1)) and kPCi x  xk D jhaika ik kai k projection can be presented in the form Tx D x C

X i 2I

!i

ˇi  hai ; xi kai k2

ai ,

(4.60)

where w D .!1 ; : : : ; !m / 2 m is a vector of weights. The proximity function f defined by (4.57) has the form f .x/ D

1 X .hai ; xi  ˇi /2 !i . 2 i 2I kai k2

(4.61)

If kai k D 1, i 2 I , then T has the form Tx D x C

X

!i .ˇi  hai ; xi/ai .

(4.62)

i 2I

If H D Rn , the operator T and the proximity function f can be written in the following, more convenient, matrix forms T x D x  A> D.Ax  b/

(4.63)

and

1 .Ax  b/> D.Ax  b/, (4.64) 2 where D WD diag. !1 2 ; : : : ; !m 2 / We can suppose without loss of generality that ka1 k kam k w 2 ri m . In this case, the proximity function f has the form f .x/ D

f .x/ D 1

1 kAx  bk2D , 2

(4.65)

where kukD D .u> Du/ 2 denotes the norm of the vector u 2 Rm , induced by the positive definite matrix D. The operator T is firmly nonexpansive and its

4.4 Simultaneous Projection

169

relaxation T is 2 -strongly quasi-nonexpansive and asymptotically regular (see  Corollary 4.4.4), and Fix T D Argminx2H f .x/ (see Theorem 4.4.6). Furthermore, both subsets are nonempty. Theorem 4.4.8. Let a function f Argminx2H f .x/ ¤ ;:

W H ! R be defined by (4.61). Then

Proof. The minimization of the proximity function f W H ! R given by (4.61) is equivalent to the following minimization problem minimize h.y/ WD 12 kyk2 p ;xiˇi / subject to i D !i .haika , i D 1; 2; : : : ; m, ik .x; y/ 2 H  Rm ,

(4.66)

where y D . 1 ; 2 ; : : : ; m / 2 Rm and the product Hilbert space H  Rm is equipped with a scalar product hh; ii defined by hh.x; y/; .v; z/ii D hx; vi C y > z, where .x; y/; .v; z/ 2 H  Rm . Note that h is continuous, convex and coercive. Furthermore, the subset X WD f.x; z/ 2 H  Rm W i D

p .hai ; xi  ˇi / !i ; i D 1; 2; : : : ; mg kai k

is closed and convex as an intersection of hyperplanes. Consequently, the function h has a minimizer .x  ; y  / on X (see Corollary 1.1.53), i.e., x  2 Argminx2H f .x/. t u De Pierro and Iusem proved a property which yields Theorem 4.4.8 in the case H D Rn (see [137, Proposition 10]).

4.4.4 Simultaneous Projection for the Linear Feasibility Problem Suppose that Ci  H are closed half-spaces, i.e., Ci WD fy 2 H W hai ; yi  ˇi g, where ai 2 H, ai ¤ 0, ˇi 2 R, i 2 I . Then the convex feasibility problem reduces to the solution of a system of linear inequalities hai ; yi  ˇi , i 2 I , PCi x D .ha ;xiˇi /C .hai ;xiˇi /C ai (see (4.7)) and kPCi x  xk D . Similarly as in the x i kai k kai k2 linear case (see Sect. 4.4.3), the simultaneous projection can be presented in the form X .hai ; xi  ˇi /C !i ai , (4.67) Tx D x  kai k2 i 2I where w D .!1 ; : : : ; !m / 2 m is a vector of weights. The proximity function f defined by (4.57) has the form

170

4 Algorithmic Projection Operators

f .x/ D

1 X Œ.hai ; xi  ˇi /C 2 !i 2 i 2I kai k2

(4.68)

(cf. (1.34)). If kai k D 1, i 2 I , then we have Tx D x 

X

!i .hai ; xi  ˇi /C ai .

(4.69)

i 2I

If H D Rn , the operator T and the corresponding proximity function f can be written in the following more convenient matrix form T x D x  A> D.Ax  b/C

(4.70)

and

1 .Ax  b/> (4.71) C D.Ax  b/C , 2 where D WD diag. !1 2 ; : : : ; !m 2 /. We can suppose without loss of generality that ka1 k kam k w 2 ri m . In this case, the proximity function f has the form f .x/ D

f .x/ D

1 k.Ax  b/C k2D . 2

(4.72)

The operator T is firmly nonexpansive and its relaxation T is 2 -strongly  quasi-nonexpansive and asymptotically regular (see Corollary 4.4.4), and Fix T D Argminx2H f .x/ (see Theorem 4.4.6). Furthermore, both subsets are nonempty. Theorem 4.4.9. Let a function f Argminx2Rn f .x/ ¤ ;.

W H ! R be defined by (4.68). Then

Proof. The minimization of the proximity function f W H ! R given by (4.68) is equivalent to the following minimization problem minimize h.y/ WD 12 kyk2 p ;xiˇi / subject to i  !i .haika ; i D 1; 2; : : : ; m; ik

i  0; i D 1; 2; : : : ; m; .x; y/ 2 H  Rm . The remaining part of the proof is similar to the proof of Theorem 4.4.8 with X replaced by X WD f.x; z/ 2 H  Rm W i 

p .hai ; xi  ˇi / ; i  0; i D 1; 2; : : : ; mg. !i kai k

Note that X is closed and convex as an intersection of closed half-spaces.

t u

4.5 Cyclic Projection

171

De Pierro and Iusem proved a property which yields Theorem 4.4.9 in case H D Rn (see [137, Proposition 13]).

4.5 Cyclic Projection Let Ci  H be nonempty closed convex subsets, i 2 I WD f1; 2; : : : ; mg. Definition 4.5.1. The operator T W H ! H defined by T WD PCm PCm1 : : : PC1

(4.73)

is called a cyclic projection (see Fig. 4.26). The cyclic projection was introduced by Stefan Kaczmarz for hyperplanes in Rn (see [223]). The properties of the cyclic projection were investigated by many authors, e.g., by Halperin [198], Bregman [42], Gurin et al. [196], Gordon et al. [189], Tanabe [321], Herman [204], McCormick [259], Censor [78], Censor et al. [86], Babenko [13], De Pierro and Iusem [137], Dax [129–132], Dye et al. [151], Bauschke and Borwein [22], Bauschke et al. [23], Popa [285], and by Deutsch and Hundal [142, 143]. In this section we present the basic properties of the cyclic projection defined by (4.73). T T Corollary 4.5.2. If i 2I Ci ¤ ;, then Fix T D i 2I Ci . T Proof. Let i 2I Ci ¤ ;. Since Fix PCi D Ci and the metric projection is strictly quasi-nonexpansive, the claim follows from Theorem 2.1.26. t u T Without the assumption i 2I Ci ¤ ; the existence of a fixed point of a cyclic projection T is not guaranteed in general. One can prove, however, that Fix T ¤ ; if all Ci  Rn are half-spaces, i 2 I (see [137, Lemma 1 and Proposition 13]). 2m Corollary 4.5.3. The cyclic projection T WD PCm PCm1 : : : PC1 is mC1 -relaxed 1 firmly nonexpansive. Consequently, if Fix T ¤ ;, then T is m -strongly quasinonexpansive and asymptotically regular. 2m Proof. The operator T is mC1 -relaxed firmly nonexpansive as a composition of m firmly nonexpansive operators (see Corollary 2.2.43). Let now Fix T ¤ ;. Then Corollary 2.2.9 yields that T is m1 -strongly quasi-nonexpansive. The asymptotic regularity follows from Corollary 3.4.6. t u T Denote Si WD PCi PCi 1 : : : PC1 , i 2 I , and S0 WD Id. If C WD i 2I Ci ¤ ;, then Fix Si  C , i 2 I , by Corollary 4.5.2. If we apply inequality (2.52) m-times, then we obtain the following estimation

kT x  zk2  kx  zk2 

m X i D1

for all x 2 H and z 2 C .

kSi x  Si 1 xk2

(4.74)

172

4 Algorithmic Projection Operators

Fig. 4.26 Cyclic projection C1

PC 1 x PC 2 PC 1 x C2

x Tx C3

Fig. 4.27 Fixed point of a composition of projections C1

PC1 x

PC 2 PC 1 x C2

x = Tx C3

Corollary 4.5.4. If at least one of the subsets Ci , i 2 I , is bounded, then the cyclic projection T WD PCm PCm1 : : : PC1 W Cm ! Cm has a fixed point z 2 Cm . Proof. Since the metric projection is nonexpansive and PCi .X / D Ci , i 2 I , the corollary follows from Theorem 2.1.13. t u A fixed point of a cyclic projection is illustrated in Fig. 4.27. Let Ci  H be closed half-spaces, i.e., Ci WD H .ai ; ˇi / D fx 2 H W hai ; xi  ˇi g, where ai 2 H, ai ¤ 0 and ˇi 2 R, i 2 I . The following theorem was proved in [137, Lemma 1 and Proposition 13]. Theorem 4.5.5. If Ci  Rn are half-spaces, i 2 I , then the cyclic projection T WD PCm : : : PCm has a fixed point.

4.5.1 Cyclic Relaxed Projection Definition 4.5.6. Let i 2 Œ0; 2, i 2 I . The operator T WD PCm ;m PCm1;m1 : : : PC1 ;1 , where PCi ;i WD Id C i .PCi  Id/, i 2 I , is called an operator of cyclic relaxed projection (see Fig. 4.28).

4.5 Cyclic Projection

173

Fig. 4.28 Cyclic relaxed projection

PC1 ,λ1 x

PC2 ,λ2 PC1 ,λ1 x

C1 C2

x C3

Tx

Corollary 4.5.7. Let i 2 .0; 2/, i 2 I , and T WD T PCm ;m PCm1 ;m1 : : : PC1 ;1 be the operator of cyclic relaxed projection. If C WD i 2I Ci ¤ ;, then Fix T D C . Proof. It follows from Theorem 2.2.21 (i) and from Remark 2.1.4 that Fix PCi ;i D Ci , i 2 I . The relaxed metric projection PCi ;i is strictly quasi-nonexpansive for i 2 .0; 2/, i 2 I , (see Corollary 2.2.23 (iii)). Therefore, the claim follows from Theorem 2.1.26. t u Corollary 4.5.8. Let i 2 Œ0; 2, i 2 I . The relaxed cyclic projection T WD PCm ;m PCm1;m1 : : : PC1 ;1 is -relaxed firmly nonexpansive, where D

2mmax .m  1/max C 2

and max D maxi 2I i . If max 2 .0; 2/ and Fix T ¤ ;, then T is asymptotically regular. Proof. The operator T is -relaxed firmly nonexpansive as a composition of i relaxed firmly nonexpansive operators, i 2 I (see Theorem 2.2.42). Now, let max 2 .0; 2/ and Fix T ¤ ;. Then  2 .0; 2/ and the asymptotic regularity follows from Corollary 3.4.6. t u One can prove that the cyclic projection is asymptotically regular without the assumption Fix T ¤ ; (see [19, Theorem 3.1]).

4.5.2 Cyclic-Simultaneous Projection Definition 4.5.9. Let Ci  H be closed convex subsets, i 2 I , and w 2 m . The operator S W X ! H defined by S WD

m X

!i Si ,

(4.75)

i D1

where Si WD PCi : : : PC1 , is called the cyclic-simultaneous projection (see Fig. 4.29).

174

4 Algorithmic Projection Operators

Fig. 4.29 Cyclicsimultaneous projection

x

S1x C1

S2x Sx

C2

S3x C3

Corollary 4.5.10. The cyclic-simultaneousPprojection S defined by (4.75) is ˇm 2i relaxed firmly nonexpansive, where ˇ D i D1 !i i C1 2 .0; 2/. Furthermore, if Fix S ¤ ;, then S˛ is strongly quasi-nonexpansive and asymptotically regular for all ˛ 2 .0; 2ˇ 1 /. P 2i Proof. Let ˇ WD m i D1 !i i C1 . It is clear that ˇ 2 .0; 2/. Since the metric projection is firmly nonexpansive, Corollary 2.2.44 yields that S is ˇ-RFNE. Therefore, Sˇ1 is FNE and .Sˇ1 / D Sˇ1 is -RFNE for all  2 .0; 2/, or, in other words, S˛ is ˛ˇ-RFNE for all ˛ 2 .0; 2ˇ 1 /. Now, the second part of the corollary follows from Corollaries 2.2.9 and 3.4.6. t u The cyclic-simultaneous projection is a special case of string-averaging projections which were studied in [91, Sect. 1], [105–107].

4.5.3 Projections with Reflection onto an Obtuse Cone Let Ci  H be closed convex subsets, i 2 I , and K TH be a closed convex and obtuse cone, i.e., K   K. Suppose that C WD K \ i 2I Ci ¤ ;. Consider the following CFP: find a point x 2 C . Denote P WD PCm PCm1 : : : PC1 , P WD Id C .P  Id/, where  2 .0; 2/, and RK WD .PK /2 D 2PK  Id, i.e., P is a relaxation of the cyclic projection P and RK is the reflection operator onto K. Definition 4.5.11. The operator RK P is called a projection-reflection and the operator RK P , where  2 .0; 2/ is called a relaxed projection-reflection (see Fig. 4.30). The operator PRK is called a reflection-projection and the operator P RK , where  2 .0; 2/ is called a reflection-relaxed projection. The following result can be found in [31, Lemma 2.1 (v)]. Lemma 4.5.12. If K  H is a closed convex and obtuse cone, then RK x 2 K for any x 2 H.

4.5 Cyclic Projection

175

Fig. 4.30 Projectionreflection

x

K RK PC 1 x

PC1 x

C C1

Proof. Let x 2 H and denote xC WD PK x. By the Moreau decomposition, we have x D xC C x , where x WD PK  x. Since K is obtuse, x 2 K   K, consequently, RK x D 2PK x  x D 2xC  .xC C x / D xC  x 2 K C K D K t u

which completes the proof.

Bauschke and Kruk studied the properties of the reflection-projection PRK which defines a reflection-projection method (see [31]). This operator is closely related to the projection-reflection operator RK P . Similarly to the equivalence (i),(ii) in Theorem 4.3.4, one can prove that .x  2 Fix RK P and y  D P x  / ” .y  2 Fix P RK and x  D RK y  /. (4.76) Corollary 4.5.13. Let K  H be a closed convex T and obtuse cone, Ci  H be closed convex subsets, i 2 I , and C WD K \ i 2I Ci ¤ ;. If  2 .0; mC1 /, m then: (i) RK P and P RK are nonexpansive, (ii) Fix RK P D Fix P RK D C , (iii) RK P jK is ˇ-strongly quasi-nonexpansive, where ˇ D

.1/mC1 . m

Proof. Since the metric projection is firmly nonexpansive (see Theorem 2.2.21 (iii)), the reflection RK is nonexpansive (see Theorem 2.2.10 (ii)) and the cyclic projection 2m P is ˛-relaxed firmly nonexpansive with ˛ D mC1 (see Corollary 4.5.3). Therefore, its relaxation P is firmly nonexpansive for D ˛ 1 D mC1 2m (see Corollary 2.2.19). Let  2 .0; mC1 /. Then P is -RFNE, where  D  D ˛ 2 .0; 2/  m (see Remark 2.1.3). Consequently, P is nonexpansive (see Theorem 2.2.10 (ii)) and strongly quasi-nonexpansive (see Corollary 2.2.9). Therefore, both operators RK P and P RK are nonexpansive as compositions of nonexpansive operators. Furthermore, Fix PCi D Ci (see Theorem 2.2.21 (i)), Fix P D Fix P D

\ i 2I

Fix PCi D

\ i 2I

Ci D C

176

4 Algorithmic Projection Operators

(see Theorem 2.1.26), Fix RK D K and Fix RK P D Fix P RK D Fix RK \ Fix P D C (see Theorem 2.1.28). Now we see that all assumptions of Theorem 2.1.51 are satisfied for S D RK . Therefore, the operator RK P jK is ˇ-strongly quasinonexpansive, where ˇD

2  ˛ .1  /m C 1 2 D D  ˛ m t u

which completes the proof.

4.5.4 Cyclic Cutter The following definition generalizes the notion of a cyclic projection. Definition 4.5.14. Let Ui W H ! H be cutters, i 2 I . The operator U WD Um Um1 : : : U1 is called a cyclic cutter. Note that the cyclic cutter U WD Um Um1 : : : U1 doe not need to be a cutter even if Ui , i 2 I , have a common fixed point (see Example 2.1.53). Corollary 4.5.15. Let Ui W H ! H be cutters with a common T fixed point, i 2 I , and U WD Um Um1 : : : U1 be a cyclic cutter. Then Fix U D i 2I Fix Ui and U is 1 m -strongly quasi-nonexpansive. Consequently, U is asymptotically regular. Proof. Note that a cutter is 1-strongly T quasi-nonexpansive (see Theorem 2.1.39). Therefore, the equality Fix U D i 2I Fix Ui follows from Theorem 2.1.26. The operator U is m1 -strongly quasi-nonexpansive as a composition of 1-strongly quasinonexpansive operators Ui , i 2 I (see Theorem 2.1.48 (ii)). Now, the asymptotic regularity of U follows from Theorem 3.4.3. t u

4.6 Landweber Operator Let H1 and H2 be two Hilbert spaces, A W H1 ! H2 be a nonzero bounded linear operator and Q  H2 be a nonempty closed convex subset. Definition 4.6.1. The operator T W H1 ! H1 defined by the equality T x WD x C

1 kAk2

A .PQ .Ax/  Ax/

is called the Landweber operator (see Fig. 4.31).

(4.77)

4.6 Landweber Operator

177

Fig. 4.31 The Landweber operator

A H2

H1

Ax s= PQAx-Ax

Tx x+A* s A*

Q

The Landweber operator is closely related to a method for the problem: find x 2 H1 such that Ax 2 Q. The method was proposed by Landweber for approximating least-squares solution of a first kind integral equation [240] (see also [37, Sect. 6.1]). Note that kAk ¤ 0, because A is a nonzero operator, consequently, the operator T is well defined. We can also take max .A A/ or max .AA / instead of kAk2 in Definition 4.6.1 if A is a compact linear operator (see Theorem 1.1.27). Define a proximity function f W H1 ! RC by f .x/ WD

 1 PQ .Ax/  Ax 2 2

(4.78)

(cf. (1.41)). The function f is convex as a composition of a linear operator A and a convex function 12 d 2 .; Q/. Note that f has the required property: f .x/ D 0 if and only if x 2 C D A1 .Q/ (cf. (1.19)).

4.6.1 Main Properties Lemma 4.6.2. Let T W H1 ! H1 be the Landweber operator defined by (4.77) and the corresponding proximity function f W H1 ! RC be defined by (4.78). Then Fix T D Argmin f .x/. x2H1

Proof. The function f is differentiable and Df .x/ D A .Ax PQ .Ax//. It follows from the necessary and sufficient optimality condition (see Corollary 1.3.3) that z 2 Argminx2H1 f .x/ if and only if A .Az  PQ .Az// D 0, i.e., T z D z. t u Theorem 4.6.3. The Landweber operator is firmly nonexpansive. Proof. Since PQ is firmly nonexpansive (see Theorem 2.2.21 (iii)), the implication (i))(iv) in Theorem 2.2.10 yields that the operator Id  PQ is firmly nonexpansive, i.e.,  2 h.u  PQ u/  .v  PQ v/; u  vi  .u  PQ u/  .v  PQ v/

178

4 Algorithmic Projection Operators

for all u; v 2 H1 . Let G WD Id  T , where T W H1 ! H1 is the Landweber operator given by (4.77). If we take u WD Ax and v WD Ay for x; y 2 H1 in the above inequality and apply the inequality kA zk  kA k  kzk and the equality kA k D kAk, then we obtain hG.x/  G.y/; x  yi D D 

1 2

kAk 1

kAk2 1 2

kAk

hA .Id  PQ /Ax  A .Id  PQ /Ay; x  yi h.Id  PQ /Ax  .Id  PQ /Ay; Ax  Ayi   .Id  PQ /Ax  .Id  PQ /Ay 2

 kA k2  .Id  PQ /Ax  .Id  PQ /Ay 2 4 kAk  1  A .Id  PQ /Ax  A .Id  PQ /Ay 2  4 kAk  2  1  1    D A .Id  PQ /Ax  A .Id  PQ /Ay   2 2 kAk kAk D

D kG.x/  G.y/k2 , i.e., G is firmly nonexpansive. By the implication (iv))(i) in Theorem 2.2.10, the Landweber operator T D Id  G is also firmly nonexpansive. t u Corollary 4.6.4. Let  2 .0; 2/ and T W H1 ! H1 be a relaxation of the Landweber operator T with nonempty Fix T . Then T is 2 -strongly quasi nonexpansive and asymptotically regular. Proof. By Theorem 4.6.3 the Landweber operator T is firmly nonexpansive. -strong quasi nonexpansivity of T follows from Corollary 2.2.9 Therefore, the 2  and the asymptotic regularity of T follows from Corollary 3.4.6. t u

4.6.2 Landweber Operator for Linear Systems Let b 2 H2 . If we take Q WD fbg, then fx 2 H1 W Ax 2 Qg is the solution set of the linear equation Ax D b. Of course, PQ .Ax/ D b, and the Landweber operator has the form 1 A .b  Ax/. (4.79) Tx D x C kAk2

4.6 Landweber Operator

179

Define the proximity function f W H1 ! RC for the linear equation Ax D b by f .x/ WD 12 kAx  bk2 . Suppose that H1 D Rn and H2 D Rm with the standard inner product and that b 2 Rm . If we take Q WD fu 2 Rm W u  bg, then fx 2 H1 W Ax 2 Qg is the solution set of a system of linear inequalities Ax  b, where A is a matrix of type m  n with rows ai , i 2 I , x 2 Rn and b 2 Rm (we suppose without loss of generality that ai ¤ 0, i 2 I ). In this case we have PQ .Ax/  Ax D .Ax  b/C (see equality (4.9)), kAk2 D max .A> A/ (see Theorem 1.1.27) and Tx D x 

1 A> .Ax  b/C . max .A> A/

(4.80)

We can also write the system of equations Ax D b or inequalities Ax  b in 1 1 1 1 an equivalent form D 2 Ax D D 2 b or D 2 Ax  D 2 b, respectively, where D is a positive definite matrix (with nonnegative elements in the second case), e.g., D WD diag.

!1 2

ka1 k

;:::;

!m kam k2

/,

(4.81)

where w D .!1 ; : : : ; !m / 2 ri m . If we apply the Landweber operator to these new systems, then we obtain 1 A> D.Ax  b/ max .A> DA/

(4.82)

1 A> D.Ax  b/C , max .A> DA/

(4.83)

Tx D x  and Tx D x 

respectively. We call the operators T given by (4.82) and (4.83), the Landweber operators related to the matrix D. The corresponding proximity functions obtain the form 2 1 1  1  f .x/ D D 2 .Ax  b/ D kAx  bk2D 2 2 and 1 f .x/ D k.Ax  b/C k2D , 2 respectively. Note that in both cases the Landweber operators and the proximity functions differ from the for the original systems. Remark 4.6.5. The Landweber operators defined by (4.82) and (4.83) related to the matrix D given by (4.81), have a nice property. One can easily show that >

A D.Ax  b/ D

m X !i i D1

kai k2

ai .ai > x  ˇi /

180

4 Algorithmic Projection Operators

and A> D.Ax  b/C D

m X !i i D1

kai k2

ai .ai > x  ˇi /C .

Therefore, the terms A> D.Ax  b/ and A> D.Ax  b/C in operators T defined by (4.82) and (4.83), respectively, do not change if we rescale equations or inequalities of the system. Moreover, max .A> DA/ also does not depend on the rescaling. Therefore, the Landweber operators defined by (4.82) and (4.83) do not change after rescaling. Note that they depend only on the vector of weights w 2 ri m . Furthermore, if A has normalized rows and w D . m1 ; m1 ; : : : ; m1 /, then D D diag. m1 ; m1 ; : : : ; m1 / and the Landweber operators (4.82) and (4.83) reduce to (4.79) and (4.80), respectively. Now we state some relationships between the Landweber operators defined by (4.82) or (4.83), where D is given by (4.81), and the simultaneous projection applied to the system Ax D b or Ax  b, respectively, where A is a matrix of type m  n with nonzero rows ai , i 2 I , x 2 Rn and b 2 Rm . Lemma 4.6.6. Let A WD Œa1 ; : : : ; am > be an m  n matrix with nonzero rows ai , i 2 I , and D WD diag. !1 2 ; : : : ; !m 2 /, where w D .!1 ; : : : ; !m / 2 m . Then ka1 k

kam k

0 < max .A> DA/  1. Proof. Note that max .˛1 A1 C ˛2 A2 /  ˛1 max .A1 / C ˛2 max .A2 /, where A1 ; A2 are positive semi-definite matrices and ˛1 ; ˛2  0. Therefore, max .A> DA/ D max .

m X !i i D1



D

kai k2

m X

!i

i D1

kai k2

m X !i i D1

kai k2

ai ai> /

max .ai ai> / max .ai> ai / D 1.

Furthermore, A> DA is a nonzero matrix. Therefore, max .A> DA/ > 0. We call the parameter

L WD

1 max .A> DA/

a Landweber extrapolation parameter or the Landweber step size.

t u

4.6 Landweber Operator

181

Corollary 4.6.7. The Landweber operators T defined by (4.82) and (4.83), where D is given by (4.81), are extrapolations of the simultaneous projection operators U defined by Ux WD x  A> D.Ax  b/ and by Ux WD x  A> D.Ax  b/C , respectively, i.e., T x D x C L .Ux  x/, where the Landweber step size L  1. Remark 4.6.8. Suppose that A is a matrix with normalized rows. If we take w D . m1 ; m1 ; : : : ; m1 / in Lemma 4.6.6, then we obtain max .A> A/ D mmax .A> DA/  m. One can show even more: max .A> A/  s, where s denotes the maximal number of nonzero elements in any column of A (see [55, Proposition 4.1] or [58, Corollary 2.8]), which is of course a stronger result than Lemma 4.6.6.

4.6.3 Extrapolated Landweber Operator for a System of Linear Equations Consider a system of linear equations Ax D b, where A is an m  n matrix with nonzero rows ai , i 2 I , x 2 Rn and b 2 Rm . Let f W Rn ! RC be a proximity function defined by a> x  ˇi 2 1X !i . i (4.84) /, f .x/ WD 2 i 2I kai k where w D .!1 ; : : : ; !m / 2 m . In this section we present an extrapolation of the Landweber operator and we show that this extrapolation is a cutter. The proximity function can be written in the form f .x/ D where D WD diag.

1 .Ax  b/> D.Ax  b/, 2

!1 ; : : : ; !m 2 / ka1 k2 kam k

f .x/ D

(see (4.64)), or, equivalently, in the form 1 > x Gx C g > x C c; 2

(4.85)

where G D A> DA, g D A> Db and c D 12 b > Db. Let T be a simultaneous projection, i.e., T x D x  A> D.Ax  b/. Let x 2 Rn and s.x/ WD T x  x D A> D.b  Ax/. (4.86) It follows from differential rules that  s.x/ D rf .x/ D Gx C g.

(4.87)

182

4 Algorithmic Projection Operators

Denote M WD Argminx2Rn f .x/. It is clear that Fix T D M ¤ ; (see Theorem 4.4.8). Let T ; WD Id C  .T  Id/ be a generalized relaxation of T with the step size function D EL W Rn ! .0; C1/ defined by

EL .x/ WD

ks.x/k2 s.x/> A> DAs.x/

(4.88)

for x … Fix T . We will prove that EL .x/  L for all x 2 Rn (see Lemma 4.6.9 below). Therefore, we call the step size EL defined by (4.88) an extrapolated Landweber step size and operator T EL with this step size—an extrapolated Landweber operator. We have s.x/> A> DAs.x/ > 0 for x … Fix T . Indeed, if 1 s.x/> A> DAs.x/ D 0, then, of course, D 2 As.x/ D 0 and, by (4.86), we have ks.x/k2 D s.x/> s.x/ D .b  Ax/> DAs.x/ 1

1

D .b  Ax/> D 2 D 2 As.x/ D 0, i.e., x 2 Fix T , a contradiction. Therefore, s.x/> A> DAs.x/ > 0 and the step size

EL .x/ is well defined. Equality (4.87) yields s.x/> A> DAs.x/ D s.x/> Gs.x/ D .Gx C g/> .G 2 x C Gg/ which leads to the following equivalent form of the step size EL .x/:

EL .x/ D

kGx C gk2 . .Gx C g/> .G 2 x C Gg/

(4.89)

Lemma 4.6.9. Let x … Fix T and z 2 Fix T . Then the step size EL .x/ defined by (4.88) satisfies the following inequalities 1

hz  x; T x  xi 1  EL .x/  > max .A DA/ kT x  xk2

(4.90)

for any z 2 Fix T . Consequently, the operator T EL is a cutter and, for all  2 .0; 2/, the operator T EL ; is 2  -strongly quasi-nonexpansive and asymptotically regular. Proof. Let EL .x/ be defined by (4.88). The first inequality in (4.90) follows from Lemma 4.6.6. We have

EL .x/ D

1 ks.x/k2 ks.x/k2  , D s.x/> A> DAs.x/ max .A> DA/ max .A> DA/ ks.x/k2

i.e., the second inequality in (4.90) is true. Applying (4.89), we write the third inequality in (4.90) in the form

4.6 Landweber Operator

183

.x  z/> .Gx C g/ kGx C gk2  , .Gx C g/> .G 2 x C Gg/ kGx C gk2

(4.91)

where z 2 Fix T D M . Note that s.z/ D 0 and Gz C g D rf .z/ D 0, by (4.87) and by the sufficient optimality condition. Since G is positive semi-definite, there exists an orthogonal matrix U and a diagonal matrix  WD diag d with d D (ı1 ; ı2 : : : ; ım /  0 such that G D U > U (see Theorem 1.1.34 (iii)). Note that maxi 2I ıi > 0, because A is a nonzero matrix. Let v D . 1 ; 2 ; : : : ; m / 2 Rm be such that v> v > 0 and let x WD U > v C z. Then Gx C g D G.U > v C z/ C g D GU > v D U > v and

kGx C gk2 D v> 2 v > 0,

i.e., Gx C g ¤ 0. Note that x … Fix T since x  T x D s.x/ D Gx C g ¤ 0. Similarly, we obtain .Gx C g/.G 2 x C Gx/ D v> 3 v > 0 and

.x  z/> .Gx C g/ D v> v.

Therefore, (4.91) can be written in equivalent forms v> v v>  2 v  > 2 > 3 v  v v  v or .v> 2 v/2  .v> v/.v> 3 v/  0: We prove that the latter inequality is true. Denote I 0 WD fi 2 I W ıi > 0g. We have .v> 2 v/2  .v> v/.v> 3 v/ X X X D. ıi2 i2 /2  . ıi i2 /. ıi3 i2 / i 2I

i 2I

i 2I

XX D .ıi2 ıj2 i2 j2  ıi ıj3 i2 j2 / i 2I j 2I

D

XX i 2I 0

D

.ıi2 ıj2 i2 j2  ıi ıj3 i2 j2 /

j 2I 0

XX i 2I 0 j 2I 0

ıi2 ıj2 .1 

ıj 2 2 / ıi i j

184

4 Algorithmic Projection Operators

X

D

ıi2 ıj2 .2 

i;j 2I 0 ;i >j

D

X

ıi ıj3 .

i;j 2I 0 ;i >j

ıj ıi  / i2 j2 ıj ıi

ıj  1/2 i2 j2  0 ıi

i.e., the third inequality in (4.90) is true. Note that T is a cutter as a firmly nonexpansive operator having a fixed point (see Theorem 2.2.5 (i) and Corollary 4.4.4), consequently, T is oriented (see Definition 2.4.4). By Corollary 2.4.5, the operator T EL is a cutter. Let  2 .0; 2/. The 2  -strong quasi nonexpansivity and asymptotic regularity of T EL ; follow from Theorems 2.1.39 and 3.4.3. t u

4.7 Projected Landweber Operator Let H1 and H2 be two Hilbert spaces. Consider the following split feasibility problem find x 2 C with Ax 2 Q, if such an x exists, where C  H1 and Q  H2 are closed convex subsets and A W H1 ! H2 is a bounded linear operator. Definition 4.7.1. An operator U W H1 ! H1 defined by U WD PC .Id C

1 2

kAk

A .PQ  Id/A/

(4.92)

is called the projected Landweber operator or an oblique projection (see Fig. 4.32). The following result is due to Byrne [55, Proposition 2.1]. Proposition 4.7.2. Let R WD PC T , where  > 0, be a projected relaxation of the Landweber operator T defined by (4.77), i.e., R x WD PC .x C

 kAk2

A .PQ  Id/Ax/,

(4.93)

and f W C ! RC be the proximity function defined by (4.78). Then Fix R D Argmin f .x/: x2C

Proof. The function f is convex as a composition of a linear operator A and a convex function 12 d 2 .; Q/. Furthermore, f is differentiable and Df .x/ D A .Ax  PQ .Ax//.

4.8 Simultaneous Cutter

185

Fig. 4.32 Projected Landweber operator

A H2

H1

x+

x C

1

A*

2

A*s

Ux A*

Ax s =PQ Ax-Ax Q

Similarly as in the proof of Corollary 1.3.5, for any  > 0, we have x

2 Argmin f ” Df .x/ 2 NC .x/ C

” Df .x/ 2 NC .x/ ” x D PC .x  Df .x// ” x D PC .x C .A .PQ .Ax/  Ax/// ” x D PC .x C

 kAk2

A .PQ  Id/Ax/ D R x

” x 2 Fix R , which completes the proof.

t u

If we take C D H1 and  D 1 in Proposition 4.7.2, then we obtain Lemma 4.6.2. Corollary 4.7.3. Let  2 .0; 2/. The projected relaxation of the Landweber 4 operator, defined by (4.93) is 4 -relaxed firmly nonexpansive. If, furthermore, 2 Fix .PC T / ¤ ;, then R is 2 -strongly quasi-nonexpansive and asymptotically regular. Proof. By Theorem 4.6.3 the Landweber operator T is firmly nonexpansive. 4 -relaxed firm nonexpansivity of R follows from Theorem 2.2.46 (i). The 4 Let Fix.PC T / ¤ ;. The 2 2 -strong quasi nonexpansivity of R follows from Theorem 2.2.46 (iii) and the asymptotic regularity of R follows from Corollary 3.4.5. t u

4.8 Simultaneous Cutter If we replace the projections PCi by cutters in the definition of a simultaneous projection (see Definition 4.4.1) and allow the weights to depend on a current point x 2 H, then we obtain a more general operator. In this section we prove that this operator is strongly quasi-nonexpansive and asymptotically regular.

186

4 Algorithmic Projection Operators

Fig. 4.33 Simultaneous cutter

x

U1x

FixU1

Ux U2x

FixU2

Definition 4.8.1. Let Ui W H ! H be cutters, i 2 I , and w W H ! m be a weight function. An operator U W H ! H defined by Ux WD

X

!i .x/Ui x

(4.94)

i 2I

is called a simultaneous cutter (see Fig. 4.33). By Corollary 2.1.49, a simultaneous cutter defined by (4.94) is a cutter if w W H ! m is an appropriate weight function. Theorem 4.8.2. Let Ui be cutters with a common fixed point, i 2 I , and U WD P ! U i 2I i i be a simultaneous cutter, where the weight function w W H ! m is appropriate. Then U is a cutter, kU x  zk2  kx  zk2  .2  /

X

!i .x/ kUi x  xk2

(4.95)

i 2I

and kU x  zk2  kx  zk2  .2  / kUx  xk2

(4.96)

for all  2 Œ0; 2, x 2 H and z 2 Fix U . Consequently, for all  2 .0; 2/, the operator U is 2  -strongly quasi-nonexpansive and asymptotically regular. Proof. Since a cutter is strongly quasi-nonexpansive (see Theorem 2.1.39) it is strictly quasi-nonexpansive T (see Remark 2.1.44 (iii)). It follows from Theorem 2.1.26 that Fix U D i 2I Fix Ui . Let  2 Œ0; 2, x 2 H and z 2 Fix U . The convexity of the function kk2 and property (2.21) yield  2   X   !i .x/.Ui x  x/  z kU x  zk2 D x C    i 2I

 2 X    D kx  zk2 C 2  !i .x/.Ui x  x/   i 2I X 2 !i .x/hz  x; Ui x  xi i 2I

4.9 Extrapolated Simultaneous Cutter

187

 kx  zk2 C 2 2

X

X

!i .x/ kUi x  xk2

i 2I

!i .x/ kUi x  xk2

i 2I

D kx  zk2  .2  /

X

!i .x/ kUi x  xk2 ,

i 2I

i.e., inequality (4.95) holds. Inequality (4.96) follows now from the convexity of the function kk2 . Let  2 .0; 2/. The 2 -strong quasi nonexpansivity of U follows  from (4.96) and from the obvious equality .2  / kUx  xk2 D

2 kU x  xk2 . 

The asymptotic regularity of U follows now from Theorem 3.4.3.

t u

4.9 Extrapolated Simultaneous Cutter Let Ui W H ! H be cutters with a common fixed point, i 2 I WD f1; 2; : : : ; mg. In this section we will show how to extend the results of Theorem 4.8.2 to a P generalized relaxation of a simultaneous cutter. We denote, as in Sect. 4.8, U WD i 2I !i Ui , where w W H ! m is a weight function. Note that the estimation of the distance kU x  zk2 given by (4.95) is stronger than that given by (4.96). Therefore, we can expect that the range of the relaxation T parameter  which guarantees the strong quasi nonexpansivity of U with respect to i 2I Fix Ui can be extended over the interval Œ0; 2. As we will see, our expectation is true if we allow the relaxation parameter to depend on the point x 2 H. Therefore, it is more convenient to use a generalized relaxation U ; W H ! H of the operator U , defined by U ; .x/ WD x C  .x/.Ux  x/

(4.97)

T (cf. (2.67)). Denote Ci WD Fix Ui and C WD i 2I Ci . If we suppose that the weight function w W H ! m is appropriate, then Theorem 2.1.26 yields Fix U D C .

4.9.1 Properties of the Extrapolated Simultaneous Cutter In this section we consider a generalized relaxations of a simultaneous cutter with a step size function W H ! .0; C1/ satisfying the following inequality

.x/  w .x/,

(4.98)

188

4 Algorithmic Projection Operators

Fig. 4.34 Extrapolated simultaneous cutter

U1x

x

FixU1

Ux U ¾x U2x

where

FixU2

P

2 i 2I !i .x/ kUi x  xk

w .x/ WD P 2   i 2I !i .x/Ui x  x

(4.99)

T for all x … i 2I Fix Ui , and the weight function w W H ! T m is appropriate. We suppose without loss of generality that .x/ D 1 for all T x 2 i 2I Fix Ui . The fact that w is appropriate ensures that Fix U ; D Fix U D i 2I Fix Ui ¤ ; (see Theorem 2.1.26 (i)), consequently, the step size w .x/ is well defined. If .x/  1 for all P x 2 H, then the operator U is an extrapolation of the simultaneous cutter U WD i 2I !i Ui . We call an operator U with a step size function  1 satisfying inequality (4.98) for all x 2 H an extrapolated simultaneous cutter (ESC) (see Fig. 4.34). The existence of an ESC follows from the convexity of the function kk2 . It turns out that an extrapolated simultaneous cutter is a cutter. P Theorem 4.9.1. Let U WD i 2I !i Ui be a simultaneous cutter with an appropriate weight function w W H ! m , U ; be a generalized relaxation of U , defined by (4.97) with the T step size function W H ! .0; C1/ satisfying inequality (4.98) for x … i 2I Fix Ui , and with  2 Œ0; 2. Then the operator U is a cutter. Consequently, for all  2 .0; 2/, the operator U ; is 2 -strongly quasi nonexpansive and asymptotically regular and kU ; x  zk2  kx  zk2  .2  / 2 .x/ kUx  xk2 for all x 2 H and for all z 2

T i 2I

(4.100)

Fix Ui .

Proof. Since a cutter is strictly quasi-nonexpansive (see Theorem 2.1.39), it holds T that Fix U ; D Fix U D i 2I Fix Ui for an appropriate weight function T wWH! m (see Remark 2.4.2 (d) and Theorem 2.1.26). Let x 2 H and z 2 i 2I Fix Ui . If T x 2 i 2I Fix Ui , the inequality hz  x; U x  xi  kU x  xk2 is clear. Now let T x … i 2I Fix Ui . It follows from (2.21) that hz  x; Ui x  xi  kUi x  xk2 , i 2 I , consequently, hz  x; Ux  xi D hz  x; D

X i 2I

X

!i .x/.Ui x  x/i

i 2I

!i .x/hz  x; Ui x  xi

4.9 Extrapolated Simultaneous Cutter

189



X

!i .x/ kUi x  xk2

i 2I

D w .x/ kUx  xk2 . Therefore, inequality (4.98) yields hz  x; Ux  xi  .x/ kUx  xk2 . Multiplying both sides by .x/ and applying the equality U x  x D .x/.Ux  x/ we obtain hz  x; U x  xi  kU x  xk2 . Therefore, U is a cutter (see Remark 2.1.31). Now Theorem 2.1.39 yields the 2  strong quasi nonexpansivity of U ; for all  2 .0; 2/. The asymptotic regularity of U ; follows from Theorem 3.4.3, and inequality (4.100) follows from the definition of U ; x. t u The properties of extrapolations of the simultaneous projection or of simultaneous cutters were studied by Pierra [284], Dos Santos [146, Sect. 4], Cegielski [62, Sect. 4.3], Kiwiel [229, Sect. 3], Bauschke [17, Sects. 7.3 and 8.3], Combettes [118, Sects. 5.4–5.8], [120, Sect. IV], [121, Sect. 2], Crombez [127, Sect. 4], Aleyner and Reich [5, Sect. 3] and by Cegielski and Censor [70, Sect. 9.5]. In [121, Proposition 2.4] a special case of Theorem 4.9.1 was proved, where it was supposed that w 2 ri m and that D w . Extrapolations of simultaneous cutters with appropriate weight functions were introduced in [70, Sect. 9.5].

4.9.2 Extrapolated Simultaneous Projection In this section we apply the results P of Sect. 4.9.1 to the generalized relaxation of the simultaneous projection U WD i 2I !i PCi , where w W H ! m is an appropriate weight function, Ci  H are nonempty closed convex subsets, i 2 I , with C WD T i 2I Ci ¤ ;. We use here the name ‘simultaneous projection’ for a more general case than in Sect. 4.4, because we allow w toP depend on x 2 H. Note that U is a special case of the simultaneous cutter U WD i 2I !i Ui , where Ui WD PCi , i 2 I . We consider a generalized relaxation U ; of U , where  2 Œ0; 2 and the step size function W H ! .0; C1/ satisfies the inequality

.x/  w .x/, where

(4.101)

P

2 i 2I !i .x/ kPCi x  xk

w .x/ WD P 2   i 2I !i .x/PCi x  x

(4.102)

190

4 Algorithmic Projection Operators

Fig. 4.35 Extrapolated simultaneous projection with the step size WD w

x

P C1 Ux

C1

U ¾x P C2

C2

for x … C and w .x/ D 1 for x 2 C . We have C D Fix U D Fix U ; , because w is an appropriate weight function (see Theorem 2.1.26 and Remark 2.4.2 (d)). Therefore, w is well defined. We call an operator U with a step size function

 1 satisfying inequality (4.101) an extrapolated simultaneous projection (see Fig. 4.35). Corollary 4.9.2. Let C ¤ ;, w W H ! m be an appropriate weight function and U W H ! H be a simultaneous projection defined by Ux WD

X

!i .x/PCi x.

(4.103)

i 2I

Furthermore, let U ; be the generalized relaxation of U , defined by (4.97) with  2 Œ0; 2 and with the step size function W H ! .0; C1/ satisfying inequality (4.101). Then U is a cutter. Consequently, for all  2 .0; 2/, U ; is 2  -strongly quasi-nonexpansive and asymptotically regular. Proof. Since the metric projection PC is a cutter with Fix PC D C (see Theorem 2.2.21 (i) and (ii)), the corollary follows directly from Theorem 4.9.1. t u

4.9.3 Extrapolated Simultaneous Projection for LFP Consider a consistent system of linear inequalities in Rn : ai> y  ˇi , i 2 I , where ai 2 Rn , ai ¤ 0 and ˇi 2 R, i 2 I . This system can be written in the matrix form Ay  b, where A is a matrix of type m  n with rows ai , i 2 I , and b 2 Rm is a vector with coordinates ˇi , i 2 I . If we take Ci WD H .ai ; ˇi / and apply (4.7) and (4.70), then we obtain the following expression for the step size w .x/ defined by (4.102): i2 h > P .ai xˇi /C ! .x/ i i 2I kai k

w .x/ D  (4.104)  , A> D.x/.Ax  b/C 2 for x … C WD fy 2 Rn W Ay  bg, where D.x/ WD diag. !1 .x/2 ; : : : ; !m .x/2 / and the ka1 k kam k The generalized relaxation U ; of weight function w W Rn ! m is appropriate. P the simultaneous projection U WD i 2I !i PCi with the step size .x/ WD w .x/ can be presented as a -relaxation of the operator Tw D U w defined by

4.9 Extrapolated Simultaneous Cutter

191

h

P i 2I

!i .x/

.ai> xˇi /C kai k

i2

> Tw x WD x    A D.x/.Ax  b/C A> D.x/.Ax  b/C 2

(4.105)

.a> xˇ /

for x … C and Tw x D x for x 2 C . Note that the terms i kai k i C , i 2 I , and A> D.x/.Ax  b/C do not change if we rescale inequalities of the system (see Remark 4.6.5), consequently, the operator Tw does not depend on the rescaling of the inequalities. Therefore, we can suppose, without loss of generality, that kai k D 1, i 2 I . Then we have P !i .x/Œ.ai> x  ˇi /C 2 > (4.106) Tw x D x  i 2I  A W .x/.Ax  b/C , A> W .x/.Ax  b/C 2 for x … C , where W .x/ WD diag w.x/. As before, C D Fix U D Fix Tw . Moreover, T˛w D Tw for any function ˛ W Rn ! .0; C1/. Therefore, we can take w W Rn ! n Rm C nf0g in (4.106) instead of w W R ! m . Corollary 4.9.3. Let a system of linear inequalities ai> x  ˇi , i 2 I; be consistent and the weight function w W Rm ! m be appropriate. Then the operator Tw given by (4.106) is a cutter. Consequently, for all  2 .0; 2/, its relaxation Tw; is 2  strongly quasi-nonexpansive and asymptotically regular. Proof. Since Tw D U w , where U is defined by (4.103) and w is defined by (4.104), the corollary follows from Corollary 4.9.2. t u

4.9.4 Surrogate Projection Similarly as in Sect. 4.9.3, we consider a consistent system of linear inequalities Ay  b. Denote Ci WD H .ai ; ˇi / D fy 2 Rn W ai> y  ˇi g, where ai 2 Rn is the i th row of the matrix A and ˇi is the i th coordinate of b 2 Rm , i 2 I . We have C WD fy 2 Rn W Ax  bg D

\

Ci :

i 2I

Let v W Rn ! m be a weight function. Let x 2 Rn . If we multiply the particular inequalities ai> y  ˇi of the system Ay  b by nonnegative weights i .x/, i 2 I , and if we add the resulting inequalities, then we obtain the inequality v.x/> Ay  v.x/> b. Denote a.x/ WD

X i 2I

i .x/ai D A> v.x/,

(4.107)

192

4 Algorithmic Projection Operators

ˇ.x/ WD

X

i .x/ˇi D v.x/> b

i 2I

and

Cv.x/ WD fy 2 Rn W v.x/> Ay  v.x/> bg.

It is clear that C  Cv.x/ for any weight function v. If a.x/ ¤ 0, then Cv.x/ is a half-space, Cv.x/ D H .a.x/; ˇ.x//, which is called a surrogate constraint for the system of linear inequalities Ax  b. Definition 4.9.4. We say that a weight function v W Rn ! m is essential (for the system Ax  b) if for any x … C it holds that v.x/> .Ax  b/ > 0

(4.108)

or, equivalently, x … Cv.x/ D H .a.x/; ˇ.x//. It turns out that for a consistent system Ay  b, an essential weight function v defines, for any x 2 Rn , a half-space (a surrogate constraint) containing the solution set C . Lemma 4.9.5. Let C ¤ ;. If v W Rn ! m is an essential weight function, then for any x … C it holds that A> v.x/ ¤ 0 and C  H .a.x/; ˇ.x//. Proof. Let x … C and v.x/> .Ax  b/ > 0. Suppose that A> v.x/ D 0. Then v.x/> b  0, because Cv.x/  C ¤ ;. Consequently, 0 < v.x/> .Ax  b/ D .A> v.x//> x  v.x/> b  0. The contradiction proves that a.x/ WD A> v.x/ ¤ 0. Now we have C  Cv.x/ D H .a.x/; ˇ.x//. t u Let v W Rn ! m be an essential weight function and Sv x be the metric projection of x 2 Rn onto the surrogate constraint H .a.x/; ˇ.x//. If x 2 C , then, of course, Sv x D x. If x … C , then Sv x D x 

.a.x/> x  ˇ.x//C ka.x/k2

a.x/

Dx

Œ.A> v.x//> x  v.x/> bC > A v.x/   A> v.x/2

Dx

Œv.x/> .Ax  b/C > A v.x/.   A> v.x/2

Inequality (4.108) yields Sv x D x 

v.x/> .Ax  b/ >   A v.x/. A> v.x/2

(4.109)

4.9 Extrapolated Simultaneous Cutter

193

Fig. 4.36 Surrogate projection

C1 C x

Sv x

C2

Note that Sv x is well defined, because A> v.x/ ¤ 0 (see Lemma 4.9.5). We call Sv W Rn ! Rn a surrogate projection (see Fig. 4.36). Note that an essential weight function v can be arbitrarily defined on C . A surrogate projection Sv is not a metric projection, because the weights vi defining the half-space H .a.x/; ˇ.x// depend on x … C . Theorem 4.9.6. Let C ¤ ;. If v W Rn ! m is an essential weight function, then Fix Sv D C and Sv is a cutter. Consequently, for any  2 .0; 2/, its relaxation Sv; is 2  -strongly quasi-nonexpansive and asymptotically regular. Proof. Let v be an essential weight function. It is clear that C  Fix Sv . We prove that Fix Sv  C . Let x … C . Then Lemma 4.9.5 and equality (4.109) yield Sv x ¤ x, i.e., x … Fix Sv . We have proved that Fix Sv D C . Let x 2 Rn and z 2 C . If x 2 C , then Sv x D x and the inequality hz  Sv x; x  Sv xi  0

(4.110)

is obvious. Let now x … C . Since Sv x D PCv x and C  Cv , inequality (4.110) follows from the characterization of the metric projection (Theorem 1.2.4). Therefore, Sv is a cutter. The second part of the theorem follows now from Theorem 2.1.39 and from Corollary 3.4.4. t u An appropriate weight function v W Rn ! m does not need to satisfy condition A> v.x/ ¤ 0 for x … C , consequently it does not need to be an essential weight function. Example 4.9.7. Let a 2 Rn , a ¤ 0. Let a1 D a2 D a; a3 D a. Consider a consistent system of inequalities a1> x  0, a2> x  1, a3> x  1. The weight function v W Rn ! 3 defined by v.x/ D . 14 ; 14 ; 12 / for all x 2 Rn is appropriate, because for any x … C we have i .x/.PCi x  x/ ¤ 0 for at least one i , but 1 .x/a1 C 2 .x/a2 C 3 .x/a3 D

1 1 1 a C a  a D 0, 4 4 2

i.e., A> v.x/ D 0. Definition 4.9.8. We say that a weight function v W Rn ! m considers only violated constraints if, for all x … C and all i 2 I , it holds x 2 Ci H) i .x/ D 0.

(4.111)

194

4 Algorithmic Projection Operators

It is clear that a weight function v W Rn ! m which considers only violated constraints is appropriate. The idea of surrogate constraints for a system of linear inequalities was introduces by Merzlyakov [260] and was continued by Oko in [277], Yang and Murty in [351], where various weight functions which consider only violated constraints were proposed, by Kiwiel [229], Kiwiel and Łopuch [233] and Cegielski [67]. The following lemma gives a sufficient condition for a weight function to be essential. Lemma 4.9.9. Let C ¤ ;. If a weight function v W Rn ! m considers only violated constraints, then v is essential and, consequently, A> v.x/ ¤ 0 for all x … C. Proof. Let a weight function v consider only violated constraints and x … C . Denote I.x/ WD fi 2 I W x … Ci g D fi 2 I W ai> x > ˇi g. Let j 2 I.x/ be such that vj .x/ > 0. The existence of such j follows from the fact that v.x/ 2 m . We have v.x/> .Ax  b/ D

X

i .x/.ai> x  ˇi / D

i 2I

X

i .x/.ai> x  ˇi /

i 2I.x/

 j .x/.aj> x  ˇj / > 0, i.e., v is essential. Now the second part of the Lemma follows from Lemma 4.9.5. t u Note that S˛v D Sv for any weight function v W Rn ! m and for any function ˛ W Rn ! .0; C1/. Furthermore, Sv does not change if we rescale the inequalities. Therefore, we can suppose without loss of generality that kai k D 1, i 2 I , and that n v W Rn ! Rm C nf0g instead of v W R ! m . More general versions of the following two results were proved in [68, Theorems 5 and 8]. The first result shows that any extrapolated simultaneous projection is a surrogate projection. Proposition 4.9.10. Let C ¤ ;, w W Rn ! m be an appropriate weight function and v W Rn ! Rm C nf0g be defined by v.x/ WD W .x/.Ax  b/C , where W .x/ WD diag w.x/, x 2 Rn . Then v considers only violated constraints. Moreover, Tw D Sv , where Tw and Sv are defined by (4.106) and (4.109), respectively. Proof. If x 2 C , then, of course, Sv x D x D Tw x. Now let x … C . We have i .x/ D !i .x/.ai> x  ˇi /C , i 2 I . By the fact that w is appropriate, we have j .x/ > 0 for at least one j 2 I , i.e., v.x/ ¤ 0. If x 2 Ci , then .ai> x  ˇi /C D 0 and i .x/ D 0, i 2 I , i.e., v considers only violated constraints. By Lemma 4.9.9, the weight function v is essential and the operator Sv is well defined. Note that

4.9 Extrapolated Simultaneous Cutter

195

.Ax  b/> C W .x/.Ax  b/ D

X

.ai> x  ˇi /C !i .x/.ai> x  ˇi /

i 2I

D

X

!i .x/Œ.ai> x  ˇi /C 2

i 2I

D .Ax  b/> C W .x/.Ax  b/C . Therefore, if we take v.x/ WD W .x/.Ax  b/C in (4.109), then we obtain P Tw x D x 

> 2 i 2I !i .x/Œ.ai x  ˇi /C  >  2 A W .x/.Ax  b/C D Sv x A> W .x/.Ax  b/C 

t u

which completes the proof.

The following result gives a relationship between the operators Sv and Tw , which is, in some sense, converse to this presented in Proposition 4.9.10. Proposition 4.9.11. Let C ¤ ;, a weight function v W Rn ! Rm C nf0g consider only violated constraints and Sv be the surrogate projection defined by (4.109). Then there is an appropriate weight function w W Rn ! m such that Sv D Tw , where Tw is the extrapolated simultaneous projection defined by (4.106). Proof. Denote I.x/ WD fi 2 I W ai> x  ˇi > 0g and define ( !i .x/ WD

.x/ for i 2 I.x/ ˛.x/1 a> ixˇ i

i

0

otherwise,

x 2 Rn , where ˛.x/ WD

X

i .x/ > a x  ˇi i 2I.x/ i

(4.112)

.

If x 2 C , then, of course, Sv x D x D Tw x. Now let x … C . By Lemma 4.9.9, we have v.x/> .Ax  b/ > 0 and A> v.x/ ¤ 0. It is clear that w.x/ 2 m and that w is an appropriate weight function. Note that !i .x/.ai> x  ˇi /C D !i .x/.ai> x  ˇi / for all i 2 I . We have A> W .x/.Ax  b/C D

X

!i .x/.ai> x  ˇi /C ai

i 2I

D

X

!i .x/.ai> x  ˇi /ai

i 2I

D ˛.x/1

X i 2I

i .x/ai D ˛.x/1 A> v.x/

196

4 Algorithmic Projection Operators

and X

!i .x/Œ.ai> x  ˇi /C 2 D

X

i 2I

!i .x/.ai> x  ˇi /.ai> x  ˇi /C

i 2I

D ˛.x/1

X

i .x/.ai> x  ˇi /C

i 2I 1

D ˛.x/ v.x/> .Ax  b/C . Therefore, P Tw x D x  Dx

> 2 i 2I !i .x/Œ.ai x  ˇi /C  >  2 A W .x/.Ax  b/C A> W .x/.Ax  b/C 

Œv.x/> .Ax  b/C > A v.x/ D Sv x   A> v.x/2 t u

which completes the proof.

Let Tw W Rn ! Rn be defined by (4.106) for an appropriate weight function w and Sv W Rn ! Rn be defined by (4.109) for an essential weight function v. Denote T WD fTw W w is appropriateg, S WD fSv W v considers only violated constraintsg and S 0 WD fSv W v is essentialg. Corollary 4.9.12. Let C ¤ ;. It holds T D S  S0.

(4.113)

Proof. The equality in (4.113) follows from Propositions 4.9.10 and 4.9.11. The inclusion in (4.113) follows from Lemma 4.9.9. t u One can easily prove that for a system Ax  b containing at least two inequalities defining two different half-spaces the inclusion in (4.113) is strict. We leave the details to the reader. Corollary 4.9.13. Let C ¤ ;. If a weight function v W Rn ! Rm C nf0g considers only violated constraints, then a surrogate projection Sv W Rn ! Rn is a cutter. Consequently, for any  2 .0; 2/, its relaxation Sv; is 2 -strongly quasi nonexpansive and asymptotically regular. Proof. The corollary follows from Lemma 4.9.9 and from Theorem 4.9.6. Since T D S, the corollary also follows directly from Corollary 4.9.3. t u

4.9.5 Surrogate Projection with Residual Selection In this section we consider a special case of the surrogate projection Sv defined by (4.109). For a matrix A with rows ai , i 2 I , and for a subset L  I , denote by AL

4.9 Extrapolated Simultaneous Cutter

197

a submatrix with rows ai , i 2 L. For simplicity, suppose that L D f1; 2; : : : ; rg, where r  m, i.e.,   AL . AD AI nL We apply the same notation as above for a subvector bL of b, for a subvector vL .x/ of the vector v.x/ and for a subvector rL .x/ of the residual vector r.x/ WD Ax  b, x 2 Rn , i.e.,  bD

     rL .x/ bL vL .x/ and r.x/ D . , v.x/ D bI nL vI nL .x/ rI nL .x/

Let x … C D fx 2 Rn W Ax  bg. Let L WD L.x/  I be such that: (i) AL has full row rank, i.e., ai , i 2 L, are linearly independent, 1 (ii) .AL A> L / .AL x  bL /  0. Note that for any x … C the weight vector 1 vL .x/ WD .AL A> L / .AL x  bL /

is essential for the system AL x  bL , consequently, the weight vector v.x/ with vI nL .x/ D 0 2 Rmr is essential for the system Ax  b. Therefore, the surrogate projection Sv x given by (4.109) is well defined. We say that a subset L.x/ satisfying the conditions (i) and (ii) above is a residual selection (RS) of I . Note that such a subset exists, e.g., L.x/ WD fj.x/g, where j.x/ 2 I.x/ WD fi 2 I W ai> x > ˇi g. Residual selections were studied in [72, Sect. 3.2] and [73, Sect. 3.1], where several constructions of maximal subsets L satisfying the conditions (i) and (ii) above were presented. If we replace the condition (ii) above by the following stronger condition: 1 (iii) .AL A>  0 and AL x  bL  0, L/

then the residual selection L.x/  I is called an obtuse cone selection (OCS). The notion OCS can be explained by the fact that for a full row matrix AL the cone generated by the rows of AL is obtuse (in Linfai W i 2 Lg) if and only if 1 .AL A>  0 (see [232, Lemma 3.1] or [65, Lemma 1.6]). A special case of an L/ obtuse cone selection is a regular obtuse cone selection (ROCS), for which the condition (iii) above is replaced by (iv) ai> aj  0 for i ¤ j , i; j 2 L, and AL x  bL  0 (see Fig. 4.37). The first inequality in (iii) says that the inverse of the Gram matrix G WD AL A> L of rows of AL is nonnegative. Nonsingular matrices with nonnegative inverses, called inverse-positive matrices, play an important role in many areas of mathematics and were studied in [35]. The properties of Gram matrices with nonnegative inverse were also studied in [232, Sect. 3] and [65, Sect. 2], where their

198 Fig. 4.37 Regular obtuse cone selection

4 Algorithmic Projection Operators C1 a1 PC 1 x x a2 C2

a5 C5 PC 3 x

PC 2 x

a3

SÀ x

a4

C PC 4 x C4

C6 a6

relations to obtuse cones are presented. We see that L is an obtuse cone selection if and only if G is inverse-positive and AL x  bL  0. The first inequality in (iv) says that G has nonpositive off-diagonal elements. Nonsingular matrices with a nonnegative inverse having this property are called Mmatrices or Minkowski matrices. The properties of such matrices were studied in [166], [326, Sect. 4], [62, Chap. 5], [232, Sect. 3] and [65, Sect. 2]. In particular, it follows from [166, Theorem 4.3] that a nonsingular Gram matrix with nonpositive off-diagonals has a nonnegative inverse. This result also follows from [62, Theorem 5.4.A], from [232, Lemma 3.2], and from [65, Corollary 2.9]. Several constructions of maximal subsets L.x/ being obtuse cone selection or regular obtuse cone selection are presented in [326, Sect. 4], [62, Chap. 5], [65, Sect. 2]. In [62, Chap. 6], [64, 230, 231] one can find applications of such selections to algorithms for convex minimization problems. Suppose that for any x … C the subset L.x/  I is a residual selection. If we 1 take vL .x/ D .AL A> L / .AL x bL / and vI nL .x/ D 0, then equality (4.109) obtains the form > 1 Sv x D x  A> (4.114) L .AL AL / .AL x  bL /. The operator Sv W Rn ! Rn defined by (4.114) is called a surrogate projection with residual selection. Further properties of the residual selection as well as its applications to the surrogate projection methods are presented in Sect. 5.13.

4.9.6 Extrapolated Simultaneous Subgradient Projection Let ci W H ! R be convex continuous functions, Ci WD fx 2 H W ci .x/  0g be nonempty and Ui be the subgradient projections relative to ci , i.e., ( c C .x/ x  i 2 gi .x/ if ci .x/ > 0 kgi .x/k Ui .x/ WD (4.115) x if ci .x/  0, where gi .x/ 2 @ci .x/ is a subgradient of ci at x, i 2 I (cf. (4.31)). It follows from Lemma 4.2.5 and from Corollary 4.2.6 that Fix Ui D Ci and that Ui is a cutter.

4.10 Extrapolated Cyclic Cutter

199

P We call the operator U WD i 2I !i Ui , where w W H ! m is a weight function, a simultaneous subgradient projection. In this section we apply Theorem 4.9.1 to the extrapolated simultaneous subgradient projection U with the step size function

W H ! .0; C1/ satisfying the following inequality 1  .x/  w .x/, where



Pm

ciC .x/ kgi .x/k

(4.116) 2

i D1 wi .x/

w .x/ WD  2 Pm  ciC .x/    i D1 wi .x/ kgi .x/k2 gi .x/

(4.117)

T for all x … i 2I Ci ¤ ;, and the weight function w is appropriate (for simplicity, .c .x// we use the convention that kgi i .x/kC D 0 if .ci .x//C D 0). Note that the step size w .x/ defined by (4.117) is a special case of the step size (4.99), where Ui is the subgradient projection defined by (4.115), i 2 I . Corollary 4.9.14. Let U ; be P a generalized relaxation of the simultaneous subgradient projection U WD i 2I !i Ui with an appropriate weight function w W H ! m , with T a step size function W H ! .0; C1/ satisfying inequality (4.116) for x … i 2I Ci ¤ ; and with  2 Œ0; 2. Then the operator U is a cutter. Consequently, kU ; x  zk2  kx  zk2  .2  / 2 .x/ kUx  xk2 D kx  zk2 

2 kU ; x  xk2 

T for all x 2 H, z 2 i 2I Fix Ui and for all  2 .0; 2/, i.e., U ; is quasi-nonexpansive and asymptotically regular.

2  -strongly

Proof. Since a subgradient projection Ui is a cutter and Fix Ui D Ci , i 2 I , the corollary follows from Theorem 4.9.1. t u The properties of a special case of the extrapolated simultaneous subgradient projection were studied by Dos Santos in [146], where D w and an appropriate weight function w is supposed to be constant, i.e., w 2 ri m . The study was continued for various convex optimization problems by Cegielski [62, Sect. 4.3], Kiwiel [229, Sects. 3 and 8], [230, 231], Kiwiel and Łopuch [233, Sect. 2], Combettes [118, Sect. 5.6], [120, Sect. V] and by Cegielski and Censor [70].

4.10 Extrapolated Cyclic Cutter Let Ui W H ! H be cutters with a common fixed point, i 2 I WD f1; 2; : : : ; mg. In this section we will extend the results of Corollary 4.5.15 to the generalized relaxation of the cyclic cutter. Similarly as in Sect. 4.9, we allow the relaxation

200

4 Algorithmic Projection Operators

parameter to depend on x 2 H. Therefore, we work with a generalized relaxation U ; of a cyclic cutter U WD Um Um1 : : : U1 , with a step size function W H ! .0; C1/ and T a relaxation parameter  2 Œ0; 2. It follows from Corollary 4.5.15 that Fix U D i 2I Fix Ui and that U is strongly quasi-nonexpansive. We will extend this result to the generalized relaxation of U , defined by U ; x WD I C  .x/.Ux  x/.

4.10.1 Useful Inequalities Let S0 WD Id and Si WD Ui Ui 1 : : : U1 , i D 1; 2; : : : ; m. Of course, U D Sm . Denote u0 WD x; ui WD Ui ui 1 and y i WD ui  ui 1 , (4.118) i D 1; 2; : : : ; m. We have ui D Si x. Lemma 4.10.1. Let Ui W H ! H be cutters with a common fixed point, i 2 I . The following inequalities hold  m  m m X 2 X 2 1 X 1   i m i i i y   hUxx; zxi  hy C: : :Cy ; y i  y  , (4.119)    2 2m i D1 i D1 i D1 where x 2 H and z 2

Tm

i D1 Fix Ui

¤ ;.

Proof. (cf. [71, Lemma 7]) Let x 2 H be arbitrary. We prove the first inequality in (4.119) by induction with respect to m. For m D 1 this inequality follows directly from the definition of a cutter and from inequality (2.21). Suppose that the first inequality in (4.119) is true for some m D k. Let w 2 H. Define V1 WD Id, Vi WD Ui Ui 1 : : : U2 for i D 2; 3; : : : ; kC1, v1 WD w, vi WD Ui vi 1 and zi WD vi vi 1 , i D TkC1 2; 3; : : : ; k C 1. Take w WD U1 x and z 2 i D1 Fix Ui . Then, of course, Si x D Vi w, ui D vi and y i D zi , i D 2; 3; : : : ; k C 1. It follows from the induction assumption that kC1 X hVkC1 w  w; z  wi  hzi C : : : C zkC1 ; zi i (4.120) Note that VkC1 w  w D from (4.120) that

PkC1 i D2

i D2

zi and w  x D y 1 . Since U1 is a cutter, it follows

hSkC1 x  x; z  xi D hVkC1 w  x; z  xi D hVkC1 w  w; z  xi C hU1 x  x; z  xi

 2  hVkC1 w  w; z  wi C hVkC1 w  w; w  xi C y 1 

4.10 Extrapolated Cyclic Cutter



kC1 X

201

hzi C : : : C zkC1 ; zi i C h

i D2

D

kC1 X

kC1 X

 2 zi ; y 1 i C y 1 

i D2

hy i C : : : C y kC1 ; y i i C

i D2

D

kC1 X

kC1 X  2 hy i ; y 1 i C y 1  i D2

hy i C : : : C y kC1 ; y i i.

i D1

Therefore, the first inequality in (4.119) is true for all m 2 N. The second inequality follows from the following equality  m  m m X 2 X X   1 1 2   y i  D  hy i C : : : C y m ; y i i  yi  .  2 2 i D1

i D1

i D1

The third inequality in (4.119) follows from the convexity of the function kk2 .

t u

4.10.2 Properties of the Extrapolated Cyclic Cutter We use the notation from the previous subsection. Define the step size function

max W H ! .0; C1/ by Pm hUx  Si 1 x; Si x  Si 1 xi

max .x/ WD i D1 (4.121) kUx  xk2 for x … Fix U and .x/ D 1 for x 2 Fix U . If we take y i WD Si x  Si 1 x, i D 1; 2; : : : ; m (cf. (4.118)), then Lemma 4.10.1 yields 1 Pm kSi x  Si 1 xk2 1

max .x/  2 i D1 , (4.122)  2 2m kUx  xk where x … Fix U . Note that m m X X hy i C : : : C y m ; y i i D hy 1 C : : : C y i ; y i i. i D1

i D1

Therefore, the step size max .x/ given by (4.121) can be equivalently written in the following form Pm hSi x  x; Si x  Si 1 xi . (4.123)

max .x/ D i D1 kUx  xk2

202

4 Algorithmic Projection Operators

Now consider the generalized relaxation U ; of the cyclic cutter U with a step size function W H ! .0; C1/ satisfying inequality .x/  max .x/ for all x … Fix U , where max .x/ is given by (4.121) or (4.123). Theorem 4.10.2. The operator U is a cutter. Consequently, for all  2 .0; 2/, the operator U ; is 2  -strongly quasi-nonexpansive and asymptotically regular. T Proof. (cf. [71, Lemma 8]) TLet x 2 H,  2 .0; 2/ and z 2 i 2I Fix Ui . It is clear that Fix U ; D Fix U D i 2I Fix Ui (see Theorem 2.1.26 and Remark 2.4.2 (d)). The first inequality in (4.119) can be written in the form hUx  x; z  xi  max .x/ kUx  xk2 .

(4.124)

Consequently, hUx  x; z  xi  .x/ kUx  xk2 . Now Corollary 2.4.5 yields that U is a cutter and U ; is 2  -strongly quasinonexpansive. The asymptotic regularity of U ; follows from Theorem 3.4.3. u t

4.11 Exercises Exercise 4.11.1. Let B.z; /  H, be a ball with the centre z 2 H and a radius  > 0. Prove that B.z; / is a nonempty closed convex subset and that ( PB.z;/ .x/ D

x zC

 kxzk .x

if kx  zk    z/ if kx  zk > .

Exercise 4.11.2. Let A  H1 and B  H2 be closed convex subsets and .x; y/ 2 A  B. Prove that NAB .x; y/ D NA .x/  NB .y/. Exercise 4.11.3. Prove that max .˛1 A1 C ˛2 A2 /  ˛1 max .A1 / C ˛2 max .A2 /, where A1 ; A2 are positive semi-definite matrices and ˛1 ; ˛2  0. Exercise 4.11.4. Prove Corollary 4.6.7 Exercise 4.11.5. Let C  H be a closed convex subset. Prove that PC D Pd.;C / , where Pd.;C / denotes a subgradient projection relative to the distance function d.; C / WD infy2C k  yk.

Chapter 5

Projection Methods

Let X  H be a nonempty closed convex subset and M  X be a subset of solutions of a convex optimization problem defined on X . Such problems were defined in Sect. 1.3. The subset M is closed and convex and, usually, T nonempty. As an example, consider the convex feasibility problem: find x 2 i 2I Ci , where Ci  H are nonempty closed convex subsets, i 2 I WD Tf1; 2; : : : ; mg, if such a point exists. If the problem is consistent, we have M D i 2I Ci ¤ ;. If the problem is inconsistent, then M is defined as a subset of minimizers of a corresponding proximity function f W H ! RC . Let xN 2 X be an approximation of a solution of a convex optimization problem. An iteration in a projection method for this problem has the form x C D U x, N where U W X ! X is an appropriate algorithmic projection operator constructed for this problem. Most of projection methods employ algorithmic operators U which are relaxations of cutters with Fix U  M or, equivalently, SQNE operators (see Theorem 2.1.39). By Corollary 2.2.9, any relaxed firmly nonexpansive operator U W X ! X with Fix U  M has this property. This is the reason why firmly nonexpansive operators are often used in projection methods. Averaged operators are also often employed in these methods. Note, however, that the class of averaged operators and the class of strictly relaxed firmly nonexpansive operators coincide (see Corollary 2.2.17). Therefore, without loss of generality, we will restrict our further analysis to relaxed cutters and to relaxed firmly nonexpansive operators. Any projection method can be presented in the form of a recurrence x0 2 X – arbitrary x kC1 D Rk x k , where Rk WD PX .Id Ck .Tk  Id//, k 2 .0; 2, and Tk W X ! H is a cutter. In most cases Fix Tk  M or Fix.PX Tk /  M , k  0. Examples of such operators were presented in Chaps. 2 and 4. If the operator Tk D T and k D  for all k  0, then we say that the method is autonomous. Otherwise, we say that the method is nonautonomous. The iteration can also be presented in the form A. Cegielski, Iterative Methods for Fixed Point Problems in Hilbert Spaces, Lecture Notes in Mathematics 2057, DOI 10.1007/978-3-642-30901-4 5, © Springer-Verlag Berlin Heidelberg 2012

203

204

5 Projection Methods

x kC1 D PX .x k C k .Tk x k  x k //.

(5.1)

If X is a closed affine subspace and Tk W X ! X , then we can omit the operator PX in (5.1) and we obtain x kC1 D x k C k .Tk x k  x k /.

(5.2)

In autonomous methods the sequence of operators fTk g1 kD0 as well as the sequence of relaxation parameters .k / are constant and these methods have the form x kC1 D PX .x k C .T x k  x k //.

(5.3)

Generally, the convergence (at least weak) of sequences generated by projection methods follows from Opial’s theorem (see Sect. 3.5) or from its generalizations (see Sects. 3.6 and 3.7). In this chapter we show how to apply these theorems in order to prove the convergence.

5.1 Alternating Projection Methods The alternating projection method (APM), also known as the von Neumann method is an iterative method for finding a common point of two nonempty closed convex subsets A; B  H, if such a point exists. One iteration of the method has the form x kC1 D PA PB x k

(5.4)

or x kC1 D T x k , where T WD PA PB is an alternating projection. The properties of this operator were described in Sect. 4.3. In this section we consider a more general method in the form x kC1 D PA .x k C k .PA PB x k  x k //,

(5.5)

where k 2 Œ0; 2 which we call a relaxed alternating projection method. We can also write x kC1 D PA Tk x k , where Tk is the k -relaxation of the alternating projection T WD PA PB . Note that if k D 1, then (5.5) reduces to (5.4).

5.1.1 General Case Corollary 5.1.1. Let A; B  H be nonempty closed convex subsets with a common point and lim inf k .2  k / > 0. Then, for an arbitrary x 0 2 A, the sequence fx k g1 kD0 generated by the relaxed alternating projection method (5.5) converges weakly to a point x  2 A \ B.

5.1 Alternating Projection Methods

205

Proof. The operator T WD PA PB is nonexpansive as a composition of nonexpansive operators PA and PB , and Fix T D A\B (see Corollary 4.3.5 (v)). Furthermore, T W A ! A is a cutter (see Lemma 4.3.2) and a demi-closed operator (see Lemma 3.2.5). Corollary 3.7.3 with X D A yields now the weak convergence of x k to a point x  2 A \ B. t u If we suppose that Fix PA PB ¤ ; instead of A \ B ¤ ; in Corollary 5.1.1, then the weak convergence holds for a tighter range of relaxation parameters. Corollary 5.1.2. Let A; B  H be nonempty closed convex subsets with Fix PA PB ¤ ; and k 2 Œ"; 32  " for some " 2 .0; 34 /. Then, for an arbitrary x 0 2 A, the sequence fx k g1 kD0 generated by the relaxed alternating projection method (5.5) converges weakly to a fixed point of the operator T WD PA PB . Proof. By Corollary 4.3.3, the alternating projection T WD PA PB is 43 -relaxed firmly nonexpansive and the operator S WD T 3 D Id C 34 .T  Id/ is firmly nonex4

pansive. Note that Fix S D Fix T , Sk D Tk , where k D 43 k (see Remarks 2.1.3 and 2.1.4) and that lim infk k .2  k / > 0. Iteration (5.5) can be written in the form x kC1 D PA Sk x k . The weak convergence of x k to a point x  2 Fix PA PB follows now from Corollary 3.7.3. t u Since iteration (5.4) is a special case of (5.5), we obtain the following result immediately. Corollary 5.1.3. Let A; B  H be nonempty closed convex subsets. If FixPA PB ¤ ;, then for an arbitrary x 0 2 H the sequence fx k g1 kD0 generated by the alternating projection method (5.4) converges weakly to a fixed point of the operator T WD PA PB . Corollary 5.1.3 is a special case of [42, Theorem 1], where the weak convergence was proved for the cyclic projection method. The strong convergence of sequences generated by the (relaxed) alternating projection method does not hold in general. An example of a closed convex cone A  H, a closed half-space B  H, with A\B D f0g and a point x 2 H for which .PA PB /k x converges weakly but does not converge strongly was presented by Hundal in [213, Theorem 1]. A simplification of Hundal’s example can be found in [258]. In the literature one can find several conditions for the strong convergence in Corollary 5.1.3, e.g., one of the subsets A; B is compact (see [112, Theorem 4 (a)]) or A; B are closed affine subspaces and d.A; B/ is attained (see [21, Theorem 4.1]). In [20, Sects. 3–6], [14, Corollary 2.4], [258, Theorem 4.1] and [235, Theorem 4.5] one can find other sufficient conditions for the strong convergence. Recall that a composition of RFNE operators is RFNE (see Theorem 2.2.37). Therefore, Corollary 5.1.3 remains true if we replace the projections PA and PB by their relaxations PA; WD .1  / Id CPA and PB; WD .1  / Id CPB , where ;  2 .0; 2/. If A is a closed affine subspace, the weak convergence in Corollary 5.1.2 is guaranteed for a broader range of the relaxation parameter. Note that in this case T .A/  A and the recurrence (5.5) can be written in the equivalent form

206

5 Projection Methods

x kC1 D x k C k .PA PB x k  x k /.

(5.6)

The following result is due to Combettes [117, Theorem 1], where A  H is supposed to be a closed subspace of H. Theorem 5.1.4. Let A  H be a closed affine subspace, B  H a nonempty closed convex subset and lim inf k .2  k / > 0. If Fix PA PB ¤ ;, then for an arbitrary x 0 2 A the sequence fx k g1 kD0 generated by the relaxed alternating projection method (5.6) converges weakly to a fixed point of the operator T WD PA PB . Proof. The operator T WD PA PB is firmly nonexpansive (see Theorem 4.3.7). Therefore, the theorem follows directly from Corollary 3.7.3. t u

5.1.2 Alternating Projection Method for Closed Linear Subspaces Suppose that A and B are closed subspaces of a Hilbert space H. Then the strong convergence holds in Corollary 5.1.3. The following theorem is due to John von Neumann [271, Theorem 13.7]. Theorem 5.1.5 (von Neumann, 1933). Let A; B  H be closed subspaces. Then, for any x 0 2 H, the sequence fx k g1 kD0 generated by the alternating projection method (5.4) converges in norm to y WD PA\B x 0 . Proof. Let x 0 2 H and x k D T k x 0 , k  0, where T WD PA PB . We divide the proof into three parts. (i) Since the metric projection is strictly quasi-nonexpansive (see Theorem 2.2.21 (iii) and Corollary 2.2.9) and A \ B ¤ ;, we have Fix T D A \ B (see Theorem 2.1.26). The weak convergence of x k to a point x  2 A \ B follows from Corollary 5.1.3. (ii) We show that x  D PA\B x 0P . Denote y 2l WD x l and y 2lC1 WD PB x l , l  0. 2k1 l i 2k lC1 It is clear that y  y D , for all i; k  0, i  2k. Let lDi y  y z 2 A \ B be arbitrary. By Theorem 2.2.30 (i), we have hy 2l  y 2lC1 ; zi D hx l  PB x l ; zi D 0 and

hy 2lC1  y 2lC2 ; zi D hPB x l  PA PB x l ; zi D 0,

l  0. Consequently, hy i  x  ; zi D limhy i  y 2k ; zi D limh k

D lim k

k

2k1 X lDi

2k1 X lDi

hy l  y lC1 ; zi D 0,

y l  y lC1 ; zi

5.1 Alternating Projection Methods

207

i  0. In particular, hPB x 0  x  ; x  i D hy 1  x  ; x  i D 0 and

hx 0  x  ; zi D hy 0  x  ; zi D 0,

i.e., x  D PA\B x 0 .   (iii) We show that limk x k  x   D 0. Since the metric projection onto a closed subspace is self-adjoint (see Theorem 2.2.30 (iii)) and idempotent (see Theorem 2.2.21 (i)), we have .T  /k T k D PB PA : : : PB PA PA PB : : : PA PB D PB .PA PB / : : : .PA PB / D PB T 2k1 . „ ƒ‚ …„ ƒ‚ … ƒ‚ … „ k-times

k-times

.k1/-times

Since x  2 Fix T , we have hx k  x  ; x k i D hT x k1  T x  ; x k i D hx k1  x  ; T  x k i. If we iterate the above equalities k-times and apply the facts that x k * x  and PB x  D x  , we obtain hx k  x  ; x k i D hx 0  x  ; .T  /k x k i D hx 0  x  ; .T  /k T k x 0 i D hx 0  x  ; PB T 2k1 x 0 i D hPB x 0  PB x  ; T 2k1 x 0 i D hPB x 0  x  ; x 2k1 i ! hPB x 0  x  ; x  i D 0. Consequently, the weak convergence x k * x  yields  2 lim x k  x   D lim.hx k  x  ; x k i  hx k  x  ; x  i/ D 0 k

k

i.e., x k ! x  D PA\B x 0 .

t u

Several proofs of the von Neumann theorem can be found, e.g., in [152], [140, Theorem 9.3] [235, Theorem 1.1] and [236]. Theorem 5.1.5 remains true if A and B are closed affine subspaces with a common point. The details are left to the reader. One can also estimate the rate of convergence of the alternating projection method x kC1 D PA PB x k . It turns out that when A and B are closed subspaces of H the rate depends on the angle between the subspaces. The following definition is due to Friedricks [170, Sect. 1.1] (see also [139, Sect. 6] and [140, Definition 9.4]). Definition 5.1.6. Let A; B  H be closed subspaces and c.A; B/ WD supfjhx; yij W x 2 A\.A\B/? ; y 2 B \.A\B/? ; kxk  1; kyk  1g

208

5 Projection Methods

The value ˛.A; B/ WD arccos c.A; B/ 2 Œ0; 2  is called an angle between the subspaces A and B. It follows from the Cauchy–Schwarz inequality that c.A; B/ 2 Œ0; 1, consequently, the the angle between the subspaces A and B is well defined. The proof of the following theorem can be found in [11, page 379] or in [140, Sect. 9.8]. Theorem 5.1.7. Let A; B  H be two closed subspaces, c WD c.A; B/, x 0 2 H, and fx k g1 kD0 be generated by the alternating projection method (5.4). Then     k x  PA\B x 0   c 2k1 x 0  PA\B x 0  , consequently fx k g1 kD0 converges geometrically, if c < 1. In papers [290, 291, 313] the angles between subspaces are also used to obtain convergence rates for iterative projection methods. Further results on the alternating projection method can be found, e.g., in [20, 21], [140, Chap. 9] and [174]. Applications of the alternating projection method in various areas of mathematics were presented in [139].

5.2 Extrapolated Alternating Projection Methods The sequences generated by the alternating projection method often converge very slowly. Such a slow convergence is observed, e.g., if A is a hyperplane and B is a ball which is tangent to A (see, e.g., [21, Example 5.3]) or A and B are two closed subspaces with c.A; B/ close to 1. In these cases the angle between x k  PB x k and PA PB x k  PB x k is close to 0 which causes very short steps PA PB x k  x k . Since the alternating projection method has many practical applications, any acceleration technique seems to be important. In this section we give several modifications of the alternating projection method which lead in practice to an acceleration of the convergence. All these methods have the form x kC1 D PA Tk ;k x k where x 0 2 A, Tk ;k W A ! H is a generalized relaxation of the alternating projection T WD PA PB , defined by Tk ;k .x/ WD x C k k .x/.PA PB x  x/ for x 2 H, where fk g1 kD0 W H ! .0; C1/ is a sequence of step size functions and k 2 Œ0; 2 is a relaxation parameter (cf. Sect. 2.4). Suppose, for simplicity, that .x/ D 1 for x 2 A \ B. We can also write x kC1 D PA .x k C k k .x k /.PA PB x k  x k //.

(5.7)

5.2 Extrapolated Alternating Projection Methods

209

If k .x k /  1, k  0, then the method described by recurrence (5.7) is called an extrapolated alternating projection method.

5.2.1 Acceleration Techniques for Consistent Problems Suppose that A \ B ¤ ;. Below we present several step size functions  W A ! Œ1; C1/ which guarantee the week convergence of sequences generated by recurrence (5.7) with k 2 Œ"; 2  " for some " 2 .0; 1/. Let T WD PA PB .

5.2.1.1 Gurin–Polyak–Raik Approach Let GPR .x/ WD

kPB x  xk2 , hPB x  x; T x  xi

(5.8)

for x 2 AnB. The step size (5.8) was proposed by Gurin et al. in [196, Sect. 3]. Recall that GPR .x/  1 for all x 2 A and that GPR .x/ D 1 if and only if PB x 2 A\B (see Lemma 4.3.13). Therefore, recurrence (5.7) with k D GPR .x k / describes an extrapolated alternating projection method. The following theorem extends the results of [196, Theorem 4]. Corollary 5.2.1. Let x 0 2 A and the sequence fx k g1 kD0 be generated by (5.7), where 1  k .x/  GPR .x/ and lim inf k .2  k / > 0. Then x k converges weakly to a point x  2 A \ B. Proof. Denote Tk WD Tk . It follows from Corollary 4.3.14 that the operator TGPR is a cutter. Therefore, Tk are also cutters, k  0, (see Lemma 2.4.7 (ii)).The operator T is nonexpansive as a composition of nonexpansive operators PA and PB . It is clear that Fix Tk D Fix T D A \ B. We have       Tk x k  x k  D k .x k / T x k  x k   T x k  x k      consequently, T x k  x k  ! 0 whenever Tk x k  x k  ! 0. Corollary 3.7.1 with S D T and X D A yields now the weak convergence of x k to a point x  2 A\B. u t Gurin et al. proved a special case of Corollary 5.2.1 with k D GPR and k D 1 for all k  0 (see [196, Theorem 4]).

210

5 Projection Methods

5.2.1.2 Bauschke–Combettes–Kruk Approach Let A  H be a closed affine subspace and x 2 A. Then it follows from Theorem 2.2.33 (i) that hPB x  x; T x  xi D hPB x  T x; T x  xi C kT x  xk2 D kT x  xk2 . Therefore, for x 2 AnB, we have GPR .x/ D

kPB x  xk2 kPB x  xk2 D hPB x  x; T x  xi kT x  xk2

The weak convergence of sequences generated by the extrapolated alternating 2 projection method (5.7) with the step size k .x/ WD kPB xxk2 for x … Fix T D kT xxk A\B, k  0, was proved by Bauschke, Combettes and Kruk in [25, Corollary 4.11].

5.2.1.3 Bauschke–Deutsch–Hundal–Park Approach Let A; B  H be closed subspaces and x 2 A. Then it follows from Theorem 2.2.30 (i) that hx; x  T xi D hx; x  PB xi C hx; PB  T xi D hx; x  PB xi D kx  PB xk2 C hPB x; x  PB xi D kx  PB xk2 . Therefore, for x … Fix T D A \ B, we have GPR .x/ D

kPB x  xk2 kT x  xk

2

D

hx; x  T xi kT x  xk2

.

The strong convergence of sequences generated by the extrapolated alternating projection method (5.7) with the step size k .x/ WD hx;xT xi2 for x … Fix T D kT xxk A \ B, k  0, was proved by Bauschke et al. in [30, Theorem 3.23]. Actually, Bauschke et al. proved that the method is faster than the alternating projection method. Furthermore, they presented the rate of convergence of the method (see [30, Theorem 3.16]).

5.2.2 Acceleration Techniques for Inconsistent Problems In this section we suppose that Fix T ¤ ;, where T WD PA PB , but we do not suppose that A \ B ¤ ;. We present two step size functions  W A ! .0; C1/

5.2 Extrapolated Alternating Projection Methods

211

which guarantee the weak convergence of sequences generated by recurrence (5.7) with k 2 Œ"; 2  " for an arbitrary " 2 .0; 1/. Denote ı WD ı.A; B/ D and

inf

x2A;y2B

kx  yk

N ı.x/ WD kT x  PB xk

for x 2 A. First we consider the general case, where A; B  H are nonempty closed convex subsets. Let CS1 .x/ WD

Q kPB x  xk C hPB x  x; T x  xi kT x  PB xk2  ı.x/ kT x  xk2

Q N for x … Fix T , where ı.x/ 2 Œı; ı.x/. Recall that CS1 .x/  inequality is strict if T x … Fix T (see Lemma 4.3.10).

1 2

and that the

Corollary 5.2.2. Let x 0 2 A and the sequence fx k g1 kD0 be generated by (5.7), where 1  k .x/  CS1 .x/ 2 and lim inf k .2  k / > 0. Then x k converges weakly to a point x  2 Fix PA PB . Proof. Denote Tk WD Tk D Id Ck .T  Id/. It follows from Theorem 4.3.11 that TCS1 is a cutter. Therefore, Tk are also cutters, k  0, (see Lemma 2.4.7 (ii)). It is clear that T is nonexpansive and that Fix Tk D Fix T . We have       Tk x k  x k  D k .x k / T x k  x k   1 T x k  x k  , 2     consequently, T x k  x k  ! 0 whenever Tk x k  x k  ! 0. Corollary 3.7.1 with S D T and X D A yields now the weak convergence of x k to a point x  2 Fix PA PB . t u The convergence of sequences generated by the extrapolated alternating projection method (5.7) with the step size function k WD CS1 , k  0, was proved in [76, Theorem 15 (i)]. Now we consider the case, where A  H is a closed affine subspace. Let CS 2 .x/ WD 1 C

Q 2 .kT x  PB xk  ı/ kT x  xk2

Q N for x … Fix T , where ı.x/ 2 Œı; ı.x/. It is clear that CS 2 .x/  1.

212

5 Projection Methods

Corollary 5.2.3. Let x 0 2 A and the sequence fx k g1 kD0 be generated by recurrence (5.7), where 1  k .x/  CS 2 .x/ and lim inf k .2  k / > 0. Then x k converges weakly to a point x  2 Fix PA PB . Proof. The corollary can be proved similarly to Corollary 5.2.2, where one should apply Theorem 4.3.16 instead of Theorem 4.3.11. t u The convergence of sequences generated by the extrapolated alternating projection method (5.7) with the step size function k WD CS 2 , k  0,was proved in [76, Theorem 15 (ii)]. In [75, 306] the problem of finding a fixed point of the alternating projection PA PB , derived from an inconsistent system of linear equations is considered and incomplete projection methods for finding Fix PA PB are proposed, where PA is replaced by an ‘incomplete projection’ which approximates PA in some sense.

5.2.3 Douglas–Rachford Algorithm Let A; B  H be closed convex subsets. The Douglas–Rachford algorithm (DR) is an iterative method for finding a common point of A and B, if such a point exists. One iteration of the method has the form 1 .RA RB x k C x k /, (5.9) 2 where RA WD 2PA  Id and RB WD 2PB  Id are reflection operators onto A and B, respectively. The iteration can also be written in the form x kC1 D T x k , where T W H ! H is the averaged alternating reflection (AAR), i.e., T WD 12 .RA RB C Id/ (see Sect. 4.3.5). The method is also called an averaged alternating reflection algorithm. The method was introduced by Douglas and Rachford in order to solve a partial differential equation describing the heat conduction problem and was studied in [26–29, 157, 161, 246], where various applications of the method were presented. Consider the following iteration x kC1 D

x kC1 D x k C

k .RA RB x k  x k /, 2

(5.10)

where k 2 Œ0; 2. If k D 1, then we obtain (5.9). We can also write x kC1 D Tk x k , where T denotes a -relaxation of the AAR operator T . The following corollary extends the result of [26, Fact 5.9]. Corollary 5.2.4. Let A; B  H be closed convex. If Fix.RA RB / ¤ ; and lim infk k .2  k / > 0, then for any x 0 2 H the sequence fx k g1 kD0 generated by (5.10) converges weakly to a fixed point of RA RB and the sequence fPB x k g1 kD0 is bounded. Furthermore, if A \ B ¤ ;, then:

5.3 Projected Gradient Method

213

(i) For any weak cluster point y of fPB x k g1 kD0 it holds that y 2 A \ B. (ii) If dim H < 1, then y WD limk PB x k exists and y 2 A \ B. Proof. By Corollary 4.3.17 (iv), the operator T WD 12 .RA RB C Id/ is firmly nonexpansive. Therefore, Corollary 3.7.3 with X D H yields x k * x  2 Fix T . k 1 The boundedness of fPB x k g1 kD0 follows from the boundedness of fx gkD0 and from the nonexpansivity of PB . Let A \ B ¤ ; and z 2 A \ B. Then Corollary 4.3.17 (i) and (iv) yields z 2 Fix T and 2  2  2  2  kC1 x  z D T x k  z  x k  z  k .2  k / T x k  x k  .     Consequently, x k  z converges and limk T x k  x k  D 0, by lim infk k .2  k / > 0. We have 1 limŒPA .2PB x k  x k /  PB x k  D lim .RA RB x k  x k / D lim.T x k  x k / D 0. k k 2 k (i) Let y be a weak cluster point of fPB x k g1 kD0 . Because B is weakly closed (see Theorem 1.1.40) and fPB x k g1  B, we have y 2 B. We prove that kD0 k 1 y 2 A. Suppose y … A and let a subsequence fPB x nk g1 kD0  fPB x gkD0 k 1 converge weakly to y. By the boundedness of fPB x gkD0 and of fx k g1 kD0 and by the nonexpansivity of PA , the sequence fPA .2PB x nk  x nk /g1 kD0 is bounded, consequently, it contains a subsequence fPA .2PB x mk  x mk /g1 kD0 which converges weakly to a point z 2 H. Since A is weakly closed, z 2 A. Because fPA .2PB x k  x k /  PB x k g1 kD0 converges to 0, it converges weakly to 0 and fPA .2PB x mk  x mk /  PB x mk g1 kD0 also converges weakly to 0. On the other hand, fPA .2PB x mk  x mk /  PB x mk g1 kD0 converges weakly to z  y ¤ 0. A contradiction shows that y 2 A. (ii) Let dim H < 1. The first part of the corollary and the nonexpansivity of PB yield     lim PB x k  PB x    lim x k  x   D 0, k



k

for some x 2 Fix RA RB , i.e., limk PB x D PB x  D y. By Corollary 4.3.17 (iii), y 2 A \ B. t u k

5.3 Projected Gradient Method Several optimization problems can be reduced to a variational inequality problem, e.g., finding the metric projection onto a nonempty closed convex subset of a Hilbert space, convex minimization problem, elliptic control problem, etc. In this section we present the most popular method for solving VIP.F ; C / where F W H ! H is a Lipschitz continuous and strongly monotone operator and C  H is a nonempty

214

5 Projection Methods

closed convex subset. The method is called a projected gradient method and has the form x kC1 D PC .x k  F x k /. (5.11) The convergence of sequences generated by the method follows from the theorem below, which shows an application of the Banach fixed point theorem and the properties of the metric projection. Theorem 5.3.1. Let C  H be closed convex, F W H ! H be -Lipschitz /. Then continuous and -strongly monotone over C , where ;  > 0, and  2 .0; 2 2 the operator T WD PC .Id F / is a contraction. Consequently, for any x 0 2 H, the sequence fx k g1 kD0 defined by (5.11) converges to the unique solution of VIP.F ; C / with a rate of the geometric progression. Proof. (cf. [358, Theorem 46.C]) Define G D Id F . Let x; y 2 H. Then kGx  Gyk2 D k.x  y/  .F x  F y/k2 D kx  yk2 C 2 kF x  F yk  2hF x  F y; x  yi  .1 C 2  2  2/ kx  yk2 D .1  /2 kx  yk2 , p where WD 1 C 2  2  2 (note that is well defined, because    and  2 .0; 2 /, and that 2 .0; 1/). Consequently, 2 kGx  Gyk  .1  / kx  yk .

(5.12)

Since PC is nonexpansive, (5.12) yields kPC .x  F x/  PC .y  F y/k D kPC Gx  PC Gyk  .1  / kx  yk , i.e., PC .Id F / is a .1  /-contraction. The Banach fixed point theorem and Theorem 1.3.8 yield that PC .Id F / has a unique fixed point xN 2 C and xN is the unique solution of VIP.F ; C /. Furthermore, the Banach fixed point theorem yields that for any x 0 2 H, the sequence fx k g1 kD0 generated by (5.11) converges geometrically to x. N t u In the literature one can find several method for solving VIP.F ; C / in particular for solving VIP.F ; Fix T /, where T W H ! H is a quasi-nonexpansive operator, for solving convex constrained minimization problems, for finding the metric projection onto the intersection of a finite family of convex subsets, for solving elliptic control problems, etc. For details we send the reader to [18,52,77,94–97,101,144,171,200, 203,211,216–218,237,244,250,251,269,273,274,276,318,320,338,342,343,345– 349, 353].

5.4 Simultaneous Projection Method

215

5.4 Simultaneous Projection Method Let fCi gi 2I  H be a finite family of nonempty closed convex subsets, where I WD f1; 2; T : : : ; mg. We consider the convex feasibility problem of finding a point x  2 C WD i 2I Ci , ifPsuch a point exists. The proximity function f W H ! RC is defined by f .x/ WD 12 i 2I d 2 .x; Ci /. The simultaneous projection method (SPM) for solving this problem has the following form – arbitrary x0 2 H P kC1 k k k x D x C k . i 2I !i PCi x  x /,

(5.13)

where fk g1 kD0  Œ0; 2 is a sequence of relaxation parameters and w D .!1 ; : : : ; !m / 2 P ri m is a vector of weights. We can also write x kC1 D Tk x k , where T WD i 2I !i PCi is the operator of a simultaneous projection and T WD Id C.T  Id/ denotes a relaxation of T . Recall that Fix T D Fix T D Argmin f .x/, x2H

where  > 0 and f W H ! R is a proximity function defined by the equality f .x/ WD

1X !i kPCi x  xk2 2 i 2I

(5.14)

(see Theorem 4.4.6) and that Fix T D C if C ¤ ; (see Corollary 4.4.2).

5.4.1 Convergence of the SPM The convergence of sequences fx k g1 kD0 generated by the simultaneous projection method (5.13) follows from the following result which is due to Combettes [117, Theorem 4]. Corollary 5.4.1. Let fx k g1 kD0  H be a sequence generated by the simultaneous projection method (5.13), where lim infk k .2  k / > 0. If Fix T ¤ ;, then x k converges weakly to a point x  2 Fix T and the following estimation holds d 2 .x kC1 ; Fix T /  d 2 .x 0 ; Fix T / 

k X

 2 l .2  l / T x l  x l  ,

lD0

k  0. If, furthermore, C WD

T i 2I

Ci ¤ ;, then

(5.15)

216

5 Projection Methods

d 2 .x kC1 ; C /  d 2 .x 0 ; C / 

k X

l .2  l /

X

 2 !i PCi x l  x l  ,

(5.16)

i 2I

lD0

k  0. Proof. Let Fix T ¤ ;. The operator T is firmly nonexpansive as a convex combination of firmly nonexpansive operators PCi (see Corollary 2.2.20). Furthermore, T is nonexpansive (see Theorem 2.2.4) and a cutter (see Theorem 2.2.5 (i)). Consequently, T  Id is demi-closed at 0 (see Lemma 3.2.5). If we take X D H, Tk WD T and S WD T in Corollary 3.7.1, then we obtain that x k converges weakly to a point x  2 Fix T and that 2  2  2  2  lC1 x  z D T x l  z  x l  z  l .2  l / T x l  x l  , l  0. Applying this inequality .k C 1/-times for l D 0; 1; : : : ; k and for z D PFix T x 0 , we obtain  2 d 2 .x kC1 ; Fix T /  x kC1  z k  2  2 X  2 D T x k  z  x 0  z  l .2  l / T x l  x l  lD0

D d 2 .x 0 ; Fix T / 

k X

 2 l .2  l / T x l  x l  .

lD0

If C ¤ ;, then estimation (5.16) can be proved in the same way as above applying Theorem 4.4.5. t u If Ci  H are closed subspaces, then the convergence in Corollary 5.4.1 is strong and x  D PC x 0 . This result is due to Reich [295, Theorem 1.7], where the convergence was proved for uniformly convex Banach spaces. The strong convergence of sequences generated by the simultaneous projection method does not hold in general. Bauschke, Matouˇskov´a and Reich, relying on a result of Hundal [213, Theorem 1], gave an example of two closed convex subsets C1 ; C2  H with a nonempty intersection, a simultaneous projection T WD 12 PC1 C 1 k 2 PC2 and a point x for which T x converges weakly but does not converge strongly [32, Theorem 5.1]. If H D Rn and Ci are hyperplanes or half-spaces, i.e., Ci WD H.ai ; ˇi / D fx 2 n R W ai> x D ˇi g or Ci WD H .ai ; ˇi / D fx 2 Rn W ai> x  ˇi g, then it follows from equalities (4.63) and (4.70) that the simultaneous projection method (5.13) can be written in the matrix form x kC1 D x k  k A> D.Ax k  b/

(5.17)

5.4 Simultaneous Projection Method

or

x kC1 D x k  k A> D.Ax k  b/C ,

217

(5.18)

respectively, where D WD The simultaneous projection method (5.13) was introduced by Cimmino in [116], where H D Rn , Ci WD H.ai ; ˇi /, i 2 I , C ¤ ; and k D 2 for all k  0. It follows from (5.17) that Cimmino’s method has the form diag. !1 2 ; !2 2 ; : : : ; !m 2 /. ka1 k ka2 k kam k

x kC1 D x k  2A> D.Ax k  b/.

(5.19)

Cimmino proved the convergence of sequences generated by (5.19) to a solution of the system Ax D b for a nonsingular nn matrix A. Special cases of method (5.13) were studied by Auslender [12], De Pierro and Iusem [134, 135, 215], Pierra [284], Combettes [117]. In particular, the (weak) convergence of the sequences generated by recurrence (5.13) to a solution of the CFP was proved by: (a) Auslender in [12], where H D Rn , C ¤ ;, !i D m1 , i 2 I , and k D 1, (b) Pierra in [284, Theorem 1.1 (i)], where C ¤ ;, !i D m1 , i 2 I , and k D 1 for all k  0, (c) De Pierro and Iusem in [134, Theorem 3], where H D Rn , Ci WD H .ai ; ˇi /, i 2 I , Fix T ¤ ; and k D  2 .0; 2/ for all k  0, (d) Iusem and De Pierro in [215, Theorem 2], where H D Rn , and k D 1 for all k  0, (e) Combettes in [117, Theorem 4], where k 2 Œ"; 2  " for some " 2 .0; 1/.

5.4.2 Projected Simultaneous Projection Methods In this section we consider the general T convex feasibility problem of finding a point x  2 X which is closest to C WD i 2I Ci , where Ci  H are nonempty closed convex subsets, i 2 I WD f1; 2; : : : mg. If X \ C ¤ ;, then the problem is to find x 2 X \ C . The proximity function f W X ! RC is defined by f .x/ WD 1 P 2 i 2I d .x; Ci /. The projected simultaneous projection method has the following 2 form x0 2 X – arbitrary P (5.20) x kC1 D PX .x k C k . i 2I !i PCi x k  x k //, where fk g1 kD0  Œ0; 2 is a sequence of relaxation parameters and w D .!1 ; : : : ; !m / 2 m is a vector of weights. If X D H, then method (5.20) is equivalent to the simultaneous projection method (5.13). We can also write P x kC1 D PX Tk x k , where T WD ! P i C i is the operator of simultaneous i 2I projection. Corollary 5.4.2. Let fx k g1 kD0  X be a sequence generated by the projected simultaneous projection method (5.20), where k 2 Œ"; 2  " for some " 2 .0; 1/.

218

5 Projection Methods

If Fix PX T ¤ ;, where T WD x  2 Fix PX T .

P i 2I

!i PCi , then x k converges weakly to a point

Proof. T is firmly nonexpansive (see Corollary 4.4.4). If we set Sk WD T , k  0, in Corollary 3.7.4, then we obtain the weak convergence of x k to a point x  2 Fix PX T . t u Consider now the following recurrence x kC1 D PX .x k C k .PX Ux k  x k //,

(5.21)

P where x 0 2 H, U WD i 2I !i PCi , w D .!1 ; : : : ; !m / 2 m and k > 0, k  0. Note the difference between (5.21) and (5.20). Corollary 5.4.3. Let fx k g1 kD0  X be a sequence generated by method (5.21), where k 2 Œ"; 32  " for some " 2 .0; 34 /. If Fix PX U ¤ ;, then x k converges weakly to a point x  2 Fix PX U . Proof. U is firmly nonexpansive (see Corollary 4.4.4). This fact together with the firm nonexpansivity of PX yield that PX U is 43 -relaxed firmly nonexpansive and that .PX U / 3 is firmly nonexpansive (see Corollary 2.2.39). Recurrence (5.21) can 4 be written in the form x kC1 D PX .x k C k .T x k  x k //, where k D 43 k , T WD .PX U / 3 D Id C 34 .PX U  Id/. Note that k 2 Œ"0 ; 2  "0  4

where "0 D 43 " 2 .0; 1/. Now, the weak convergence of x k to a point x  2 Fix T D Fix PX U follows from Corollary 3.7.3. t u

5.5 Cyclic Projection Methods Let fCi gi 2I  H be a finite family of nonempty closed convex subsets, where  I WD f1; T 2; : : : ; mg. Consider the convex feasibility problem of finding a point x 2 C WD i 2I Ci , if such a point exists. The cyclic projection method for solving this problem has the following form – arbitrary x0 2 H x kC1 D PCm : : : PC1 x k .

(5.22)

If we replace the operators PCi in (5.22) by their relaxations PCi ;i , where i 2 .0; 2/, i 2 I , then we obtain the following, more general method

5.5 Cyclic Projection Methods

219

x0 2 H – arbitrary kC1 k D PCm ;m : : : PC1 ;1 x , x

(5.23)

which we call the cyclic relaxed projection method. We can also write x kC1 D T x k , where T WD PCm ;m : : : PC1 ;1 is the cyclic relaxed projection. Denote S0 WD Id and Si WD PCi ;i : : : PC1 ;1 for i D 1; 2; : : : ; m. We have Sm D T .

5.5.1 Convergence Corollary 5.5.1. Let fx k g1 kD0  H be a sequence generated by the cyclic relaxed projection method (5.23), where i 2 .0; 2/, i 2 I . If Fix T ¤ ;, then x k converges weakly to a point x  2 Fix T and the following estimation holds d 2 .x kC1 ; Fix T /  d 2 .x 0 ; Fix T / 

k X  2 ˛ T x l  x l  ,

(5.24)

lD0 max where ˛ D 2 and max WD maxfi W i 2 I g. If C WD mmax for all i 2 I , then

d 2 .x kC1 ; C /  d 2 .x 0 ; C / 

T i 2I

Ci ¤ ; and i D 1

k X X   Si x l  Si 1 x l 2 .

(5.25)

lD0 i 2I

Proof. Let Fix T ¤ ;. It follows from Corollary 4.5.8 that T is -RFNE, where  D 2mmax and that T is asymptotically regular. Since T is nonexpansive, it follows .m1/max C2 from Theorem 3.5.3 that x k converges weakly to x  2 Fix T . Corollary 2.2.9 yields max the ˛-strong quasi nonexpansivity of T , where ˛ WD 2 D 2 . Therefore, T is  mmax ˛-SQNE, i.e.,  l      T x  z2  x l  z2  ˛ T x l  x l 2 for all z 2 Fix T , l D 0; 1; : : : If we take z D PFix T x 0 and apply the above inequality for l D 0; 1; : : : ; k, we obtain  2 d 2 .x kC1 ; Fix T /  x kC1  z k  2  2 X  2 D T x k  z  x 0  z  ˛ T x i  x i  i D0

D d 2 .x 0 ; Fix T / 

k X  2 ˛ T x i  x i  . i D0

220

5 Projection Methods

If C ¤ ;, then we obtain estimation (5.25) in the same way as above if we apply inequality (4.74). t u The weak convergence of sequence generated by the cyclic projection method was proved by Bregman [42, Theorem 1]. Sufficient conditions for strong convergence as well as for geometric convergence were given by Gurin et al. [196]. Other conditions for the strong convergence can be found in [14]. De Pierro and Iusem considered the method for a inconsistent convex feasibility problem in Rn (see [137]) and studied the convergence of sequences generated by the method. The rate of convergence for the method in a Hilbert space was studied in [142, 143, 290, 291]. The paper [23] contains a review of the results on the cyclic projection method in a Hilbert space. The cyclic projection method for a consistent system of n linear equations in Rn was introduced by Stefan Kaczmarz [223]. Therefore, the cyclic projection method is also known as the Kaczmarz method. Kaczmarz supposed that the equations are independent or, equivalently, that the solution is uniquely defined and proved the convergence to the solution of sequences generated by the method [223]. Altman proved the equivalence of the method with the Gauss–Seidel method for the system A> Ax D A> b [7, Corollary in Sect. 2]. Halperin considered the cyclic projection method for a finite system of subspaces Vi  H, i 2 I , and proved that for  any x 2 H it holds that limk!1 .PVm PVm1 : : : PV1 /k x  PV x  D 0 (see [198, Theorem 2]). Note that this generalizes the von Neumann Theorem 5.1.5. Gordon, Bender and Herman proposed a method for solving a system of linear equations [189] which is equivalent to the Kaczmarz method, and called it an algebraic reconstruction technique (ART). Tanabe proved the convergence for an arbitrary system of linear equations in Rn [321, Corollary 9]. The Kaczmarz method in Hilbert space was studied by McCormick in [259]. Since 1970 the Kaczmarz method has been applied to computerized tomography and in radiation therapy (see, e.g., [81, 82, 100, 189]). Further properties of the Kaczmarz method as well their modifications can be found in [9, 13, 34, 41, 81, 86, 129–132, 162, 197, 201, 239, 247, 268, 285–287].

5.5.2 Projection-Reflection Method Let Ci  H be closed convex subsets, i 2 I , and K TH be a closed convex and obtuse cone, i.e., K   K. Suppose that C WD K \ i 2I Ci ¤ ;. Consider the following CFP: find x 2 C . Denote P D PCm PCm1 : : : PC1 , P WD Id C.P  Id/, where  2 .0; 2/, and RK WD .PK /2 D 2PK  Id, i.e., P is a relaxation of the cyclic projection P and RK is the reflection operator onto K. Consider the iterative method defined by the following recurrence

5.5 Cyclic Projection Methods Fig. 5.1 Projectionreflection method and reflection-projection method

221

K yk

k

x =RK y

k

xk+1=RK Pµxk C yk+1=Pµ RK y k C1

x kC1 D RK P x k ,

(5.26)

where x 0 2 H. We can also write x kC1 D R x k , where R WD RK P is a -relaxed projection-reflection operator (see Definition 4.5.11). We call the method a projection-reflection method (see Fig. 5.1). Since K is obtuse, x k 2 K for all k  1 (see Lemma 4.5.12). Therefore, without loss of generality, we can suppose that x 0 2 K and that R WD RK P in (5.26) is restricted to K. Corollary 5.5.2. Let K  H be a closed T convex and obtuse cone, Ci 0 H be closed convex subsets, i 2 I , C WD K \ i 2I Ci ¤ ;,  2 .0; mC1 m / and x 2 K. Then the sequence generated by the projection-reflection method (5.26) converges weakly to a point x  2 C . Proof. The operator R jK is nonexpansive and strongly quasi-nonexpansive (see Corollary 4.5.13 (i) and (iii)). Consequently R jK is asymptotically regular (see Theorem 3.4.3). By Corollary 4.5.13 (ii), we have Fix R D C . Let fx k g1 kD0 be generated by the projection-reflection method (5.26). We see that all assumptions of Opial’s Theorem 3.5.1 are satisfied. Therefore, x k converges weakly to x  2 C . u t Consider now sequences fy k g1 kD0 generated by the recurrence y kC1 D P RK y k

(5.27)

with y 0 2 H. We call the method a reflection-projection method. The method is closely related to the projection-reflection method (5.26). If we take x 0 WD RK y 0 , k k we obtain that the sequence fx k g1 kD0 defined by x WD RK y is generated by (5.26). kC1 k Furthermore, y D P x (see Fig. 5.1). Bauschke and Kruk considered the reflection-projection method (5.27) in H D Rn with  D 1 [31] and proved that the sequences generated by the method converge to a point x  2 C (see [31, Theorem 3.1]). The following corollary generalizes this result. Corollary 5.5.3. Let K  H be a closed T convex and obtuse cone, Ci 0 H be closed convex subsets, i 2 I , C WD K \ i 2I Ci ¤ ;,  2 .0; mC1 / and y 2 H. m Then the sequence generated by the reflection-projection method (5.27) converges weakly to a point y  2 C . 0 0 k 1 Proof. Let fy k g1 kD0 be generated by (5.27), x WD RK y and fx gkD0 be generated k k by the projection-reflection method (5.26). We have x D RK y and y kC1 D P x k . By Corollary 4.5.13 (i) and (ii), P RK is nonexpansive and Fix P RK D C .

222

5 Projection Methods

The operator R is strongly quasi-nonexpansive (see Corollary 4.5.13 (iii)), consequently R is asymptotically regular (see Theorem 3.4.3). This fact together with the nonexpansivity of P yield  kC1      y  y k  D P x k  P x k1   x k  x k1      D Rk x 0  Rk1 x 0  ! 0. Therefore,     .P RK /kC1 y 0  .P RK /k y 0  D y kC1  y k  ! 0, i.e., P RK is asymptotically regular. We see that all assumptions of Opial’s Theorem 3.5.1 are satisfied. Therefore, y k converges weakly to a point y  2 C . u t

5.6 Successive Projection Methods Let fCi gi 2I  H be a finite family of nonempty closed convex subsets Ci , i 2 I WD f1; 2; : : : ; mg, with a common T point. We consider the convex feasibility problem of finding a point x  2 C WD i 2I Ci , if such a point exists. The successive projection method or sequential projection method for solving this problem has the following form – arbitrary x0 2 H x kC1 D x k C k .PCik x k  x k /,

(5.28)

1 where fk g1 kD0  Œ0; 2 is a sequence of relaxation parameters and fik gkD0  I is a control sequence or, shortly, control. Note that ik can depend on the current approximation x k 2 H. In this case we identify ik with ik .x k /. It is convenient to define ik .x/ for any x 2 H, i.e., to define a sequence of functions fik g1 kD0 W H ! I . We will, however, relate this sequence of functions to recurrence (5.28). If the sequence of functions fik g1 kD0 is constant, i.e., ik D i for all k  0, then the function i W H ! I is called a control function or, shortly, control. In this case we write ik D i.x k /. A comprehensive overview of the sequential projection methods can be found in [78] and in [108, Chap. 5].

5.6.1 Convergence The convergence of sequences generated by (5.28) depends on the properties of the control sequence fik g1 kD0 . The following definition is due to Gurin et al. (see [196, Sect. 1]).

5.6 Successive Projection Methods

223

Definition 5.6.1. A control sequence fik g1 kD0 W H ! I is called an approximately remotest set control if for any sequence fx k g1 kD0 generated by (5.28) the following implication holds       lim PCik x k  x k  D 0 H) lim max PCi x k  x k  D 0. k

k

i 2I

(5.29)

Definition 5.6.2. A control sequence fik g1 kD0 W H ! I is called an approximately semi-remotest set control if for any sequence fx k g1 kD0 generated by (5.28) the following implication holds       lim PCik x k  x k  D 0 H) lim inf PCi x k  x k  D 0 for all i 2 I . k

k

(5.30)

Remark 5.6.3. Since I is a finite subset, the equality on the right hand  side of (5.29) can be written in the following equivalent form: limk PCi x k  x k  D 0 for all i 2 I . Therefore, implication (5.29) is equivalent to the following one       lim PCik x k  x k  D 0 H) lim PCi x k  x k  D 0 for all i 2 I . k

k

Corollary 5.6.4. Let C ¤ ; and fx k g1 kD0  H be a sequence generated by the successive projection method (5.28), where lim infk k .2  k / > 0. k (i) If fik g1 kD0 is an approximately remotest set control, then x converges weakly  to a point x 2 C . (ii) If H is finite dimensional and fik g1 kD0 is an approximately semi-remotest set control, then x k converges to a point x  2 C .

Proof. Setting X D H, Tk WD PCik and S WD PCi in Corollary 3.7.1, we obtain x k * x  2 Fix PCi D Ci , i 2 I , in (i) and x k ! x  2 Fix PCi D Ci , i 2 I , in (ii). t u

5.6.2 Control Sequences Below we present some control functions or control sequences which satisfy implication (5.29) or at least (5.30). It follows from Corollary 5.6.4 that the successive projection methods with such controls and with relaxation parameters k 2 Œ"; 2  " for some " 2 .0; 1/ generate sequences converging, at least weakly, to a point x  2 C if C ¤ ;. Definition 5.6.5. A control function i W H ! I is called a remotest set control if   i.x/ D argmax PCj x  x  . j 2I

224

5 Projection Methods

Definition 5.6.6. A control sequence fik g1 kD0 W H ! I is called an almost remotest set control if there exists a constant ˛ 2 .0; 1 such that       PCik x k  x k   ˛ max PCj x k  x k  . j 2I

Note that a remotest set control is an almost remotest set control. Furthermore, an almost remotest set control is approximately remotest set control. Definition 5.6.7. Let Cj WD fx 2 H W haj ; xi  ˇj g, j 2 I , be half-spaces, where aj 2 H, aj ¤ 0, and ˇj 2 R , j 2 I . A control i W H ! I is called a maximal residual control if i.x/ D argmax.haj ; xi  ˇj /. j 2I

  Note that if aj  D 1, j 2 I , then a maximal residual control is a remotest set control. Lemma 5.6.8. Let Cj D fx 2 H W haj ; xi  ˇj g, j 2 I , be half-spaces, where aj 2 H, aj ¤ 0, and ˇj 2 R , j 2 I . The maximal residual control is an almost remotest set control. Consequently, it is an approximately remotest set control.     Proof. Let ˛ WD minj 2I aj , ˇ WD maxj 2I aj  and i W H ! I be a maximal residual control. We have   haj ; xi  ˇj   / hai.x/ ; xi  ˇi.x/ D max.haj ; xi  ˇj / D max.aj  aj  j 2I j 2I  ˛ max j 2I

for any x …

T j 2I

  haj ; xi  ˇj   D ˛ max PCj x  x  aj  j 2I

Cj . Therefore,

    hai.x k / ; x k i  ˇi.x k / ˛      max PCj x  x  PCi.xk / x k  x k  D ai.x k /  ˇ j 2I and the control function i W H ! I is an approximately remotest set control.

t u

Definition 5.6.9. We say that a control sequence fik g1 kD0  I is cyclic if for all k  0 it holds fik ; ikC1 ; : : : ; ikCm1 g D I . Without loss of generality we can suppose that the cyclic control sequence has the form ik D k.mod m/ C 1, k  0. Note that a successive projection method with the cyclic control is closely related to the cyclic projection method in the following sense. Let fx k g1 kD0 be generated by

5.6 Successive Projection Methods

225

the cyclic projection method and fy k g1 kD0 be generated by the successive projection method with the cyclic control. If x 0 D y 0 , then x k D y km for all k 2 N. The following definition generalizes the notion of the cyclic control (cf. [243], [244, Definition 2.3] and [78, Definition 3.4]). Definition 5.6.10. We say that a control sequence fik g1 kD0 is almost cyclic if there exists a constant s  m (called an almost cyclicality constant) such that I  fik ; ikC1 ; : : : ; ikCs1 g for all k  0. An almost cyclic control is also called quasi-periodic (see, e.g., [8,153]). A more general type of control, sometimes called quasi-cyclic is studied in [6, 331]. The following definition generalizes the notion of an almost cyclic control (cf. [2, 8, 153]). Definition 5.6.11. We say that a control sequence fik g1 kD0 is repetitive if for any j 2 I and for any k  0 there exists l  k such that il D j . Various variants of successive projection methods with repetitive control were studied, e.g., in [2, 70, 111, 164]. In [164, Sect. 3] the repetitive control was called admissible. Note that a cyclic control sequence fik g1 kD0 is m-almost cyclic and an almost cyclic control sequence with the almost cyclicality constant s D m is cyclic. Furthermore, any almost cyclic control sequence is repetitive. In the sequel we will apply a cyclic control, almost cyclic control and repetitive control also in more general recurrences than (5.28), where the projections operators will be replaced by cutters. In the next sections we present results which are stronger than the following one (see Theorems 5.8.11, 5.8.14, 5.9.3 and 5.9.4). Lemma 5.6.12. Let fx k g1 kD0  H be generated by the successive projection method (5.28). (i) If fik g1 kD0 is almost cyclic, then it is an approximately remotest set control. (ii) If H is finite dimensional, C ¤ ;, lim infk k .2  k / > 0 and fik g1 kD0 is repetitive, then it is an approximately semi-remotest set control. Proof. (i) Let the sequence fik g1 kD0 be almost cyclic with s  m being an almost cyclicality constant. Suppose that     lim PCik x k  x k  D 0. k

(5.31)

Let i 2 I and rk 2 f0; 1; : : : ; s  1g be such that i D ikCrk , k  0. By the triangle inequality, we have

226

5 Projection Methods k 1  kCr  rX  kClC1  x k  x k   x  x kCl 

lD0

D

rX k 1

    kCl PCikCl x kCl  x kCl 

lD0



s2 X

    kCl PCikCl x kCl  x kCl  ,

lD0

consequently,

  lim x kCrk  x k  D 0.

(5.32)

k

The definition of the metric projection and the triangle inequality yield,     PC x k  x k   PC x kCrk  x k  i i      PCi x kCrk  x kCrk  C x kCrk  x k        D PCikCr x kCrk  x kCrk  C x kCrk  x k  . k

  Therefore, (5.31) and (5.32) yield limk PCi x k  x k  D 0, i.e., fik g1 kD0 is approximately regular. (ii) Let H be finite dimensional, C ¤ ;, lim infk k .2  k / > 0 and fik g1 kD0 be repetitive. Since (5.28) is a special case of (3.9) with X D H and Tk WD PCik , by Corollary 3.7.1, we obtain  2 2  2  kC1   x  z  x k  z  k .2  k / PCik x k  x k  for any z 2 C . Therefore, lim infk k .2  k / > 0 yields     lim PCik x k  x k  D 0. k

1 1 Let i 2 I . Since fik g1 kD0 is repetitive, there is a subsequence fnk gkD0  fkgkD0 such that ink D i , k  0. Now we have

          lim inf PCi x k  x k  D lim inf PCin x k  x k  D lim PCik x k  x k  D 0, k

k

k

k

i.e., fik g1 kD0 is an approximately semi-remotest set control sequence.

t u

Since the cyclic control is almost cyclic, it is an approximately remotest set control. Therefore, Corollary 5.6.4 yields that the sequences generated by the cyclic projection method with lim infk k .2 k / > 0 converge (at least weakly) to a point x  2 C if C ¤ ;.

5.6 Successive Projection Methods

227

5.6.3 Examples Example 5.6.13. Consider the successive projection method (5.28) with the cyclic control fik g1 kD0 and with k 2 Œ"; 2  " for some " 2 .0; 1/. The method (also called a cyclic projection method) was studied by: (i) Kaczmarz in [223], where H D Rn , Ci WD fx 2 Rn W hai ; xi D ˇi g, i D 1; 2; : : : ; n, ai 2 Rn are linearly independent and k D 1 for all k  0. Kaczmarz proved that for any starting point x 0 2 Rn the sequence fx k g1 kD0 generated by the method converges to a unique solution of the system hai ; xi D ˇi , i D 1; 2; : : : ; n; (ii) Tanabe in [321], where H D Rn , Ci WD fx 2 Rn W hai ; xi D ˇi g, i 2 I , (with possibly an empty intersection) and k D 1 for all k  0. Tanabe proved the convergence of any sequence generated by the method (see [321, Corollary 9]); (iii) Bregman in [42], where Ci  H are closed convex subsets with C ¤ ; and k D 1 for all k  0. Bregman proved that any sequence generated by the method converges weakly to a point x  2 C (see [42, Theorem 1]); (iv) CensorTet al. in [86], where H D Rn , Ci D fx 2 Rn W hai ; xi D ˇi g, i 2 I , C D i 2I Ci D ; and k D  2 .0; 2/. Let x 0 2 Rn be arbitrary, x k be generated by the successive projection method with cyclic control and x  ./ D limk x km . Censor et al. proved that if  # 0, then x  ./ converges to a least squares solution (see [86, Theorem 1]). Small relaxation parameters for the Kaczmarz method were also used in [201]; (v) Gurin et al. in [196, Theorem 1], where Ci  H, i 2 I , are closed convex subsets with C ¤ ; and k 2 Œ"; 2  " for some " 2 .0; 1/ and for all k  0. Gurin et al. proved the strong convergence of sequences generated by the method if the subsets Ci , i 2 I , have a special structure (e.g., they are half-spaces or are uniformly convex or C has a nonempty interior). Example 5.6.14. Let C ¤ ;. Consider the successive projection method (5.28) with the approximately remotest set control and with k 2 Œ"; 2  " for some " 2 .0; 1/. Corollary 5.6.4 yields x k * x  2 C . (i) The convergence properties of the successive projection method with the remotest set control were studied by Agmon in [1] and by Motzkin and Schoenberg in [266], where H D Rn , Ci D fx 2 Rn W hai ; xi  ˇi g, k D  2 .0; 2/. Agmon proved that the sequences generated by the method converge linearly [1, Theorem 3]. Motzkin and Schoenberg also considered the case  D 2 if C has a nonempty interior and proved the finite convergence in this case. (ii) Bregman in [42] considered the convex feasibility problem with infinitely many subsets Ci  H, i 2 I , having a nonempty intersection, and studied successive projection methods for this problem. In [42, Theorem 2] the convergence x k * x  2 C was proved for the remotest set control (the existence of such control was supposed). Bregman also considered a special case of the problem with infinitely many half-spaces Ci  H, Ci WD fx 2 H W

228

5 Projection Methods

hai ; xi  ˇi g, i 2 I , where fai W i 2 I g is bounded. In [42, Theorem 3] the convergence x k * x  2 C was proved for the maximal residual control (the existence of such control was supposed). (iii) Goffin in [186] considered the linear feasibility problem in Rn and studied sufficient conditions for finite convergence of the sequential projection method with the remotest set control [186, Theorems 7.1 and 7.5]. (iv) Gurin et al. in [196] considered the convex feasibility problem with infinitely many subsets Ci  H, i 2 I , having a nonempty intersection. In [196, Theorems 1 and 2] the convergence x k ! x  2 C was proved for an approximately remotest set control, where Ci satisfy some conditions, e.g., Ci , i 2 I , are half-spaces or uniformly convex subsets, or the intersection has an open interior. (v) Dye and Reich in [152] studied successive projection methods for Ci  H, i 2 I , being one-dimensional subspaces.

5.7 Landweber Method and Projected Landweber Method Let A W H1 ! H2 be a nonzero linear bounded operator and Q  H2 be a nonempty closed convex subset. The Landweber method (LM) for finding x 2 H1 satisfying Ax 2 Q (if such an x exists) is defined by the recurrence x kC1 D x k C

k kAk2

A .PQ .Ax k /  Ax k /,

where x 0 2 H1 is arbitrary and k 2 Œ0; 2. We can also write x kC1 D Tk x k . where T is a -relaxation of the Landweber operator T WD 1 2 A .PQ  Id/A. kAk Landweber proposed the method with k D  2 .0; 2/, k  0, for solving an integral equation (see [240]). The projected Landweber method (PLM) is an iterative method for solving the following split feasibility problem: find x 2 C such that Ax 2 Q, where C  H1 , Q  H2 are nonempty closed convex subsets of Hilbert spaces H1 ; H2 and A W H1 ! H2 is a nonzero linear bounded operator, if such a point exists. The iterative step of the projected Landweber method has the form x kC1 D PC .x k C

k kAk2

A .PQ .Ax k /  Ax k //,

(5.33)

where k 2 Œ0; 2. We can also write x kC1 D PC Tk x k , where T is the -relaxation of the Landweber operator T or x kC1 D Rk x k , where R WD PC T is the projected -relaxation of the Landweber operator. It is clear that if C D H1 , then the projected Landweber method reduces to the Landweber method. The projected Landweber method was studied by Byrne [55] for H1 and H2 being Euclidean spaces and for k D  2 .0; 2/, k  0. Actually, Byrne called the method a CQ algorithm

5.7 Landweber Method and Projected Landweber Method

229

and proved the convergence of sequences generated by the method to a solution of the split feasibility problem, if such a solution exists (see [55, Theorem 2.1]). Generalized versions of the projected Landweber method were studied by Xu [344] for a multiple set split feasibility problem and by Zhao and Yang [361]. Theorem 5.7.1. Let the projected Landweber operator defined by (4.92) have a fixed point. If lim infk k .2  k / > 0, then, for an arbitrary x 0 2 H1 , the sequence fx k g1 kD0 generated by the projected Landweber method (5.33) converges weakly to a point z 2 Fix U . Proof. It follows from Theorem 4.6.3 that the Landweber operator T WD Id C

1 kAk2

A .PQ  Id/A is firmly nonexpansive. The weak convergence of x k to a fixed point of the projected Landweber operator U follows now from Corollary 3.7.4 (iii) with S D T , X D C and k D k , k  0. t u Variants of the projected Landweber method for the consistent split feasibility problem, i.e., such that C \ A1 .Q/ ¤ ;, were studied in [292, 293, 350], where projection onto C and Q were replaced by projections onto closed convex sets Ck  C and Qk  Q. Recall that x  2 C is a fixed point of the projected Landweber operator U if and only if it is a minimizer of the proximity function f W C ! R defined by 2  f .x/ WD 12 PQ .Ax/  Ax  (see Proposition 4.7.2). Consequently, Theorem 5.7.1 gives sufficient conditions for the weak convergence of sequences generated by the projected Landweber method to a solution of the split feasibility problem. Suppose now that H1 D Rn , H2 D Rm and that A is a nonzero matrix of type m  n. If Q D fbg, then the projected Landweber method for finding a solution x  2 C of the linear equation Ax D b has the form x kC1 D PC .x k C

k A> .b  Ax k // max .A> A/

(cf. equality (4.79)). If A has nonzero rows ai 2 Rn , i 2 I , then we can 1 1 write the system Ax D b in an equivalent form D 2 Ax D D 2 b, where D D diag. !1 2 ; !2 2 ; : : : ; !m 2 / and w D .!1 ; !2 ; : : : ; !m / 2 ri m . The projected ka1 k ka2 k kam k Landweber method for the modified system has the form x kC1 D PC .x k C

k A> D.b  Ax k // max .A> DA/

(5.34)

(cf. (4.82). If Q D fy 2 Rm W y  bg, then, similarly as above, the projected Landweber method for finding a solution x  2 C of the linear inequality Ax  b has the form k A> .Ax k  b/C / x kC1 D PC .x k  max .A> A/

230

5 Projection Methods 1

(cf. (4.80)) and the projected Landweber method for the modified system D 2 Ax  1 D 2 b has the form x kC1 D PC .x k 

k A> D.Ax k  b/C / max .A> DA/

(5.35)

(cf. (4.83). If we take a positive definite matrix D with nonnegative elements and C D Rn in (5.35), then we obtain a method which was studied in [87, equality (12) and Theorem 2] (see also [322], [132, Sect. 4], [98], [220, Theorem 2], [160, 222] for related results for a system of linear equations). Note that method (5.34) for 1 1 solving the system D 2 Ax D D 2 b is an extrapolation of the simultaneous projection method (5.17) for solving the system Ax D b (see Corollary 4.6.7). Similarly, 1 1 method (5.35) for solving the system D 2 Ax  D 2 b is an extrapolation of the simultaneous projection method (5.18) for solving the system Ax  b, where the extrapolation parameter (step size) L D  .A1> DA/  1. max

5.8 Simultaneous Cutter Methods In this section we consider the common fixed point problem for a finite family of cutters U D fUi gi 2I , where the operators Ui W H ! H, i 2 I WD f1; 2; : : : ; mg, have a common fixed point. We study the convergence ofPsequences generated by relaxations of simultaneous cutters of the form V WD i 2J !i Vi , where V WD fV g is a finite family of cutters V W H ! H, i 2 J , with the property i i 2J i T T Fix V  Fix U and w W H !

is an appropriate weight function. i i jJ j i 2J i 2I One iteration of the recurrence is defined by x C D V x D x C .

X

!i .x/Vi x  x/,

i 2J

where x 2 H is the current approximation of a solution,  2 Œ0; 2 is a relaxation parameter and w.x/ D .!1 .x/; : : : ; !jJ j .x//. A simple example of such iteration is the simultaneous projection method applied to the convex feasibility problem. In this case U D V D fPCi gi 2I , w 2 m is a vector of constant weights and one iteration of the method has the form X x C D U x D x C . !i PCi x  x/. i 2I

In this section we consider, however, more general methods, where in different iterations different families V WD fVi gi 2J and different weight functions w W H !

jJ j can be applied. We will study the convergence of sequences generated by the following recurrence

5.8 Simultaneous Cutter Methods

x0 2 H – arbitrary P kC1 k k k k k k D x C k . i 2Jk !i .x /Vi x  x /, x

231

(5.36)

k where V k WD fVik gT i 2 Jk , i 2Jk is a sequence of T finite families of cutters Vi W H ! H, 1 k with the property i 2Jk Fix Vi  i 2I Fix Ui ¤ ; for all k  0, fk gkD0  Œ0; 2 is a sequence of relaxation parameters, fwk g1 kD0 W H ! mk is a sequence k of appropriate weight functions, mk WD jJk j and wk .x/ D .!1k .x/; :::; !m .x//. k We call the method defined by (5.36) a simultaneous cutter method (SiCM). An iteration in (5.36) can also be written in the form

x kC1 D x k C k .V k x k  x k /, P k k k where V k x WD i 2Jk !i .x/Vi x for x 2 H. It is clear that V is a cutter (see Corollary 2.1.49). Iterations of type (5.36) were studied by several authors (see [70] and the references therein).

5.8.1 Assumptions on Weight Functions In this section we define some classes of weight functions w W H ! jJ j related to a family of cutters V WD fVi gi 2J and some classes of sequences of weight functions k k fwk g1 kD0 W H ! jJk j related to a sequence of families of cutters V WD fVi gi 2Jk , k 1 k  0. It turns out that if we employ sequences fw gkD0 which belong to these classes, then we can apply convergence results presented in Sects. 3.6 and 3.7 in order to prove the convergence of sequences generated by the simultaneous cutter method. The assumption that the weight functions wk are appropriate is not sufficient for the convergence. Remark 5.8.1. (5.36) can be:

A sequence of weight functions fwk g1 kD0 applied in recurrence

(a) Such that the weights !ik .x k / essentially depend on the iteration’s counter k and on the current point x k , i 2 I , k  0, (b) A sequence of constant weight functions; in this case !ik .x k / D !ik , i 2 I , k  0, (c) A constant sequence; in this case !ik .x k / D !i .x k /, i 2 I , k  0, (d) A constant sequence of constant weight functions; in this case !ik .x k / D !i , i 2 I , k  0. In the practical realization of a simultaneous algorithm there is no difference between cases (a), (b) and (c) and we can use the notation !ik instead of !ik .x k / in (a) and instead of !i .x k / in (c). Nevertheless, it is useful to distinguish the weights which depend on the current point x and the weights which depend on iteration’s counter. Furthermore, some properties of the simultaneous cutter methods depend

232

5 Projection Methods

on the properties of corresponding simultaneous cutter operators, where appropriate weight functions play an important role. Definition 5.8.2. Let V WD fVi gi 2J be a finite family of cutters Vi W H ! H, i 2 J , and ˇ > 0 be a constant. We say that a weight function w W H ! jJ j is ˇ-regular with respect to the family fUi gi 2I , or, shortly, regular if for any x 2 H there exists j 2 J such that  2 !j .x/ Vj x  x   ˇ max kUi x  xk2 . i 2I

(5.37)

T T Suppose that i 2J Fix Vi  i 2I Fix Ui . Note that a weight function which is regular with respect to the family U WD fUi gi 2I is appropriate with respect to the family V WD fVi gi 2J (cf. Definition 2.1.25). It is also clear that if V  U, then a constant weight function w with all positive weights !i , i 2 J , is regular. Example 5.8.3. Let V WD U, I.x/ WD fi 2 I W x … Fix Ui g and m.x/ WD jI.x/j be the number of elements of I.x/, where x 2 H. The following weight functions w W H ! m , where w.x/ D .!1 .x/; !2 .x/; : : : ; !m .x//, x 2 H, are regular: (i) Positive constant weights, i.e., w.x/ WD w 2 ri m

(5.38)

for all x 2 H. To verify that w is regular it suffices to choose j 2 Argmaxi 2I kUi x  xk and take ˇ D mini 2I !i in Definition 5.8.2. In particular, !i .x/ D m1 , i 2 I , x 2 H is regular. (ii) Constant weights for violated constraints, i.e., ( !i .x/ WD

P i

for i 2 I.x/

0

for i … I.x/,

j 2I.x/ j

(5.39)

i 2 I , x 2 H, where v D .1 ; : : : ; m / 2 ri m . To verify that w is regular it suffices to choose j 2 Argmaxi 2I kUi x  xk and take ˇ WD mini 2I !i in Definition 5.8.2. In particular, a weight function ! with ( !i .x/ D

1 m.x/

0

for i 2 I.x/ for i … I.x/,

(5.40)

i 2 I , x 2 H, is regular. The weights of the form (5.39) were applied in [87, Sect. 2] and [215] for some variants of the simultaneous projection method. (iii) Weight functions w W H ! m , where the weights !i .x/ are proportional to kUi x  xk, i.e.,

5.8 Simultaneous Cutter Methods

233

( !i .x/ WD

P

kUi xxk kUj xx k

j 2I

for x … for x 2

0

T j 2I

Fix Uj

j 2I

Fix Uj ,

T

(5.41)

i 2 I , x 2 H. To verify that w is regular it suffices to choose j 2 Argmaxi 2I kUi x  xk and take ˇ WD m1 in Definition 5.8.2. (iv) Weight functions w W H ! m satisfying the condition wi .x/  ı for all i 2 I.x/

(5.42)

x 2 H, for some constant ı 2 .0; m1 . To verify that w is regular it suffices to choose j.x/ 2 Argmaxi 2I kUi x  xk and take ˇ WD ı in Definition 5.8.2. The weights satisfying (5.42) were applied in [120, Sect. III] and in [119, Sect. 1]. Note that the weight functions defined by (5.38) and by (5.39) satisfy (5.42). (v) Weight functions w W H ! m which consider only almost violated constraints, i.e., !i .x/ D 0 for all i … J .x/, where   J .x/ WD fj 2 I W Uj x  x   max kUi x  xkg i 2I

(5.43)

for some 2 .0; 1, i 2 I , x 2 H. To verify that w is regular it suffices 2 to choose j WD j.x/ 2 J .x/ with !j .x/  m1 and take ˇ WD m in Definition 5.8.2. The P existence of such j follows from the fact that !i .x/  0 for all i 2 J .x/ and i 2J .x/ !i .x/ D 1. In particular, the following weight functions satisfy (5.43): (a)

( !i .x/ WD

  1 if i D argmaxj 2I Uj x  x  0 in other cases,

(5.44)

i 2 I , x 2 H, where Ui D PCi for a closed convex subset Ci  H, i 2 I . In this case w defines a remotest set control (cf. Definition 5.6.5). (b)

 !i .x/ WD

1 if i D j.x/ 0 in other cases,

(5.45)

where j.x/ 2 J .x/ for some 2 .0; 1, i 2 I , x 2 H. If Ui D PCi for a closed convex subset Ci  H, i 2 I , then any sequence of weight functions fwk g1 kD0 of this form (with independent of k) is an almost remotest set control (cf. Definition 5.6.6). The next definitions extend the notion of a regular weight function to a sequence of weight functions. Definition 5.8.4. Let V k WD fVik gi 2Jk , be a sequence of families of cutters Vik W H ! H, i 2 Jk , k  0. We say that a sequence of weight functions wk W H ! jJk j is regular (with respect to the family U WD fUi gi 2I ), if there exists a sequence

234

5 Projection Methods

k k fˇk g1 kD0  .0; C1/ with lim infk ˇk > 0 such that w applied to the family V is ˇk -regular with respect to the family U, k  0.

It is clear that for a regular weight function w applied to the family U the constant sequence wk WD w is regular. Definition 5.8.5. Let V k WD fVik gi 2Jk be a sequence of cutters Vik W H ! H, i 2 Jk , k  0, and the sequence fx k g1 kD0 be generated by recurrence (5.36). We say that a sequence of appropriate weight functions wk W H ! jJk j applied to the sequence of families fV k g1 kD0 is: (i) Approximately regular (with respect to the family U WD fUi gi 2I ) if there exists a sequence fik g1 kD0 with ik 2 Jk for all k  0, such that the following implication holds  2   lim !ikk .x k / Vikk x k  x k  D 0 H) lim Ui x k  x k  D 0, for all i 2 I , k

k

(5.46) (ii) Approximately semi-regular (with respect to the family U WD fUi gi 2I ) if there exists a sequence fik g1 kD0 with ik 2 Jk for all k  0, such that the following implication holds  2   lim !ikk .x k / Vikk x k  x k  D 0 H) lim inf Ui x k  x k  D 0, for all i 2 I . k

k

(5.47) A sequence of weight functions fwk g1 kD0 applied to recurrence (5.36) is also called a control sequence or a control. If fwk g1 kD0 satisfies (i) or (ii), then we also say that the control in recurrence (5.36) is approximately regular or approximately semi-regular. Remark 5.8.6. Let x k be generated by recurrence (5.36). (i) T By Corollary 3.7.1 (i), the sequence fx k g1 er monotone with respect to kD0 is Fej´ P k k k Fix U , because V WD ! V is a cutter as a convex combination i i 2I i 2Jk T i i T of cutters (see Corollary 2.1.49) and i 2I Fix Ui  i 2Jk Fix Vik . Therefore, x k is bounded. Corollary 2.1.37 yields  k k    V x  x k   z  x k  i   T for any z 2 i 2I Fix Ui , consequently, Vik x k  x k  is bounded. Since !ik .x k / is also bounded, we have the following equivalence    2 lim !ikk .x k / Vikk x k  x k  D 0 ” lim !ikk .x k / Vikk x k  x k  D 0 (5.48) k

k

for any ik 2 Jk , k  0. Therefore, implication (5.46) is equivalent to the following one

5.8 Simultaneous Cutter Methods

235

    lim !ikk .x k / Vikk x k  x k  D 0 H) lim Ui x k  x k  D 0 for all i 2 I , k

k

(5.49) and implication (5.47) is equivalent to the following one     lim !ikk .x k / Vikk x k  x k  D 0 H) lim inf Ui x k  x k  D 0 for all i 2 I . k

k

(5.50) (ii) The following condition yields the approximate regularity of fwk g1 kD0 : There is a sequence fik g1 with i 2 J for all k  0, such that k k kD0 lim inf !ikk .x k / > 0 k

and     lim Vikk x k  x k  D 0 H) lim Ui x k  x k  D 0 for all i 2 I . k

k

(5.51)

In particular, implication (5.51) holds if  k k    V x  x k   ˇ max Ui x k  x k  ik i 2I

(5.52)

for all k  0 and for a constant ˇ > 0. If Jk D I , k  0, then the following conk k dition also implies the approximate regularity of fwk g1 kD0 : lim infk !i .x / > 0 k k and for all i 2 I with !i .x / > 0 it holds     k k V x  x k   ˇ Ui x k  x k  for all k  0. i

(5.53)

If Ui D PCi for closed convex subsets Ci  H, i 2 I , then we say that an algorithm satisfying the latter condition is linearly focusing (cf. [22, Definition 4.8]). Lemma 5.8.7. Let V k WD fVik gi 2Jk be a sequence of families of cutters Vik W H ! H, i 2 Jk , k  0, and a sequence fx k g1 kD0 be generated by recurrence (5.36). If the sequence of weight functions wk W H ! jJk j : (i) Is regular, then wk is approximately regular; k (ii) Contains a regular subsequence fwnk g1 kD0 , then w is approximately semiregular. k 1 Proof. (i) Let a sequence fwk g1 kD0 applied to the sequence of families fV gkD0 be ˇk -regular, where lim infk ˇk > 0. Then, for an arbitrary k, there exists ik 2 Jk such that  2  2 !ikk .x k / Vikk x k  x k   ˇk max Ui x k  x k  . i 2I

 2 If !ikk .x k / Vikk x k  x k  ! 0, then, of course,

236

5 Projection Methods

 2 lim ˇk max Ui x k  x k  D 0 i 2I

k

and, due to the positivity of lim infk ˇk , we have  2 lim max Ui x k  x k  D 0. i 2I

k

    Consequently, limk maxi 2I Ui x k  x k  D 0, i.e., limk Ui x k  x k  D 0 for all i 2 I . k 1 (ii) Let a subsequence fwnk g1 kD0  fw gkD0 applied to the sequence of families nk V , k  0, be regular, i.e., for an arbitrary k, there exists ink 2 Jnk such that 2    !innk .x nk / Vinn k x nk  x nk   ˇnk max kUi x nk  x nk k2 k

i 2I

k

and lim infk ˇnk > 0. If  2 lim !ikk .x k / Vikk x k  x k  D 0 k

then, of course,

2    lim !innk .x nk / Vinn k x nk  x nk  D 0, k

k

k

consequently, lim ˇnk max kUi x nk  x nk k2 D 0. i 2I

k

By the positivity of lim infk ˇnk , we have lim max kUi x nk  x nk k2 D 0, i 2I

k

i.e.,

 2 lim inf max Ui x k  x k  D 0 k

i 2I

  or, in other words, lim infk Ui x k  x k  D 0 for all i 2 I .

t u

Now we present theorems which give sufficient conditions for important control sequences to be approximately (semi-)regular. Consider a special case of recurrence (5.36), where Jk D I , Fix Vik  Fix Ui for all k  0 and for all i 2 I . We can write recurrence (5.36) in the form x kC1 D x k C k

X

!ik .Vik x k  x k /,

(5.54)

i 2I

where k 2 Œ0; 2, k  0. Denote, for simplicity, !ik D !ik .x k /. Let Ik  I be nonempty and !ik D 0 for i … Ik , k  0. Then (5.54) is equivalent to

5.8 Simultaneous Cutter Methods

237

x kC1 D x k C k

X

!ik .Vik x k  x k /,

(5.55)

i 2Ik

where we can suppose without loss of generality that wk W H ! jIk j . Recurrence (5.55) describes two stage algorithms, where fIk g1 kD0 is a control sequence for the outer stage and fwk g1 kD0 is a control sequence for inner stages. The application of such algorithms is convenient if m D j I j is a big number. In this case we can split I onto smaller subsets. The control Ik is, actually, a subset of I on which the algorithm works in kth iteration. Consider the following condition for sequences generated by recurrence (5.55): For any k  0, there is lk 2 Ik such that  2   lim !lkk Vlkk x k  x k  D 0 H) lim max PFix Ui x k  x k  D 0. k

k

i 2Ik

(5.56)

Note that the latter condition is weaker than the approximate regularity of fwk g1 kD0 with respect to the family fPFix Ui gi 2I , because Ik  I , consequently maxi 2Ik   PFix U x k  x k   maxi 2I PFix U x k  x k . i i Definition 5.8.8. (cf. [22, Definition 3.18]) We say that the sequence fIk g1 kD0  I is p-intermittent, where p 2 N, or intermittent if I D Ik [ IkC1 [ : : : [ IkCp1 for any k  0. A simple example of an intermittent sequence is an almost cyclic control (cf. Definition 5.6.10). 1 Theorem 5.8.9. Let fx k g1 kD0 be generated by (5.54), fIk gkD0  I be intermittent. Suppose that for any k  0 there is lk 2 Ik , k  0, satisfying condition (5.56). If wki D 0 for all i … Ik , k  0, then fwk g1 kD0 is approximately regular with respect to the family of cutters U WD fUi gi 2I .   Proof. Let wki D 0 for all i … Ik , k  0. Define ik WD argmaxi 2Ik Vik x k  x k ,  2   k  0. Suppose that limk !lkk Vlkk x k  x k  D 0. Denote Ci WD Fix Ui , i 2 I . By property (5.56), we have

  lim max PCi x k  x k  D 0. k

i 2Ik

(5.57)

Corollary 2.1.37, the assumption Fix Vik  Fix Ui D Ci for all i 2 I and k  0, and the definition of the metric projection yield   k    V y  y    y  y k P   kPCi y  yk , i Fix V i

y 2 H, consequently,       k k V x  x k  D max V k x k  x k   max PC x k  x k  . i ik i i 2Ik

i 2Ik

(5.58)

238

5 Projection Methods

Inequality (5.58) and equality (5.57) imply   lim Vikk x k  x k  D 0.

(5.59)

k

Let i 2 I . Since Ik is intermittent, there is p  1 such that i 2 IkCrk for some rk 2 f0; 1; : : : ; ; p  1g. By the triangle inequality, P the convexity of the norm, the definition of ikCl , l D 0; 1; : : : ; p1, the equality j 2IkCl !jkCl D 1 and by (5.59), we have k 1  kCr  rX  kClC1  x k  x k   x  x kCl 

lD0

D

    X   kCl kCl kCl kCl   kCl  !j .Vj x  x /  j 2IkCl

rX k 1 lD0



rX k 1

kCl

p1 X

kCl

D

X

   kCl kCl  !jkCl VikCl x  x  kCl

j 2IkCl

lD0 p1 X

    !jkCl VjkCl x kCl  x kCl 

j 2IkCl

lD0



X

   kCl kCl  kCl VikCl x  x !0 kCl

lD0

as k ! 1, i.e.,

  lim x kCrk  x k  D 0. k

(5.60)

Further, by the definition of the metric projection and by the triangle inequality,         PC x k  x k   PC x kCrk  x k   PC x kCrk  x kCrk  C x kCrk  x k  . i i i (5.61)   k  D 0. Now Since i 2 IkCrk , equality (5.57) yields limk PCi x kCrk  x kCr  PC x k  x k  D 0. Since Ui inequalities (5.61) and equality (5.60) imply lim k i   are cutters, Corollary 2.1.37 yields limk Ui x k  x k  D 0, i 2 I , i.e., condition (5.46) is satisfied, which means that fwk g1 t u kD0 is approximately regular. Let now Ik WD fi 2 I W !ik > 0g. 5.8.10. We say that a control sequence fwk g1 kD0 intermittent if the sequence fIk g1 kD0 , where Ik

Definition p  1, or p-intermittent, i.e., I D Ik [ IkC1 [ : : : [ IkCp1 .

(5.62) is p-intermittent, where is defined by (5.62), is

5.8 Simultaneous Cutter Methods

239

If jIk j D 1 or, equivalently, wk D eik , k  0, for a sequence fik g1 kD0  I , then 1 the fact that fwk g1 is intermittent means that fi g is an almost cyclic control k kD0 kD0 sequence. Suppose now that fx k g1 kD0 is generated by recurrence (5.54) having the following k 1 property: For any subsequence fx nk g1 kD0  fx gkD0 it holds   lim Vink x nk  x nk  D 0 H) lim kPFix Ui x nk  x nk k D 0, k

k

(5.63)

for all i 2 I (actually, it suffices to consider only i 2 Ik in (5.63)). Theorem 5.8.11. Let fx k g1 kD0 be generated by recurrence (5.54) having property k (5.63). If fwk g1 kD0 is intermittent and !i  ı for all i 2 Ik , k  0 and for some k 1 constant ı > 0, then fw gkD0 is approximately regular with respect to the family of cutters U WD fUi gi 2I . k Proof. Let fwk g1 kD0 be p-intermittent, where p  1, and ı > 0 be such that !i  ı for all i 2 Ik , k  0. Define

  ik D argmax !ik Vik x k  x k  ; i 2Ik

 2 k  0. Suppose that limk !ikk Vikk x k  x k  D 0. By equivalence (5.48), we have   lim !ikk Vikk x k  x k  D 0. k

(5.64)

Let i 2 I and rk 2 f0; 1; : : : ; ; p  1g be such that i 2 IkCrk . By the triangle inequality, the convexity of the norm, the definition of ikCl , l D 0; 1; : : : ; p  1, and by (5.64), we have k 1  kCr  rX  kClC1  x k  x k   x  x kCl 

lD0

D

rX k 1 lD0



rX k 1

   X    kCl kCl kCl kCl   kCl  !j Vj x x  j 2IkCl  kCl

lD0

m

p1 X lD0

X

    !jkCl VjkCl x kCl  x kCl 

j 2IkCl

   kCl kCl  kCl !ikCl  x kCl  ! 0 VikCl x kCl

240

5 Projection Methods

as k ! 1, consequently,   lim x kCrk  x k  D 0.

(5.65)

k

Further, by the definition of the metric projection and by the triangle inequality, we have     PC x k  x k   PC x kCrk  x k  i i      PCi x kCrk  x kCrk  C x kCrk  x k  ,

(5.66)

where Ci D Fix Ui , i 2 I . Since i 2 IkCrk , we have !ikCrk  ı. By the definition of ikCrk , we have         lim VikCrk x kCrk  x kCrk   ı 1 lim !ikCrk VikCrk x kCrk  x kCrk  k k   kCrk  k  kCrk kCrk  ı 1 lim !ikCr x  x V  D 0, i kCr kCr k

k

k

    i.e., limk VikCrk x kCrk  x kCrk  D 0. By condition (5.63),   lim PCi x kCrk  x kCrk  D 0: k

  Inequalities (5.66) and equality imply now limk PCi x k  x k  D 0. Since  (5.65)  Ui are cutters, we have limk Ui x k  x k  D 0 (see Corollary 2.1.37), i 2 I , i.e., condition (5.46) is satisfied, which means that fwk g1 t kD0 is approximately regular. u Remark 5.8.12. Since a sequence generated by (5.54) is bounded as a Fej´er monotone sequence, it has a weak cluster point y  . Therefore, property (5.63) and the demi-closedness of PFix Ui  Id at 0 yields that y  2 Fix Ui . This leads to the assumption that algorithm (5.54) is focusing (see [22, Definition 3.7], where this property was applied to firmly nonexpansive operators). Furthermore, property (5.63) applied to operators Vik being metric projections leads to the assumption that algorithm (5.54) is strongly focusing (see [22, Definition 4.8]). Consider now the following recurrence x kC1 D x k C k

X

!ik .Ui x k  x k /,

(5.67)

i 2Ik

where Ik WD fi 2 I W !ik > 0g, k  0, which is a special case of (5.54) with V k D U, k  0. Define Ki WD fk  0 W i 2 Ik g, i 2 I . Definition 5.8.13. We say that fIk g1 kD0 is repetitive if for any i 2 I the subset Ki is infinite. We say that a control sequence fwk g1 kD0 is repetitive if the sequence fIk g1 kD0 is repetitive.

5.8 Simultaneous Cutter Methods

241

An extended classification of control sequences for infinite I can be found in [119, Sect. 3]. If j Ik jD 1 or, equivalently, wk D eik , k  0, for a sequence fik g1 kD0  I , 1 then fwk g1 is repetitive if and only if fi g is a repetitive control in the sense k kD0 kD0 of Definition 5.6.11. Note that Theorems 5.8.9 and 5.8.11 hold for anyTsequence of relaxation parameters k 2 Œ0; 2 and without the assumption that i 2ITFix Ui ¤ ;. It turns out that a repetitive control is approximately semi-regular if i 2I Fix Ui ¤ ; and if we restrict the choice of k to an interval Œ"; 2  " for some " 2 .0; 1/. Theorem 5.8.14. Let Ui W H ! H, i 2 I , be cutters with a common fixed point and fx k g1 kD0 be generated by recurrence (5.67), where lim infk k .2  k / > 0 and !ik  ı for all i 2 Ik , k  0 and for some ı > 0. If the control sequence fwk g1 kD0 is repetitive, then fwk g1 is approximately semi-regular. kD0 Proof. By Theorem 4.8.2, we have X  2 2  2  kC1 x  z  x k  z  k .2  k / !ik Ui x k  x k  . i 2Ik

If we iterate the above inequality k-times, we obtain k X  2  2 X 2  kC1 x  z  x 0  z  l .2  l / !il Ui x l  x l  . lD0

Consequently,

1 X

k .2  k /

X

i 2Il

 2 !ik Ui x k  x k  < 1.

i 2Ik

kD0

Since the sum of an absolutely convergent series does not depend on the order of summands, we have m X X

 2 k .2  k /!ik Ui x k  x k 

i D1 k2Ki

D

1 X

k .2  k /

X

 2 !ik Ui x k  x k  < 1.

i 2Ik

kD0

Therefore,

X

 2 k .2  k /!ik Ui x k  x k  < 1,

k2Ki

i 2 I , and lim

k!1;k2Ki

 2 k .2  k /!ik Ui x k  x k  D 0,

242

5 Projection Methods

i 2 I . Since lim infk k .2  k / > 0, it holds lim infk!1;k2Ki k .2  k / > 0, consequently,  2 lim !ik Ui x k  x k  D 0, k!1;k2Ki

for all i 2 I . The assumption !ik  ı > 0 for all   i 2 Ik , k  0, yields now k k  limk!1;k2Ki Ui x  x D 0, i 2 I , i.e., lim infk Ui x k  x k  D 0 for all i 2 I which means that fwk g1 t u kD0 is approximately semi-regular.

5.8.2 Convergence Theorem Theorem 5.8.15. Suppose that: Ui W H ! H, i 2 I , are cutters with a common fixed point, Ui  Id are demi-closed at 0, i 2 I , k k k V k are T D fVi gi 2J T families of cutters Vi W H ! H, i 2 Jk , with the property k i 2Jk Fix Vi  i 2I Fix Ui , k  0, k 1 (d) fw gkD0 W H ! jJk j is a sequence of appropriate weight functions, (e) lim infk k .2  k / > 0, (f) fx k g1 kD0 is generated by recurrence (5.36). (a) (b) (c)

If the sequence of weight functions fwk g1 kD0 applied to the sequence of families V : k

(i) Is approximately regular with respect to the family U WD fUi gi 2I , then x k converges weakly to a common fixed point of Ui , i 2 I ; (ii) Is approximately semi-regular with respect to the family U WD fUi gi 2I and H is finite dimensional, then x k converges to a common fixed point of Ui , i 2 I . Proof. (cf. [70, Theorem 9.27]) Let V k W H ! H be defined by X

V k x WD

!ik .x/Vik x

i 2Jk

and Tk be the k -relaxation of the operator V k , i.e., Tk x WD Vkk x D x C k .V k x  x/.

(5.68)

The operators V k are cutters (see Corollary 2.1.49), consequently, Tk are strongly quasi-nonexpansive (see Theorem 2.1.39). By Theorem 2.1.26, we have Fix Tk D Fix V k D

\ i 2Jk

Fix Vik 

\ i 2I

Fix Ui ,

5.8 Simultaneous Cutter Methods

consequently,

1 \

243

Fix Tk 

\

Fix Ui .

i 2I

kD0

Let " > 0 be such that lim infk k > " and lim infk .2  k / > " and let z 2 T " i 2I Fix Ui . For a sufficiently large k we have 2  k  2 . Now, it follows from Theorem 4.8.2 that, for sufficiently large k, 2  2  kC1 x  z D Tk x k  z X  2  2  x k  z  k .2  k / !ik .x k / Vik x k  x k  i 2Jk

 2  2  x k  z  k .2  k / V k x k  x k   2 2  k   Tk x k  x k 2 D x k  z  k  k 2 "  2  x  z  Tk x k  x k  . 4     Therefore, x k  z decreases, consequently, V k x k  x k  ! 0, and, for an arbitrary ik 2 Jk ,  2 X k k  k k 2 !ikk .x k / Vikk x k  x k   !i .x / Vi x  x k  ! 0

(5.69)

i 2Jk

as k ! 1. (i) Suppose that fwk g1 kD0 is approximately regular with respect to the family U WD fUi gi 2I . Letik 2 Jk , k   0, be such that implication (5.46) holds. Then (5.69) yields limk Ui x k  x k  D 0 for all i 2 I . Therefore, condition (3.11) is satisfied for Tk WD V k and for S WD Ui , i 2 I . We have proved that all assumptions of Corollary 3.7.1 (ii) are satisfied for X D H, Tk WD V k and for S WD Ui , i 2 I . Therefore, x k converges weakly to a common fixed point of Ui , i 2 I . (ii) Suppose that H is finite dimensional and fwk g1 kD0 is approximately semiregular with respect to the family U WD fUi gi 2I . Let ik 2  Jk , k  0,  be such that implication (5.47) holds. Then (5.69) yields lim infk Ui x k  x k  D 0 for all i 2 I . Therefore, condition (3.12) is satisfied for Tk WD V k and for S WD Ui , i 2 I . We have proved that all assumptions of Theorem 3.7.1 (iii) are satisfied for X D H, Tk WD V k and for S WD Ui , i 2 I . Therefore, x k converges to a common fixed point of Ui , i 2 I . t u

244

5 Projection Methods

Remark 5.8.16. (i) We can also consider recurrence (5.36), where k depends on x 2 H, i.e., x0 2 H – arbitrary P x kC1 D x k C k .x k /. i 2Jk !ik .x k /Vik x k  x k /,

(5.70)

where k W H ! .0; 2 is a sequence of relaxation functions. It follows from the proof of Theorem 5.8.15 that the theorem remains true if we suppose that lim infk k .x/  " and lim supk k .x/  2  " for all x 2 H and for some " 2 .0; 1/. (ii) We can apply different relaxation parameters for different operators Vik , i 2 I , in recurrence (5.36) (we suppose, for simplicity, Jk D I for all k  0) which leads to the following recurrence – arbitrary x0 2 H P x kC1 D x k C i 2I ki .ik .x k /Vik x k  x k /,

(5.71)

k where fk g1 D kD0  .0; 2 is a sequence of relaxation vectors, i.e.,  k k k 1 k .1 ; : : : ; m /, and fv gkD0 is a sequence of weight functions v W H ! m . Note, however, that recurrence (5.71) can be reduced to (5.70) (or to (5.36) if we apply constant weight functions vk ) if we substitute the sequence of k 1 weight functions fvk g1 kD0 and the sequence of relaxation vectors f gkD0 to 1 k 1 the following sequences fw gkD0 W H ! m and fk gkD0 W H ! .0; 2, where X k .x/ D ki ik .x/ (5.72) i 2I

and !ik .x/ D ki ik .x/=k .x/;

(5.73)

i 2 I and x 2 H (cf. Remark 4.4.3). Note that this transformation maintains the assumptions on weight functions and on relaxation parameters presented before. In particular, if lim infk ki  " and lim supk ki  2  " for all i 2 I and for some " 2 .0; 1/, then lim infk k .x/  " and lim supk k .x/  2" for all x 2 H. Furthermore, if vk is an approximately regular sequence of weight functions and lim infk k .2  k / > 0, then wk is also approximately regular. Therefore, we apply recurrence (5.36) instead of (5.71) in Theorem 5.8.15 in order to simplify the notation. (iii) If we replace the assumption on the demi-closedness of Ui  Id at 0, i 2 I and the assumption on the approximate regularity of fwk g1 kD0 in Theorem 5.8.15 by the following one:   k k  k k k 2 D 0 H) all weak cluster points of fx k g1 lim kD0 lie in T k !ik .x / Vik x  x Fix U , i i 2I then the theorem remains true (cf. Remark 3.7.5).

5.8 Simultaneous Cutter Methods

245

5.8.3 Examples Example 5.8.17. (Iusem and De Pierro [215]) Consider recurrence (5.36), where Jk D I , k D 1, Vik D PCi , for all k  0, Ci  H are closed convex subsets, k i 2 I , the sequence fwk g1 kD0 of weight functions w W H ! m is constant, i.e., k w D w, and w has the form ( P i for i 2 I.x/ j 2I.x/ j !i .x/ WD 0 for i … I.x/, where v D T .1 ; 2 ; : : : ; m / 2 ri m and I.x/ WD fi 2 I W x … Ci g. Suppose that C WD i 2I Ci ¤ ;. Since wk is regular (see Example 5.8.3 (ii)), it follows from Theorem 5.8.15 that x k * x  2 C . The convergence was proved by Iusem and de Pierro [215, Corollary 4] for H D Rn . Iusem and de Pierro have also k considered the case C D ; and have P proved the local convergence of x to a fixed point of [ the operator T WD i 2I !i PCi with Fix T ¤ ; for a starting B.z; 12 mini 2I kPCi z  zk/, see [215, Theorem 3]). Note that point x 0 2 z2Fix T

this convergence can be proved simply as follows. If x 0 2 B.z; r/ for some z 2 Fix T , where r WD 12 mini 2I kPCi z  zk, then !i .x 0 / D i , i 2 I . The quasi nonexpansivity of T (see Corollary 4.4.4) yields that x 1 2 B.z; r/. One can prove by induction with respect to k that !i .x k / D i for all i 2 I and for all k. The convergence follows now from Corollary 5.4.1. Example 5.8.18. (Aharoni and Censor [3]) Consider recurrence (5.36), where , Jk D I , Vik D PCi for closed convex subsets Ci  Rn , i 2 I , with H D RnT C WDP i 2I Ci ¤ ;, k 2 Œ"; 2  " for some " 2 .0; 1/, k  0, wk 2 m k with 1 kD0 !i D C1, i 2 I . By Theorem 4.4.5, we have m X  kC1 2  2  2 x  z  x k  z  k .2  k / !ik PCi x k  x k  i D1

If we iterate this inequality k-times, we obtain k m X 2  2 X  2  kC1 x  z  x 0  z  l .2  l / !il PCi x l  x l  . lD0

i D1

Consequently, 1 m X X

 2 k .2  k /!ik PCi x k  x k 

i D1 kD0

D

1 X kD0

k .2  k /

m X i D1

 2 !ik PCi x k  x k  < C1

246

5 Projection Methods

and

1 X

 2 k .2  k /!ik PCi x k  x k  < C1

kD0

for any i 2 I . The assumption lim infk k .2  k / > 0 yields now 1 X

 2 !ik PCi x k  x k  < C1,

kD0

  i 2 I . Since kD0 !ik D C1, we have lim infk PCi x k  x k  D 0, i 2 I , i.e., wk is approximately semi-regular. Theorem 5.8.15 (ii) yields now the convergence x k ! x  2 C . (See [3, Theorem 1] for a different proof of convergence). P1

Example 5.8.19. (Butnariu and Censor [53]). Consider recurrence (5.36), where H D Rn , Jk D I , Vik D PCi , Ci  H are closed convex subsets, k  0, i 2 I , lim infk k > 0, lim supk k < 2, wk 2 T

m has a subsequence converging to a point w 2 ri m . Suppose that C WD i 2I Ci ¤ ;. Let " > 0 be such that k 1 !i > " for all i 2 I . Then there exists a subsequence fwnk g1 kD0  fw gkD0 nk " " nk 1 such that !i > 2 for all i 2 I and k  0, consequently, fw gkD0 is 2 -regular (see Example 5.8.3 (i)). Therefore, fwk g1 kD0 is approximately semi-regular (see Lemma 5.8.7 (ii)). Theorem 5.8.15 (ii) yields now x k ! x  2 C . If we suppose that all cluster points w of fwk g1 kD0 have the property !i  ı for all i 2 I and for some ı 2 .0; m1 , then fwk g1 kD0 is regular (see Example 5.8.3 (iv)) and the weak convergence holds in general Hilbert spaces H. Example 5.8.20. (Fl˚am and Zowe [168]) Consider recurrence (5.36), where Jk D k k I for all k, lim infk k > 0 and lim supk k < 2 and V Ti are cutters with Fix Vi  Ci for closed convex subsets Ci  H with C WD i 2I Ci ¤ ;, satisfying the inequality     k k V x  x k   ˛ PC x k  x k  , (5.74) i i i 2 I , k  0, for some constant ˛ > 0. Furthermore, suppose that the sequence of weights fwk g1 kD0 satisfies the following conditions: (i) lim supk !ik > 0, i 2 I , (ii) !ik .PCi x k  x k / ¤ 0 ) !ik > ı > 0. If we take Ui WD PCi , i 2 I , then it follows from (5.74) and from (i) to (ii) that the sequence of weights fwk g1 kD0 is regular. We see that all assumptions of Theorem 5.8.15 (i) are satisfied. Therefore, x k * x  2 C . The convergence was proved by Fl˚am and Zowe [168, Theorem 1] for H D Rn . Actually, Fl˚am and Zowe considered recurrence (5.71), but it can be reduced to (5.36), as it was explained in Remark 5.8.16 (ii). Example 5.8.21. Consider recurrence (5.55) with Vik WD PCi , i 2 I , k  0, for closed convex subsets Ci  H having a common fixed point, where k 2 Œ"; 2  " for some " 2 .0; 1/, Ik WD fi 2 I W !ik > 0g and !ik  ı for all i 2 Ik such that x k … Ci , k  0 (recurrences of this form with countable I were considered by Combettes

5.8 Simultaneous Cutter Methods

247

and Puh [125, Sect. 3]). Then condition   (5.56) is automatically satisfied (it ksuffices 1 to take lk WD argmaxi 2Ik PCi x k  x k ). If fIk g1 kD0 is intermittent, then fw gkD0 is approximately regularT (see Theorem 5.8.9) and it follows from Theorem 5.8.15 (i) that x k * x  2 C D i 2I Ci . Example 5.8.22. (Simultaneous subgradient projection method) Consider recurrence (5.36), where Jk D I for all k  0, lim infk k > 0 and lim supk k < 2 and Vik D Vi WD Pci is a subgradient projection relative to a continuous convex function ci W H ! R which is globally Lipschitz continuous on bounded subsets, i 2 I , k  0. Recall that Vi are cutters (see Corollary 4.2.6) and Vi  Id are demiclosedT at 0 (see Theorem 4.2.7). Denote Ci WD S.ci ; 0/, i 2 I , and suppose that C WD i 2I Ci ¤ ;. Let fwk g1 kD0 be a sequence of appropriate weight functions. (a) If fwk g1 kD0 is approximately regular, then it follows from Theorem 5.8.15 (i) that x k converges weakly to a point x  2 C . (b) If fwk g1 kD0 is approximately semi-regular and H is finite dimensional, then it follows from Theorem 5.8.15 (ii) that x k converges to a point x  2 C . Example 5.8.23. (Censor and Elfving [87]) Consider a consistent system of linear inequalities Ax  b and a method defined by the recurrence x kC1 D x k C k .Tk x k  x k /, where Tk W Rn ! Rn is given by Tk x WD x 

1 A> Mk .Ax  b/C , max .A> Mk A/

with Mk WD Dk MDk for a positive definite matrix M with nonnegative elements and a diagonal matrix Dk WD diag.d1k ; d2k ; : : : ; dmk / given by  dik

WD

1 if i 2 I k 0 otherwise,

and Ik WD fi 2 I W ai> x k  ˇi > 0g. Denote by Ak the submatrix of A with rows ai , i 2 Ik , by bk the subvector of b with coordinates ˇi , i 2 Ik , and by Rk the principal submatrix of M with rows and columns i 2 Ik . Further, denote C WD fx 2 Rn W Ax  bg, Ck WD fx 2 Rn W Ak x  bk g. Of course, C  Ck . Then Tk is the Landweber operator for the system Ak x  bk relative to the positive definite matrix Rk . Therefore, Fix Tk D Ck (see Lemma 4.6.2) and Tk is firmly nonexpansive (see Theorem 4.6.3), consequently, Tk is a cutter. Let U be an orthogonal matrix, G D diag.g1 ; g2 ; : : : ; gm / be such that gi > 0, i 2 I , and M D U > GU (the existence of U and G follows from Theorem 1.1.34 (iii)). Denote k Gk WD Dk GDk . It is clear that Gk D diag.g1k ; g2k ; : : : ; gm / where  gik WD

gi if i 2 I k 0 otherwise.

248

5 Projection Methods

We leave it to the reader to check that Mk D Dk U > GUDk D U > Gk U and

1

(5.75)

1

Mk2 D U > Gk2 U:

(5.76)

Denote y.x/ D .1 .x/; 2 .x/; : : : ; m .x//> D UAx. By Theorem 1.1.27, we have    12 2  max .A Mk A/ D Mk A  D sup >

kxkD1

   12 2 M Ax  D sup  k 

kxkD1

 2  > 12  U G UAx  k  

 2 X  12   D sup G D sup  y.x/ gik .i .x//2  k  kxkD1

kxkD1 i 2I

X

 sup

kxkD1 i 2I

 1 2   gi .i .x//2 D sup G 2 y.x/ kxkD1

2 2  1  1     D sup G 2 UAx  D sup U > G 2 UAx  kxkD1

kxkD1

 1 2  1 2     D sup M 2 Ax  D M 2 A D max .A> MA/. kxkD1

Let S W Rn ! Rn be the Landweber operator for the system Ax  b related to the matrix M , i.e., S has the form S x WD x 

1 A> M.Ax k  b/C . max .A> MA/

Note that Mk .Ax k b/C D M.Ax k b/C . Because the system Ax  b is equivalent 1 1 1 to M 2 Ax  M 2 b, Lemma 4.6.2 yields that Fix S D fx 2 Rn W M 2 Ax  1 M 2 bg D C . Therefore,  >  1 A Mk .Ax k  b/C  max .A> Mk A/  >  1 A Mk .Ax k  b/C   > max .A MA/   > 1 A M.Ax k  b/C  D max .A> MA/   D S x k  x k  ,

   Tk x k  x k  D

i.e., condition (3.11) is satisfied and C D Fix S  Fix Tk . We see that all assumptions of Corollary 3.7.1 (iii) are satisfied. Therefore, x k !  x 2 Fix S D fx 2 Rn W Ax  bg.

5.8 Simultaneous Cutter Methods

249

5.8.4 Block Iterative Projection Methods Consider the convex feasibility problem: find a common point of a finite family of nonempty closed convex subsets Ci  H, i 2 I . If m is a large number, it is convenient to split the set I into smaller blocks and to perform the operations of the projection method in two stages: first on blocks and then on the family of blocks. Let I D L1 [ L2 [ : : : [ Lr , where Lj are nonempty subsets which are pairwise disjoint, j 2 J D f1; 2; : : : ; rg. Then each i 2 I can be transformed in a unique way to a pair .j; l/ such that j 2 J and l 2 Lj . The subset Ci canT now beTpresented as Cj l and the convex feasibility problem obtains the form: x  2 j 2J l2Lj Cj l . Consider the following two stage recurrence x kC1 D Ujkk x k ,

(5.77)

where jk 2 J is a control for the outer loop and Ujkk is a relaxed simultaneous projection which “employs” only projections PCjk ;l for l 2 Ljk , i.e., Ujkk has the form X Ujkk WD Id Ck . !ik PCjk ;l  Id/, (5.78) l2Ljk

where k 2 Œ0; 2 and wk W H ! jLj j is a sequence of weight functions applied in k the inner loop. The sequence f.jk ; wk /g1 kD0 defines a two stage control. In particular, the following operator Ujkk can be used in the inner loop of a two stage recurrence (5.77): Ujkk WD PCjk ;lk (5.79) where lk 2 Ljk is a control sequence for the inner loop. Note that PCjk ;lk is a special case of the operator Ujkk defined by (5.78), where k D 1 and wk D elk . We can identify the weight function wk W H ! jLj j with a weight function k

vk W H ! m defined by  ik

WD

!ik if i 2 Ljk 0 otherwise.

(5.80)

5.8.4.1 Double Layer Control Sequence Consider the two stage recurrence (5.77), where the control sequence: fjk g1 kD0  J applied in the outer stage is an almost cyclic control, i.e., J  fjk ; jkC1 ; : : : ; jkCp1 g

(5.81)

250

5 Projection Methods

for some p  r and, for all k, Ujkk is defined by (5.78), where the weight function wk W H ! jLj j is ˛-regular with respect to the family fPCjk ;l gl2Ljk , where ˛ 2 k .0; 1, i.e., there exists lk 2 Ljk such that  2  2     !lkk PCjk ;lk x k  x k   ˛ max PCjk ;l x k  x k  , l2Ljk

(5.82)

k  0. Examples of such weight functions were presented in Sect. 5.8.1. We call the sequence f.jk ; wk /g1 kD0 defined above a double layer control. A special case of this control is the following: ˛ D 1, k D 1, wk D elk 2 ext jLj j and lk 2 Ljk is a k

remotest set control, i.e., lk D lk .x k / 2 Ljk is such that         PCjk ;lk x k  x k  D max PCjk ;l x k  x k  . l2Ljk

(5.83)

The double layer control f.jk ; wk /g1 kD0 can be identified with a sequence of weight functions fvk g1 kD0 defined by (5.80). Corollary 5.8.24. The double layer control is approximately regular with respect to the family U WD fPCi gi 2I . Proof. Recurrence (5.77), where Ujkk is defined by (5.78), is a special case of (5.55), where Ik D Ljk , V k D U WD fPCi gi 2I . Furthermore, (5.82) yields condition (5.56). 1 Since fjk g1 kD0 is almost cyclic, fIk gkD0 is intermittent. The claim follows now from Theorem 5.8.9. t u Theorem 5.8.25. Let C ¤ ; and x k be generated by recurrence (5.77) with a double layer control and with a sequence of relaxation parameters fk g1 kD0 such that lim inf k .2  k / > 0. Then for an arbitrary x 0 2 H the sequence fx k g1 kD0 converges weakly to a point x  2 C . Proof. It follows from Corollary 5.8.24 that the double layer control is approximately regular. The claim follows now from Theorem 5.8.15 (i). t u Several block iterative projection methods are presented in [3, 53, 81, 229], [118, Chap. 5], [108, Sect. 5.6], [5, 59, 159].

5.9 Sequential Cutter Methods In this section we consider the common fixed point problem for a finite family of cutters Ui W H ! H, i 2 I WD f1; 2; : : : ; mg, having a common fixed point, and sequences generated by the following recurrence x0 D x 2 H – arbitrary x kC1 D x k C k .Uik x k  x k /

(5.84)

5.9 Sequential Cutter Methods

251

1 where fk g1 kD0  .0; 2 is a sequence of relaxation parameters and fik gkD0  I is a control sequence.

Definition 5.9.1. Let fx k g1 kD0 be generated by recurrence (5.84). A control sequence fik g1 W H ! I is called: kD0 (i) Approximately regular if the following implication holds     lim Uik x k  x k  D 0 H) lim Ui x k  x k  D 0 for all i 2 I , k

k

(5.85)

(ii) Approximately semi-regular if the following implication holds     lim Uik x k  x k  D 0 H) lim inf Ui x k  x k  D 0 for all i 2 I . k

k

Note that recurrence (5.84) is a special case of (5.36), where wk D eik , k  0. Therefore, a control sequence fik g1 kD0 is approximately (semi-)regular means that feik g1 is an approximately (semi-)regular sequence of weight functions with kD0 respect to the family U WD fUi gi 2I . If Ui are metric projections, then an approximately (semi-)regular control coincides with an approximately (semi-)remotest set control (cf. Definitions 5.6.1 and 5.6.2). If, furthermore, fik g1 kD0 is an almost remotest set control (see Definition 5.6.6), then the sequence feik g1 kD0 is a regular sequence of weight functions (see Example 5.8.3 (v)).

5.9.1 Convergence Theorem If we take V k D U WD fUi gi 2I and wk WD eik , k  0, in Theorem 5.8.15, then we obtain the following result. Corollary 5.9.2. Let Ui W H ! H, be cutters with a common fixed point and such that Ui  Id are demi-closed at 0, i 2 I , and the sequence fx k g1 kD0 be generated by recurrence (5.84), where lim inf k .2  k / > 0. If the control sequence fik g1 kD0 is: (i) Approximately regular, then x k converges weakly to a common fixed point of Ui , i 2 I . (ii) Approximately semi-regular and H is finite dimensional, then x k converges to a common fixed point of Ui , i 2 I .

5.9.2 Control Sequences for Sequential Cutter Methods Corollary 5.9.2 shows the importance of approximate (semi-)regularity of the control sequence fik g1 kD0 for the convergence of the sequences generated by the

252

5 Projection Methods

sequential cutter method. In this section we present some control sequences which are approximately (semi-)regular. The results presented in this section are special cases of Theorems 5.8.11 and 5.8.14, where V k D U D fUi gi 2I and wk D eik . Corollary 5.9.3. Let Ui W H ! H be cutters having the following property: for any bounded sequence fy k g1 kD0  H     lim Ui y k  y k  D 0 H) lim PFix Ui y k  y k  D 0, k

k

(5.86)

i 2 I . Let fx k g1 kD0 be generated by recurrence (5.84). If the control sequence fik g1 is almost cyclic, then fik g1 kD0 kD0 is approximately regular. A family of operators fUi gi 2I has property (5.86) if, e.g., kUi y  yk  ı kPFix Ui y  yk

(5.87)

for a constant ı > 0, and for arbitrary i 2 I and y 2 H. Of course, the latter property is satisfied for operators Ui being strict relaxations of metric projections PCi onto nonempty closed convex subsets Ci , i 2 I . Therefore, Corollary 5.9.3 yields in particular that an almost cyclic control sequence applied to a successive projection method (5.28), where lim infk k .2  k / > 0, is an approximately remotest set control sequence (see Lemma 5.6.12 (i)). Corollary 5.9.4. Let Ui W H ! H, i 2 I be cutters with a common fixed point and fx k g1 kD0 be generated by recurrence (5.84), where lim infk k .2  k / > 0. If the 1 control sequence fik g1 kD0 is repetitive, then fik gkD0 is approximately semi-regular.

5.9.3 Examples Example 5.9.5. (Aharoni, Berman and Censor [2]) Consider method (5.84), where H is finite dimensional and the cutters Ui satisfy the demi-closedness principle, i 2 I , (in [2] it was supposed that Ui have property (5.87), i 2 I ), k 2 Œ; 2 for some  2 .0; 1/ and the control fik g1 kD0 is repetitive. By Corollary 5.9.4 the control fik g1 is approximately semi-regular. Note that for a cutter Ui with property (5.87), kD0 the operator Ui  Id is closed at 0, because, by the nonexpansivity of the metric projection, the operator PFix Ui  Id is closed at 0, i 2 I (seeTLemma 3.2.5). Therefore, Corollary 5.9.2 (ii) yields the convergence x k ! x  2 i 2I Fix Ui . The convergence was proved by Aharoni et al. [2, Theorem 1] (see also [108, Sect. 5.5]), where the method was called the .ı; /-algorithm. An interesting convergence result for method (5.84) with k D 1 and Ui W Rn ! Rn being paracontractions (continuous strictly quasi-nonexpansive operators) and with repetitive control was obtained by Elsner et al. [164, Theorem 1] (see also [163, Theorem 1] for linear case). This result is not covered by Corollaries 5.9.2 (ii) and 5.9.4. Conversely, the

5.10 Extrapolated Simultaneous Cutter Methods

253

convergence of sequences in Rn generated by (5.84) with repetitive control does not follow from the [164, Theorem 1], because the cutters are not supposed to be continuous. Example 5.9.6. Consider method (5.84), where lim infk k .2  k / > 0 and Ui , i 2 I , are firmly nonexpansive operators with a common fixed point. Then Ui  Id are demi-closed at 0, i 2 I . 1 (i) If Ui have property (5.86), i 2 I , and the control sequence fi Tk gkD0 is almost k  cyclic, then Corollaries 5.9.2 (i) and 5.9.3 yield x * x 2 i 2I Fix Ui (see also [331, Theorem 2] and [6], where a quasi-cyclic order is considered, which is slightly more general than the almost cyclic control). (ii) If H is finite dimensional and the control sequenceT fik g1 kD0 is repetitive, then k  Corollaries 5.9.2 (ii) and 5.9.4 yield x ! x 2 i 2I Fix Ui (see also [22, Example 3.14], [331, Theorem 1] and [6, Theorem 3.1] for related results).

Example 5.9.7. (Sequential subgradient projection method) Consider method (5.84), where lim infk k .2  k / > 0 and Ui , i 2 I , are subgradient projections, i.e., Ui D Pci , where ci W H ! R are continuous convex functions, i 2 I , which n are globally Lipschitz continuous on bounded subsets (this T holds if, e.g., H D R ). Denote Ci WD S.ci ; 0/, i 2 I , and suppose that C WD i 2I Ci ¤ ;. Since Ui are cutters (see Corollary 4.2.6) and Ui  Id are demi-closed at 0 (see Theorem 4.2.7), it follows from Corollary 5.9.2 (i) that x k converges weakly to a point x  2 C if the 1 control sequence fik g1 kD0 is approximately regular (e.g., if fik gkD0 is almost cyclic). k Similarly, it follows from Corollary 5.9.2 (ii) that x converges to a point x  2 C , if H is finite dimensional and the control sequence fik g1 kD0 is approximately semiregular (e.g., if fik g1 is repetitive). A special case of the sequential subgradient kD0 projection method was presented in [102], where H D Rn and the control is almost cyclic. In [136] a finitely convergent modification of the method of [102] was presented. In [80] the method of [102] was applied for solving interval linear inequalities. For a survey on subgradient projection methods, see [79].

5.10 Extrapolated Simultaneous Cutter Methods We consider the common fixed point problem T for a finite family of cutters Ui W H ! H, i 2 I WD f1; 2; : : : ; mg, with i 2I Fix Ui ¤ ;. In this section we study convergence of sequences which are generated by generalized relaxations of simultaneous cutters of the form T x WD V; x D x C .x/.V x  x/,

(5.88)

where  2 Œ0; 2,  W H ! .0; P1/ is a step size function, V W H ! H is a simultaneous cutter, i.e., V x WD i 2J i .x/Vi x, V WD fVi gi 2J is a finite family of T T cutters, i 2J Fix Vi  i 2I Fix Ui , and v W H ! jJ j is a weight function with

254

5 Projection Methods

v.x/ D .1 .x/; : : : ; jJ j .x// 2 jJ j . In particular, we can take V D U WD fUi gi 2I . In this case T x D U; x D x C .x/.Ux  x/, P where Ux WD i 2I !i .x/Ui x and w W H ! m . A sequence of operators Tk of the form (5.88) describes, for any x 0 2 H, the recurrence x kC1 D Tk x k , i.e., x kC1 D x k C k k .x k /.

X

ik .x k /Vik x k  x k /,

(5.89)

i 2Jk

where k 2 Œ0; 2, fk g1 kD0 W H ! .0; 1/ is a sequence of step size functions, fvk g1 W H !

jJk j is a sequence of weight functions, i.e., vk .x/ D kD0 k k .1 .x/; : : : ; jJk j .x// 2 jJk j and V k WD fVik gi 2Jk , k  0, is a sequence of families T T of cutters with i 2Jk Fix Vik  i 2I Fix Ui for all k  0. The iterative method described by (5.89) is called an extrapolated simultaneous cutter method (ESCM). If we take, in particular, V k WD U for all k  0, we obtain the following recurrence x kC1 D x k C k k .x k /.

k X

ik .x k /Ui x k  x k /,

(5.90)

i 2I k where fvk g1 kD0 W H ! m is a sequence of weight functions and v .x/ D k k .1 .x/; : : : ; m .x//, x 2 H. We can appropriately apply Definition 5.8.5 and Remark 5.8.6 also for sequences generated by recurrence (5.89). If Ui are projections, then the methods defined by recurrence (5.90) are known in the literature as extrapolated simultaneous projection methods or extrapolated methods of parallel projections. Various variants of extrapolated simultaneous cutter (in particular projection) methods were studied in [146, 284, 351], [62, Chaps. 4 and 5], [175, 229, 247], [17, Chap. 8], [118, Chap. 5], [89, 90, 98, 99, 119, 120, 176, 230– 233, 304, 305], [10, Theorem 1 and Corollary 4.4], [67–70, 92, 190].

5.10.1 Assumptions on Step Sizes Let V WD fVi gi 2J be a finite family of cutters with a common fixed point and v W H ! jJ j be an appropriate weight function. Define a step size function v W H ! R by the equality v .x/ WD

8 <

P

i 2J

P

k :1

i .x/kVi xxk2

i 2J

i .x/Vi xx k

2

if x … if x 2

T i 2J

Fix Vi ,

i 2J

Fix Vi .

T

Since cutters are strictly 2.1.39), it follows from P quasi-nonexpansive (see Theorem T Theorem 2.1.26 that i 2J i .x/Vi x ¤ x for x … i 2J Fix Vi , consequently, v is

5.10 Extrapolated Simultaneous Cutter Methods

255

well defined. Furthermore, it follows from the convexity of the function kk2 that v .x/  1 for all x 2 H. Definition 5.10.1. Let Vi W H ! H, i 2 J , be cutters with a common fixed point and v W H ! jJ j be a weight function which is appropriate with respect to the family V WD fVi gi 2J . We say that a step size function  W H ! .0; C1/ is ˛-admissible with respect to the family fVi gi 2J , where ˛ 2 .0; 1, or, shortly, admissible, if ˛v .x/  .x/  v .x/ (5.91) T for all x … i 2J Fix Vi .

5.10.2 Convergence Theorem Theorem 5.10.2. Suppose that: (a) Ui W H ! H, i 2 I , are cutters with a common fixed point, (b) Ui  Id, i 2 I , are demi-closed at 0, (c) V k WD fVik gi 2Jk are families of cutters Vik W H ! H, i 2 Jk , with the properties \ \ Fix Vik  Fix Ui (5.92) i 2Jk

and

i 2I

  max Vik x  x   max kUi x  xk i 2Jk

i 2I

(5.93)

for all x 2 H, k  0, and for some constant > 0. (d) fvk g1 kD0 W H ! jJk j is a sequence of appropriate weight functions, (e) The step size k W H ! .0; C1/ is ˛-admissible with respect to V k , k  0, for some ˛ 2 .0; 1, (f) lim infk k .2  k / > 0, (g) fx k g1 kD0 is generated by recurrence (5.89). Then: (i) If the sequence of weight functions fvk g1 kD0 applied to the sequence of families V k is regular with respect to the family U WD fUi gi 2I , then x k converges weakly to a common fixed point of Ui , i 2 I . (ii) If the sequence of weight functions fvk g1 kD0 applied to the sequence of families V k contains a subsequence of weight functions which is regular with respect to the family U WD fUi gi 2I and H is finite dimensional, then x k converges to a common fixed point of Ui , i 2 I . Proof. (cf. [70, Theorem 9.35]) Let V k W H ! H be defined by V k x WD

X i 2Jk

ik .x/Vik x

256

5 Projection Methods

and Tk be a generalized relaxation of V k , i.e., Tk x WD Vkk ;k x D x C k k .x/.V k x  x/.

(5.94)

The operators V k areTcutters (see Corollary 2.1.49). By Theorem 2.1.26, we have Fix Tk D Fix V k D i 2Jk Fix Vik (note that a cutter is strictly quasi-nonexpansive T T by Theorem 2.1.39). By (5.92), 1 i 2I Fix Ui . Let " 2 .0; 1/ and kD0 Fix Tk  k0  0 be such that k 2 Œ"; 2  " for all k  k0 . By Theorem 4.9.1, the operator Vkk is a cutter, consequently Tk is a k -relaxed cutter. Theorem 2.1.39 implies  kC1 2  2 2  k   x Tk x k  x k 2  z  x k  z  k   T for all z 2 m all k  k0 . Consequently, fx k g is bounded, x k  z is i D1  Fix kUi and  monotone and Tk x  x k  ! 0. (i) Let ˇ > 0, k1  k0 and jk 2 f1; 2; : : : ; Jk g be such that  2   jkk .x/ Vjkk x  x   ˇ max kUi x  xk2 i 2I

for any x 2 H  and k k1 . Since k is ˛-admissible, the norm kk is a convex   function and Vjk x  x   maxi 2I kUi x  xk for all j 2 Jk , we have     Tk x k  x k  D k k .x k / V k x k  x k    P k k  k k k 2   i 2Jk i .x / Vi x  x V k x k  x k   k ˛ P  2 k k k k k  i 2Jk i .x /Vi x  x   P k k  k k k 2 i 2Jk i .x / Vi x  x  D k ˛ P   k .x k /V k x k  x k  i 2Jk

i

i

2    jkk .x k / Vjkk x k  x k     k ˛ P k k  k k k i 2Jk i .x / Vi x  x  2 ˇ maxi 2I Ui x k  x k   .  k ˛ P . i 2Jk ik .x k // maxi 2I Ui x k  x k  D

  "˛ˇ max Ui x k  x k  i 2I

for all k  k0 . Consequently,

5.10 Extrapolated Simultaneous Cutter Methods

257

    Tk x k  x k   "˛ˇ max Ui x k  x k  i 2I

(5.95)

  and Ui x k  x k  ! 0 for all i 2 I . Therefore, condition (3.6) is satisfied for Uk D Tk and S D Ui , i 2 I . We have proved that all assumptions of Theorem 3.6.2 (i) are satisfied for S D Ui , i 2 I . Therefore, x k converges weakly to a common fixed point of Ui , i 2 I . (ii) Suppose that H is finite dimensional and fvk g1 kD0 contains a ˇ-regular subsequence fvnk g1 . Let ˇ > 0, k  k and j 2 I be such that 1 0 nk kD0  2   jnnk .x/ Vjnnk x  x   ˇ max kUi x  xk2 . k

k

i 2I

Similarly as in (i) one proves that kTnk x nk  x nk k 

"˛ˇ max kUi x nk  x nk k . i 2I

  Consequently, lim infk Ui x k  x k  D 0 for all i 2 I . If we take Uk D Tk and S D Ui , i 2 I , in Theorem 3.6.2 (ii), we obtain the convergence of x k to a fixed point of Ui for all i 2 I . t u The operators Tk defined by (5.94) do not need to be continuous, because we have not assumed that the operators Vik as well as the weight functions ik are continuous. This allows application of operators from an essentially broader family than the nonexpansive ones. We have supposed in Theorem 5.10.2 that fvk g1 kD0 is regular (in (i)) or that fvk g1 contains a regular subsequence (in (ii)). These assumptions kD0 seem not to be too restrictive. The assumption on the step sizes guarantees that Tk is a relaxed cutter. All these assumptions allow using algorithms which can treat the violated and the nonviolated constraints in a more flexible way and provide long steps. Remark 5.10.3. In [119, Sect. 1] and [120, Sects. IV and V] Combettes presented three methods which are quite similar to ESCM: two extrapolated methods of parallel projections (EMOPP and EPPM2) and an extrapolated method of parallel subgradient projection (EMOPSP): (i) In EMOPP (see [119, Definition 1.2]) it is supposed that Jk D I , Vjk D PC k , i where PC k is the metric projection onto a closed convex superset Cik of Ci i with d.x k ; Cik /  d.x k ; Ci /, (5.96) for all i 2 I , k  0, and for some constant  > 0, and the weights vk 2 m satisfy condition (5.42), k  0. (ii) In EPPM2 (see [120, Sect. IV.C]) it is supposed that Jk D I , Vjk D PC k , where i PC k is the metric projection onto a closed and convex superset Cik of Ci , i 2 I , i

258

5 Projection Methods

k  0, and v 2 m . The supersets Cik of Ci are chosen in such a way that the k 1 following condition is satisfied: for any subsequence fx nk g1 kD0  fx gkD0 and for all i 2 I it holds     .x nk * x and lim PC nk x nk  x nk  D 0/ H) x 2 Ci . (5.97) k

i

(iii) In EMOPSP (see [120, Sect. V]) it is supposed that Jk  I , Jk ¤ ;, Vik D Pfi are subgradient projections relative to a (lower semi-)continuous convex function fi W H ! R, ik  ı > 0 for xk … Ci ; i 2 Jk :

(5.98)

The above methods EMOPP and EMOPSP are covered by ESCM, where Vjk are cutters, j 2 Jk , k  0, satisfying (5.92) and (5.93) and fvk g1 kD0 is a sequence of appropriate weight functions. Note that (5.96) with ik satisfying ik  ı > 0 for xk … Ci , i 2 Jk , implies that fvk g1 kD0 is a regular sequence of weight functions (see Example 5.8.3 (iv) and Definition 5.8.4). In EPPM2 condition (5.97) is slightly weaker (only in an infinite dimensional case) than the approximate regularity of vk satisfying (5.98), but Theorem 5.10.2 enables application of more general sequences of weight functions than the ones satisfying (5.98). Furthermore, Theorem 5.10.2 k does T not krequire Ci T Ci , i 2 Jk , k  0. The theorem requires only the inclusion i 2Jk Ci  C WD i 2I Ci to be satisfied, k  0.

5.10.3 Extrapolated Simultaneous Subgradient Projection Method Let ci W H ! R be convex functions which are globally Lipschitz continuous on T bounded subsets, Ci WD fx 2 H W ci .x/  0g, i 2 I and C WD i 2I Ci ¤ ;. The extrapolated simultaneous subgradient projection method (ESSPM) is a special case of an ESCM defined by recurrence (5.90), where Ui are subgradient projections, i.e., Ui are defined by (4.115), i 2 I . By Corollary 4.2.6, Lemma 4.2.5 and by Theorem 4.2.7, Ui are cutters, Fix Ui D Ci and Ui  Id are demi-closed at 0, i 2 I . Let vk W H ! m be an appropriate weight function and a step size function k W H ! .0; C1/ be defined by P i 2I

.c .x//

ik .x/. kgi i .x/kC /2

k .x/ WD  2 ,  P .c .x//  i 2I ik .x/ kgi .x/kC2 gi .x/ i

k  0 (for simplicity, we use the convention that ESSPM is defined by

.ci .x//C kgi .x/k

D 0 if .ci .x//C D 0). The

5.11 Extrapolated Cyclic Cutter Method

x kC1 D x k  k k .x k /

259

X i 2I

.ci .x k //C k ik .x k /   gi .x /, gi .x k /2

(5.99)

where x 0 2 H is arbitrary and k 2 .0; 2/, k  0. We can also write x kC1 D Ukk ;k x k , where Ukk ;k WD Id Ck k .U k  Id/ is a generalized relaxation of the simultaneous subgradient projection U k WD

X

ik Pci ,

i 2I

k  0. Theorem 5.10.4. Let x 0 2 H be arbitrary and x k be generated by recurrence (5.99), where lim infk k .2  k / > 0. k (i) If fvk g1 kD0 is regular, then x converges weakly to a common fixed point of Ui , i 2 I. (ii) If fvk g1 kD0 contains a subsequence of weight functions which is regular and H is finite dimensional, then x k converges to a common fixed point of Ui , i 2 I .

Proof. The step sizes k are 1-admissible. Therefore, the theorem follows from Theorem 5.10.2. t u The ESSPM was studied by Dos Santos in [146], where positive constant weights w 2 ri m were considered and the convergence in the finite dimensional case was proved [146, Sect. 5]. Furthermore, the method was studied by Combettes in [120, Sect. V] (cf. Remark 5.10.3 (iii)).

5.11 Extrapolated Cyclic Cutter Method Let Ui W H ! H be cutters with a common fixed point, U D Um Um1 : : : U1 be a cyclic cutter and x 0 2 H. Consider sequences fx k g1 kD0 generated by the recurrence x kC1 D x k C k k .x k /.Ux k  x k /,

(5.100)

where k 2 .0; 2, the step size functions k W H ! .0; C1/ satisfy the inequality ˛  k .x/  max .x/

(5.101)

260

5 Projection Methods

1 for all x 2 H with ˛ 2 .0; 2m ,

P max .x/ WD

i 2I hUx

 Si 1 x; Si x  Si 1 xi kUx  xk2

,

where Si WD Ui Ui 1 : : : U1 for i D 1; 2; : : : ; m and S0 WD Id (cf. (4.121)). The existence of the step size k .x/ follows from inequality (4.122). Since P m

i 2I

kSi x  Si 1 xk2 kUx  xk2

1

(see again (4.122)) we can use in particular step size functions k satisfying the inequality P kSi x  Si 1 xk2 m˛ i 2I  k .x/  max .x/ (5.102) kUx  xk2 for all x 2 H. We can also write recurrence (5.100) in the form x kC1 D Uk ;k x k , where Uk ;k WD Id Ck k .U  Id/ is a generalized relaxation of the cyclic cutter U with the step size function k satisfying inequalities (5.101).

5.11.1 Convergence In [71] one can find a slightly weaker version of the following theorem. Theorem 5.11.1. Let fx k g1 kD0 be generated by (5.100), where k .x/ satisfies (5.101) and let z 2 Fix U . Then the following inequality holds  kC1 2  2  2 x  z  x k  z  k .2  k /˛ 2 Ux k  x k  .

(5.103)

Moreover: (i) If U  Id is demi-closed at 0 and lim infk k .2  k / > 0, then x k * x  2 Fix U . (ii) If the step size function k satisfies inequalities (5.102), then  Pm  k k 2 2  / i D1 Si x  Si 1 x .  2 Ux k  x k  (5.104) If, furthermore, Ui  Id are demi-closed at 0, i D 1; 2; : : : ; m, and lim infk k .2  k / > 0, then x k * x  2 Fix U . 2  2  kC1 . x  z  x k  z  k .2  k /m2 ˛ 2

Proof. Since a T cutter is strictly quasi-nonexpansive (see Theorem 2.1.39), we have Fix U D i 2I Fix Ui . Note that Uk ;k x k  x k D k k .x k /.Ux k  x k /.

5.11 Extrapolated Cyclic Cutter Method

261

By Theorem 4.10.2, the operator Uk is a cutter and 2  2  2 2  k    kC1 U ; x k  x k 2 x  z D U; x k  z  x k  z  k k k  k 2  2 D x  z  k .2  k /k2 .x k / Ux k  x k   2  2  x k  z  k .2  k /˛ 2 Ux k  x k  .   Therefore, x k  z is decreasing, x k is bounded and   k Ux  x k  ! 0.

(5.105)

nk 1 Let x  2 H be a weak cluster point of the sequence fx k g1 kD0 and fx gkD0  k 1  fx gkD0 be a subsequence which converges weakly to x .

(i) Suppose that U  Id is demi-closed at 0. Condition (5.105) yields x  2 Fix U .  The weak convergence of the whole sequence fx k g1 kD0 to x follows now from Lemma 3.3.3. (ii) Suppose that the step size function k satisfies inequality (5.102). Then inequality (5.104) can be proved similarly to (5.103). Consequently,   Si x k  Si 1 x k  ! 0

(5.106)

for i D 1; 2; : : : ; m, because lim infk k .2  k / > 0 and Ux k  x k is bounded. Suppose that Ui  Id are demi-closed at 0, i D 1; 2; : : : ; m. Condition (5.106) for i D 1 yields kU1 x nk  x nk k D kS1 x nk  S0 x nk k ! 0. Due to the demi-closedness of U1  Id at 0, we have U1 x  D x  . Since k.U1 x nk  U1 x  /  .x nk  x  /k D kU1 x nk  x nk k ! 0 and x nk * x  , we have U1 x nk * U1 x  D x  . By (5.106) for i D 2, kU2 .U1 x nk /  U1 x nk k D kS2 x nk  S1 x nk k ! 0. Since U2  Id is demi-closed at 0 and U1 x nk * x  , U2 x  D x  . In a similar way we obtain that Ui x  D x  for i D 3; : : : ; m. Therefore, Ux  D Um : : : U1 x  D x  . We conclude that the subsequence fx nk g1 kD0 converges weakly to a fixed point of the operator U . The weak convergence of the whole sequence fx k g1 kD0 to x  2 Fix U follows now from Lemma 3.3.3. t u

262

5 Projection Methods

Even if we take  WD max and  D 2, the generalized relaxation U; needs not to be an extrapolation of U because max needs not to be greater or equal to 12 . 1 We only know that max  2m (see (4.122)). Note, however, that U is m1 -SQNE (see 2m Corollary 4.5.15), consequently, it is a mC1 -relaxed cutter (see Corollary 2.1.43) and U 1Cm is a cutter (see Remark 2.1.3). Since Umax is a cutter (see Theorem 4.10.2), 2m

; max g. Lemma 2.4.7 (i) yields that U is also a cutter, where  D maxf 1Cm 2m 2m Therefore, U; with  2 . 1Cm ; 2 is an extrapolation of U . Interesting schemes for accelerated cyclic projections for subspaces of H can be found in [177] and in [141].

5.11.2 Accelerated Kaczmarz Method for a System of Linear Equations Consider a consistent system of linear equations hai ; xi D ˇi ; i 2 I ,

(5.107)

where ai 2 H, ai ¤ 0 and ˇi 2 R, i 2 I . Suppose for simplicity that WD H.ai ; ˇi / D fx 2 H W hai ; xi D ˇi g, i 2 I . By kai k D 1. Let Ci T assumption, C WD i 2I Ci ¤ ;. Let Ui WD PCi , i 2 I . We have Ui x D x  .hai ; xi  ˇi /ai , i 2 I (see (4.1)). Of course, Fix Ui D Ci and Ui a cutter as the metric projection (see Theorem 2.2.21 (ii)), i 2 I . Since the metric projection is strictly (see Corollary 2.2.24), Theorem 2.1.26 T quasi-nonexpansive T yields Fix U D i 2I Fix Ui D i 2I Ci D C . We apply method (5.100) with k WD max and k D 1 for all k  0 to system (5.107). By inequality (5.103), any sequence generated by (5.100) is Fej´er monotone with respect to C . By (4.123), Pm max .x/ D

i D1 hSi x

 x; Si x  Si 1 xi

kUx  xk2

.

Denote u0 WD x and ui WD Ui ui 1 D Si x, i D 1; 2; : : : ; m. Since hUi y; ai i D ˇi for any y 2 H and for all i 2 I , we have hSi x; ai i D hUi ui 1 ; ai i D ˇi , i 2 I , consequently, m X i D1

m X hSi x  x; Si x  Si 1 xi D hSi x  x; Ui ui 1  ui 1 i i D1 m X D hSi x  x; .ˇi  hai ; ui 1 i/ai i i D1

5.12 Surrogate Constraints Methods

263

D

m X .ˇi  hai ; ui 1 i/.hSi x; ai i  hx; ai i/ i D1

m X D .ˇi  hai ; xi/.ˇi  hai ; ui 1 i/ i D1

and we obtain the following form for the step size Pm max .x/ D

i D1 .ˇi

 hai ; xi/.ˇi  hai ; ui 1 i/ kUx  xk2

,

(5.108)

where x … Fix U . It follows from Theorem 5.11.1 that for any x 0 2 H, the kC1 sequence fx k g1 D Uk x k with k D max is kD0 generated by the recurrence x Fej´er monotone with respect to C and converges weakly to a solution of the system (5.107). Moreover, by Theorem 3.8.4, the sequence converges strongly. Remark 5.11.2. For any u 2 H and z 2 Ci , we have hUi u u; zui D kUi u  uk2 . Therefore, it follows from the proof of Lemma 4.10.1 that in this case the first inequality in (4.119) is, actually, an equality. Consequently, we have an equality in (4.124) and the operator U with the step size function given by (4.123) has the property hU x  x; z  U xi D 0 for all z 2 C , or, equivalently, hU x  x; z  xi D kU x  xk2 . Consequently, the step size function  WD max is optimal in the following sense: kU x  zk D min kx C ˛.Ux  x/  zk . ˛

One can expect that this property leads in practice to an acceleration of the convergence to a solution of the system (5.107), of sequences generated by the recurrence x kC1 D Umax x k in comparison to the classical Kaczmarz method. A different acceleration scheme for the Kaczmarz method was presented in [247, Sect. 3.1].

5.12 Surrogate Constraints Methods In this section we consider a consistent system of linear inequalities Ay  b, where A is a matrix of type m  n and b 2 Rm . Let ai 2 Rn denote the i th row of the matrix A and ˇi 2 R denote the i th coordinate of b, i.e., A D Œa1 ; : : : ; am > and b D Œˇ1 ; : : : ; ˇm > . Suppose, for simplicity, that kai k D 1, i T 2 I . Denote Ci WD H .ai ; ˇi / D fy 2 Rn W ai> y  ˇi g, i 2 I , and C WD i 2I Ci . The surrogate constraints method (SCM) is described by the recurrence x kC1 D x k  k

vk .x k /> .Ax k  b/ > k k A v .x /,   A> vk .x k /2

(5.109)

264

5 Projection Methods

where x 0 2 Rn , k 2 Œ0; 2 and fvk g1 kD0 is a sequence of essential weight functions, i.e., weight functions vk W Rn ! Rm nf0g satisfying (4.108) for any x … C . It follows from Lemma 4.9.5 that recurrence (5.109) is well defined. We can also write x kC1 D Svk ;k x k , n n where fSvk g1 kD0 is a sequence of surrogate projection operators Svk W R ! R defined by (4.109) and Svk ;k WD Id Ck .Svk  Id/ is the k -relaxation of the operator Svk , k  0. Recall that Svk x k is the projection of x k onto a surrogate constraint Hk WD H .ak .x k /; ˇk .x k //, where ak .x k / WD A> vk .x k / and ˇk .x k / WD vk .x k /> b (see Sect. 4.9.4) and that Svk is a cutter with Fix Svk D C (see Theorem 4.9.6). Various versions of the surrogate projection method were studied by Merzlyakov [260], Oko [277], Yang and Murty [351], Kiwiel [229–231], Kiwiel and Łopuch [233], Cegielski [67, 68] and by Dudek [150].

5.12.1 Proper Control Definition 5.12.1. Let a sequence fx k g1 kD0 be generated by the surrogate constraints method (5.109). We say that a sequence of weight functions (or the control) fvk g1 kD0 is proper if   lim vk .x k /> .Ax k  b/ D 0 H) lim inf .Ax k  b/C  D 0 k

k

(5.110)

Now we apply Definition 5.8.5 (ii) to V k WD V D U D fPCi gi 2I with Ci D H .ai ; ˇi /, i 2 I . We see that fvk g1 kD0 is approximately semi-regular if and only if there exists a sequence fik g1  I such that the following implication holds kD0 lim vkik .x k /> Œ.ai>k x k  ˇik /C 2 D 0 H) lim inf max.ai> x k  ˇi /C D 0 (5.111) k

k

i 2I

It follows from Remark 5.8.6 (i) and from the equivalence of all norms in Rn that condition (5.111) is equivalent to   lim vkik .x k /> .ai>k x k  ˇik /C D 0 H) lim inf .Ax k  b/C  D 0. k

k

(5.112)

If vk considers only violated constraints (see Definition 4.9.8), then vk .x k /> .Ax k  b/ D vk .x k /> .Ax k  b/C D

X

vki .x k /> .ai> x k  ˇi /C .

i 2I

Therefore, the assumption that a sequence of weight functions fvk g1 kD0 which consider only violated constraints is approximately semi-regular is stronger than

5.12 Surrogate Constraints Methods

265

(5.110). This fact follows from a simple observation that if a sum of positive quantities goes to zero, then all summands go to zero. If the weight functions vk consider only violated constraints, k  0, then it follows from Proposition 4.9.11 that recurrence (5.109) defines an extrapolated simultaneous projection method. In this case one can apply Theorem 5.10.2 in order to prove the convergence of a sequence generated by (5.109). In the next subsection we prove that the convergence follows without assumptions that vk considers only violated constraints and that the corresponding sequence of weight functions fwk g1 kD0 (see (4.112)) is approximately semi-regular. It suffices to suppose instead that fvk g1 kD0 is proper.

5.12.2 Convergence Theorem Theorem 5.12.2. Let x 0 2 Rn and a sequence fx k g1 kD0 be generated by the surrogate constraints method (5.109), where lim infk k .2  k / > 0 and the k sequence of weight functions fvk g1 kD0 is proper. Then x converges to a solution  x of the system Ax  b. Proof. The operator Svk is a cutter and Fix Svk D C , k  0 (see Theorem 4.9.6). Therefore, equality (4.109) and Theorem 2.1.39 yield #2 "  kC1 2  k 2 vk .x k /> .Ax k  b/ x       z  x  z  k .2  k / , A> vk .x k /   for any z 2 C . Consequently, x k  z converges as a decreasing sequence, x k is bounded and vk .x k /> .Ax k  b/   lim D 0. A> vk .x k / k We can suppose without loss of generality that vk .x k / 2 m . Since x k is bounded, A> vk .x k / is also bounded. Therefore, limk vk .x k /> .Ax k  b/ D 0. By (5.110), we have m X  2 lim inf Œ.ai> x k  ˇi /C 2 D lim inf .Ax k  b/C  D 0: k

k

i D1

This implies   lim inf PCi x k  x k  D lim inf.ai> x k  ˇi /C D 0 k

k

for all i 2 I . Corollary 3.7.1 (iii) for X D Rn , Tk WD Svk and S WD PCi yields now the convergence of x k to x  2 Fix PCi D Ci for any i 2 I , i.e., x  2 C . t u

266

5 Projection Methods

5.12.3 Examples of Proper Control Now we give examples of proper sequences of weight functions vk W Rn ! Rm C nf0g. We suppose for simplicity that A has normalized rows, i.e., kai k D 1, i 2 I . Example 5.12.3. Let fx k g1 kD0 be generated by the surrogate constraints method k n m (5.109), where fvk g1 kD0 is a sequence of weight functions v W R ! RC nf0g k k > n k defined by v .x/ WD W .x/ .Ax  b/C , x 2 R , k  0, with W WD diag wk k n and fwk g1 kD0 being a sequence of weight functions w W R ! m which are approximately semi-regular with respect to fPCi gi 2I with Ci WD H .ai ; ˇi /, i 2 I . It is clear that vk considers only violated constraints, consequently, vk is essential (see Lemma 4.9.9). We show that the sequence fvk g1 kD0 is proper. We have k k k vk .x k /> .Ax k  b/ D .Ax k  b/> C W .x /.Ax  b/ X D !ik .x k /Œ.ai> x k  ˇi /C 2 i 2I

If limk vk .x k /> .Ax k  b/ D 0, then, for any sequence fik g1 kD0  I , we have lim !ikk .x k /Œ.ai>k x k  ˇik /C 2 D 0. k

  k k  D 0, i.e., Since fvk g1 kD0 is approximately semi-regular, lim infk PCi x  x lim infk .ai> x k  ˇi /C D 0 for all i 2 I , or, equivalently, lim infk .Ax k  b/C  D 0. In particular, a constant sequence vk WD v is proper, where v W Rn ! Rm C nf0g is defined by v.x/ WD W .x/.Ax  b/C , for x … C , W .x/ WD diag w.x/ and w has one of the following forms: (i) w is constant with positive weights, i.e., w.x/ DW w 2 ri m for all x 2 Rn . Note that w is regular (see Example 5.8.3 (i)), consequently, it is approximately regular (see Lemma 5.8.7 (i)). If we take w D . m1 ; m1 ; : : : ; m1 /, then we obtain i .x/ D m1 .ai> x ˇi /C , i.e., the weights vi .x/ are proportional to the residua of violated constraints. Since the rescaling of weights does not change the operator Sv defined by (4.109), we can apply, equivalently, i .x/ WD .ai> x  ˇi /C or i .x/ WD

P

.ai> xˇi /C > j 2I .aj xˇj /C

. The latter weights were proposed by Yang and Murty

[351, page 167]. (ii) w considers only almost violated constraints, i.e., !j .x/ D 0 for all j 2 I such that .aj> x  ˇj /C < max.ai> x  ˇi /C , i 2I

where 2 .0; 1. Note that w is approximately regular (see Example 5.8.3 (v) and Lemma 5.8.7 (i)). If we take D 1, then the surrogate constraints method reduces to a successive projection method with the remotest set control.

5.12 Surrogate Constraints Methods

267

Example 5.12.4. Let fx k g1 kD0 be generated by the surrogate constraints method (5.109), where fvk g1 is a sequence containing a subsequence fvnk g1 kD0 kD0 of weight functions which consider only almost remotest constraints, i.e., ink .x/ D 0 for all i … J .x/; where J .x/ WD fj 2 I W .aj> x  ˇj /C  max.ai> x  ˇi /C g i 2I

x 2 Rn , with 2 .0; 1. We show that fvk g1 kD0 is proper. Suppose for simplicity that vk .x/ 2 m , x 2 Rn . Let jnk D jnk .x nk / 2 J .x nk / be such that jnnk .x nk /  m1 , k k  0. The existence of such j follows from the fact that vnk .x nk / 2 m . We have vnk .x nk /> .Ax nk  b/ D

X

ink .x nk /.ai> x nk  ˇi /

i 2I

D

X

i 2J

ink .x nk /.ai> x nk  ˇi /C

.x nk /

 jnnk .x nk /.aj>n x nk  ˇjnk /C k

k

1  max.ai> x nk  ˇi /C . m i 2I > k Therefore, limk vk .x k /> .Ax k  b/ D 0 yields  lim infk maxi 2I .ai x  ˇi /C D 0 k   which is equivalent to lim infk .Ax  b/C D 0. Note that the successive projection method with the remotest set control is a special case of the method with D 1.

Example 5.12.5. Let fx k g1 kD0 be generated by the surrogate constraints method (5.109), where fvk g1 is a sequence of essential weight functions containing a kD0 subsequence fvnk g1 satisfying the following condition kD0  ink .x nk /

 ı for some i 2 Argmaxi 2I .ai> x nk  ˇi / D 0 for i … I.x nk /,

(5.113)

for all x nk … C , where I.x/ WD fi 2 I W ai> x > ˇi g, k  0, and ı > 0 is a small nk predefined constant. We show that fvk g1 … C and jnk 2 I.x nk / kD0 is proper. Let x nk nk be such that i .x /  ı and .aj>n x nk  ˇjnk /C D max.ai> x nk  ˇi /C . i 2I

k

Then we have vnk .x nk /> .Ax nk  b/ D

X

ink .x nk /.ai> x nk  ˇi /

i 2I

D

X

i 2I.x nk /

ink .x nk /.ai> x nk  ˇi /C

268

5 Projection Methods

 jnnk .x nk /.aj>n x nk  ˇjnk /C k



ı max.ai> x nk i 2I

k

 ˇi /C .

> k Suppose that limk vk .x k /> .Axk  b/ D 0. Then i 2I .ai x  ˇi /C D 0  lim infk max k k 1   which is equivalent to lim infk .Ax  b/C D 0, i.e., fv gkD0 is proper.

Remark 5.12.6. Yang and Murty proved the finite convergence of sequences fx k g1 kD0 generated by the surrogate constraints method (5.109) with weight functions vk satisfying the condition  ik .x k /

 ı for i 2 I" .x k / D 0 for i … I" .x k /,

(5.114)

> k for predefined small T constantsn "; ı>> 0, where I" .x/ WD fi 2 I W ai x  ˇi > "g k and x … C" WD i 2I fx 2 R W ai x  ˇi > "g (see [351, Theorem 3.3]) Note that nk 1 if a sequence fvk g1 kD0 satisfies condition (5.114), then any subsequence fv gkD0 nk satisfies condition (5.113) for all x … C" . We leave it to the reader to check that the result of [351, Theorem 3.3] can be derived from Example 5.12.5.

5.13 SCM with Residual Selection In this section we present surrogate constraints methods for solving a consistent system of linear inequalities Au  b, which employ a special kind of surrogate projection operators. For any x … C the surrogate projection Sv x given by (4.109) is determined by an essential weight v D v.x/ (see Definition 4.9.4) related to a constraints selection L WD L.x/  I WD f1; 2; : : : ; mg. We use the same notation as in Sect. 5.12.

5.13.1 General Properties We start with some relationships between surrogate projections and metric projections onto solution sets of equality and inequality systems. Let x … C be a current approximation of the solution, L WD L.x/  I be a nonempty subset of indices (a constraints selection) and let and r WDj L j. Let AL denote the submatrix of A with rows ai , i 2 L, and bL —the subvector of b with coordinates ˇi , i 2 L. Further, denote \ CL WD Ci D fu 2 Rn W AL u  bL g i 2L

5.13 SCM with Residual Selection

269

and HL WD fu 2 Rn W AL u D bL g. Suppose, for simplicity, that L D f1; 2; : : : ; rg, i.e.,  AD

     bL vL .x/ AL . ,bD and v.x/ D AI nL bI nL vI nL .x/

Denote SvL x WD x 

vL .x/> .AL x  bL / > AL vL .x/   > A vL .x/2 L

(cf. (4.109)). Lemma 5.13.1. Let x … C and L WD L.x/  I be such that AL has full row rank. Then PCL x D PHL x if and only if 1 .AL A> L / .AL x  bL /  0.

Proof. Consider the following convex minimization problem minimize 12 ku  xk2 subject to AL u D bL u 2 Rn

(5.115)

whose solution is uN D PHL x. The corresponding KKT-point .Nu; y/ N 2 Rn  Rr has the form > 1 uN D x  A> L .AL AL / .AL x  bL / 1 yN D .AL A> L / .AL x  bL / 1 u; y/ N is also a KKT-point (i) Suppose that .AL A> L / .AL x  bL /  0. Then .N related to the convex minimization problem

minimize 12 ku  xk2 subject to AL u  bL u 2 Rn

(5.116)

whose solution is u D PCL . Therefore, uN D u . (ii) Suppose that uN D PHL x D PCL x, i.e., > 1 uN D x  A> L .AL AL / .AL x  bL /.

(5.117)

270

5 Projection Methods

Let y  be a vector of Lagrange multipliers for problem (5.116). Then .Nu; y  / satisfies the related KKT-system, consequently,  uN D x  A> L y D 0.

This equality together with (5.117) give  > > 1 A> L y D AL .AL AL / .AL x  bL /.

Since AL has full row rank, we have 1 y  D .AL A> L / .AL x  bL /

which is nonnegative, because y  is a vector of Lagrange multipliers for problem (5.116). u t Lemma 5.13.2. Let x … C , L WD L.x/  I be such that AL has full row rank and vL WD vL .x/ 2 RrC be essential for the system AL u  bL i.e., v> L .AL x  bL / > 0. Then SvL x D PHL x if and only if 1 vL D ˛.AL A> L / .AL x  bL /

(5.118)

for some ˛ > 0. Consequently, if any of both conditions is satisfied, then SvL x D PCL x. Proof. ((H) Let vL be defined by (5.118). Applying (4.109) and (4.5) with A WD AL and v.x/ WD vL .x/, we obtain S vL x D x 

v> L .AL x  bL / >  > 2 AL vL  A vL 

Dx

v> L .AL x  bL / > AL vL > v> L AL AL vL

L

> 1 D x  A> L .AL AL / .AL x  bL / D PHL x.

(H)) Suppose that SvL x D PHL x. By (4.5) and (4.109) with A WD AL , we have v> > > 1 L .AL x  bL / >  > 2 AL vL D AL .AL AL / .AL x  bL /. A vL  L

Since AL has full row rank, 1 vL D ˛.AL A> L / .AL x  bL /,

5.13 SCM with Residual Selection

271

where ˛D

 > 2  A vL  L

v> L .AL x  bL /

which is positive, because vL is essential for the system AL u  bL and A> L vL ¤ 0 (see Lemma 4.9.5). u t

5.13.2 Description of the Method Now we describe one iteration of the surrogate constraints method with residual selection. Suppose that x k is obtained in the kth iteration. The next iteration consists of two steps: Step 1 (Constraints selection). Determine a subset Lk WD L.x k / such that (a) AL has full row rank and (b) The vector 1 k vLk WD vLk .x k / D .ALk A> Lk / .ALk x  bLk /

(5.119)

is nonnegative and essential for the system ALk u  bLk . Step 2 (Actualization). Evaluate x kC1 D x k C k .SvLk x k  x k /,

(5.120)

where k 2 .0; 2/. By Lemmas 5.13.1 and 5.13.2, we have SvLk x k D PCLk x k D PHLk x k . A simple way to obtain a constraints selection satisfying conditions (a) and (b) above is to take Lk WD L.x k / D fik g D fi.x k /g, where ik WD argmaxf

ai> x k  ˇi W i 2 I g, kai k

(5.121)

consequently, vk D v.x k / D ıik . It is clear that vk is essential for the system Au  b. The surrogate constraints method with Lk and vk defined by (5.121) reduces to the successive projection method with the remotest set control. It turns out that the surrogate constraints method with residual selection Lk satisfying conditions (a) and (b) above provides steps which are no shorter than in the latter method if ik 2 Lk . This property leads to the convergence of sequences generated by (5.120). Theorem 5.13.3. Let x 0 2 Rn and a sequence fx k g1 kD0 be generated by the surrogate constraints method (5.120), where lim infk k .2  k / > 0, vLk is given

272

5 Projection Methods

by (5.119), Lk is obtained by a residual selection and Lk contains ik defined by (5.121). Then x k converges to a solution x  of the system Ax  b. Proof. By Lemma 5.13.2 and by the inclusion CLk  Cik , we have               SvLk x k  x k  D PCLk x k  x k   PCik x k  x k   PCi x k  x k  for all i 2 I . Consequently, condition (3.11) with Tk WD SvLk and S WD PCi , i 2 I , is satisfied. Because ALk has full row rank, ALk A> Lk is positive definite and vLk is essential. Since for any v the surrogate projection Sv is a cutter with Fix Sv D C (see Theorem 4.9.6), Corollary 3.7.1 yields the convergence of x k to a point x  2 C WD fx 2 Rn W Ax  bg. t u In the next subsection we present several constructions of constraints selection L.x/ for x … C leading to full row rank matrix AL for which the vector vL WD 1 .AL A> L / .AL x  bL / is nonnegative and essential for the system AL u  bL . More general constructions of this type and their applications to the convex minimization problems can be found in [62–65, 72, 73, 154, 155, 230–232].

5.13.3 Obtuse Cone Selection Let x … C . In an obtuse cone selection a subset L WD L.x/ is constructed recursively 1 in such a way that AL x > bL , AL has full row rank and .AL A>  0. Therefore, L/ r the vector vL 2 R given by 1 vL WD .AL A> L / .AL x  b/

is nonnegative and essential for the system AL u  bL . Consequently, the vector v WD .vL ; vI nL / with vI nL D 0 2 Rmr is essential for the system Au  b. The name “obtuse cone selection” can be explained in the following way. Denote CL WD conefai W i 2 Lg and suppose that AL has full row rank. Then CL is obtuse (in 1 Linfai W i 2 Lg) if and only if .AL A>  0 (see [232, Lemma 3.1] or [65, L/ Lemma 1.6]). Denote K D K.x/ WD fi 2 I W ai> x > ˇi g. For any z 2 C and for w WD x  z we have AK w > 0. Let L  K be a current selection such that the matrix AL has full 1 row rank and .AL A>  0. The following lemmas give conditions under which L/ 0 the update L WD L [ flg  K maintains the described above properties of AL , i.e., 1 AL0 has full row rank and .AL0 A>  0. Denote L0 /   AL . AL0 WD ai> Recall that AC L denotes the Moore–Penrose pseudoinverse of AL which for a full > > 1 row matrix AL has the form AC L D AL .AL AL / . The proof of the following Lemma can be found in [65, Corollary 2.7].

5.13 SCM with Residual Selection

273

n Lemma 5.13.4. Let A> L0 w > 0 for some w 2 R . If AL has full row rank, C 1 1  0 and ai> AL  0, then AL0 has full row rank and .AL0 A>  0. .AL A> L/ L0 /

Let x D x k … C . If L D flg, where l 2 K, then, of course, AL has full row rank 1 (as a nonzero vector al ) and .AL A> D kal k2  0. If there is i 2 K such that L/ > ai al  0, then 2 > ai> AC  0. L D ai al kal k Applying Lemma 5.13.4 we see that L0 has the same properties as L: the matrix AL0 1 has full row rank and .AL0 A>  0. By repeated application of Lemma 5.13.4 L0 / 0 with L WD L we obtain a new subset L0  K such that AL0 has full row rank and 1 .AL0 A>  0. L0 / Let L WD L.x/ be constructed by an obtuse cone selection. Since AL x > bL , 1 the vector vL WD .AL A> L / .AL x  b/ has nonnegative coordinates. Therefore, all assumptions of Lemma 5.13.2 are satisfied, consequently, SvL x D PHL x D PCL x. If in each iteration of (5.120) we start the above defined construction of the subset a> x k ˇ Lk WD L.x k / with lk WD argmaxf i kai k i W i 2 I g, then Theorem 5.13.3 yields the convergence of sequences generated by (5.120) to a solution of the system Ax  b.

5.13.4 Regular Obtuse Cone Selection Now we present a special case of the obtuse cone selection. The corollary below follows from Lemma 5.13.4. We use the same notation as in Sect. 5.13.3. n Corollary 5.13.5. Let A> L0 w > 0 for some w 2 R . If AL has full row rank, > 1 1 .AL AL /  0 and AL ai  0, then AL0 has full row rank and .AL0 A>  0. L0 / 1 Proof. Suppose that AL has full row rank, .AL A>  0 and AL ai  0. Then, by L/ Lemma 5.13.4, we have > > > 1 ai> AC  0. L D ai AL .AL AL / 1 Consequently, AL0 has full row rank and .AL0 A>  0. L0 /

t u

Let x … C . Applying Corollary 5.13.5 and starting with L D flg, where l 2 K, one can construct recursively a maximal subset L  K such that ai> aj  0 for all 1 i; j 2 L, i ¤ j . Then, of course, AL has full row rank and .AL A>  0. The L/ subset L constructed in this way is called a regular obtuse cone selection (ROCS). The regular obtuse cone selection is easier to perform than the general obtuse cone selection presented in Sect. 5.13.3 and the residual selection presented in Sect. 5.13.2. Application of all these methods leads in practice to an essential acceleration of the convergence, in particular, if the solution set is “flat” (see the numerical results presented in [72, 73, 155]).

274

5 Projection Methods

5.14 Exercises Exercise 5.14.1. Show that Theorem 5.1.5 remains true if A and B are closed affine subspaces with a common point. Exercise 5.14.2. Show that the result of [351, Theorem 3.3] can be derived from Example 5.12.5. Exercise 5.14.3. Prove equalities (5.75) and (5.76).

References

1. S. Agmon, The relaxation method for linear inequalities. Can. J. Math. 6, 382–392 (1954) 2. R. Aharoni, A. Berman, Y. Censor, An interior point algorithm for the convex feasibility problem. Adv. Appl. Math. 4, 479–489 (1983) 3. R. Aharoni, Y. Censor, Block-iterative projection methods for parallel computation of solutions to convex feasibility problems. Lin. Algebra Appl. 120, 165–175 (1989) 4. D. Alevras, M.W. Padberg, Linear Optimization and Extensions. Problems and Solutions (Springer, Berlin, 2001) 5. A. Aleyner, S. Reich, Block-iterative algorithms for solving convex feasibility problems in Hilbert and in Banach spaces. J. Math. Anal. Appl. 343, 427–435 (2008) 6. A. Aleyner, S. Reich, Random products of quasi-nonexpansive mappings in Hilbert space. J. Convex Anal. 16, 633–640 (2009) 7. M. Altman, On the approximate solution of linear algebraic equations. Bulletin de l’Acad´emie Polonaise des Sciences Cl. III 3 , 365–370 (1957) 8. I. Amemiya, T. Ando, Convergence of random products of contractions in Hilbert space. Acta Sci. Math. (Szeged) 26, 239–244 (1965) 9. R. Ansorge, Connections between the Cimmino-method and the Kaczmarz-method for solution of singular and regular systems of equations. Computing 33, 367–375 (1984) 10. G. Appleby, D.C. Smolarski, A linear acceleration row action method for projecting onto subspaces. Electron. Trans. Numer. Anal. 20, 253–275 (2005) 11. N. Aronszajn, Theory of reproducing kernels. Trans. Am. Math. Soc. 68, 337–404 (1950) 12. A. Auslender, Optimisation, m´ethodes num´eriques (Masson, Paris, 1976) 13. V.N. Babenko, Convergence of the Kaczmarz projection algorithm. Zh. Vychisl. Mat. Mat. Fiz. 24, 1571–1573 (1984) (in Russian) 14. J.B. Baillon, R.E. Bruck, S. Reich, On the asymptotic behavior of nonexpansive mappings and semigroups in Banach spaces. Houston J. Math. 4, 1–9 (1978) 15. S. Banach, Sur les op´erations dans les ensembles abstraits et leur application aux e´ quations int´egrals. Fundamenta Mathematicae 3, 133–181 (1922) 16. H.H. Bauschke, A norm convergence result on random products of relaxed projections in Hilbert space. Trans. Am. Math. Soc. 347, 1365–1373 (1995) 17. H.H. Bauschke, Projection Algorithms and Monotone Operators, Ph.D. Thesis, Department of Mathematics, Simon Fraser, University, Burnaby, BC, Canada, 1996 18. H.H. Bauschke, The approximation of fixed points of compositions of nonexpansive mapping in Hilbert space. J. Math. Anal. Appl. 202, 150–159 (1996) 19. H.H. Bauschke, The composition of the projections onto closed convex sets in Hilbert space is asymptotically regular. Proc. Am. Math. Soc. 131, 141–146 (2002)

A. Cegielski, Iterative Methods for Fixed Point Problems in Hilbert Spaces, Lecture Notes in Mathematics 2057, DOI 10.1007/978-3-642-30901-4, © Springer-Verlag Berlin Heidelberg 2012

275

276

References

20. H.H. Bauschke, J. Borwein, On the convergence of von Neumann’s alternating projection algorithm for two sets. Set-Valued Anal. 1, 185–212 (1993) 21. H.H. Bauschke, J. Borwein, Dykstra’s alternating projection algorithm for two sets. J. Approx. Theor. 79, 418–443 (1994) 22. H.H. Bauschke, J. Borwein, On projection algorithms for solving convex feasibility problems. SIAM Rev. 38, 367–426 (1996) 23. H.H. Bauschke, J.M. Borwein, A.S. Lewis, The method of cyclic projections for closed convex sets in Hilbert space. Contemp. Math. 204, 1–38 (1997) 24. H.H. Bauschke, P.L. Combettes, A weak-to-strong convergence principle for Fej´er-monotone methods in Hilbert spaces. Math. Oper. Res. 26, 248–264 (2001) 25. H.H. Bauschke, P.L. Combettes, S.G. Kruk, Extrapolation algorithm for affine-convex feasibility problems. Numer. Algorithms 41, 239–274 (2006) 26. H.H. Bauschke, P.L. Combettes, D.R. Luke, Phase retrieval, error reduction algorithm, and Fienup variants: A view from convex optimization. J. Opt. Soc. Am. A 19, 1334–1345 (2002) 27. H.H. Bauschke, P.L. Combettes, D.R. Luke, Hybrid projection-reflection method for phase retrieval. J. Opt. Soc. Am. A 20, 1025–1034 (2003) 28. H.H. Bauschke, P.L. Combettes, D.R. Luke, Finding best approximation pairs relative to two closed convex sets in Hilbert spaces. J. Approx. Theor. 127, 178–192 (2004) 29. H.H. Bauschke, P.L. Combettes, D.R. Luke, A strongly convergent reflection method for finding the projection onto the intersection of two closed convex sets in a Hilbert space. J. Approx. Theor. 141, 63–69 (2006) 30. H.H. Bauschke, F. Deutsch, H. Hundal, S-H. Park, Accelerating the convergence of the method of alternating projections. Trans. Am. Math. Soc. 355, 3433–3461 (2003) 31. H.H. Bauschke, S.G. Kruk, Reflection-projection method for convex feasibility problems with an obtuse cone. J. Optim. Theor. Appl. 120, 503–531 (2004) 32. H.H. Bauschke, E. Matouˇskov´a, S. Reich, Projection and proximal point methods: convergence results and counterexamples. Nonlinear Anal. 56, 715–738 (2004) 33. M.H. Bazaraa, H.D. Sherali, C.M. Shetty, Nonlinear Programming, Theory and Algorithms, 3rd edn. (Wiley, Hoboken, 2006) 34. M. Benzi, C.D. Meyer, A direct projection method for sparse linear systems. SIAM J. Sci. Comput. 16, 1159–1176 (1995) 35. A. Berman, R.J. Plemmons, Nonnegative Matrices in the Mathematical Sciences (Academic, New York, 1979) 36. V. Berinde, Iterative Approximation of Fixed Points (Springer, Berlin, 2007) 37. M. Bertero, P. Boccacci, Introduction to Inverse Problems in Imaging (Institute of Physics Publishing, Bristol, 1998) 38. D.P. Bertsekas, Nonlinear Programming (Athena Scientific, Belmont, 1995) 39. D. Blatt, A.O. Hero, Energy-based sensor network source localization via projection onto convex sets (POCS). IEEE Trans. Signal Process. 54, 3614–3619 (2006) 40. J.M. Borwein, A. Lewis, Convex Analysis and Nonlinear Optimization, Theory and Examples (Springer, New York, 2000) 41. R. Bramley, A. Sameh, Row projection methods for large nonsymmetric linear systems. SIAM J. Sci. Stat. Comput. 13, 168–193 (1992) 42. L.M. Bregman, Finding the common point of convex sets by the method of successive projection (in Russian). Dokl. Akad. Nauk SSSR 162, 487–490 (1965); English translation in: Soviet Math. Dokl. 6, 688–692 (1965) ¨ Abbildung von Mannigfaltigkeiten. Math. Ann. 71, 97–115 (1912) 43. L.E.J. Brouwer, Uber 44. F.E. Browder, Fixed-point theorems for noncompact mappings in Hilbert space. Proc. Nat. Acad. Sci. USA 53, 1272–1276 (1965) 45. F.E. Browder, Nonexpansive nonlinear operators in a Banach space. Proc. Nat. Acad. Sci. USA 54, 1041–1044 (1965) 46. F.E. Browder, Convergence of approximants to fixed points of nonexpansive nonlinear mappings in Banach spaces. Arch. Rational Mech. Anal. 24, 82–90 (1967)

References

277

47. F.E. Browder, Convergence theorems for sequences of nonlinear operators in Banach spaces. Math. Zeitschr. 100, 201–225 (1967) 48. F.E. Browder, W.V. Petryshyn, The solution by iteration of nonlinear functional equations in Banach spaces. Bull. Am. Math. Soc. 72, 571–575 (1966) 49. R.E. Bruck, Nonexpansive projections on subsets of Banach spaces. Pac. J. Math. 47, 341–355 (1973) 50. R.E. Bruck, Random products of contractions in metric and Banach spaces. J. Math. Anal. Appl. 88, 319–332 (1982) 51. R.E. Bruck, S. Reich, Nonexpansive projections and resolvents of accretive operators in Banach spaces. Houston J. Math. 3, 459–470 (1977) 52. R.S. Burachik, J.O. Lopes, B.F. Svaiter, An outer approximation method for the variational inequality problem. SIAM J. Contr. Optim. 43, 2071–2088 (2005) 53. D. Butnariu, Y. Censor, On the behavior of a block-iterative projection method for solving convex feasibility problems. Int. J. Comp. Math. 34, 79–94 (1990) 54. D. Butnariu, Y. Censor, P. Gurfil, E. Hadar, On the behavior of subgradient projections methods for convex feasibility problems in Euclidean spaces. SIAM J. Opt. 19, 786–807 (2008) 55. C. Byrne, Iterative oblique projection onto convex sets and the split feasibility problem. Inverse Probl. 18, 441–453 (2002) 56. C. Byrne, A unified treatment of some iterative algorithms in signal processing and image reconstruction. Inverse Probl. 20, 103–120 (2004) 57. C.L. Byrne, Applied Iterative Methods (AK Peters, Wellesley, 2008) 58. C.L. Byrne, Bounds on the largest singular value of a matrix and the convergence of simultaneous and block-iterative algorithms for sparse linear systems. Int. Trans. Oper. Res. 16, 465–479 (2009) 59. T.D. Capricelli, P.L. Combettes, Parallel block-iterative reconstruction algorithms for binary tomography. Electron. Notes Discr. Math. 20, 263–280 (2005) 60. G. Casssiani, G. B¨ohm, A. Vesnaver, R. Nicolich, A geostatistical framework for incorporating seismic tomography auxiliary data into hydraulic conductivity estimation. J. Hydrol. 206, 58–74 (1998) 61. J. Cea, Optimisation: th´eorie et algorithmes (Dunod, Paris, 1971); Polish translation: Optymalizacja: Teoria i algorytmy (PWN, Warszawa, 1976) 62. A. Cegielski, Relaxation Methods in Convex Optimization Problems (in Polish). Monographs, vol. 67, Institute of Mathematics, Higher College of Engineering, Zielona G´ora, 1993 63. A. Cegielski, in Projection Onto an Acute Cone and Convex Feasibility Problems, ed. by J. Henry i J.-P. Yvon. Lecture Notes in Control and Inform. Sci., vol. 197 (Springer, London, 1994), pp. 187–194 64. A. Cegielski, A method of projection onto an acute cone with level control in convex minimization. Math. Program. 85, 469–490 (1999) 65. A. Cegielski, Obtuse cones and Gram matrices with nonnegative inverse. Lin. Algebra Appl. 335, 167–181 (2001) 66. A. Cegielski, A generalization of the Opial’s theorem. Contr. Cybern. 36, 601–610 (2007) 67. A. Cegielski, Convergence of the projected surrogate constraints method for the linear split feasibility problems. J. Convex Anal. 14, 169–183 (2007) 68. A. Cegielski, Projection methods for the linear split feasibility problems. Optimization 57, 491–504 (2008) 69. A. Cegielski, Generalized relaxations of nonexpansive operators and convex feasibility problems. Contemp. Math. 513, 111–123 (2010) 70. A. Cegielski, Y. Censor, in Opial-Type Theorems and the Common Fixed Point Problem, ed. by H.H. Bauschke, R.S. Burachik, P.L. Combettes, V. Elser, D.R. Luke, H. Wolkowicz. FixedPoint Algorithms for Inverse Problems in Science and Engineering, Springer Optimization and Its Applications, vol. 49 (Springer, New York, 2011), pp. 155–183 71. A. Cegielski, Y. Censor, Extrapolation and local acceleration of an iterative process for common fixed point problems. J. Math. Anal. Appl. 394, 809–818 (2012)

278

References

72. A. Cegielski, R. Dylewski, Selection strategies in projection methods for convex minimization problems. Discuss. Math. Differ. Incl. Contr. Optim. 22, 97–123 (2002) 73. A. Cegielski, R. Dylewski, Residual selection in a projection method for convex minimization problems. Optimization 52, 211–220 (2003) 74. A. Cegielski, R. Dylewski, Variable target value relaxed alternating projection method. Comput. Optim. Appl. 47, 455–476 (2010) 75. A. Cegielski, A. Suchocka, Incomplete alternating projection method for large inconsistent linear systems. Lin. Algebra Appl. 428, 1313–1324 (2008) 76. A. Cegielski, A. Suchocka, Relaxed alternating projection methods. SIAM J. Optim. 19, 1093–1106 (2008) 77. A. Cegielski, R. Zalas, Methods for variational inequality problem over the intersection of fixed point sets of quasi-nonexpansive operators (2012) Numer. Funct. Anal. Optim. (in print) 78. Y. Censor, Row-action methods for huge and sparse systems and their applications. SIAM Rev. 23, 444–466 (1981) 79. Y. Censor, Iterative methods for convex feasibility problems. Ann. Discrete Math. 20, 83–91 (1984) 80. Y. Censor, An automatic relaxation method for solving interval linear inequalities. J. Math. Anal. Appl. 106, 19–25 (1985) 81. Y. Censor, Parallel application of block-iterative methods in medical imaging and radiation therapy. Math. Program. 42, 307–325 (1988) 82. Y. Censor, Binary steering in discrete tomography reconstruction with sequential and simultaneous iterative algorithms. Lin. Algebra Appl. 339, 111–124 (2001) 83. Y. Censor, M.D. Altschuler, W.D. Powlis, On the use of Cimmino’s simultaneous projections method for computing a solution of the inverse problem in radiation therapy treatment planning. Inverse Probl. 4, 607–623 (1988) 84. Y. Censor, A. Ben-Israel, Y. Xiao, J.M. Galvin, On linear infeasibility arising in intensitymodulated radiation therapy inverse planning. Lin. Algebra Appl. 428, 1406–1420 (2008) 85. Y. Censor, T. Bortfeld, B. Martin, A. Trofimov, A unified approach for inversion problems in intensity-modulated radiation therapy. Phys. Med. Biol. 51, 2353–2365 (2006) 86. Y. Censor, P.P.B. Eggermont, D. Gordon, Strong underrelaxation in Kaczmarz’s method for inconsistent systems. Numer. Math. 41, 83–92 (1983) 87. Y. Censor, T. Elfving, New methods for linear inequalities. Lin. Algebra Appl. 42, 199–211 (1982) 88. Y. Censor, T. Elfving, A multiprojection algorithm using Bregman projections in a product space. Numer. Algorithm 8, 221–239 (1994) 89. Y. Censor, T. Elfving, Block-iterative algorithms with diagonal scaled oblique projections for the linear feasibility problems. SIAM J. Matrix Anal. Appl. 24, 40–58 (2002) 90. Y. Censor, T. Elfving, Iterative algorithms with seminorm-induced oblique projections. Abstr. Appl. Anal. 8, 387–406 (2003) 91. Y. Censor, T. Elfving, G.T. Herman, in Averaging Strings of Sequential Iterations for Convex Feasibility Problems, ed. by D. Butnariu, Y. Censor, S. Reich. Inherently Parallel Algorithms in Feasibility and Optimization and their Applications (Elsevier, Amsterdam, 2001), pp. 101–113 92. Y. Censor, T. Elfving, G.T. Herman, T. Nikazad, On diagonally relaxed orthogonal projection methods. SIAM J. Sci. Comput. 30, 473–504 (2008) 93. Y. Censor, T. Elfving, N. Kopf, T. Bortfeld, The multiple-sets split feasibility problem and its applications for inverse problems. Inverse Probl. 21, 2071–2084 (2005) 94. Y. Censor, A. Gibali, Projections onto super-half-spaces for monotone variational inequality problems in finite-dimensional spaces. J. Nonlinear Convex Anal. 9, 461–475 (2008) 95. Y. Censor, A. Gibali, S. Reich, The subgradient extragradient method for solving variational inequalities in Hilbert space. J. Optim. Theory Appl. 148, 318–335 (2011) 96. Y. Censor, A. Gibali, S. Reich, Extensions of Korpelevich’s extragradient method for the variational inequality problem in Euclidean space. Optimization 61, 1119–1132 (2012)

References

279

97. Y. Censor, A. Gibali, S. Reich, Strong convergence of subgradient extragradient methods for the variational inequality problem in Hilbert space. Optim. Meth. Software 26, 827–845 (2011) 98. Y. Censor, D. Gordon, R. Gordon, Component averaging: An efficient iterative parallel algorithm for large and sparse unstructured problems. Parallel Comput. 27, 777–808 (2001) 99. Y. Censor, D. Gordon, R. Gordon, BICAV: A block-iterative, parallel algorithm for sparse systems with pixel-related weighting. IEEE Trans. Med. Imag. 20, 1050–1060 (2001) 100. Y. Censor, G.T. Herman, On some optimization techniques in image reconstruction from projections. Appl. Numer. Math. 3, 365–391 (1987) 101. Y. Censor, A.N. Iusem, S.A. Zenios, An interior point method with Bregman functions for the variational inequality problem with paramonotone operators. Math. Program. 81, 373–400 (1998) 102. Y. Censor, A. Lent, Cyclic subgradient projections. Math. Program. 24, 233–235 (1982) 103. Y. Censor, A. Motova, A. Segal, Perturbed projections and subgradient projections for the multiple-sets split feasibility problem. J. Math. Anal. Appl. 327, 1244–1256 (2007) 104. Y. Censor, A. Segal, The split common fixed point problem for directed operators. J. Convex Anal. 16, 587–600 (2009) 105. Y. Censor, A. Segal, On the string averaging method for sparse common fixed point problems. Int. Trans. Oper. Res. 16, 481–494 (2009) 106. Y. Censor, A. Segal, Sparse string-averaging and split common fixed points. Contemp. Math. 513, 125–142 (2010) 107. Y. Censor, E. Tom, Convergence of string-averaging projection schemes for inconsistent convex feasibility problems. Optim. Meth. Software 18, 543–554 (2003) 108. Y. Censor, S.A. Zenios, Parallel Optimization, Theory, Algorithms and Applications (Oxford University Press, New York, 1997) ¨ 109. A.E. C ¸ etin, H. Ozaktas ¸ , H.M. Ozaktas, Resolution enhancement of low resolution wavefields with POCS algorithm. Electron. Lett. 9, 1808–1810 (2003) 110. W. Chen, D. Craft, T.M. Madden, K. Zhang, H.M. Kooy, G.T. Herman, A fast optimization algorithm for multicriteria intensity modulated proton therapy planning. Med. Phys. 7, 4938– 4945 (2010) 111. W. Chen, G.T. Herman, Efficient controls for finitely convergent sequential algorithms. ACM Trans. Math. Software 37, 1–23 (2010) 112. W. Cheney, A.A. Goldstein, Proximity maps for convex sets. Proc. Am. Math Soc. 10, 448– 450 (1959) 113. C.E. Chidume, Quasi-nonexpansive mappings and uniform asymptotic regularity. Kobe J. Math. 3, 29–35 (1986) 114. Ch. Chidume, Geometric Properties of Banach Spaces and Nonlinear Iterations (Springer, London, 2009) 115. H. Choi, R.G. Baraniuk, Multiple wavelet basis image denoising using Besov ball projections. IEEE Signal Process. Lett. 11, 717–720 (2004) 116. G. Cimmino, Calcolo approssimato per le soluzioni dei sistemi di equazioni lineari. La Ricerca Scientifica, II 9, 326–333 (1938) 117. P.L. Combettes, Inconsistent signal feasibility problems: Least-square solutions in a product space. IEEE Trans. Signal Process. 42, 2955–2966 (1994) 118. P.L. Combettes, in The Convex Feasibility Problem in Image Recovery, ed. by P. Hawkes. Advances in Imaging and Electron Physics, vol. 95 (Academic, New York, 1996), pp. 155–270 119. P.L. Combettes, Hilbertian convex feasibility problem: Convergence of projection methods. Appl. Math. Optim. 35, 311–330 (1997) 120. P.L. Combettes, Convex set theoretic image recovery by extrapolated iterations of parallel subgradient projections. IEEE Trans. Image Process. 6, 493–506 (1997) 121. P.L. Combettes, in Quasi-Fej´erian Analysis of Some Optimization Algorithm, ed. by D. Butnariu, Y. Censor, S. Reich. Inherently Parallel Algorithms in Feasibility and Optimization and their Applications (Elsevier, Amsterdam, 2001), pp. 115–152

280

References

122. P.L. Combettes, Solving monotone inclusions via compositions of nonexpansive averaged operators. Optimization 53, 475–504 (2004) 123. P.L. Combettes, P. Bondon, Hard-constrained inconsistent signal feasibility problems. IEEE Trans. Signal Process. 47, 2460–2468 (1999) 124. P.L. Combettes, S.A. Hirstoaga, Equilibrium programming in Hilbert spaces. J. Nonlinear Convex Anal. 6, 117–136 (2005) 125. P.L. Combettes, H. Puh, Iterations of parallel convex projections in Hilbert spaces. Numer. Funct. Anal. Optim. 15, 225–243 (1994) 126. G. Crombez, A geometrical look at iterative methods for operators with fixed points. Numer. Funct. Anal. Optim. 26, 157–175 (2005) 127. G. Crombez, A hierarchical presentation of operators with fixed points on Hilbert spaces. Numer. Funct. Anal. Optim. 27, 259–277 (2006) 128. Y.-H. Dai, Fast algorithms for projection on an ellipsoid. SIAM J. Optim. 16, 986–1006 (2006) 129. A. Dax, A note of the convergence of linear stationary iterative process. Lin. Algebra Appl. 129, 131–142 (1990) 130. A. Dax, Linear search acceleration of iterative methods. Lin. Algebra Appl. 130, 43–63 (1990) 131. A. Dax, On hybrid acceleration of a linear stationary iterative process. Lin. Algebra Appl. 130, 99–110 (1990) 132. A. Dax, The convergence of linear stationary iterative processes for solving singular unstructured systems of linear equations. SIAM Rev. 32, 611–635 (1990) 133. L. Debnath, P. Mikusi´nski, Hilbert Spaces with Applications, 2nd edn. (Academic, San Diego, 1999) 134. A.R. De Pierro, A.N. Iusem, A simultaneous projections method for linear inequalities. Lin. Algebra Appl. 64, 243–253 (1985) 135. A.R. De Pierro, A.N. Iusem, A parallel projection method of finding a common point of a family of convex sets. Pesquisa Operacional 5, 1–20 (1985) 136. A.R. De Pierro, A.N. Iusem, A finitely convergent “row-action” method for the convex feasibility problem. Appl. Math. Optim. 17, 225–235 (1988) 137. A.R. De Pierro, A.N. Iusem, On the asymptotic behavior of some alternate smoothing series expansion iterative methods. Lin. Algebra Appl. 130, 3–24 (1990) 138. F. Deutsch, in Applications of von Neumann’s Alternating Projections Algorithm, ed. by P. Kenderov. Mathematical Methods in Operations Research (Sophia, Bulgaria, 1983), pp. 44–51 139. F. Deutsch, in The Method of Alternating Orthogonal Projections, ed. by S.P. Singh. Approximation Theory, Spline Functions and Applications (Kluwer Academic, The Netherlands, 1992), pp. 105–121 140. F. Deutsch, Best Approximation in Inner Product Spaces (Springer, New York, 2001) 141. F. Deutsch, in Accelerating the Convergence of the Method of Alternating Projections via a Line Search: A Brief Survey, ed. by D. Butnariu, Y. Censor, S. Reich. Inherently Parallel Algorithms in Feasibility and Optimization and their Application, Studies in Computational Mathematics, vol. 8 (Elsevier Science, Amsterdam, 2001), pp. 203–217 142. F. Deutsch, H. Hundal, The rate of convergence for the cyclic projections algorithm, I. Angles between convex sets. J. Approx. Theor. 142, 36–55 (2006) 143. F. Deutsch, H. Hundal, The rate of convergence for the cyclic projections algorithm, II. Norms of nonlinear operators. J. Approx. Theor. 142, 56–82 (2006) 144. F. Deutsch, I. Yamada, Minimizing certain convex functions over the intersection of the fixed point sets of nonexpansive mappings. Numer. Funct. Anal. Optim. 19, 33–56 (1998) 145. J.B. Diaz, F.T. Metcalf, On the set of subsequential limit points of successive approximations. Trans. Am. Math. Soc. 135, 459–485 (1969) 146. L.T. Dos Santos, A parallel subgradient projections method for the convex feasibility problem. J. Comput. Appl. Math. 18, 307–320 (1987) 147. W.G. Dotson Jr., On the Mann iterative process. Trans. Am. Math. Soc. 149, 65–73 (1970)

References

281

148. W.G. Dotson, Fixed points of quasi-nonexpansive mappings. J. Austral. Math. Soc. 13, 167– 170 (1972) 149. J. Douglas, H.H. Rachford, On the numerical solution of heat conduction problems in two or three space variables. Trans. Am. Math. Soc. 82, 421–439 (1956) 150. R. Dudek, Iterative method for solving the linear feasibility problem. J. Optim Theor. Appl. 132, 401–410 (2007) 151. J. Dye, M.A. Khamsi, S. Reich, Random products of contractions in Banach spaces. Trans. Am. Math. Soc. 325, 87–99 (1991) 152. J.M. Dye, S. Reich, On the unrestricted iteration of projections in Hilbert space. J. Math. Anal. Appl. 156, 101–119 (1991) 153. J. Dye, S. Reich, Unrestricted iterations of nonexpansive mappings in Hilbert space. Nonlinear Anal. 18, 199–207 (1992) 154. R. Dylewski, Selection of Linearizations in Projection Methods for Convex Optimization Problems (in Polish), Ph.D. thesis, University of Zielona G´ora, Institute of Mathematics, 2003 155. R. Dylewski, Projection method with residual selection for linear feasibility problems. Discuss. Math. Differ. Incl. Contr. Optim. 27, 43–50 (2007) 156. M.G. Eberle, M.C. Maciel, Finding the closest Toeplitz matrix. Computat. Appl. Math. 22, 1–18 (2003) 157. J. Eckstein, D.P. Bertsekas, On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program. 55, 293–318 (1992) 158. I. Ekeland, R. T´emam, Convex Analysis and Variational Problems (North-Holland, Amsterdam, 1976) 159. T. Elfving, A projection method for semidefinite linear systems and its applications. Lin. Algebra Appl. 391, 57–73 (2004) 160. T. Elfving, T. Nikazad, Stopping rules for Landweber-type iteration. Inverse Probl. 23, 1417– 1432 (2007) 161. V. Elser, I. Rankenburg, P. Thibault, Searching with iterated maps. Proc. Natl. Acad. Sci. USA 104, 418–423 (2007) 162. L. Elsner, I. Koltracht, P. Lancaster, Convergence properties of ART and SOR algorithms. Numer. Math. 59, 91–106 (1991) 163. L. Elsner, I. Koltracht, M. Neumann, On the convergence of asynchronous paracontractions with application to tomographic reconstruction from incomplete data. Lin. Algebra Appl. 130, 65–82 (1990) 164. L. Elsner, I. Koltracht, M. Neumann, Convergence of sequential and asynchronous nonlinear paracontractions. Numer. Math. 62, 305–319 (1992) 165. F. Facchinei, J.-S. Pang, Finite-Dimensional Variational Inequalities and Complementarity Problems, Volume I, Volume II (Springer, New York, 2003) 166. M. Fiedler, V. Pt´ak, On matrices with non-positive off-diagonal elements and positive principal minors. Czech. Math. J. 12, 382–400 (1962) 167. S. Fitzpatrick, R.R. Phelps, Differentiability of the metric projection in Hilbert space. Trans. Am. Math. Soc. 270, 483–501 (1982) 168. S.D. Fl˚am, J. Zowe, Relaxed outer projections, weighted averages and convex feasibility. BIT 30, 289–300 (1990) 169. R. Fletcher, Practical Methods of Optimization (Wiley, Chichester, 1987) 170. K. Friedricks, On certain inequalities and characteristic value problems for analytic functions and for functions of two variables. Trans. Am. Math. Soc. 41, 321–364 (1937) 171. M. Fukushima, A relaxed projection method for variational inequalities. Math. Program. 35, 58–70 (1986) 172. E.M. Gafni, D.P. Bertsekas, Two metric projection methods for constrained optimization. SIAM J. Contr. Optim. 22, 936–964 (1984) 173. A. Gal´antai, Projectors and Projection Methods (Kluwer Academic, Boston, 2004) 174. A. Gal´antai, On the rate of convergence of the alternating projection method in finite dimensional spaces. J. Math. Anal. Appl. 310, 30–44 (2005)

282

References

175. U. Garc´ıa-Palomares, Parallel projected aggregation methods for solving the convex feasibility problem. SIAM J. Optim. 3, 882–900 (1993) 176. U. Garc´ıa-Palomares, A superlinearly convergent projection algorithm for solving the convex inequality problem. Oper. Res. Lett. 22, 97–103 (1998) 177. W.B. Gearhart, M. Koshy, Acceleration schemes for the method of alternating projections. J. Comput. Appl. Math. 26, 235–249 (1989) 178. C. Geiger, Ch. Kanzow, Numerische Verfahren zur L¨osung unrestingierter Optimierungsaufgaben (Springer, Berlin, 1999) 179. C. Geiger, Ch. Kanzow, Theorie und Numerik restringierter Optimierungsaufgaben (Springer, Berlin, 2002) 180. J.R. Giles, Convex Analysis with Application in Differentiation of Convex Functions (Pitman Advanced Publishing Program, Boston, 1982) 181. P.E. Gill, W. Murray, M.H. Wright, Numerical Linear Algebra and Optimization (AddisonWesley, Redwood City, 1991) 182. W. Glunt, T.L. Hayden, R. Reams, The nearest ‘doubly stochastic’ matrix to a real matrix with the same first moment. Numer. Lin. Algebra Appl. 5, 475–482 (1998) 183. K. Goebel, Concise Course on Fixed Points Theorems (Yokohama Publishing, Yokohama, 2002); Polish translation: Twierdzenia o punktach stałych (Wydawnictwo UMCS, Lublin, 2005) 184. K. Goebel, W.A. Kirk, Topics in Metric Fixed Point Theory (Cambridge University Press, Cambridge, 1990); Polish translation: Zagadnienia metrycznej teorii punkt´ow stałych (Wydawnictwo UMCS, Lublin, 1999) 185. K. Goebel, S. Reich, Uniform Convexity, Hyperbolic Geometry, and Nonexpansive Mappings (Marcel Dekker, New York, 1984) 186. J.L. Goffin, The relaxation method for solving systems of linear inequalities. Math. Oper. Res. 5, 388–414 (1980) 187. J.L. Goffin, On the finite convergence of the relaxation method for solving systems of inequalities. Operations Research Center, Report ORC 71–36, University of California, Berkeley, 1971 188. D. G¨ohde, Zum Prinzip der kontraktiven Abbildung. Math. Nachr. 30, 251–258 (1965) 189. R. Gordon, R. Bender, G.T. Herman, Algebraic reconstruction techniques (ART) for threedimensional electron microscopy and X-ray photography. J. Theoret. Biol. 29, 471–481 (1970) 190. D. Gordon, R. Gordon, Component-averaged row projections: a robust, block-parallel scheme for sparse linear systems. SIAM J. Sci. Comput. 27, 1092–1117 (2005) 191. A. Granas, J. Dugundji, Fixed Point Theory (Springer, New York, 2003) 192. K.M. Grigoriadis, A.E. Frazho, R.E. Skelton, Application of alternating convex projection methods for computation of positive Toeplitz matrices. IEEE Trans. Signal Process. 42, 1873– 1875 (1994) 193. K.M. Grigoriadis, R.E. Skelton, Low-order control design for LMI problems using alternating projection methods. Automatica 32, 1117–1125 (1996) 194. K.M. Grigoriadis, R.E. Skelton, Alternating convex projection methods for discrete-time covariance control design. J. Optim. Theor. Appl. 88, 399–432 (1996) 195. J. Gu, H. Stark, Y. Yang, Wide-band smart antenna design using vector space projection methods. IEEE Trans. Antenn. Propag. 52, 3228–3236 (2004) 196. L.G. Gurin, B.T. Polyak, E.V. Raik, The method of projection for finding the common point in convex sets. Zh. Vychisl. Mat. Mat. Fiz. 7, 1211–1228 (1967) (in Russian); English translation in: USSR Comput. Math. Phys. 7, 1–24 (1967) 197. R. Haller, R. Szwarc, Kaczmarz algorithm in Hilbert space. Studia Math. 169, 123–132 (2005) 198. I. Halperin, The product of projection operators. Acta Sci. Math. (Szeged) 23, 96–99 (1962) 199. H.W. Hamacher, K.-H. K¨ufer, Inverse radiation therapy planning – a multiple objective optimization approach. Discrete Appl. Math. 118, 145–161 (2002) 200. S.-P. Han, A successive projection method. Math. Program. (Ser. A) 40, 1–14 (1988) 201. M. Hanke, W. Niethammer, On the acceleration of Kaczmarz’s method for inconsistent linear systems. Lin. Algebra Appl. 130, 83–98 (1990)

References

283

202. Y. Haugazeau, Sur les in´equations variationnelles et la minimisation de fonctionnelles convexes (Th`ese, Universit´e de Paris, Paris, 1968) 203. H. He, S. Liu, H. Zhou, An explicit method for finding common solutions of variational inequalities and systems of equilibrium problems and fixed points of an infinite family of nonexpansive mappings. Nonlinear Anal. 72, 3124–3135 (2010) 204. G.T. Herman, A relaxation method for reconstructing objects from noisy X-rays. Math. Program. 8, 1–19 (1975) 205. G.T. Herman, Fundamentals of Computerized Tomography: Image Reconstruction from Projections, 2nd edn. (Springer, London, 2009) 206. G.T. Herman, W. Chen, A fast algorithm for solving a linear feasibility problem with application to intensity-modulated radiation therapy. Lin. Algebra Appl. 428, 1207–1217 (2008) 207. N.J. Higham, Computing a nearest symmetric positive semidefinite matrix. Lin. Algebra Appl. 103, 103–118 (1988) 208. N.J. Higham, Computing the nearest correlation matrix – a problem from finance. IMA J. Numer. Anal. 22, 329–343 (2002) 209. J.-B. Hiriart-Urruty, C. Lemar´echal, Convex Analysis and Minimization Algorithms, Vol I, Vol II (Springer, Berlin, 1993) 210. J.-B. Hiriart-Urruty, C. Lemar´echal, Fundamentals of Convex Analysis (Springer, Berlin, 2001) 211. S.A. Hirstoaga, Iterative selection methods for common fixed point problems. J. Math. Anal. Appl. 324, 1020–1035 (2006) 212. J. H¨offner, P. Decker, E.L. Schmidt, W. Herbig, J. Rittler, P. Weiß, Development of a fast optimization preview in radiation treatment planning. Strahlentherapie und Onkologie 172, 384–394 (1996) 213. H.S. Hundal, An alternating projection that does not converge in norm. Nonlinear Anal. 57, 35–61 (2004) 214. J.K. Hunter, B. Nachtergaele, Applied Analysis (World Scientific, Singapore, 2000) 215. A.N. Iusem, A.R. De Pierro, Convergence results for an accelerated nonlinear Cimmino algorithm. Numer. Math. 49, 367–378 (1986) 216. A.N. Iusem, A.R. De Pierro, A simultaneous iterative method for computing projections on polyhedra. SIAM J. Contr. Optim. 25, 231–243 (1987) 217. A.N. Iusem, A.R. De Pierro, On the convergence properties of Hildreth’s quadratic programming algorithm. Math. Program. (Ser. A) 47, 37–51 (1990) 218. A.N. Iusem, B.F. Svaiter, A row-action method for convex programming. Math. Program. 64, 149–171 (1994) 219. B.K. Jennison, J.P. Allebach, D.W. Sweeney, Iterative approaches to computer-generated holography. Opt. Eng. 28, 629–637 (1989) 220. M. Jiang, G. Wang, Development of iterative algorithms for image reconstruction. J. X-Ray Sci. Tech. 10, 77–86 (2002) 221. M. Jiang, G. Wang, Convergence studies on iterative algorithms for image reconstruction. IEEE Trans. Med. Imag. 22, 569–579 (2003) 222. B. Johansson, T. Elfving, V. Kozlov, Y. Censor, P.-E. Forss´en, G. Granlund, The application of an oblique-projected Landweber method to a model of supervised learning. Math. Comput. Model. 43, 892–909 (2006) 223. S. Kaczmarz, Angen¨aherte Aufl¨osung von Systemen linearer Gleichungen. Bulletin International de l’Acad´emie Polonaise des Sciences et des Lettres A35, 355–357 (1937); English translation: S. Kaczmarz, Approximate solution of systems of linear equations. Int. J. Contr. 57, 1269–1271 (1993) 224. A.C. Kak, M. Slaney, Principles of Computerized Tomographic Imaging (IEEE, New York, 1988) 225. I.G. Kazantsev, S. Schmidt, H.F. Poulsen, A discrete spherical x-ray transform of orientation distribution functions using bounding cubes. Inverse Probl. 25, 105009 (2009)

284

References

226. D. Kinderlehrer, G. Stampacchia, An Introduction to Variational Inequalities and Their Applications (Academic, New York, 1980) 227. W.A. Kirk, A fixed point theorem for mappings which do not increase distances. Am. Math. Mon. 72, 1004–1006 (1965) 228. Yu.N. Kiseliov, Algorithms of projection of a point onto an ellipsoid. Lithuanian Math. J. 34, 141–159 (1994) 229. K.C. Kiwiel, Block-iterative surrogate projection methods for convex feasibility problems. Lin. Algebra Appl. 215, 225–259 (1995) 230. K.C. Kiwiel, The efficiency of subgradient projection methods for convex optimization. I. General level methods. SIAM J. Contr. Optim. 34, 660–676 (1996) 231. K.C. Kiwiel, The efficiency of subgradient projection methods for convex optimization, II. Implementations and extensions. SIAM J. Contr. Optim. 34, 677–697 (1996) 232. K.C. Kiwiel, Monotone Gram matrices and deepest surrogate inequalities in accelerated relaxation methods for convex feasibility problems. Lin. Algebra Appl. 252, 27–33 (1997) 233. K.C. Kiwiel, B. Łopuch, Surrogate projection methods for finding fixed points of firmly nonexpansive mappings. SIAM J. Opt. 7, 1084–1102 (1997) 234. A. Kiełbasi´nski, H. Schwetlick, Numerical Linear Algebra (in German) (Verlag Harri Deutsch, Thun, 1988); Polish translation: Numeryczna algebra liniowa (WNT, Warszawa, 1992) 235. E. Kopeck´a, S. Reich, A note on the von Neumann alternating projections algorithm. J. Nonlinear Convex Anal. 5, 379–386 (2004) 236. E. Kopeck´a, S. Reich, Another note on the von Neumann alternating projections algorithm. J. Nonlinear Convex Anal. 11, 455–460 (2010) 237. G.M. Korpelevich, The extragradient method for finding saddle points and other problems. Ekonomika i Matematicheskie Metody 12, 747–756 (1976) 238. M.A. Krasnosel’ski˘ı, Two remarks on the method of successive approximations (in Russian). Uspehi Mat. Nauk 10, 123–127 (1955) 239. S. Kwapie´n, J. Mycielski, On the Kaczmarz algorithm of approximation in infinitedimensional spaces. Studia Math. 148, 5–86 (2001) 240. L. Landweber, An iteration formula for Fredholm integral equations of the first kind. Am. J. Math. 73, 615–624 (1951) 241. S. Lee, P.S. Cho, R.J. Marks, S.Oh, Conformal radiotherapy computation by the method of alternating projections onto convex sets. Phys. Med. Biol. 42, 1065–1086 (1997) 242. S.-H. Lee, K.-R. Kwon, Mesh watermarking based projection onto two convex sets. Multimedia Syst. 13, 323–330 (2008) 243. A. Lent, in A Convergent Algorithm for Maximum Entropy Image Restoration with a Medical X-ray Application, ed. by R. Shaw. Image Analysis and Evaluation (SPSE, Washington DC), pp. 249–257 244. A. Lent, Y. Censor, Extensions of Hildreth’s row-action method for quadratic programming. SIAM J. Contr. Optim. 18, 444–454 (1980) 245. A.W.-C. Liew, H. Yan, N.-F. Law, POCS-based blocking artifacts suppression using a smoothness constraint set with explicit region modeling. IEEE Trans. Circ. Syst. Video Tech. 15, 795–800 (2005) 246. P.L. Lions, B. Mercier, Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16, 964–979 (1979) 247. C. Liu, An acceleration scheme for row projection methods. J. Comput. Appl. Math. 57, 363– 391 (1995) 248. Y.M. Lu, M. Karzand, M. Vetterli, Demosaicking by alternating projections: theory and fast one-step implementation. IEEE Trans. Image Process. 19, 2085–2098 (2010) 249. P.-E. Maing´e, Inertial iterative process for fixed points of certain quasi-nonexpansive mappings. Set-Valued Anal. 15, 67–79 (2007) 250. P.-E. Maing´e, Extension of the hybrid steepest descent method to a class of variational inequalities and fixed point problems with nonself-mappings. Numer. Funct. Anal. Optim. 29, 820–834 (2008)

References

285

251. P.-E. Maing´e, New approach to solving a system of variational inequalities and hierarchical problems. J. Optim. Theor. Appl. 138, 459–477 (2008) 252. W.R. Mann, Mean value methods in iteration. Proc. Am. Math. Soc. 4, 506–510 (1953) 253. S¸t. M˘arus¸ter, The solution by iteration of nonlinear equations in Hilbert spaces. Proc. Am. Math. Soc. 63, 69–73 (1977) 254. S¸t. M˘arus¸ter, Quasi-nonexpansivity and the convex feasibility problem. An. S¸tiint¸. Univ. Al. I. Cuza Ias¸i Inform. (N.S.) 15, 47–56 (2005) 255. S¸t. M˘arus¸ter, C. Popˆırlan, On the Mann-type iteration and the convex feasibility problem. J. Comput. Appl. Math. 212, 390–396 (2008) 256. S¸t. M˘arus¸ter, C. Popˆırlan, On the regularity condition in a convex feasibility problem. Nonlinear Anal. 70, 1923–1928 (2009) 257. E. Masad, S. Reich, A note on the multiple-set split convex feasibility problem in Hilbert space. J. Nonlinear Convex Anal. 8, 367–371 (2007) 258. E. Matouˇskov´a, S. Reich, The Hundal example revisited. J. Nonlinear Convex Anal. 4, 411– 427 (2003) 259. S.F. McCormick, The methods of Kaczmarz and row orthogonalization for solving linear equations and least squares problems in Hilbert space. Indiana Univ. Math. J. 26, 1137–1150 (1977) 260. Yu.I. Merzlyakov, On a relaxation method of solving systems of linear inequalities (in Russian). Zh. Vychisl. Mat. Mat. Fiz. 2, 482–487 (1962) 261. D. Michalski, Y. Xiao, Y. Censor, J.M. Galvin, The dose-volume constraint satisfaction problem for inverse treatment planning with field segments. Phys. Med. Biol. 49, 601–616 (2004) 262. W. Mlak, Introduction to Hilbert Spaces (in Polish) (PWN, Warsaw, 1982) 263. W. Mlak, Hilbert Spaces and Operator Theory (Kluwer Academic, Boston, 1991) 264. J.-J. Moreau, D´ecomposition orthogonale d’un espace hilbertien selon deux cˆones mutuellement polaires. C. R. Acad. Sci. Paris 255, 238–240 (1962) 265. J. Moreno, B. Datta, M. Raydan, A symmetry preserving alternating projection method for matrix model updating. Mech. Syst. Signal Process. 23, 1784–1791 (2009) 266. T.S. Motzkin, I.J. Schoenberg, The relaxation method for linear inequalities. Can. J. Math. 6, 393–404 (1954) 267. J. Musielak, Introduction to Functional Analysis (in Polish) (PWN, Warszawa, 1989) ´ 268. J. Mycielski, S. Swierczkowski, Uniform approximation with linear combinations of reproducing kernels. Studia Math. 121, 105–114 (1996) 269. N. Nadezhkina, W. Takahashi, Strong convergence theorem by a hybrid method for nonexpansive mappings and Lipschitz-continuous monotone mappings. SIAM J. Optim. 16, 1230–1241 (2006) 270. F. Natterer, The Mathematics of Computerized Tomography (Wiley, Chichester, 1986) 271. J. von Neumann, in Functional Operators – Vol. II. The Geometry of Orthogonal Spaces. Annals of Mathematics Studies, vol. 22 (Princeton University Press, Princeton, 1950) (Reprint of mimeographed lecture notes first distributed in 1933) 272. O. Nevanlinna, S. Reich, Strong convergence of contraction semigroups and of iterative methods for accretive operators in Banach spaces. Israel J. Math. 32, 44–58 (1979) 273. N. Ogura, I. Yamada, Non-strictly convex minimization over the fixed point set of an asymptotically shrinking nonexpansive mapping. Numer. Funct. Anal. Optim. 23, 113–137 (2002) 274. N. Ogura, I. Yamada, Nonstrictly convex minimization over the bounded fixed point set of a nonexpansive mapping. Numer. Funct. Anal. Optim. 24, 129–135 (2003) 275. S. Oh, R.J. Marks, L.E. Atlas, Kernel synthesis for generalized time-frequency distributions using the method of alternating projections onto convex sets. IEEE Trans. Signal Process. 42, 1653–1661 (1994) 276. J.G. O’Hara, P. Pillay, H.-K. Xu, Iterative approaches to finding nearest common fixed points of nonexpansive mappings in Hilbert spaces. Nonlinear Anal. 54, 1417–1426 (2003) 277. S.O. Oko, Surrogate methods for linear inequalities. J. Optim. Theor. Appl. 72, 247–268 (1992)

286

References

278. Z. Opial, Nonexpansive and Monotone Mappings in Banach Spaces. Lecture Notes 67-1, Center for Dynamical Systems, Brown University, Providence, RI, 1967 279. Z. Opial, Weak convergence of the sequence of successive approximations for nonexpansive mappings. Bull. Am. Math. Soc. 73, 591–597 (1967) 280. S.C. Park, M.K. Park, M.G. Kang, Super-resolution image reconstruction: A technical overview. IEEE Signal Process. Mag. 20, 21–36 (2003) 281. J. Park, D.C. Park, R.J. Marks, M. El-Sharkawi, Recovery of image blocks using the method of alternating projections. IEEE Trans. Image Process. 14, 461–474 (2005) 282. S.N. Penfold, R.W. Schulte, Y. Censor, A.B. Rosenfeld, Total variation superiorization schemes in proton computed tomography image reconstruction. Med. Phys. 37, 5887–5895 (2010) 283. W.V. Petryshyn, T.E. Williamson Jr., Strong and weak convergence of the sequence of successive approximations for quasi-nonexpansive mappings. J. Math. Anal. Appl. 43, 459– 497 (1973) 284. G. Pierra, Decomposition through formalization in a product space. Math. Program. 28, 96– 115 (1984) 285. C. Popa, Least-squared solution of overdetermined inconsistent linear systems using Kaczmarz’s relaxation. Int. J. Comp. Math. 55, 79–89 (1995) 286. C. Popa, Extensions of block-projections methods with relaxation parameters to inconsistent and rank-deficient least-squares problems. BIT 38, 151–176 (1998) 287. C. Popa, R. Zdunek, Kaczmarz extended algorithm for tomographic image reconstruction from limited data. Math. Comput. Simulat. 65, 579–598 (2004) 288. S. Prasad, Generalized array pattern synthesis by the method of alternating orthogonal projections. IEEE Trans. Antenn. Propag. 28, 328–332 (1980) 289. J.L. Prince, A.S. Willsky, A geometric projection-space reconstruction algorithm. Lin. Algebra Appl. 130, 151–191 (1990) 290. E. Pustylnik, S. Reich, A.J. Zaslavski, Convergence of infinite products of nonexpansive operators in Hilbert space. J. Nonlinear Convex Anal. 11, 461–474 (2010) 291. E. Pustylnik, S. Reich, A.J. Zaslavski, Convergence of non-cyclic infinite products of operators. J. Math. Anal. Appl. 380, 759–767 (2011) 292. B. Qu, N. Xiu, A note on the CQ algorithm for the split feasibility problem. Inverse Probl. 21, 1655–1665 (2005) 293. B. Qu, N. Xiu, A new halfspace-relaxation projection method for the split feasibility problem. Lin. Algebra Appl. 428, 1218–1229 (2008) 294. S. Reich, Weak convergence theorems for nonexpansive mappings in Banach spaces. J. Math. Anal. Appl. 67, 274–276 (1979) 295. S. Reich, A limit theorem for projections. Lin. Multilinear Algebra 13, 281–290 (1983) 296. S. Reich and I. Shafrir, The asymptotic behavior of firmly nonexpansive mappings. Proc. Am. Math. Soc. 101, 246–250 (1987) 297. S. Reich, A.J. Zaslavski, Attracting mappings in Banach and hyperbolic spaces. J. Math. Anal. Appl. 253, 250–268 (2001) 298. R.T. Rockafellar, Convex Analysis (Princeton University Press, Princeton, 1970) 299. R.T. Rockafellar, Monotone operators and the proximal point algorithm. SIAM J. Contr. Optim. 14, 877–898 (1976) 300. W. Rudin, Functional Analysis, 2nd edn. (McGraw-Hill, New York, 1991); Polish translation: Analiza funkcjonalna (PWN, Warszawa, 2002) 301. A.A. Samsonov, E.G. Kholmovski, D.L. Parker, C.R. Johnson, POCSENSE: POCS-based reconstruction for sensitivity encoded magnetic resonance imaging. Magn. Reson. Med. 52, 1397–1406 (2004) 302. J. Schauder, Der Fixpunktsatz in Funktionalr¨aumen. Studia Math. 2, 171–180 (1930) 303. D. Schott, A general iterative scheme with applications to convex optimization and related fields. Optimization 22, 885–902 (1991) 304. H.D. Scolnik, N. Echebest, M.T. Guardarucci, M.C. Vacchino, A class of optimized row projection methods for solving large nonsymmetric linear systems. Appl. Numer. Math. 41, 499–513 (2002)

References

287

305. H.D. Scolnik, N. Echebest, M.T. Guardarucci, M.C. Vacchino, Acceleration scheme for parallel projected aggregation methods for solving large linear systems. Ann. Oper. Res. 117, 95–115 (2002) 306. H.D. Scolnik, N. Echebest, M.T. Guardarucci, M.C. Vacchino, Incomplete oblique projections for solving large inconsistent linear systems. Math. Program. 111, 273–300 (2008) 307. A. Segal, Directed Operators for Common Fixed Point Problems and Convex Programming Problems, Ph.D. Thesis, University of Haifa, Haifa, Israel, 2008 308. H.F. Senter, W.G. Dotson Jr., Approximating fixed points of nonexpansive mappings. Proc. Am. Math. Soc. 44, 375–380 (1974) 309. A. Serbes, L. Durak, Optimum signal and image recovery by the method of alternating projections in fractional Fourier domains. Comm. Nonlinear Sci. Numer. Simulat. 15, 675– 689 (2010) 310. N.T. Shaked, J. Rosen, Multiple-viewpoint projection holograms synthesized by spatially incoherent correlation with broadband functions. J. Opt. Soc. Am. A 25, 2129–2138 (2008) 311. G. Sharma, Set theoretic estimation for problems in subtractive color. Color Res. Appl. 25, 333–348 (2000) 312. K.K. Sharma, S.D. Joshi, Extrapolation of signals using the method of alternating projections in fractional Fourier domains. Signal Image Video Process. 2, 177–182 (2008) 313. K.T. Smith, D.C. Solman, S.L. Wagner, Practical and mathematical aspects of the problem of reconstructing objects from radiographs. Bull. Am. Math. Soc. 83, 1227–1270 (1977) 314. R.A. Soni, K.A. Gallivan, W.K. Jenkins, Low-complexity data reusing methods in adaptive filtering. IEEE Trans. Signal Process. 52, 394–405 (2004) 315. H. Stark, P. Oskoui, High resolution image recovery from image-plane arrays, using convex projections. J. Opt. Soc. Am. A 6, 1715–1726 (1989) 316. H. Stark, Y. Yang, Vector Space Projections. A Numerical Approach to Signal and Image Processing, Neural Nets and Optics (Wiley, New York, 1998) 317. J. Stoer, R. Bulirsch, Introduction to Numerical Analysis, 3rd edn. (Springer, New York, 2002) 318. C. Sudsukh, Strong convergence theorems for fixed point problems, equilibrium problems and applications. Int. J. Math. Anal. (Ruse) 3, 1867–1880 (2009) ´ 319. S. Swierczkowski, A model of following. J. Math. Anal. Appl. 222, 547–561 (1998) 320. W. Takahashi, Y. Takeuchi, R. Kubota, Strong convergence theorems by hybrid methods for families of nonexpansive mappings in Hilbert spaces. J. Math. Anal. Appl. 341, 276–286 (2008) 321. K. Tanabe, Projection method for solving a singular system of linear equations and its applications. Numer. Math. 17, 203–214 (1971) 322. K. Tanabe, Characterization of linear stationary iterative processes for solving a singular system of linear equations. Numer. Math. 22, 349–359 (1974) 323. G. Tetzlaff, K. Arnold, A. Raabe, A. Ziemann, Observations of area averaged near-surface wind- and temperature-fields in real terrain using acoustic travel time tomography. Meteorologische Zeitschrift 11, 273–283 (2002) 324. Ch. Thieke, T. Bortfeld, A. Niemierko, S. Nill, From physical dose constraints to equivalent uniform dose constraints in inverse radiotherapy planning. Med. Phys. 30, 2332–2339 (2003) 325. J. van Tiel, Convex Analysis, An Introductory Text (Wiley, Chichester, 1984) 326. M.J. Todd, Some Remarks on the Relaxation Method for Linear Inequalities, Technical Report, vol. 419, Cornell University, Cornell, Ithaca, 1979 327. Ph.L. Toint, Global convergence of a class of trust region methods for nonconvex minimization in Hilbert space. IMA J. Numer. Anal. 8, 231–252 (1988) 328. W. Treimer, U. Feye-Treimer, Two dimensional reconstruction of small angle scattering patterns from rocking curves. Physica B 241–243, 1228–1230 (1998) 329. J.A. Tropp, I.S. Dhillon, R.W. Heath, T. Strohmer, Designing structured tight frames via an alternating projection method. IEEE Trans. Inform. Theor. 51, 188–209 (2005) 330. M.R. Trummer, SMART – an algorithm for reconstructing pictures from projections. J. Appl. Math. Phys. 34, 746–753 (1983)

288

References

331. P. Tseng, On the convergence of the products of firmly nonexpansive mappings. SIAM J. Optim. 2, 425–434 (1992) 332. A. Van der Sluis, H.A. Van der Vorst, in Numerical Solution of Large Sparse Linear Algebraic Systems Arising from Tomographic Problems, ed. by G. Nolet. Seismic Tomography (Reidel, Dordrecht, 1987) 333. V.V. Vasin, A.L. Ageev, Ill-Posed Problems with A Priori Information (VSP, Utrecht, 1995) 334. S. Webb, Intensity Modulated Radiation Therapy (Institute of Physics Publishing, Bristol, 2001) 335. S. Webb, The Physics of Conformal Radiotherapy (Institute of Physics Publishing, Bristol, 2001) 336. R. Webster, Convexity (Oxford University Press, Oxford, 1994) 337. R. Wegmann, Conformal mapping by the method of alternating projections. Numer. Math. 56, 291–307 (1989) 338. R. Wittmann, Approximation of fixed points of nonexpansive mappings. Arch. Math. 58, 486–491 (1992) 339. P. Wolfe, Finding the nearest point in a polytope. Math. Program. 11, 128–149 (1976) 340. B.J. van Wyk, M.A. van Wyk, A POCS-based graph matching algorithm. IEEE Trans. Pattern Anal. Mach. Intell. 26, 1526–1530 (2004) 341. Y. Xiao, Y. Censor, D. Michalski, J.M. Galvin, The least-intensity feasible solution for aperture-based inverse planning in radiation therapy. Ann. Oper. Res. 119, 183–203 (2003) 342. H.-K. Xu, Iterative algorithms for nonlinear operators. J. Lond. Math. Soc. 66, 240–256 (2002) 343. H.-K. Xu, An iterative approach to quadratic optimization. J. Optim. Theor. Appl. 116, 659– 678 (2003) 344. H.-K. Xu, A variable Krasnosel’ski˘ı-Mann algorithm and the multiple-set split feasibility problem. Inverse Probl. 22, 2021–2034 (2006) 345. I. Yamada, in The Hybrid Steepest Descent Method for the Variational Inequality Problem Over the Intersection of Fixed Point Sets of Nonexpansive Mappings, ed. by D. Butnariu, Y. Censor, S. Reich. Inherently Parallel Algorithms in Feasibility and Optimization and their Applications, Studies in Computational Mathematics, vol. 8 (Elsevier Science, Amsterdam, 2001), pp. 473–504 346. I. Yamada, N. Ogura, Hybrid steepest descent method for variational inequality problem over the fixed point set of certain quasi-nonexpansive mappings. Numer. Funct. Anal. Optim. 25, 619–655 (2004) 347. I. Yamada, N. Ogura, Adaptive projected subgradient method for asymptotic minimization of sequence of nonnegative convex functions. Numer. Funct. Anal. Optim. 25, 593–617 (2004) 348. I. Yamada, N. Ogura, N. Shirakawa, in A Numerically Robust Hybrid Steepest Descent Method for the Convexly Constrained Generalized Inverse Problems, ed. by Z. Nashed, O. Scherzer. Inverse Problems, Image Analysis and Medical Imaging, American Mathematical Society, Contemp. Math., vol. 313 (2002), pp. 269–305 349. I. Yamada, N. Ogura, Y. Yamashita, K. Sakaniwa, Quadratic optimization of fixed points of nonexpansive mappings in Hilbert space. Numer. Funct. Anal. Optim. 19, 165–190 (1998) 350. Q.Z. Yang, The relaxed CQ algorithm solving the split feasibility problem. Inverse Probl. 20, 1261–1266 (2004) 351. K. Yang, K.G. Murty, New iterative methods for linear inequalities. JOTA 72, 163–185 (1992) 352. Q. Yang, J. Zhao, Generalized KM theorems and their applications. Inverse Probl. 22, 833– 844 (2006) 353. Y. Yao, Y.-C. Liou, Weak and strong convergence of Krasnoselski–Mann iteration for hierarchical fixed point problems. Inverse Probl. 24, 015015, 8 (2008) 354. D. Youla, Generalized image restoration by the method of alternating orthogonal projections. IEEE Trans. Circ. Syst. 25, 694–702 (1978) 355. M. Yukawa, I. Yamada, Pairwise optimal weight realization – Acceleration technique for settheoretic adaptive parallel subgradient projection algorithm. IEEE Trans. Signal Process. 54, 4557–4571 (2006)

References

289

356. M. Zaknoon, Algorithmic Developments for the Convex Feasibility Problem, Ph.D. Thesis, University of Haifa, Haifa, Israel, 2003 357. E.H. Zarantonello, in Projections on Convex Sets in Hilbert Space and Spectral Theory, ed. by E.H. Zarantonello. Contributions to Nonlinear Functional Analysis (Academic, New York, 1971), pp. 237–424 358. E. Zeidler, Nonlinear Functional Analysis and Its Applications, III – Variational Methods and Optimization (Springer, New York, 1985) 359. J. Zhang, A.K. Katsaggelos, in Image Recovery Using the EM Algorithm, ed. by V.K. Madisetti, D.B. Williams. Digital Signal Processing Handbook (CRC Press LLC, Boca Raton, 1999) 360. D.F. Zhao, The principles and practice of iterative alternating projection algorithm: Solution for non-LTE stellar atmospheric model with the method of linearized separation. Chin. Astron. Astrophys. 25, 305–316 (2001) 361. J. Zhao, Q. Yang, Several solution methods for the split feasibility problem. Inverse Probl. 21, 1791–1799 (2005)

•

Glossary of Symbols

H hx; yi kxk hx; yiG p kxkG WD hx; xiG ^.x; y/ X I WD f1; 2; :::; mg xC ; x RnC ; Rn m jJ j V? B.x; / C0 bd C int C cl C Lin S aff S H.a; ˇ/ H .a; ˇ/ HC .a; ˇ/ fC ; f Argminx2X f .x/ argminx2X f .x/

Hilbert space Inner product of x; y 2 H Norm of x 2 H induced by h; i Inner product of x; y 2 Rn induced by a positive definite matrix G The norm of x 2 Rn induced by h; iG Angle between nonzero vectors x; y 2 H A nonempty closed convex subset of H Finite subset of indices Positive and the negative part of x 2 Rn Nonnegative and the nonpositive orthant Standard simplex in Rm The number of elements of a finite subset J Subspace orthogonal to a subspace V  H Ball with a centre x and radius > 0 Complement of a subset C  H Boundary of a subset C  H Interior of a subset C  H Closure of a subset C  H Linear subspace generated by S  H Affine subspace generated by S  H Hyperplane fx 2 H W ha; xi D ˇg Half-space fx 2 H W ha; xi  ˇg Half-space fx 2 H W ha; xi  ˇg Positive and the negative part of a function f Subset of minimizers of f W X ! R Minimizer of f W X ! R

A. Cegielski, Iterative Methods for Fixed Point Problems in Hilbert Spaces, Lecture Notes in Mathematics 2057, DOI 10.1007/978-3-642-30901-4, © Springer-Verlag Berlin Heidelberg 2012

291

292

S.f; ˛/ epi f f 0 .x; s/ Df , f 0 , DT rf .x/ r 2 f .x/ diag v A> AC cone S conv S ri C C NC .x/ TC .x/ @f .x/ gf .x/ PC Pa T Fix T d.; C / d.A; B/ L.H1 ; H2 / max .A/ Fej T Sep T

Glossary of Symbols

Sublevel set of a function f at a level ˛ 2 R Epigraph of a function f Directional derivative of a function f at x in a direction s Derivative of a function f or of an operator T Gradient of a function f at x Hessian of a function f at x Diagonal matrix with a vector v at the main diagonal Matrix transposed to a matrix A Moore–Penrose pseudoinverse of a matrix A Conical hull of a subset S  H Convex hull of a subset S  H Relative interior of a subset C  H Polar cone to C  H Normal cone to a convex subset C  H at x 2 H Tangent cone to a convex subset C  H at x 2 H Subdifferential of a function f at x 2 H Subgradient of a function f at x 2 H Metric projection onto a subset C  H Metric projection onto Linfag, where a 2 H Relaxation of an operator T Subset of fixed points of an operator T Distance function to a subset C  H Distance between subsets A; B  H Space of all bounded linear operators A W H1 ! H2 Largest eigenvalue of a nonnegative operator A W H ! H See page 46 See page 55

Glossary of Acronyms

AAR APM AR ART AV CFPP CFP CMP DR EMOPP EMOPSP EPPM ESC ESCM ESSPM FM FNE -FNE GCFP KKT-point LFP LLSP LM LSFP NE OCS

Averaged alternating reflection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 Alternating projection method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204 Asymptotically regular (operator) . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Algebraic reconstruction technique . . . . . . . . . . . . . . . . . . . . . . . . . . 220 Averaged (operator) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Common fixed point problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Convex feasibility problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Convex minimization problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Douglas–Rachford (algorithm) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212 Extrapolated method of parallel projections . . . . . . . . . . . . . . . . . . 257 Extrapolated method of parallel subgradient projections . . . . . . . . 257 Extrapolated parallel projection method . . . . . . . . . . . . . . . . . . . . . . 257 Extrapolated simultaneous cutter . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 Extrapolated simultaneous cutter method . . . . . . . . . . . . . . . . . . . . 254 Extrapolated simultaneous subgradient projection method . . . . . . 258 Fej´er monotone (operator) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Firmly nonexpansive (operator) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 -firmly nonexpansive (operator) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Generalized convex feasibility problem . . . . . . . . . . . . . . . . . . . . . . . 33 Karush–Kuhn–Tucker point . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Linear feasibility problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Linear least square problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Landweber method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 Linear split feasibility problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Nonexpansive (operator) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Obtuse cone selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

A. Cegielski, Iterative Methods for Fixed Point Problems in Hilbert Spaces, Lecture Notes in Mathematics 2057, DOI 10.1007/978-3-642-30901-4, © Springer-Verlag Berlin Heidelberg 2012

293

294

PLM QNE RFNE -RFNE ROCS RS SCM SiCM SFM SFP SNE SPM sQNE C -sQNE SQNE ˛-SQNE VIP

Glossary of Acronyms

Projected Landweber method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 Quasi-nonexpansive (operator) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Relaxed firmly nonexpansive (operator) . . . . . . . . . . . . . . . . . . . . . . . . 65 -relaxed firmly nonexpansive (operator) . . . . . . . . . . . . . . . . . . . . . . . 65 Regular obtuse cone selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 Residual selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 Surrogate constraints method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 Simultaneous cutter method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 Strongly Fej´er monotone (operator) . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 Split feasibility problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Strongly nonexpansive (operator) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 Simultaneous projection method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 Strictly quasi-nonexpansive (operator) . . . . . . . . . . . . . . . . . . . . . . . . . . 47 C -strictly quasi-nonexpansive (operator) . . . . . . . . . . . . . . . . . . . . . . . 47 Strongly quasi-nonexpansive (operator) . . . . . . . . . . . . . . . . . . . . . . . . 56 ˛-strongly quasi nonexpansive (operator) . . . . . . . . . . . . . . . . . . . . . . . 56 Variational inequality problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Index

.ı; /-algorithm, 252

actualization, 39 adjoint operator, 9 admissible control sequence, 225 admissible step size function, 255 affine hull, 13 affine operator, 11 affine subspace, 6 algebraic reconstruction technique, 220 algorithm, 39 algorithmic mapping, 39 algorithmic operator, 39 algorithmic projection operator, 129 almost cyclic control, 225 almost remotest set control, 224 alternating projection, 147 alternating projection method, 204 angle, 6 angle between subspaces, 208 angle between vectors, 6 appropriate weight function, 50 approximately regular control, 234 approximately regular control sequence, 251 approximately regular weight function, 234 approximately remotest set control, 223 approximately semi-regular control, 234 approximately semi-regular control sequence, 251 approximately semi-regular weight function, 234 approximately semi-remotest set control, 223 approximating sequence, 39 asymptotically regular sequence of operators, 116 operator, 111

attracting operator, 47 autonomous method, 105 autonomous projection method, 203 averaged alternating reflection, 161 averaged alternating reflection algorithm, 212 averaged operator, 74 averaged quasi-nonexpansive mapping, 54

ball, 5 Banach fixed point theorem, 41 Banach theorem on contractions, 41 band, 133 Browder–G¨ohde–Kirk theorem, 42

closed operator, 107 coercive function, 7 common fixed point problem, 27 compact operator, 9 concave function, 14 constrained minimization problem, 24 constraints functions, 28 constraints qualification, 26 constraints sets, 28 contraction, 41 control function, 222 control sequence, 222, 234 convex cone, 13 convex feasibility problem, 27 convex function, 14 convex hull, 13 convex minimization problem, 25 convex subset, 12 convexity of the norm, 2 CQ algorithm, 229 cutter, 53

A. Cegielski, Iterative Methods for Fixed Point Problems in Hilbert Spaces, Lecture Notes in Mathematics 2057, DOI 10.1007/978-3-642-30901-4, © Springer-Verlag Berlin Heidelberg 2012

295

296

Index

cyclic control, 224 cyclic cutter, 176 cyclic projection, 171 cyclic projection method, 218 cyclic relaxed projection, 172 cyclic relaxed projection method, 219 cyclic-simultaneous projection operator, 173

firmly contractive operator, 65 firmly nonexpansive operator, 65 firmly quasi-nonexpansive operator, 54 fixed point, 11 focusing algorithm, 240 Fr´echet-differentiable function, 7

demi-closed operator, 107 demi-closedness principle, 107 derivative of a function, 7 of an operator, 10 diagonal matrix, 11 differentiable function, 7 operator, 10 differential, 7 of an operator, 10 directional derivative, 7 distance function, 15 distance to a subset, 15 Douglas–Rachford algorithm, 212

Gˆateaux-derivative, 8 Gˆateaux-differentiable function, 8 Gˆateaux-differential, 8 general convex feasibility problem, 33 generalized relaxation, 97 global minimizer, 24 global minimum, 7 gradient, 8 Gram matrix, 12

eigenvalue, 9 ellipsoid, 137 elliptic operator, 9 epigraph, 7 "-optimal solution, 25 essential weight function, 192 Euclidean space, 3 extrapolated alternating projection method, 209 extrapolated Landweber operator, 182 extrapolated Landweber step size, 182 extrapolated methods of parallel projections, 254 extrapolated simultaneous cutter, 188 extrapolated simultaneous cutter method, 254 extrapolated simultaneous projection method, 254 extrapolated simultaneous projection operator, 190 extrapolated simultaneous subgradient projection method, 258 extrapolation, 97

F -attracting mapping, 48 Fej´er monotone operator, 45 sequence, 108

half-space, 6 hard constraints, 33 Hessian, 8 Hilbert space, 1 hyperplane, 6

ice-cream cone, 140 idempotent operator, 11 identity operator, 8 image, 11 intermittent control, 238 intermittent sequence of subsets, 237 inverse image, 11 inverse strongly monotone operator, 74 inverse-positive matrices, 197 iteration of an operator, 11 iterative method, 39 iterative procedure, 39

Kaczmarz method, 220 Karush–Kuhn–Tucker point, 25 kernel, 11 Krasnosel’ski˘ı–Mann theorem, 115

Landweber extrapolation parameter, 180 Landweber method, 228 Landweber operator, 176 Landweber operator related to a matrix, 179 Landweber step size, 180 linear feasibility problem, 30

Index linear least squares problem, 32 linear split feasibility problem, 35 linearization, 16 linearly dependent vectors negative, 2 positive, 2 linearly focusing algorithm, 235 Lipschitz constant, 10 Lipschitz continuous operator, 10 local minimizer, 24 local minimum, 7 lower semi-continuous function, 7

maximal residual control, 224 metric projection, 17 minimizer, 6 monotone operator, 10 Moore-Penrose pseudoinverse, 12 Moreau decomposition, 13 multiple-sets split feasibility problem, 35

ni-firmly nonexpansive operator, 73 nonautonomous method, 105 nonautonomous projection method, 203 nonexpansive operator, 41 nonnegative operator, 9 norm of an operator, 9 normal cone, 13 normal equations, 32 normal solution, 32 null space, 11

oblique projection, 130, 184 obtuse cone, 13 obtuse cone selection, 197, 272 Opial space, 107 Opial’s property, 107 Opial’s theorem, 114 optimal solution, 24 orbit, 11 orbit of an operator, 39 oriented operator, 98 orthant nonnegative , 5 nonpositive, 5 positive, 5 orthogonal projection, 11 orthogonal subspace, 5 over-projection, 77 over-relaxation, 40

297 paracontraction, 48 parallelogram law, 2 partial derivative, 8 polar cone, 13 polytope, 12 positive definite matrix, 11 positive operator, 9 positive semi-definite matrix, 11 positively homogeneous function, 15 product Hilbert space, 4 projected gradient method, 214 projected Landweber method, 228 projected Landweber operator, 184 projected relaxation, 89 projected simultaneous projection method, 217 projection, 11 projection method, 203 projection vector, 17 projection-reflection method, 221 operator, 174 proper control, 264 proper sequence of weight functions, 264 proximity function, 28 pseudo-contractive operator, 73

quasi-nonexpansive operator, 47 quasi-periodic control, 225

range, 11 reflection, 40, 77 reflection-projection method, 221 perator, 174 reflection-relaxed projection operator, 174 regular obtuse cone selection, 197, 273 regular weight function, 232 relative interior, 13 relaxation, 40 relaxation parameter, 40 relaxed alternating projection method, 204 relaxed cutter, 53 relaxed firmly nonexpansive operator, 65 relaxed metric projection, 77 relaxed projection-reflection operator, 174 relaxed separator, 53 repetitive control sequence, 225, 240 repetitive sequence of subsets, 240 residual selection, 197 residual vector, 31

298 residuum of a constraint, 31 retract, 11 retraction, 11

Schwarz inequality, 1 Schwarz theorem, 8 segment, 6 self-adjoint operator, 9 separating operators, 54 separator, 53 sequential projection method, 222 simultaneous cutter, 186 simultaneous cutter method, 231 simultaneous projection, 162 simultaneous projection method, 215 simultaneous relaxed projection, 162 simultaneous subgradient projection, 199 Slater constraints qualification, 26 soft constraints, 33 sphere, 6 split feasibility problem, 34 standard scalar product, 3 standard simpleks, 5 step size function, 97 strict convexity of the norm, 2 strict relaxation, 40 strictly attracting operator, 47 strictly convex function, 14 strictly Fej´er monotone operator, 46 sequence, 108 strictly nonexpansive operator, 41 strictly quasi-nonexpansive operator, 47 strictly relaxed firmly nonexpansive operator, 65 strongly attracting operator, 56 strongly convex function, 14 strongly Fej´er monotone sequence, 108 strongly focusing algorithm, 240

Index strongly monotone operator, 10 strongly nonexpansive operator, 91 strongly oriented operator, 98 strongly quasi-nonexpansive operator, 56 subdifferentiable function, 16 subdifferential, 16 subgradient, 16 subgradient projection, 144 sublevel set, 7 subset of minimizers, 6 successive projection method, 222 surrogate constraint, 192 surrogate constraints method, 263 surrogate constraints method with residual selection, 271 surrogate projection, 193 with residual selection, 198

tangent cone, 14 triangle inequality, 2

under-projection, 77 under-relaxation, 40 uniformly F -attracting mapping, 56 unit vector, 5 unitary operator, 9 update, 39

variational inequality problem, 27 violated constraints, 50 von Neumann method, 204

weak cluster point, 4 weak convergence, 4 weak limit, 4 weakly lower semi-continuous function, 7 weight function, 50