An Algorithmic Approach to Nonlinear Analysis and Optimization (Mathematics in Science and Engineering 63)

K(x, 5, .&)I

on [O, 11. A, m

- UZ(5)l

on B

fm

u1 , u2

do

11

on

-f(4 < 1 1 P


y =

I

-

Ij u1 - u2 /I

4 5 ) l d5

PYX,

Also, =

=

I f"(.l)

-

jZK(x, 5 , f ( 4 ) d5 K (x,

f2(4 < h2P2

1'

K(5,77,4?))4)d5,

5,

1I i

JZ

0

0

UI(77)

- %(77)l

4

<m 2 y

; (1.2.14)

10

1.

1 h impm/m!

for

I h I m /3”/m! < 1.

h

on B.

m

1.2.11

uo

Extensions of the Fixed Point Argument

f B,

CB

B.

by

B,

1.2.5

on B, ,

B, : B, -+ B

Lemma 1.2.15.

(1.2.16)

6

OL

B6(a)

u.

Proof.

u

llf(4 - v

B, .

B, ,

< llf(4 -f(.)ll

+

<.llu--l/+(1 or6 (1 -

-v

I1

-46

< +

B, .

1 B,

1.2.5 open

1.2.15

strict

f B, on B u

u by

r, - a),

OL

1.2.

11

CONTRACTION MAPPINGS

B,(u) 6

u

=

1.2.15

r/(1 - a),

B, .

(s, -)

on

B , --+ B

on

s

Definition 1.2.17. A uniformly contractive on B

(s,

4

411 < 01 II 24 - v /I

Ilf(S7

B

u, v

s E S;0

sE

)

(1.2.18)

< cy < 1. I

f ( s , *)

Theorem 1.2.19.

CB

B,

f (-, u )

0

B

B, by

B,

.)

on S, uo

- a)6

uo(s)

.

u E B,

S

s

s

on Proof.

of 1.2.15. S B.

by u

B,

uo(s)

s

uo (., u )

s

11 s - t /I < r ) ,

7 a 0 ( 4 -f(t, E >0 f ( s , uo(s)) = uo(s) t

uo(t)

( t , -)

UO(S))ll

< fz of B). f ( t , .) E. As 1.2.19,

uo(s) uo(s) 11 s - t

~ / ( 1- a).

uo

I

I/ < r )

12

1.

ITERATIVE METHODS ON NORMED LINEAR SPACES

The Inverse Mapping Theorem

inverse

mapping theorem

f

C1 =

0.

x =0

on

on

=s

x.

g(s, x)

on R1

0

p.

< a < 1,

YI x


B,

y 6

on

x

g(s, .)

on B ,

B , by

g

I

I < 6, .

by on B,l

-

1.2.19,

g(s, xO(s)) = xO(s),

f(x"(s)) = s.

a ) 6,

s

.

1.2.

13

CONTRACTION MAPPINGS

(x) = s

by

f'

f'(xn)

A Useful Lemma

by

A u = & Au =

z,

( I f A)-l

+v

&Au

+ z, f A)-l

u =

(Ih - A ) = X

u.

(I -),AX -

Ih - A

h

A,

p

p

Au by

B

jl A ]I < 1.

n by n

=

1

<

v u =

=

A

/I B I/ < 1 .

- I,

A

Bu

I/ A / / < j h j. I/.

+v A.

on

Lemma 1.2.20.

A A

B

14

1.

ITERATIVE METHODS ON NORMED LINEAR SPACES

11 B - A < 6 A-lB = I -

B

B

=

-

- B).

- B) = I -

/I I/ = II

-

B)ll < 1

6

=I -

B

I

-

=

EXERCISES

1.2.1. x

of on

I/

1.2.2.

11

I/ < 1, m

(I -

=

1 A", n=O

m

+ CO.

B

1.2.3. 1.2.4.

f 1.2.5.

C1

K

on

f # 0.

R1 =

K.

0

of

1.3.

15

DIFFERENTIATION ON NORMED SPACES

1.2.6.

B

uo

on u , + ~= (u,)

1.3.

uo

B,(uo).

uo E B ,

.

DIFFERENTIATION ON NORMED SPACES

FrCchet Differentiability

x

on R1, f [ ( x u ) -f ( x ) ] / u

+

u -+0.

on

f’(x)

x, df(x, u ) = f ’ ( X ) . u u

I u I -+ 0. on R1

Definition 1.3.2.

F. X

]I u 11 +0

F)

U on U

E. x

U

1.

16

E

ITERATIVE METHODS O N NORMED LINEAR SPACES

F,

a)

I

x.

df

on E

x, h on R1. x.

x

h

by

E

f

Theorem 1.3.4. h h

F;

E,

x

h(x, u )

=f ( u ) ,

x. Proof.

x

p

Af(.r, u ) Yx, u )

- P(X,

=

f(.

+

I/

u)il

u) -f(x).

u ) - 4 ( x , .)I/

I/ u

I/ u /I L /I

4% u ) - P(X, u)ll 4 i/ u /I

// u //

+ 0.

E

fix u1

u = xul (z

u,

p

11 h(x, u ) - P(X, .)I1 u I/

I/ u Ij -+ 0

a -+

-

+,

’

I‘ u1 /I

u1) - P(Xx, %)I/

E.

=0

h

Af(x, by h(x, u )

X(x, u1) - p(x, u,)ll

0 I/

u1

0

x. u) =

=

p

0 u A f ( x ,u ) = f ( u )

=f(u).

FrCchet, Ann. SCI.Ecole Norm. Sup. 42, 293-323 (1925).

h =0 (1.3.3)

1.3.

17

1.3.2

E

h(x, .) h (x, u )

on E 5.2.10), v

= (u, v),

U C E.

x

E.

v

x E U,

R1

h(x, u )

=

u

. v,

z, = f ’ ( x ) .

by

on on

Lemma 1.3.5.

U

En.

on U

Vf(x)

gradient

x Proof.

4% u) =

+ u1 -4 + + +f(%+ u1 ,..., xn +

f(*l

9

+f(%

x2

Y...?

u1, x2

-f(.1

>...,4

+

+

,-..,Xn) -f(% u1 , x2 ,-..,xn) ... Un) -f(.1 u1 ,..., XfL-1 %I-1 4.

u 2 , x3

+

+

2

[A1 ,p.

4 x 9

af (x1 + 51 4 = Ul 3x1

2

*2

,..., Xn>

18

1.

ITERATIVE METHODS ON NORMED LINEAR SPACES

0 < ti < ui

I/ u

+0

u = (ul

,..., u ~ ) ~ ;

I/ u /I -+ 0

/I 5 / /

--j

0 by (1.3.3),

q x , ).

En on

).

=

Em U En,

m

by

on U

u ) = /Ax).,

Jr

m by n

/AX)

by

Jacobian

( 1.3.6)

==

__

__

ax,

ax,

...

__

A(x, -)

1.3.

19

DIFFERENTIATION ON NORMED SPACES

Em)

U.

x

Jf

Of F)

u

U

Of

u:

f

on

x,

Jf

on U

C1.

on

By

v

En,z,

Directional Derivatives

u -+ 0

1.3.2

E. A u -+ 0

by by u

=

aul

O(

-+

0

(E

u1

on En. Definition 1.3.7. u

x

11 u I/

=

by

f

Lemma 1.3.9.

on on U

C1

U fu =

Proof.

I/ u /I = 1,

01 -+

0

by

(Vf>u ) . on U

1.3.5

11 au /I

--f

0.

1

20

1.

ITERATIVE METHODS O N NORMED LINEAR SPACES

4

f, u

x

(b

E7&,

u

E

hyperplane.

a

E2

( u, x) = UlXl

+ upq

En

(u,x) = 6

if

b

=b

E3

FIG. 1.3.1. ’0.

(u, x)

Half-spaces and hyperplanes in

E2. Shaded area

IS

half-space

1.3.

21

E2 u13c1

a half-space En (u, - b 01. of x S, by

<

+

u2x2 -

b

u

20

u

(u ,

En

S C1

-

b >, 0

by

f = tangent hyperplane

x

u = 0.

(11

x

u /I = 1)

u

x, 1.3.2

&Vf(x) E2).

FIG. 1.3.2.

E2. F

x : (V’(.Y),

ZL)

< 0.

22

1. Of(.)

no of

f. u

//

=

1,

-11

Vf(x)il

< ( V f ( 4 9 4 d /I +rVf(x)

ti =

Vf(x)ll].

y =

x,

by u

=

-Vf(x)/il Vf(x)iI (Vf(X),u )

Of(.)

=

( V f ( x ) ,24)

=

-/I

Vfll

<0

01.

u

<0

u

=

-Vf(x);

1.3.2).

a

( V f ( x ) ,u )

u

<0

+ nu) < f ( x ) . by

x.

x,

=

u)

<0

u)

Steepest Descent with Respect t o Non-Euclidean N o r m s

on En do on

u), on En

on En.

/I u IIG

= (u, Gu)1’2

>0

1.3.

23

DIFFERENTIATION ON NORMED SPACES

x,

?” u). =

I/ u ;1

=

E2

I/ u (I2

=

E2.

11 u (IG

u

=

1

f,

x,

u)

+A h

Of(.)

u :1

- 1 ) = 0,

11 u ;1

+ 2XGu = 0

-

1

=

0 2Gu]

(u, Gu)

u =

(1 V f ( x ) \ l c - l .

y

x

I/ I I G , u

En

on

=

=I

Taylor’s Theorem and Some Applications

U

on R1.

C2 p. 961)

on U , 4(x,

4 = df(x, 4 + ,f”(O,

&,

4 =f(.

U2

+ 4 -f(4

(1.3.1 1)

24

1.

ITERATIVE M E T H O D S

< 0

4

N O R M E D LINEAR SPACES

on

x

5

x $- u

77 <

=

x

+ qu,

En, E

C

Definition 1.3.12. conrex on u,

z‘ E

I

C,

,xu

+

C -

LY), z1

E

C on

E*L. Theorem 1.3.13. C En. A&,

5

x ).

x

=

+u

<

HI(.)

C2on U,

+ #, Hr(S)u), x x +u

(Vf(x),).

on

0 < 7)

f

(1.3.14)

5 = x + qu by

Hessian

iy

...

%XI ax,

H,

(1.3.15)

=

__

sxn ax,

C’,

x.

4 ( x , ).

1

pp.

..

=

(WS), 4.

( 1.3.16)

is

1.3.13 on R1

En

B,(x)

on R1

S

x

1.3.

25

DIFFERENTIATION ON NORMED SPACES

on En. df(xo, u) 3 0 u df(xo, u ) > 0

1.3.17.

xo on En

weak

strong

Bs(xo); xo xo

x';

B, .
global.

xo

on R1).

C1 on

1.3.18.

En

on U

x0

Vf(x0)

0.

=

1.3.13,

Proof. 0

11 u 11 < 6. 11 u 11 < 6.

< 4 ( x 0 , 4 = ( V f ( 0 ,u )

u = -a Vf(xo) -.(Vf(C),

Vf(x0N

Vf(xo) # 0, ( V f ( 0 ,Vf(x0N a:

--f

0

a:

-

>0

3 0.

II Vf(x0)I12

Cl), -a(Vf(S), V f ( X 0 ) ) < 0 01:

= 0.

En,

7 u

f,

=

0

=

x

+

I x, u

En

NU

x C1

(Y

(Vf(v),u )

=

0.

u

/I u /IG,

x u

q

26

1.

by x,

by f

=

EXERCISES

1.3.1.

on En

on

C1

1.3.2.

/j

IIG

En, S

x on

u

by

Of

u

1.3.3. B,(x) = {x I /I x - x I/

En;

< S}

on R1

1.3.4. 1.3.5.

(u, Gu),

1.3.6.

C2,

1.3.7.

f

H, C1 on E2 f ( x ,y ) y

11.

?

1.4.

27

GRADIENT TECHNIQUES

11

YoZ:&f(x,

d

Y)) = 6(,F:&f(X,

Y ) ) = 0. ?

1.4.

GRADIENT T E CHN I Q U ES

Newton’s Method

on

Of

=

0.

f :R1+ R1 = 0.

C1

xo

by by

xo

xo x1

xo

(xn}

1.4.1 xo by

f

(1.3.1

linear

f(x)

#

= f(xo>

+f’(xo)(x

- X”) =

0

by x1 (1.4.1)

28

g(x)

1.

x

:

-

f ’ # 0.

‘f‘(x)

1.2, g

x* (xrL}

xo.

Q

1.4.1.

x0

of

xo E

B,

{xn>

xo

C2; Q

f‘

1.4.

29

GRADIENT

by (1.4.2)

xo .

f’

1.4.1 xo = 0

(x) = x2

FIG. 1.4.2.

,..., u,) f

;

En + En,

= 0,

i

=

1 ,..., n.

1.4.2,

30

1.

C1. un+i

= un

-

J;'(u%>f(un),

J, on En;

C2 Of

=

xo Of : En + En

0 C2,

by :

=

H, .

u,+l = u, - FZ;l(u,) Vf(un).

on En

Ht

J,

Descent M e t hods book

Vf uo

(xn}

downhill

f

=

unfl

u,

Au = -H-l f

Vf(u0)

f

u,,

f

du,

1.4.

31

GRADIENT TECHNIQUES

u1

(Vf(uA m u , ) V f ( U 0 ) ) uo

u

= 0.

+ 01du

uo

01,

uo + 0 1 ~d u

du

>0

uo u1

u2 by

al,

01,

du. {u,}

{a,}

by un+1 = U n

-

%fG1(un) W

uo

n

)

modiJied

optimal

{a,}

f

du

3).

a’s

V’(U,+~) -H3Un)

VfW

01,

du =

u,

-HFl(U)

,

n by n

Ht’(u,)]

Hr(un) Lemma 1.4.7.

Proof. (Vf(u),d u ) =

(Vf(u),

du

=

-H Of(.)

Of

u

du

Vf(U)

<0

du

u

Of(.)

Of

Vf(u)) > 0. H.

I

=

u1

32

1.

SPACES

Y f by u,+1 = u,,

{H,} n by n technique

- .-

anfI;l Y f ( u n ) ,

{H;'}] gradient descent

ZI;' T f ( u 1 8 ) 11 ilH,L

u,, {xn)

H,

Hf(xrL), 11, = I metliod of steepest descent

Au,,

2

=

Z L , , +~ u,

.[f(un)) f ( ~ ~ -.+: f~( u) l l ) ]

n

y

Y f ( u n )+ 0

--f

a3

u,, + uo. An

C2 on E n quadratic

u by

u

by

on quadratic on

I on E n , O f ( u ) = f ( ~-1- A u )

=

('Tf(u),Au)

4-~ ( O U ,

'I

Of(.) AZL= -ZZF1 Tf(u) Of

-E(Tf(M), I / -

If. Yf(2l))

I

g'( Y)

=:

0,

'

2.1

2)(Yf(U), I 1 -'GIl-1 T f ( U ) ) Y)

0.

:

= g(.)

1.4.

33

GRADIENT TECHNIQUES

H

H

=

=I

01

)

=

1.

f,

H,(u)

u

C2

uo

H,(u) n

Theorem 1.4.11.

- -

f

u*

H,(uo) = an

1

on En

f {Hn} 0 < A, < 11 H;' /I f2 = {u I f ( U )

< Ao

f(u0)).

by Vf

u,,

{un>

uo f(un)

-ff(uo).

u,

uo

Q,

+ uo.

Proof.

p

>0

f

11 u 11

f ( u )4 +co

---f

Q

co. j j uk

/I

co

--f

Q

f(uA)
<

un Y+ uo

f ( ~ ~ + f~( u) n ) . y

fun} uo

by

Q.

f y =f(uo).

1.4.7

34

1.

SPACES

Yf(uo)= 0.

2.2.14

2.2.18

e

on 0

52,

< ( W n ) , Un

tl

< f(Un+d

-

-f @ n )

0

C f ( u o ) # 0.

12 + cx).

?).

+0

u,,~ - u,

Un)

~

K1T u n ) )

(Vf(Un),

or, =

))

ff,-'Gfy

Cf

/I H i 1 11

on Q .

.

< Xo. by

( W U n ) ,

K1W

2

>0

26 >0

3 A0 I1

+0

Vf(un)-+ V f ( u o )

n,

/I Y ,

n ) )

6'

u,+~ - u,,

=

%H,l 0.t %)

-

n. ti,,

~u, +

0

V f ( u o )= 0.

u,

u" u

(uJ

+ uo,

a

u',

u'

u' = uo.

I

1.4.11 11;'

-

I.

3 E2 A,s:

A,

1 A,xi

> A,

f >:

by

f

==

0.

A, A, A,

Cf

by

1.4.

35

GRADIENT

A,, A,,A , ,..., (A,

Vf

A,.

on En

Lemma 1.4.12.

H,

(1.4.8)

Vf.

=

Proof.

f(.) u0

Vf.

=c

1.4.11 Vf(zi,)

=

u0 01,

= A,),

=

1

a

+

( a , ).

+ h(u,

+ Gu,

= U,

f 0=a

+ GuO

I

-

(1.4.13)

(1.4.13),

u0 - u0 =

-01~

Vf(u0) =

V f (u,)

f ( u ) = f (uo) +

Vf(u,)

uo u

=

- u0

u,.

u,

I,

yo

=

1,

1.

36

n

---f

E n by

/I u ;1

=

En,

2

2

Z/Cu pp. f ( u ) -= c

g(2)

+ ( a , + A(u, U)

-=f(dG-’z)= c

Gu)

( Z / E ’ a , z ) -+ f ( 2 , 2).

~’~

g

z,, zo

~

uo

2

N nO

-

uo -

=

‘C,g(zo)

Yf(uo)

Z/cuo.

zn Tg(z) =

1.4.12

by

Tg by Tf(u)].

E”

Ij u

En

-

ii u

:=

(.\/cu, Z / c u )

-:

“u “2 “

“z

zi

{Efr<j 3, * \V.

IBM

/. Res. Develop.

{H,} 10,

&-l,

1.4.

37

GRADIENT

-H-l

Vf

11 l l H

En

3, by

C2 by

on

A Digression on 111 Conditioning

Ax

=

b

E2

by

5)

A A by

da$ A

+ a:2

A by

by

daEl

+ ai2.

by (1.4.1

1.

38

LINEAR SPACES

by the

A)

by

A

1.4.3

b.

1.4.4.

1.4.3 x

., 6,

1.4.4

b by

b)

x.

1.4.3.

of

1.4.

39

GRADIENT

Err i n b>

Line X

OIIX,t 0

12 2

FIG. 1.4.4.

:bl

of

Ax

=

b.

by

A A

-

aI2

=

A

a11a22-

A

A

aZ2

A

=

1

%Q12

a,,

0.

+

a12%!2

=d

a 1 1

+ azz)

= 0;

-

a12 aI2 = 0

40

1.

LINEAR

a I 2= :

?)I.

=

0, A A

< u11a22

=

A

1.

C:I b

A

A ill conditioned. h

-

XI)

=

2 by 2

h

A 13

7~

el, =

-

2h

A

=

0,

2,

h,/A,

Ax

0,

A,

-

A, =

A,

=

~

h,

4

-

0

hl/X2

A

=

1

1.

+

E2

,4

pp. -{- h,x,Ae, .

x h2 = 0 ( A

+

=

A

. A, X,/h,

--t

on

m,

,

1.4.

41

GRADIENT

by

H

-H Of

H

H

no 1.4.4,

x 6

b

no

f ( x ) = (x,

Ax) =

A,/X, on on

-A-l Of A A

-kl

Of

-

no

A A A (x,Ax):

by

Ax

=

A,/&

by by

b

A x

-

by

0,

by by

A.

42

1.

A-l Of

-

ogy

3,

A Question of Stability

dx dt

=

f(4

f : En-+ En

z ( t ) for

equilibrium solution f ( z ( t ) )= 0, t 3 0.

z

0

/I

9 t + +a,

=

+0

x

x(0)

Q.

V

Theorem 1.4.17.

n Q

:= {x

C1

1

V(x)

on

En

< V(X,)}. V

0 on Q.

x(t)

1.4.

43

GRADIENT TECHNIQUES

x(0)

i =f(x)

= xo

V V(x(t)) = ( V V , f ) < 0

on Q,

11 x 11 + 00.

v(0)= 0. x ( t ) +0

t+

+

Y ( x )4 +GO 00.

on R1,

Proof.

<0

V(x(t))-

t

> 0.

< V(.(t)) < V(%)

0

Q

L2.

x

x I] + co,

V ( x )-+ +00

V(x(t))

t

+

Y

+ co.

V ( x ( t ) )+ 0

Q

{x(tn)}

xo L2 t, -+00 v ( x ( t ) )-+ =0

v(xo)

i

E

>0

=

0. tk

x ( t ) +0

+

+ oc,

t

+ co;

11 4t;)ll 2 E .

a

{x(tA)}

f : En

El

4

C1

xo

g(x)

=

x

-

Vf(x)]

2 =

f(4 -f(X0),

V ( x ) >, 0

xo

V

=

X) =

- 1 1 Of112 < 0

=

44

1. x

# xo

1.4.17 Q

xo

Ilfliz 1.7, V 1.4.17.

by by

EXERCISES

C 2 on R1 xn. geometric progression,

f

1.4.1.

s,

-

I

xo

~

<

x,, -+ xo

,

s,,- .tn i2.

N

1.4.2.

on E"

1.4.3. f(.)

+ (a,

=c

.Y)

1

by

;(.T,

so f ( X ) = .f(.Y")

/I

A.

.x

--:

~

-~(T~(.\."),1')

s,,.

C'ovipf. R e d . 25 (1847)

-

;(/I,

Ch),

1.5. LEAST

45

SQUARES APPROXIMATION

on En

1.4.4.

Of = 0 1.4.5.

on En

(u) =f

A

1.4.6.

A

Og(z) =

2 by 2

< u11a22

A

A A. AJA,

1.4.7.

Ax) Ax) A , , A,

2 by 2

A

1.4.8.

=

A.

A-l A-l

A.

=

A 1.4.9.

z(t)

z

A-l. a? = f

=0 En.

V ( t ) >, 0 t

1.4.10.

1.5.

C1

v(t)-+

V

0

t+

+ co.

LEAST SQUARES APPROXIMATION

f= be

C1

on En

(u)= p,

46

1.

ITERATIVE METHODS O N NORMED L I N E A R SPACES

p E Em.

m by

n

f”ul )...)u,)

m

=

n

-

p

= 0.

(1.5.1)

Jf

{all}

m

Jf

m


f ( u ) - p. uo

ilf(u)

-

/3

I/

i ~ f ( u o) p 11

=

01

least squures solution

uo

/3

=

>n

uo

1.5.

47

LEAST SQUARES APPROXIMATION

Some Remarks on Generalized Inverses

En,Em ( V ,W )

A V

by V , W , m by n

W

V, W

V,

M

V

M

n

-

r

by M I . V

V

=M

0M i

vo E M

V

v uo E M I .

vo

N

M

Pvo = 0. on N

Pv, = v o

I -P P

M. P

+ no,

V)

P

A

by P ) ,

P 2= P

P = PT. by

18, 21, 22, 41, 4.4

As el

e,

,..., e,

,..., e ,

M

P

ML, r

Ax

=

AE by m >, n

b

V, W),

n a i j x j = bi

i = 1,..., m.

,

j=1

kl all

a22

... ...

am2

..*

a12

"')("')

=

a2n

amn

xn

(b').

(1.5.5)

bm

n

b

6,

R(A)]. is

x

b,

=

0, b,,

6, x

A

.

x.

A b

N(A)],

=

0,

4x

1.

ON

A

V,

A. TT’

V

A ) by

n by m

A R(nlr) N ( A ) l .

AT. A’, N(AT)

m

n, “(A)

=

A

=

R(A)]

m =n AE

R ( A ) -= It’

V, W )

A rn

Az:

0.

A

~:

on u

-~

7’

21,

-

iz

n, v

< n, N(A)

c’ i\’(A)-L, z’ t :V(A):

Au

N(A),

u

=

Av.

=

n’(A)l V)lli(A)

A

on V / N ( A ) 21 on

A

N(A)L

on R ( A )

iV(A)’by A. A

2

on

E

Definition 1.5.7. TJ7, W )

generalized ineerse W,

by

R(A)

A+ Ar.

W, A+

1.5.

A+ A+ w

49

LEAST SQUARES APPROXIMATION

AT

AT

R(A) R(A)l. 1.5.1.

w = Av ATw

A+w = v,

v

N(A)I,

FIG.1.5.1. The range and null spaces of the adjoint and generalized inverse of A .

+

v0 v0, Avo = 0,

z, =

W,

w

=

wo

+

vo N ( A ) vo E N(A)L = R(A+). A+Av = A+Az,O = vo = PR(a+) A+ N(A)]. wo wo N(A+) wo R ( A ) so

AA+w = AA+w0 = W O

A

== PR(a)

N(A+)].

by E. H. Moore, Bull. Amer. Math. SOC.26, 394-395 (1920).

50

1.

ITERATIVE METHODS ON NORMED LINEAR SPACES

A+. by A+AA+= A'

AA+A = A (AA+)*= AA+ (A+A)' = A+A.

( 1.5.10)

A+

A

1.5.7 N ( A ) = (0) A+A =

A R(A+)= N ( A ) I = V . m = n, AA+ = I A+ = A-l. ATA Q A = I. N(A)=

AQw

m

>n

A

A

4.4.8). R(A)

=

w

(ATA)-l AT,

W

A+,

by

A+ = (ATA)-'A*.

1)

by

A

An Iterative Procedure for Finding Least Squares Solutions

m

=

n

Jf

Jr J f

, Jr

on g : En + El

by g(u) = Ilf(u) - p [I2,

Proc. Cambridge Philos. SOC.51, 406-413 (1955).

1.5.

51

LEAST SQUARES APPROXIMATION

m

{Ofi}

v llf(4 B 112 -

=2Jf(U)T(f(4

-

B). by

uo

Au-P=O,

Au

- /3

jj Au0 - /3

I( Au

- /3

uo

11

= 2AT(Au0- /?) = 0

A).

11

C1

u uo

2.2 Au = ,t? AT(Au0- 8) uo

= 0.

En by UO =

(1

u,, - A+f(u0),

f ( u ) = Au - /3.

by un = ug - A+Au,

A T A 0 = ATAun- ATAAfAu,

+ ATAA+/3

= ATAA+P

= AT(AA+)TP = (AA+A)T/3.

AT(Au0- 8)

=0

/I Auo - P 11

or

=

As

0. uo

m =n u0

(m

=

A-l/3,

2 n), UO =

(ATA)-'ATP.

A A

+ A+P,

52

1.

f

by un , 1

:=

u,,

(1.5.16)

an/r+(~4f(~J,

~

{I,,}

du,,

= z/,+~ -

un).

u,,

m rn

=

>n

J,

n

JT'

=n

(1

ilu

=

-./;(u)f(u)

YiAZL)

=

2J:Wf(U).

Fg, 4

=

-2a(f,

JfJTf).

],IT

Y

by

Y

J f

r

( f , J f J f ' f )=

(f'). 1-1

(Tg, 1121)

- .0

L~ >;

0.

du Tg(u) : 01. f : E" + E n

ZL

g : Ett -+ E' un J;(u").f(Zl") =

Y;

',f(u")l:' -= 0.

g(u)

2

11 f(u)I!'.

1.5.

53

LEAST SQUARES APPROXIMATION

JT

=

1;.' g,

f on En

Vf. H, uo

by

EXERCISES

1.5.1.

P

on

ilf

M,

V

I'

M.

1.5.2.

A N(A)=

A

W)

Ai

1.5.3.

A

Bj

B,

( A i ,Bj). R. Tewarson, Comput.

I. 10, 41 1-413

(1968).

AB

54

1.

ON

1.5.4.

A A =

SPACES

AT

=

AT =

A AA+.

W).

1.5.5.

1.2.20 by

1.5.6. m by n

(V,W). 1.5.7.

b E R(A),

Ax = b u

V,

x = A'6

+ ( I - A+A)u

- A+A

1.6.

on N ( A )

N(A)J-1.

TWO-POINT BOUNDARY VALUE PROBLEMS

a

xz

= f"X1

,...) x, , t ) ,

i = 1,..., n

TI.

t

);I?(

x(t) =

t

on two-point boundary value problem).

of

1.6.

55

TWO-POINT BOUNDARY VALUE PROBLEMS

x -+ A? -f (x, -)

x(.)

B TI

on

C1

by (1.6.2)

g(x, .) = A?

-

f(x,

B

g

0 ) .

B [B

on

TI

g.

of

g

C1

f

u

B

V’i(x,

x

=)

f i

x],

t

t.

TI,

11 u Ij + 0,

56

1.

D B)

D

: x ---f xo

B

x

D

g B by g'(x, .)

=

I1

.).

-

g xn

i-1

k'(% >

=% -

,j: x

+

[g'(.,

3

.)]-'g(x,

.)

on

x(.)

B As x,,

xu,

t

{xn} no

x, xn

2 linear

by I<. illcG111 (1963).

Proc. Internat. Astronaut. Congr., 14th, Paris,

1.6.

57

TWO-POINT BOUNDARY VALUE PROBLEMS

(1.6.4), %+l

=

>

.) %+l

-

+f(% .)

.) xn

7

9

n z i = A(i)u

+ b(t),

4 4 = Jf(u(t)?4 b b(4

=

J f ( 4 4 , 4 44 + f ( u ( t ) ,

t).

Solution of Linear Two-Point Boundary Value Problems

A xl,

x,,

b on. on

u(0)

TI.

n

-

K zi = A ( t ) u k d(0) 1 < k.

<

k

u( T )

yj,

c

58

1.

SPACES

up. k

+ UP(.)

ajuj(*) 3=1

al,..., ak.

L

k by k

k

uj(T).

k

olW(T)

+ uP(T)

j=1

K

c

by

k

La = /3

(1.6.9)

a,

xo ,

6 A , 6,

xl,

11 xTc.,l- x, / / + 01.

11 x,

-

xo 11

TI 4

0,

59

1.6. up(0)by

k

d,..., ak up.

x1 ,

(Y

a' =

x2 ,

i"), ak'

x3,

L

01

1.4 L

by uj(0).

d(0)

d(.) t,

d(-) L. by

d(.) on

d(0)

60

1.

n

t

=

=

3

t,,

u,v

pp.

on

f = t, .

u'( T ) ,

L by by

[5].

I,

L

1.6. La

=

61

p

01.

An Example

by a

x2=

r(4

t

=

dx:(t)

x2 --

(1.6.12)

r

+x;(4 + g ( t )

21.

E

by =

x1(2) = 0,

1.076000,

~ ~ (= 2 0.576000, )

=

= 0,

x3(2) = 0.997661

62

1.

by

1.7.

F U R T H E R REMARKS ON STABILITY

1.4.17 by

1892. A

A, by

=f ( x ) .

2

f“x>

=

4 + g”x>, f

g(x)

f.

aii

=

*

=

afi(0)/axi ,

A

n by n

A x +g(x).

g

i - Ax g

* @

by

York, 1965. IEEE Trans. Circuit Theory 97, by

lo

York, 1961.

=

0.

1.7.

FURTHER REMARKS

i- Ax =

63

STABILITY

0

A

stability

matrix).

En p.

A

Theorem 1.7.2.

11 g(x)lI

71

< 7 11 x I\

>0

x E B,

B,

.

Eq. n by

Proof.

P

n

ATP + PA

-8

(1.7.3)

V ( x ) = (x, Px). on En. If x(.)

p.

C1

V ( X ( t ) ) = (k,Px) = (x, =

=z

V

+ (x, P3i)

+ (x, PAX)+ ( g ( 4 p1. + (x, Pg(4 + 2(x, Pg(.)>.

A=Px)

4 x 7

81 .

If A, , A,

P, -(x, Qx)

< -A,

11 x /I2

(2,

@>d (217AI - ' A d 1

x

Pg) < A I/ x l12.

/I2

P <0 V(0) = 0.

1.4.17

B,.

I

Estimation of Domains of Attraction

1.7.2

P

(1.7.4)

64

1. by

V

V(x)

:

by

1.7.2.

En.

Px)=

P < 0,

1.4.17,

A

1.7.2

V(x)

P < 0.

x ErL

V(x) = =

0.

1.7.1

Contour

V=o

( x , P x ) = constont

Equilibrium point

E2

1.7.1.

n

V(x)

=

2. =

0

0,

on

J2 llA.

SIAM

I. 16,

1020-1023

1.7.

FURTHER REMARKS

65

STABILITY

1.4.17. by i= Ax

+ g(x)

by

E2 =

I

A

P

A P

=

0 %

P = X1kl + x2x2 -

-x;

-

+ x,x;,

x;

V ( x )=

=

0,

X

vv +AVV XI

x2

=0

4-h(x2, - 2x1) = 0

+ A(2x1x,

~

2.4

= 0.

G. Geiss, GAEC Rep. R M 3161 (1966). For details in the numerical testing the outlined method, see G. Geiss, “Analysis and Design Nonlinear Systems Liapunov’s Method.” PrenticeHall, Engelwood Cliffs, New Jersey, 1970. l2

l3

66

1.

ITERATIVE METHODS O N NORMED LINEAR SPACES

9 2

= -.

=

=0

+

I

-

27

2

by by p

= f( p ) ,

p

f(p) A by =0

by

As

CHAPTER 2

CONSTRAI NE D 0PTI MIZATIO N ON F

2.1.

INTRODUCTION

on E”

penalty

1943 by 2.3

2.4 by by

67

2.

68

En

OPTIMIZATION

by

3.

2.5.

SEMICONTINUITY A N D C O N V E X I T Y

2.2.

f on semicontinuous x E li

Definition 2.2.1.

R

S

y

E

>0

6

b'

c' E

-f

s.

f

>0

zipper semi-

I

continuous

f by (2.2.2), - E

x < f ( x ) -f ( y )

<

E.

by

x

by book

2.2.

69 {x,} xo.

2.2.3, x,

3

1 n 3N})

N+

by

-

2.2.4.

I

co

a

d

SAf)= {x 2.2.5.

&I. I U

on

A

S,(f)

(Y.

( / y_I xIj -0

>f(x)

(b) Q

Proof.

=

> a).

).(fI

{x

x

E Q,

f(x) -

E.

>a

&f(x) -E

y

S,( f)

x

> a]

01

f(y)>f(x)

<

x

y

B,(x).

= f ( x ) - E,

-E

B8(x),

<

>a

Q

x

f(x)

E

6 >0 B6(x).

(y)

+

E

on

11 y

11 -+0

-x

x,

f ( y n ) < f ( x ) - E,

11 y,

yn

E

6,

-+

- x

I/ < 6,

0.


E


I 2.2.6. by u t ( x ) = 0

uf x

E

So(f)

by u f ( x ) = 1

I

2.

70

CONSTRAINED OPTIMIZATION O N

En

2.2.6

uf

So(f ) S,(f2u) = (x

If%

=

u

0). on f

f

on U ,

=

01

2u

Sg

Proof.

So f (x)

U

on

Lemma 2.2.7.

>0

x

01

6, . 0

E

>0

<

<

S,C

f , f(y)2

f (y) > 0

So SG .

y

-

y 6

x

E

y So ,

=

,6,).

u(y) >f "x) u ( x ) - E.

B,(x) y

I Examples of Semicontinuous Functions

n on En

2.2.

71

AND

R1,

f on

C1

by

L( f )

We

on

on [0, 11

C'

L( f )

by 1 . fn(x) = - .

277x

max

O<x<1

n ---t

oc).

y)

L(0) = 0. by on

Lemma 2.2.8.

<

n

Em)

m by

Y

Em)

Proof.

A

(A,} A, < Y,

A,

Y

A

2.

72

I/ A ,

~

ON

0,

A

En

A, A B, A, B, 4


up

A

B].


r.

I

on

Lemma 2.2.9. n.

I

A,

---f

2 by 2 A,

A ,<

A,

=

A,

i" J 0 -

A, #

2

A,

4

0,

Convex Functions

C

B

on

C Definition 2.2.10.

x, y

to

C

I

A

0

f on

< /I< 1.

B

73

2.2.

on

y

x

f(y)

on

f(x).

strictly

on

Lemma 2.2.12. S,( f )

a.

Proof.

x,y

E

0

S,(f)

< 13 < 1. I on

Theorem 2.2.13.

Proof. f(xo)

xo

0 < /3

x

+ (1

xO)

-

< Pf(4 + (1

~

< :

P)fkO) < f(."
B,(xO) x =

on C 1,

/3x1+ (1

- p) xo E

x

p

B,(xO)

f ( x )
C.

E

Q,

I Theorem 2.2.14. x,y

C1 on E

+

on -

p)


2.

74

CONSTRAINED OPTIMIZATION O N

En

by f(x

-t P(Y

-

4)

0 < 0 < 1.

+

(Vf(x OP(Y -%Y

- -4

(2.2.16)

I

j?-+ 0 2.2.14 Corollary 2.2.17.

f on

A C1

C f(x)

Of

Proof. Of (xo) = 0 > f ( x o ) by

x

I C2

Theorem 2.2.18.

C

H,

on C Proof.

0


g(a)

= 0.

x, y

E

on f

75

2.2. u = ax

+ (1 - a ) y

1.3.13), = a[(V.(u),

).(g

+ (1

-

-4

a"f(4,

5 , ~ on x - u = (1 - a ) ( x - y )

+ $(ff,(Ox - u, x - 4 1 Y -4

+

Y

-

%Y

-

41,

u x, y , y - u = -a(x -

H, on C, Existence of Minima

f ( x ) > -a].

En

on 5

on

{xn>E S

S,

on

Lemma 2.2.19.

f ( x n ) --+ -a. xnv-+xo xo. f(xo)

<

=

-a,

> -a. 2.2.19 f ( x ) 3 y,

on S.

y

(xn}


76

2.

En

OPTIlLlIZATION

f(x7,)

minimizing sequence.

y.

y on

(xn)

by

-+

+

ccj

/I x I/

En

on

f

Theorem 2.220, S

f

---f

f

xo E S. (xn}. Pyoof.

{xn} S

by {xn},

s,,+ xo

on

y

u

xo

\n

xo

S

do

S,

(xn),

1 x '~

f(xn)

---f

-+

f ( x n )< y -+

-:~

-

+ co

11 x, 11

bound

In).

x 11 + 2.2.20

S 1. 2.2.20

5, .f

S

y = f ( x ) on ). s

~

0

j'=

0,

so

2.2.

77

do do

bound 2.2.20 2.2.20 on R1

EXERCISES

2.2.1.

uniform norm

on

2.2.2.

< m,

g,

2.2.3.

S,(g,) 2.2.4.

L? L?

g,

A

n by n

(x,

Ax)

on En.

f

2.2.5.

C1 2.2.6.

En.

on

V ( f 2 u )=

g

2u

u C on

on g2u

g2 n by n

2.2.7. Y

g >, 0

2.

78

En

CONSTRAINED OPTIMIZATION

r

A P

1

<

P


-

1.

if

An-+ A , A (1 P I/ < 1; 2.2.8.

use

on En,

A

11 B

B

-

A 11 < E

E

1.2. 2.2.7

2.3.

A PENALTY ARGUMENT

Some Examples of Constrained Problems

of

gi < 0.

on En by

f

-f.

=

1.7

Y(x)= v(x)

Px),

0. x E En =

x =

[?j

2 0

xvl

book

x

30

k

x

(yi) by

2.3.

79

A PENALTY ARGUMENT

so

a, b,

k

i=l

(Yi- (.Xi

+ +c y . bXi

a, b, c

on E3. 1.5.

As

1.6

f

x =

and

f(x, t) =

(\f

YX>

)

4

3(x, 4

i =f(x, t) =

1 x2( x2(0), xz(0)

=

0, x3(1 )

=

1.

t

=

on 1.

~ ~ (=0 1, ) x 2 , x3

I1 41)Il

=

4 ) l , I %(1)

-

1 1).

x,(O), x3(0)

on

11 x(

E2

x2(

=

0, xg(

=

1.

x(0) on E 2

4 will

80

2.

En

CONSTRAIXED OPTIMIZATION

objective

f on En

cost

rn

s

rn

< n.

do

f , gj 2.2.6.

u

>

g3uj

u,(x)

=

uq,(x).

gj

=

0,

<0 g2u. = g?u2 = 0. 3 1 1 3

<

\TC

zii(x) =

1 feasible

Definition 2.3.2.

I

constraint set

En S,(gsuj)

2.2.7

The Penalty Concept

on

En

[6]:

rigid constiaints are only idealized limiting cases of aery large restoring forces.

2.3.

E2 x2

81

A PENALTY ARGUMENT

+y2

on

x +y

- 1=

(x,y )

=

by f&,y)

= x2

0.

(3, do

+ n(x + y

+y2

~

1)2

f, Of,

0

=

+y

x + n ( x f y - 1) = y + n ( x

1) = 0.

n

x=y=-

1

bound,

n.

4,

x = y -+

-

2.3.5.

K

ki K,

---f

m by m (K,)

K,

co

bound

II + 00.

by (gu, K,gu), gu =

gu

1

( RmUm

gi)

giuj ui

m

fn =

g u y j. n, by

f+

k;,P).

m

2.

82

En

CONSTRAINED OPTIMIZATION

fn (gu,K,gu)

f

n+

00

K penalty constants.

on on

5.

En 2.3

2.6. on f

f on f

on

by

f

Theorem 2.3.5.

En,

f

(gj} on /I x I/ + 00

-+

Q0

feasible set Q, defined by the set of vectors x f o r which the constraints

Kn-+ co { f,) on a,. f n ( x n ) +f (xo),

(xn}

{x,}

xo

xo on Q

=

n QJ. xo; xo

{xn}

f (xn)-+f (xo)

xo

2.2.7

Proof.

A. V.

I.

-+

g$ui by

{xn}

g(x,)

2.2.20

Math. Phys. 41, 291-299 (1962).

0.

2.3.

{f,}

d,

.

Q = Q, n $2,

d

= fn(xn)

=

Q

d,


< d. -+

{xn>

xo

3

f(x,,> f(X,I,)

I/ x 11 -+ 03,

f03

9, .

=f ( x o )

>f(X0)

-

1

v.

<

(gu, K n Y g u ) x," .

(gu)2+ 0

gu(x0)

by

v + 03

< 0.

9,

xo

f. {x,}

=

xo,


n.

fn(xn) + y

y =d

f(.,)

-+f(xo) g(x,)

d, xo

xo XI

f ( d ) = d, d, < d,,

(gu, K,gu) + 0.

I

{x,}

bound

< d.

{x,,>,

d

83

A PENALTY ARGUMENT

Q,


fn(xn)+d.

n

+ GO

x,+xo,

x,

84

En

2.

1%

x

= ,xz -- 3’2

+y

-

n((x

4

1

--

0

~

y

-

x

1)Z

+y

(.%

+y

E2

2

~

=

0.

-

n.

Yf,

by x(n)

2

n

=

v(n)

’

-

CC.

--f

3n

=

2n) ’

f ( 4 8 ) >Y ( 4 f f G ,

3

3

y ( n ) )+ co. \Ve

2.3

(see we s

0

J

Definition 2.3.6. actize binding

j xo

.,

gj(.xo)= 01.

QJ $2,

LJJ,

=

(x

&(X)

i

E

J)

I

2.3.

A

85

ARGUMEKT

Q,

g,

, Q,

Q,

{x 1 gj

g,

5 0,j E J }

< 0.

xo,

gj

Corollary 2.3.7. xo o n Q,

J

s =0

{xn}

2.3.5

j 4 J;

Proof.

g,(xo) < 0

.xn E Q,, .

n

{xTL}

x,

xo.

---f

<0

Q, . 2.3.5.

8=

xo 6 Q, , by Q, n Q,

on Q

f, d, = fn(x,) =

=

n

:=

R

d

=

g,

=

< 0,

-y by

=

< d.

I

2.3.1.

J

d,

d.

>0

n j E J .

=

E2

g,

=

by x y - 1

+

< 0, g,

(1,

3).

g, g,

2.3.7

x > 0. g,. by m

>n

(1, Q,

g,

= g, =

0. you

=

-x

< 0,

2.

86

CONSTRAINED OPTIMIZATION ON

En linear

Ax - 6

=

0

n

i = l , ..., m.

Ca,,x,-bbi=O, 3=1

A4 m by n A n 6.

b

<

m

=

m

n

m

A

> n, n,

< n,

tY

,= 0

FIG. 2.3.1. A typical feasible

inequality constraints by Ax m n

-

b

linear

< 0, x

2.4.

THE KUHN-TUCKER AND LAGRANGE MULTIPLIER RULES

by

-I,

n by n m

b

=

0.

> n + 1.

E2

2.3.1, by

m 2n

+ 1.

E X E RC ISES

E2 0, x by

2.3.1.

x =y =

2.3.2. x

+ y2) + y - 1 = 0, x < 4,

+ y2),

by

+y

-

1 = 0,

E2, by

by

2.4.

T H E K U H N - T U C K E R AND L A G R A N G E MULTIPLIER RULES

f,gi xo

a

2.3. of m m

=s

=s


by

C1

on

En

87

88

2.

En by g 2 = 0

R1

2.2.6

uj

:

f,g, C' on

by Q;). Q = Q, n

Global versus Local

xo xo fo

fo B,(xO) x1

f"X)

+ (p/2)1\6 = f(.x)

I P 11

2

x0

on >f(xo) s

~

x"

a.

>0

p

12.

2f ( x 0 ) ,

x t B,(xn), x

xo

f2

xo

f" =

f,

xo by

fo,

on f2. Cfo = Cf

xo,

x".

T h e Lagrange Rule

n by m g, (rn 4 Tf".

+

by

n by

by

Ygl,..., YgTn, Tfo.

2.4.

THE KUHN-TUCKER AND LAGRANGE MULTIPLIER RULES

r,

89

r

We

C1

Theorem 2.4.2.

Sr,

on

xo

En xo

hi (2.4.3)

V f ( x o )= j
(3.4.3)

r

G

xo,

SZ, SZ,

x0

2.3, 2.3.5

f

x,

xo

x0

on

O)

Q, by x,

.

n,

Sr,

C1,

xo

x,

by r n by r

n.

r

Gr

xo r xo.

2.2.8), n.

Y

x,

r x,

(2.2.4) (2.4.5)

90

En

2.

An,i.

r

n,

2.2.9, r

Vf

G(xo)

=

<

G,(xo)

on Vgl

=

<m -1

<

G,(x,)

,...,

= r,

xo,

by

3.2.6,

n by m

m xo.

I Xi

,...,

< rn,

Em, A

2.4.2 m

=

n

-

1

n by n x,,

f,gj

by

cl, Gr(x,)

---f

Gf(xo) = 0,

Gf(xO)= m, The Abnormal Case

2.4.2 =

G(xo) < on

Gl(xo). x0

A,

no

xo.

Of

no

2.4.

THE KUHN-TUCKER

f,gi

Theorem 2.4.6.

91

AND LAGRANGE MULTIPLIER RULES

U

on

C1

En

xo

A,

Vj(X0)

+ 1 Aj

= 0.

j <m

If

xo

=

A, f 0

xo),

<

G(xo) = m.

{hi}

G, ,

A,

=

0 (abnormality)

G (2.4.7):h

=

xo, x2 $- y 2

< rn

h

E2,

x

0

< m. I

+ y - 1 = 0, by

(3.4.3)

2 by 1 h

h x =y

=

3

=

3,

(2.4.3), 2x = 2y = A.

x

iy -1

=

0

92

2.

CONSTRAINED OPTIMIZATION ON

En

The Kuhn-Tucker Rule

x").

=

as =

> 0. n by 1 xo)

j EJ

by

Tfo.

by

2.4 Theorem 2.4.8.

f , gj

Q:

on xo

E" regularity assumption (Tg,(sO), h ) i 0

h

E

j

El',

E

J,

hi 2 0

so,

2.4.2

Proof.

Q,

f

Q,O

{xn) * €1.

A . Tucker, Pror. (1961).

,\luth.

Sttrtistics, 2mf, Herkelry, 48

2.4.

THE KUHN-TUCKER AND LAGRANGE MULTIPLIER RULES

93

xo, =

0

VfO(xn) =

(2.4.10)

Cg,(xn),

An,,

3 <m

=

=

0

k,,Jg!(x,) uj (x n) 3 0. $ J, xo x, xo. j J n >, N .

n

QJ,

--f

as

Og2 h)

6 >0 by x, , n >, N ,

<

<0

n

x,

--f

( V f o ,h )

xo.

A4 on Vfo

Of

M 3 (Vjo(xn),h)

8

An,j

(2.4.11 )

2 8h.j .

jEJ

An,j by A n , j , An,$

n + co.

0,

gj

---f

Aj

30

C1,

x,

+ xo

=

Ei,

xo,

I

Aj

A n , j = kn,jgi(xn) ~ j ( x n ) .

3, on

A

2.4.8,

s

> 0, h)

<0

2.

94

En

Some Pathologial Examples

E2 g(x,y)

= y2 - ( 3 -

=

0.

g

2.4.1. by (Vgl,h)

<0

( V y , , h) ,< 0.

2.4.2 by

to

2.4.

THE KUHN-TUCKER

AND LAGRANGE MULTIPLIER RULES

95

by ~ X X , Y ) = x2

+YZ

+~ ( Y Z

-

(x

(2.4.12)

1)3)2,

-

n x(n), y ( n )

Of, x(n)

y(n) = 0

x(n) -+1

+ 3n(x(n)

-

1)5

= 0,

n + GO.

E2

vis d vis

f

y)

gl(.,Y)

=

x,

= (Y

-

2)

-

g2(x,y) = -(y - 2)

-

-

x)3

(Vg,, h) = -h,

<0

(3 - x)3

< 0. 2.4.1

by

is

by

E2,

(Vg,,h) = h, Vf

=

(3,2),

h, = 0.

(Vf,h) = h, < 0

96

En

2.

h.

A, , A ,

(3, fn

u1 , u, n 1, 1.

u1 , u2

on

0, 0

u1 = uq = 1

1,0

0, 1 Ofn = 0

n

Vfn = 0

( x ( n ) ,y ( n ) )

by

(3,

2.3.7. 2.4.6

2.4 2.6, by

by

by

EXERCISES

A

2.4.1. by n

n

x E E” &(x, A x ) ,

2.4.

THE KUHN-TUCKER AND LAGRANGE MULTIPLIER RULES

(b, x =

b E En,

1

=

97

A-lb (b, A-lb) .

2.4.2.

11 x 11

on

=

1, x

Ax) En.

A xo

A,

Ax). 2.4.3.

+ (y

-

E2 1)2 x2 y

-

~

<0

g,

2.4.4.

x

+y

-

g,

?

C1 on En. fu

f

< 0.

2

3.4.1 2.4.5.

by

u

(1 u 11

u

=

(Vf(x),u ) ,

1 x,

by u = 1.3.

f,

u -Vf.

,h) < 0

2.4.6. gj

< 0, x0

x

+ p h E B,(xo)]. gj

xo.

C1

xo =

B,(xo) xo p h

+

0. p

2.

98

CONSTRAINED OPTIMIZATION ON

En

2.6

2.4.7.

C1

maximizes xi 3 0.

on

En,

< 0,

>0

xo.

2.4.8.

by (2.4.13),

fn

u1 = u2 = 0 u1 = 1, u2 =

2.5.

=0

u1 = 0, u2 = 1

0.

T H E REGULARITY ASSUMPTION

,h) < 0 s

a,

Lemma 2.5.1.

En. i
,..., ulrL h

(ui, h )

Proof.

<_

0

i > k, 0

( a i , nj) ==

En

< k < vn.

( a i ,h)

>0

Em

{ui)

{Zi)

> 0,

6,.

Ej

h (m -

by

{ui}

2.5.

Ef

ui

Ej

99

REGULARITY ASSUMPTION

Em.

u; E E;,

uj E

(ui

, u3) =

i+j

(ai,u3) f 0

Ej,

us

( a j , us) = 1

> k.

(ui, uj”) =


j

Cj

2.5.1

A ( a i , h)

<0

k


(ai

{ai>

,h ) > 0

k = 0, m

=

0

<j < k 2.5.1 ;

2 (ui, h)

FIG.2.5.1.

< 0.

The nonernpty intersection of half-spaces.

s = 0, Q,

{x 1 gi < 0 , j E J>.

,

C1

Q,

2.4.

100

2.

I?"

2.5.2.

C'

E

(Cg, , h )

0

gj

h

En

xo.

J

j

QJ J,

2.4.8

h

==

x

-

0.

=

x E R, .

x,,,

gj

=

2.2.14

01.

A Sufficiency Argument

on =

2.5.2

Q,

0,

s2,

QJ

2.5.3.

f

gj on Q,

xo

x". h

=

x

xo, h) 0

-

,<

x E QJ. j J.

2.5.2 x

2.2.14

by

2.5.

gj(xo) = 0

2.2.14 f(XO)

x

Q,

101

THE REGULARITY ASSUMPTION

+

<

< f(4

k 3 ( 4

3sJ

.

A Multiplier Rule for Mixed Constraints

1

+ < m,

> 0,

I

c'

f , gj

Theorem 2.5.4.

En 0 <s

s

(I +

n by

on xo

< m.

xo

Xi ,

=

1

3€J

j<s

E

xj

(2.5.5)

<

Xj xo

Proof. n by r

Gr

{Vgj(xo)} by G'(xo) r

Vgi(xo) 2.4.4. s

independent independent <m.

1

by

r

2.2.8, xo.

G'(x)

G(x)

r

102

2.

CONSTRAINED OPTIMIZATION O N

G(x,)

xo, xn+xo.

n

En

G'(x,)

=

Gr

=

x0,

Y

x

k). 2.4.8

2.4.2 Vfo(xn)

by

r.

Gf(x,)

G,(xo) = r Xi 3 0 j

2.4.2,

A,,i

+ Xi

,

j < s

J 3. -gj, dj+s =

-g9

,

<0

j

j

g

2.4.2

J s

< 0,

by

E

< S. =

0.

2.4.8,

.

h

An,j

=

gl(xn)

<0

0

j

Xn.j - A n , j + s

1

<

{xn},

by {x,},

gl(x,)

>, 0,

0

2.5. $n,l+s =

103

THE REGULARITY ASSUMPTION

0

=

< 0.

2 n.

j = 1, {xn}

<

s'

< s'

s - s' 2.5.1,

An,2

50

An,j 3 0

k = s' h En

,h) < 0

<j

m =

< Y,

I +s =r , h) > 0

2.4.8,

hj 3 0

j

j

E

J

+ Xi

s'

hj

<

2.4.2, xo

I 2.5.4

xo by

, h) < 0

E

J,

2.

104

En

CONSTRAINED OPTIMIZATION

2.5.4 by

s =

m

s =

< Vf on 2.4.6, (m

0

xo,

no

+

,

=

A, f 0

< m.

<

,

A,

=

0

Gibb’s Lemma

by 2.5.4

by

C1

book by

n

C x, = 1

and

x,

>, 0.

i=l

Danskin, “The Theory of Max-Min.” Springer, Berlin, 1967.

on En,

2.6.

BIBLIOGRAPHIC NOTES AND OTHER REMARKS

2.5.4

(2.5.5)

p

Vf(x0) = p v

> 0,

x:

,...,

Vgi = Xi

=

if.

x;

-

-

,..., O ) T

hi

30

f: -1 V f ( x 0 ) = PO,

0

pn

n

xs =

p

0,

+ hi 2 p. V f ( x o ) = po,

V f ( x o ) 3 PO,

hi V’(x0)

2 p.

if xo > 0 if xo 2 0.

(2.5.7)

2.4.7.

2.6.

BIBLIOGRAPHIC NOTES A N D O T H E R REMARKS

Remarks on Section 2.3

2.3.5 by

of

of

by

by by

4

A. A. Goldstein and Kripke, Numer. Math. 6, 47-48 (1964). C. Operations Res. 9, 169-184 (1961).

2.

106 x E En

En

CONSTRAINED OPTIMIZATION Y

>0

s =

SZ,

gj).

3, interior

x, f,

Y

SZ, . Y

x, xo book.6 Remarks on Section 2.4

on by

n by m m.

of

3. by

2.5.4

.4.

York, 1968.

2.6.

107

BIBLIOGRAPHIC NOTES AND OTHER REMARKS

by

on pp. 152 by

on on 1969

xo.

( jE J ) , h) < 0,

En

h

B,(xo) y

x on

gi(x)

y

h

< 0.

xo,

2.4.6

by

h)

< 0. book

by book,

book by book by et

aZ.7

et al. by g(x)

=

0.

En. =

’

b,

b E En.

R. Duffin, E. Peterson, and C. Zener, “Geometric Programming.” Wiley, New York, 1967. 8 A . Ben-Israel, SIAM Rev. 10, 235-236 (1968).

108

En

2.

a ab

=

g(.-

b).

-

f,(x, b )

x,(b),

of

{xn>

-n(g(x,,) 2.4.2,

X(b),

-

b)

=

A,

f,(x,(b)) +f(x0(6)); 2.3.5. (2.6.1)

X

f

X on

6 (6 n, X

A,

xu,

=

0

Remarks on Section 2.5

F a, p

a

En

cone

x, y

r, ax + py r,

3 0. En. r

En

by ( a , x) 3 0 by

ai ,

109

2 . 6 . BIBLIOGRAPHIC NOTES AND OTHER REMARKS

r Xi) ai

by (ui ,y )

.A

x

=

n by m

>0

m

x ATy 3 0

y

X 20 2 01. A

x

book by

et u1.’

r

r,

r

2 0.

(x,y ) ( a i ,y )

xE

X Ah; A

ATy 3 0

=

En (2.4.3)

(2.4.9)

linear

VfO(xn)

(2.4.10)

3

by

Yf(xo) (2.4.9). (2.4.5)

by

Vfo(xn) Vj(xo)

(2.4.3). Farkas, J . Reine Angew. Math. 124, 1-27 (1901).

CHAPTER 3

COMPUTATIONAL TECHNIQUES FOR CONSTRAINED OPTIMIZATION ON E"

3.1.

iWTRODUCTION

book

2

by by

A

3.6 on on 110

3.2.

IMPLEMENTATION OF THE PENALTY ARGUMENT

111

by

3.5. En. by

3.5). 2.

En

n

no

3.2.

I M PL EM E NT AT ION O F T H E PENALTY A R G U M E N T

on

K

(2.3.4)

K

2.3.5 K. K

bound.

112

3.

COMPUTATIONAL TECHNIQUES

k,,jgj(xn)uj(x,)

hnj)

k,,j K

no

K, ?) K.

do

no

I(

3.5

K by

K by

n =

k,,igj(xn)

good

hi

-

I hj

gj = 0

<E

E

n,

(3.2.1)

2.6

-

0

k,,j

no

.

hj

n,

on

kn,j

by

3.2.

IMPLEMENTATION OF THE PENALTY ARGUMENT

113

Quadratic Approxi mations

by f(x)=c

+

n

b,xi

+4

i=l

f(x)

x

=

(xl

c

,..., xn)T

A

aijx,xj . i

(3.2.2)

j

+ ( b , x) + $(x, Ax),

n by n

aij .

A is ’

R(Y) = X I Y ;

+ h2Y; + ... + Any: +

(3.2.3)

c1,

A.

hi

8

E2, z

x

by 8 x =

Uz

(x, A x )

by (azz- all)

z (z, UTAUz)

28

+

28. =

UTz)

2% a11 - a 2 2 ’

zlzz 8

3.

114 A,

U

,

a;,

=

COMPUTATIONAL TECHNIQUES

A,

A

A.

= ah2

f (U'z)

/3,

+ a)2 + h,(z, + /3)2 + c, , y1 = z1+ a, y 2 = z , + /3 A,

A,

by

(3.2.3).

A,, f ( U T z )= h y ;

+ by, -

C 2on

En,

S

f

xo

by

(3.2.4)

of

(3.2.2). Of

xo

(3.2.3)

=

xo. V f ( x o ) = 0. A, 3 A, > 0,

+ x)

/3

-

c

extremal

=

>

no

=

2 if

/3

by xo

/3 - c < 0. 2.2.3)

by

n

xo

xo

Of (xo) = 0,

2.2.17). b =0 =

1)

=

xo

1.4.4) 1.4

on

of

3.2.

115

IMPLEMENTATION OF THE PENALTY ARGUMENT

by

on

f. 1.4,

by

== x k

xk+l

G-' Of (xk),

Pk

=

- aRpk >

Pk =

Vf(xk)

0 1 ~

go

ak

Pk

f.

f B,(xo) n

by A, 3 A,

>0 A1

=

xo xo

+ x,

01

=

= A,

,

1.

of

AJA, x0

AJA,

Of or

&/A2--+

co

+ 00:

3.

116

COMPUTATIONAL TECHNIQUES

A, f A,)

y Al/A,

---f

co by

by on R1

x

FVe

by 1;

= x2.

do

2 f n ( x ) - x2 -1

n((x u(x)) 3.2.1 n)

U(X) =

(x

-

1

x 1 f, <.

U(X)

x =

1.

3.2. fn

n + m,

fn, K.

z

1.4. (x,

Gx)

g(z)

7

(2,z )

= Ox,

B

=

1/G.

3.2.

117 x z

11 z 1 '

by

=

(z,Z) = (Bx,Bx) = ( x , G x ) = 11 x

\4( -

x 1

11%.

"( x ,

\

\

\

\

x 1

0

FIG. 3.2.1. An illustration of steep valleys introduced by penalty functions.

x

x

=

Ax

B

A

diagonal

A f ( ~=)g(Z) = (z,A - l G k ' z ) .

x

= B-lz.

118

3.

COMPUTATIONAL TECHNIQUES

A

A-lGA-l

=

dE,

A

3.2.2 A-lGA-l on

Summary of an Algorithm

K (3.2.5),

f, by

xo.

{xk}

En Of,

/ / Vf,,Ij < e l ,

el

= 0.

I/ xk+l - xk 11 < e2 , 1 f,(xk+l) - f,(xk)i < e3 ;

1 gi(xk)uj(xk)I < E,

K xk , 01;

3.5. x”

f x”

Of

-

is Vf,

=

0

3.2.

119

IMPLEMENTATION OF THE PENALTY ARGUMENT

f(x)

< f(3i;) < f ( 2 )- 6

6,

on

x” no

of

E X E RClSES

3.2.1.

f(x,y) = 1OO(y - .2)2

+ (1

-

xy.

(3.2.6)

on

(- 1.2, bog

(1, ?

3.2.2.

real diagonal n by n A X A-1GA-1

f

A:,

i=l

CEIAi = n

A-lGA-l 1

A-1GA-1 of A )

3.

120 3.2.3.

3.2.2 by

4.2

3.3.

0.2.

A CONJUGATE DIRECTION METHOD

by V f , H , Of (x,,)

xk

by fx g,, .

,f x x , f

on on En

by xk+l

= xk - OikEfkgk,

k

= 0 , 1,...,

(3.3.1)

Hk x,

gh

H,,

f dxh

=

1.4.7.

x ~+ xk~ ak

f

-

H,g, ; '"h

xk x.n) = 0.

3.5.

Hk

by

x,,

-Hkgk xh

a,

J ~ ( X ~ +< ~ )f ( x k ) .

3.3.

121

A CONJUGATE DIRECTION METHOD LY.

dfk

=

f ( X k ) - f(Xk+l)

(3.3.2).

01

Hk

=

H?’(xk),

H, Hk

Hk

Hk -+ G-l V f ( x o )= 0.

=

1,

x k + xo

( x , x)lI2 by f

(x, H ; ’ x ) ~ / ~ ;

xk -Hkgk ,

-gk . u

En

( V f ( x k ) ,u ) ,

(u, H;’u)’12

u

,4

+

Hi’u)

=

1)

-

1,

= 0,

h g,

+ 2AHi’u

= 0.

h =

[y

-YH&

9

(gk , Hkgk)p1/2].

Hk

122

3.

COMPUTATIONAL TECHNIQUES

{Hk)

Wkl. Definition 3.3.4,

{pi}

A

G

i # j.

0

( p i , G p j )=

I , )c

(

En, Lemma 3.3.5.

3.3.3).

{pi}

n

{pi> Proof.

xy=lcipi = 0 n

%(Pi GP,) = Ci(P, , GP,),

0=

7

i=l

ci = 0

I

G by

A

P

n by n

pi. Lemma 3.3.6.

p , ,..., p ,

( p i , Gpi)

so

on

G

Proof.

{qi} qi.

U

P

=

=

1.

{pi} P

G

UQ,

3.3.

A CONJUGATE DIRECTION METHOD

123

i = 1,..., n.

=i

%kUik

(3.3.7)

*

k=l

( p i ,Gp,) {pi}

UTU, UTU = I

U

[CF=luij = ( p i , Gpi)=

UTU UTU

p.

<

U U U

=

[ P

U)2 < 1.

=

P

1. UTU

UTU p.

by

2A: I)

UTU

=

i=l

ufi

=

= n,

i = l j-1

Xi > 0

UTU.

n n

UTU =

=

i=l

by

on

p. .

I

)

A:

Xi

=

1

I

UTU UTU

1

=

124

3.

COMPUTATIONAL TECHNIQUES

-H,g,

on En

(3.3.1)

Vf

n

=

0.

by

f on En

Theorem 3.3.8.

{pj}

xo,

by ~

k

+ xk ~ =

-

C Y ~ P ~ k, = 0, 1 , 2,...

x,

ak

V f ( x o ) = 0.

= xo

f

Proof.

+ $(Ox, Gdx),

=f ( x o )

Ax xnP1

Of

= x - xo,

Ax.

=

nk

p k ) = 0.

+ cr,-lpn-l

x,

=

(3.3.9) 11-1

xn = x k + l

QiPz

~

.

2=k+l

by n-1

g n = gk+1 -

C

a i G ~ t .

?=!,-A1

p, ,

{pi}

n-1

('GI

> P k )

=k k f l

9

Pk) -

C

4P k

3

GPJ = 0

a=h+l

by

g, %i =

j


-

0

1

{pj}

1.

En.

3.3.5,

g, = 0 x, = xo.

j

pi

g,

G(xn - xo),

:

i

j,

(3.3.9)

3.3.

125

A CONJUGATE DIRECTION METHOD

The Davidon-Fletcher-Powell Algorithm

3.3.8

H,

{Hkgk) (3.2.1)

n

H, H,

xo,

H, ,

k Hk+l

Ax,

=

=

0, 1, ...,

= Hk

- x,

=

I

(3.3.1) - Bk

,

Ag,

. (Ax, ,gk+l)= 0 [a,

= gkfl - g,

so

Ax,Ax,T (3.3.1) Ax,

Hk by

by by

3.3.1 1

3.3.13 (3.3.1)

(3.3.10) not

1.4.7). W. C. Davidon, Variable metric method for minimization, At. Energy Cumm. Rep. 5990 (1959).

3.

126

COMPUTATIONAL TECHNIQUES

H,

Theorem 3.3.11.

by

H, -Hkgk Proof.

-gk -Hkgk

xk

(-g, , u ) ( g k , Hkgk) > 0. - ( g k , H,g,) xk ( g , , H,gJ > 0

En),

g, -Hkgk

0:

>0

,

-Hkgk -H&k

H, -aHkgk

> 0.

&k

Hk Hk is

Hk+l . q

=

p

.\/pkAg, p.

a,

> 0.

=


(3.3.10)

(3.3.12)

p 3

Agk)

q.

=

=

9

-(Ax,

(dxk ,gk.+l)

gk+l)

-

,gk)

>gk)

3.3.

(3.3.12),

x x ; ( x , Hk+lX) > 0 q. Hk+1

( x , Hk+lx) > 0,

Ag,

127

A CONJUGATE DIRECTION METHOD

p

1 f

Theorem 3.3.13. v f ( x o ) = 0.

{-Hkgk)

n

H,

=

f

Proof.

f ( 4= . f ( X 0 ) Ax

=

x - xo,

Of = gk+l

'gk AX,

+ $((Ax),G ( W ,

= xk+l - xk

Hk+,G AX,

-

gk

=

Hk Agk

(3.3.14)

Ax,

=

9

.

= H,+i

Agk

+ AX, - Hk Agk

(3.3.10)], Hk+,GAxk

HIG A x ,

=

= AXk,

=

Hyl A x ,

.

(3.3.15)

k

01

AX,)

=

k

=

Ax,

(H;'Ax, ,AX,)

( A x i , G A x j ) = 0,

(3.3.16)

.

(3.3.1 5)

O,(i<j
1.

=

k

O z - al(gl,

=

= 0,1,...

Ax,,

HjGAx,=Axi,

-a,g,

k

0

=

-cilH,gl ( A x , ,8,)

=

( A X ,, G AX,).

< i <j < k ,

1. (3.3.17)

(3.3.16)

k 3 1.

(3.3.17)

128

3.

COMPUTATIONAL TECHNIQUES

k xl+l = A X , i- A.yj-1

0

<

k

+ 1.

+ ... t A ~ t + +i

<j, gJ+1 --

gt+1

=

G(x,+l -

XZ+l)9

3

1

(Ax, ,

AX,%)= 0

(Ag, , H,G A X , ) = (Og,,A X , ) = ( A X , ,

AX,) = 0

(AX,

9

g1+d

=

( A x , &+1)-L 3

s=2+1

j

j

< k.

(3.3.10)

< k [Eq. (3.3.17)], E€,+,GA X ,

0<

<j

=

H,G A X , = A X ,

< k. €I,+,G A X ,

= AX,

(3.3.15) (AX,

9

~

+

k 1. (Ax,,g,+l) = 0

( A x , > G(X,+l - xzt 1 ) ) = ( A X , ,G ( A y , ... AX,,,))

g,+1) =

+ +

=0

3.3.

129

A CONJUGATE DIRECTION METHOD

(3.3.18)

7&+1) = 0

0

< i < j < k. Also, Ax, = Hj+,G Ax, 0 = W,+& A%

¶

&+I)

= (Ax2

j

* GH?+l&+l)

by

j

< k.

by

Axjtl

< k.

=

-

ai+lHj+lgj+l,

(Ax, ,

k

+ 1,

(Axi , g , ) = 0 (Axi)

g, H,G A x ,

=

=

i

=

0, 1,..., n

0

=0

(-Hkgk}

> 0.

=

i # j,

a p j ( H i g i , GHjgj) = ( A x i , G A x j )

aimj

- ~ l ~ + ~ ,

Axjtl)

-

conjugate 1.

g,

En,

0. n

Ax,

Axi

=

-aiHigi.

H,G

H,

1

=

3.3.1 1

3.3.13

H,

f . Ho = I ,

xo

f -Hkgk H, xo. xo,

H;l(xo).

k

Hk

-

130

3.

COMPUTATIONAL TECHNIQUES

xo,

3.5

fik

3.6).

H, Hkgk

-lI&k

gk .

2 by 2

Hk

H&k

y1

E2.

3.3.1,

f

t

y2

FIG. 3.3.1. Descent algorithm restricted to certain directions.

3.3. -Hkgk,

131

A CONJUGATE DIRECTION METHOD

Cxk

xk

xk+l ;

no

g,

Hk.

H,g,

Hk Hk

Hk Hk+l =

1. Hk

do

3.2.2: Hk A

AHkA A

by

z = Ax

gk

f(klzk). 3.5

EXERCISES

3.3.1.

u)

(u, H-lu) = 1.

3.

132

COMPUTATIONAL

3.3.2.

3.3.6

I,

U T U

UTU = I.

?

3.3.3.

p,, p, ( p i , Gp,)

(y).

+ (a, x) f i(x, Gx)

f

3.3.4.

E2

,

=

1 :r

1 1 .

‘Vf

=

0

10

H,

3.4.

M

=

I.

GRADIENT PROJECTION

linear

Some Results f r o m Matrix Theory

n by n

3.4.

A, D (r

133

GRADIENT PROJECTION

r

+ s = n). K

L

)

g = (M N *

93 =

I , , I,

AK

+ BM

=

I , , AL

+ BN

Y,

s.

0, 'CK + L)M = 0, CL on by BD-l K,

K

=

=

( A - BD-lC)-l.

-D- ' C K .

N

xzz

(D- CA-lB)-'

L

=

-A-'BN

9?Ol

93

=

K K

=

A-'

+ AplB(D

-

CA-'B)-'CA-'

0%

A

D

=

1,.

134

3.

COMPUTATIONAL TECHNIQUES

of a

(3.4.1)

(3.4.5) K

n by n

Lemma 3.4.6.

D n by n m by m n by m rn by n.

A

(3.4.5), B

K-l

by K-1

< n.

A - BD-lC.

A

Lemma 3.4.8.

m

=

n by rn ATA an

m by m

rn by

Proof.

ATA

(3.4.7)

by v l ,..., v , .

A Grammian

n

=

1

n

i.(

,)i.

X*X,

20

i=l j=1

x

I[ x I/

=

=

1

(xl,..., x,)'. by

V(x) X

h x,

A

{vi}. X

A

x 2.4.2). I:=,xivi = 0

r

=

0

r # 0. I

a

3.4.8

I. Schur, Crelles J. 147, 217 (1917).

of ( F - X I )

by

=

vi 0

3.4.

N

Lemma 3.4.9. n by n

I35

GRADIENT PROJECTION

n by m

m.

NTG-lN

Proof.

3.4.4) rank C

+ rank B

n

-

< rank CB < min(rank c, rank B ) ,

Y by n B n by m. r = n, B n by m CB m. A =T p N . I

(3.4.10)

(3.4.10) 3.4.8

Projected Gradients with Respect to Non-Euclidean Norms

m

f


on En

n

gi(.)

=

1 u , ~ x+~ bi = 0

(3.4.11)

i=1

&(.) {ai>

+ bi

= (a2 ,).

n by m

m

g

(3.4.11) g m N m.

= 0,

by (ai, x)

=

NTx

N.

+ b = 0,

x

gi . Vgi = ai , Vgi ,

E

En,b Em

N

+ bi = 0.

on N

b

( n - 1)-

=

m

SZ

by 12. n

-

m.

P -Vf(xo) 1.3.13),

f(."

+ h) -f(.")

SZ. =

(Vf(0,h),

136

3.

5

COMPUTATIONAL TECHNIQUES

on

x0

xo

C' Q

h

01

>0

ah

B,(xn)]. 0

+ h. xo (3.4.11). + ah) - f ( x O ) 3 0

h


-

aP Vf(X")

-f ( X 0 )

p

(P

=

=

-PVf(xO). -a(P Of([), P Of(.")).

=

I1 p Vf(X0)lIZ 3 0 C',

( P V f ( 0 ,p

5

W(X0))

26 >0

xO. -a(P Vf(C), p Of(.")

<0

ah E B,(xo).

O(

/I P~f(XO)II = 0.

Theorem 3.4.12.

f E C1 on SZ.

x0 1'

Vf(X0) =

(3.4.13)

0,

Q.

Q

=

En,

P

=

I

Vf(xo)= 0. on En =c

by

+ (a,X) + &(x, Gx) x0

x O f ( x n ) - V f ( x ) = G(x" - x).

Of(.)

=

a

+ Gx

(3.4.14)

3.4.

137

GRADIENT PROJECTION

3.4.12,

As

V'(x0) Q.

x0

ai = Vgi

Vf(xo)

91 by

Vgi

52.

Vf(xo)= Nc

N

'7gi

(3.4.14) x

Q,

x0

Q

xo - x

NT(zo- X) W ( x 0 - x) =

c m G-l(Nc - Of(.)).

-x =

= 0 = NTG-'Nc

-

NTG-l

N

0

Q.

N

by

3.4.9, NTG-lN c

=

(NTG-lN)-'NTGplY'(x).

XO

P

=I

=x

-

-

(3.4.15)

PG-' V ~ ( X ) ,

(3.4.15)

no

I

(m =

(3.4.16),

En, by 0

=

I, 0

S;, @ 0

En

=

(3.4.16)

G-lN(NTG-lN)-lNT.

138

3. x

x1 E a

E

COMPUTATIONAL TECHNIQUES

En x2 E Q. G-lNc

a

x1

+x2,

1S).

Vgi)

c

m

N NTG-lN

NTG-lNc.

x1

SZ, NTx = NTx,

=

c = (NTG-lN)-'NT~

x1 = G-'N(NTG-'N)-'NT~.

P by p

G-lN(NTG-lN)-lNT.

Px,

a

Q.

a.

Q

=

I

Px,

x1

-

P

=

=

0

P

(3.4.17)

P P Q,

projection P

=I .

a=

= I,

G

P

(3.4.18)

P =I

-

N(NTN)-lNT.

(3.4.18)

by

(3.4.16)

by

(NTN)-I N T (3.4.18) I - P = N(NTN)-l N T Q. N NTx = 0 x E Q) N,

NT

a:

(3.4.15) s

xo

9 NTx + b = 0, on Q,

NTx = 0. b # 0. x

NTy = 0

b # 0. y f( y) = c

=

x

+ a,

x is

NTx

+b =0

a = N(NTN)-l b.

+ (a, y ) + *( y , Gy)

3.4. =

(3.4.15) XO

+

=

139

GRADIENT PROJECTION

Vf(x)

+ y,

- y = --PG-l

XO -

+

-PG-l V ~ ( X ) G-'N(NTG-'N)-'6

=

+ G-lN(NTG-'N)-lb,

x = -PG-l

- 01.

(3.4.19)

(3.4.15) (3.4.15)

En by 11 x IjG =

1) IIc

(x,

Vf,

a, (3.4.15).

m

f


C1

11 x ! I H - ,

on En (x, H-lx)l12,

=

H xo =

0.

g h > - Sj(X0) = (&(%)? 4,

x

d x = x - xo . on of

gj(xo).

(3.4.20)

(3.4.20) =

=

0,

(V&(3C,),

4 + &(XO)

NTAx +g

= 0,

=0

(3.4.21)

(3.4.22)

140

3.

N

COMPUTATIONAL TECHNIQUES

n by m

m

N7x

// dx !lH-l

1

=

.

+b =0

(3.4.22) (3.4.18). (G'(xo), Ax) N

(3.4.22). m.

N by

11 A x

2H-l

-

1

=

0) is

Ax. (2.4.3) A, , A, ,..., A, rn

Cf ( x ~ ) 2X"H

A, Cgl(xo) = 0

A X -C 1=1

Cf ( x g ) X

--

2AnH-l A X

-t

y

~

--= H

(3.4.23)

11 A x

hi.

m

2 4 Ax

= 0,

=

1

'Cf 4 HNA,

--2Ao, y = -11

Tf

+ Nh , i H . Ax

-y-'(iV*H

'if

L

NTAx

LVTZ{NA)= -g.

+g = 0 3.4.9

( AVrIZA\r)-l

x AX

=

-y-'(I

-

.Y - .Y"

-

( V W X ) '(yg

-

N7H

IlL\'(AVTIlA\7)-1N1) ti -

P

-y-'PIl =

I

-

).

4 HAV(NTHN)-'g

1 I€L\J(lvTHlv) 'g, HiV(NTHLV)-'iVT,

(3.4.24) (3.4.25)

141

3 . 4 . GRADIENT PROJECTION

G-l (--H Of is

H

I/ llH-l p.

y

11 A x [I2 - p

on A x g(xo) = 0,

=

0 by

Xk+l

=

Xk - CikPkHkgk

+

c k

k

,

= 0,1,...,

(3.4.26)

(Pk

C,

P

x,),

ak

Ax, on

{H,)

A

by by NTx = 0,

(n

Sr C En. xk+l

=

x k - f f k H k vf(xk)?

ak

xo E Sr.

Hk

Ha Hk

Sr G, nHnPm= PG-l

xo

P

=

-

PG-' V f ( x o ) = 0 G-lN(NTG-'N)-l NT. no

P =I PG-l V f ( x 0 ) = 0 G-l Of

=

V f ( x o ) = 0.

I/ ItG, by

Goldfarb, Extension of Davidon's variable matrix method to See also maximization under linear constraints, Meeting SIAM, Iowa City, May 1966.

142

3.

(G-l

=

COMPUTATIONAL TECHNIQUES

I). Hk

xk Vf(x0) = 0

k + co.

NTHN, N An Acceleration Scheme

K on

2.3

K

n

f

f, f, 3.2)

-

H,

hj

K

-

n xo

HT'

xo.

H,

et al.

h i , H;'(xo), B. Rosen, SZAM J. 8, 181-217 (1960).

+ *(g, K,g).

3.4.

143

GRADIENT PROJECTION

m

< v,

v

N

xo.

m xo

of

f

g

m X

m

m

+v Vf

+ Nh = 0

(3.4.27)

g = o

f,

xo.

Vfn x,

fn(xn) g, = N , Hn

-

-

+ NKg = 0

H?'(x*)

-

+ NK.)zl-'

=

(3.4.28)

n

HL~(X,),

= [fxz

+ N 3 g + NKNTl-l,

vm by m H,, x, x,

+m

-

= gi(x,)

X

Kg

Xj

v

Vf

n

f (xo)

[(fn)xxI-'

N,

=

+ XTg)==+ NKNT)-l

((f

.

(3.4.29) v

+m

A

by 1.4. fz

+ Nh +

(.fix

+ N d ) d x + NAh g

A x , AX

x,

kn,j

= 0,

+ NTAx = 0

(3.4.30)

144

3.

COMPUTATIONAL TECHNIQUES

x,.

Y

+ m by v + m NKN' NT

H-'

-

N 0

1-'

=

H

(

HN(NTHN)p1NTH HN(NTHN)p' (NWN)-1NW K -(NmV-1

1.

-

(3.4.31)

6Z by D-I no

Ax A X = -PH Of

y =

= x,+~

+ x,

+ HiV(NTHN)plg,

11.

(3.4.32)

K H

=

K

K

E X E RClSES

SZ

3.4.1. n

-

m.

m

3.5. et al.

3.4.2.

145

NUMERICAL EXAMPLES

+

H K

by

K

H H

+ AH = H

-

hi

Kg

AH

HN(NTHN

-

K A),

+ AK.

+ (AK)-')-lNTH.

(3.4.33)

ki # 01.

(AK)-l 3.4.3.

C

3.4.4.

B

T

by n

n by m

B : En -+ En

C B : E n 4 En, B + En).

C:

3.5. NUMERICAL EXAMPLES A N D ONE-DIMENSIONAL MINIMIZATION Search by Golden Section

(Y

Ax

=

-a:H

Ax (Y

by

Ax

L,

=

L, L,

no

Ax

01

=

0, 1.

01

=

1,

146

3. =

01

COMPUTATIONAL TECHNIQUES

2 by

<

xg + A x L,.

B <

a:.

> fc,

[B, D]. L,

=

L, ,

01

L,

on

on L, a:

B, C book by

21,

D

=

=

lj-1

j

L, 3 L, 3

> l2 > ... .

1,

4+

2 2.

9

B<

by 1,

C

L,

1.1

B L,

B’

L,

< C’

=

l3

on.

n

Lnp1, by B < B =C

on on

.

lvL-, = 21,

p.

ln-a lnP3 = 31,

=

+ I,

lnP1

+ 21,

=

51,,

= 31, =

81, ,

by fo

=I1 =

ff = fj-1

1

+

fi-2

(3.5.2)

,

j 3 2.

3.5.

147

NUMERICAL EXAMPLES

{f j > 1, 1, 2, 3, 5, 8, 13, 21, 34, 55,

89, 144,... . Lemma 3.5.3. =fj+Jn,

1

j

Proof.

ln-(k+l)

j =

1% l2 = fn-,In by

+

= ln-k

=

<j < n

Il

(3.5.4)

< k. =fk+lln

ln-(k-l)

k

+ 1.

+

fkln

.

I Il

=

1.

= 1,

n-1

In

-

n l l / f n , I,

=

=

=

In

fnln

11, fll (f n - , / f n ) . I ,

=

Il/fn.

=

144.

D

0.

C by by

(3.5.5)

2 1,

by

li+l, I = - - lj-1 -

lj

lj+l

L l

1

+ r = lj-l/li+l

T =

‘5 2

=

7=------+

T

T~.

+

1

r2

1.618033989 ....

(3.5.6)

148 lj

=

3. ~

l

~

COMPUTATIONAL TECHNIQUES

+ (3.5.5)], ~ 1,-1

==

71,

= T21,+1

,..., 1,

= Tn-'ln

4

In=-.

7

by

p. Tntl -

n,f,,

I,

-

f n

?+I

=

( -7)-(n+l)

Id.

1,

by

ll/fn

by

n n >, 1 9 / f l = 1.1708 .

lJ~~-l, Z1/5-"-',

(3.5.7)

fi

1,rn+l/d5

1 . -

by

by a priori

I,

=

ll/f,,,

(3.5.7)

on n). n,

B

by by ll/r

B

D I,, .

C

Some Numerical Results

y)

=

y ,by

E2

+ (1 - x)'. 3.2. by

3.5.

149

NUMERICAL EXAMPLES

17 (- 1.2, 1).

on

f

=

-

XI)+

+

-

-

+

11 Vf /I <

x:)~ -1 (1

-

+ (x, -

+

-

-~3)'

I)(x~

-

1, 1, =

38

0

(-3, on

11 Of I/ <

f<

-

1, - 3 ,

-

on

Iterate

XI

x3

Jc2

f

Jc4

~

0 1

6 12 18 24 32 38

- 3.OOOOOO 0.291019 0.809222 1.08393 1.38862 1.30185 1.01086 I .ooooo

.oooooo

-1

0.675471 1. I 344 1 1.94672 1.72897 1.01925 1.ooooo

- 3.OOOOOO

.oooooo

0.758285 0.837651 0.046068 0.240571 0.986669 1.ooooo

0.272845 0.9942 1 2 1.ooooo

x:

-x1xgx3x4 x i - xg = 0.

-1

+ xg

-

1

=

0, x:x4

0,

0.8,

19191.9 134.292 10.0278 3.88486 2.20103 0.742350 0.001 563 0.000000

- x3 =

B, 0.8,

(f

65 12

=

3.

150

/I Of

j/

COMPUTATIONAL TECHNIQUES

< (3.4.32),.f

I/ Of / / <

-

a

16

16 (0.793700, 0.707106, 0.592731,

0.840896). 10

x2

11 Of /I <

10

xg

f

45

=

-0.250299,

(3.4.32) x2 100

by x4 100

f

123

11 Of /I <

=

-0.250299

1818

82 on

of

3.2.

C f

= X:

-

+ llx, $- xg

by

ki= 16 ;

kj

0.8, 60,

:

2/z,<2)

24

f

-

2/2

3.6.

BIBLIOGRAPHIC

11 Of jl < (&!,

151

AND REMARKS

0, fl) 34

f

-

kj = 180, 7.8. K.

to

EXERCISE

3.5.1.

A

3.6.

B I B L I O G R A P H I C N O T E S A N D REMARKS

Remarks on Section 3.2

up

A

1963,

by

2

by

f gi < 0, s <

m

< m,

=

< < (3.6.1)

I

> 0,

by -r

C sd<m

-gj(x).

3.

152

COMPUTATIONAL TECHNIQUES

f,by

Y

x,,

=

{x

I

< 0, s < j < m].

x,,

x, .

x,

(3.6.1) Y

fr no r

f,

--j

0.5

,< s by interior

S S S xo . g,(xn)

xo

> 0,

xn ,

Y.

xI c :

A.

TP-155(1965).

j

>s

Res. Anal. Corp. Tech. Rep. TP-96 (1963);

3.6.

153

BIBLIOGRAPHIC NOTES AND REMARKS

Remarks on Section 3.3

3.3.6 by by

by

(3.3.9)

[9]. Ai Bi

=

on

Higi c

by f(x)= n by n

+ (a, x) + &(x,

A;:

x1

by

A,,

x1

= xo -

aoH0 V’(xo).

Y

Ho

by r .

B

Y

Powell, Comput. /. 7, 155-169 (1964). Fletcher and C. H. Reeves, Comput. /. 7, 149-153 (1964). Hestenes and E. Stiefel, N a t . Bur. Standards Rep. 1 6 5 9 (1952). B G . E. Myers, J. Optimization Theory Appl. 2 , 209-219 (1968). lo W. C. Davidon, A t . Energy Comm. Rep. A N L - 5 9 9 0 (1959). lL Pearson, Res. Anal. Corp. Tech. Rep. T P - 3 0 2 (1968). sM.

’

n

154

3. H,

=

H, x1

n

+ I - B(BTB)-’ Br. H,, -+ H , , Y =

- Y

H,,

n,

=

G-l

by by

H Hi

[Eq. Hi+,

A,

=

.

-

Bi

A x k Ax: Agk) 9

I/ Ax, 11/ 1 1 dgk I/ x + Ax, y.

Hkfl

A Ax A Ax I/ A x k ~Ag,Ij ~ / ~ ~ Hk - Bk ,

-

--f

y

Hk+l

3.4.

l2

IAS,

Y.

1.4.5).

dg + A-’ Ag

Remarks on Section 3.4

l3

of

Math. Comp. 22, 665-666 (1966).

-

y2, (Hk -

y

Bk)Ag,

A,,

Hk .

=

0.

3.6.

155

NOTES AND REMARKS

n-1

Ai.

= i=O

n-m-1

PG-l

Ai,

=

h.2)

i=O

P S

,...,

Ax,,

n -m

STGS = A, n - m by n

-

A ( A x i , Axi).

m n by n

n by m

S by S

n by

1

=

m

Q. G-lu {ui}

u =

u

CyIy ciui,

Q,

=0

(u,

Q.

S

S

= A,

l4 D. Goldfarb, Extension of Davidon's variable metric method to maximization under linear constraints, Meeting SIAM, Iowa City (May 1966).

3.

156

COMPUTATIONAL

n--nr-1

A x i Ax: (Axi,CAx,)

1=0

n-m-1

Axi

P

=

=

Agi .

m

=

0,

I

H,. et al.

II

on

=

I.

11, K.

Remarks on Section 3.5

by by

et

on En 15

lo l7

,

irr York, 1962. *VASA Rep. 67-FM-172 CAEC H r p . RE-300

3.6.

157

BIBLIOGRAPHIC NOTES AND REMARKS

of by

3.5.

on

A 54

(-3, - 1 , -3, - 1 )

<

on

2.3),

ei

6,.

The 6,

(3.6.3) on

6,

by

(3.6.3) by

in

Sj

by

l8

/. 8, Res. Anal. Corp. Rep. TP-302

Comput.

ao

21 22

23 24 25

A t . Energy Res. Estab. Rep. TP-340.

Tech. Rep. Comput. Comput. 111,

7, 10, 14,

158

3.

COMPUTATIONAL TECHNIQUES

no by by

As

4.2). As by

et

~ 1 . ~ 7

Barron, AZAA Preprint 67-600 (1967). E. C. Stewart, Kavanough, and H. Brocker, Study for a global search algorithm for optimal control, Congr. IAAC, 5th, Lausanne (1967). 26

27

CHAPTER 4

CONSTRAINED OPTIMIZATION IN FUNCTION SPACE

4.1.

INTRODUCTION

91

by 1943

159

by

160

4.

5. 4.2

on no

4.3.

4.2.

PROBLEMS O F O P T I M A L C O N T R O L

by

A Problem Concerning Evasive Tactics

body

fz

r

SZ

r. P ( r )

P(r) on

q

x,

r

=

cos 0

=

8,

by

30

by

(4.2.1)

4.2.

T)

8(t)

P(T). P(T)

161

PROBLEMS OF OPTIMAL CONTROL

Q,

'p

r.

of

of

y

En,

on

by

<0

'p.

by g(x, , xp)

Q

by

52 g(xl ,x2) g(x,

, x,)

<0

=

mR. T

on

Q).

by (4.2.3)

state x3(tj)

8 on dzfferential constraints

, x2) < 0.

r.

minimize (4.2.3),

tf]

r,

c

tj

by B. 0. Koopman, Hide and seek games, Conf. NATO Search and Detection, La Spezia, Italy (1963).

162

4.

CONSTRAINED OPTIMIZATION IN FUNCTION SPACE

A Problem in Production Scheduling

y(t). d

by

t

s(t) T

[ y ( t )- d ] dt

s(t) =

(4.2.4)

0

T c( y )

h(s) c( y )

g(

j 30

=

=

g( 9). uy, h(s) = bs,

(a,b

TI (4.2.5) =

q

q.

y)

3 0, y

J,

0.

z by t = bs

-tuy

+ max(q, 0).

(4.2.6)

q minimize z( T )

TI (4.2.6)

S=y-d

(4.2.7) Y=Cr

z(0) = 0,

y(0) constraints

-s

< 0,

-y by

< 0.

state

S. Karfin, l’he IVIathematical Theory of Inventory Processes, in “Modern Mathematics for the Engineer” F. Beckenbach, Vol. 11. NlcGraw-Hill, York. 1961.

4.2.

163

PROBLEMS OF OPTIMAL CONTROL

A Problem in Rocketry

D,

g

by

x

=

v (4.2.8)

h T(t)

m

v

0 < T ( t )< M . T(t) tr] (4.2.8), 0

-h(t,)] v(O), m(0)

<

[minimize h(0) = 0 T ( t ) M.

<

z

z(x(t,), t,,)]by

j

=

controlfunction u. 8, q, T. i= f ( x , u, t ) , x(0) 1,..., m,

u

x u)

=

x

= ci

x

I x,(t,) - ci 1.

t, tf

< 0,

164

4.

CONSTRAINED OPTIMIZATION I N FUNCTION SPACE

Admissable Controls and the Ritz Procedure

optimal control

z

u

by uo.

u uo. u

uo uo

L,

on

tr].

L, u ( t ; p),

u(t)

-

p

r

u

2r

1

r

Y

-1

switching points);

t

tf

p,

m

=

2r. on

tr].

u ( * ,p)

/3

by

Em. z

L, Em

m.

z by

u i= f ( x , Z L , t )

on L , . u u ( . , /3) u 'CV,,,

z

z(u)

7

'\T. Kitz,

I.

z(x(t, , u ) , t r ) . d = z(uo)

/3

E

C L, >0 z ( u m )- z(uo) < E

Retne A n g ~ z c .Math. 135, 1-61 (1908).

E

4.2.

165 on

m

L,. z(zZ,,)

<

z(u(., p)),

z(GwJ = z(u,). z(G,) =

u(., 6)

W,.-l

z z(Gm)

01

W, . d

--f

< 8 < ~(a,) < z(M,) < z(u') + z(G,)

-+

W, Em.

,8

m

Wmp1C W , 8 ,,

E

E

+c o

W,

Wm-l 6 >, d.

E =

d

+

E,

d.

z

L,

by

L, ,

uo

W,

by uo u( ., ,8);

by

uo

Constraints and Penalty Functions g(x, u )

by

unconstrained

do 5. Theorem 4.2.9.

f,h: on

H C L, .

< 0, on L, .

z.

166

4.

H

{u,}

{f,)

by

fn(Un)

-

H.

{un}

uo

hi = 0, f(u">.

u, + uo

I

gi(x, u )

< 0.

h(u) = 0

g(x(t), u ( t ) )

gj

t).

< 0 on g <0

12 = 0

tr]

4.2.9 by 2). g(x, u )

< 0,

x,+~ by

g

<0

~ ~ += ~ 0( by t ~ )

4.2.

167

OF OPTIMAL

a

uo z ( x ( t f ,u ) , tr)

j

=

~ , + ~ (t ~01 ) = 0. minimize A? = f ( x , u, t ) ,

tr)= 0

terminal constraints

x(0) 1,..., m.

a priori.

tf

by

let t = i = f ( x , u, t )

Also dx,+,/ds

=

=

0

11.

s

0.

.

t,

=

1.

=

xntl .

on Some Feasible Algorithms

good

by

on

3,

on by R. S. Long, AIAA J. 3, 1351-1352 (1965).

4.

168

z

u ( - ,p), i=

z

u, t ) .

z

u, by on do

z. z

2.3 by

on

z(x(ti))

on ETn by by by

by

i

=

0,

..., N

~

\V. 2nd,

1.

4.2.

169 -

f

=

0

N-1

Z(/3) = Z(X(t,), tf)

+g 1

-f”2

(4.2.14)

i=O

on

En

2. (4.2.15)

u

f

u, /3,

do

up

et d 6 ) . 1.6

by by

et a1.,6

by

et al., York, 1962. ‘P. Control Systems 3, E. R. Operations lies. 14, H. Progr. Astronaut. 17, 401-434 lo A. SIAM Rev. 10,

Advan.

170

4.

by

et

on up

1966

by

by

EXERCISES

L,(Q),

4.2.1 on R1.

L? L,

L, ,

by

L,

Co

Co(Q)

4.2.2 max h(u(t)),

21

h

o
u

by

11 S. AC-12, 132-138 (1967).

l2

l3

A.

Proc. IEEE Prof. Op. Automatic Control

A.

IEEE Conv. Rec.

4.3. 4.3.

171

THE NEYMAN-PEARSON MULTIPLIER RULE

T H E NEYMAN-PEARSON MULTIPLIER RULE

2.5 xo E En

C1

f, n

xi 2 0

xi = 1,

and

X

V’(xo)

A.

a

by of

2

H

C L,

rp /3, y

En.

SZ qo

oi

(4.3.3)

f,g p, l4

J. Neyman and E. Pearson, Philos. Trans. Roy. SOC. Ser. A 231,289-337 (1933).

172

4. y Q, p:

by by

4.2.9

hi by

u

2.2.

=

0 on

y-(x) 3 P(x)

on

on u1 , u2

ut3

u3

p, y

hj(v) 0. = 1

on

hi on f,g,

I/ Q,,

---f

0 (il

v

on (/3 - ~ ( x ) )ZL,(.Z') ~ on ,8 = -a. ?f ]+, ?g/+ 2g/+ # 0 on Q) on L2(Q).

L2

4.3.

173

Gateaux Differentiability

I on

Definition 4.3.7.

H,

1

p

u

(4.3.8)

dl

U

p

I

H,

I

on U.

1. on U ,

I

on

1.3.9

U. I

I

I

Lemma 4.3.9. dZ(p0, u ) =

0

po

+ tu.

p

t

po

I t 1 <6

1 t I < 8.

by

dF(p, u )

Z(po

=

dl(P,, u)Itxo = p(0) = g'(0)

6

> 0,

t

=

+ tu) = q(t)

0.

p(t).

q'(t) dl(rp0, u )

U,

H.

u

Proof. dl(p, u ) = p ( t )

on

I

H.

U

= 0.

I

po,

WP)= R f (P,, 4 dx. do

by p

l5

f 4.3.1

Gateaux, Bull. SOC.Math. France 50, 1-21 (1922).

4.

174

CONSTRAINED OPTIMIZATION IN FUNCTION SPACE

7

< t + 0, by I/ qu I/ o + F on L,

gradient

f on En,

C1 Of(%)

df(x, h)

=

(Of (x), h )

A

hi

ag/ap,

4.3.1 A Multiplier Rule

4.3.14 n = 1 , 2,

by FAY)) = F ( v ) p,

+ ;( a d +

%(pi)

+ h:(F))

H

on

(4.3.10) {yn}

H. yo.

(F,}

by

4.2.9

af /&p

on L, , on f

F(q) y

4.3.2).

h$

4.3.3 on F.

4.3.THE

H m

>, 1 ,

175

NEYMAN-PEARSON MULTIPLIER RULE

Wm,,L L,

=

Wm,z(sZ)

Cm(sZ)

IIVII:

=

1 I/ m/1:

(4.3.11)

9 <m

(11

L,

on Rn

Dj

j;

5.3 {F,} L, norm convergent

by

jl 9, ]Im \<

5.4

5.3 5.4.2)].

v,,

F,

4.3.9.

G

An,3 A,

A&x)

30

>, 0

j

=

An,3 q0

90

af lay,

ag/+

11 q,,

-

y o /lo -+0

1, 2.

u3

176

4.

CONSTRAINED OPTIMIZATION I N FUNCTION SPACE

E

P(x)

An,l

x

< /3

EE

n

Am,3

x, n.

An,2

An,3

A

qn

/3 < qo < y.

on

< qn(x')< y ( x )

2

by

on E .

/3

An,,

=

0

n

p O ( x )3 0. po(x) 3 0 on qn pO = 0 qo(x) > P(x)

L, An,, po =

L, yo((.)< y ( x ) .

0

Theorem 4.3.14.

yo (P)(x)7x)

dx

W'm,z(Q), m >, 1 (Q

--a3

< P(x) <

2f

En),

< y(x) < +a dx

P, y

>, y .

<

L, agjaq # 0

on L, .

4.3.THE

po , po 3

TO

< y.

177

NEYMAN-PEARSON MULTIPLIER RULE

0

L,

> /3

on

po

h

po

30

on

by

A

(4.3.15)

/3 < TO < y

on Q,

y =

h

on

=

-A),

tp'

+ co

(4.3.15)

/3 = 0

g(T, x)

(4.3.15)

=

on L, on

2f

Of

En

2.6 F on

Cl),

=

T,

178

4.

CONSTRAINED OPTIMIZATION IN FUNCTION SPACE

4.3.14

Po(4

n,

- -4w

- p.n(x))

44 F,.

yn

D

Rl),

4.3.14 by by

et al.

An Example in the Theory of Search

rp

D C En D p(x)

20 by x

SZ

by 1 - e-7'")

by F(cp) =

p ( x ) ( l - e-q(z)) dx. R

p

p dx = a. R

rp -+f a ,

4.3. v(x)

179

THE NEYMAN-PEARSON MULTIPLIER RULE

SZ

<

4.3.14,

vo -p(x)e-w(2)

r

(4.3.17)

POW - Po@) -

p(x)e-wo(2) = A

p(x) -

vo(x) =

h

> 0.

=

0 < yo(x) <

h on

0,

-p(4

= Po(x) -

p ( x ) < A.

yo =

-p(x)cM

p(x)

= -pO(x)

-h

> heM. d(x) =

jR

p(x) -

0
h

d(4dx

I

3

< MI E I

I

(4.3.18)

!

y o > 0,

on a

I

I.

A.

h 1 ~

I Eo I

(-a

by

+ M / EM 1) + 1

EM = (x 1 p(x)

do

dx =

A,

(4.3.19)

Ell

> heM)

Eo =

- EM ;

h

=

0,

180

4.

SPACE

p(x) = 0

qo = 0

q0 =

p ( x ) > 0,

M

An Example of a Control Problem with Quadratic Cost and Linear Dynamics

1 j T [ ~ ( t d)t ], ~ 0

i

x(0)

= q,

=

1

1, x ( T ) = 2, T 3 3,

I

i= p X(t) =

1

+ j t p(s) ds

j T (1 i'p(s) ds)2dt.

F(p) = 'z

' 0

0

q

x p.

1

-

ds

=

=

1

$, ~ ( s ds )

C,

2,

u(s) ds.

x(s, t ) = 0

3t d F ( y ~u, )

=

1;

~ ( tdt)

j T~ ( s ,t ) u(s) ds 0

=

u(s) ds

x(t) S

dt,

< 1.

4.3.

181

THE NEYMAN-PEARSON MULTIPLIER RULE

F S'x(t) dt. S

1"x ( t ) dt = pn(s) A + J,'x(t) dt

f ' x ( t ) dt

-

Jn

/3

=

-

po(s) -

y =

J O

A

x

xn(t)

TO,

G(s) =

j'x,(t) dt

= @(s)

1 < @(s)

<

- pn(s)

n

1

Eo =

-

-+ A.

=

0 on En En.

po = pn = =

@(s)

(4.3.21)

0

0 on Eo.

qo

on TI.

q~~

on

xo i= q,

up

qn =

1

1

=

~ ( t= ) 1

t

t >, 1. tl] C

En,

V(S)

ds

TI

1~9 1=1 p,, =

+ Stn

31

~

t,

pn TO

on

=

>, A.

0 t,

t,,

,

x(t)

> A.

t,]

cpo =


0 s

> t, . TO

+ 1,

xo( T )

T

2 =

on

tl]

=

1,

,< A. by

182

4.

io=

CONSTRAINED OPTIMIZATION IN FUNCTION SPACE

s = 1

xo < 0

by xo

< h on

t,];

s

s

so t, , G(s)

of

yo

2.

xo

FIG.4.3.1. Optimum trajectory for a control problem with quadratic s

t,

=

1,

t,

=

T

3

=

by qo = 0 on

2

> 3,

t2].

t, QP :=

T

-

+1

t,

,

xo

t,

2. xo(T ) =

q0

by

1

cost.

1. xo(t) =

T

h

yo

O < S < l l<s
2.

t,

=

4.3.

183

THE NEYMAN-PEARSON MULTIPLIER RULE

T = 3, cpo

EXERCISES

G

4.3.1.

(/3

ht

.

on - cp)2

u1

cp

p. 117]),

h:(vO + tu) t

=

-2

n

(&)

on cpo

u1

- cpO(x)

+ yu(x)) u l ( x ) u ( x ) dx,

+ qu,

0
< t.

t

of u1 yo.

u1

-2(/3

- rpo)ul.

4.3.2. =

4 dx,

R

F

on L,(sZ) on Lz(sZ).

p.

4.3.3.

hj2 on L,.

(4.3.4)-(4.3.6)

af

-+

0,

184

4. Fatou's lemma, yn(x) 3 0 yrL(x)

dx not continuous

a,(%)

---f

n(x)

L, ,

qn +0

[/3

ul(x)

-

3 0,

h: h; . 2.2.6,

h: 4.3.4.

by 2

x(0)

=

1.

I(?) = 4

= -fix

+ cp,

TI

q

(x2

$- I$) dt.

0

J by x

J.

(P

po

4.3.9

dJ(cpo, u ) = 0. y o , xo by

k+Px--g)=O zi

x(0)

K

7

1,

pu

~

~

x =0

0. t, K 2 - 2K

cpo =

=

R+

Kx, ,

K -

1

=

0

K ( T ) = 0.

CHAPTER 5

W E A K CONVERGENCE IN HILBERT SPACE

5.1.

INTRODUCTION

3 on En

on no

5.2

is

on on En. convex

on

5.3,

convex by

1. 5.4. book. 185

5.

186

WEAK CONVERGENCE I N HILBERT SPACE

WEAK CONVERGENCE

5.2.

1.1

H

(u,

11 u I/ = (u, u)li2,

H,

u, ZI

H

H L2(Q) L,

on

Rn;

i2

by (u,V )

11 u,

{a,}

u

(un> H.

-

-u

u,

=

J

U ( X ) V(X) dx.

H

I/ -+ 0

u

u) I(u,

< I/ u 11

H. (u,

-

11 v (1

u,

---+

{u,) =

L,(Q) ZI

Q

=

11,

(u,

, nnx} on

L, {un>

/I

/I m x

11

=

nnx -

dq, by

Lemma 5.2.1.

H. Proof.

nmx

I/

=; ‘v

I].

0

er

u,

-

5.2.

187

WEAK CONVERGENCE

u,

II 24 I/ <

II 24, 11.

I

(5.2.2)

u, -+ u

on H

by

11 u 11

H

A

Theorem 5.2.3.

u,

I

-

u,

<

u

11 u, 11

<M

5.2.3,

H norm

not

i2 = Lz(Q) 1)

B,

nnx)/2/.r!2

do Weak Compactness

H,

Theorem 5.2.4.

Proof. M {vi}

(u,,

, vz)

{u,} i(u,

11 u, 11

,

<M v

H. (u,,

, vl).

(u,

, of

H.

188

5.

WEAK CONVERGENCE I N HILBERT SPACE

{un}. by tun,}. {un,} gs

j

( u n k ,vi)

k 3j.

z'

H,

by

vj l(unk - u n s >

.)I

< ;(unk

+-

> 2, - ~ 2 ) ;

l(uns

< 2MI' Z'

3

+

l(unk - u n 8 9

vj)l

ZJ - z'j)i

- vj

I1 +

uns

~

9

I),.

{unk}

(uric, c)

H.

e

I

(un$

5.2.4 B. A

fun,} B by

B

Theorem 5.2.5.

H

B

B Proof. u =

0. u,,

-

B

{un}

H.

u

{un}

I(% n

unl = u l .

0, 3

unl)J

3 n, > n, , I(%

, u,J

<1

n, .

<4

n 2 n:%> n, . nh+l I(un,un,)i j = 1 ,..., k .

I(%

< I/k 5.2.3,

S . Mazur, Stzcdia M a t h . 4, 70-84 (1933).

3

<1

u,*)I

n

3

> nk

I/ u, 11

an

5.2.

189

WEAK CONVERGENCE

M.

by

on

+ ... + unk II + anz + ... + unk.1,

unl

Un,

< kM2 + 2.1 + 44 + +

-

.*-

1 ~

< K(M2 +

k /K) &lku,9

B

(Ilk)

B

unj

B

1

B, by

be 5.2.4

5.2.5.

B

Theorem 5.2.6.

H,

I

B

B

B En

B. 5.2.3),

En

5.2.6 by

B&O)

= {a

I /I U

- a0 /I

<

I).

L, Weak Semicontinuity

I/ u,

-

u 11 -+

0

u,

-

5.2.5 u.

on H 5.2.5

190

5.

WEAK CONVERGENCE I N HILBERT SPACE

f

Theorem 5.2.7.

f

on H .

Proof.

S, a.

<

S , ( f ) = {u 1 f ( u ) S,(f) 2.2.3 by 2.2.8. 2.2.3 by

f

B}.

a;

I

f

2.2.14, En.

f Theorem 5.2.8. B on f ( u ) -++a [I u /I 4co on B.

B Proof. f 5.2.6. 2.2.14

H . If €3

f by

B

f

f

I

f

5.2.8

Corollary 5.2.9.

Proof.

by

I

5.2.8

-f 5.2.7

5.2.1

by

5.2.

191

WEAK CONVERGENCE

As continuous. =

II 24. II.

< M I / u 11

j f I).(

f on H

M

H.

u

on

Theorem 5.2.10.

H

=

(u, v )

H.

v

Proof.

H,

u, w u

+ w 112 +

112

u -

= 211 u

H.

(u,> u,

f(Un>

,

+

+ 211

112

20

112,

1)

f

M If(u,)/ -+ M ,

// u, / /

by

a

=

f (u,)

+ < MI1un + urn>

=f ( U n

urn

---t

M

M

=

11

(5.2.10),

ll un - urn /I2 = 4 - II un + urn /I2 +4---

4M2 M2

-

0.

(u,)

)I u, - uo I/ -+ 0

uo

H.

F. Reisz, Compt. Rend. 144, 615-619, 734-736 (1907).

192

5. f(u,)

WEAK

f(uo).

=

2,

=

Muo.

( u , 2,)

=

by A

u = uo.

H

u

0

=

f ( u ) = (u,

on A. M2= =

[I(A. u )' ] ~ < M 2 ' u0 - XU ~

=

2X(U, UO)

~

+ xyu, u)),

A

u

11'

11 uo 11

X

=

-2X(u, u") -tP ( U , u) 2 0.

X ( u , uo) =

=

u u1

0

0

uo

A'); f(uJ

- l(u, uo))3

= (u , uo)/(u,u),

0 CYUO

= ( u , TI) on = ul. 01

A.

u

f(u)/M,

A.

u

H A.

uo

f

f ( u ) = (u,

u.

I

5.2.10 on If

EXERCISES

5.2.1.

n

Lz(Q)

rnx)/1/1, 2 ,... .

u =

11

Q =

(u,}

=

(u,

//

n # m, 5.2.2.

5.2.3.

//

u,

u,

/I2

=

/I 411 u /I.

, u m ) l / z= 0

1

I/ u,

-u

on En

11 + 0

5.3.

FIXED

u1 ,..., u,

H

K

5.2.4.

u1

0 1 ~

ai 3

0

193

OF NONEXPANSIVE MAPS

a1

+

K,

+ --.+ a, u,

+

01,

=

K

1.

Eq.

5.2.5.

?

5.3.

F I X E D P O I N T S O F N O N E X P A N S I V E MAPS

1

<4u 0 ,<

(Y

-

1,

< 1. nonexpansive

01

on

< 11 U

llf(4

-

v

/I

u , u.

5.3.1 5.3.4

[4]. Lemma 5.3.1.

B H.

B g ( B ) = {g(u) I

f g

u

=

on H

I -f

I

B}

Proof. u, v

H (go4

g(4 U v) = 11 u - 2: )i2 -

~

u

- a)

>, 0.

(5.3.2)

194

5.

WEAK CONVERGENCE I N HILBERT SPACE

(urn}

+ vo

vo

5.2.6,

{un}, u,

by u

0

< (g(u) - g(u,),

u

-

u, = uo

+ av,

uo

uo

H.

(5.3.2),

u,)

(g(.)

by

-

-

> 0,

OL

vo, u

n.

- 240)

(5.3.3)

30

H

v by

u by ua

(g(u,) - uo, v) 3 0.

As

CY

(5.3.3), (g(uo),v)

0,

-+

g

on H . (a0 - g(."),

v

=

v)

v,

< 0.

vo - g(uo)

I/ oo - g(uo)li < 0

no = g(uo).

g(B)

I f

Theorem 5.3.4.

on

H

f Proof.

g

=

0

I - f.

f o ; ( u )= a f ( u ) . 11 ).(af - f,(v)il


< a /I u

-

fa by on B

fa

v 11.

H.

1.2.5 u,

durn)

= ua

-

f(%)

= (a -

-

a.

0

5.3. 01 3

195

FIXED POINTS OF NONEXPANSIVE MAPS

1, 5.3.1

g(B). uo

I

= uo.

g(B) B

g(uo) = 0.

f

5.3.4 B.

on

B

B

fa

B,

a

B

af(u)

<1 a,

by

of

+ zl =

on L,(sZ),

J

Y ,u(Y))

R

L,

v

by %+I = m n >

uo

+ As

+ u,

(5.3.6)

H.

f

Theorem 5.3.7.

on

H.

v

H

by

(urn>

uo

Proof,

(5.3.5)

H. by g(u) = f ( u )

g

g

+ v.

uo u0

g. If uo

z

n,

j( u,

(1 <

I( u,

5.

196

SPACE

WEAK

6 {un}: =

n

in1

c, =

1'

U,

~

U, 11.

n aU3),

J>A

uI ,

6

B,(u,)

c = (J c,. A 2 1

C, C,

{u]}

C, C, ,

C

k,

u

g(u)

I

j

U,+lll

~

k.

=

K

j

B,(u,)

' z(u)

g(uJ I

~

<"U

~

U)

C g(C,>

< 8, j 3k

B,(u,)

g(u)

I'

+ 1,

g 5.3.4, g

by

1

u

=

f ( u ) + z' {un)

u

{un)

n do on

C" Q

R".

( 7 4 , E)?,l

on C"

(D'% 1)'C)o

= 3 cni

3

by

5.3.

197

FIXED P O I N T S OF NONEXPANSIVE MAPS

( ,

L,

Di

by D3y =

82, 212

Ij 1

=

j,

... sin 2171

+ + j,

n

*--

C" Wmv2(52) Wm+1,2 C Wm,'

on C" by

Wms2(52).

( , )m WOT~,

3 0.

1 by u =f(u)

by untl n -+ CO. If u, + uo

{u,}

f(u,+,) - f(un)

+

--f

+ v,

f

Theorem 5.3.9.

f(.")

m

Wm,2 m 3 1.

Wm-lr2

=

1,

WmJ

Lemma 5.3.8.

n

=

u.

Dju

0

Wm+1,2,

=

on Wms2 f ( u n ) v,

+

{u,}

Wm,2,

uo =

21.

Proof.

<M

I/ U, I ( , + ~

5.3.8. uo

g

=

f(u)

I1 Un+1

Wma2 by

u n U + u'

+v

-

IIm

5.3.7, WmJ

< /I

% l

l!m

(5.3.10)

R. Rellich, Nuchr. Akud. Wiss. Gottingen Math. Phys. KZ. 3&35 (1930).

5.

198

WEAK CONVERGENCE I N HILBERT SPACE

-

n. ll(f(Un,)

-f(U')llm

u -+ co

.m"+J

-f(Un,)

uny+l= f ( u , ) {un}

0.

Wm,2

un,+, -+ uo

f(un,+,)-+ f ( u o )

f,

-

+ u).

uo.

0,

(5.3.10) Wm,2. I

uo

EXERCISES

f

5.3.1. 5.3.4. 5.3.1. 5.3.2.

C,

k

=

5.3.3.

nk>lC,

1, 2,...,

C Ck+l

k,

(Jk>l

C,

on [a, b]

{un}

{un}

v+

co

u

{un}

5 . 4 . T H E PENALTY ARGUMENT I N HILBERT SPACE

199

5.3.4.

5.3.4

5.4.

T H E P E N A L T Y A R G U M E N T IN H I L B E R T SPACE

4

on H

gi = 0.

m

by {u 1 gj(u) = 0 {f,} by

B, j

=

1,..., m}.

H

(5.4.1)

{f,} bound.

on

by Theorem 5.4.2.

f, g2 B

on

H,

B B,.

f

{f,} on B

{u,}

uo {un}

*

Beltrami,

1. Comput. System Sci.

B f ( u ) + +GO /I u 11 + 00 B = H). fn(u,) -+ f(uO), on B, . uo;

1, 323-329 (1967).

u0

5.

200

IN

Proof. 5.2.8

by

{un)

(f,}

.f:

d = BonB

on B, n B =f n b )

d,

2f n ( 4 2f @ n )


< d. ---f

{un) f ( u n ) d. uo B, n B.

<

unv

-1

d

2

+a

11 u 11

{un} --j

cx),

bound {u,}

f

=f(uo)

f(%") >f(uO) - 1 v.

s;(%") > 95(.")

3 0. j ( u o ) < d,

B,

uo

f.

uo

u0

5.5.

20 1

AN EXISTENCE QUESTION I N OPTIMIZATION

uo,

{u,}

B,

u‘ =

d,

< d,, < d

d,

1.

n

fn(un) -+ y


n

3

co

I

y = d.

d,

f; n.

H

=

Wm,2(SZ), SZ

R”, Wm+lJ(Q),

{un} by

{u,}

on f,gg do

no

Wm+l,2

5.4.2

4.3

5.5.

AN EXISTENCE QUESTION I N OPTIMIZATION

by

FZ

ct al. by

H x H

H x H

202

5. H: by

u, u

{u, u>

{u, v}

+ {u’, v’} = {u + u’, v + v‘} a{u,u } = {mu,av}.

H x H on H x H

(0, 0). ({u, 4, {a’, INU,

.Ill

4) = (u, El’>

=

+ II

(11 u 112

by

+ (v,4 l12)1’2. H x H

H

xH A,B

H, ll{Un

9

A x B

H x H.

{un , u,} vn}

-

{u, u }

A x B,

{u, vll12 = I/ un - 7.4

ilf(u)ll < k 11 u 11

Proof.

k

-

vn

-

/I2

-

0

H

>0

u

HI. u,

-

is u,

f(u).

H

u

g by g(u) = g

( f ( u ) ,u ) . u,

+ /I

A x B,

Lemma 5.5.1.

-

112

g

u (f(Un),

v>

-

( f ( u >v, )

1 a

> 0,

by

J ( 4 = I1 ‘u -f(412 + all u 112

(5.5.2)

5.5. u,

v

203

A N EXISTENCE QUESTION IN OPTIMIZATION

L,(Q),

Q= on L,(Q).

TI,

TI)

u

(u),

J(u)

v; u

J? J(u) 3 a l l ~ 1 1 ~

H

U

u

J(u) = y >, all u \I2.

6

=

J(uo),

y

2 a I/ u

uo 6 =

B,

= {u

I II u 11

B,

L,(Q)

{ f ( u ) I u E B,} L,

L, x L,

B, {v - f ( u ) , mu}

<

u 3 {a - f ( u ) , au}

L, x L,

5.2.9,

EXERCISES

B,

5.5.1. 5.5.2. 5.5.3.

5.4.2

H x H

r = ( f ( u ) 1 11 u I/ < a>, J,

APPENDIX

COMPUTER PROGRAM FOR THE SOLUTION O F TWO-POINT BOUNDARY VALUE PROBLEMS

1.6

117, on

360,167 on

1.6,

1.6

21 0.01 204

205

COMPUTER PROGRAM

by x =

42)

(XI

)...)xa)T,

xZ(0) =

=

1.076,

=

0.5760,

0,

=

= 0.9976.

by xl(t)

=

-0.538t

+

+ x2(0), 0.498t + sC,(O),

x 2 ( t ) = 0.288t x3(t) =

x4(t)

k by k

xl(t)

x&t)

x3(t).

L,

by

1.6.

50 L.

by

206

APPENDIX

n

=

3 u =

(,!)

ZI =

of

t

tt

x3(0)

c3

by

L

L,

=

x = (xl, x 2 , x3)=]. tt x , ( t f ) = c1 x2(tr) = c 2 ,

207

COMPUTER PROGRAM

tf

E =

lo2

=

+2 * on linear

six

208

APPENDIX

n

4

+4

(H;,

+

f;'i

el =

(012

+

'i.

;w?

to

.

t

on

209

COMPUTER PROGRAM

a

by a's

2000 by

2000 Under no circumstances

210

APPENDIX

FJ 2

=

FJ

f ( x , t).

N

FJ

x

X I ( I ,I T ) ,

1

IT F(J)

F(J) XI(Z, ZT).

9 Y(I),

Y(N

+ 3)

Y(2N Y(3)

of

+ Y(N

+

by

20

no

( N < 20). by

8 6 by by

t, no

by

21 1

COMPUTER PROGRAM

by

200-500 on

=

2

= 200,

0.01.

Data to be Supplied D23.16). first

IK

of initial

IKK

last initial

IFK

first

N

IFKK

final last

N

MH

final

of

1.OD-15) 1.OD-09)

(KE

=

6

4)

Output.

(1)

D-23.16

212

;

(5)

XT

CT(I) -YT(I,

4

by

by

213

COMPUTER PROGRAM

OflURLE P R E C I S I C N C I ~ 2 O l ~ C T I Z O l ~ ~ C ~ 2 O ~ Z O l ~ X I l 2 O ~ 2 ~ 5 l ~ X T l Z O ~ 2 O 5 l ~ 1 V I l i O ~ 2 O l r k l 2 C ~ Z O l ~ E l Z O l r C I 4 o r C T d l 2 O l ~ A L P ~ l ~ 2 O l t A L P ~ 2 l 2 U l ~

ZVl2ZtliAI2Ct2Cl~Rl2O~~Ol~CVl4Ol~XPl2Ol

C C C C C C C C C

c C C

n o L R L E PRECISIC~ CIFF~CIF.CCAX.G~CFC,ENCPM DOUBLE P R E C I S I C h H M I ~ ~ H C A X ~ V A L U E I E P S ~ ~ E F S ~ ~ E P S ~ E P S I ~ H P DOUBLE F R E C I S I C h T I h * T I h h r T V L v S T E P COMMON X I t K E V CCCCCh h CCPMCh STEP EXTERNAL C E R I k O R B I T CHANGE PRCBLEC h E h T O N - R A P H S C L C E T H C C F C P S C L V I h G F I R S T C R C E R h C h L I N E A R 6. V. P. N I S THE h U C B E R OF V b R I A P L E S v I I T C I K K IS L C C l T I C N O F C O N C I T I O N S S P E C I F I E D A T 110, CH I S L U P B E R C F I h T E R V A L S . I F K T O I F K K IS L C C A T I O N OF C C N C I T I C h S S P E C I F I E C A T T - 1 . € P S I I S T C L E R A L C E Ch h C R l r S t E P S IS T C L E R A N C E F C R P A T R I X I N V E R S I O N E P S 1 IS T O L E R A N C E L S E C I h I N T E G R 4 f I C h S C H E C E A e I S KE IS THE C R C E R C F T H E P R E C I C T O P t h P C IS hUCBER C F T I M E S CORRECTOP IS LSEC I T E A C H S T E P READ(5rlOCl ~ ~ I K ~ I K K I I F K ~ I F K Y ~ C ~ ~ J I T 100 F O R M A T ( 1 5 1 H M = D F L O A T (PHI R E A O ~ 5 s l O l lE P S I E P S I * T b C 101 F O R M A T (023.161 READ ( 5 t l C C I K E i h R C READ I5v101l E P S l R E A C 1 5 r l C l l (CI( l t 1 = 1 t N l e I C T ( J I t I - l t h l EPSZ=EPSl K-0 MODE=C NV=N LOP.1 S T E P - l b L /HW W I N = E 7EP HWAX=SlEP VALLE=TVL NK-IFKK-IFKC1 MP=MH+l OK 10 J = l . N Y 10 A L P H 2 I J l = O . C C * C C

* * * * * * * * * * *

* * * * * * * * * * * *

INSERT I N I T I A L STAPTIhG E C L L T I C I HERE I N I T I A L S C L L T I O N IS L I h E A R F I T e E T U E E h e C U h C A R I E S O C 15 I = l r N 19 X I I I I 1 l = C I ( I l DO 4 0 J - 2 . C H X1141Jl=-C.53EC+OC X I 15s Jl.C.ZEEO+CO XI(ttJI=0.4SEe3C*CC 4C CChTIhCE 00 4 1 1 = 2 , C F TIh=DFLCATII I T hh= 1I h* 5 T E P- S T E P

x~ii1~i=~-a.~3ec+C~I*T1~~+~l~lrll X I ~ ~ . I I = O . ~ ~ ~ O + O O ~ T I ~ ~ + X I I ~ ~ ~ I XI13.1 l=C.45€€30*CC*11hh~X1~3.11

41 C O h T I N U E X I I 4 , r F 1-0. 53ec*cc XI ( ~ . ~ ~ i = c . z e e c + o c

~II~.cPI=c.~~~~~c+co

C C

I N I T I A L S T A R l l h C SCLLTICh EhCS kERE

* * * * * * * * * * * ICNT.1

**.*********

214

APPENDIX

115C CONTINLE I F I I K .GT.1 I J K = l I F I I K . E C . 1) J K - I K K L 1 DO 1 J = l t h K

O@

z

I-lrh

V I I I .J) =C. CCCC@

2 CONTINLE Y I I J Y * J I 21- C D t C C JK=JKE1 1 CONTIhUE UR I T E 1 6 . 2 1 1 I 2 1 1 F O R C A T I ' I L I T I A L HCMCG. S C L U T I C L S b R E ' I UR I T E 16,225I I l V 1 I 1 v J I v JZ1 shK) v 1.1 sh I 22C F U R C A T 1 / ~ 3 ~ 3 Z r D 2 3 . 1 b i l )

C C

c

**********.*

* * * * * * * * * * *

I N S E R T C A L L L I N I L C A R C A h C GC T C ZOO0 C A R C F E R E CALL L I h A L l C I ~ C T ~ V I r A L P F l ~ ~ F ~ E F S ~ E P S I ~ K E ~ N R C ~ E P S l ~ I K ~ I K K ~ I F l I F K K rHC,TUL ,ALPHZI GO 70 2 C O C

********.***

* * * * * * * * * * *

KEV=1

DC 3 J J - 5 C . P H r 5 0 DC 4 J S l t h K

00 C I = l . h L- I LZ 5 V I L I ~ V I I I ~ J I Yl1~=o.coocc VIII-STEP CALL A 8 I S I O E R l U ~ V r D V q IVALbEt K 1 OC 1 C L L * l . J J CALL 181 16 CONTINLE DO

OPq

H C IN.FMAX,

EPSlt EPS2t

t I=ltN

L=IEZ VC ( I I J I = V I L I t ElI~JI=VlLI 4 CONTIhLE C C C

* * * * * * * * * * *

**** I h S E R T C A L L CRTHO C A R E C A L L ORTHO l E * V r V l r k t h % ) * * * * * * * * * * *

* * * * * * * * * * * *

A F T E R T F I S CARC I F C E S I R E O

I*+***

* * * * * * * * * * * *

3 CONTINLE *RITE It*Zl!I 2 1 5 F P R C A T l / / l ' F I N A L I L T E G R A T I C N C F C R T F C h C R C L L I L E C FOMOG. SOLUTICNS 1S T A R T I N G h 1TH' 1) U R I TE l b r 2 2 5 ) I l V 1 11 r J t r J z l * N K I ~ 1 x r1h l UQITElt~216I i l t F O R C A T l / / l a HCCCG. S C L L T I C h S AT E h C C F I h T E C P I T I C N I P E ' I I U R I T € 16.225 I I I V C 1 I,J I J-1 .hK I , 1=1 . h l C B E C I N I h T E G R A T I O N OF P A R T I C L L A R S C L U T I C N V I lI=O.CO*CC vlZI=STEP KEV-2 DO 11 1 . l . h

,

L-Id2 11 V 1 L I . X I I 1 r l I CALL ABI S 1 0 E R I V i V r O V i h V . K E thRC rPCCE ~ L C P i ~ C l h i F C A X ~ E P S l . E P S 2 . \V&LCE v K I oc 1 E L L = l r C k C A L L Ah1

COMPUTER PROGRAM

ie

215

CONTINlE DC 1 2 I.1.N

C=f c z

12 X P I I I = V l l J *RITE l6r220) 2 2 J F O R C A T I I I t ' PARTICULAR S f L L T I C h AT EhC C f f h T E R V 0 L ' ) I URITtlC.iC6l (XP(IltI=lrhl KEV=i DC 1 3 J X l r h K JK=IF K 00 1 4 I = l , h K A I I , J ) =VC I J K , J ) 14 JK=JK&l 1 2 CCNTINLE W I T € lC.2IP) C A I R I X 7 C B E I h V E R l E C T C S C L V E F C R l L P H b S IS')) 218 F0RYAlI//l' h R I T E l t i 2 2 5 l (lAII.J)tJ~l,hKI.I~l~hK) JK=IFK DC 32 I x l t N I CTAll )=C7(JKl-XFlJI) J Y = J U 61 3 2 C O N T INCE OC 3C I = l . N K O C 30 J = 2 v h K 9 1 1 .l)=CTAIIl RIIrJI=C.COCCC 30 C C h T I L U E CALL C C R O U T ( A ~ R I C I E P S ~ ~ Y I ~ , ~ O . I ~ C ) 00 3 1 I = l . h K 4LFFl(I )=R(I 111 3 1 CChTlhUE icoo CChTINLE 00 1 5 I * l . h K 1 5 A L P H Z I 1I = A L P k Z ( I I + A C F H l I 1 I kAITE(C~Z10 2 1 4 FCRYPI111I' A L P H A 5 A R F * ) ) WRITE I C , Z o t I I A L P H Z I K ) . Y = 1 .hK I B E G I N I N T E G R A T I C N OF PAR?. P L C S ACPCA T I C E S WCCCC. S O L U T I O N S C 00 20 J x l t h C I A IJ)=C.OOtCC DC 2 1 I = l . h K C I A I J ) = A L P C Z I I * V I ( J * I l*C I A ( J I 21 C C h T I h L E ZC C C h l l h L E 00 22 J = I K , I K U 22 C I A I J I = C I I J I hRITElt.221) 2 7 1 fORCAlI//l' T C T A L f C L C 7 l t h A T S T A R T C F I h T E R V A L 1 5 ' 1 1 h R I T € I t * Z C C 1 IC A I I = I vh OC 3 1 I l l . & ?5 X T l l , l ) = C I A l I l DC 2 3 I f l r l v L.IC2 22 VIL)=CIAII) UFV-2 V I 1)=C.EO+CC vlZ)=slEP CALL A B I S I C E R I V ~ V I C V . ~ V ~ K E ~ ~ R C ~ C C C F ~ L C P ~ P ~ ~ L ~ ~ C A X ~ E P S ~ ~ E P LVALUE . K l 2 4 J=L.CP CALL A e I

n c 2: 1 i 1 . h t2 L

APPENDIX JP=J+1

XTIIrJPI=V(LI 2 = CflhTINLF 24 C Z h T f N C E hRITE I e t Z i ~ l

222 F ~ R P A I I / / l ' l C I A L S O L U T I C h A T E C C C F 222 C

27 2e

2E

25

1197 20c 20 1

iui 20 ? 20:

20t

37 1VALLE r K I CC ? @ LL=l.PH CALL b e 1 oc 2 5 I = l * h

L=I*2

LP=LL+l

xIII,LPl=v~Ll

IhT€RVbL

IS'II

217

COMPUTER PRO GRAM 39 C O N T I N C E

3P C C h l l k C E h R I TE 16.226 h C h L I N E b P S C L L l I C h d T E h C C F I h T E P V I L IS')) URITEltrZCC ( X I ( 1 .MP) . I = 1 , h i hRllE Its222 h R I l E l t r 2 C 4 ) ~Cl(l),I=lrNl

22t FORCAT(//(

@

204 F O R C A l l / / / ~ 3 > ~ 0 2 3 . 1 6 ~ ~ UR I T E I e ,2 z e I I C 1 ~ 2 2 8 F C R M A l ( / / l l H 7C'TA.L I l E R A l E S 1 1 5 1 GO 10 1 1 9 5

llSe CONTINLE hRIfE(6r227)

227 F O R k ' A T ( / / ? ? H 1155 CONTINCE END

IChT h C CCNVEREEhCE.

TCTAL

IlE

TES =

218

APPENDIX DC 5 K = l r K R

KI=KGl *(1l=1 D C 4 KU=l.K

DC L 1. l . h SCP=G.CO

KPCsIGhZ OC t K C = l r K KCCrKPCGh t 5LP=$CMECIKCI*V (KCPI 5 Y l I & Z l = V IIEZIGH*SLC I 111=v I l l C P CALL O E R I V O I '3C E 4 I . l . h hI.hZt1

lEMPl=VINII KMI=NI OC P UC=lrK UCI.YCI&h

TECP=VIUCII VIUCIl=TECFl e TECPL-lEPF1 - I E C P KCISKMICN 8 4 YIUCI)=lE*Fl 4 CO hlIhLE

COhSluc2l=C.OC 00 7 K K X l t K KUJ-K-UKC2 7 CChSlUKJl=CChSlKKJl-CCh5lKKJ-1) DC S I = l t N

urI=htcl DC S K U = l r K l U Q - K 1-KK C 1

KClrKClGh UCIh-KCI-h

SUM*O.CC DCS9 UJ-1.KC UClh=KCIhth S9 SLC=SUCCCChSlMJl*V

IKMINI

5 YlKCIl*ZLC

00 30 1 . l . h 3c YlIG2l=oIlIl DC 10 1 - l . h

hiI=h22GI N I =h2CI 10 V l N I l = l ( h 2 I l 3 CONlIhUE KRl-KR YIl)=T RETLRh ENTRV b e 1 IFlIV.EC.0) V ( 1 )-SAVE

GC TO 101

Vl2J=H DO 100 I m l t h NU R 3 I = N K R Z C h C I NUR4I=hKR3ISh NI-NZCI YfIC2l=VlhKR31) 100 V l N l l = V l h K R 4 I l I V-0 101 K R l l = K R T C l

COMPUTER PROGRAM 4 5 H=VI2I o r 32 1 l 1 . h SIJC=C.CC hKP21 =hKRZ&I hKR?I =hWQZlGh kKR4I=hKR?IGh V(hKRZII=V(ICZI KCC=h2 &I h l =KYC

V(hKP41 I=V(hIl

OC 3 1 K C = l 1 K R 1 1 KCC-KCPth

33 S L C = S L P & C ( K C I * V f K P C l 1FIhOF.EC.OIGC TO36 Ih.hCI OVI INI=CV I1I

O V I I I=CV(I v 4 I E i l = 0 1( I I GC T C I 2

lCH*SLc

3c v ( I & Z l = 1 l I C i l ~ H * S L C 32 V ( h K R 3 I I =V I 1 C Z I LCLR-C IF(hOPl?7.3€~37 38 V ( I I = V ( l l C V ( Z I GO TC ? F 37 O T = C l C V I i l vllI=cl 3 5 CALL C t R I \ I V I IF(hCR.LT.CIGC TO 5 0

KRlZ=KPl1&1 DC 2 4 1 i l . h SLM=C.CC KCC=EtZC I T€CF=V I K P C I OC 2 5 I ( C = l , K E l Z KCC=KCC&h TECPl=TECP-VlLCCl SLC=SLCECPIKCI*TECF ? t TEYP=TEPPl I F f h D P - E C - C l G C T C 4C

IH=hEI C V I IJ.CV 1 I h I EH*SLC v(IEzl=cv (I I cc T O 3 4 4C h K R Z 1 = h K Q Z C I V l l & Z l = V ( h K R 2 I l&H*SLC ?4 COhTlhCE

EPS'=EPSZ/(Z.CC+*WPTl OC 4 1 J = l . h P C hKLh=C NDL@=C 42 I - l r h KKR?I=hKR2EhtI TECP=V(hKP?I I TErPl=CAHS(V(I&2I-TEcF)

oc

IECP=CbeS(lECPI IF(TE~Pl.GT.EPSS*lECFlhCL@=hCUe~~ IFITE*Pl.Gl.EP~l*TECF)hWCh,hKChtl

42 CCh~IhLE I F ~ ~ N C ~ . E C . C I G Ci r 44 IF(J.hE.hPCI GC T C l C C l IF(CE1Hl 4 t 144146

219

220 IGU1

APPENDIX t P

4 i

1.l.h

NKR41 =hKPZGhChCI h l =hZ&I 47 Y(hKP411=VlhIl 4 4 CALL C F R I b I Y I

IF(NOCH.EC.C.~hC.rET~.~E.~I

I F I C K C ~ . E C . C . ~ R . J . E C . ~ R G Ic c DC 4 2 l = l t h 4 K R 5 l =hMRZCPf,l YIhLR?I l=YIIC2l hl=hZCI ‘YKP41 = h K Q Z E h C h & I IFlhOP.tC.Cl GC T C 4 5

GC

TC 4 8

50

D V ~ I l = D V l I l ~ H * C l K ~ T Z l * ~ V l k I l - V ~ h C P 4 I ~ l Y l 1L i l = D Y ( 1 ) GC ir 4 2 4 5 Y I I C 2 I = V I l & i l G H * C l K Q l 2 l * l \ ( h l l-V(hKR41 11 43 CrhlINLE 4 1 COklINLE 4 E I F l P A @ S l Y I 2 I l . L E . H ~ 4 ~ 2 lL D L P = 1 c.L DC t ! l = l , h

hI=hZtl

6.2 63

61 61

TEPFl=V(hll K“1 =NI OC 6 ; Kf”l.KPI1 KkI=KCIf,h TEPP=V(KCIl V l K ~ l I = l E P F l lF~Pl=TEMFl-lECF K P l = K P I Ch V~YCII=lECPl KRl=PlhClKRll~KRl IF1LCLt.hE.lIPFTURh vl2l=Vl2lG\l21 r=Y121 KN=NZ CChS(II‘1.CC CChS(21-c.DC PO t 5 K r l r K P l l KW=KPGh C C N S l K t 2 I =C.CC KC=KRIl-KEl KO=CIhCIK&lrCCl IFlMC.EC.11 GC TO C 1 DC C l K K = 2 r K C KKJ=KC-KKC2 CChSlKKJ~=2.CC*CONSiCKJ)-CC~SlK~J-~l CChSI11=2.DC*CCNSl11 D C t S 151.h KPI=KCCI KC1h=KCI KPI-KPlth

sw=c.co

be 65 65 4t

DC 6 e K J - l v K C K P I N = K P INGN SLC=SL~CCCh~l~Jl*VlK~lhl V ~ K P I I - ~ L P COhTlNLE RETURN I F l O d ~ ~ l V ~ 2 I l . L T . ~ ~ l G h C2 ) T C VI I l = v l l ~ - t l 2 I 1.1-V 1 2 I V(

i I = v I 2 I /2.

5C

COMPUTER PROGRAM

oc

54 53

52 60

52 1-1.n IFfN0P-EC.OI GC T r 5 4 NI=N&I DV(IJ=OV(hlJ GO TO 5 3 NYRZI=hKIZ&I V ( I & 2 J*V( hKP21J NKR41=hKP2CZ*NCl Ih=h2&1 h21=lhth V(INl=V(NWR4IJ Y(N2IJ=V(IhJ CONTINLE GC T C 45 IF(KRT.EC.CI JH=l DC 3 2 C JJSIrJW oc 7 2 i = l . h KJ=O KC*ICh22 DO 7 4 U = l . K R l l

suC=a.oc KCPIKC KC=KCth DO It J = K , K R l l KCC=KCY&h KJ=KJCl 7 t SUC=SLCtHC(KJJ*V(KCCJ V t K W J =Slit4 K J = K J t 5 - K P 11 7 4 COhlINCiE ?2 COhlINCE 1 t C CONIINLE GO TO 45 ENlUV herb IV.1 111 K = K V DO 90 i = i . h NI=N2CI NKR3I=hlRZthCI NKR4I=NKR3ICh V ( h K R ? I J=V ( I C2) CC V ( N K R 4 I ) = V ( h I J V M A X = C C A X l ( O A B S ( V A L L E l ,l.CO) S A V E = V ( 1) H=TlZJ IF(K.hE.CJ GO TO $1 VIIJ=VALbE V(ZI=bALCE-5LbE cc TC 52 91 Kh=K&h2 V(ZI=(V~LLE-V(*CZJJ/V(Kh) VIlJ=SAbEEVIZJ 5 2 00 5 3 I.1.N h1 = N Z t I NKR?I=hKR2IChCI NKR4I=hKR31Ch 53 V ( I C 2 J = V ( h K R 3 1 J C V ( Z J * V ( h K R 4 1 ) IF(hCR.LT.OJ PElLRh

Do 54 h c = i , h a c CALL C E R I V ( V J

DC 5 5 I = l . h Nl=NZCI hKR2I=hKRZtl

22 1

222

APPENDIX h N P ? I - h K P Z I Ch hKF4lrhKR3IEh

V(hKRiIl=Y(IG2l 95

VlltZl=VlhKR?llL~l2l~~VlhIlCVlh~F4lll/2. IFlK.EC.CI G C TC F t I F I C A ~ S l V A L L E - V l K ~ 2 1 l . ~ ~ . € F ~ l * VC ~C 4 T~ Cl 5 7

S t DT SP 1;l.h hKR31 = h M R 2 t h 6 1 I F l C A B S ~ Y l h M R ~ I l - V I l t Z I ) . C T . E P S l * V ~ ~ XCl t

T C 97

5 f COhTlhlE RETCRh

57 IFIK.EC.01 G C TC S 4 hL=hZCK NKC?K=hKRZth&K NKR4K=hKRlKEh Y l i l = 2 . D O + l V A L L E ~ V l ~ ~ ~ 3 K l l / l V l h ~ ~ ~ V l h K R 4 K ~ ~ $4 CCNTIkLE RETLRN ENlRV b e 1 0 I F l O 4 ~ S I V l Z l I . L l . ~ C I P ) GC T C 1 1 C JH.1 1 1 3 I F l C A B S f V l ~ ~ ~ . L T ~ C l @ S ~ ~ / Z . CCOC I )T C 1 1 2 Vl2l=H/2. G C TC: C C 112 JH=JH&l H=C/Z.

GO TO 1 1 3 1 1 C KV=C VALUE=VlllCVIEl GO T O 111

ENC SLBRCLlIhE C C R C L T I L . R ~ C I E P S ~ ~ ~ K C P T ~ ~ C ~ ~ ~ ~ N C I

D C U B L E P R E C i SlOh A W RrC .SI P .CT S E F S DICEhSICh A I L C l r r h O l ~ I r R I h C l ~ r 1 )

C=nCPT IND-C

IF ( P I 1 C t i f . 2 5 10 M = N DC 2C I z 1 . h DC 1 5 J x l r h 15 R I l . J I = C . 20 R l l r l l = l . 2 5 1c.c

II=C l - C A R 5 1 A 11.1 I I 3 5 I.2.h

nc

lFll-CbBSIAll~l~~~30r35r35~35 3c 11.1

T=CABSIAIIllll 3. C C h T I h L E IF(IIl4C~65r4C 4c

lC=lC&l If1 ~ 1 4 5 ~ ~ 5 . 4 5 4 5 oc $0 J = ~ . c SIR( 1, J l R (1 JI qR ( I I v J I 5C R I I I . J I - S C f DC to J - l r h S=AtI.J) A l I ~ J l ~ d l l I i J l CC A l l l . J l = S

COMPUTER PROGRAM

IF l I I - K ~ l l C ~ 1 3 ~ r l l C 1 1 L IC=lCt1

IF ( C I11 t 9 12 5 ,115

223

APPENDIX

C

c c

c

C C

c

225

COMPUTER PROGRAM C C

ENC R I C H 1 HAht

SIDES

* * * * * * * * * * * nc

ic

* * * * * * * * * * * *

i=i.h

L=I*NP B P I L I =- B P I L 1 * f I I 1 10 C O k T I N l i E O C E JX1.N L=J+NP VILI=V(L1*RFILJ

e CO~TIRLE QETURh 5 CChllhlE

C C

* * * * * * * * * + *

* * * * * * + ** + *

I N S E R T O R I G I L A L h C h L l h E A R PRCPLEC F E P E

RP=OS4RT(Vll)**Z+VI4)**2*VI5)**21

C C

v(s)=vI6) V I 1c 8.V I 7 1 Y I111 = * I € 1 VIl21=-VI3)/PP V I 13 1 =-v 1 4 ) I P P VI14I-VlL~lPP EhO k C h L l h E A R P R C E L E C

********.**

* * * * * * * + + + * *

RElCRL EN0

C C

S L B R O L l l h E CRTHC ( E r V r V l r h t h R ) OOLBLE P R E C I S l C h E l 2 C ~ 2 O 1 v ~ ~ 2 0 v 2 0 1 v V I ~ Z O ~ Z O 1 ~ ~ E T A I 2 O v Z O ~ r V V I 2 O 1 OOLBLE P R E C I S I C h 21V.VTE S L B P C L l l h E 1 C C R T ~ C h C P t A L f Z ECCLUChS CF E lhTC V v PERFCRW SAME O P E P A T I C h L C h C C L L V N S CF V l DC 1 i = i v n V l I , l ) = E ( I * l1 1 CDhTlhUE DC 4 J * 2 t h K JL*J-1 DO L I = l r J L SAt=O.CDSCC DC 7 L.1.n IF10ABSIbILII~1.LT.1.00-35~ GC T C 7 SPV=V(LvI)*VIL,l1CSAV 7 C0)uTINbE VV(1 )=SAM VTE=~.COC~O OC € I l = l . h I F I C A B S l V I l I ~ I ~ 1 ~ L f ~ l ~ O O ~ 3 5 ~ C R ~ C ~ ~ S I E l I l ~ CC J 1 1T O~ L8 T ~ l ~ O D ~ 3 5 ~ V T E * V I I I v I 1*E ( I I v J ) t \ T E e CONlINlE GC T C 5 IFIYV(I).LT.1.00-35~ BETAIJvl)~blE/VbII) F CcnTIhuE 00 S LJ*l.h V I L J v J ) =E (LJ. J 1 9 COlYTlNLE DO 1 C JJ=l.JL OC 11 IJ.1.N

VlIJvJI=VIIJsJ)-BElAIJ~JJ~*~IIJIJJ) VIlIJvJ~~VI~IJvJ1~BElAlJ~JJ1*VIllJvJJ~ 11 C O N T I N C E l a CONTIN~,E

226

APPENDIX 4 CONTINLE DO 1 4 J x l r N K

SAV-0.

c c caa

DC 1 2

Krlth

GC T C !2 SA\~V(YIJI+V~~~JIESAV 1 2 COhTlNLE VV t J J - 0 S C R 1( 5 1 \ I OC 1 3 Y.1.h IF(VV(Jl.Ll.1.CD-351 GC T O 1 3 V ( K v J l = V ( R e J 1 / V V I Jl YI(K,JlrVIIY.JI/VV(JI 1 3 CONTINLE 1 4 CONTINUE RETURN EN0

IF~OAES(\(K~JI1.LT~l.OO-351

SLBRCUTINE L I h L L ~ C I ~ C T ~ V l ~ I L P h 1 ~ C ~ ~ E P S ~ E P S I ~ U E ~ N R C t E P S l t l K ~ I K K

1I F K ~ I F K K ~ H C ~ f V L i I L P H 2 1 COLELE PREC IS I O N C I (201 . C T ( 2 0 1 r X I ( Z 0 * 2 0 5 l 1 0 L F t ' Z ( Z O ) *rC(i?O,ZO)r l Y I ~ Z 0 ~ 2 0 1 ~ C I I l 2. C0T)1 ( 2 0 ) r A L P h l l 2 0 ) . V V 1 ( 2 0 . 2 0 ) r 2 V I 2 2 5 1 ~ A ( 2 C ~ Z G I ~ R ( 2 0 . 2 ,CCIV ( + O ) r X P ( 2 0 1 COUBLE P R E C I S I O N D I F v E L C R C * E P S C . DOUBLE P R E C I S I C h C C l h i h M L l r V L L U E ~ E F S 1 r E P S Z t E P S r E P S f t H * DOUBLE P R E C l S I C h l \ L , S T E P COMMON P I t K E V CCCMCN N CCCCCh STEP EXlERhLL DERIb 205 F O R C A l ( / ( 3 ( 3 t ~ 0 2 3 . 1 6 l 1 1 206 F O R M I l ( / ( 3 ~ * C 2 3 . 1 6 1 I EPSZ=EPSl

u=c

CCOE=C h\=h LOF-1

EPSL=l.OO+CZ*EPSI M P =WH+ 1 hK-IFKK-IFKC1 DC 5 1 J = l r h K DO 5 1 I a 1 . h 51 Y V I ( I ~ J 1 ~ Y I ( I r J I OC 5 2 I v l t N 5 2 ALPH2(11=0.0C+GO ICNT=l HMIN-SIEP

HMAX=STEP VA L L E 11 b L 1 1 5 0 CONTINLE KEY=l oc 7 J l l . h K

oc e

I=~.N

L.162

e

Y(Ll-VVIllrJl

cokTINiE V~11=c.aocoo Y t 2) =STEP CALL I B I S ~ C L R I V ~ Y ~ C V ~ h V ~ I E ~ h R C ~ C C C E ~ L O P ~ H ~ I N ~ ~ C A X ~ E P S l ~ E P S lVALUE 9 Y 1 OC 17 L L I l r C H CLLL 101

227

COMPUTER PROGRAM

C

17 CDHTINLE DC 9 1-l.h L.162 9 VClI*J)=VlL) ? C0)uTINLE IFlICKT.GT.1) GC T C 1 1 5 1 B E G I N I h l E t R l T I C N OF P A R T I C C L A R S C L U T I O N VIlJ=C.COtCC v I 2 ) =STEP KEVS2 co 11 1 l l . h L-162 11 v l L I = x I l I . 1 ~ CALL ARIS f O E R I V . V t C t . h V ~ Y E i h R C ~ ~ C C E ~ L ~ P ~ ~ C I h ~ ~ ~ ~ X ~ E F S l ~ € P S 2 ~ lVALCEtKl D C 1 E LL=l.CC. CALL A81

1E CChTINlE OG 1 2 1.l.h L.162 12 XPII )=VILI 1191 C O h T I h L E KEV=2 00 1 3 J = l t h l c JK=IFK Or 1 4 I * l . h K AII*JI*VC(JK.J) 1 4 JN=J*&l I? CGhTIhLE JK=IFK DC 4 2 I=l.hK CTAII I=Cl(JY)-I JK) J K = J K Ll ?i CONTINLE DO 20 I * l * h C OC 30 J = Z t h K R I I , 1) = C l b I I I Q I I.JJ=C.CCECC 20 C C h T I h L E CALL CCROLl I L tRtC .EPZthK*l tZOvIhC1 DC 3 1 I = l . N K ALPPlfIJ=RfItl) 31 CflNllNLE C B E G I N I h T E G R b T I C N OF P A R T . P L L S A L P k d T I C E S F C C C C . OC 20 J w 1 . h CIAlJI=C.COECC no 2 1 l = 1 * h * C I A I J l X A L P C . 1 I ) * V V I I J. I I + C I A I J 1 21 CONTINLE 2C CGhTINCE

SOLUTIONS

DC 3 c I - l r h

c I b l I ~ = c I b l I ~ t ~ I l I , l ~ 36 C C N T I N L E 00 22 J = I K , I K K 22 C I A I J ) = C I I J ) kR I T € f C v 2 2 1 8 2 2 1 F O R Y A T I / / I ' l C l A L SC L U T I C h 4 7 S T A P T C F I h T E R V d L I S ' ) I URlTE I t * Z C 6 1 ( C I A I )*I=l,hi flC 2 3 1.l.h L.162

i l YIL)=cIbIII V I ll=o.CDtcC Vli)='clFP

228

APPENDIX CALL APIS l C E S I V ~ V ~ C ~ r h V ~ K E ~ h R C t r C C E t L D P t H r l h . r r 4 X ~ E P S l ~ E P S Z ~ lVALUEtU I

OC

24 J X l , C k

CALL A e I 2 4 CChllhLE oc 2 5 1 i l . h

L=IL2 2' C

2E

24 ??

1157

1

C

e

x P l I l = ~ I L l CflhTIhLE l C h l =I C N ? G 1 IFIIChl.G?.tI GO T O 1 1 5 7 C C W P L T E AhC CWECU h C E C S JU=IFK EhCFM=C. C C t C C 120 2 0 I = l r h l c CIF=OABSlCll JK I-LP (JUI I IF(CIF.CE.EhCFC1 ENOFW=CIF JK=J K t l CONTIhlE IF(ENCQC.LE.EPSL1 CC T C 1 1 9 7 00 33 J = l , h U cc 3 4 I ' l r h YVlllrJI~ALFHlIJI*VVI(lrJI CChTIhlE CCh11ht.E CC TC l l S C CCNlIhLE IF(IK.Gl.11 JK=1 IF(IK.EC.11 JM=IKUE1 DC 1 J = l r h K 4 L P ) I l l J I = C I A 1J K I JU=JU&l COhTIhCE RElURh ENC I N S E R T ? t o C E L I C I T E R CARC F C L L C b E D @ V S V S I h CARC FERE

1 3

1 3

2cc 5C 1 .c 1.0

i.C

c

C-15 0-C9

otoo

4

1.C 1.C7t 0.c 0. c

c.c

0.c 0.c

0.c C.576

C.SS?ttl

c.c c.c

C-07

otoo otoo DtOO

otoo otoo otoo otoo

o+co o*oo

ctoo otoo

0.c NUPBER CF SCLRCE

SlAlECEhTS

240K i J Y l E 5 F C R E a E C L T I G h

962

&

Books Al. 6

7 on

B1. by on

270

book

1.

J. up

1963.

22 9

230 Papers* 1.

2. 3. 4.

5. 6.

7. 8.

9. 10.

11.

0. (1954). Reend. Circ. Mat. PaZermo (Ser. 11) 3, 363-397. E. (1969). A J . Math. Anal. Appl. 26, 297-306. F. 165). PYOC.Nut. Acad. Sci. U.S.A. 53, 1272-1276. F. E., (1966). by Bull. Amer. Math. SOC.72, 571-575. (1966). S I A M Rev. 8, 309-321. (1943). Bull. Amer. Math. SOC. 49, 1-23. A,, (1963). A on SIAM J. 1 1 , 442-447. (1968). Comput. J. 10, 392-399. (1963). A Comput. J. 6, 163-168. (1968). A Ind. Eng. Chem. Fund. 7, 142-151. P. (1966). An

J . Astronaut. Sci. 13, 166-169. 12. 13.

0. (1957).

111, Operations Res. 5, 613-626. by A I A A J. 2, 1761-1766.

(1964).

* book.

on

12, 23, 24, 107, 183,229 170

50, 125, 153, 230 229 16

154, 157 158

173

169, 199, 230 36,40, 60,62,63, 123, 126, 159, 178, 201, 229, 230 107 157 158 193, 230 170 82 105 44 60, 230 67, 80,

65 230

141, 154, 155, 230 44, 105, 107, 229 230

107, 159, 229 47, 48, 229 153 64 230

229, 230

169

J., 104

62 162

125, 153

158 142, 145, 154, 156, 157,

168, 230 C . A., 50, 230 107, 109

168, 230 56, 61, 169, 230 229

109 106, 152 156

O., 161, 178, 230

105

92 23 1

232

AUTHOR INDEX

170

153 191 197 164 B., 142

36 L., 141, 154, 230 62 62 S., 167

134 E . C . , 158 G. 111, 157 153 A., 169

106, 152 56, 61, 169, 230 A. 82 S.,188 170 156 S., 170 E. 49 Munick, 169 G. 153, 154, 156, 157

53 A., 92 156

171 171 153, 157 50 107, 193, 230 169 125, 153, 157, 230

A,, 170 50,230 230

146, 148, 151, 229 157 C., 107,

SUBJECT INDEX see

104 see

see

7, 56, 63 43, 63 66 3 84

80 see

by 5 , 8, 10, 15

11

5 138 177 33

131, 153

17, 19, 174

122

139

124 29 24

by 141, on

En,

139,

151

40

18, 119

see

5 , 11

131, 150, 154

24,

on

09 233

234

SUBJECT INDEX

59 3

30, 53 12 193 133, 144 by 18 61

81, 84, 11 92,

13, 116,

107

23, 65, 140

see

58 46 see

107 175, 198, 201 184

on 136 76, 190, 165 163 see

I11 134, 145

08

68 44

184, 186 35

175,

SUBJECT INDEX

235

Strong (Frkchet) differentiability, defined, 15-16 Surjective maps, 3

see Conjugate direction algorithms Volterra integral equation, solution of, 9-10

Tangent hyperplane, defined, 21 Taylor’s theorem, 24

Weak compactness, 187-189 Weak (Gateaux) differentiability, 19-20, 173

Variable

metric

gradient methods,