Optimal Design of Control Systems: Stochastic and Deterministic Problems (Pure and Applied Mathematics: A Series of Monographs and Textbooks 221)

= Q(d/dy)
!?- =i-c(z)

dz

(2.2.57)

(here the time-invariant loss function /s = fs(z) is determined by analogy with (2.2.9) and 7 is the stationary error.) The optimal control (2.2.58) 3 differs from (2.2.20) only by the^ position puoiuiuii of ui the oiic switch owiu^ii point puniu z 7 , which can

be obtained from the condition dz

v r/

.

To complete the solution of the synthesis problem, we need to calculate z3. By analogy with Section 2.2.1, the switch point z3 divides the z-axis into two parts: the domain z > z3, where df$/dz > 0, it* = um, and

Eq. (2.2.57) has the form

and the domain z < z 3 , where df^/dz the form

w

/ A \

< 0, u* = um, and Eq. (2.2.57) has

AtF + um -^- = 7 - c ( z ) .

(2.2.60)

At the switch point z3 the derivatives of the functions / + (z) and f ~ ( z ) vanish:

120

Chapter II

To solve the linear equations (2.2.59) and (2.2.60) explicitly, we need to specify the form of the linear operator L(d/dz). Assume that the density of the transition probability Tr(y,y') at the jump moment is given by the formula f

1I exp(-fc / L 2 |y I / -2/1)n ffor //
^(s) = 71 IT-•——— fc2 —— + Afc s - s22 '

(2.2.63)

A* = fc2

After the change s —> d/dz, we obtain the following expression for the operator L(d/dz) from (2.2.63) and (2.2.54):

(2.2.64) With regard to (2.2.64), we can write Eqs. (2.2.59) and (2.2.60) in the form x(_d__

dz

*,\df£ I dz

/,,

.. d

Introducing the functions

we transform the system (2.2.65) as follows:

=

um ) dz

,

where •U, 7rn/ 1[_

(U,^. 1 "7

(17 U,/(,

JI

Relations (2.2.61) and (2.2.66) imply the following matching conditions for the functions
(k2um + \&k)p+(z3r) = (-k22um + \&k)
(2.2.69)

Exact Methods for Synthesis Problems

121

The characteristic equations corresponding to Eqs. (2.2.67) are

umj

\

um J

=0.

(2.2.70)

By fj,^ and ^ we denote the roots of the characteristic equation for the function (p+ (z) (correspondingly, by /j^ and /j,^ the roots of the characteristic equation for (f>~(z)). A straightforward verification shows that if

A

(2.2.71)

then (1) all roots /uf )2 are real,

(2) each characteristic equation in (2.2.70) has roots of opposite signs (for definiteness, in what follows, we assume that ^ and /j,^ are positive, and respectively, fj,^ < 0). REMARK. Note that condition (2.2.71) must be satisfied, since this is the existence condition for the stationary tracking considered here. Indeed, the expression on the left-hand side in (2.2.71) is equal to the absolute value of the mean rate of the regular displacement in the command signal y(t) caused by random jumps. Obviously, this rate of change cannot exceed the limit admissible speed of the servomotor. Inequality (2.2.71) just confirms

this fact.

D

Taking into account these properties of the roots of the characteristic equations (2.2.70), one can readily see that the solutions of Eqs. (2.2.67) have the form ±

(z)=

X

f r

"?('-*') ± ( '}

'

~ (/if - / 4 ) L / f o o 6 fZ

±, ^ a. 1 ei*2(*-*)c±(z')dz'\.

(2.2.72)

Using (2.2.72), from the matching conditions (2.2.69), we obtain the following equation for the required switch point z3:

(k2um + AAfc) I" r° elt+yc+(z3_ ZT 04 ~ / 4 ) [J-oo6

}

y

, y

f°° rf9+(3_ ZT Jo

] \

e^yc~(z3r~y)dy\.

(2.2.73)

122

Chapter II

For the quadratic penalty function c(z) = z 2 , Eq. (2.2.73) has an exact solution. Actually, taking into account (2.2.68), we can rewrite Eq. (2.2.73) in the form

t-y-

Oh - A*2~)

f°° M~»/o

L./-OO

>

e"*»c(y) dy + / Jo

1 J

e"^c(t/) dy] = 0, J

(2.2.74)

where

c(y) = co + cl2/ + c 2 y 2 ,

CQ = fc 2 (z 3 ) 2 + 2Afcz 3 - 2,

{ 2i.jL. i 0 1

Calculating the integrals in (2.2.74), we obtain

(-k2um+XAk)\

/I

1N

/

c

2

T —:—T———XT— Ko ~

/ / ' I

— ^2 J

L

v \

^i

1

~

2

i *J '

1

4

//~/

,

-

\ ( III ' 1 "

A*2

1/^

Since /Lt^"2 and /z^ 2 satisfy the quadratic equations (2.2.70), after easily transformation, we obtain the following explicit formula for the switch point: /o n

v • •«

Using (2.2.69), (2.2.72), and (2.2.77), we can readily calculate the stationary specific error 32

r

mAk

+ X)

2Afc

3

2

~ ~ ~ ~

.

(2.2.78)

Exact Methods for Synthesis Problems

123

If instead of condition (2.2.71), we have a stronger inequality

A

F~~ I

then we can substantially simplify (2.2.77) and (2.2.78) by expanding these expressions in power series in the small parameter e = XAk/umk2 and taking only the leading terms of these expansions. In this case, instead of (2.2.77) and (2.2.78), we have Afc/2 jfe2

1

'

7 - K3)2-

(

J

(2-2.80)

For the first time, formulas (2.2.79) and (2.2.80) were derived by somewhat different methods in [176]. §2.3.

Optimal control of the population size

Numerous investigations deal with the dynamics of animal and microorganism populations and control of the population size. For example, an

extensive literature on this subject can be found in [51, 73, 87, 89, 133, 142, 186, 187, 189]. Various mathematical evolutional models depending on the environmental conditions of biological populations are used for describing the variations in the population size. We begin with a brief review of such models. The main attention is given to the models that we shall consider in this book later. 2.3.1. Models describing the population dynamics. Apparently, Malthus was the first who considered the following model for the population

dynamics in 1798: J = az.

(2.3.1)

Here x = x ( t ) is the population size11 at time t, and the constant number a, called the growth factor, is determined as the difference between the birthrate and the death-rate factors. If the birth-rate is larger than the death-rate (a > 0), then, according to the Malthus model (2.3.1), the population size must grow infinitely. llr

The variable x is assumed to be continuous, although the number of individuals in

the population can be only integer. However, if the number of individuals is sufficiently large, then the continuous model (2.3.1) can be used. In this case, the variable x is treated as the population density, that is, as the number of individuals per unit area (or volume) of the population habitat.

124

Chapter II

Usually, this result is not confirmed, which shows that the model (2.3.1) is not perfect. Nevertheless, the basic idea of this model, namely, the assumption that the rate of the population variation is proportional to the current population size proved to be very fruitful. Many more realistic

models were constructed on this basis by introducing the corresponding corrections to the growth factor a. So, for example, if we assume that in (2.3.1) the growth factor a depends

on the population size x as = a(x) = -fj,

ln(x/K)

/

x\

or a = a(x) = r (I - — J ,

then we obtain the Gompertz model (1825)

dx

* - '^ or the Verhulst model (1838)

Equation (2.3.3) is often called the logistic equation. The positive constants r and K are usually called the natural growth factor and the capacity of the medium. Models for more complicated systems of interacting populations are also based on the Malthus model (2.3.1). Assume that in the same habitat there are two different populations of sizes xi and x%, respectively. Let each of these populations be described by the Malthus type equations

at

,

at

= bx2.

, . (2.3.4)

Now we assume that individuals in the second population (predators) can exist only if they eat individuals (prey) from the first population.12 In this case, it is natural to assume that the growth factors a and b in (2.3.4) have the form

12

This model is usually illustrated by the assumption that x\ denotes a community

of hares bares and X? a community of wolves. Hares need vegetable food, and wolves feed on hares •es (and (and only onlv on hares). hares)

Exact Methods for Synthesis Problems

125

Thus, we arrive at the two equations

dxi

—-— = (ai — 02^2)*!,

at

dx2

—;— = ( — t>i + ozxzjx^

at

,

,

(2.3.5)

which are well-known Lotka-Volterra equations that model the behavior of the simplest system of interacting populations, the "predator-prey" model. These equations were studied in detail by V. Volterra [187], who found many remarkable properties of their solutions.

The multidimensional generalization of the Lotka-Volterra model has the form

•fcr

/.

^. .. V

r=i)2,...,n.

(2.3.6)

The dynamics of system (2.3.6) depends on the form of the matrix A = \\aij\\™. If this ma trix is antisymmetric, i.e., if a,-j = — oj,- and an = 0, then Eq. (2.3.6) describes a conservative model for the population interaction. If the quadratic form arsxrxs is positive definite, then model (2.3.6) is called dissipative. Further generalizations of models for the population dynamics were related to more detailed descriptions of the interaction between individuals in the population. For example, in many actual situations, the growth factor a depends on the population size at some preceding moment of time rather than on the current population size. In these cases, it is expedient to use the Hutchinson model (1948) M\

h > 0,

(2.3.7)

which is a generalization of the logistic model (2.3.3). In 1976 Gushing proposed the following more general model, in which both discrete and distributed delays are taken into account: f

x(t)=rx(t)\l-

I

f00

Jo

1

x(t-s)dK(s)\,

1

t > 0,

(2.3.8)

where K(s) is a nondecreasing bounded function and the integral on the right-hand side is of the Stieltjes type. In some special cases, it is necessary to take into account the spatial distribution of the population size. In these cases, the state of the population is described by the density function D(t,x,y] at the point (x,y). If the movement of individuals within the habitat is a diffusion process, then instead of (2.3.7), we have the Hutchinson model with diffusion —— (ft x y } =

dt

' '>

126

Chapter II

Equations (2.3.1)-(2.3.9) model the behavior of isolated biological communities (autonomous models). If there are external actions on the population, then some additional terms responsible for these actions appear on the right-hand sides of Eqs. (2.3.1)-(2.3.9) As usual, we distinguish two

types of external actions: purposeful controlled actions, which can be used to control the population size, and noncontrolled random perturbations. Let us consider a population described by the model (2.3.3). If there are external actions, say, some individuals are taken away from the population, then we obtain the controlled logistic model x = r

/ x\ (l-K — \x-qux, V /

(2.3.10)

where the function u = u(t) > 0 is the intensity of the catching process and

the number q > 0 is the catchability coefficient. In this case, the value

ft,

Q=q

u(t)x(t)dt

(2.3.11)

•>«!

gives the number of individuals caught during the time interval [ti,^]-13 In a similar way, the Lotka-Volterra equations can be generalized to the following controlled system:

xi — (ai - a2x2)xi - qiuixi, •

i

X2 = ( — Oi +

i

\

(Z.O.J-.ZJ

Here the control functions «i = ui(t) > 0 and u^ = u-2(t) > 0 are, respectively, the intensities of catching prey and predators. If individuals are removed only from one population, then we have u\(t) = 0 or U2(t) = 0 in

(2.3.12). If the population behavior is substantially influenced by noncontrolled

random perturbations, then the dynamics of the population is described by stochastic differential equations. For example, in some problems, the population behavior can be satisfactory described by the stochastic logistic model

x = r 1- K- x-qux + Wx£(t), (2.3.13) V J where £ ( t ) is the scalar Gaussian white noise (1.1.31) and the number B > 0 determines the intensity of random perturbations. Many other stochastic models used to describe the dynamics of various biological systems can be found in [51, 83]. 13

Note that Eq. (2.3.10) can be used not only in the case of "mechanical" removal

(catching, shooting, etc.) but also in the case where the population size is controlled by treating the habitat with chemical agents.

Exact Methods for Synthesis Problems

127

2.3.2. Statement of the problem. In this section we consider the optimal control of the size of a population described by the controlled logistic model (2.3.10). The statement of this problem is borrowed from the books [35, 68], where this problem is formulated in conformity to the problem of fisheries management.

We shall assume that the state x = x(t) of the controlled system (2.3.10) characterizes the general quantity (or the mean density) of fish at time t in some chosen habitat. We also assume that the intensity of fishing is bounded by a given value um > 0. In this case, the mathematical model for the dynamics of fish population has the form . (, x\ as = r I 1 — — I a? — qux, \ K)

0 < u(t)
t> 0,

(2.3.14)

x(Q) = ZQ > 0.

By p > 0 we denote the price of the unit mass of caught fish and by c > 0 the price of unit "efforts" u for fishing. Then it is natural to estimate the "quality" of control by the functional fT

I(u)= I (pqx(t)-c)u(t)dt, Jo

(2.3.15)

which, with regard to (2.3.11), gives the profit provided by the fishing process defined by the control function u(i): 0 < t < T. The problem is to find the optimal control u*(t): 0 < t < T for which the functional (2.3.15) attains its maximum. Following [35, 68], instead of (2.3.15), we shall estimate the quality of control (i.e., of fishing) by the functional in which the terminal time T —t oo. In this case, an additional "killing" factor appears in the integrand to ensure the convergence, namely, ,.00

I(u) = I Jo

st

e-

(pqx(t)-c)u(t)dt,

(2.3.16)

where S > 0 is a given positive number. As a result, we arrive at the following problem: for an arbitrary initial state x(Q) = x$ of the controlled system (2.3.14), to find a control function 0 < Ut.(t) < um: t > 0 (or

0 < w* ( x ( t ) } < um: t > 0) for which the functional (2.3.16) attains its maximum on the trajectories of system (2.3.14). REMARK. If the initial population size XQ does not exceed the capacity K of the medium, then it follows from Eq. (2.3.14) that for any time moment

128

Chapter II

t > 0 and any admissible control u(t), the population size has the same property, x(t) < K. Therefore, this problem is well posed if the parameters p, q, and c in the functional (2.3.16) satisfy the condition

— = Xl
(2.3.17)

Otherwise, (x± > K), this problem has only a trivial solution u*(t) = 0, t > O.14 Therefore, in what follows, we assume that the inequality (2.3.17) is satisfied. We also assume that qum > r.

D

2.3.3. The solution of problem (2.3.14), (2.3.16). If we define the function F(x) of the maximum future profit by the relation

U

oo

-|

e~st (pqx(t) - c)u(t) dt z(0) = x , j

(2.3.18)

then, using the standard procedure described in §1.3, we obtain the Bellman equation

max (J^U^Ufn

< I j

ra!

(l ~ IT) ~~ qux\ —r— — SF + (pqx — c)u > = 0 ^

-**- '

I CtiC

(2.3.19)

J

corresponding to problem (2.3.14), (2.3.16). It follows from Eq. (2.3.19) that, depending on the current state (the

population size) x of the system (2.3.14), to perform the optimal control we need to choose u*(x) = 0 for all points x £ R1 C R+ at which the function

p(x) =pqx-c-qx—— ax

(2.3.20)

is negative. Conversely, at all points x G R2 C R+ where 0, we

need to take the maximum admissible control ut(x) = um. If tp(xt) = 0 at a point cc* (in view of continuity, the point x* separating R1 from R2 is the limit point of these domains), then the optimal control u f ( x * ) for Eq. (2.3.19) is formally undetermined. However, one can see that the choice

of any admissible control 0 < u < um at the point x* does not affect the solution of Eq. (2.3.19).

Now let us consider how the population size in system (2.3.14) varies with time. Let x(0) = XQ < xi. Obviously, in this case, there exists an

initial half-interval [0,t*) at all points of which we must set ut(t) = 0. This statement immediately follows from the fact that the expression pqx — c in

the parentheses in (2.3.16) is negative for x(t] close to XQ. Thus, we have 14

Since x\ > K, we have pqx(t) — c < 0 for all t.

Exact Methods for Synthesis Problems

129

x(t) e R1 for all t £ [0,t«). Hence, it follows from Eq. (2.3.14) with u = 0 that, on the interval [0,**), the population size x(t) increases monotonically up to the value xt = x(t*) that separates the sets R1 and R 2 . At the point x f , as was already noted, the control may take any arbitrary value. It is expedient to take this value equal to

and keep it constant for t > t#. It follows from (2.3.14) that the control (2.3.21) preserves the population size xt. REMARK. For u(xt) ^ MI, the representative point of system (2.3.14), starting from the state a?*, comes either to the set R1 (for u(xt) > MI) or to the set R2 (for u(xt) < «i) during an infinitely small time interval. Then the control u = 0 (or u = um) immediately returns the representative point to the state a;*. Thus, for u(x+) ^ ui, the population size ** is preserved by infinitely rapid switches of control (this is the sliding mode). Though, as follows from (2.3.19), the value of the functional (2.3.16) for this control remains the same as for u(xt) = -MI, the constant control u(t) = u(xt) = HI, t > <*, is more convenient, since in this case the existence problem does not arise for the solution x(t), t > t f , of Eq. (2.3.14). The optimal control 0

for

0 < t < t*

«={(1-V)

fo,

«>«„

(x(t)

< a;*),

<"-22>

realizes the generalized solution x*(t) of Eq. (2.3.14) in the Filippov sense (see §1.1). D Thus, for x(0) = XQ < xi the optimal control (2.3.22) is a piecewise function shown in Fig. 21 together with the plot of the function x*(t), which shows the change of the population size corresponding to this control. It

remains to find the moment tt at which the catching of individuals starts or, which is the same, to find the size (density) xt = xt(tf} of the population that we need to keep constant in the area of active catching. These variables can readily be obtained by calculating the functional (2.3.16) and taking its maximum with respect to t«. Indeed, for the control (2.3.22), the functional (2.3.16) is equal to -

K J

. (2.3.23)

We can calculate its maximum with respect to tt by using the fact that x* = z*(i*) as a function of tt satisfies Eq. (2.3.14) for u = 0. After the

Chapter II

130

FIG. 21 differentiation, from the extremum condition

we obtain the following equation for the optimal size x* of the population: x

t - ± ( ± + ( l - i } K X\ x _ ^ - n • 2\pq + ( 2Mr ' r)") *

(

f2324^ '

This equation has only one positive solution

J

(2.3.25)

which has a physical meaning. Note that, in view of (2.3.17), the value xt determined by (2.3.25) always satisfies the inequality x^ < xt < K.

We also note that the condition

XQ < %i, introduced for the sake of obviousness, does not influence the

Exact Methods for Synthesis Problems

131

choice of the optimal control. This strategy is completely determined by x f , and according to this strategy, we do not catch individuals if the current population density x(t) < xf and start catching with constant intensity (2.3.21) if the population size attains the value xt (2.3.25). We can readily calculate the profit function (2.3.18) corresponding to

this strategy. Integrating the equation in (2.3.14) with a;(0) = x and u = 0, we obtain

-rK

x (w t ) = —————-.——r, x + (K - x)e~Tt

t > 0. ~

v(2.3.26)

'

Using (2.3.26), we see that the condition

xK

— = a;,

X -f {J\ — XJC

(2.3.27)

' '

allows us to find the moment tt. From (2.3.23) and (2.3.27), we explicitly calculate the profit function -Z*

(2.3.28) for x < x*. To solve problem (2.3.14), (2.3.16) completely, it remains to consider the case x(0) = XQ > x*, that is, the case where the initial population size is larger than the optimal size (2.3.25). First, we note that, in view of (2.3.28), the profit function F(x) monotonically increases on the interval 0 < x < x* from zero to the maximum value

(x,) =

dq

(pqx* - c) 1 \ KJ

(2.3.29)

We also note that the function ^(x) has only one maximum point

and since the "killing" factor 6 in (2.3.16) is strictly positive, we always

have the strict inequality x 2 > x*. Now if x(0) = XQ = x2 > £*, then using the constant control (2.3.30)

Chapter II

132

we can keep the population size at a level of x? for which the functional (2.3.16) attains the value

I(u(x2)) = However, the constant control (2.3.30) is not optimal. One can readily see that the functional (2.3.16) can take values larger than l(u(x.2)) = if>(x2) if, instead of (2.3.30), we use the piecewise constant control function

0
(2.3.31)

shown in Fig. 22.

A

x(t)

FIG.

22

We choose a time interval A during which the control um is applied, so that at the end of A the population size attains the value (2.3.25), that is,

«(A) = a;*.15 The time interval A for the control um is determined by the equation , - ________x 2 K(r-qu m )________ «t'* —

p

:

:——————^—;——————TT"-

rx2+[K(r-qum)-rx2\e(iu™-r^

(Z.O.OZ)

15

The inequality I(u&(t)) > 7(«(a;2)) = *l>(xi) is obtained by calculating the func-

tional (2.3.16) with regard to Eq. (2.3.14), where control has the form (2.3.31). Here we do not perform the corresponding elementary but cumbersome calculations and leave

them to the reader as an exercise.

Exact Methods for Synthesis Problems

133

Obviously, control functions of the form (2.3.31) can be used not only for the initial population size x(0) = x% but also for arbitrary initial sizes x(0) = x > xf. In this case, we must perform the change £2 —>• x only in Eq. (2.3.32) for the length A of the initial pulse um. One can easily verify

that (2.3.20) implies tp(x) > 0 for all x > x*. Therefore, the optimal control as a function of the current population size (the synthesizing function) for problem (2.3.14), (2.3.16) has the form

for

for for

0 < x < a;*,

x = x#,

(2.3.33)

x > a;*,

where xt is determined by (2.3.25). Formula (2.3.33) gives the mathematical expression for the control strategy that is well known in the theory of optimal fisheries management [35, 68]. The key point in this strategy is the existence of an optimal size xt of fish population given by (2.3.25). The goal of control is to achieve the optimal size xf of the population as soon as possible and to preserve the size x* by using the constant control (2.3.21). This control strategy maximizes

the profit obtained by fishing if the profit is estimated by the functional (2.3.16). In conclusion, we note that the results presented in this section can be generalized to the case in which the dynamics offish population is subject

to the retarded equation (or the equation with delay) =r(l- ^--V] X(t) - qu(t)x(t); K \ J

here we have studied the controlled Hutchinson model. For the results related to this case, see [99].

The stochastic version of problem (2.3.14), (2.3.16), when the behavior of the population is described by the stochastic equation (2.3.13), will be

described in §6.3. §2.4. Stochastic problem of optimal fisheries management

Now let us consider the problem of optimal fisheries management that differs from the problem considered in §2.3 by the stochastic character of the model used for the description of the population dynamics. We assume that the behavior of a fish population is subject to the stochastic differential equation of the form

x ( t ) = [r- qu(t)]x(t) + V2Bx(t)£(t),

x(0)

= z0,

0 < u(t) < um,

t > 0,

(2.4.1)

134

Chapter II

where £ ( t ) is a scalar Gaussian white noise (1.1.31), B > 0 is a given positive number, the natural growth factor r > 0 and the catchability coefficient q > 0 have the same meaning as similar coefficients in (2.3.10), (2.3.13), and (2.3.14). Equation (2.4.1) is a special case (as K ->• oo) of Eq. (2.3.13) and, in accordance with the classification presented in Section 2.3.1, the model described by Eq. (2.4.1) can be called a controlled stochastic Malthus

model. Just as in §2.3, the size x ( t ) of the fish population is controlled by catching a part of this population. In this case, the catching intensity u(t ) has an upper bound um, and therefore, the set of all nonnegative measurable bounded functions of the form u(t) : [0, oo) —>• [0, um] is considered as the

set of admissible controls. The goal of control is to maximize the functional (2.3.16), which, in view of the random character of the functions x(t) and u(t), is replaced by the corresponding mean value. As a result, we have the problem r

/.oo

I(u) = E \

-|

e~5t (pqx(t) - c}u(t) dt ->•

I JO

J

max

.

(2.4.2)

«(t)6[0,« m ] «>0

In what follows, we assume that the decay index 6 in (2.4.2) satisfies the

condition 8 > r. We shall solve problem (2.4.1), (2.4.2) by using the standard procedure of the dynamic programming approach described in §1.4. We define the profit function for problem (2.4.1), (2.4.2) by the relation F(x)=

r r°° e~

max E / «(«)e[o,« m ] Uo t>o

Si

i

(pqx(t) - c)u(t) dt \ x(Q) = x , \

(2.4.3)

where £[(•) [ x(0) = x] denotes the conditional mathematical expectation of (•). As was shown in [113, 175], the second-order derivative of the profit function (2.4.3) is continuous. It follows from Theorem 3.1.5 in [113] that for all x G R + — [0, oo), this function has the upper bound

F(x)
(2.4.4)

where N > 0 is a constant.

The Bellman equation (Fx - ^, Fxx = f^f) max [Bx2 Fxx + (r + B - qu)xFx - SF + (pqx - c)u] = 0

0<«<« m

(2.4.5)

for the profit function (2.4.3) can be obtained as usual (see §1.4). It should

be pointed out that a symmetrized stochastic integral (see [174] and §1.2)

Exact Methods for Synthesis Problems

135

was used for writing (2.4.5). This led to an additional term B in the parentheses in (2.4.5), that is, in the coefficient of xFx.le Equation (2.4.5) allows us to find the optimal control ut as a function u i f ( x ) of the current states of system (2.4.1). First, we note that, according to (2.4.5), the set of all admissible states of (2.4.1) can be divided into the following two subsets (just as in §2.3):

the subset R 1 , where
the subset R 2 , where f>(x) > 0 and u*(x) = um. The boundary between these two subsets is determined by the relation pqx - c - qxFx - 0.

(2.4.6)

The further calculations show that, in this problem, there exists a unique point x* satisfying (2.4.6). Therefore, the subsets R1 and R2 are the intervals R1 = [0, x*) and R2 = (x*,oo). Thus the optimal control in the synthesis form Ut = u*(x) becomes uniquely determined at all points x <E R+

except for the point x*. It follows from (2.4.6) that we can use any admissible control u(z*) 6 [0,itm] at the point x*. Therefore, the optimal control function M*(X) can be represented in the form ( \ ' ,,

u*(x)=<

(2.4.7)

I Mm,

if

X > X*,

and the final solution of the synthesis problem is reduced to calculating the coordinate of the switch point X*. To calculate x*, we need to solve the Bellman equation (2.4.5). As was already noted, the second-order derivative of the profit function F(x) is continuous, thus the profit function F(x) satisfying (2.4.5) can

be obtained by using the matching method (see §2.2). In what follows, we describe the procedure for solving the Bellman equation (2.4.5) and calculating the coordinate of the switch point xf in detail. By F*(x) and F2(x) we denote the profit function F(x) on the intervals

R1 = [0, x,) and R2 = (a:,, oo). It follows from (2.4.5) and (2.4.7) that the functions Fl and F2 satisfy the linear equations

- SF1 = 0, 2

Bx F

2

2

x

0 < x < x,, 2

+ (r + B - qum)xF - SF + (pqx

- c)um = 0,

(2.4.8) x > a;,.

(2.4.9) 16 If the stochastic differential equation in (2.4.1) is the Ito equation, then the second term in the Bellman equation (2.4.5) has the form (r — qu)xFx.

136

Chapter II

Since the profit function F(x) is sufficiently smooth (recall that the secondorder derivative of F(x) is continuous), both functions F1 and F2 must satisfy the condition (2.4.6) at the switch point x*. Taking into account the fact that F(0) = 0 according to (2.4.1) and (2.4.3), we have the following additional boundary condition for the function F!(x):

Fl(0) = 0.

(2.4.10)

The boundary conditions (2.4.6), (2.4.10), and the upper bound (2.4.4) allow us to obtain the explicit analytic solution of Eqs. (2.4.8) and (2.4.9). Equation (2.4.8) is the well-known homogeneous Euler equation. Its general solution has the form

,

(2.4.11)

where AI and A? are constants, and k\ and k\ satisfy the characteristic equation

Bk(k-l) + (r + B)k-S = 0.

(2.4.12)

The constants AI and A^ are determined by two boundary conditions (2.4.10) and (2.4.6) at the points x = 0 and x = a;,. Since the roots

of Eq. (2.4.12) have opposite signs, we conclude that to satisfy the condition (2.4.10), we need to set A2 equal to zero in (2.4.11). The constant c

~k

can be calculated by substituting Fl(x) = Aixk> into (2.4.6) and taking into account the fact that (2.4.6) is valid at the switch point x*. Thus, the

solution of Eq. (2.4.8) is given by the formula

>

0<x<x*.

(2.4.14)

The inhomogeneous Euler equation (2.4.9) can be solved in a similar way. By using the standard method of variation of parameters, we obtain the general solution PqX

Exact Methods for Synthesis Problems where

137

___________ *? = ^ [««m - r + >/(««m - r) 2 + 4B<5] ,

Y ^2 = ^[?w™ -

r

Lti

___________ ~ V(lum -r)2 + 4BS]

(2.4.16)

satisfy the characteristic equation

Bk2 - (qum -r)k-S = 0, and Ay and A± are arbitrary constants.

Since k2 is positive, we must set the constant A$ equal to zero (otherwise, formula (2.4.15) contradicts the upper bound (2.4.4)). The constant A^ can

be calculated from condition (2.4.6) at the switch point x f . Substituting F2(x) (determined by (2.4.15) with A3 = 0) into (2.4.6) instead of the function F, we obtain A

.

(S-r-B)p

——— ________ 1____________________ '___________ rp

4 — r 7/ r

n

,

\

i_fc*

^

__

*

c

_______ /w

T 7

-kl

2,

*

This implies the following expression for the function F2(x):

6 - r - B + qum

l\(6-r-B)px*

c]fx\k'

+ 72" r————— JT^ ——— ~ T

kl [S-r-B + qum

— )

JJV**/

i

x >x

~

*-

(2.4.17)

The two functions F1(x) (2.4.14) and F2(x) (2.4.17) determine the profit function F(x) satisfying the Bellman equation (2.4.5) for all x £ R+ = [0,oo). These functions contain a parameter a;*, which remains unknown.

We can calculate x* by using the continuity property of the profit function F(x). Each of the functions Fl and F2 is continuous. Hence, to ensure the continuity of F(x), it suffices to satisfy the condition

Fl(x*) = F2(x*)

(2.4.18)

at the switch point x». It follows from (2.4.6), (2.4.8), and (2.4.9) that (2.4.18) is equivalent to the condition

F2,(x.) = F2x(x*) at the switch point x*.

(2.4.19)

138

Chapter II

Calculating the second-order derivative of the functions (2.4.14) and (2.4.17), we derive the following equation for x» from (2.4.19):

p(6 — r — B)

c

Hence, the switch point xt is determined by the explicit formula T _ JLtjfi

c(d — r — tf + qum)

——

/1 I

*\

*

I Zf.rt.ZdJ I

pq[S — r — B + ,ki~k2\qum] Formula (2.4.20) and the optimal control algorithm (2.4.7) constitute

the complete analytic solution of the stochastic problem (2.4.1), (2.4.2) of optimal fisheries management. Some final comments and remarks. It is of interest to compare (2.4.20) and (2.3.25), which is the optimal size of the population in the deterministic problem (2.3.14), (2.3.16) of optimal control considered in §2.3. Denoting (2.3.25) by xf, we may expect that the equality

lim x* = lim xt

K-s-oo

B-X)

(2.4.21)

is valid due to continuity reasons (since the deterministic version of problem (2.4.1), (2.4.2) formally coincides with problem (2.3.14), (2.3.16) as K ->• oo). We can verify (2.4.21) by straightforward calculations of the limits on both sides. Indeed, using (2.3.25), we readily calculate the limit on the left-hand side of (2.4.21) for S > r,

lim i, =

K-+OO

f

pq(S - r)

(2.4.22) ^

'

The same result is obtained by calculating the limit of (2.4.20) as B —>• 0, since fej-l v (S-r)(qum-r) k\ — k\ B->o which follows from (2.4.13) and (2.4.16). Formula (2.4.21) shows how the results obtained in this section for problem (2.4.1), (2.4.2) are related to similar results for problem (2.3.14), (2.3.16) obtained in Section 2.3.3 by quite different methods.

There is another interesting specific feature of problem (2.4.1), (2.4.2). Namely, the standard "classical" approach of the dynamic programming

Exact Methods for Synthesis Problems

139

that leads to the exact solution of the stochastic problem (2.4.1), (2.4.2) does not allow us to solve the synthesis problem (that is, to find the switch point Xf) for the deterministic version of problem (2.4.1), (2.4.2), that is, in the case where there are no random perturbations in Eq. (2.4.1). This fact can readily be verified if we consider the deterministic version of the

Bellman equation (2.4.5). max \(r - qu)xFx - SF + (pqx - c)u] = 0

(2.4.23)

0
and calculate the functions

F'( z ) = l(f>z.-£)(±.J :c I

i •"

d \o-l

,

(2.4.24)

-«'l <- I + T^ i / \7 i \ •*-* / a[(r-qu )(a-l) m

I —

q\ \a;*

* r - qum

(2.4.25)

which, in this case, determine the profit function F(x) on the intervals R1 = [0, »*) and R 2 = [a;*, oo). Contrary to the stochastic case in which the continuity condition (2.4.18) for the functions (2.4.14) and (2.4.17) determines the unique switch point (2.4.20), one can readily verify that the same continuity condition F 1 ( x f ) = F2(x*) for the functions (2.4.24) and (2.4.25) holds for any point xf € (0, oo). Therefore, the control problem considered can serve as an example illustrating the well-known idea (see [113, 175]) that the dynamic programming approach is more suited for solving control problems with stochastic

models of plants (which, by the way, describe the actual reality more adequately) . REMARK. If the equation in (2.4.1) is understood as the Ito stochastic equation, then the Bellman equation for problem (2.4.1), (2.4.2) differs from (2.4.5) and has the form max [Bx2Fxx + (r - qu)xFx — SF + (pqx — c)u] = 0.

0<«<« m

The way for solving this equation is quite similar to the above procedure for solving Eq. (2.4.5). However, the population size xf that determines

the switch point for the optimal control (2.4.7) differs from (2.4.20) and is

140

Chapter II

given by the expressions

_

=

c(S - r + qum)

s B + qum - r - y/(B + qum - r) 2 + 46B . D

CHAPTER III

APPROXIMATE SYNTHESIS OF STOCHASTIC CONTROL SYSTEMS WITH SMALL CONTROL ACTIONS

Various approximate synthesis methods can be useful if the Bellman equation cannot be solved exactly. Chapters III-VI deal with some of these

methods. Approximate methods are usually efficient if the initial statement of the optimal control problem contains a small parameter. Quasioptimal control algorithms are constructed by using either the corresponding procedures of successive approximations or asymptotic expansions of the loss function in powers of a small parameter of the problem. The choice of a method for constructing an approximate solution of the synthesis problem essentially

depends on the choice of a parameter that is considered to be small. For example, in this chapter, the values of control actions are assumed to be small. Chapter IV is about the Bellman equation with small diffusion coefficients. In Chapter V, we consider control problems for oscillating systems with small attenuation decrement. In Chapter VI, the role of small parameters is played by the a posteriori covariances of unknown coefficients in the plant equations. Let us formulate the main idea of the approximate synthesis method studied in this chapter. As was already noted, the method is based on the assumption that control actions on the plant P are relatively small. Prom

the physical viewpoint, this assumption means that the effect of the control actions on the phase trajectories of the system is small, and therefore the system dynamics is similar to noncontrolled motion. In particular, this

assumption holds for control problems with constraints if the noises acting on the plant are of large intensity. Indeed, let us assume that the controlled (unperturbed) plant is a stable

mechanical system. Then large random perturbations lead to large deviations of the system from the equilibrium state. In this case, some "internal" inertial and elastic forces arise in the system. These forces can significantly

exceed the (bounded) control forces whose effects on the system turn out to be relatively small.1 1

Note that in this book we do not consider deterministic synthesis problems for

141

142

Chapter III

From the formal mathematical viewpoint, the fact that control actions are small leads to a small parameter in the nonlinear term in the Bellman equation. To verify this fact, let us consider the synthesis problem for the servomechanism (Fig. 10) governed by the Bellman equation (1.4.21). Assume that the dimensions of the region U of admissible controls are bounded by a small value of the order of e. For definiteness, we assume that U is either an r-dimensional parallelepiped (RT D U = (u: u,-| < umi, i — 1, 2, . . . , r; maxj umi = e)) or an r-dimensional ball of radius e, that is, (R, D U = (u: £J=1 «? < e ) ) . In the first case, according to (1.3.22), the solution of the synthesis problem is given by the formula (the control algorithm)

= - { « m l , - . - , W m r } s i g n ( QT (t) -r— (t, X, y) 1,

\

dx

J

(3.0.1)

where the vector of partial derivatives dF/dx is calculated by solving the equation

LF(t, x,y) = -ci ( x , y ) + eu^ ^

w

~ - r, *,,,

(3.0.2)

Here um denotes the r-vector (column) um/e and L is a linear operator of the form 2

"'^^ u -

(see

32

In the second case (where U is a ball), the optimal control has the form (1.3.23))

systems controlled by small forces. Such systems called weakly controllable in [32] were studied in [32, 137]. 2 Recall that relations (3.0.1) and (3.0.2) follow from the Bellman equation (1.4.21)

with c(x,y,u) = ci(x,y) and Ax(t,x) = a(t,x) + Q(t)u; {uml,.. .,umr} denotes a diagonal (r X r)-matrix; for a column A with components AI ,..., Ar, the expressions sign A and \A\ denote j--columns with components sign-A,- and \A{\ (i = 1,..., r), respectively.

Approximate Synthesis of Stochastic Control Systems

143

where the vector dF/dx is the gradient of the loss function satisfying the equation

(

op

1

ft]?\ ~^

QQT-

•

(3.0.4)

If we denote the nonlinear terms in Eqs. (3.0.2) and (3.0.4) in the same way, then we can write both equations in the form

,

(3.0.5)

where $(f, dF/dx) is a given nonlinear function of its arguments. As a rule, equations of the type (3.0.5) cannot be solved exactly. However, the presence of a small parameter in the nonlinear term of this equation yields a rather natural way of solving this equation approximately. To this end, one can use the method of successive approximations in which the

zero-order approximation Fo(t, x,y) satisfies the equation

LF0 = -c1(x,y)

(3.0.6)

and the successive approximations Fk(t,x,y) can be calculated recurrently by solving the sequence of linear equations of the form

A =1,2,....

(3.0.7)

If we know the solution Fk(t, x, y) of the equation for the /sth approximation (fc = 0, 1, . . .), then we can perform an approximate synthesis of the controlled system by setting the quasioptimal control algorithm as (3.0.8)

In this chapter we consider an approximate method for the synthesis of optimal systems whose "algorithmic" essence is given in formulas (3.0.6)-

(3.0.8).3 Needless to say, the practical use of procedure (3.0.6)-(3.08) in special problems leads to additional problems of constructivity and efficiency of 3 The approximate synthesis algorithm (3.0.6)-(3.08) is a modification of the wellknown Bellman method of successive approximations [14, 16]. This method was used by

W. Fleming for solving some stochastic problems of optimal control [55]. The procedure

(3.0.6)-(3.0.8) is a special case of the Bellman method if the trivial strategy uo(t,x,y) = 0 is used as the initial "generating" control strategy in the Bellman method.

144

Chapter III

this approximate synthesis method. In this chapter we shall discuss these problems in detail. All related material is divided in sections as follows. First (§§3.1-3.3), we consider some methods for calculating the successive approximations for stationary synthesis problems. We write out approximate solutions (that correspond to the first two approximations) for some

special control systems with various types of disturbances affecting the system. In §3.1 and §3.2, we consider random perturbations of the white noise type. In §3.3 the results obtained in §3.1 and §3.2 are generalized to the case of correlated noises. In §3.4 we study nonstationary problems and estimate the error of the approximate synthesis (3.0.6)-(3.0.8) for the first two approximations. In §3.5 we study asymptotic properties of successive approximations (3.0.7), (3.0.8) as k —>• oo. We show that, under some special conditions, as k —> oo the sequence F& is convergent to the exact solution of the Bellman equation, and the corresponding quasioptimal control algorithms (3.0.8) to the optimal control ut(t,x,y). In this case, the convergence Uk -> uf is understood in the sense of convergence of values of the functional to be minimized. Finally, in §3.6 the method of successive approximations (3.0.6)^(3.0.8) is used for approximate synthesis of some stochastic control systems with distributed parameters. §3.1.

Approximate solution of stationary synthesis problems

3.1.1. Let us consider the problem of optimal damping of oscillations in a dynamic system subject to random perturbations of the white noise type (Fig. 13). Let the plant P be described by the following system of linear stochastic differential equations with constant coefficients:

(3.1.1)

Here x = x ( t ) is an n- vector (column) of current phase variables of the system (xi(t), . . . , xn(t )) , u = u(t) is an r-vector (column) of control actions (ui(t), . . . ,u r (t)), £(t) is an n-vector (column) of random perturbations with independent components (£i(<), . . • ,£»»(<)) of the standard white noise type (1.1.31), and A, Q, and
J

(3.1.2)

Approximate Synthesis of Stochastic Control Systems

145

where c(x) > 0 is a given convex penalty function attaining the absolute minimum c(0) = 0 at the point x — 0 (the restrictions on c(z) are discussed in detail in §3.4 and §3.5). Let admissible controls be bounded and small. We assume that all components of the control vector u satisfy the conditions

K < £umi,

i=l,...,r,

(3.1.3)

where e > 0 is a small parameter and w m i , . . . , umr > 0 are given numbers

of order 1. The system shown in Fig. 13 is a special case (the input signal y(t) = 0) of the servomechanism shown in Fig. 10. Therefore, the Bellman equation

for problem (3.1.1)-(3.1.3) readily follows from (1.4.21), and taking into account the relations Ay(t, y) = 0, By(t, y) = 0, Ax(t, x, u) = Ax + Qu, and c(x,y,u) = c(x), we obtain dF(t x}

' +LF(t,X)+ Ot

min

T

gT

+ c(x) = 0.

(3.1.4)

Here L denotes a linear elliptic operator of the form

(3 L5)

-

where, according to (1.4.16), the matrix B = aaT , and, as usual, the sum in the last expression on the right-hand side of (3.1.5) is taken over repeated indices from 1 to n. It follows from (3.0.1) and (3.0.2) that in this case the optimal control has the form

(3.1.6) where the loss function F(t, x) satisfies the equation

dF —— (t, a;) + LF(t, x) - -c(x) + eu

(3.1.7)

Some methods for solving Eq. (3.1.7) will be considered in §3.4. In the present section, we restrict our consideration to stationary operating conditions of the stabilization system in question. It follows from §1.4 that the stationary mode of stabilization (damping) can take place if T —)• oo, where T is the terminal instant of the operation interval (the upper

Chapter III

146

integration limit in (3.1.2)). Obviously, in this case, stationary operating conditions exist only if the unperturbed motion of the plant P is stable, that is, in other words, if the real parts of the eigenvalues of the matrix A in (3.1.1) are negative. In what follows, we assume that these conditions

are satisfied. If we define the stationary loss function f ( x ) by the relation (see (1.4.29), (2.2.9))

then (3.1.7) implies the following time-invariant equation for f ( x ) :

Lf(x)

= 7 - c(x) +

(3.1.8)

where the parameter 7 characterizing stationary "specific losses" together

with the function f ( x ) can be found by solving Eq. (3.1.8). We shall solve Eq. (3.1.8) by the method of successive approximations. The scheme for calculating (3.0.6), (3.0.7), applied to the time-invariant equation (3.1.8) leads to the sequence of equations

(3.1.9)

Lfo = 7° - c ( x ) ,

Lfk = Jk- c(x)

Q1

dx

(3.1.10)

It follows from (3.1.9) and (3.1.10) that each time when we are calculating the next approximation, we need to solve a linear inhomogeneous elliptic equation of the form

Lf(x) =

(3.1.11)

We shall consider a method for solving Eq. (3.1.11) with a given function
in eigenfunctions of some Sturm-Liouville problem [179]. 3.1.2. The passage to the adjoint equation. Let us consider the operator

that is the dual of the operator (3.1.5). The equation

— (<, x) = L*p(t,x)

(3.1.13)

Approximate Synthesis of Stochastic Control Systems

147

is the Fokker-Planck equation (1.1.67) for the n-dimensional Gaussian Markov process x(i) [45, 167, 173]. The assumption that the matrix A is stable implies that this process has a stationary density function po(x) such that

L*p0(x) = 0.

(3.1.14)

In this case the stationary probability density po(x) has the form

po(x) = V

. /

1

«

T ——— exp [L - \(x Px)} J . 2 - i

(3.1.15) '

where P"1 is the matrix of covariances of components of the vector x.

We shall present a possible method for solving Eq. (3.1.13) and calculating the matrix P in (3.1.15). The diffusion Markov process x ( t ) described by (3.1.13) satisfies the system of linear stochastic differential equations

x = Ax +
(3.1.16)

describing the uncontrolled motion of the plant (3.1.1). We pass from x to new variables y related to x by the linear transformation

x = Vy

(3.1.17)

with a nondegenerate matrix V. As a result, instead of (3.1.16), we have the following system for the new variables:

(3.1.18) where

A = V~iAV,

a = V-1a.

(3.1.19)

We choose V so that the matrix A is diagonal

I={A1,A2,...,An}.

(3.1.20)

As is known [62], the matrix V always exists and can readily be constructed if the eigenvalues of the matrix A are simple, that is, if the characteristic equation of the matrix A,

det(j4. - \E) = 0,

(3.1.21)

has different roots AI, A 2 , . . . , A n . In this case, the columns of the matrix V are the eigenvectors vl of the matrix A that satisfy the linear equations

Aff=\ivi,

i=l,...,n.

(3.1.22)

148

Chapter III

The system (3.1.18) can readily be solved in the case of (3.1.20). Indeed, writing (3.1.18) in rows, we obtain

yt = Aiyi + rli(t),

£=l,...,n,

(3.1.23)

where the random functions rjt(i) — &tk(,k(t) are processes of the white noise type and have the characteristics

Eife(*) = 0,

Em(t)rjm(t - T) = BtmS(r)-

£ , m = l , . . . , n . (3.1.24)

Here Btm is an element of the matrix B = cfcfT. obtain

yt(t) = yt0e^-^ +

Solving Eq. (3.1.23), we

e^-r,^') dt',

ylo = W (t 0 ),

(3.1.25)

Jto

and taking into account (3.1.24), derive the following expressions for the means and covariances:

(3.1.26)

which determine the transition probability p(y(t) process y(t). It follows from (3.1.26) that

Ey/ = 0,

Eytym = -

y(to)) of the Gaussian

B m

^ .

(3.1.27)

*t + *m

in the stationary case as t —> oo, since, by assumption, ReA^ < 0 (i = 1, . . . , n) for all roots of the characteristic equation (3.1.21) of the matrix A. It follows from (3.1.27) that the stationary density function po(y) can be written in the form

po(y) =

1

=exp[-i(y 1

r

Py)],

^/(27r)«detP-

where each entry of the matrix P"1 is given by the formula

(3.1.28)

Approximate Synthesis of Stochastic Control Systems

149

The stationary density po(y) satisfies the stationary Fokker-Planck equation

L*Po(y) = --(XiViPo) + ^jj-(BijPo) dyi

2 ayidyj

= 0.

(3.1.30)

Since the random processes y(t) and x(t) are related by the linear transformation (3.1.17), the comparison of (3.1.15) with (3.1.28) yields the formula

P = VTPV,

(3.1.31)

which together with (3.1.29) allows us to calculate the matrix P. Now let us return to the Fokker-Planck equation (3.1.14). If the operator (3.1.5) satisfies the potentiality condition (see §4, Section 5 in [173]), then the operator equality4 POL

= L*PQ

(3.1.32)

readily follows from (3.1.5), (3.1.12), and (3.1.14). However, even if the

potentiality conditions are not satisfied and (3.1.32) does not hold, one can choose an operator L\ satisfying a similar relation PoL

= L\PQ.

(3.1.33)

One can readily see that the operator L\ has the form5

ri = - ( G , ^ +

(^),

(3-1.34)

where the matrix G = ||G,-j||" is similar to the transpose matrix AT from (3.1.1) and (3.1.5),

G=P-1ATP.

(3.1.35)

The similarity transform (3.1.35) employs the matrix P from (3.1.15). Relation (3.1.33) allows us to replace Eq. (3.1.11) by a similar equation

for the dual operator. In other words, it follows from (3.1.11) and (3.1.33) that the problem of finding f ( x ) in (3.1.11) is equivalent to the problem of finding z(x) in the equation

L\z(x) = ip(x),

(3.1.36)

where z(x), i>(x), and the functions /(a;), V>(x) from Eq. (3.1.11) satisfy the relations

z(x)=po(x)f(x), 4

if>(x) = Po(x)if>(x).

(3.1.37)

As usual, the operator equality is understood in the sense that it is an ordinary

r e l a t i o n p g ( x ) L w ( x ) = L*po(x)w(x) for any sufficiently smooth function w(x). 5 The verification of (3.1.33) is left to the reader as an exercise.

150

Chapter III

3.1.3. The solution of equations (3.1.36) and (3.1.11). Let us consider the following problem of finding the eigenfunctions zs (x) and eigenvalues A s of the operator L\ (the Sturm-Liouville problem):

L\zs = \,z,.

(3.1.38)

Since L\ is the Fokker-Planck operator, its eigenfunctions zs must satisfy the zero conditions at infinity (as x —> oo).

By passing from x to new variables y (x = Vy) and acting in a way similar to (3.1.17)-(3.1.31), we can transform the operator (3.1.34) to the form £;

=-(A,.tt)+ I ( B

t f

) ,

(3.1.39)

where Bij is an element of the matrix B = a (TT , "a = V cr, V is a nonde__ _ i __ __ generate matrix such that the transformation V GV makes the matrix G diagonal, G = {Ai, . . ., A n }, and A; are roots of Eq. (3.1.21).6 In the new variables the stationary Fokker-Planck equation has the form

*

- 0-

^-

This equation differs from (3.1.30) only by the matrix of diffusion coefficients; therefore, the stationary probability density p0(y) is determined by the formulas

pQ(y) =

1

_= exp [ - |(j/TPy)] ,

(3.1.41)

y(27r)"detP

1 similar to (3.1.28) and (3.1.29). Differentiating (3.1.40) appropriately many times, we see that

_

+ m 2 A 2 + • • • + mn \n) —^——jr^Po = 0,

(3.1.43)

According fco (3.1.35), the matrix G is similar to the transpose AT. Since all similar and transpose matrices have the same eigenvalues, the characteristic equation det(G — AB) = 0 for the matrix G coincides with (3.1.21)).

Approximate Synthesis of Stochastic Control Systems

151

where mi, m?,..., mn are any arbitrary integers between 0 and oo. It follows from v(3.1.43) that the functions -J^j—Q 1" Pn an-d the num' dy-i ---dyn bers (miAi + • • • + m n A n ) can be treated as the eigenfunctions zs and the

eigenvalues Xs of problem (3.1.38), respectively. By using (3.1.41), we can write the functions zs in more detail as follows:7 Zs

= (_l)™i+"-+™»tf rai ... ra J y )exp [- f(y T Pj/)].

(3.1.44)

Here Hmi,_mn(y) — Hmi...mn(yi: • • -,2/n) denote multidimensional Hermitian polynomials (for instance, see [4]) that, by definition, are equal to Hm^,...,mn(y) = (-l) mi +"+ m « exp [f (j/TPt/)]

"

(3.1.45)

It follows from the general theory [4] for Hermitian polynomials with real variables y that these polynomials form a closed and complete system of functions, and an arbitrary function from a sufficiently large class (these functions grow at infinity not faster than any finite power of \y\) can be expanded in an absolutely and uniformly convergent series in this system of functions. Furthermore, the polynomials H are orthogonal to another

group of Hermitian polynomials G given by the formula

"-|(/i T P"V)]-

(3.1.46)

Here the variables fj, and y satisfy the relation or

=

and the orthogonality condition itself has the form

r00

i"*

J — oo

J — oo

_

MetP

..Sv,m,

(S1/imi is the Kronecker delta). 7

The constant coefficient [(2ir) n detP "'j- 1 / 2 in (3.1.44) is omitted.

(3.1.48)

152

Chapter III

However, we often need to use a complex matrix V for the change of variables x —> y (for instance, see the problem in §3.2.). To pass to complex variables, we need to verify some additional statements from the general

theory [4], which hold for real variables. In particular, it is necessary to verify the orthogonality conditions (3.1.48), which are the most important in practical calculations.

This was verified in [107], where it was shown that all properties of the polynomials H and G remain valid for complex variables if only all functions HVl...Vn(y), Gmi...mn(y), exp[|(yTPy)], and exp[|(/iTP fj,)] are considered as functions of the initial real variables x of the problem. To this end, we need to make the change of variables y = V x in all these functions. In particular, in this case, the orthogonality condition (3.1.48) has the form oo

/

/»oo

... / -oo

e-^xTp^HVl...

J — oo

..£„„„,„,

(3.1.49)

where the matrices PI and P satisfy the relation

~P = VTP1V

(3.1.50)

similar to (3.1.31). Thus, we obtain the following algorithm for constructing the solution f ( x ) of Eq. (3.1.11). First, we seek a stationary density po(x) satisfying (3.1.14) and an operator L\ satisfying (3.1.33). Then we transform prob-

lem (3.1.11) to problem (3.1.36). After this, to find the eigenfunctions and eigenvalues of problem (3.1.38), we need to calculate the matrix V that transforms the matrix G to the diagonal form { A i , . . . , A n } by the siml-

larity transform V GV. Next, using the known A,- and V and (3.1.42), we calculate the matrices P and P that determine the stationary distribution (3.1.41). The expression obtained for p0(y) enables us to find the eigenfunctions zs = zmi...mm (3.1.44) for problem (3.1.38) and the orthogonal polynomials G TOl ... TOn (3.1.46). Finally, we seek the function z(x) satisfying (3.1.36) in the form of the series with respect to the eigenfunctions: oo z x

( ) =

X) mj....m n =0

a

m1...mnZmi. ..mn(x),

(3.1.51)

Approximate Synthesis of Stochastic Control Systems

153

where omi...ran are unknown coefficients; the eigenfunctions zmi,,.mn(x) can be calculated by formulas (3.1.44) with y = V x. If we also represent the right-hand side ij)(x) — po(x)(x) in (3.1.36) as the series oo

po(x)
^

bmi...mnzmi...mn(x),

(3.1.52)

where, in view of (3.1.49),

*— r...r

A/det PI (27r) n / 2 mi... !.. .m .,,„„. ^_00 00 n\ J_

Jj.oo _,

T-l i x G r o i ... T O/nT(F~ z)dxi...dz n

(3.1.53)

then we can calculate the unknown coefficients oroi...TOn in (3.1.51) by the formula ^m i...mn

\

ami...mn - T———— ,

\

.

, \

A mi ... mn = Aimi + ••• + Xnmn,

/ o i c x \

(3.1.54)

which follows from (3.1.38) and (3.1.43). Now we see that (3.1.37) implies the expression

for the solution of the initial equation (3.1.11). The algorithm obtained for solving (3.1.11) can be used for calculating

the successive approximations (3.1.9) and (3.1.10) It remains only to solve the problem of how to choose the stationary losses 7^ (k=0,l,2, ... ) in

Eqs. (3.1.9) and (3.1.10). 3.1.4. Calculation of the parameters Vs (k - 0,1,2,...). The structure of the solution (3.1.55) and a natural requirement that the stationary loss function f ( x ) must be finite imply that there is a unique method for choosing 7*. Indeed, since, according to (3.1.54), the eigenvalue AQO...O = 0, the coefficient aoo...o in (3.1.46) is finite if a necessary condition 60o.. .0 = 0 is satisfied, or, more precisely, (in view of (3.1.53) and (3.1.46)) if we have oo

/

/»oo

.../ •oo

J — oo

PQ(X)
154

Chapter III

This relation, (3.1.9) and (3.1.10) imply the following expressions for the stationary losses 7*: OO

/

/»OO

.../ -OO

c(x)pQ(x)dxl...dxn,

(3.1.56)

J — OO

Tdfk-i(x)

Q'

Po(x)dxi...dxn, dx = 1,2,.... (3.1.57)

Thus, we have completely solved the problem of how to calculate the successive approximations (3.1.9), (3.1.10) for the stationary operating conditions

of the optimal stabilization system. If the loss function f k ( x ) in the fcth approximation is calculated, then the quasioptimal control Uk(x) in the feth. approximation is completely defined,

namely, in view of (3.0.8) and (3.1.6), we have

(3.1.58) In the next section, using this general algorithm for approximate synthesis, we shall calculate a special system of optimal damping of random oscillations when the plant is a linear oscillating system with one degree of freedom. §3.2. Calculation of a quasioptimal regulator for the oscillatory plant

In this section we consider the stabilization system shown in Pig. 13, in which the plant P is an oscillatory dynamic system described by the equation

x + /3x + x = u + VB£(t),

(3.2.1)

where the absolute value of the scalar control u is bounded,

\u\ < e,

(3.2.2)

the scalar random process £(t) is the standard white noise (1.1.31), and /?,

B, and s are given positive numbers ((3 < 2). Equations of the type of (3.2.1) describe the motion of a single mass

point under the action of elastic forces, viscous friction, controlling and random perturbations. The same equation describes the dynamics of a direct-current motor controlled by the voltage applied to the armature when

Approximate Synthesis of Stochastic Control Systems

155

the load on the shaft varies randomly. Examples of other actual physical objects described by Eq. (3.2.1) can be found in [2, 19, 27, 136]. For system (3.2.1), (3.2.2), it is required to calculate the optimal regulator (damper) C (see Fig. 13), which will damp, in the best possible way with respect to the mean square error, the oscillations constantly arising

in the system due to random perturbations £(t). More precisely, as the optimality criterion (3.1.2), we shall consider the functional

(x2(t)

=E

x2 (t)) dt] , J

(3.2.3)

which has the meaning of the mean energy of random oscillations in system (3.2.1). Note that the mean square criterion (3.2.3) is used most frequently and this criterion corresponds to the most natural statement of the optimal

damping problem [1, 50]. However, there are other statements of the problem with penalty functions other than the function c(x] = x2 + x2 exploited in (3.2.3). From the viewpoint of the method used here for solving the synthesis problem, the choice of the penalty function is of no fundamental importance. To make the problem (3.2.1)-(3.2.3) consistent with the general statement treated in §3.1, we write Eq. (3.2.1) as the following system of two first-order equations for the phase coordinates x\ and x% (these variables can be considered as the displacement x\ = x and the velocity X2 = x):

(3.2.4)

= —zi —(3x2

Using the vector-matrix notation, we can write system (3.2.4) in the form (3.1.1), where A, Q, and
A=

0

0

Q=

(3.2.5)

According to §3.1, under the stationary operating conditions (T -» oo in (3.2.3)), the desired optimal damper C (Fig. 13) is a relay type regulator described by the equation (see (3.1.6))

u*(xi,x2) = -esign

-— . \OX2'

(3.2.6)

Here / = /(xi^xz) is the loss function satisfying the stationary Bellman equation (see (3.1.8))

df

(3.2.7)

Chapter III

156 where, according to (3.1.5) and (3.2.5),

L — x2-— - (/3x2 +

B d2

-— + IT -5-

(3.2.

ox-2

The equation

21

>.i-j.i.

determines a switching line for the optimal control action (from u — +£ to u = —£ or backwards) on the phase plane (KI, x2). The goal of the present

section is to obtain explicit expressions for the control algorithm (3.2.6) and the switching line (3.2.9). To this end, it is necessary to solve Eq. (3.2.7). We shall solve this equation by the method of successive approximations

discussed in §3.1. First, we shall prepare the mathematical apparatus for calculating the successive approximations. A straightforward verification shows that the stationary distribution with the density function po(x) = po(xi,x2), satisfying the equation (see (3.1.14))

f

=

has the form

(3.2.10) Hence, the matrices P and P~l in (3.1.15) are equal to

2/3 B

B

I 0 0 1

i o o i

(3.2.11)

It follows from (3.2.11) and (3.1.35) that in this case the matrix G of the

operator (3.1.34) coincides with the transpose matrix AT, that is, according to (3.2.5), we have " 0 -1 G= (3.2.12) 1 -0 and the operator (3.1.34) has the form

d

o

B d2 f\ s\ 2

(3.2.13)

One can readily see that the same probability density (3.2.10) satisfies the stationary equation Llpo(xi,x2) = 0. Therefore, in this case, the matrix PI from (3.1.49) and (3.1.50) coincides with the matrix P determined by (3.2.11).

Approximate Synthesis of Stochastic Control Systems

157

The matrix V that reduces (3.2.12) to the diagonal form by the similarity transform is equal to

1

V =

1

—AI

—A 2 (3.2.14)

This expression and formulas (3.1.50) and (3.2.12) imply B

B

Correspondingly, the inverse matrix P

B 2(4-,

2//J

(3.2.15)

-A2

has the form

A2 2//J 2//3 A!

(3.2.16)

The matrices (3.2.15) and (3.2.16) allow us to calculate the two-dimensional Hermitian polynomials .

Gim

r 1 / T7"^" \ T

, e i _,

f)l+m

^

r

1 / T1 ~^ \ i

exp

= (-!)

(3.2.18)

VL = Py.

Then these polynomials must be represented as functions of xi and x? by using the formula x = Vy and expression (3.2.14) for the matrix V. Table 3.2.1 shows some first polynomials H and G. In this case, in view of (3.1.51)-(3.1.55), (3.2.7), (3.2.10), and (3.2.11), the solutions of the equations of successive approximations (3.1.9), (3.1.10) can be written in terms of the Hermitian polynomials H^m(xi,X2) as the

series mA 2 '

(3.2.19) where the coefficients b\m are calculated by the formulas

(3.2.20) £

/V K __

/

-

9

V* ——
^1

9

I


^2 i ^

> 1.

Chapter III

158

TABLE 3.2.1 Polynomials H #00 = 1

-

-

2/3

-H

, p

2/3

//3

\

X2 #10 — -Pnyi + P\iyi — MI — ~TT B »i+ (^-y ) w

-"01 = -i 122/1 + -T22J/2 — M2 — ~^" ±> rr

"p

i

T-T-

n

.

rr

~p

i

'Xl+(P

+js}x,

,,2

-"20 = ~-» 11 + Ml

,,2

-"02 — — •« 22 T P2 #30

— Ml —

~ PllfJ-2

~

#12 =

#03 = A*2 — 3P22/x2 #40 = Ml ~ 3PnPl 2

#31 = MlM2

o-

_ ,,2,,2

rr

3

#"22 — MlM2 3Pl 2 P 2 2

#"13 — PlM2 #04

— fJ-2 ~ 6P22M 2

Polynomials G

GOO = 1

1 2JS 1 2^ Expressions for the polynomials G 2 O j G n , . . . can be obtained from the corresponding expressions for Him by the change Ppq —>• Ppq , fj,p —>• yp. Before we pass to the straightforward calculation of the successive ap-

proximations f k , we make the following two remarks about some singular-

Approximate Synthesis of Stochastic Control Systems

159

ities of the series on the right-hand side of (3.2.19).

REMARK 3.2.1. In practice, the series (3.2.19) is usually replaced by a finite sum. The number of terms of the series (3.2.19) left in this sum is determined by the rate of convergence of the series (3.2.19). Here we do

not discuss this question (see, for example, [26, 166, 179]). However, in our case, the series (3.2.19) cannot be truncated in an arbitrary way, since it contains complex terms such as the polynomials Him and the coefficients

aklm (this follows from (3.2.14)-(3.2.18) and (3.2.20)). At the same time, the loss function /jt(£i, £2) represented by this series has the meaning of a

real function for real arguments. Therefore, truncating the series (3.2.19), we must remember that a finite sum of this series determines a real function

only if the last terms of this sum contain all terms with Him of a certain group (namely, all Htm with l + m = s, where s is the highest order of the

polynomials left in the sum (3.2.19)).

D

REMARK 3.2.2. Equation (3.2.7), as well as the corresponding equations of successive approximations (3.1.9) and (3.1.10), is centrally symmetric (such equations remain unchanged after the substitution (x\, £2) —>• (—£1, —£2))- Therefore, the series (3.2.19) must not contain terms for which the sum (I + TO) is odd, since the polynomials Htm with odd (I + TO) are

not centrally symmetric (see Table 3.2.1). If we take this fact into account, then the body of practical calculations is considerably reduced.

D

In what follows, we present the first two approximations calculated according to (3.1.9) and (3.1.10) and the quasioptimal control algorithms «o(£i,£2) and «i(£i,CB2) corresponding to these approximations. The zero approximation. First of all, let us calculate the parameter 7° of specific stationary losses in the zero approximation. From (3.1.56) with regard to c(x) = x\ + x\ and (3.2.10), we have 0

O

f

r^

II

/•OO

/

2

*) \

F

/3 /

9

9 \ "I

KB JJ_ 0 0 Calculating the integral, we obtain

7° = j.

(3.2.21)

In view of Remark 3.2.2, the first coefficients a°0 and a^ in the series (3.2.19) are equal to zero.8 The coefficients b^Q, b®^ and 6§2 can be calculated by using the formulas for G2o, GII, and G02 from Table 3.2.1 and 8

The same result can be obtained if we formally calculate the coefficients 6j0 and fejjj

by using (3.2.20).

160

Chapter III

(3.2.16). Then, according to (3.2.20), the coefficient b% has the form O<x\

=

f3

x exp [ - f (x2 + x2)} (7° - xl - x2,), (3.2.22) The integral in (3.2.22) can readily be calculated, thus, taking into account (3.2.21) and (3.2.14), we obtain

In a similar way, we can easily find

(3 2 24)

--

All other coefficients b®m with l-\-m > 2 are zero in view of the orthogonality

condition (3.1.49). According to (3.2.19), it follows from (3.2.23) and (3.2.24) that a° =--^-/^-^

(3.2.25) 02

_

**'

IP

,rV

Finally, using the formulas for H20, #11, and H02 from Table 3.2.1 and (3.2.25), we obtain the loss function in the zero approximation t I

\ _

Orj-

| a0 r r

JOv*!) 2-2J — a20-"20 i

ll-"ll +

| a0 r r

02"02

. Ix| _ -B(/? 2 +4)

(3.2.26) This relation and condition (3.2.9) imply the following equation for the zero-approximation switching line F0:

zi+z

2

= 0.

(3.2.27)

Approximate Synthesis of Stochastic Control Systems

161

In this case, the quasioptimal control algorithm UQ(X] in the zero approximation has the form

u0(x) = -g sign (xi + ^x2} .

(3.2.28)

REMARK 3.2.3. The loss function f o ( x i , x 2 ) in the zero approximation (without a constant term) and the parameter of stationary losses (3.2.21) can be calculated in a different way, without using the method considered above. Indeed, if we first seek the solution of the zero approximation equation (L is the operator (3.2.8)) Lf0 = 7° - xl - x\

(3.2.29)

as the quadratic form

fo(xi, x2) = with unknown coefficients /in, hi2, and h22, then, substituting this expression into (3.2.39), we obtain four equations for ha, /ii2, h22, and 7°. However, higher approximations cannot be obtained by this simple reasoning. D

The first approximation. It follows from (3.1.10) and (3.2.26) that in the first approximation we need to solve the equation

2x2 This equation can be solved by analogy with the zero-approximation equation (3.2.29), but the calculations are much more cumbersome due to more complicated expression on the right-hand side.

First, we employ (3.1.57) and (3.2.21) to find the specific stationary losses 1

R

cR

^

f r°°

//

9

f / 3 / 2

2\~l

T

then, after the integral is calculated, we obtain

The coefficients a\0, a\^... in (3.2.19) are calculated by (3.2.19) and (3.2.20) with regard to the formulas for Glm from Table 3.2.1. We omit

162

Chapter III

the intermediate calculations and write the final expression for /I(:EI, £2)Taking only the first terms in the series (3.2.19) up to the fourth order inclusively (that is, omitting the terms for which (I + m) > 4), we obtain the following expression for the loss function in the first approximation: = VX

\ + px\ + puxixi + pmx\ + const . (3.2.31) Here9

ea/3 25

P31 = -^5-,

Pii =

£CX. l+T,

_ ea B

_ ea/3 125

P22 = -=-,

_ ea 165

Pl3 = 77T^, 1

O^O!

^2 = ^ - ^ ,

P04 = T^, ,

/o i->\ — 1 /9

« = (7r/35)

(32.32)

.

The condition f32
(3.2.32). From (3.2.9) and (3.2.31) we obtain the following equation for the switch-

ing line F1 in the first approximation:

It follows from the continuity conditions that for small e the switching

line F1 is close to F° determined by Eq. (3.2.27). Therefore, if we set £ 2 = —(/3/2)xi in the terms of the order of e in (3.2.33), then we obtain errors of the order of e2 in the equation for F1. Using this fact and formulas (3.2.32) and (3.2.33), we arrive at the following expression with accuracy up to the terms of the order of O(e1}:

Figure 23 shows the position of the switching lines F° and F1 on the phase plane (EI, x%). The switching line (3.2.34) determines the quasioptimal control algorithm in the first approximation:

«i(i) = -esign L +

l+

£«a;i -

-Zi •

(3-2.35)

We do not calculate the coefficients v and p and the constant term "const" in (3.2.31) since they do not affect the position of the switching line and the control algorithm in the first approximation.

Approximate Synthesis of Stochastic Control Systems

163

— —e

W* = £

FIG. 23

FIG. 24 This algorithm can easily be implemented with the help of standard blocks of analog computers. The corresponding block diagram of a quasioptimal control system for damping of random oscillations is shown in Fig. 24, where 1 and 2 denote direct-current amplifiers with amplification factors

gg/32 " 45

164

Chapter III

In conclusion, we dwell on another statement that follows from the calculations of the first approximation.

Namely, all expressions with a

small parameter contain this parameter in the form of the product ea = £/^TT/3B, This statement concerns the loss function (3.2.31), the switching line (3.2.34), and the formula (3.2.30) for stationary specific losses, which can be written in the form

or, more briefly,

if the condition /?2
fact, by the parameter e/ ' ^JirfjB. If we recall that, by the conditions of problem (3.2.2), the parameter e determines the values of admissible control, then it turns out that this variable need not be small for the method of successive approximations to be efficient. Only the relation between the limits of the admissible control and the intensity of random perturbations B is important. All this confirms our assertion made at the beginning of this chapter that the method of successive approximations considered here is convenient

for solving problems with bounded controls when the intensity of random perturbations is large.

§3.3. Synthesis of quasioptimal controls in the case of correlated noises

Now we shall show how the method of successive approximations studied in this chapter can be used for constructing quasioptimal controls when random actions on the system are not white noises. Instead of the system shown in Pig. 13, we shall consider a stabilization system of a somewhat more general form (see Fig. 25), where in addition to random actions £(<) on the plant we also take into account the noise r)(t] in the feedback circuit.

Let the controlled plant P be described, just as in §3.1, by the system of linear differential equations with constant coefficients (3.3.1)

Approximate Synthesis of Stochastic Control Systems

165

y(t)

FIG. 25 where XT = (zi, . . ., xn), UT = (MI, . . . , ur), (,T (t) = (&(<), . . .,&«(<)), and the constant matrices vl, Q, and cr are of dimensions n x n, n x r, and

n x ?7i, respectively. Block 1 in Fig. 25 is assumed to be a linear inertialess device described by the equation

y(t) = Cx(t) + Dr](t),

(3.3.2)

where yT = (j/i, . . . , yi, r = (771, . . . , rjt), and C and D are constant matrices of dimensions I x n and I x I, respectively, (det D ^ 0). The goal of control is to minimize a functional of the form - E

c(x(t},u(t}}dt\.

(3.3.3)

We assume that the random perturbations £(<) and r](t) affecting the system are independent diffusion processes with drift coefficients

= Hr),

(3.3.4)

and matrices of local diffusion coefficients B^ and B^ (G and B^ are mxm dimensional constant matrices; H and Bn are £ x £ dimensional constant matrices; the matrices B£ and B^ are symmetric, 5^ is a nonnegative definite matrix and B^ is a positive definite matrix). It is well known that in this case the diffusion processes £(t) and rj(t) are Gaussian. The stated problem is a special case of the synthesis problem treated in

§1.5. This problem is characterized by the fact that the controlled process x(t) is not a Markov process (in contrast, say, with problems considered in §3.1 and §3.2; moreover, x(t) is a nonobservable process), and therefore, to describe the controlled system shown in Fig. 25, we need a special space Xt

Chapter III

166 of states. This space was called the space of sufficient (see

coordinates in §1.5

also [171]). As was shown in §1.5, in this case, as sufficient coordinates,

we must use a system of parameters that determine the current a posteriori probability density of nonobserved stochastic processes:

The a posteriori density (3.3.5) satisfies a stochastic partial differential equation, which is a special case of Eq. (1.5.39). It follows from §1.5 that to write an equation for the density (3.3.5), we need to use a priori probability characteristics of the (n + m + £)-dimensional stochastic Markov process It follows from (3.3.1), (3.3.2), and (3.3.4) that the combined process has the drift coefficients

ax = Ax + Qu + a!;, ay — Ayxx Ayyy +

CQu,

(3.3.6)

and the matrix of local variances

B-

0...0 0...0

0 . . . 00 . . . 0 0...0 0...0

0...0 0...0

0...00...0 0...0

0...0 0...0

0...0

0...0

0...0

(3.3.7)

0...0

B

The matrices introduced in (3.3.6) and (3.3.7) are Ay,

= CA-

Ayy

=

DHD-^,

C,

Ayt: = Ca, By = DBnDT.

(3.3.8)

10 In this case, the control u in (3.3.1) is assumed to be a given known vector at each time instant t.

Approximate Synthesis of Stochastic Control Systems

167

Using (3.3.6) and (3.3.7), we obtain the following equation for the a posteriori probability density (3.3.5):11

dp(t,z)

I

92p(t,z)

ot

2*

Ozoz

(

T D—1

—^—- = -Spg ; — — 1 dy SJV

2\

p C

'

OCLy

d ,

,.

' - Sp^-^(azp(t,z))

-,\ , F . T D - LI /

rp

_l

dy -\- op ——Tfr — C.psd-u &v

^y

_

\

+ \y* B (ay - cEpsay)

uz

r

c

Ody

\

/

*

^y '— ^~p» ^P Tj—^r J P\t) Zi.

^y

J

*'

/ O O O\

{o.o.y ]

Here p(i,z) = pt(x,^) denotes the a posteriori density (3.3.5), z denotes the vector (#,£), az is the vector composed of the vector-columns ax and ct£, the matrix Bz is a part of the matrix (3.3.7) consisting of its first (n + m) rows and columns, Eps denotes the a posteriori averaging of the corresponding expressions (that is, the integration with respect to z with the density p(t, z ) ) .

It follows from (3.3.6)-(3.3.8) that the matrix B2 is constant, the components of the vector az are linear functions of z, and the expression in the square brackets in (3.3.9) linearly and quadratically depends on z. Therefore, as shown in §1.5 (see also [170, 175]), the a posteriori density p(t, z)

satisfying (3.3.9) is Gaussian, that is, (t))TK-l(t)(z

-

- *(<))],

(3.3.10)

if the initial (a priori] density p(0, z) = po(z) is Gaussian (this is assumed in the sequel).

Substituting (3.3.10) into (3.3.9), one can obtain a system of differential equations for the parameters ~z and K~* of the a posteriori probability density (3.3.10). One can readily see that this system has the form

V [y-ay(z,y,u)},

K~l = -2K~1BZK-1 - R-1V - VTK~i - W

(3.3.11)

(3.3.12)

(in our special case, the system (1.5.52) acquires the form (3.3.11), (3.3.12)). If instead of K~[ we use the inverse matrix K (which is the matrix of a posteriori covariances), then the system (3.3.11), (3.3.12) can be written in the form

z = az(z, u) + fflB~\y T

- ay(z, y, u)],

K = 1BZ + VK + K V + KWK. n

(3.3.13) (3.3.14)

To derive (3.3.9) from (1.5.39), we need to recall that, according to the notation

used in (1.5.39), the vector Aa coincides with the vector a z , the vector A& with ay, and

the structure of the diffusion matrix (3.3.7) implies the following relations between the matrices: \\Ba/3\\ = Bz, \\Baa.\\ = 0, \\Fffp\\ = B~l, and ||B,p|| = By.

Chapter III

168 Here
= Ayxkx( A V = 0

a G

yrr _

~

I ^yx-°y AT R"1 ^y A

I

AT R-1 A

»£ »

y*

T

(3.3.15)

"1

AT f!~1A

st y

y

where, in turn, kxx, kx£,... are elements of the block covariance matrix K,

kxx

kxe

(the dimensions of a block are determined by the dimensions of its subscripts; for example, kx£ is of dimension n x TO). The loss function for problem (3.3.1)-(3.3.3)

F(t,zt,Kt) = min Eps

:(X(T),W(T))C!T y*

u(r) t
= min Eps

C X T

= z t , JiC(t) =

u(r) t
(3.3.16) is completely determined by the time instant t and the current values of the parameters ( z t , K f ) of the a posteriori density (3.3.10) at this instant of time. It follows from the definition given in §1.5 that (z(f), K(t)) are sufficient coordinates for problem (3.3.1)-(3.3.3). The Bellman equation (1.5.54) for the function (3.3.16) can readily be obtained in the standard way from the Eqs. (3.3.13), (3.3.14) for the sufficient coordinates. However, it should be noted that, in this case, the system

(3.3.13), (3.3.14) has a special feature that allows us to exclude the a posteriori covariance K(t) from sufficient coordinates. The point is that, in contrast, say, with a similar system (1.5.53), the matrix equation (3.3.14) is independent of controls u and in no way related to the system of differential equations (3.3.13) for the a posteriori means ?(<). This allows us first to solve the system (3.3.14) and calculate the matrix of a posteriori covariances K(t) in the form of a known function of time on the entire control

interval 0 < t < T (we solve (3.3.14) with the initial matrix K(0) = K0, where KQ is the covariance matrix of a priori probability density po(z)). If K(t) is assumed to be known, then in view of (3.3.8) and (3.3.15) we

can also assume that the matrix crz in (3.3.13) is a known function of time, crz(t), and the loss function (3.3.16) depends on the set (i,2t). Therefore,

Approximate Synthesis of Stochastic Control Systems

169

instead of Eq. (1.5.54) for the loss function F(t, 2), we have the Bellman equation of the form

= mn

[c(x,v.)N(z,K(t))dz\ v

\

(3.3.17) (here N(Z,K(t)} denotes the normal probability density (3.3.10) with the vector of mean values ~z and the covariance matrix K(t)).

Just as in §3.1 and §3.2, Eq. (3.3.17) becomes simpler if we consider the stationary operating conditions for the stabilization system shown in

Fig. 25. The stationary operating conditions established during a long operating time interval (which corresponds to large time t) can exist if only there exists a real symmetric nonnegative definite matrix Kf such that

K*WK* + K*VT + VK* + 1BZ = 0

(3.3.18)

and this constant matrix K* is an asymptotically stable solution of (3.3.14). Let us assume that this condition is satisfied. Denoting the mean "control losses" per unit time under the stationary operating conditions, as

usual, by 7, we can define the stationary loss function /(z) = Km [F(t, z) - -j(T - t)], T-KX>

for which, from (3.3.17), we derive the time-invariant Bellman equation

[

rt f

r

aj(z, w ) — + / c(x,u)N(z, K*) dz

In (3.3.19) (TZ> is the matrix az (see (3.3.8) and (3.3.15)), where kxx, k^x,... are the corresponding blocks of the matrix Kf determined by (3.3.18). In some cases, it is convenient to solve Eq. (3.3.19) by the method of successive approximations treated in §3.1 and §3.2. The following example shows how this method can be used. Let us consider the simplest version of

the synthesis problem (3.3.1)-(3.3.3) in which Eqs. (3.3.1), (3.3.2) contain scalar variables instead of vectors and matrices. In (3.3.3) we write the penalty function c(x, u) in the form

c(x,u) = x +e~lu2,

(3.3.20)

170

Chapter III

where e > 0 is a small parameter. From the "physical" viewpoint, this penalty function means that the control actions are penalized much more strongly than the deviations of the phase coordinate x(i) of the control system (3.3.1) from the equilibrium state x — 0. For simplicity, we set Q =
x = — ax -\- £ + u + — [ y — (h — a)x — £ — u • (3.3.21)

£, = —
I

^XX

ff

——

^QifoxX

I

^fox^

I

'

_-j

I

5

-^-,

%£, ——

\®>

' 9)*^x£

ffx = (h - a)kxx + kx^

(3.3.22)

Bn
In this case the Bellman equation (3.3.19) has the form

„..,, s

dxdt

+

4

* eeJ

=

,. (3.3.23,

Here the constants ax* and cr^« are ffx,

= (h- a)k*xx + fc* c ,


where the constant covariances k*x, k*x^ and fc|^ form the stationary solution of the system of differential equations (3.3.22). Passing to the new variables x± = (^/B^/

(3.3.24)

where r =
\+ f x\N(bx1,k*,)dx = f , (3.3.25) J J

Approximate Synthesis of Stochastic Control Systems

171

where b = &x* /\/B^. Taking into account the formulas r°°

I

x\N(bx-L,klx)dx

J — CO

°°

/

x[exp -00

1

--——(v-bx *KXX

V

_ ^rt T.L* ) +I 6' a . 1•*•| Il _ 2 F (\ _ - ^/ rL JT (3.3.26)

27T J -

and minimizing the expression in the square brackets, we obtain from (3.3.25) the optimal control for the stationary stabilization conditions:

uf(xi, x2) = ~~~(xii

a;2

)'

(3.3.27)

where the function /(KI,^) satisfies the nonlinear elliptic equation l, x 2 )

= 7 - "(xi) +

2

-

(3-3.28)

Equation (3.3.28) is similar to Eqs. (3.1.8) and (3.2.7), therefore, in this case, we can use the same method of approximate synthesis as in §3.1 and §3.2. Then the quasioptimal control U]t(xi,X2) in the fcth approximation is determined by the formula

uk(xi,x2) = - — -/^(x1,X2), 20

where the functions fk(xi,x2) approximations

k = 0,1,2,...,

(3.3.29)

satisfy the linear equations of successive

Lfk(xi,x2)=uk(xi,X2),

* = 0,1,2,...,

(3.3.30)

, x2) = U>Q(XI) = 7° -

In this case, the calculations of successive approximations fk(xi,x2) are completely similar to those discussed in §3.1 and §3.2. Therefore, here

Chapter III

172

we restrict our consideration to a brief description of the calculation of fk(xi,xz). We only dwell upon the distinctions in formulas. The operator (3.3.24) can be written in the form (3.1.5) if A = \\Aij\\\ and B = \\Bij\\^ in (3.1.5) are understood as the matrices

A=

1 1 1 1

—a 0

-g

The stationary density po(x) satisfying (3.1.14) has the form (3.1.15), and the matrices P and P~1, as one can readily see, have the form

P ~_ A ,/2

-afj,p

-a/ip -(r +

(3.3.31)

p/ > 1 , = a + g, v — a- g — r, and p = r + 2g). Using (3.3.31), we can find the matrix (see (3.1.35)) p[cf-(r + gY}

p(1a? - 2ar — gr) - g/j,2

(3.3.32)

as well as the matrix p/2gfj, l/2g

V =

(3.3.33)

By the similarity transform, this matrix reduces the matrix (3.3.32) to the diagonal form

\i 0

0 A2

= -a,

A 2 = -g.

(3.3.34)

It follows from (3.1.44), (3.1.51), and (3.1.55) that solutions of the equations of successive approximations (3.3.30) can be represented as the series (3.3.35)

where Him(xi,X2) are two-dimensional Hermitian polynomials calculated __ _ i __ _ -i by the formulas (3.2.17) with y = V x (the matrix V is inverse to (3.3.33)).

Approximate Synthesis of Stochastic Control Systems

173

The coefficients aktm are calculated by the formula (see (3.2.19))

fc

_ ——tfm——_——^m_ ) l\i + m\2 la + mg

(3.3.36)

and the coefficients

,,

00

I 0 /* X* det^P G/ m (/i) ^=T7Tpx ex? [ - ^(xTPx)]uk(x) dXldx2 !m, JJ

(3.3.37) are expressed in terms of the group of Hermitian polynomials Gim(x\, x-z) orthogonal to Him(xi, x-i) and calculated by (3.2.18). Parallel to the calculations of the successive approximations to the loss

function (3.3.35), we calculate specific stationary losses -yfc (corresponding to the Mh approximation) from the condition &QO = 0. So, in the zero approximation we have

hence, performing simple calculations and taking into account (3.3.31), we obtain

g) 2

Next, using the obtained value of 7° and formulas (3.3.26), (3.3.30), (3.3.36), and (3,3.27), we can calculate any desired number of coefficients a°m in the series (3.3.35). With the help of these coefficients, we can construct an approximate expression for the function fo(xi,X2), which allows

us to derive an explicit formula for the quasioptimal control algorithm UQ(XI,X^) in the zero approximation and to calculate the variables -y1, /i (£1,2:2), and MI (3:1,2:2) related to the first approximation. Here we write explicit formulas neither for fo(xi, Xi) nor for /i(a;i, £2), since they are very cumbersome. We only remark that in this case all quasioptimal control algorithms (3.3.29) are nonlinear functions of the phase variables (2:1,2:2); moreover, the character of nonlinearity is determined by

the number of terms left in the series (3.3.35) for the calculations.

Chapter III

174

Thus, from the preceding it follows that the methods for calculations of stationary operating conditions of the stabilization system (Fig. 13) can readily be generalized to the case of a more general system with correlated noise (Fig. 25) if the noise is a Gaussian Markov process. In this case, the

optimal system is characterized by the appearance of an optimal filter in the regulator circuit; this filter is responsible for the formation of sufficient coordinates. In our example (Fig. 25), where x, y, u, £, and f] are scalar, this filter is described by Eqs. (3.3.21). The circuit of functional elements of this closed-loop control system is shown in Fig. 26.

f

——>

" 1 ' -, >

—J

s(i )

Jo

"NT1

M

x(<)

W -*

t( t) ^\

3

Zi

<—

y

5^

Jo

,_ 4<

6

<

7<

f—

d/dt

'

8-

y(*)

!>?(* x

«—

FIG. 26 Blocks P and 1 are units of the initial block diagram (Fig. 25). The rest of the diagram in Fig. 26 determines the structure of the optimal controller. One can see that this controller contains standard linear elements of analog computers such as integrators, amplifiers, adders, etc. and one nonlinear converter NC, which implements the functional dependence (3.3.29).

Units of the diagram marked by ">" and having the numbers 1,2,... ,8 are amplifiers with the following amplification factors Kt:

K5 = -1,

K6 = a - h,

Kr = -1,

K& = h.

Approximate Synthesis of Stochastic Control Systems

175

§3.4. Nonstationary problems. Estimates of the quality of approximate synthesis

3.4.1. Nonstationary synthesis problems. If equations of a plant

are time-dependent or if the operating time T of a system is bounded, then the optimal control algorithm is essentially time-varying, and we cannot find this algorithm by using the methods considered in §§3.1-3.3. In this case, to synthesize an optimal system, it is necessary to solve a time-varying Bellman equation, which, in general, is a more complicated problem. However, if the plant is governed by a system of linear (time-varying) equations, then we can readily write solutions of the successive approximation equa-

tions (3.0.6), (3.0.7) in quadratures. Let us show how this is done. Just as in §3.1,

problem for the stabilization system (Fig.

we consider the synthesis

13) with a plant P described by

equations of the form

x = A(i)x + Q(t)u

+ ar(t)£(t),

(3.4.1)

where x is an n-dimensional vector of phase coordinates, u is an r-dimensional vector of controls, A(t), Q(t), and
n-dimensional standard white noise (1.1.34). To estimate the quality of control, we shall use the following criterion of the type of (1.1.13): (3.4.2)

and assume that the absolute values of the components of the control vector u are bounded by small values (see (3.1.3)):

K <£umi,

i= l,...,r.

(3.4.3)

According to (3.1.6) and (3.1.7), the optimal control u*(t,x) for problem (3.4.1)-(3.4.3) is given by the formula u,(t,x) = -{ew m i,...,£« mr }sign ( QT(t)—-(t,x) ) , \ dx / where the loss function F(t,

(3.4.4)

x) satisfies the equation

LtiXF(t, x) = -c(x)

- £$ (t, —— (<, x)}, \ ox )

(3.4.5)

176

Chapter III

with Lt,x denoting a linear parabolic operator of the form

—+-1

d

fit* L/tt-

*} LJ

For the function 3>(t,dF/dx], we have the expression (3.4.7)

In this case, the function F(t,x) must satisfy (3.4.5) for all x £ R n , 0 < t < T, and be a continuous continuation of the function (3.4.8)

as t -> T (see (1.4.22)). The nonlinear equation (3.4.5) is similar to (3.0.5) and, according to (3.0.6) and (3.0.7), can be solved by the method of successive approximations. To this end, we need to solve the sequence of linear equations

Lt,,F0(t,x) = -c(x),

Lt,,Fk(t, x) = -c(x)

(3.4.9)

- e$ (t, ^±(t, x)}, dx V >

k = 1, 2, . . . (3.4.10)

(all functions Fk(t,x) determined by (3.4.9) and (3.4.10) must satisfy condition (3.4.8)). Next, if we take Fk(t,z) as an approximate solution of Eq. (3.4.5) and substitute Fk into (3.4.4) instead of F, we obtain a quasioptimal control algorithm Uk(t, x) in the fcth approximation. Let us write the solutions Fj. (i, a;), k = 0, 1, 2, . . . , in quadratures. First, let us consider Eq. (3.4.9). Obviously, its solution Fo(t,x) is equal to the value of the cost functional

c(x(T))dT+i>(x(T))

x(t)=x

(3.4.11)

on the time interval [t, T] provided that there are no control actions. In this case, the functional on the right-hand side of (3.4.11) is calculated along the trajectories x(r), t < r < T, that are solutions of the system of stochastic

differential equations

X = A(T)X

+
x ( t ) = x,

describing the uncontrolled motion of the plant (u = 0 in (3.4.1)).

(3.4.12)

Approximate Synthesis of Stochastic Control Systems

177

It follows from §1.1 and §1.2 that the solution of (3.4.12) is a continuous Markov process X(T) (a diffusion process). This process is completely determined by the transitive probability density function p(x, t; z, T), which determines the probability density of the random variable z = X(T) if the stochastic process x(t) was in the state x(f) — x at the preceding time moment t. Obviously, by using p(x,t;z,r), we can write the functional (3.4.11) in the form

FQ(t,x)= f

dr I

Jt

p(x,t-z,T)c(z)dz+

Jnn

f

p(x,t;z,T)4>(z)dz.

(3.4.13)

JRn

On the other hand, we can write the transitive density p(x, t; z, T) for the diffusion process X(T) (3.4.12) as an explicit finite formula if we know the

fundamental matrix X(t, T) for the nonperturbed (deterministic) system z = A(t)z. Indeed, since Eqs. (3.4.12) are linear, the stochastic process X(T) satisfy-

ing this equation is Markov and Gaussian. Therefore, for this process, the transitive probability density has the form

p(x,t;z,T) = [(2ir)B det DJ-^expI-f (z - a)TD~l(z - a)], where a = Ez = E(X(T)

(3.4.14)

x ( t ] = a;) is the vector of mean values and

D = E[(z — Ez)(z — Ez) T ] is the covariance (dispersion) matrix of the random vector z = X(T). On the other hand, using the fundamental matrix X(t,T)12 we can write the solution of system (3.4.12) in the form (the Cauchy formula)

x(r) = X(t,r)x+ f Jt

X(s,T)
Hence, performing the averaging and taking into account properties of the white noise (1.1.34), we obtain the following expressions for the vector a

and the matrix D:

a = Ex(r)=X(t,T)x,

(3.4.15) T T

D = E [ ( x ( r ) - a))((xx((rr))--aa)) ]

!

Jt

X(s,T)o-(s)Et(s)£T(S')o-T(s')XT(s',T)dsds'

12

Recall that the fundamental matrix X(t,r),T > t, is a nondegenerate nxn matrix

whose columns are linearly independent solutions of the system Z ( T ) = A(T)Z(T), so

that X(t, t) = E, where E is the identity matrix. Methods for constructing fundamental matrices and their properties are briefly described on page 101 (for details, see [62, 111]).

178

Chapter III X(s,T)cr(s)6(s'

-s)
X(s,T)B(s)XT(s,T)ds,

=

B(s)=a(s}ffT(s).

J*

(3.4.16)

Formulas (3.4.13)-(3.4.16) determine the solution F0(t,x) of the zeroapproximation equation (3.4.9), satisfying (3.4.8), in quadratures. It follows from (3.4.13)-(3.4.16) that the function F0(t,x) is infinitely many times differentiable with respect to the components of the vector x if the functions c(z) and ij)(z) belong to a rather wide class (it suffices that the functions c ( z ) exp(— |z T D~ 1 z) and ^>(z) exp(— ^zTD~iz) were absolutely integrable [25]). Therefore, by analogy with (3.4.13), we can write the solution Fk(t,x) of the successive approximation equations (3.4.10), satisfying

(3.4.8), in the form

Fk(t,x)=

dr Jt

JRn

= l,2,...

(3.4.17)

To obtain explicit formulas for the functions Fo(t, x), Fi(t, x),. .., which allow us to write the quasioptimal control algorithms Uo(t, x), «i(<, a;), ...

as finite analytic formulas, we need to have the analytic expression of the matrix X(t,r) and to calculate the integrals in (3.4.13) and (3.4.17). For autonomous plants (the case where the matrix A(t) in (3.4.1) and (3.4.12) is constant, A(t) = A = const), the fundamental matrix X(t,r) has the form of a matrix exponential:

X(t,r) = e A ( r -*>

(3.4.18)

whose elements can be calculated by standard methods. On the other hand, it is well known that fundamental matrices of nonautonomous systems can be constructed, as a rule, by numerical methods.13 Thus for A(t) ^ const, it is often difficult to obtain analytical results. If the plant equation (3.4.1) contains a constant matrix A(i) EE A —

const, then formulas (3.4.13) and (3.4.17) allow us to generalize the results obtained in §§3.1-3.3 for the stationary operating conditions to the timevarying case. For example, let us consider a time-varying version of the

problem of optimal damping of random oscillations studied in §3.2. 13 Examples of special matrices A(t) for which the fundamental matrix of the system x = A(t)x can be calculated analytically, can be found, e.g., in [139].

Approximate Synthesis of Stochastic Control Systems

179

Just as in §3.2, we shall consider the optimal control problem (3.2.1)(3.2.3). However, in contrast with §3.2, we now assume that the terminal time (the upper limit T of integration in the functional (3.2.3)) is a finite

fixed value. By writing the plant equation (3.2.1) in the form of system (3.2.4), we see that problem (3.2.1)-(3.2.3) is a special case of problem

(3.4.1)-(3.4.3) if

0

4= a(t) = a =

0

1

Q(t) = Q = 1

-i -p

0

0

C(x) =

o VB

2 2 Xl+X2,

i>(3

(3.4.19) = 0.

Therefore, it follows from the general scheme (3.4.4)-(3.4.10) that in this case the optimal control has the form

u#(t,xi,x2) = -esign f - — ( t , x i , x 2 ) 1,

(3.4.20)

where for 0 < t < T the function F(t,xi,x2) satisfies the equation

Lt>xF(t, xi, x2) = -x\-

(3.4.21)

0z 2 V "

and vanished at the terminal point, that is,

F(T,X!,x2) = Q.

(3.4.22)

According to (3.4.6) and (3.4.13), the operator Lt,x in (3.4.21) has the form

d

d Lt,x = ^T

8

B d2 2 dx\

-—— + — ——r.

dx2

Let us calculate the loss function Fg(t, xi, x2) of the zero approximation. In view of (3.4.9), (3.4.21), and (3.4.22), this function satisfies the linear equation

with the boundary condition

F0(T,xi,x2) = Q.

(3.4.24)

According to (3.4.13), the function F0(t,Xi,x2) can be written in quadratures ,-T

F0(t,xi,x2)=

dr Jt

(3.4.25)

Chapter III

180

where the transition probability density p(x,t;z, T) is given by (3.4.14). It follows from (3.4.15) and (3.4.16) that to find the parameters of the transition density we need to calculate the fundamental matrix (3.4.18). Obviously, the roots AI and A2 of the characteristic equation dei(A — \E) = 0 of the matrix A given by (3.4.19) are

Ai,2 = --

From this and the Lagrange-Silvester formula [62] we obtain the following expression for the fundamental matrix (3.4.18) (here p = (T — t ) ) :

-(A-X1E) -(A-X2E) (Ai-A2 (A2• sin Sp + 6 cos 6p sin 6p — sin 6p 6 cos Sp — | sin 6p (3.4.26)

X(t,r) =

It follows from (3.4.15), (3.4.16), and (3.4.26) that in this case the vector of means a and the variance matrix D of the transitive probability density (3.4.14) have the form

x\ cosSp + |(a:2 + f

(3.4.27)

On

D12

_B

Di 2 B

D 22

25' 6

B


(3.4.28)

Substituting (3.4.14) into (3.4.25) instead ofp(x,t;z,r), integrating, taking into account (3.4.27) and (3.4.28), and performing some easy calcula-

Approximate Synthesis of Stochastic Control Systems

181

tions, we obtain the following final expression for the function Fo(t, x^, x2):

xi

-l-

B(/32+4:); ^2

—e-^sm2S-p(B - 2(3x1 - 4xlX2) 80

(3.4.29)

"? cos 2 W/32 - 4)z? + 40*1*2 + 4zi - 5/3], where ~p = T — t. Let us briefly discuss formula (3.4.29). If we consider the terms on the right-hand side of (3.4.29) as function of "reverse" time ~p — T — t, then these terms can be divided into three groups: infinitely increasing, damping, and independent of ~p as ~p —> oo. These three types of terms have the following physical meaning. The only infinitely growing term (B/(3)~p in (3.4.29) shows how the mean losses (3.4.11) depend on the operating time in the mode of stationary operating conditions. Therefore, the coefficient B//3 has the meaning of the specific mean error 7, which was calculated in §3.2 by other methods and for which we obtained 7° = B//3 in the zero approximation (see (3.2.21)). Next, the terms independent of ~p (in the braces in (3.4.29)) coincide with the expression for the stationary loss function obtained in §3.2 (formula (3.2.26)). Finally, the damping terms in (3.4.29) characterize the deviations of operating conditions of the control system from the stationary ones. Using (3.4.29), we can approximately synthesize the optimal system in the zero approximation, where the control algorithm uo(t, Xi,x2) has the form (3.4.20) with F replaced by FQ. The equation

8F0

=

2x2

— I"

I

2x

+

2cos 2Sp(/3xi + x2) = 0

(3.4.30)

determines the switching line on the phase plane (xi, x 2 ). Formula (3.4.30) shows that this is a straight line coinciding with the K-axis as ~p —)• 0 and rotating clockwise as ~p -> oo (see Fig. 27) till the limit value xi + 2x2/(3 — 0 corresponding to the stationary switching line (see (3.2.27)). Formulas (3.4.29) and (3.4.30) also allow us to estimate whether it is important to take into account the fact that the control algorithm is timevarying. Indeed, (3.4.29) and (3.4.30) show that deviations from the stationary operating conditions are observed only on the time interval lying at

Chapter III

182

FIG. 27 the distance ~ (3~l from the terminal time T. Thus, if the general operating time T is substantially larger than this interval (say, T 3> 3//3), then we can

use the stationary algorithm on the entire interval [0,T], since in this case the value of the optimality criterion (3.2.3) does not practically differ from the optimal value. This fact is important for the practical implementation of optimal systems, since the design of regulators with varying parameters is a rather sophisticated technical problem. 3.4.2. Estimates of the approximate synthesis performance. Up to this point in the present chapter, we have studied the problem of how to find a control syste close to the optimal one by using the method of successive approximations. In this section we shall consider the problem of how the quasioptimal system constructed in this way is close to the optimal

system, that is, the problem of approximate synthesis performance. Let us estimate the approximate synthesis performance for the first two (the zero and the first) approximations calculated by (3.0.6)-(3.0.8). As an example, we use the time-varying problem. (3.4.1)-(3.4.3). We assume that

the entries of the matrices A(t), Q(t), and cr(i) in (3.4.1) are continuous functions of time defined on the interval 0 < t < T. We also assume that the penalty functions c(x) and i}>(x) in (3.4.2) are continuous and bounded for all x £ R n . Then [124] there exists a unique function F(t, x) that satisfies

the Cauchy problem (3.4.5), (3.4.8) for the quasilinear parabolic equation (3.4.5)14 This function is continuous in the strip HT = {\x\ < oo, 0 < t < T} 14

We shall use the following terminology: Eq. (3.4.5) is called a quasilinear (semi-

linear) parabolic equation, the problem of solving Eq. (3.4.5) with the boundary condi-

Approximate Synthesis of Stochastic Control Systems

183

and continuously differentiable once with respect to t and twice with respect to x for 0 < t < T; its first- and second-order derivatives with respect to x are bounded for x 6 HTOne can readily see that in this case

\F(t,x)-F1(t,x)\~e2,

\F(t,x)-F0(t,x)\~e,

(3.4.31)

and hence, for small £, the functions F0(t, x) and Fi(t, x) nicely approximate

the exact solution of Eq. (3.4.5). To prove relations (3.4.31), let us consider the functions So(t,x) = F(t, x) - F0(t, x) and Si(t, x) = F(t, x) - Fi(t, x). It follows from (3.4.5), (3.4.9), and (3.4.10) that these functions satisfy the equations

LS0 = -e* (t, ^-} ,

S0(T, x) = 0,

(3.4.32)

(3.4.33)

Equations (3.4.32) and (3.4.33) differ from (3.4.9) only by the expressions on the right-hand sides and by the initial data. Therefore, according to

(3.4.13), the functions So and Si can be written in the form T

/»

/)/?

dr /

/

S1(t,x)=ef Jt

\

p(x,t;z,T)$ (r,—— (T,Z) ) dz, v

jRn

az

(3.4.34)

'

dr f p (z,i;z,r)[*(r,^)-*(T,^)]dz. v Jnn L v Oz ' J (3 4 35)

Since the function $ is continuous (see (3.4.7)) and the components of the vector dF/dx are bounded, we have |$(T, dF/dz)\ < P for all r 6 [0, T]; hence, we have the estimate

\S0(t ,x)\<e

dr I Jt

p(x,t;z,T)$

dz,

jRn

<eP Ji

dr I p(x,t;z,T)dz = •/R«

(3.4.36)

tion (3.4.8) is called the Cauchy problem, and the boundary condition (3.4.8) itself is sometimes called the "initial" condition for the Cauchy problem (3.4.5), (3.4.8). This terminology corresponds to the universally accepted standards [61, 124] if (as we shall do in §3.5) in Eq. (3.4.5) we perform a change of variables and use the "reverse" time p = T — t instead of t. In this case, the backward parabolic equation (3.4.5) becomes a "usual" parabolic equation, and the boundary value problem (3.4.5), (3.4.8) takes the form of the standard Cauchy problem.

Chapter III

184

The first relation in (3.4.31) is thereby proved. To prove the second relation in (3.4.31), we need to estimate the difference S'0 = (dF/dxi) - (dF0/dxi). To this end, we differentiate (3.4.32) with respect to x;. As a result, we obtain the following equation for the function SQ: (3.4.37)

(in fact, the derivative on the right-hand side of (3.4.37) is formal, since the

function $ (3.4.7) is not differentiable). Using (3.4.13) for SQ, we obtain T

C

dr

(3.4.38)

JRn

Integrating (3.4.38) by parts with respect to z,- and taking into account (3.4.14) and (3.4.15), we arrive at

S'0 = £ f

Jt

dr I

Jnn

p(x,t;z,r)D-j1(zj

-Xjk(t,r)xk}$(r,

-f) dz. (3.4.39)

\ oz /

From (3.4.39) we obtain the following estimate for S10:

\So\ < £P I Jt

dr

I ^i? \zi ~ XJk(t, r)xk\p(x, t; z, T) dz = ePVi, Jnn .

(3.4.40)

J=l Now we note that since Q(i) in (3.4.7) is bounded, the function $(<, satisfies the Lipschitz condition with respect to y:

(3.4.41) Using (3.4.40), (3.4.41), and (3.4.35), we obtain dz <sN f Jt

dr f

p(x,t;z,T}y^\S*0\dz

JRn

i=1

n

<£2NPV(T-t),

V

(3.4.42) 1=1

Approximate Synthesis of Stochastic Control Systems

185

which proves the second relation in (3.4.31). In a similar way, we can also estimate the difference dF/dxi — dFi/dxt = S{. Indeed, just as (3.4.39) was obtained from (3.4.32), we use (3.4.33) to obtain

S{=e ?{=ef Jt

dr dr f

p(x,t-,z,T)D(z-Zkxk)*T,-*T,

dz.

JRn

(3.4.43) This relation and (3.4.40), (3.4.41) for the function S{ readily yield the estimate f> 17

|S1I =

fl Z7

< £2PVViN,

(3.4.44)

which we shall use later. According to (3.0.8), in this case the quasioptimal controls uo(t, x) and MI(<, x) are determined by (3.4.4), where instead of the loss function F(t, x) we use the successive approximations F0(x,t) and Fi(x,t), respectively. By Go(t, a:) and Gi(x,t) we denote the mean values of the functional (3.4.11) calculated on the trajectories of the system (3.4.1)

x = A(T)X + Q(T)MJ(T, a;) +
v(+\ —T \ I — 5

i — 0 1 — ' '

with the use of the quasioptimal controls uo(t, x) and u-\_(t, x). The functions Gi(t, a;), i — 0, 1, estimate the performance of the quasioptimal control algorithms Wi(<, x), i = 0,1. Therefore, it is clear that the approximate synthesis may be considered to be justified if there is only a small difference between the performance criteria Go(t, x) and Gi(i, x) of the suboptimal systems and the exact solution F(t, x} of Eq. (3.4.5) with the initial condition (3.4.8). One can readily see that the functions GO and GI satisfy estimates of type (3.4.31), that is,

\F(t1x)-G0(t1x)\~£,

\F(t,x)-G1(t,x)\~e2.

(3.4.45)

Relations (3.4.45) can be proved by analogy with (3.4.31). Indeed, the functions GO and GI satisfy the linear partial differential equations [45, 157]

LGi(t,x) = -c(x)-euf(t,x)QT(t)-(t,x),

(3.4.46)

Chapter III

186

This fact and (3.4.9), (3.4.10) imply the following equations for the functions H0 = F0 - Go and HI = FI - Gj.:

H0(T, x) = 0,

LH0 =

LH^^W^-*(t^}\,

(3.4.47)

F1(r,,) = o.

(3.4.48)

Since wf Q T '^j£- = $(*, ^")> Eq. (3.4.48) can be rewritten as follows:

(3.4.49) It follows from (3.4.4) that Eqs. (3.1.46), (3.4.49) are linear parabolic equations with discontinuous coefficients. Such equations were studied in [80, 81, 144]. It was shown that if, just as in our case, the coefficients in (3.1.46), (3.1.49) have discontinuities of the first kind, then, under our assumptions about the properties of A(t), Q(t), c ( x ) , and ip(x), the solutions of Eqs. (3.4.46), (3.4.49) and their first-order partial derivatives are bounded.

Using this fact, we can readily verify that the right-hand sides of (3.4.47) and (3.4.49) are of the order of e and e 2 , respectively. For Eq. (3.4.47), this statement readily follows from the boundedness of the components of the

vectors dGo/dx and MO and the elements of the matrix Q. The right-hand side of (3.4.49) can be estimated by the Lipschitz condition (3.4.41) and the inequality

dFi

dF0

8F

dF

which follows from (3.4.40) and (3.4.44). Therefore, for the functions H0 and HI we have

\HO\

•e2.

(3.4.50)

To prove (3.4.45), it suffices to take into account the inequalities |F-Go

FO-GO\< F- FO\ + \HO\,

\F-Gi and to use (3.4.31) and (3.4.50). Thus, relations (3.4.45) show that if the Bellman equation contains a small parameter in nonlinear terms, then the difference between the quasioptimal control system calculated by (3.0.6)-(3.0.8) and the optimal control system is small and, for sufficiently small e, we can restrict our calculations to a small number of approximations. We need either one (the zero)

Approximate Synthesis of Stochastic Control Systems

187

or two (the zero and the first) approximations. This depends on the admissible deviation of the quasioptimal system performance criteria Gi(t, x) from the loss function F(t,x). In conclusion, we make two remarks about (3.4.45).

REMARK 3.4.1. One can readily see that all arguments that lead to the estimates (3.4.45) remain valid for any types of nonlinear functions in (3.4.5) that satisfy the Lipschitz condition (3.4.41). Therefore, in particular, all statements proved above for the function <S> (3.4.7) automatically hold for equations of the form (3.0.4) with r-dimensional ball taken as the set U of admissible controls, instead of an r-dimensional parallelepiped. D REMARK 3.4.2. The estimates of the approximate synthesis accuracy considered in this section are based on the assumption that the solutions of the Bellman equation and their first-order partial derivatives are bounded. At first glance it would seem that this assumption substantially narrows the class of problems for which the approximate synthesis procedure (3.0.6)(3.0.8) can be justified. Indeed, the solutions of Eqs. (3.4.5), (3.4.9), (3.4.10), and (3.4.46) are unbounded for any x G Rn if the functions c(x) and i>(x) infinitely grow as x —>• oo. Therefore, for example, we must eliminate frequently used quadratic penalty functions from consideration. However, if we are interested in the solution of the synthesis problem in a given bounded region XQ of initial states x(0) of the control system, then the procedure (3.0.6)-(3.0.8) can also be used in the case of unbounded penalty functions. This statement is based on the following heuristic arguments. Since the plant equation (3.4.1) is linear and the matrices A(t), Q(t), and ff(t) and the control vector u are bounded, we can always choose a sufficiently large number R such that the probability

P{sup 0 < t < T \x(t)\ > R} becomes arbitrary small [11, 45, 157] for any fixed domain XQ of the initial states x ( Q ) . Therefore, without loss of accuracy, we can replace the unbounded functions c(x) and i{>(x) in (3.4.2) (if, in a certain sense, these functions grow as x\ — R —> oo slower than the probability P{sup 0 < t < T x(i)\ > R} decreases as R —>• oo) by the expressions

c(x] maXj; c(x) \x\ = R,

for for for

ij)(x)

for

\x\ < R, \x\ > R, x

< R, > R,

\x\ = R, for which the solutions of Eqs. (3.4.5), (3.4.9), (3.4.10), and (3.4.46) satisfy the boundedness assumptions. D

188

Chapter III

The question of whether procedure (3.0.6)-(3.0.8) can be used for solving the synthesis problems with unbounded functions c(x) and i/>(x) in the functional (3.4.2) will be rigorously examined in the next section. §3.5. Analysis of the asymptotic convergence of

successive approximations (3.0.6)— (3.0.8) as k —>• oo

The method of successive approximations (3.0.6)-(3.0.8) can also be used for the synthesis of quasioptimal control systems if the Bellman equation

does not contain a small parameter in nonlinear terms. Needless to say that (in contrast with Section 3.4.2 in §3.4) the first two approximations, as a rule, do not approximate the exact solution of the synthesis problem sufficiently well. We only hope that the suboptimal system synthesized on

the basis of (3.0.9) is close to the optimal system for large k. Therefore, we need to investigate the asymptotic behavior as k —>• oo of the functions

Fk(t,x) and U k ( t , x ) in (3.0.6)-(3.0.8). The present section deals with this problem. Let us consider the time-varying synthesis problem of the form (3.4.1)™ (3.4.3) in a more general setting. We assume that the plant is described by

the vector-matrix stochastic differential equation of the form

x = a(t, x) + q(t)u + a(t, x ) £ ( t ) .

(3.5.1)

Here x is an n-dimensional vector of phase coordinates of the system, u is an r-dimensional vector of controls, £(t) is an n-dimensional vector of random actions of the standard white noise type (1.1.34), a(t, x) is a given vectorfunction of the phase coordinates x and time t, and q(t) and
such that for t > t0, 0 < t0 < T, the stochastic equation (3.5.1) has a unique solution x(t) satisfying the condition x(t0) = XQ at least in the weak sense (see §IV.4 in [132]). As an optimality criterion, we take the functional (3.4.2),

U

T

-I

c(x(t))dt + il>(x(T))\. J

(3.5.2)

Here c ( x ) and ij)(x) are given nonnegative scalar penalty functions whose

special form is determined by the character of the problem considered (the requirements on c ( x ) and ij)(x) are given later). The constraints on the domain of admissible controls have the form

(1.1.22), weC/,

(3.5.3)

Approximate Synthesis of Stochastic Control Systems

189

where U C Rr is a closed bounded convex set in the Euclidean space R r . It is required to find a function it, = ut(t,x(t)} satisfying (3.5.3) such that the functional (3.5.2) calculated on the trajectories of system (3.5.1) with the control M* attains its minimum value.

In accordance with the dynamic programming approach, solving this problem is equivalent to solving the Bellman equation that, for problem

(3.5.1)-(3.5.3), reads (see §1.4)

Here a(t, x] is a column of functions with components (see (1.2.48))

-—-^ami, Z Oxm

t=l,...,n.

(3.5.5)

Recall that we assumed in §1.2 that throughout this book all stochastic differential equations written (just as (3.5.1)) in the Langevin form [127] are symmetrized [174].

By definition, the loss function F in (3.5.4) is equal to

F = F(t,x) = min u(T)£U

Here E[(-)

T ( r r E-H / C(X(T))

dr

llJt

x(t) = x] means averaging over all possible realizations of the

controlled stochastic process x(r) =; XU^(T) (r > t) issued from the point

x at T = t. It follows from (3.5.6) that

F(T,x) = j(x).

(3.5.7)

Passing to the "reverse" time p = T — t, we transform Eq. (3.5.4) and the condition (3.5.7) to the form

LF(p,x) = -c(x) - mm \uTqT(p)~(p, x ] \ , F(0,z) = '0(z).

In (3.5.8) we have the following notation:

(3.5.8) (3.5.9)

190

Chapter III

ai(p,x) = a,i(x,T — p), q(p) =q(T — p), b t j ( p , x ) is a general element of the matrix |
x

c(x).

(3.5.11)

In addition, we obtain the function

u*(p,x) =
(3.5.12)

that satisfies the condition T T, ^^F

dF

~ and solves the synthesis problem (after we have solved Eq. (3.5.11) with

the initial condition (3.5.9)). The form of the functions